From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5100743006; Tue, 8 Aug 2023 09:07:35 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1DB0B43247; Tue, 8 Aug 2023 09:07:35 +0200 (CEST) Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) by mails.dpdk.org (Postfix) with ESMTP id 778CF40A87 for ; Tue, 8 Aug 2023 09:07:32 +0200 (CEST) Received: by mail-lf1-f51.google.com with SMTP id 2adb3069b0e04-4fe28f92d8eso8378938e87.1 for ; Tue, 08 Aug 2023 00:07:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1691478452; x=1692083252; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=kCbsdZ08fOMmYA0OR+Lbb2oC3DIiLJapdg04s55k58w=; b=U4vTkK+qtc4uM8PKcsTRdQFI9r3d1IGz74lWvgtU3pYmlZC8TghbYVfp1XCZegihKh B1rcUnuqkcLFy7Hy53ViRLBICruejOuqKeJc403OhXXZKzyzq+IzwJFyUKzD7fo71bKP gG0+xy74nWvw5iCoH/VhfHb4bGhqUO2rS+hpK19aReICJZ1eIz6PoBG+jNvtqr9pdsQi MXTMNc50LHv/G2qXuX8TtEbyCAs5sCR3MQDFAXFdJ2U8PnX986rLgqSDjyKwJ9EnKncz bMJXXDIwSuAOnYlPhp+uR2PPce/QU/NeKGPskXcixAhiNP9aeYb6S9X4HUetnXmnYspX K08A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691478452; x=1692083252; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=kCbsdZ08fOMmYA0OR+Lbb2oC3DIiLJapdg04s55k58w=; b=Croy7Kgf29eHcaWtV1AiCrlXYOW54ZmvsCmttQ5B/6mVhLIlBd4AypoqRTKsN0wA1f UeiEs621Te3HNtAxaxpIVHl/AEgdOtgRuhdQy6xjP3WTL9oBHaFquvNQQ4jFHxnkVWPy p0jjMfLeA5eSIb/1oqrhnSepipMIsiOjO+zi4tuT6pxBmfsohUDW+uOun6/Mmifk9LYV 5MVMf2nQ3PZVO3ZgpO2Lc+T6AaQy5EUMZFV8z3IEd4zI0BrgmzC42mekJizi9e/JYcyw kAAyyVNJ/3j/TuJatm63EIekhuStvkbGvbtIn68oV5L6EFmOrws6CYTx6JsbVlKepUy/ 41Kw== X-Gm-Message-State: AOJu0Yz1VV6hxmGbNZGTaELZzFcXZE9Fpinmifqe6u5eVFVfX8F/obiW xAoyKZmld+NqgpdQE4qK1fd/qiARKRS7a4EdLy+xosSF0ldjpAKlLcmvvw== X-Google-Smtp-Source: AGHT+IGG0rFN67R5z5bM1aMe1/YoYkPJpXrckweiYhTUCv31RQoqbcJKHd9ERWfvHIXXwbQ2WI3ciTqVVSAx6lLANoQ= X-Received: by 2002:a05:6512:3256:b0:4f8:7124:6803 with SMTP id c22-20020a056512325600b004f871246803mr6398957lfr.35.1691478451854; Tue, 08 Aug 2023 00:07:31 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?Q?Juraj_Linke=C5=A1?= Date: Tue, 8 Aug 2023 09:07:20 +0200 Message-ID: Subject: Re: Intel QAT 8970 accel card on ARM Ampere Server To: Ruifeng Wang Cc: Patrick Robb , Honnappa Nagarahalli , Dharmik Jayesh Thakkar , "ci@dpdk.org" , nd Content-Type: multipart/alternative; boundary="00000000000044bf6d0602640475" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --00000000000044bf6d0602640475 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable We've talked about this some more and the best way to move forward is to rebuild the ubuntu kernel. It should be fairly straightforward according to their wiki page . The page mentions a fairly old release (19.04), but was updated a year ago so the instructions are likely still valid. However, I don't have the link to the kernel patch that Honnappa mentioned. @Honnappa Nagarahalli @Ruifeng Wang , can you please provide a reference for the patch? Since the patch is small, there shouldn't be problems with applying it. Let us know whether this is doable. Regards, Juraj On Fri, Aug 4, 2023 at 11:48=E2=80=AFAM Ruifeng Wang = wrote: > Hi Patrick, > > > > Thanks for reaching out and my apologies for delayed response. > > > > We noticed that some information is missing regarding using QAT with DPDK > on Arm. > > The DPDK document will be updated to include the missing part. > > Will get back on this later. > > > > Best regards, > > Ruifeng > > > > *From:* Patrick Robb > *Sent:* Tuesday, August 1, 2023 1:14 AM > *To:* Ruifeng Wang ; Honnappa Nagarahalli < > Honnappa.Nagarahalli@arm.com>; Juraj Linke=C5=A1 > *Cc:* Dharmik Jayesh Thakkar ; ci@dpdk.org > *Subject:* Intel QAT 8970 accel card on ARM Ampere Server > > > > Hi Ruifeng, Honnappa, Juraj, > > > > The Intel QAT 8970 accelerator card has arrived to the Community Lab, and > we've installed it on the Ampere server. Presumably, we should test both > crypto and compress operations (and their respective performance metrics)= . > To that end, there are also DTS testsuites for testing QAT crypto/compres= s > functions. These testsuites make use of the crypto perf dpdk app and the > compress perf dpdk app. If you want, you can setup the DTS stuff yourself= , > both on the system side, and the Jenkins side (you are allowed to submit > PRs on our gitlab now), but we can also do that on the lab side as we > probably have more experience. I do, however, have a question about the Q= AT > kernel driver and corresponding PMDs. > > > > compress suite: > https://git.dpdk.org/tools/dts/tree/test_plans/compressdev_qat_pmd_test_p= lan.rst > > crypto suite: > https://git.dpdk.org/tools/dts/tree/test_plans/crypto_perf_cryptodev_perf= _test_plan.rst > > > > For reference, the DPDK docs page explaining QAT driver capabilities and > building the QAT PMDs (crypto sym, crypto asym, and compress) is here: > https://doc.dpdk.org/guides/cryptodevs/qat.html#building-qat > > Some notes before I get to my main question: > > -The 8970 is a C62x device > > -OpenSSL (arm requires it for QAT) is installed > > -3 PFs are visible from lspci (expected) > > -SRIOV is enabled > > > > However, although the system is on a valid kernel version for the QAT > driver, the kernel module for QAT is not loaded, so in trying to set up > testing, I am unable to create the 16 VFs for the 3 PFs respectively, lik= e > the example below: > > > > echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs > echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs > echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs > > > > There is also an option to download the firmware from the kernel firmware > repo and copy the qat binaries to /lib/firmware and start the qat modules > from there. I wasn't able to resolve the situation with this method, but = it > also could have been user error on my part. > > > > There is an option to install using the IDZ QAT Driver > , > but it should not be required given the kernel version the Ampere server = is > on, and I don't want to go down the road of relying on this "fall back" > method without consulting you first. Is it possible that there is anythin= g > specific to running a QAT device on ARM specifically which I am missing > here? The DTS testsuite testplans actually seem to recommend going down > this road in general, but the DPDK docs say to use the kernel driver, so = I > don't know. > > > > In any case, one of you should be able to login to the Ampere server in > situations like this, or just in general. Ruifeng/Juraj I see you both ha= ve > accounts on our IdM system, so you should have access. Please let me know > if you need renewed vpn cert configs and I will send you one. If you do > login, know this system could be running CI testing at any time. I can > always schedule time for it to be offline and available for maintenance i= f > you want to do anything which could be disruptive to testing. > > > > I also CC'd Dharmik on this as I see he sent an email regarding QAT > support on aarch64 in June. > > > > Let me know if you have any thoughts on the QAT kernel driver part. > > > > Thanks, > > Patrick > --00000000000044bf6d0602640475 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
We've talked about this some more and the best wa= y to move forward is to rebuild the ubuntu kernel. It should be fairly stra= ightforward according to their wiki page. The page mentions a fairly old release (19= .04), but was updated a year ago so the instructions are likely still valid= .

However, I don't have the link to the kernel= patch that Honnappa mentioned.=C2=A0= @Honnappa Nagarahalli=C2=A0@Ruifeng Wang<= /a>, can you please provide a reference for the patch?

=
Since the patch is small, there shouldn't be problems with applyin= g it. Let us know whether this is doable.

Regards,=
Juraj

On Fri, Aug 4, 2023 at 11:48=E2=80=AFAM Ruifeng Wang <Ruifeng.Wang@arm.com> wrote:

Hi Patrick,

=C2=A0

Thanks for reaching out and my apologies for delayed= response.

=C2=A0

We noticed that some information is missing regardin= g using QAT with DPDK on Arm.

The DPDK document will be updated to include the mis= sing part.

Will get back on this later.

=C2=A0

Best regards,

Ruifeng

=C2=A0

From: Patrick Robb <probb@iol.unh.edu>
Sent: Tuesday, August 1, 2023 1:14 AM
To: Ruifeng Wang <Ruifeng.Wang@arm.com>; Honnappa Nagarahalli <Honnappa.Nagarah= alli@arm.com>; Juraj Linke=C5=A1 <juraj.linkes@pantheon.tech><= br> Cc: Dharmik Jayesh Thakkar <DharmikJayesh.Thakkar@arm.com>; ci@dpdk.org
Subject: Intel QAT 8970 accel card on ARM Ampere Server

=C2=A0

Hi Ruifeng, Honnappa, Juraj,

=C2=A0

The Intel QAT 8970 accelerator card has arrived to t= he Community Lab, and we've installed it on the Ampere server. Presumab= ly, we should test both crypto and compress operations (and their respectiv= e performance metrics). To that end, there are also DTS testsuites for testing QAT crypto/compress functions. These t= estsuites make use of the crypto perf dpdk app and the compress perf dpdk a= pp. If you want, you can setup the DTS stuff yourself, both on the system= =C2=A0side, and the Jenkins side (you are allowed=C2=A0to submit PRs on our gitlab now), but we can also do that on = the lab side as we probably have more experience. I do, however, have a que= stion about the QAT kernel driver and corresponding PMDs.

=C2=A0

=C2=A0

For reference, the DPDK docs page explaining QAT dri= ver capabilities and building the QAT PMDs (crypto sym, crypto asym, and co= mpress) is here:=C2=A0https://doc.dpdk.org/guides/cryptodev= s/qat.html#building-qat

Some notes before I get to my main question:

-The 8970 is a C62x device

-OpenSSL (arm requires it for QAT) is installed

-3 PFs are visible from lspci (expected)

-SRIOV is enabled

=C2=A0

However, although the system is on a valid kernel ve= rsion for the QAT driver, the kernel module for QAT is not loaded, so in tr= ying to set up testing, I am unable to create the 16 VFs for the 3 PFs resp= ectively, like the example below:=C2=A0

=C2=A0

echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)= /sriov_numvfs
echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs
echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs=

=C2=A0

There is also an option to download the firmware fro= m the kernel firmware repo and copy the qat binaries to /lib/firmware and s= tart the qat modules from there. I wasn't able to resolve the situation= with this method, but it also could have been user error on my part.=C2=A0

=C2=A0

There is an option to install using the IDZ QAT Driver, but it should not be required given the kernel version = the Ampere server is on, and I don't want to go down the road of relyin= g on this "fall back" method without consulting you first. Is it = possible that there is anything specific to running a QAT device on ARM specifically which I am missing here? The DTS testsuit= e testplans actually seem to recommend going down this road in general, but= the DPDK docs say to use the kernel driver, so I don't know.=

=C2=A0

In any case, one of you should be able to login to t= he Ampere server in situations like this, or just in general. Ruifeng/Juraj= I see you both have accounts on our IdM system, so you should have access.= Please let me know if you need renewed vpn cert configs and I will send you one. If you do login, know this syste= m could be running CI testing at any time. I can always schedule time for i= t to be offline and available for maintenance if you want to do anything wh= ich could be disruptive to testing.=C2=A0

=C2=A0

I also CC'd Dharmik on this as I see he sent an = email regarding QAT support on aarch64 in June.

=C2=A0

Let me know if you have any thoughts on the QAT kern= el driver part.=C2=A0

=C2=A0

Thanks,

Patrick=C2=A0

--00000000000044bf6d0602640475--