From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 07B2D43038; Fri, 11 Aug 2023 23:18:24 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DAC0940E03; Fri, 11 Aug 2023 23:18:23 +0200 (CEST) Received: from mail-ot1-f45.google.com (mail-ot1-f45.google.com [209.85.210.45]) by mails.dpdk.org (Postfix) with ESMTP id 5C22940144 for ; Fri, 11 Aug 2023 23:18:22 +0200 (CEST) Received: by mail-ot1-f45.google.com with SMTP id 46e09a7af769-6bcae8c4072so2075763a34.1 for ; Fri, 11 Aug 2023 14:18:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1691788701; x=1692393501; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=iIN6dPBq8xvolb+1S1a5oejerJWXhL4afDE3V4Jp1xY=; b=CTfkL40lmUB8jIZ5gNPNrXc7mnnxdy9zcj+HmrShUa1IgT/E1BlSXf8ooTDdAWJuT2 Mkcsgm2niV/OQ3Ctgc4jpTubcKmSqIxC4S8ztKPds72AQicL/J5x2J93DcuwOBpf65vN omQCyXmZQW7ydMDd9ca6DKOsOl6o5qV1MenEQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691788701; x=1692393501; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=iIN6dPBq8xvolb+1S1a5oejerJWXhL4afDE3V4Jp1xY=; b=dEYr7gCDJrWxViMTqqm/B8x1Nxyb3IF4dBzPJ6dv/uv0NEH3J8VfxL6PHVtEW3p1Tt uUmZH9ilMUflv+dL7nIVLnJjkHO6TGESTlvtz3W7SQn/bwmCqN72wkyTOqhs9owN1aXG k1v1YR/uu7DeTpA0Y49TolzJuWevNU9SHSW06EIdacjCDgdibm42J0HA6VZ/+GWeiBdk LI/EqTzGWus6fH6X68bv/Gnjnwn/PxSisiPHy1DXUA/p5qsMWmmjq6qSpkyIy2IyzHtU f+yAn1Z9OehU8BWWwIJ8uKwMfS/0mZPwWETr7Jtg2hGjMNsEahS3yXd5EMdEYka0n9kT Kfbw== X-Gm-Message-State: AOJu0YxacZIBEjoaeVPIIaDdOH04Wt2/CiO8KUaX+Hg6cM7N0ohqMbx3 joE76kCyQx3MhYXaqaRQ0PHDkPQIsSbolQqGRTizHw== X-Google-Smtp-Source: AGHT+IHZRr8m3TW+I4yMTuVntCvpf2EN772Zk6CPUQBJUaGPqB0CgPT/UyPn23wXSWQLvE9OepCfxd+FLN+viM2a3ac= X-Received: by 2002:a05:6870:d1c6:b0:19f:6fae:d5fc with SMTP id b6-20020a056870d1c600b0019f6faed5fcmr2883158oac.33.1691788701353; Fri, 11 Aug 2023 14:18:21 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Patrick Robb Date: Fri, 11 Aug 2023 17:18:10 -0400 Message-ID: Subject: Re: Intel QAT 8970 accel card on ARM Ampere Server To: Ruifeng Wang Cc: =?UTF-8?Q?Juraj_Linke=C5=A1?= , Honnappa Nagarahalli , Dharmik Jayesh Thakkar , "ci@dpdk.org" , nd Content-Type: multipart/alternative; boundary="00000000000094605e0602ac40a7" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --00000000000094605e0602ac40a7 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sorry about the wait on my reply guys. Thanks for the information. So I download the 2 diffs from that thread, make a patch with them. Then where and how do I apply it? Then I install the packages needed per the ubuntu page, and then I can skip down to the "Building The Kernel" section? And then we're all set I think, and we just have to setup DTS and associated Jenkins pipelines. Do you want me to back anything up in advance of this? I don't know if that is needed or not, but Ampere is currently live doing testing for CI, so I want to act in a safe manner. I will try to address this first thing on Monday and get back to you. On Tue, Aug 8, 2023 at 3:11=E2=80=AFAM Ruifeng Wang = wrote: > *From:* Juraj Linke=C5=A1 > *Sent:* Tuesday, August 8, 2023 3:07 PM > *To:* Ruifeng Wang > *Cc:* Patrick Robb ; Honnappa Nagarahalli < > Honnappa.Nagarahalli@arm.com>; Dharmik Jayesh Thakkar < > DharmikJayesh.Thakkar@arm.com>; ci@dpdk.org; nd > *Subject:* Re: Intel QAT 8970 accel card on ARM Ampere Server > > > > We've talked about this some more and the best way to move forward is to > rebuild the ubuntu kernel. It should be fairly straightforward according = to their > wiki page . The page > mentions a fairly old release (19.04), but was updated a year ago so the > instructions are likely still valid. > > > > However, I don't have the link to the kernel patch that Honnappa > mentioned. @Honnappa Nagarahalli @Ruifeng > Wang , can you please provide a reference for the > patch? > > *[Ruifeng]* Here is the kernel patch set: > https://lkml.org/lkml/2022/6/17/328 > > > > Since the patch is small, there shouldn't be problems with applying it. > Let us know whether this is doable. > > > > Regards, > > Juraj > > > > On Fri, Aug 4, 2023 at 11:48=E2=80=AFAM Ruifeng Wang wrote: > > Hi Patrick, > > > > Thanks for reaching out and my apologies for delayed response. > > > > We noticed that some information is missing regarding using QAT with DPDK > on Arm. > > The DPDK document will be updated to include the missing part. > > Will get back on this later. > > > > Best regards, > > Ruifeng > > > > *From:* Patrick Robb > *Sent:* Tuesday, August 1, 2023 1:14 AM > *To:* Ruifeng Wang ; Honnappa Nagarahalli < > Honnappa.Nagarahalli@arm.com>; Juraj Linke=C5=A1 > *Cc:* Dharmik Jayesh Thakkar ; ci@dpdk.org > *Subject:* Intel QAT 8970 accel card on ARM Ampere Server > > > > Hi Ruifeng, Honnappa, Juraj, > > > > The Intel QAT 8970 accelerator card has arrived to the Community Lab, and > we've installed it on the Ampere server. Presumably, we should test both > crypto and compress operations (and their respective performance metrics)= . > To that end, there are also DTS testsuites for testing QAT crypto/compres= s > functions. These testsuites make use of the crypto perf dpdk app and the > compress perf dpdk app. If you want, you can setup the DTS stuff yourself= , > both on the system side, and the Jenkins side (you are allowed to submit > PRs on our gitlab now), but we can also do that on the lab side as we > probably have more experience. I do, however, have a question about the Q= AT > kernel driver and corresponding PMDs. > > > > compress suite: > https://git.dpdk.org/tools/dts/tree/test_plans/compressdev_qat_pmd_test_p= lan.rst > > crypto suite: > https://git.dpdk.org/tools/dts/tree/test_plans/crypto_perf_cryptodev_perf= _test_plan.rst > > > > For reference, the DPDK docs page explaining QAT driver capabilities and > building the QAT PMDs (crypto sym, crypto asym, and compress) is here: > https://doc.dpdk.org/guides/cryptodevs/qat.html#building-qat > > Some notes before I get to my main question: > > -The 8970 is a C62x device > > -OpenSSL (arm requires it for QAT) is installed > > -3 PFs are visible from lspci (expected) > > -SRIOV is enabled > > > > However, although the system is on a valid kernel version for the QAT > driver, the kernel module for QAT is not loaded, so in trying to set up > testing, I am unable to create the 16 VFs for the 3 PFs respectively, lik= e > the example below: > > > > echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs > echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs > echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs > > > > There is also an option to download the firmware from the kernel firmware > repo and copy the qat binaries to /lib/firmware and start the qat modules > from there. I wasn't able to resolve the situation with this method, but = it > also could have been user error on my part. > > > > There is an option to install using the IDZ QAT Driver > , > but it should not be required given the kernel version the Ampere server = is > on, and I don't want to go down the road of relying on this "fall back" > method without consulting you first. Is it possible that there is anythin= g > specific to running a QAT device on ARM specifically which I am missing > here? The DTS testsuite testplans actually seem to recommend going down > this road in general, but the DPDK docs say to use the kernel driver, so = I > don't know. > > > > In any case, one of you should be able to login to the Ampere server in > situations like this, or just in general. Ruifeng/Juraj I see you both ha= ve > accounts on our IdM system, so you should have access. Please let me know > if you need renewed vpn cert configs and I will send you one. If you do > login, know this system could be running CI testing at any time. I can > always schedule time for it to be offline and available for maintenance i= f > you want to do anything which could be disruptive to testing. > > > > I also CC'd Dharmik on this as I see he sent an email regarding QAT > support on aarch64 in June. > > > > Let me know if you have any thoughts on the QAT kernel driver part. > > > > Thanks, > > Patrick > > --=20 Patrick Robb Technical Service Manager UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 www.iol.unh.edu --00000000000094605e0602ac40a7 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Sorry about the wait on my reply guys.

= Thanks for the information. So I download the 2 diffs from that thread, mak= e a patch with them. Then where and how do I apply it?=C2=A0

=
Then I install the packages needed per the ubuntu page, and then= I can skip down to the "Building The Kernel" section? And then w= e're all set I think, and we just have to setup DTS and associated Jenk= ins pipelines.=C2=A0

Do you want me to back anythi= ng up in advance of this? I don't know if that is needed or not, but Am= pere is currently live doing testing for CI, so I want to act in a safe man= ner. I will try to address this first thing on Monday and get back to you.= =C2=A0



On Tue, Aug 8, 2023 at 3:11=E2=80= =AFAM Ruifeng Wang <Ruifeng.Wang= @arm.com> wrote:

From: Juraj Linke=C5=A1 <juraj.linkes@pant= heon.tech>
Sent: Tuesday, August 8, 2023 3:07 PM
To: Ruifeng Wang <Ruifeng.Wang@arm.com>
Cc: Patrick Robb <probb@iol.unh.edu>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm= .com>; Dharmik Jayesh Thakkar <DharmikJayesh.Thakkar@arm.com>; ci@dpdk.org; nd <nd@arm.com>
Subject: Re: Intel QAT 8970 accel card on ARM Ampere Server

=C2=A0

We've talked about this some more and the best w= ay to move forward is to rebuild the ubuntu kernel. It should be fairly str= aightforward according to their wiki page. The page mentions a fairly old release (19.04), b= ut was updated a year ago so the instructions are likely still valid.

=C2=A0

However, I don't have the link to the kernel pat= ch that Honnappa mentioned.=C2=A0@Honnappa Nagarahalli=C2=A0@Ruifeng Wang, can you please pro= vide a reference for the patch?

[Ruifeng] Here is the kernel patch set: https://lkml.org/lkml/2022/6/17/328

=C2=A0

Since the patch is small, there shouldn't be pro= blems with applying it. Let us know whether this is doable.

=C2=A0

Regards,

Juraj

=C2=A0

On Fri, Aug 4, 2023 at 11:48=E2=80=AFAM Ruifeng Wang= <Ruifeng.Wang= @arm.com> wrote:

Hi Patrick,

=C2=A0

Thanks for reaching out and my apologies for delayed= response.

=C2=A0

We noticed that some information is missing regardin= g using QAT with DPDK on Arm.

The DPDK document will be updated to include the mis= sing part.

Will get back on this later.

=C2=A0

Best regards,

Ruifeng

=C2=A0

From: Patrick Robb <probb@iol.unh.edu>
Sent: Tuesday, August 1, 2023 1:14 AM
To: Ruifeng Wang <Ruifeng.Wang@arm.com>; Honnappa Nagarahalli <Honnappa.Nagarah= alli@arm.com>; Juraj Linke=C5=A1 <juraj.linkes@pantheon.tech>
Cc: Dharmik Jayesh Thakkar <DharmikJayesh.Thakkar@arm.com>; ci@dpdk.org
Subject: Intel QAT 8970 accel card on ARM Ampere Server

=C2=A0

Hi Ruifeng, Honnappa, Juraj,

=C2=A0

The Intel QAT 8970 accelerator card has arrived to t= he Community Lab, and we've installed it on the Ampere server. Presumab= ly, we should test both crypto and compress operations (and their respective performance metrics). To that end, there are also DT= S testsuites for testing QAT crypto/compress functions. These testsuites ma= ke use of the crypto perf dpdk app and the compress perf dpdk app. If you w= ant, you can setup the DTS stuff yourself, both on the system=C2=A0side, and the Jenkins side (you are allo= wed=C2=A0to submit PRs on our gitlab now), but we can also do that on the l= ab side as we probably have more experience. I do, however, have a question= about the QAT kernel driver and corresponding PMDs.

=C2=A0

=C2=A0

For reference, the DPDK docs page explaining QAT dri= ver capabilities and building the QAT PMDs (crypto sym, crypto asym, and co= mpress) is here:=C2=A0https://doc.dpdk.org/guides/cryptodev= s/qat.html#building-qat

Some notes before I get to my main question:

-The 8970 is a C62x device

-OpenSSL (arm requires it for QAT) is installed

-3 PFs are visible from lspci (expected)

-SRIOV is enabled

=C2=A0

However, although the system is on a valid kernel ve= rsion for the QAT driver, the kernel module for QAT is not loaded, so in tr= ying to set up testing, I am unable to create the 16 VFs for the 3 PFs respectively, like the example below:=C2=A0=

=C2=A0

echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)= /sriov_numvfs
echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs
echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs=

=C2=A0

There is also an option to download the firmware fro= m the kernel firmware repo and copy the qat binaries to /lib/firmware and s= tart the qat modules from there. I wasn't able to resolve the situation with this method, but it also could have been user e= rror on my part.=C2=A0

=C2=A0

There is an option to install using the IDZ QAT Driver, but it should not be required given the kernel version = the Ampere server is on, and I don't want to go down the road of relyin= g on this "fall back" method without consulting you first. Is it = possible that there is anything specific to running a QAT device on ARM specifically which I am missing here? The DTS testsuit= e testplans actually seem to recommend going down this road in general, but= the DPDK docs say to use the kernel driver, so I don't know.=

=C2=A0

In any case, one of you should be able to login to t= he Ampere server in situations like this, or just in general. Ruifeng/Juraj= I see you both have accounts on our IdM system, so you should have access. Please let me know if you need renewed vpn cert co= nfigs and I will send you one. If you do login, know this system could be r= unning CI testing at any time. I can always schedule time for it to be offl= ine and available for maintenance if you want to do anything which could be disruptive to testing.=C2=A0<= /u>

=C2=A0

I also CC'd Dharmik on this as I see he sent an = email regarding QAT support on aarch64 in June.

=C2=A0

Let me know if you have any thoughts on the QAT kern= el driver part.=C2=A0

=C2=A0

Thanks,

Patrick=C2=A0



--

Patrick Robb

Technic= al Service Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu


<= p dir=3D"ltr" style=3D"color:rgb(34,34,34);line-height:1.2;margin-top:0pt;m= argin-bottom:0pt">

--00000000000094605e0602ac40a7--