From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4A8C7430BE; Mon, 21 Aug 2023 10:46:02 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1FDF340EE3; Mon, 21 Aug 2023 10:46:02 +0200 (CEST) Received: from mail-ed1-f54.google.com (mail-ed1-f54.google.com [209.85.208.54]) by mails.dpdk.org (Postfix) with ESMTP id 7142040A7D for ; Mon, 21 Aug 2023 10:46:00 +0200 (CEST) Received: by mail-ed1-f54.google.com with SMTP id 4fb4d7f45d1cf-523bf06f7f8so3652590a12.1 for ; Mon, 21 Aug 2023 01:46:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1692607560; x=1693212360; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=7+5z+DJ+bBxkGIF0GJYBE9ndX6FfLUJAH4mLBkURDmg=; b=b9pCc0/vPzze0UdxEIRaBvTYR8OYbY9b7MjH1zjKQbx2QnBp1L900KgJ8v6XU75vuB +dAU9ohYUJCd6sXPz7ATZ/61xnuNMAvXVpria2uJCtwEfZKsil05W5Y92PrjRCddj/gM ikOH/dqBVGl7eiyrrmcoilEGi9OfixEd2W7KDUp15dTHe2n6LkhkArNs3/v+zOfAGtE/ CpKqn+wg2XwxZm3fzeBdxfng+qiMqm+44NxpnwDvrxTpVefDnl0iXzI5DN86js/P9/Am Ttk9/E/VJxxi3kRCtQF8lF3lUquf7WGBnhT95ool26IGz3OPz5dXCuWR6UwxzPdv/O/N tHdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692607560; x=1693212360; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=7+5z+DJ+bBxkGIF0GJYBE9ndX6FfLUJAH4mLBkURDmg=; b=AfhEt1Fu89q+pkDoY2CvdOmiEB8bPfC4+MQ3zIjllsgQV4109SKYYFHX4GnOxBltWK tfm1Y3Up9s3ItFeF6YIBZ3oqxMN/E0h59OHAumZmgBKq+q2pnV9Q9B37JPxew3PLXbLX FY5c76XxpPc8Kzlgo+PDWPmM5hD6QScg94ykNT4ou89L7Da5mx6aeO4Ogb2s0MhZkIwC MOerlUxDPc0jcDX/7HPbVVQhLpHLGrY61eODFSkRVRNHKaf/xuEXl3UiQW/Cr0aIg2n8 vV5Mcy5Hl2jLPz4/YqRMG7cBHO8WpepDhIJFBEw2V0tOuY9yXL0GkDjfTuxlsqFhsjgi Pcug== X-Gm-Message-State: AOJu0Yxkui7mtw58EQqHkE+iPdyg2GU3iRdZ4pZ9MUV2yPN5VEGuDxxu L4wUs/6Sc2xKEXFcrH38tuqWl/S1b+qnZfUBot+7Z9Cd1ZoW+KSzFyUdJg== X-Google-Smtp-Source: AGHT+IFHzEQau/TFsIDr2jSxpaiIpaT3x4e70lxH0qCozL/ILZTYGVl4GbHMBqPcyCdmvpGUgEp4dzqlt3Xs7sdGDoM= X-Received: by 2002:aa7:d9d8:0:b0:525:734a:bf30 with SMTP id v24-20020aa7d9d8000000b00525734abf30mr5110742eds.36.1692607559621; Mon, 21 Aug 2023 01:45:59 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?Q?Juraj_Linke=C5=A1?= Date: Mon, 21 Aug 2023 10:45:38 +0200 Message-ID: Subject: Re: Intel QAT 8970 accel card on ARM Ampere Server To: Patrick Robb Cc: Ruifeng Wang , Honnappa Nagarahalli , Dharmik Jayesh Thakkar , "ci@dpdk.org" , nd Content-Type: multipart/alternative; boundary="00000000000056000a06036ae8a5" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --00000000000056000a06036ae8a5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Patrick, On Fri, Aug 11, 2023 at 11:18=E2=80=AFPM Patrick Robb w= rote: > Sorry about the wait on my reply guys. > > Thanks for the information. So I download the 2 diffs from that thread, > make a patch with them. Then where and how do I apply it? > > First get the ubuntu repo: - git clone git://kernel.ubuntu.com/ubuntu/ubuntu-.git 22.04 is jammy, but looking at https://kernel.ubuntu.com/git/, it's not under ubuntu/ubuntu-jammy.git, but rather ubuntu-stable/ubuntu-stable-jammy.git. It also seems the repo's been redirected: git clone git://kernel.ubuntu.com/ubuntu-stable/ubuntu-stable-jammy.git Cloning into 'ubuntu-stable-jammy'... fatal: remote error: **REPOSITORY RELOCATED** Updated URL: https://git.launchpad.net/~ubuntu-kernel-stable/+git/jammy Local path: /ubuntu-stable/ubuntu-stable-jammy.git Cloning the new URL worked for me. Then we need to checkout the tag that corresponds to the running kernel (uname -r), apply the patch and build the kernel with the running config (in /boot/config-$(uname -r)), possibly enabling the QAT driver if needed. > Then I install the packages needed per the ubuntu page, and then I can > skip down to the "Building The Kernel" section? And then we're all set I > think, and we just have to setup DTS and associated Jenkins pipelines. > > Do you want me to back anything up in advance of this? I don't know if > that is needed or not, but Ampere is currently live doing testing for CI, > so I want to act in a safe manner. I will try to address this first thing > on Monday and get back to you. > > The backup should not be needed, at least in principle, as we can always reinstall the original kernel packages. > > > On Tue, Aug 8, 2023 at 3:11=E2=80=AFAM Ruifeng Wang wrote: > >> *From:* Juraj Linke=C5=A1 >> *Sent:* Tuesday, August 8, 2023 3:07 PM >> *To:* Ruifeng Wang >> *Cc:* Patrick Robb ; Honnappa Nagarahalli < >> Honnappa.Nagarahalli@arm.com>; Dharmik Jayesh Thakkar < >> DharmikJayesh.Thakkar@arm.com>; ci@dpdk.org; nd >> *Subject:* Re: Intel QAT 8970 accel card on ARM Ampere Server >> >> >> >> We've talked about this some more and the best way to move forward is to >> rebuild the ubuntu kernel. It should be fairly straightforward according= to their >> wiki page . The page >> mentions a fairly old release (19.04), but was updated a year ago so the >> instructions are likely still valid. >> >> >> >> However, I don't have the link to the kernel patch that Honnappa >> mentioned. @Honnappa Nagarahalli @Ruifeng >> Wang , can you please provide a reference for the >> patch? >> >> *[Ruifeng]* Here is the kernel patch set: >> https://lkml.org/lkml/2022/6/17/328 >> >> >> >> Since the patch is small, there shouldn't be problems with applying it. >> Let us know whether this is doable. >> >> >> >> Regards, >> >> Juraj >> >> >> >> On Fri, Aug 4, 2023 at 11:48=E2=80=AFAM Ruifeng Wang >> wrote: >> >> Hi Patrick, >> >> >> >> Thanks for reaching out and my apologies for delayed response. >> >> >> >> We noticed that some information is missing regarding using QAT with DPD= K >> on Arm. >> >> The DPDK document will be updated to include the missing part. >> >> Will get back on this later. >> >> >> >> Best regards, >> >> Ruifeng >> >> >> >> *From:* Patrick Robb >> *Sent:* Tuesday, August 1, 2023 1:14 AM >> *To:* Ruifeng Wang ; Honnappa Nagarahalli < >> Honnappa.Nagarahalli@arm.com>; Juraj Linke=C5=A1 >> *Cc:* Dharmik Jayesh Thakkar ; ci@dpdk.or= g >> *Subject:* Intel QAT 8970 accel card on ARM Ampere Server >> >> >> >> Hi Ruifeng, Honnappa, Juraj, >> >> >> >> The Intel QAT 8970 accelerator card has arrived to the Community Lab, an= d >> we've installed it on the Ampere server. Presumably, we should test both >> crypto and compress operations (and their respective performance metrics= ). >> To that end, there are also DTS testsuites for testing QAT crypto/compre= ss >> functions. These testsuites make use of the crypto perf dpdk app and the >> compress perf dpdk app. If you want, you can setup the DTS stuff yoursel= f, >> both on the system side, and the Jenkins side (you are allowed to submit >> PRs on our gitlab now), but we can also do that on the lab side as we >> probably have more experience. I do, however, have a question about the = QAT >> kernel driver and corresponding PMDs. >> >> >> >> compress suite: >> https://git.dpdk.org/tools/dts/tree/test_plans/compressdev_qat_pmd_test_= plan.rst >> >> crypto suite: >> https://git.dpdk.org/tools/dts/tree/test_plans/crypto_perf_cryptodev_per= f_test_plan.rst >> >> >> >> For reference, the DPDK docs page explaining QAT driver capabilities and >> building the QAT PMDs (crypto sym, crypto asym, and compress) is here: >> https://doc.dpdk.org/guides/cryptodevs/qat.html#building-qat >> >> Some notes before I get to my main question: >> >> -The 8970 is a C62x device >> >> -OpenSSL (arm requires it for QAT) is installed >> >> -3 PFs are visible from lspci (expected) >> >> -SRIOV is enabled >> >> >> >> However, although the system is on a valid kernel version for the QAT >> driver, the kernel module for QAT is not loaded, so in trying to set up >> testing, I am unable to create the 16 VFs for the 3 PFs respectively, li= ke >> the example below: >> >> >> >> echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs >> echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs >> echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs >> >> >> >> There is also an option to download the firmware from the kernel firmwar= e >> repo and copy the qat binaries to /lib/firmware and start the qat module= s >> from there. I wasn't able to resolve the situation with this method, but= it >> also could have been user error on my part. >> >> >> >> There is an option to install using the IDZ QAT Driver >> , >> but it should not be required given the kernel version the Ampere server= is >> on, and I don't want to go down the road of relying on this "fall back" >> method without consulting you first. Is it possible that there is anythi= ng >> specific to running a QAT device on ARM specifically which I am missing >> here? The DTS testsuite testplans actually seem to recommend going down >> this road in general, but the DPDK docs say to use the kernel driver, so= I >> don't know. >> >> >> >> In any case, one of you should be able to login to the Ampere server in >> situations like this, or just in general. Ruifeng/Juraj I see you both h= ave >> accounts on our IdM system, so you should have access. Please let me kno= w >> if you need renewed vpn cert configs and I will send you one. If you do >> login, know this system could be running CI testing at any time. I can >> always schedule time for it to be offline and available for maintenance = if >> you want to do anything which could be disruptive to testing. >> >> >> >> I also CC'd Dharmik on this as I see he sent an email regarding QAT >> support on aarch64 in June. >> >> >> >> Let me know if you have any thoughts on the QAT kernel driver part. >> >> >> >> Thanks, >> >> Patrick >> >> > > -- > > Patrick Robb > > Technical Service Manager > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > www.iol.unh.edu > > > --00000000000056000a06036ae8a5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Patrick,

On Fri, Aug 11, = 2023 at 11:18=E2=80=AFPM Patrick Robb <probb@iol.unh.edu> wrote:
Sorry abou= t the wait on my reply guys.

Thanks for the information.= So I download the 2 diffs from that thread, make a patch with them. Then w= here and how do I apply it?=C2=A0


First get the ubuntu repo:
22.04 is jammy, but looking at https://kernel.ubuntu.com/git/, it&= #39;s not under ubuntu/ubuntu-jammy.git, but rather=C2=A0ubuntu-stable/ubun= tu-stable-jammy.git. It also seems the repo's been redirected:

Clon= ing into 'ubuntu-stable-jammy'...
fatal: remote error: **REPOSITORY= RELOCATED** =C2=A0Updated URL: https://git.launchpad.net/~ubuntu-kernel-stable= /+git/jammy Local path: /ubuntu-stable/ubuntu-stable-jammy.git

Cloni= ng the new URL worked for me. Then we need to checkout the tag that corresp= onds to the running kernel (uname -r), apply the patch and build the kernel= with the running config (in /boot/config-$(uname -r)), possibly enabling t= he QAT driver if needed.
=C2=A0
Then I install the p= ackages needed per the ubuntu page, and then I can skip down to the "B= uilding The Kernel" section? And then we're all set I think, and w= e just have to setup DTS and associated Jenkins pipelines.=C2=A0
=
Do you want me to back anything up in advance of this? I don= 't know if that is needed or not, but Ampere is currently live doing te= sting for CI, so I want to act in a safe manner. I will try to address this= first thing on Monday and get back to you.=C2=A0


The backup should not be needed, at least= in principle, as we can always reinstall the original kernel packages.
=C2=A0


<= div dir=3D"ltr" class=3D"gmail_attr">On Tue, Aug 8, 2023 at 3:11=E2=80=AFAM= Ruifeng Wang <Ruifeng.Wang@arm.com> wrote:

From: Juraj Linke=C5=A1 <juraj.linkes@pant= heon.tech>
Sent: Tuesday, August 8, 2023 3:07 PM
To: Ruifeng Wang <Ruifeng.Wang@arm.com>
Cc: Patrick Robb <probb@iol.unh.edu>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm= .com>; Dharmik Jayesh Thakkar <DharmikJayesh.Thakkar@arm.com>; ci@dpdk.org; nd <nd@arm.com>
Subject: Re: Intel QAT 8970 accel card on ARM Ampere Server

=C2=A0

We've talked about this some more and the best w= ay to move forward is to rebuild the ubuntu kernel. It should be fairly str= aightforward according to their wiki page. The page mentions a fairly old release (19.04), b= ut was updated a year ago so the instructions are likely still valid.

=C2=A0

However, I don't have the link to the kernel pat= ch that Honnappa mentioned.=C2=A0@Honnappa Nagarahalli=C2=A0@Ruifeng Wang, can you please pro= vide a reference for the patch?

[Ruifeng] Here is the kernel patch set: https://lkml.org/lkml/2022/6/17/328

=C2=A0

Since the patch is small, there shouldn't be pro= blems with applying it. Let us know whether this is doable.

=C2=A0

Regards,

Juraj

=C2=A0

On Fri, Aug 4, 2023 at 11:48=E2=80=AFAM Ruifeng Wang= <Ruifeng.Wang= @arm.com> wrote:

Hi Patrick,

=C2=A0

Thanks for reaching out and my apologies for delayed= response.

=C2=A0

We noticed that some information is missing regardin= g using QAT with DPDK on Arm.

The DPDK document will be updated to include the mis= sing part.

Will get back on this later.

=C2=A0

Best regards,

Ruifeng

=C2=A0

From: Patrick Robb <probb@iol.unh.edu>
Sent: Tuesday, August 1, 2023 1:14 AM
To: Ruifeng Wang <Ruifeng.Wang@arm.com>; Honnappa Nagarahalli <Honnappa.Nagarah= alli@arm.com>; Juraj Linke=C5=A1 <juraj.linkes@pantheon.tech>
Cc: Dharmik Jayesh Thakkar <DharmikJayesh.Thakkar@arm.com>; ci@dpdk.org
Subject: Intel QAT 8970 accel card on ARM Ampere Server

=C2=A0

Hi Ruifeng, Honnappa, Juraj,

=C2=A0

The Intel QAT 8970 accelerator card has arrived to t= he Community Lab, and we've installed it on the Ampere server. Presumab= ly, we should test both crypto and compress operations (and their respective performance metrics). To that end, there are also DT= S testsuites for testing QAT crypto/compress functions. These testsuites ma= ke use of the crypto perf dpdk app and the compress perf dpdk app. If you w= ant, you can setup the DTS stuff yourself, both on the system=C2=A0side, and the Jenkins side (you are allo= wed=C2=A0to submit PRs on our gitlab now), but we can also do that on the l= ab side as we probably have more experience. I do, however, have a question= about the QAT kernel driver and corresponding PMDs.

=C2=A0

=C2=A0

For reference, the DPDK docs page explaining QAT dri= ver capabilities and building the QAT PMDs (crypto sym, crypto asym, and co= mpress) is here:=C2=A0https://doc.dpdk.org/guides/cryptodev= s/qat.html#building-qat

Some notes before I get to my main question:

-The 8970 is a C62x device

-OpenSSL (arm requires it for QAT) is installed

-3 PFs are visible from lspci (expected)

-SRIOV is enabled

=C2=A0

However, although the system is on a valid kernel ve= rsion for the QAT driver, the kernel module for QAT is not loaded, so in tr= ying to set up testing, I am unable to create the 16 VFs for the 3 PFs respectively, like the example below:=C2=A0=

=C2=A0

echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)= /sriov_numvfs
echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs
echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs=

=C2=A0

There is also an option to download the firmware fro= m the kernel firmware repo and copy the qat binaries to /lib/firmware and s= tart the qat modules from there. I wasn't able to resolve the situation with this method, but it also could have been user e= rror on my part.=C2=A0

=C2=A0

There is an option to install using the IDZ QAT Driver, but it should not be required given the kernel version = the Ampere server is on, and I don't want to go down the road of relyin= g on this "fall back" method without consulting you first. Is it = possible that there is anything specific to running a QAT device on ARM specifically which I am missing here? The DTS testsuit= e testplans actually seem to recommend going down this road in general, but= the DPDK docs say to use the kernel driver, so I don't know.=

=C2=A0

In any case, one of you should be able to login to t= he Ampere server in situations like this, or just in general. Ruifeng/Juraj= I see you both have accounts on our IdM system, so you should have access. Please let me know if you need renewed vpn cert co= nfigs and I will send you one. If you do login, know this system could be r= unning CI testing at any time. I can always schedule time for it to be offl= ine and available for maintenance if you want to do anything which could be disruptive to testing.=C2=A0<= /u>

=C2=A0

I also CC'd Dharmik on this as I see he sent an = email regarding QAT support on aarch64 in June.

=C2=A0

Let me know if you have any thoughts on the QAT kern= el driver part.=C2=A0

=C2=A0

Thanks,

Patrick=C2=A0



--

Patrick Robb

Technic= al Service Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu


<= p dir=3D"ltr" style=3D"color:rgb(34,34,34);line-height:1.2;margin-top:0pt;m= argin-bottom:0pt">

--00000000000056000a06036ae8a5--