From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <ci-bounces@dpdk.org> Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4A8C7430BE; Mon, 21 Aug 2023 10:46:02 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1FDF340EE3; Mon, 21 Aug 2023 10:46:02 +0200 (CEST) Received: from mail-ed1-f54.google.com (mail-ed1-f54.google.com [209.85.208.54]) by mails.dpdk.org (Postfix) with ESMTP id 7142040A7D for <ci@dpdk.org>; Mon, 21 Aug 2023 10:46:00 +0200 (CEST) Received: by mail-ed1-f54.google.com with SMTP id 4fb4d7f45d1cf-523bf06f7f8so3652590a12.1 for <ci@dpdk.org>; Mon, 21 Aug 2023 01:46:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1692607560; x=1693212360; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=7+5z+DJ+bBxkGIF0GJYBE9ndX6FfLUJAH4mLBkURDmg=; b=b9pCc0/vPzze0UdxEIRaBvTYR8OYbY9b7MjH1zjKQbx2QnBp1L900KgJ8v6XU75vuB +dAU9ohYUJCd6sXPz7ATZ/61xnuNMAvXVpria2uJCtwEfZKsil05W5Y92PrjRCddj/gM ikOH/dqBVGl7eiyrrmcoilEGi9OfixEd2W7KDUp15dTHe2n6LkhkArNs3/v+zOfAGtE/ CpKqn+wg2XwxZm3fzeBdxfng+qiMqm+44NxpnwDvrxTpVefDnl0iXzI5DN86js/P9/Am Ttk9/E/VJxxi3kRCtQF8lF3lUquf7WGBnhT95ool26IGz3OPz5dXCuWR6UwxzPdv/O/N tHdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692607560; x=1693212360; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=7+5z+DJ+bBxkGIF0GJYBE9ndX6FfLUJAH4mLBkURDmg=; b=AfhEt1Fu89q+pkDoY2CvdOmiEB8bPfC4+MQ3zIjllsgQV4109SKYYFHX4GnOxBltWK tfm1Y3Up9s3ItFeF6YIBZ3oqxMN/E0h59OHAumZmgBKq+q2pnV9Q9B37JPxew3PLXbLX FY5c76XxpPc8Kzlgo+PDWPmM5hD6QScg94ykNT4ou89L7Da5mx6aeO4Ogb2s0MhZkIwC MOerlUxDPc0jcDX/7HPbVVQhLpHLGrY61eODFSkRVRNHKaf/xuEXl3UiQW/Cr0aIg2n8 vV5Mcy5Hl2jLPz4/YqRMG7cBHO8WpepDhIJFBEw2V0tOuY9yXL0GkDjfTuxlsqFhsjgi Pcug== X-Gm-Message-State: AOJu0Yxkui7mtw58EQqHkE+iPdyg2GU3iRdZ4pZ9MUV2yPN5VEGuDxxu L4wUs/6Sc2xKEXFcrH38tuqWl/S1b+qnZfUBot+7Z9Cd1ZoW+KSzFyUdJg== X-Google-Smtp-Source: AGHT+IFHzEQau/TFsIDr2jSxpaiIpaT3x4e70lxH0qCozL/ILZTYGVl4GbHMBqPcyCdmvpGUgEp4dzqlt3Xs7sdGDoM= X-Received: by 2002:aa7:d9d8:0:b0:525:734a:bf30 with SMTP id v24-20020aa7d9d8000000b00525734abf30mr5110742eds.36.1692607559621; Mon, 21 Aug 2023 01:45:59 -0700 (PDT) MIME-Version: 1.0 References: <CAJvnSUCeHvf7dLo4KiV0GVrgMVaGT9zWbYDmUP-WMpj=5Bv3mw@mail.gmail.com> <AS8PR08MB7080BFD3E3C508EED8C88FE19E09A@AS8PR08MB7080.eurprd08.prod.outlook.com> <CAOb5WZaGmQVLuLo9=ZBVtRh6-2=9vtS-foJN3GZerjaq17e6ow@mail.gmail.com> <AS8PR08MB70800D8F0315FFCE194C4FB99E0DA@AS8PR08MB7080.eurprd08.prod.outlook.com> <CAJvnSUB0fOmVBng=omuvP0sBMmhz3=gFO-SVXzNzBYp9d-zKHQ@mail.gmail.com> In-Reply-To: <CAJvnSUB0fOmVBng=omuvP0sBMmhz3=gFO-SVXzNzBYp9d-zKHQ@mail.gmail.com> From: =?UTF-8?Q?Juraj_Linke=C5=A1?= <juraj.linkes@pantheon.tech> Date: Mon, 21 Aug 2023 10:45:38 +0200 Message-ID: <CAOb5WZZ9eR9JBv825RLo1zqYM0hcD=Fozzqt4m7JeGUeVSO0gA@mail.gmail.com> Subject: Re: Intel QAT 8970 accel card on ARM Ampere Server To: Patrick Robb <probb@iol.unh.edu> Cc: Ruifeng Wang <Ruifeng.Wang@arm.com>, Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>, Dharmik Jayesh Thakkar <DharmikJayesh.Thakkar@arm.com>, "ci@dpdk.org" <ci@dpdk.org>, nd <nd@arm.com> Content-Type: multipart/alternative; boundary="00000000000056000a06036ae8a5" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions <ci.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/ci>, <mailto:ci-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/ci/> List-Post: <mailto:ci@dpdk.org> List-Help: <mailto:ci-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/ci>, <mailto:ci-request@dpdk.org?subject=subscribe> Errors-To: ci-bounces@dpdk.org --00000000000056000a06036ae8a5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Patrick, On Fri, Aug 11, 2023 at 11:18=E2=80=AFPM Patrick Robb <probb@iol.unh.edu> w= rote: > Sorry about the wait on my reply guys. > > Thanks for the information. So I download the 2 diffs from that thread, > make a patch with them. Then where and how do I apply it? > > First get the ubuntu repo: - git clone git://kernel.ubuntu.com/ubuntu/ubuntu-<release codename>.git 22.04 is jammy, but looking at https://kernel.ubuntu.com/git/, it's not under ubuntu/ubuntu-jammy.git, but rather ubuntu-stable/ubuntu-stable-jammy.git. It also seems the repo's been redirected: git clone git://kernel.ubuntu.com/ubuntu-stable/ubuntu-stable-jammy.git Cloning into 'ubuntu-stable-jammy'... fatal: remote error: **REPOSITORY RELOCATED** Updated URL: https://git.launchpad.net/~ubuntu-kernel-stable/+git/jammy Local path: /ubuntu-stable/ubuntu-stable-jammy.git Cloning the new URL worked for me. Then we need to checkout the tag that corresponds to the running kernel (uname -r), apply the patch and build the kernel with the running config (in /boot/config-$(uname -r)), possibly enabling the QAT driver if needed. > Then I install the packages needed per the ubuntu page, and then I can > skip down to the "Building The Kernel" section? And then we're all set I > think, and we just have to setup DTS and associated Jenkins pipelines. > > Do you want me to back anything up in advance of this? I don't know if > that is needed or not, but Ampere is currently live doing testing for CI, > so I want to act in a safe manner. I will try to address this first thing > on Monday and get back to you. > > The backup should not be needed, at least in principle, as we can always reinstall the original kernel packages. > > > On Tue, Aug 8, 2023 at 3:11=E2=80=AFAM Ruifeng Wang <Ruifeng.Wang@arm.com= > wrote: > >> *From:* Juraj Linke=C5=A1 <juraj.linkes@pantheon.tech> >> *Sent:* Tuesday, August 8, 2023 3:07 PM >> *To:* Ruifeng Wang <Ruifeng.Wang@arm.com> >> *Cc:* Patrick Robb <probb@iol.unh.edu>; Honnappa Nagarahalli < >> Honnappa.Nagarahalli@arm.com>; Dharmik Jayesh Thakkar < >> DharmikJayesh.Thakkar@arm.com>; ci@dpdk.org; nd <nd@arm.com> >> *Subject:* Re: Intel QAT 8970 accel card on ARM Ampere Server >> >> >> >> We've talked about this some more and the best way to move forward is to >> rebuild the ubuntu kernel. It should be fairly straightforward according= to their >> wiki page <https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel>. The page >> mentions a fairly old release (19.04), but was updated a year ago so the >> instructions are likely still valid. >> >> >> >> However, I don't have the link to the kernel patch that Honnappa >> mentioned. @Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com> @Ruifeng >> Wang <Ruifeng.Wang@arm.com>, can you please provide a reference for the >> patch? >> >> *[Ruifeng]* Here is the kernel patch set: >> https://lkml.org/lkml/2022/6/17/328 >> >> >> >> Since the patch is small, there shouldn't be problems with applying it. >> Let us know whether this is doable. >> >> >> >> Regards, >> >> Juraj >> >> >> >> On Fri, Aug 4, 2023 at 11:48=E2=80=AFAM Ruifeng Wang <Ruifeng.Wang@arm.c= om> >> wrote: >> >> Hi Patrick, >> >> >> >> Thanks for reaching out and my apologies for delayed response. >> >> >> >> We noticed that some information is missing regarding using QAT with DPD= K >> on Arm. >> >> The DPDK document will be updated to include the missing part. >> >> Will get back on this later. >> >> >> >> Best regards, >> >> Ruifeng >> >> >> >> *From:* Patrick Robb <probb@iol.unh.edu> >> *Sent:* Tuesday, August 1, 2023 1:14 AM >> *To:* Ruifeng Wang <Ruifeng.Wang@arm.com>; Honnappa Nagarahalli < >> Honnappa.Nagarahalli@arm.com>; Juraj Linke=C5=A1 <juraj.linkes@pantheon.= tech> >> *Cc:* Dharmik Jayesh Thakkar <DharmikJayesh.Thakkar@arm.com>; ci@dpdk.or= g >> *Subject:* Intel QAT 8970 accel card on ARM Ampere Server >> >> >> >> Hi Ruifeng, Honnappa, Juraj, >> >> >> >> The Intel QAT 8970 accelerator card has arrived to the Community Lab, an= d >> we've installed it on the Ampere server. Presumably, we should test both >> crypto and compress operations (and their respective performance metrics= ). >> To that end, there are also DTS testsuites for testing QAT crypto/compre= ss >> functions. These testsuites make use of the crypto perf dpdk app and the >> compress perf dpdk app. If you want, you can setup the DTS stuff yoursel= f, >> both on the system side, and the Jenkins side (you are allowed to submit >> PRs on our gitlab now), but we can also do that on the lab side as we >> probably have more experience. I do, however, have a question about the = QAT >> kernel driver and corresponding PMDs. >> >> >> >> compress suite: >> https://git.dpdk.org/tools/dts/tree/test_plans/compressdev_qat_pmd_test_= plan.rst >> >> crypto suite: >> https://git.dpdk.org/tools/dts/tree/test_plans/crypto_perf_cryptodev_per= f_test_plan.rst >> >> >> >> For reference, the DPDK docs page explaining QAT driver capabilities and >> building the QAT PMDs (crypto sym, crypto asym, and compress) is here: >> https://doc.dpdk.org/guides/cryptodevs/qat.html#building-qat >> >> Some notes before I get to my main question: >> >> -The 8970 is a C62x device >> >> -OpenSSL (arm requires it for QAT) is installed >> >> -3 PFs are visible from lspci (expected) >> >> -SRIOV is enabled >> >> >> >> However, although the system is on a valid kernel version for the QAT >> driver, the kernel module for QAT is not loaded, so in trying to set up >> testing, I am unable to create the 16 VFs for the 3 PFs respectively, li= ke >> the example below: >> >> >> >> echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs >> echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs >> echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs >> >> >> >> There is also an option to download the firmware from the kernel firmwar= e >> repo and copy the qat binaries to /lib/firmware and start the qat module= s >> from there. I wasn't able to resolve the situation with this method, but= it >> also could have been user error on my part. >> >> >> >> There is an option to install using the IDZ QAT Driver >> <https://www.intel.com/content/www/us/en/developer/topic-technology/open= /quick-assist-technology/overview.html>, >> but it should not be required given the kernel version the Ampere server= is >> on, and I don't want to go down the road of relying on this "fall back" >> method without consulting you first. Is it possible that there is anythi= ng >> specific to running a QAT device on ARM specifically which I am missing >> here? The DTS testsuite testplans actually seem to recommend going down >> this road in general, but the DPDK docs say to use the kernel driver, so= I >> don't know. >> >> >> >> In any case, one of you should be able to login to the Ampere server in >> situations like this, or just in general. Ruifeng/Juraj I see you both h= ave >> accounts on our IdM system, so you should have access. Please let me kno= w >> if you need renewed vpn cert configs and I will send you one. If you do >> login, know this system could be running CI testing at any time. I can >> always schedule time for it to be offline and available for maintenance = if >> you want to do anything which could be disruptive to testing. >> >> >> >> I also CC'd Dharmik on this as I see he sent an email regarding QAT >> support on aarch64 in June. >> >> >> >> Let me know if you have any thoughts on the QAT kernel driver part. >> >> >> >> Thanks, >> >> Patrick >> >> > > -- > > Patrick Robb > > Technical Service Manager > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > www.iol.unh.edu > > > --00000000000056000a06036ae8a5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Hi Patrick,</div><div><br></div><div>On Fri, Aug 11, = 2023 at 11:18=E2=80=AFPM Patrick Robb <<a href=3D"mailto:probb@iol.unh.e= du">probb@iol.unh.edu</a>> wrote:<br></div><div class=3D"gmail_quote"><b= lockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-le= ft:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr">Sorry abou= t the wait on my reply guys.<div><br></div><div>Thanks for the information.= So I download the 2 diffs from that thread, make a patch with them. Then w= here and how do I apply it?=C2=A0</div><div><br></div></div></blockquote><d= iv><br></div><div>First get the ubuntu repo:</div><div><ul style=3D"color:r= gb(51,51,51);font-family:"Ubuntu Beta",UbuntuBeta,Ubuntu,"Bi= tstream Vera Sans","DejaVu Sans",Tahoma,sans-serif;font-size= :13px"><li style=3D"margin-bottom:10px;list-style-type:none"><pre style=3D"= width:864px;overflow-x:auto">git clone git://<a href=3D"http://kernel.ubunt= u.com/ubuntu/ubuntu-">kernel.ubuntu.com/ubuntu/ubuntu-</a><release coden= ame>.git</pre></li></ul><div><span style=3D"white-space:pre"><font face= =3D"arial, sans-serif">22.04 is jammy, but looking at </font></span><a href= =3D"https://kernel.ubuntu.com/git/">https://kernel.ubuntu.com/git/</a>, it&= #39;s not under ubuntu/ubuntu-jammy.git, but rather=C2=A0ubuntu-stable/ubun= tu-stable-jammy.git. It also seems the repo's been redirected:</div></d= iv><div><br></div></div><blockquote style=3D"margin:0 0 0 40px;border:none;= padding:0px"><div class=3D"gmail_quote"><div><font face=3D"monospace">git c= lone git://<a href=3D"http://kernel.ubuntu.com/ubuntu-stable/ubuntu-stable-= jammy.git">kernel.ubuntu.com/ubuntu-stable/ubuntu-stable-jammy.git</a></fon= t></div></div><div class=3D"gmail_quote"><div><font face=3D"monospace">Clon= ing into 'ubuntu-stable-jammy'...</font></div></div><div class=3D"g= mail_quote"><div><font face=3D"monospace">fatal: remote error: **REPOSITORY= RELOCATED** =C2=A0Updated URL: <a href=3D"https://git.launchpad.net/~ubunt= u-kernel-stable/+git/jammy">https://git.launchpad.net/~ubuntu-kernel-stable= /+git/jammy</a> Local path: /ubuntu-stable/ubuntu-stable-jammy.git</font></= div></div></blockquote><div class=3D"gmail_quote"><div><br></div><div>Cloni= ng the new URL worked for me. Then we need to checkout the tag that corresp= onds to the running kernel (uname -r), apply the patch and build the kernel= with the running config (in /boot/config-$(uname -r)), possibly enabling t= he QAT driver if needed.</div><div>=C2=A0</div><blockquote class=3D"gmail_q= uote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,2= 04);padding-left:1ex"><div dir=3D"ltr"><div></div><div>Then I install the p= ackages needed per the ubuntu page, and then I can skip down to the "B= uilding The Kernel" section? And then we're all set I think, and w= e just have to setup DTS and associated Jenkins pipelines.=C2=A0</div><div>= <br></div><div>Do you want me to back anything up in advance of this? I don= 't know if that is needed or not, but Ampere is currently live doing te= sting for CI, so I want to act in a safe manner. I will try to address this= first thing on Monday and get back to you.=C2=A0</div><div><br></div></div= ></blockquote><div><br></div><div>The backup should not be needed, at least= in principle, as we can always reinstall the original kernel packages.</di= v><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0p= x 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div d= ir=3D"ltr"><div></div><div><br></div></div><br><div class=3D"gmail_quote"><= div dir=3D"ltr" class=3D"gmail_attr">On Tue, Aug 8, 2023 at 3:11=E2=80=AFAM= Ruifeng Wang <<a href=3D"mailto:Ruifeng.Wang@arm.com" target=3D"_blank"= >Ruifeng.Wang@arm.com</a>> wrote:<br></div><blockquote class=3D"gmail_qu= ote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,20= 4);padding-left:1ex"><div> <div lang=3D"EN-US"> <div> <div style=3D"border-top:none;border-right:none;border-bottom:none;border-l= eft:1.5pt solid blue;padding:0mm 0mm 0mm 4pt"> <div> <div style=3D"border-right:none;border-bottom:none;border-left:none;border-= top:1pt solid rgb(225,225,225);padding:3pt 0mm 0mm"> <p class=3D"MsoNormal"><b>From:</b> Juraj Linke=C5=A1 <juraj.linkes@pant= heon.tech> <br> <b>Sent:</b> Tuesday, August 8, 2023 3:07 PM<br> <b>To:</b> Ruifeng Wang <<a href=3D"mailto:Ruifeng.Wang@arm.com" target= =3D"_blank">Ruifeng.Wang@arm.com</a>><br> <b>Cc:</b> Patrick Robb <<a href=3D"mailto:probb@iol.unh.edu" target=3D"= _blank">probb@iol.unh.edu</a>>; Honnappa Nagarahalli <<a href=3D"mail= to:Honnappa.Nagarahalli@arm.com" target=3D"_blank">Honnappa.Nagarahalli@arm= .com</a>>; Dharmik Jayesh Thakkar <<a href=3D"mailto:DharmikJayesh.Th= akkar@arm.com" target=3D"_blank">DharmikJayesh.Thakkar@arm.com</a>>; <a = href=3D"mailto:ci@dpdk.org" target=3D"_blank">ci@dpdk.org</a>; nd <<a hr= ef=3D"mailto:nd@arm.com" target=3D"_blank">nd@arm.com</a>><br> <b>Subject:</b> Re: Intel QAT 8970 accel card on ARM Ampere Server<u></u><u= ></u></p> </div> </div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <div> <div> <p class=3D"MsoNormal">We've talked about this some more and the best w= ay to move forward is to rebuild the ubuntu kernel. It should be fairly str= aightforward according to <a href=3D"https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel" target=3D"_bl= ank">their wiki page</a>. The page mentions a fairly old release (19.04), b= ut was updated a year ago so the instructions are likely still valid.<u></u= ><u></u></p> </div> <div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> </div> <div> <p class=3D"MsoNormal">However, I don't have the link to the kernel pat= ch that Honnappa mentioned.=C2=A0<a href=3D"mailto:Honnappa.Nagarahalli@arm= .com" target=3D"_blank">@Honnappa Nagarahalli</a>=C2=A0<a href=3D"mailto:Ru= ifeng.Wang@arm.com" target=3D"_blank">@Ruifeng Wang</a>, can you please pro= vide a reference for the patch?<u></u><u></u></p> <p class=3D"MsoNormal"><b>[Ruifeng]</b> Here is the kernel patch set: <a hr= ef=3D"https://lkml.org/lkml/2022/6/17/328" target=3D"_blank"> https://lkml.org/lkml/2022/6/17/328</a><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> </div> <div> <p class=3D"MsoNormal">Since the patch is small, there shouldn't be pro= blems with applying it. Let us know whether this is doable.<u></u><u></u></= p> </div> <div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> </div> <div> <p class=3D"MsoNormal">Regards,<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">Juraj<u></u><u></u></p> </div> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <div> <div> <p class=3D"MsoNormal">On Fri, Aug 4, 2023 at 11:48=E2=80=AFAM Ruifeng Wang= <<a href=3D"mailto:Ruifeng.Wang@arm.com" target=3D"_blank">Ruifeng.Wang= @arm.com</a>> wrote:<u></u><u></u></p> </div> <blockquote style=3D"border-top:none;border-right:none;border-bottom:none;b= order-left:1pt solid rgb(204,204,204);padding:0mm 0mm 0mm 6pt;margin-left:4= .8pt;margin-right:0mm"> <div> <div> <div> <p class=3D"MsoNormal">Hi Patrick,<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> <p class=3D"MsoNormal">Thanks for reaching out and my apologies for delayed= response.<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> <p class=3D"MsoNormal">We noticed that some information is missing regardin= g using QAT with DPDK on Arm.<u></u><u></u></p> <p class=3D"MsoNormal">The DPDK document will be updated to include the mis= sing part.<u></u><u></u></p> <p class=3D"MsoNormal">Will get back on this later.<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> <p class=3D"MsoNormal">Best regards,<u></u><u></u></p> <p class=3D"MsoNormal">Ruifeng<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> <div style=3D"border-top:none;border-right:none;border-bottom:none;border-l= eft:1.5pt solid blue;padding:0mm 0mm 0mm 4pt"> <div> <div style=3D"border-right:none;border-bottom:none;border-left:none;border-= top:1pt solid rgb(225,225,225);padding:3pt 0mm 0mm"> <p class=3D"MsoNormal"><b>From:</b> Patrick Robb <<a href=3D"mailto:prob= b@iol.unh.edu" target=3D"_blank">probb@iol.unh.edu</a>> <br> <b>Sent:</b> Tuesday, August 1, 2023 1:14 AM<br> <b>To:</b> Ruifeng Wang <<a href=3D"mailto:Ruifeng.Wang@arm.com" target= =3D"_blank">Ruifeng.Wang@arm.com</a>>; Honnappa Nagarahalli <<a href= =3D"mailto:Honnappa.Nagarahalli@arm.com" target=3D"_blank">Honnappa.Nagarah= alli@arm.com</a>>; Juraj Linke=C5=A1 <<a href=3D"mailto:juraj.linkes@= pantheon.tech" target=3D"_blank">juraj.linkes@pantheon.tech</a>><br> <b>Cc:</b> Dharmik Jayesh Thakkar <<a href=3D"mailto:DharmikJayesh.Thakk= ar@arm.com" target=3D"_blank">DharmikJayesh.Thakkar@arm.com</a>>; <a href=3D"mailto:ci@dpdk.org" target=3D"_blank">ci@dpdk.org</a><br> <b>Subject:</b> Intel QAT 8970 accel card on ARM Ampere Server<u></u><u></u= ></p> </div> </div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> <div> <p class=3D"MsoNormal">Hi Ruifeng, Honnappa, Juraj,<u></u><u></u></p> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">The Intel QAT 8970 accelerator card has arrived to t= he Community Lab, and we've installed it on the Ampere server. Presumab= ly, we should test both crypto and compress operations (and their respective performance metrics). To that end, there are also DT= S testsuites for testing QAT crypto/compress functions. These testsuites ma= ke use of the crypto perf dpdk app and the compress perf dpdk app. If you w= ant, you can setup the DTS stuff yourself, both on the system=C2=A0side, and the Jenkins side (you are allo= wed=C2=A0to submit PRs on our gitlab now), but we can also do that on the l= ab side as we probably have more experience. I do, however, have a question= about the QAT kernel driver and corresponding PMDs.<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">compress suite: <a href=3D"https://git.dpdk.org/tools/dts/tree/test_plans/compressdev_qat_p= md_test_plan.rst" target=3D"_blank"> https://git.dpdk.org/tools/dts/tree/test_plans/compressdev_qat_pmd_test_pla= n.rst</a><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">crypto suite: <a href=3D"https://git.dpdk.org/tools/dts/tree/test_plans/crypto_perf_crypt= odev_perf_test_plan.rst" target=3D"_blank"> https://git.dpdk.org/tools/dts/tree/test_plans/crypto_perf_cryptodev_perf_t= est_plan.rst</a><u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">For reference, the DPDK docs page explaining QAT dri= ver capabilities and building the QAT PMDs (crypto sym, crypto asym, and co= mpress) is here:=C2=A0<a href=3D"https://doc.dpdk.org/guides/cryptodevs/qat= .html#building-qat" target=3D"_blank">https://doc.dpdk.org/guides/cryptodev= s/qat.html#building-qat</a><br> <br> Some notes before I get to my main question:<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">-The 8970 is a C62x device<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">-OpenSSL (arm requires it for QAT) is installed<u></= u><u></u></p> </div> <div> <p class=3D"MsoNormal">-3 PFs are visible from lspci (expected)<u></u><u></= u></p> </div> <div> <div> <p class=3D"MsoNormal">-SRIOV is enabled<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">However, although the system is on a valid kernel ve= rsion for the QAT driver, the kernel module for QAT is not loaded, so in tr= ying to set up testing, I am unable to create the 16 VFs for the 3 PFs respectively, like the example below:=C2=A0<u></u><u>= </u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)= /sriov_numvfs<br> echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs<br> echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs<u></u><u>= </u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">There is also an option to download the firmware fro= m the kernel firmware repo and copy the qat binaries to /lib/firmware and s= tart the qat modules from there. I wasn't able to resolve the situation with this method, but it also could have been user e= rror on my part.=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">There is an option to install using the <a href=3D"https://www.intel.com/content/www/us/en/developer/topic-technolo= gy/open/quick-assist-technology/overview.html" target=3D"_blank"> IDZ QAT Driver</a>, but it should not be required given the kernel version = the Ampere server is on, and I don't want to go down the road of relyin= g on this "fall back" method without consulting you first. Is it = possible that there is anything specific to running a QAT device on ARM specifically which I am missing here? The DTS testsuit= e testplans actually seem to recommend going down this road in general, but= the DPDK docs say to use the kernel driver, so I don't know.<u></u><u>= </u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">In any case, one of you should be able to login to t= he Ampere server in situations like this, or just in general. Ruifeng/Juraj= I see you both have accounts on our IdM system, so you should have access. Please let me know if you need renewed vpn cert co= nfigs and I will send you one. If you do login, know this system could be r= unning CI testing at any time. I can always schedule time for it to be offl= ine and available for maintenance if you want to do anything which could be disruptive to testing.=C2=A0<u><= /u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">I also CC'd Dharmik on this as I see he sent an = email regarding QAT support on aarch64 in June.<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">Let me know if you have any thoughts on the QAT kern= el driver part.=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">Thanks,<u></u><u></u></p> </div> <div> <p class=3D"MsoNormal">Patrick=C2=A0<u></u><u></u></p> </div> </div> </div> </div> </div> </div> </div> </blockquote> </div> </div> </div> </div> </div> </div></blockquote></div><br clear=3D"all"><div><br></div><span class=3D"gm= ail_signature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signatu= re"><div dir=3D"ltr"><p dir=3D"ltr" style=3D"line-height:1.2;margin-top:0pt= ;margin-bottom:0pt"><font color=3D"#000000" face=3D"Arial"><span style=3D"f= ont-size:13.3333px;white-space:pre-wrap">Patrick Robb</span></font></p><p s= tyle=3D"color:rgb(34,34,34);line-height:1.2;margin-top:0pt;margin-bottom:0p= t"><span style=3D"font-size:10pt;font-family:Arial;color:rgb(0,0,0);backgro= und-color:transparent;vertical-align:baseline;white-space:pre-wrap">Technic= al Service Manager</span></p><p dir=3D"ltr" style=3D"color:rgb(34,34,34);li= ne-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style=3D"font-size:10= pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical= -align:baseline;white-space:pre-wrap">UNH InterOperability Laboratory</span= ></p><p dir=3D"ltr" style=3D"color:rgb(34,34,34);line-height:1.2;margin-top= :0pt;margin-bottom:0pt"><span style=3D"font-size:10pt;font-family:Arial;col= or:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-sp= ace:pre-wrap">21 Madbury Rd, Suite 100, Durham, NH 03824</span></p><p dir= =3D"ltr" style=3D"color:rgb(34,34,34);line-height:1.2;margin-top:0pt;margin= -bottom:0pt"><span style=3D"font-size:10pt;font-family:Arial;color:rgb(17,8= 5,204);background-color:transparent;vertical-align:baseline;white-space:pre= -wrap"><a href=3D"http://www.iol.unh.edu/" style=3D"color:rgb(17,85,204)" t= arget=3D"_blank">www.iol.unh.edu</a></span></p><p dir=3D"ltr" style=3D"colo= r:rgb(34,34,34);line-height:1.2;margin-top:0pt;margin-bottom:0pt"><br></p><= p dir=3D"ltr" style=3D"color:rgb(34,34,34);line-height:1.2;margin-top:0pt;m= argin-bottom:0pt"><span style=3D"font-size:10pt;font-family:Arial;color:rgb= (51,51,51);background-color:transparent;vertical-align:baseline;white-space= :pre-wrap"><img src=3D"https://lh4.googleusercontent.com/7sTY8VswXadak_YT0J= 13osh5ockNVRX2BuYaRsKoTTpkpilBokA0WlocYHLB4q7XUgXNHka6-ns47S8R_am0sOt7MYQQ1= ILQ3S-P5aezsrjp3-IsJMmMrErHWmTARNgZhpAx06n2" width=3D"150" height=3D"37" st= yle=3D"border: none;"></span></p></div></div> </blockquote></div></div> --00000000000056000a06036ae8a5--