Re: Intel QAT 8970 accel card on ARM Ampere Server

["Juraj Linkeš" <juraj.linkes@pantheon.tech>]

[-- Attachment #1: Type: text/plain, Size: 7856 bytes --]

Hi Patrick,

On Fri, Aug 11, 2023 at 11:18 PM Patrick Robb <probb@iol.unh.edu> wrote:

> Sorry about the wait on my reply guys.
>
> Thanks for the information. So I download the 2 diffs from that thread,
> make a patch with them. Then where and how do I apply it?
>
>
First get the ubuntu repo:

   -

   git clone git://kernel.ubuntu.com/ubuntu/ubuntu-<release codename>.git


22.04 is jammy, but looking at https://kernel.ubuntu.com/git/, it's not
under ubuntu/ubuntu-jammy.git, but
rather ubuntu-stable/ubuntu-stable-jammy.git. It also seems the repo's been
redirected:

git clone git://kernel.ubuntu.com/ubuntu-stable/ubuntu-stable-jammy.git
Cloning into 'ubuntu-stable-jammy'...
fatal: remote error: **REPOSITORY RELOCATED**  Updated URL:
https://git.launchpad.net/~ubuntu-kernel-stable/+git/jammy Local path:
/ubuntu-stable/ubuntu-stable-jammy.git


Cloning the new URL worked for me. Then we need to checkout the tag that
corresponds to the running kernel (uname -r), apply the patch and build the
kernel with the running config (in /boot/config-$(uname -r)), possibly
enabling the QAT driver if needed.


> Then I install the packages needed per the ubuntu page, and then I can
> skip down to the "Building The Kernel" section? And then we're all set I
> think, and we just have to setup DTS and associated Jenkins pipelines.
>
> Do you want me to back anything up in advance of this? I don't know if
> that is needed or not, but Ampere is currently live doing testing for CI,
> so I want to act in a safe manner. I will try to address this first thing
> on Monday and get back to you.
>
>
The backup should not be needed, at least in principle, as we can always
reinstall the original kernel packages.


>
>
> On Tue, Aug 8, 2023 at 3:11 AM Ruifeng Wang <Ruifeng.Wang@arm.com> wrote:
>
>> *From:* Juraj Linkeš <juraj.linkes@pantheon.tech>
>> *Sent:* Tuesday, August 8, 2023 3:07 PM
>> *To:* Ruifeng Wang <Ruifeng.Wang@arm.com>
>> *Cc:* Patrick Robb <probb@iol.unh.edu>; Honnappa Nagarahalli <
>> Honnappa.Nagarahalli@arm.com>; Dharmik Jayesh Thakkar <
>> DharmikJayesh.Thakkar@arm.com>; ci@dpdk.org; nd <nd@arm.com>
>> *Subject:* Re: Intel QAT 8970 accel card on ARM Ampere Server
>>
>>
>>
>> We've talked about this some more and the best way to move forward is to
>> rebuild the ubuntu kernel. It should be fairly straightforward according to their
>> wiki page <https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel>. The page
>> mentions a fairly old release (19.04), but was updated a year ago so the
>> instructions are likely still valid.
>>
>>
>>
>> However, I don't have the link to the kernel patch that Honnappa
>> mentioned. @Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com> @Ruifeng
>> Wang <Ruifeng.Wang@arm.com>, can you please provide a reference for the
>> patch?
>>
>> *[Ruifeng]* Here is the kernel patch set:
>> https://lkml.org/lkml/2022/6/17/328
>>
>>
>>
>> Since the patch is small, there shouldn't be problems with applying it.
>> Let us know whether this is doable.
>>
>>
>>
>> Regards,
>>
>> Juraj
>>
>>
>>
>> On Fri, Aug 4, 2023 at 11:48 AM Ruifeng Wang <Ruifeng.Wang@arm.com>
>> wrote:
>>
>> Hi Patrick,
>>
>>
>>
>> Thanks for reaching out and my apologies for delayed response.
>>
>>
>>
>> We noticed that some information is missing regarding using QAT with DPDK
>> on Arm.
>>
>> The DPDK document will be updated to include the missing part.
>>
>> Will get back on this later.
>>
>>
>>
>> Best regards,
>>
>> Ruifeng
>>
>>
>>
>> *From:* Patrick Robb <probb@iol.unh.edu>
>> *Sent:* Tuesday, August 1, 2023 1:14 AM
>> *To:* Ruifeng Wang <Ruifeng.Wang@arm.com>; Honnappa Nagarahalli <
>> Honnappa.Nagarahalli@arm.com>; Juraj Linkeš <juraj.linkes@pantheon.tech>
>> *Cc:* Dharmik Jayesh Thakkar <DharmikJayesh.Thakkar@arm.com>; ci@dpdk.org
>> *Subject:* Intel QAT 8970 accel card on ARM Ampere Server
>>
>>
>>
>> Hi Ruifeng, Honnappa, Juraj,
>>
>>
>>
>> The Intel QAT 8970 accelerator card has arrived to the Community Lab, and
>> we've installed it on the Ampere server. Presumably, we should test both
>> crypto and compress operations (and their respective performance metrics).
>> To that end, there are also DTS testsuites for testing QAT crypto/compress
>> functions. These testsuites make use of the crypto perf dpdk app and the
>> compress perf dpdk app. If you want, you can setup the DTS stuff yourself,
>> both on the system side, and the Jenkins side (you are allowed to submit
>> PRs on our gitlab now), but we can also do that on the lab side as we
>> probably have more experience. I do, however, have a question about the QAT
>> kernel driver and corresponding PMDs.
>>
>>
>>
>> compress suite:
>> https://git.dpdk.org/tools/dts/tree/test_plans/compressdev_qat_pmd_test_plan.rst
>>
>> crypto suite:
>> https://git.dpdk.org/tools/dts/tree/test_plans/crypto_perf_cryptodev_perf_test_plan.rst
>>
>>
>>
>> For reference, the DPDK docs page explaining QAT driver capabilities and
>> building the QAT PMDs (crypto sym, crypto asym, and compress) is here:
>> https://doc.dpdk.org/guides/cryptodevs/qat.html#building-qat
>>
>> Some notes before I get to my main question:
>>
>> -The 8970 is a C62x device
>>
>> -OpenSSL (arm requires it for QAT) is installed
>>
>> -3 PFs are visible from lspci (expected)
>>
>> -SRIOV is enabled
>>
>>
>>
>> However, although the system is on a valid kernel version for the QAT
>> driver, the kernel module for QAT is not loaded, so in trying to set up
>> testing, I am unable to create the 16 VFs for the 3 PFs respectively, like
>> the example below:
>>
>>
>>
>> echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs
>> echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs
>> echo 16 > /sys/bus/pci/drivers/c6xx/(pci address)/sriov_numvfs
>>
>>
>>
>> There is also an option to download the firmware from the kernel firmware
>> repo and copy the qat binaries to /lib/firmware and start the qat modules
>> from there. I wasn't able to resolve the situation with this method, but it
>> also could have been user error on my part.
>>
>>
>>
>> There is an option to install using the IDZ QAT Driver
>> <https://www.intel.com/content/www/us/en/developer/topic-technology/open/quick-assist-technology/overview.html>,
>> but it should not be required given the kernel version the Ampere server is
>> on, and I don't want to go down the road of relying on this "fall back"
>> method without consulting you first. Is it possible that there is anything
>> specific to running a QAT device on ARM specifically which I am missing
>> here? The DTS testsuite testplans actually seem to recommend going down
>> this road in general, but the DPDK docs say to use the kernel driver, so I
>> don't know.
>>
>>
>>
>> In any case, one of you should be able to login to the Ampere server in
>> situations like this, or just in general. Ruifeng/Juraj I see you both have
>> accounts on our IdM system, so you should have access. Please let me know
>> if you need renewed vpn cert configs and I will send you one. If you do
>> login, know this system could be running CI testing at any time. I can
>> always schedule time for it to be offline and available for maintenance if
>> you want to do anything which could be disruptive to testing.
>>
>>
>>
>> I also CC'd Dharmik on this as I see he sent an email regarding QAT
>> support on aarch64 in June.
>>
>>
>>
>> Let me know if you have any thoughts on the QAT kernel driver part.
>>
>>
>>
>> Thanks,
>>
>> Patrick
>>
>>
>
> --
>
> Patrick Robb
>
> Technical Service Manager
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> www.iol.unh.edu
>
>
>

[-- Attachment #2: Type: text/html, Size: 16344 bytes --]