RE: Intel QAT 8970 accel card on ARM Ampere Server

[Dharmik Jayesh Thakkar <DharmikJayesh.Thakkar@arm.com>]

[-- Attachment #1: Type: text/plain, Size: 4469 bytes --]

Thank you for the details, Patrick!
Yeah can you please update the grub and vfio settings and see if it works. I don’t think it should have any implications on other NICs.

From: Patrick Robb <probb@iol.unh.edu>
Sent: Tuesday, October 10, 2023 11:00 AM
To: Dharmik Jayesh Thakkar <DharmikJayesh.Thakkar@arm.com>; David Marchand <david.marchand@redhat.com>
Cc: Ruifeng Wang <Ruifeng.Wang@arm.com>; Juraj Linkeš <juraj.linkes@pantheon.tech>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; ci@dpdk.org; nd <nd@arm.com>
Subject: Re: Intel QAT 8970 accel card on ARM Ampere Server



On Mon, Oct 9, 2023 at 11:56 PM Dharmik Jayesh Thakkar <DharmikJayesh.Thakkar@arm.com<mailto:DharmikJayesh.Thakkar@arm.com>> wrote:
Hi Patrick,

Can you provide the grub settings? Is iommu.passthrough=1 included?

Sure. I'm not sure if you just wanted the kernel cmdline options or the whole grub config, but I assume you just meant kernel cmdline. Let me know if you meant more.

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=39-79 nohz_full=39-79 rcu_nocbs=39-79 processor.max_cstate=1 intel_pstate=disable console=ttyS0,115200 console=tty0"

But, iommu.passthrough=1 is not included, so I can add that if we need to. Do you know that this won't have any bad implications for the (intel, nvidia, broadcom) NICs which we test on this server?


Also, is qat_c62xvf loaded as well?
qat_c62xvf is built in to the kernel also.



Finally, a few guidelines on the vfio driver:
At times, we need to configure the vfio driver.
On kernel vers. 5.9+ we need to load the vfio-pci driver with the additional parameter disable_denylist=1
Unload the vfio-pci driver if it is already loaded so that we can reload it with the correct parameters :
sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo modprobe -r vfio_virqfd; sudo modprobe -r vfio
If you can't unload the vfio driver because it's been built into the kernel, you'll have to find another way to change VFIO parameters, or to rebuild your kernel with VFIO_PCI set as a module. Failing to do that, you might encounter issues later on when you try to bind the VFs to VFIO.

Load the vfio-pci driver and bind it to QAT VFs device ids:
sudo modprobe vfio-pci disable_denylist=1 enable_sriov=1 vfio-pci.ids=8086:37c9

Enable no-iommu-mode:
echo "1" | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
 /sys/module/vfio/parameter is missing ?
If /sys/module/vfio/parameters does not exist, you might be missing the kernel module VFIO_NOIOMMU

Automatically set VFIO params on boot
It's possible to set these parameters automatically on boot by creating a /etc/modprobe.d/vfio-pci.conf file with the parameters :
cat /etc/modprobe.d/vfio-pci.conf
options vfio enable_unsafe_noiommu_mode=1
options vfio-pci disable_denylist=1 enable_sriov=1 vfio-pci.ids=8086:37c9

We haven’t encountered this issue in the past, so just making sure the configuration is correct. I don’t think having the driver static/loadable should make a difference, I will try with building statically on my setup.

Thank you!

Okay, this should be fine. Like I said, we are also running tests on NICs on this server. So, in our Jenkinsfiles scripts for running the testing, I will add a preliminary step only for QAT tests which runs:
sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo modprobe -r vfio_virqfd; sudo modprobe -r vfio
sudo modprobe vfio-pci disable_denylist=1 enable_sriov=1 vfio-pci.ids=8086:37c9
echo "1" | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
(then run QAT tests)

And if running on NICs, have a preliminary step which runs
sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo modprobe -r vfio_virqfd; sudo modprobe -r vfio
sudo modprobe vfio

David does this also sound reasonable to you, per your comment about isolating this setting to QAT card testing?

Dharmik if this all sounds okay and you can confirm the iommu.passthrough change is fine, I will proceed. Thank you for providing the assistance.

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[-- Attachment #2: Type: text/html, Size: 10876 bytes --]