On Wed, Oct 11, 2023 at 4:14 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:


On Tue, Oct 10, 2023 at 5:59 PM Patrick Robb <probb@iol.unh.edu> wrote:


On Mon, Oct 9, 2023 at 11:56 PM Dharmik Jayesh Thakkar <DharmikJayesh.Thakkar@arm.com> wrote:

Hi Patrick,

 

Can you provide the grub settings? Is iommu.passthrough=1 included?


Sure. I'm not sure if you just wanted the kernel cmdline options or the whole grub config, but I assume you just meant kernel cmdline. Let me know if you meant more.

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=39-79 nohz_full=39-79 rcu_nocbs=39-79 processor.max_cstate=1 intel_pstate=disable console=ttyS0,115200 console=tty0"

But, iommu.passthrough=1 is not included, so I can add that if we need to. Do you know that this won't have any bad implications for the (intel, nvidia, broadcom) NICs which we test on this server?
 

Just a note here, Patrick. The iommu kernel and intel_pstate parameters aren't supported on arm, so you can remove those. And when iommu.passthrouh=1, IOMMU is bypassed and intel_iommu doesn't do anything (and maybe isn't supported on arm, but that's not clear from the docs), so that can be removed as well.

Thanks Dharmik and Juraj. Updated kernel cmdline args: 
BOOT_IMAGE=/vmlinuz-5.15.82+ root=/dev/mapper/ubuntu--vg--1-ubuntu--lv ro default_hugepagesz=1G hugepagesz=1G hugepages=32 isolcpus=39-79 nohz_full=39-79 rcu_nocbs=39-79 processor.max_cstate=1 iommu.passthrough=1 console=ttyS0,115200 console=tty0

I added the iommu.passthrough option and tried again, to no avail. FYI I am still using the guidance here: https://doc.dpdk.org/guides/cryptodevs/qat.html along with your added steps.

root@arm-ampere-dut:~# echo 16 > /sys/bus/pci/drivers/c6xx/0000:03:00.0/sriov_numvfs
Segmentation fault (core dumped)

As you know the above setting of the 48 VFs is a prerequisite to binding the VFs to vfio-pci. But, I did run through loading the custom vfio and there were no issues, so once we clear this initial hurdle we should be fine.