On Wed, Oct 11, 2023 at 4:14 AM Juraj Linkeš wrote: > > > On Tue, Oct 10, 2023 at 5:59 PM Patrick Robb wrote: > >> >> >> On Mon, Oct 9, 2023 at 11:56 PM Dharmik Jayesh Thakkar < >> DharmikJayesh.Thakkar@arm.com> wrote: >> >>> Hi Patrick, >>> >>> >>> >>> Can you provide the grub settings? Is iommu.passthrough=1 included? >>> >> >> Sure. I'm not sure if you just wanted the kernel cmdline options or the >> whole grub config, but I assume you just meant kernel cmdline. Let me know >> if you meant more. >> >> GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G >> hugepages=32 iommu=pt intel_iommu=on isolcpus=39-79 nohz_full=39-79 >> rcu_nocbs=39-79 processor.max_cstate=1 intel_pstate=disable >> console=ttyS0,115200 console=tty0" >> >> But, iommu.passthrough=1 is not included, so I can add that if we need >> to. Do you know that this won't have any bad implications for the (intel, >> nvidia, broadcom) NICs which we test on this server? >> >> > > Just a note here, Patrick. The iommu kernel and intel_pstate parameters > aren't supported on arm, so you can remove those. And when > iommu.passthrouh=1, IOMMU is bypassed and intel_iommu doesn't do anything > (and maybe isn't supported on arm, but that's not clear from the docs > ), > so that can be removed as well. > Thanks Dharmik and Juraj. Updated kernel cmdline args: BOOT_IMAGE=/vmlinuz-5.15.82+ root=/dev/mapper/ubuntu--vg--1-ubuntu--lv ro default_hugepagesz=1G hugepagesz=1G hugepages=32 isolcpus=39-79 nohz_full=39-79 rcu_nocbs=39-79 processor.max_cstate=1 iommu.passthrough=1 console=ttyS0,115200 console=tty0 I added the iommu.passthrough option and tried again, to no avail. FYI I am still using the guidance here: https://doc.dpdk.org/guides/cryptodevs/qat.html along with your added steps. root@arm-ampere-dut:~# echo 16 > /sys/bus/pci/drivers/c6xx/0000:03:00.0/sriov_numvfs Segmentation fault (core dumped) As you know the above setting of the 48 VFs is a prerequisite to binding the VFs to vfio-pci. But, I did run through loading the custom vfio and there were no issues, so once we clear this initial hurdle we should be fine.