On Tue, Oct 10, 2023 at 5:59 PM Patrick Robb <probb@iol.unh.edu> wrote:

>
>
> On Mon, Oct 9, 2023 at 11:56 PM Dharmik Jayesh Thakkar <
> DharmikJayesh.Thakkar@arm.com> wrote:
>
>> Hi Patrick,
>>
>>
>>
>> Can you provide the grub settings? Is iommu.passthrough=1 included?
>>
>
> Sure. I'm not sure if you just wanted the kernel cmdline options or the
> whole grub config, but I assume you just meant kernel cmdline. Let me know
> if you meant more.
>
> GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G
> hugepages=32 iommu=pt intel_iommu=on isolcpus=39-79 nohz_full=39-79
> rcu_nocbs=39-79 processor.max_cstate=1 intel_pstate=disable
> console=ttyS0,115200 console=tty0"
>
> But, iommu.passthrough=1 is not included, so I can add that if we need to.
> Do you know that this won't have any bad implications for the (intel,
> nvidia, broadcom) NICs which we test on this server?
>
>

Just a note here, Patrick. The iommu kernel and intel_pstate parameters
aren't supported on arm, so you can remove those. And when
iommu.passthrouh=1, IOMMU is bypassed and intel_iommu doesn't do anything
(and maybe isn't supported on arm, but that's not clear from the docs
<https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt>),
so that can be removed as well.

From what I can tell, using iommu.passthrough=1 is the standard, so if
there are any negative implications, we should investigate them, but there
shouldn't be anything major.


>
>>
>> Also, is qat_c62xvf loaded as well?
>>
> qat_c62xvf is built in to the kernel also.
>
>
>>
>>
>
>>
>> Finally, a few guidelines on the vfio driver:
>>
>> At times, we need to configure the vfio driver.
>>
>> On kernel vers. 5.9+ we need to load the vfio-pci driver with the
>> additional parameter *disable_denylist=1*
>>
>> Unload the vfio-pci driver if it is already loaded so that we can reload
>> it with the correct parameters :
>> *sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo
>> modprobe -r vfio_virqfd; sudo modprobe -r vfio*
>>
>> If you can't unload the vfio driver because it's been built into the
>> kernel, you'll have to find another way to change VFIO parameters, or to
>> rebuild your kernel with VFIO_PCI set as a module. Failing to do that, you
>> might encounter issues later on when you try to bind the VFs to VFIO.
>>
>> Load the vfio-pci driver and bind it to QAT VFs device ids:
>> *sudo modprobe vfio-pci disable_denylist=1 enable_sriov=1
>> vfio-pci.ids=8086:37c9*
>>
>> Enable no-iommu-mode:
>> *echo "1" | sudo tee
>> /sys/module/vfio/parameters/enable_unsafe_noiommu_mode*
>>
>>  /sys/module/vfio/parameter is missing ?
>>
>> If /sys/module/vfio/parameters does not exist, you might be missing the
>> kernel module VFIO_NOIOMMU
>>
>>
>>
>> *Automatically set VFIO params on boot*
>>
>> It's possible to set these parameters automatically on boot by creating a
>> */etc/modprobe.d/vfio-pci.conf *file with the parameters :
>> *cat /etc/modprobe.d/vfio-pci.conf*
>> *options vfio enable_unsafe_noiommu_mode=1*
>> *options vfio-pci disable_denylist=1 enable_sriov=1
>> vfio-pci.ids=8086:37c9*
>>
>>
>>
>> We haven’t encountered this issue in the past, so just making sure the
>> configuration is correct. I don’t think having the driver static/loadable
>> should make a difference, I will try with building statically on my setup.
>>
>>
>>
>> Thank you!
>>
>>
>> Okay, this should be fine. Like I said, we are also running tests on NICs
> on this server. So, in our Jenkinsfiles scripts for running the testing, I
> will add a preliminary step only for QAT tests which runs:
> *sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo
> modprobe -r vfio_virqfd; sudo modprobe -r vfio*
> *sudo modprobe vfio-pci disable_denylist=1 enable_sriov=1
> vfio-pci.ids=8086:37c9*
> *echo "1" | sudo tee
> /sys/module/vfio/parameters/enable_unsafe_noiommu_mode*
> (then run QAT tests)
>
> And if running on NICs, have a preliminary step which runs
> *sudo modprobe -r vfio_iommu_type1; sudo modprobe -r vfio_pci; sudo
> modprobe -r vfio_virqfd; sudo modprobe -r vfio*
> *sudo modprobe vfio*
>
> David does this also sound reasonable to you, per your comment about
> isolating this setting to QAT card testing?
>
> Dharmik if this all sounds okay and you can confirm the iommu.passthrough
> change is fine, I will proceed. Thank you for providing the assistance.
>
>