Re: help

[Ferruh Yigit <ferruh.yigit@amd.com>]

Hi Igor,

VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you
please try with:
- enable 'enable_unsafe_noiommu_mode' flag
- Force PA mode via '--iova-mode=pa' eal parameter


Also to be able to figure out AMD IOMMU support level, can you please
provide:
- AMD part number
- ESXi/hypervisor version
- ESXi & VM kernel logs


Thanks,
Ferruh

On 7/20/2023 5:21 PM, Jochen Behrens wrote:
> +Ronak from the ESX team
> 
>  
> 
>  
> 
> In our usage, we do set amd_iommu=off in the boot command line from
> grub. (Or intel_iommu=off for Intel processors.)
> 
>  
> 
>                 Jochen
> 
>  
> 
> *From: *Thomas Monjalon <thomas@monjalon.net>
> *Date: *Thursday, July 20, 2023 at 6:00 AM
> *To: *Igor de Paula <igordptx@gmail.com>
> *Cc: *users@dpdk.org <users@dpdk.org>, Jochen Behrens
> <jbehrens@vmware.com>, Nipun Gupta <nipun.gupta@amd.com>, Nikhil Agarwal
> <nikhil.agarwal@amd.com>, Ferruh Yigit <ferruh.yigit@amd.com>
> *Subject: *Re: help
> 
> !! External Email
> 
> +Cc some AMD maintainers, they can have an idea about IOMMU settings.
> 
> 
> 20/07/2023 14:44, Igor de Paula:
>> I have enabled it in the host and in the BIOS for AMD...
>> In the Bios I changed to amd_iommu=on and in the host it's the same for
>> either.
>>
>> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>>
>> > 20/07/2023 11:35, Igor de Paula:
>> > > The weird thing is that it only happens when I am using a host with an
>> > AMD
>> > > processor. It doesn't happen when I use a host with an Intel processor.
>> >
>> > So it's probably a matter of BIOS settings for the IOMMU?
>> >
>> >
>> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon <thomas@monjalon.net>
>> > > wrote:
>> > >
>> > > > +Cc the vmxnet3 maintainer.
>> > > >
>> > > > Please Jochen, do you have an idea what's wrong below?
>> > > >
>> > > >
>> > > > 20/07/2023 11:25, Igor de Paula:
>> > > > > This is because it can't negotiate the IOMMU type with any port.
>> > > > >
>> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon <thomas@monjalon.net
>> > >
>> > > > wrote:
>> > > > >
>> > > > > > Hello,
>> > > > > >
>> > > > > > The first error is "Cause: Error: number of ports must be even"
>> > > > > >
>> > > > > >
>> > > > > > 03/05/2023 18:13, Igor de Paula:
>> > > > > > > I am running a VM inside a VMWARE server (vSphere).
>> > > > > > > My goal it to set up DPDK with two HW ports, and set up a
>> > > > virtio_user to
>> > > > > > > interact with the kernel stack.
>> > > > > > > In another app I have it working but instead of virtio_user I am
>> > > > running
>> > > > > > > KNI, it works in IOVA-PA mode.
>> > > > > > > I am looking to replace the KNI.
>> > > > > > >
>> > > > > > > When I try to set up virtio_user port as in the doc:
>> > > > > > >
>> > > > > >
>> > > >
>> > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path <https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path>
>> > > > > > > I get a error it can't run in PA mode.
>> > > > > > >
>> > > > > > >
>> > > > > > > When I try to run as VA mode from a parameter, I get the
>> > following
>> > > > > > errors:
>> > > > > > > EAL: lib.eal log level changed from info to debug
>> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0
>> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0
>> > > > > > > EAL: Support maximum 128 logical core(s) by configuration.
>> > > > > > > EAL: Detected 2 lcore(s)
>> > > > > > > EAL: Detected 1 NUMA nodes
>> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3'
>> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21'
>> > > > > > > EAL: Checking presence of .so 'librte_eal.so'
>> > > > > > > EAL: Detected static linkage of DPDK
>> > > > > > > EAL: Ask a virtual area of 0x7000 bytes
>> > > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000)
>> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> > > > > > > EAL: DPAA Bus not present. Skipping.
>> > > > > > > EAL: VFIO PCI modules not loaded
>> > > > > > > EAL: Selected IOVA mode 'VA'
>> > > > > > > EAL: Probing VFIO support...
>> > > > > > > EAL: IOMMU type 1 (Type 1) is supported
>> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported
>> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported
>> > > > > > > EAL: VFIO support initialized
>> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes
>> > > > > > > EAL: Virtual area found at 0x100007000 (size = 0x5b000)
>> > > > > > > EAL: Setting up physically contiguous memory...
>> > > > > > > EAL: Setting maximum number of open files to 1048576
>> > > > > > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
>> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0
>> > > > > > hugepage_sz:1073741824
>> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
>> > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000)
>> > > > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
>> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
>> > > > > > > EAL: Virtual area found at 0x140000000 (size = 0x2000000000)
>> > > > > > > EAL: VA reserved for memseg list at 0x140000000, size 2000000000
>> > > > > > > EAL: Ask a virtual area of 0x2000 bytes
>> > > > > > > EAL: Virtual area found at 0x2140000000 (size = 0x2000)
>> > > > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB
>> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes
>> > > > > > > EAL: Virtual area found at 0x2180000000 (size = 0x2000000000)
>> > > > > > > EAL: VA reserved for memseg list at 0x2180000000, size 2000000000
>> > > > > > > EAL: TSC frequency is ~2350000 KHz
>> > > > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0])
>> > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1])
>> > > > > > > EAL: Trying to obtain current memory policy.
>> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0
>> > > > > > > EAL: Restoring previous memory policy: 0
>> > > > > > > EAL: request: mp_malloc_sync
>> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
>> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
>> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
>> > > > > > initializing
>> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
>> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
>> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not
>> > > > > > initializing
>> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
>> > > > > > > EAL: Bus (pci) probe failed.
>> > > > > > > EAL: lib.telemetry log level changed from disabled to warning
>> > > > > > > EAL: Error - exiting with code: 1
>> > > > > > >   Cause: Error: number of ports must be even
>> > > > > > > EAL: request: mp_malloc_sync
>> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > For some reason the HW ports won't setup. From what I understand
>> > > > > > > net_vmxnet3 should work with VA mode.
>> > > > > > > I enabled I/OMUU for the VM.
>> > > > > > > The weird thing even when enabled, I still have the
>> > > > > > > enable_unsafe_noiommu_mode flag on.
>> > > > > > > And because it's on the this:
>> > > > > > >
>> > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev);
>> > > > > > >
>> > > > > > > return PA mode, and it fails.
>> > > > > > >
>> > > > > > > When I disable it by modifying
>> > > > > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get
>> > another
>> > > > > > error.
>> > > > > > > The error is that it doesn't find a suitable IOMMU type:
>> > > > > > > Just putting the relevant message:
>> > > > > > >
>> > > > > > >
>> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB
>> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1
>> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
>> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
>> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such
>> > device)
>> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type
>> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used
>> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1
>> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3
>> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
>> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
>> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such
>> > device)
>> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type
>> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used
>> > > > > > >
>> > > > > > >
>> > > > > > > I can see where it fails:
>> > > > > > >
>> > > > > > >
>> > > > > > > const struct vfio_iommu_type *
>> > > > > > > vfio_set_iommu_type(int vfio_container_fd)
>> > > > > > > {
>> > > > > > > unsigned idx;
>> > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
>> > > > > > > const struct vfio_iommu_type *t = &iommu_types[idx];
>> > > > > > >
>> > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
>> > > > > > > t->type_id);
>> > > > > > > if (!ret) {
>> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n",
>> > > > > > > t->type_id, t->name);
>> > > > > > > return t;
>> > > > > > > }
>> > > > > > > /* not an error, there may be more supported IOMMU types */
>> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error "
>> > > > > > > "%i (%s)\n", t->type_id, t->name, errno,
>> > > > > > > strerror(errno));
>> > > > > > > }
>> > > > > > > /* if we didn't find a suitable IOMMU type, fail */
>> > > > > > > return NULL;
>> > > > > > > }
>> > > > > > >
>> > > > > > >
>> > > > > > > The ioctl returns -1.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > I tried many things.
>> > > > > > > Enabling/disabling IOMMU.
>> > > > > > > Modifying the enable_unsafe_noiommu_mode flag.
>> > > > > > > Running in PA mode.
>> > > > > > > Any help will be much appreciated.
>> >
>> >
>> >
>> >
>> >
>> >
>>
> 
> 
> 
> 
> 
> 
> !! External Email: This email originated from outside of the
> organization. Do not click links or open attachments unless you
> recognize the sender.
>