VMXNET3 driver IOVA support

[Igor de Paula <igordptx@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 5438 bytes --]

Hi,
I am using a Vmware host (ESXI 7.0) where I have IOMMU enabled. I am using
virtio_user ports as an exception path so I am trying to use IOVA as VA.
Just as an example, I am trying to run the l2fw example application with
--iova-mode=va.
Although I shows there is support for IOMMU, it fails to negotiate the
IOMMU type, here is the output of: ./dpdk-l2fwd --iova-mode=va
--log-level=eal,8

EAL: lib.eal log level changed from info to debug
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 2 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Checking presence of .so 'librte_eal.so.21.3'
EAL: Checking presence of .so 'librte_eal.so.21'
EAL: Checking presence of .so 'librte_eal.so'
EAL: Detected static linkage of DPDK
EAL: Ask a virtual area of 0x7000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x7000)
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: DPAA Bus not present. Skipping.
EAL: VFIO PCI modules not loaded
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
*EAL: IOMMU type 1 (Type 1) is supported*
EAL: IOMMU type 7 (sPAPR) is not supported
EAL: IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x5b000 bytes
EAL: Virtual area found at 0x100007000 (size = 0x5b000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
EAL: Creating 2 segment lists: n_segs:128 socket_id:0 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x2000 bytes
EAL: Virtual area found at 0x100062000 (size = 0x2000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x2000000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x2000000000)
EAL: VA reserved for memseg list at 0x140000000, size 2000000000
EAL: Ask a virtual area of 0x2000 bytes
EAL: Virtual area found at 0x2140000000 (size = 0x2000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x2000000000 bytes
EAL: Virtual area found at 0x2180000000 (size = 0x2000000000)
EAL: VA reserved for memseg list at 0x2180000000, size 2000000000
EAL: TSC frequency is ~2350000 KHz
EAL: Main lcore 0 is ready (tid=7f3f3a1e5c00;cpuset=[0])
EAL: lcore 1 is ready (tid=7f3f391de400;cpuset=[1])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 1024MB
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 net_vmxnet3
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:0b:00.0 on NUMA socket -1
EAL:   probe driver: 15ad:7b0 net_vmxnet3
EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)
EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device)
EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such device)
EAL: 0000:0b:00.0 failed to select IOMMU type
EAL: Requested device 0000:0b:00.0 cannot be used
EAL: PCI device 0000:13:00.0 on NUMA socket -1
*EAL:   probe driver: 15ad:7b0 net_vmxnet3*


*EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device)EAL: Set
IOMMU type 7 (sPAPR) failed, error 19 (No such device)EAL: Set IOMMU type 8
(No-IOMMU) failed, error 19 (No such device)*
EAL: 0000:13:00.0 failed to select IOMMU type
EAL: Requested device 0000:13:00.0 cannot be used
EAL: lib.telemetry log level changed from disabled to warning
MAC updating enabled
EAL: Error - exiting with code: 1
  Cause: No Ethernet ports - bye
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was shrunk by 1024MB

The negotiation for IOMMU type happens in rte_vfio_setup_device(), it calls
the function:
set_iommu_type() in the following snippet:

/*
* pick an IOMMU type and set up DMA mappings for container
*
* needs to be done only once, only when first group is
* assigned to a container and only in primary process.
* Note this can happen several times with the hotplug
* functionality.
*/
if (internal_conf->process_type == RTE_PROC_PRIMARY &&
vfio_cfg->vfio_active_groups == 1 &&
vfio_group_device_count(vfio_group_fd) == 0) {
const struct vfio_iommu_type *t;

/* select an IOMMU type which we will be using */
t = vfio_set_iommu_type(vfio_container_fd);
if (!t) {
RTE_LOG(ERR, EAL,
"%s failed to select IOMMU type\n",
dev_addr);
close(vfio_group_fd);
rte_vfio_clear_group(vfio_group_fd);
return -1;
}

In there there is a ioctl call:
unsigned idx;
for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
const struct vfio_iommu_type *t = &iommu_types[idx];

int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
t->type_id);

which fails.

Its also weird for me the when using ./usertool/dpdk-devbind.py --status
I gett the following output:
Network devices using DPDK-compatible driver
============================================

*0000:0b:00.0 'VMXNET3 Ethernet Controller 07b0' drv=vfio-pci
unused=vmxnet30000:13:00.0 'VMXNET3 Ethernet Controller 07b0' drv=vfio-pci
unused=vmxnet3*

Network devices using kernel driver
===================================
0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' if=eth0 drv=vmxnet3
unused=vfio-pci *Active*

Meaning the driver that is connected is vfio-pci. So why is it probing
VMXNET3?

Does anyone have any experience in running IOVA as VA in VMWARE host?

[-- Attachment #2: Type: text/html, Size: 6104 bytes --]