[dpdk-users] Error when running DPDK in ESXi Centos guest.

[Paul Sprague <psprague@centripetalnetworks.com>]

Hello!

We're trying to run DPDK example apps in a guest machine running Centos
7.5.  The host is ESXi version 6.5.

I'm building dpdk on the guest machine where I'm trying to run it.  I've
tried both DPDK versions 18.05 and 18.08.

We have created five interfaces on esxi for connection to our guest.  One
management port and four data ports.  We're binding theses four data ports
to DPDK.  The ports are all VMXNET3 interfaces.  They are basically setup
like the VMXNET3 interfaces in [
https://doc.dpdk.org/guides/nics/vmxnet3.html], using a vswitch to connect
to a physical interface.  However note that we do not have any VF
interfaces as shown in this document, only VMXNET3 interfaces.

Fundamentally, the roadblock we are hitting is that the VMXNET3 interfaces
are failing initialization when starting the DPDK example app.  Here is
what we see:

[root@rg-vm ~]# ./dpdk-18.08/examples/packet_ordering/build/packet_ordering
-c 0x0e0 -- -p 0xf
EAL: Detected 24 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Probing VFIO support...
EAL: PCI device 0000:04:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 15ad:7b0 net_vmxnet3
*eth_vmxnet3_dev_init(): Incompatible hardware version: 0*
*EAL: Requested device 0000:04:00.0 cannot be used*

We see this for all four interfaces that we are trying to bind to DPDK.
However, strangely, *sometimes* after a reboot, the first two interfaces
initialize correctly.  But after that first attempt, all four interfaces
then fail the same way.

Here are the commands we're using to setup DPDK.

modprobe uio
insmod ./dpdk-18.08/build/build/kernel/linux/igb_uio/igb_uio.ko
./dpdk-18.08/usertools/dpdk-devbind.py --bind=igb_uio 04:00.0
./dpdk-18.08/usertools/dpdk-devbind.py --bind=igb_uio 0c:00.0
./dpdk-18.08/usertools/dpdk-devbind.py --bind=igb_uio 13:00.0
./dpdk-18.08/usertools/dpdk-devbind.py --bind=igb_uio 1b:00.0

Note that we have also tried using the uio_pci_generic with the same
results.  We have not been able to get the vfio-pci driver to bind to the
VMXNET3 interfaces.

I'm not sure it matters, but the physical interfaces on the other side of
the vswitch that we're connecting to are:
17:00.0 Ethernet controller: Intel Corporation I350 Gigabit Fiber Network
Connection (rev 01)

We are running on a Dell R540.  Also note that when we run Centos 7.5 with
DPDK on this hardware without VMWare, everything works fine.  Also if we
run in VMWare, but "passthrough" the i350 interfaces to the VM (instead of
using vswitch and vmxnet) everything also works fine in that case.

I've tried updating the kernel (3.10) to the latests (4.18) but still get
the same error.

If I try to read the version register (VSSR) (the one that causes this
error) in the vmxnet3 pci bar registers (before I bind to DPDK) using
ethtool, it looks fine (non zero).  I've googled around a lot but can't
seem to find much help on this.

Any help would be greatly appreciated.  Thanks!
Paul Sprague
Centripetal Networks