[dpdk-dev] DPDK vDPA application on Mellanox ConnectX 6Dx

[William Tu <u9012063@gmail.com>]

Hi,

We are setting up a testbed using mellanox connectX 6Dx
MCX621102AN-ADAT (2x25G) and
are playing with vDPA features.  We setup two machines (traffic gen and KVM)
and connect them back-to-back with two connectX 6Dx.

At this moment we don't know which component might be misconfigured
(qemu, vdpa app, or HW nic). Any feedback for debugging is appreciated!

At KVM machine we follow [1] to setup vdpa on NIC, basically
1) Enable switchdev
  echo 0000:02:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind
  echo switchdev > /sys/class/net/enp2s0f0/compat/devlink/mode
  echo 0000:02:00.2 > /sys/bus/pci/drivers/mlx5_core/bind

$ lspci
  02:00.0 Ethernet controller: Mellanox Technologies MT28841
  02:00.1 Ethernet controller: Mellanox Technologies MT28841
  02:00.2 Ethernet controller: Mellanox Technologies MT28850
  02:00.3 Ethernet controller: Mellanox Technologies MT28850

[11350.951711] mlx5_core 0000:02:00.0: E-Switch: Enable:
mode(OFFLOADS), nvfs(2), active vports(3)
[11351.032413] mlx5_core 0000:02:00.0 enp2s0f0_p0: Link up
[11351.226525] enp2s0f0_pf0vf1: renamed from eth0
[11351.403649] IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0f0_p0: link becomes ready
[11351.403951] IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0f0_pf0vf0: link
becomes ready
[11351.404162] IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0f0_pf0vf1: link
becomes ready

[1]https://docs.mellanox.com/pages/viewpage.action?pageId=25146841#OVSOffloadUsingASAP%C2%B2Direct-ovs-dpdkhwoffloadsOVS-DPDKHardwareOffloads

2) run the DPDK's vdpa appliaction, create /tmp/sock-virtio0
$ ./vdpa -w 0000:02:00.2,class=vdpa --log-level=pmd,info -- -i
EAL: Detected 12 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Probe PCI driver: net_mlx5 (15b3:101e) device: 0000:02:00.2 (socket 0)
EAL: Probe PCI driver: mlx5_vdpa (15b3:101e) device: 0000:02:00.2 (socket 0)
vdpa> create /tmp/sock-virtio0 0000:02:00.2
VHOST_CONFIG: vhost-user server: socket created, fd: 65
VHOST_CONFIG: bind to /tmp/sock-virtio0
vdpa> VHOST_CONFIG: new vhost user connection is 68
VHOST_CONFIG: new device, handle is 0
...
VHOST_CONFIG: virtio is now ready for processing.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 3
mlx5_vdpa: mlx5_vdpa_virtq.c:424: mlx5_vdpa_virtq_enable(): Update
virtq 3 status disable -> enable.
mlx5_vdpa: mlx5_vdpa_virtq.c:133: mlx5_vdpa_virtq_stop(): Query vid 0
vring 3: hw_available_idx=0, hw_used_index=0
mlx5_vdpa: mlx5_vdpa_virtq.c:264: mlx5_vdpa_virtq_setup(): vid 0: Init
last_avail_idx=0, last_used_idx=0 for virtq 3.
VHOST_CONFIG: virtio is now ready for processing.

3) start the VM
$ qemu-system-x86_64 --version
QEMU emulator version 4.2.1
$ qemu-system-x86_64 -enable-kvm -smp 5 -cpu host -m 4G -drive \
    file=/var/lib/libvirt/images/vdpa-vm.qcow2 \
    -serial mon:stdio \
    -chardev socket,id=charnet1,path=/tmp/sock-virtio0 \
    -netdev vhost-user,chardev=charnet1,queues=2,id=hostnet1 \
    -device virtio-net-pci,mq=on,vectors=6,netdev=hostnet1,id=net1,mac=e4:11:c6:d3:45:f2,bus=pci.0,addr=0x6,page-per-vq=on,rx_queue_size=1024,tx_queue_size=1024
\
    -object memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on
\
    -numa node,memdev=mem -mem-prealloc -name "vdpa-vm"

Once VM boots, inside vm, I can see virtio device, and I bring it up, turns
on promisc mode.
vdpa@vdpa-vm:~$ ethtool -i ens6
driver: virtio_net
version: 1.0.0

4) At traffic gen, start to send packet with dst mac e4:11:c6:d3:45:f2
however, on the VM side, the packet stats is always 0.

On the KVM host side, I saw packets arrived at PF, but not sending to
vf0 (0000:02:00.2)
Average:        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s
rxcmp/s   txcmp/s  rxmcst/s   %ifutil
Average:    enp2s0f0_p0 31544608.00      0.00 1971539.12      0.00
 0.00      0.00      0.00     64.60
Average:    enp2s0f0_pf0vf1      0.00      0.00      0.00      0.00
  0.00      0.00      0.00      0.00

$ mlnx_perf -i enp2s0f0_p0
                    rx_packets: 15,235,681
                      rx_bytes: 914,140,860 Bps      = 7,313.12 Mbps
                  rx_csum_none: 15,235,680
                rx_cache_reuse: 7,617,792
                       ch_poll: 238,057
              rx_out_of_buffer: 19,963,082
      rx_vport_unicast_packets: 35,198,770
        rx_vport_unicast_bytes: 2,252,721,216 Bps    = 18,021.76 Mbps

So I install tc rule
$ tc filter add dev enp2s0f0_p0 protocol ip parent ffff: \
    flower skip_sw action mirred egress redirect dev enp2s0f0_pf0vf0
  filter protocol ip pref 49152 flower chain 0 handle 0x1
  eth_type ipv4
  skip_sw
  in_hw
    action order 1: mirred (Egress Redirect to device enp2s0f0_pf0vf0) stolen
    index 1 ref 1 bind 1 installed 14 sec used 0 sec
    Action statistics:
    Sent 31735712192 bytes 495870503 pkt (dropped 0, overlimits 0 requeues 0)

With the rule, packets show up in KVM's vf, but still no packet
inside VM (ip -s link show) show all zero.

Appreciate for any suggestion to debugging.
Thanks in advance.
William & Sergey