Re: [dpdk-dev] DPDK vDPA application on Mellanox ConnectX 6Dx

[William Tu <u9012063@gmail.com>]

Hi,

We finally get it worked out! It was due to our misconfiguration in the
dpdk vdpa sample application. We continue measuring the performance
using a dpdk traffic gen (64B UDP) and measure at VM's rx queue.
We wonder if the number below (around 1.8Mpps per core) is expected?

At hypervisor:
qemu and dpdk vdpa app is using 100%
16391 root      20   0 9930.6m  33380  14032 S 100.3  0.1   3:13.86
qemu-system-x86
16046 root      20   0  0.136t  13228   7448 S  95.7  0.0   2:47.43 vdpa

At VM (ens6 is the vdpa virtio dev)
root@vdpa-vm:/dpdk# mlnx_perf -i ens6
Initializing mlnx_perf...
Sampling started.
            rx_queue_0_packets: 1,824,640
              rx_queue_0_bytes: 109,478,400 Bps      = 875.82 Mbps
              rx_queue_0_kicks: 28,367

Thanks
William

On Mon, Jul 13, 2020 at 7:29 AM William Tu <u9012063@gmail.com> wrote:
>
> Hi,
>
> We are setting up a testbed using mellanox connectX 6Dx
> MCX621102AN-ADAT (2x25G) and
> are playing with vDPA features.  We setup two machines (traffic gen and KVM)
> and connect them back-to-back with two connectX 6Dx.
>
> At this moment we don't know which component might be misconfigured
> (qemu, vdpa app, or HW nic). Any feedback for debugging is appreciated!
>
> At KVM machine we follow [1] to setup vdpa on NIC, basically
> 1) Enable switchdev
>   echo 0000:02:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind
>   echo switchdev > /sys/class/net/enp2s0f0/compat/devlink/mode
>   echo 0000:02:00.2 > /sys/bus/pci/drivers/mlx5_core/bind
>
> $ lspci
>   02:00.0 Ethernet controller: Mellanox Technologies MT28841
>   02:00.1 Ethernet controller: Mellanox Technologies MT28841
>   02:00.2 Ethernet controller: Mellanox Technologies MT28850
>   02:00.3 Ethernet controller: Mellanox Technologies MT28850
>
> [11350.951711] mlx5_core 0000:02:00.0: E-Switch: Enable:
> mode(OFFLOADS), nvfs(2), active vports(3)
> [11351.032413] mlx5_core 0000:02:00.0 enp2s0f0_p0: Link up
> [11351.226525] enp2s0f0_pf0vf1: renamed from eth0
> [11351.403649] IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0f0_p0: link becomes ready
> [11351.403951] IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0f0_pf0vf0: link
> becomes ready
> [11351.404162] IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0f0_pf0vf1: link
> becomes ready
>
> [1]https://docs.mellanox.com/pages/viewpage.action?pageId=25146841#OVSOffloadUsingASAP%C2%B2Direct-ovs-dpdkhwoffloadsOVS-DPDKHardwareOffloads
>
> 2) run the DPDK's vdpa appliaction, create /tmp/sock-virtio0
> $ ./vdpa -w 0000:02:00.2,class=vdpa --log-level=pmd,info -- -i
> EAL: Detected 12 lcore(s)
> EAL: Detected 1 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: Probe PCI driver: net_mlx5 (15b3:101e) device: 0000:02:00.2 (socket 0)
> EAL: Probe PCI driver: mlx5_vdpa (15b3:101e) device: 0000:02:00.2 (socket 0)
> vdpa> create /tmp/sock-virtio0 0000:02:00.2
> VHOST_CONFIG: vhost-user server: socket created, fd: 65
> VHOST_CONFIG: bind to /tmp/sock-virtio0
> vdpa> VHOST_CONFIG: new vhost user connection is 68
> VHOST_CONFIG: new device, handle is 0
> ...
> VHOST_CONFIG: virtio is now ready for processing.
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
> VHOST_CONFIG: set queue enable: 1 to qp idx: 3
> mlx5_vdpa: mlx5_vdpa_virtq.c:424: mlx5_vdpa_virtq_enable(): Update
> virtq 3 status disable -> enable.
> mlx5_vdpa: mlx5_vdpa_virtq.c:133: mlx5_vdpa_virtq_stop(): Query vid 0
> vring 3: hw_available_idx=0, hw_used_index=0
> mlx5_vdpa: mlx5_vdpa_virtq.c:264: mlx5_vdpa_virtq_setup(): vid 0: Init
> last_avail_idx=0, last_used_idx=0 for virtq 3.
> VHOST_CONFIG: virtio is now ready for processing.
>
> 3) start the VM
> $ qemu-system-x86_64 --version
> QEMU emulator version 4.2.1
> $ qemu-system-x86_64 -enable-kvm -smp 5 -cpu host -m 4G -drive \
>     file=/var/lib/libvirt/images/vdpa-vm.qcow2 \
>     -serial mon:stdio \
>     -chardev socket,id=charnet1,path=/tmp/sock-virtio0 \
>     -netdev vhost-user,chardev=charnet1,queues=2,id=hostnet1 \
>     -device virtio-net-pci,mq=on,vectors=6,netdev=hostnet1,id=net1,mac=e4:11:c6:d3:45:f2,bus=pci.0,addr=0x6,page-per-vq=on,rx_queue_size=1024,tx_queue_size=1024
> \
>     -object memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on
> \
>     -numa node,memdev=mem -mem-prealloc -name "vdpa-vm"
>
> Once VM boots, inside vm, I can see virtio device, and I bring it up, turns
> on promisc mode.
> vdpa@vdpa-vm:~$ ethtool -i ens6
> driver: virtio_net
> version: 1.0.0
>
> 4) At traffic gen, start to send packet with dst mac e4:11:c6:d3:45:f2
> however, on the VM side, the packet stats is always 0.
>
> On the KVM host side, I saw packets arrived at PF, but not sending to
> vf0 (0000:02:00.2)
> Average:        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s
> rxcmp/s   txcmp/s  rxmcst/s   %ifutil
> Average:    enp2s0f0_p0 31544608.00      0.00 1971539.12      0.00
>  0.00      0.00      0.00     64.60
> Average:    enp2s0f0_pf0vf1      0.00      0.00      0.00      0.00
>   0.00      0.00      0.00      0.00
>
> $ mlnx_perf -i enp2s0f0_p0
>                     rx_packets: 15,235,681
>                       rx_bytes: 914,140,860 Bps      = 7,313.12 Mbps
>                   rx_csum_none: 15,235,680
>                 rx_cache_reuse: 7,617,792
>                        ch_poll: 238,057
>               rx_out_of_buffer: 19,963,082
>       rx_vport_unicast_packets: 35,198,770
>         rx_vport_unicast_bytes: 2,252,721,216 Bps    = 18,021.76 Mbps
>
> So I install tc rule
> $ tc filter add dev enp2s0f0_p0 protocol ip parent ffff: \
>     flower skip_sw action mirred egress redirect dev enp2s0f0_pf0vf0
>   filter protocol ip pref 49152 flower chain 0 handle 0x1
>   eth_type ipv4
>   skip_sw
>   in_hw
>     action order 1: mirred (Egress Redirect to device enp2s0f0_pf0vf0) stolen
>     index 1 ref 1 bind 1 installed 14 sec used 0 sec
>     Action statistics:
>     Sent 31735712192 bytes 495870503 pkt (dropped 0, overlimits 0 requeues 0)
>
> With the rule, packets show up in KVM's vf, but still no packet
> inside VM (ip -s link show) show all zero.
>
> Appreciate for any suggestion to debugging.
> Thanks in advance.
> William & Sergey