[dpdk-users] Intel XXV710 SR-IOV packet loss

["Miroslav Kováč" <miroslav.kovac@pantheon.tech>]

Hello,


We are trying a setup with intel 25 GB card XXV710 and sr-iov. We need sr-iov to sort packets based on vlan in between the VFs. We are using trex on one machine to generate packets and multiple VPPs (each in docker container, using one VF) on another one. Trex machine contains the exact same hardware.


Each VF contains one vlan with spoof checking off and trust on and specific MAC address. For example ->


vf 0 MAC ba:dc:0f:fe:ed:00, vlan 1537, spoof checking off, link-state auto, trust on



We are generating packets with VF destination MACs with the corresponding VLAN. When sending packets to 3 VFs trex shows 35 million tx-packets and Dpdk stats on the trex machine show that 35 million were in fact sent out:


##### DPDK Statistics port0 #####
{
    "tx_good_bytes": 2142835740,
    "tx_good_packets": 35713929,
    "tx_size_64_packets": 35713929,
    "tx_unicast_packets": 35713929
}


rate= '96%'; pktSize=       64; frameLoss%=51.31%; bytesReceived/s=    1112966528.00; totalReceived=   17390102; totalSent=   35713929; frameLoss=   18323827; bytesReceived=    1112966528; targetDuration=1.0


However VPP shows only 33 million rx-packets:

VirtualFunctionEthernet17/a/0     2      up          9000/0/0/0
rx packets               5718196
rx bytes               343091760
rx-miss                  5572089

VirtualFunctionEthernet17/a/1     2      up          9000/0/0/0
rx packets               5831396
rx bytes               349883760
rx-miss                  5459089

VirtualFunctionEthernet17/a/2     2      up          9000/0/0/0
rx packets               5840512
rx bytes               350430720
rx-miss                  5449466

Sum of rx packets and rx-miss is 33,870,748. About 2 million is missing.



Even when I check VFs stats I see only 33 million to come (out of which 9.9 million are rx-missed):


root@protonet:/home/protonet# for f in $(ls /sys/class/net/enp23s0f1/device/sriov/*/stats/rx_packets); do echo "$f: $(cat $f)"; done | grep -v ' 0$'

/sys/class/net/enp23s0f1/device/sriov/0/stats/rx_packets: 11290290
/sys/class/net/enp23s0f1/device/sriov/1/stats/rx_packets: 11290485
/sys/class/net/enp23s0f1/device/sriov/2/stats/rx_packets: 11289978



When increasing the number of VFs the number of rx-packets on VPP is actually decreasing. Up to 6 or 7 VFs I still receive somewhere around 28-33 million packets, but when I use 8 VFs all the sudden it drops to 16 million packets (no rx-miss any more). The same goes with trunk mode:


VirtualFunctionEthernet17/a/0     2      up          9000/0/0/0
rx packets               1959110
rx bytes               117546600


VirtualFunctionEthernet17/a/1     2      up          9000/0/0/0
rx packets               1959181
rx bytes               117550860

VirtualFunctionEthernet17/a/2     2      up          9000/0/0/0
rx packets               1956242
rx bytes               117374520
.
.
.
Approximately the same amount of packets for each VPP instance which is 2 million packets * 8 = 16 million packets out of 35 million sent. Almost 20 million are gone



We are using vfio-pci driver.


The strange thing is that when I use only PF, no sr-iov VFs are on and I try the same vpp setup I can see all 35 million packets to come across.


This leads us to believe that there could be something wrong the sr-iov on XXV710 but we don't know how to debug this any further - The packets seem to be lost somewhere in the NIC when using sr-iov and we don't know of any dpdk or linux tool that could help us with locating the lost packets.


Regards,

Miroslav Kovac