[dpdk-dev] Vmxnet3-pmd not receiving packets after random time under high load

[dpdk-dev] Vmxnet3-pmd not receiving packets after random time under high load

[Daniel Kan]

I dug deeper into the problem vmxnet-pmd not capturing packets. Whenever pmd stops capturing, it does not respect the number of rxd. For example, I set rxd to 512. The pkt mbuf mempool I allocate is 4482, which is much bigger than rxd. I would expect ~512 mbufs to be removed from mempool at any moment in time. When not able to capture new packets is imminent, pmd would use much more than 512 to the point of exhausting all the mbufs in mempool. After this point, pmd stops capturing packet and does not recover; it never enters the while loop below. 

vmxnet3_rxtx.c
457	while (rcd->gen == rxq->comp_ring.gen) {

This issue occurs quite randomly, which leads me to believe there is some race condition. It occurs when packets are arriving faster than the application can process. 

I understand that vmxnet3 pmd preallocates rxq descriptors during initialization. Yet, even when it’s receiving packets correctly, it uses a little more than rxq descriptors. I find the number of mbufs in mempool very interesting. 

Before vmxnet3 init
4482

After vmxnet3 init
3970

When receiving packets properly
ranges between 3920 - 3970

Prior to stop receiving packets
ranges between 0 and 3970

After the last packet is received
4481

I appreciate any comment/help on this. Thanks. 

Dan


On Mar 11, 2014, at 12:55 PM, Daniel Kan <dan@nyansa.com> wrote:

> I’m unable to get RSS to work properly with vmxnet3-pmd. The first issue is that the number of rxqs must be power of 2. Otherwise, rte_eth_dev_start() fails due to inability to activate vmxnet3 NIC. This is not too big of a deal, but physical NICs don’t have this requirement. 
> 
> The second issue is that RSS is just not working at all for me. The rxmode is set to ETH_MQ_RX_RSS and rss_hf = ETH_RSS_IPV4_TCP | ETH_RSS_IPV4_UDP | ETH_RSS_IPV4 | ETH_RSS_IPV6. The same configuration works for a real NIC. When I checked mb->pkt.hash, the value is all zeroed out. 
> 
> Even if I disabled RSS, I found the performance of vmxnet3-pmd to be quite poor, peaking out at 600k pps with 64 byte packet, while libpcap can do 650k pps. 
> 
> Lastly, there is a stability issue. On a number of occasions, vmxnet3-pmd stops receiving packets after some random time and several million packets. 
> 
> I’m not sure if anyone else is having as much issue as I’m, I will give vmxnet3-usermap a try. 
> 
> Finally, does either vmxnet3-usermap or vmxnet3-pmd work well for non-Intel-based underlying physical NIC? 
> 
> Thanks. 
> 
> Dan
> 
>