82599ES 10-Gigabit NIC cannot at 10G speed for a long time

82599ES 10-Gigabit NIC cannot at 10G speed for a long time

[Sun Yalou]

[-- Attachment #1: Type: text/plain, Size: 966 bytes --]

Hi there,I'm developing a custom virtual switch using OVS-DPDK(v19.11.10). 
In order to achieve 10Gbit/s, I modified the OVS pmd mode from 
run-to-completion to pipeline. I used 1 thread as rx thread, 4 threads as 
worker thread and 1 thread as tx thread. it's a load balance archiecture.
I found the forwarding rate can reach 10Gbit/s in first 2 minite, and then 
drop to 5Gbit/s . My traffic generator keeps send packets at 10G rate. I 
debugged and found that packet loss occurred in `rte_eth_tx_burst`, then it 
invoke `ixgbe_xmit_pkts_vec` -> `ixgbe_xmit_fixed_burst_vec` , in 
`ixgbe_xmit_fixed_burst_vec`it should be `txq->nb_tx_free` < `nb_pkts`, it 
means there is not enough TX descriptor to be allocated.
I have tried increasing socket-mem when starting program and reducing the 
size of batch sent, neither worked. I wonder if i am missing something 
important? This question has been bugging me for weeks.
Does anyone know this issue? Thanks, BR, Yalou

[-- Attachment #2: Type: text/html, Size: 2006 bytes --]

Re: 82599ES 10-Gigabit NIC cannot at 10G speed for a long time

[Stephen Hemminger]

On Thu, 24 Nov 2022 04:42:06 +0000
Sun Yalou <sunyalou@bupt.edu.cn> wrote:

> Hi there,I'm developing a custom virtual switch using OVS-DPDK(v19.11.10). 
> In order to achieve 10Gbit/s, I modified the OVS pmd mode from 
> run-to-completion to pipeline. I used 1 thread as rx thread, 4 threads as 
> worker thread and 1 thread as tx thread. it's a load balance archiecture.
> I found the forwarding rate can reach 10Gbit/s in first 2 minite, and then 
> drop to 5Gbit/s . My traffic generator keeps send packets at 10G rate. I 
> debugged and found that packet loss occurred in `rte_eth_tx_burst`, then it 
> invoke `ixgbe_xmit_pkts_vec` -> `ixgbe_xmit_fixed_burst_vec` , in 
> `ixgbe_xmit_fixed_burst_vec`it should be `txq->nb_tx_free` < `nb_pkts`, it 
> means there is not enough TX descriptor to be allocated.
> I have tried increasing socket-mem when starting program and reducing the 
> size of batch sent, neither worked. I wonder if i am missing something 
> important? This question has been bugging me for weeks.
> Does anyone know this issue? Thanks, BR, Yalou


Curious why you think pipeline is faster, since processing same
packet repeatedly by multiple cores causes cache misses.