RE: dpdk Tx falling short

["Lombardo, Ed" <Ed.Lombardo@netscout.com>]

Hi Stephen,
Here are comments to the list of obvious causes of cache misses you mentiond.

Obvious cache misses.
 - passing packets to worker with ring - we use lots of rings to pass mbuf pointers.  If I skip the rte_eth_tx_burst() and just free mbuf bulk, the tx ring does not fill up.
 - using spinlocks (cost 16ns)  - The driver does not use spinlocks, other than what dpdk uses.
 - fetching TSC  - We don't do this, we let Rx offload timestamp packets.
 - syscalls?  - No syscalls are done in our driver fast path.

You mention "passing packets to worker with ring", do you mean using rings to pass mbuf pointers causes cache misses and should be avoided?

Thanks,
Ed

Also, never ever use floating point.
-----Original Message-----
From: Stephen Hemminger <stephen@networkplumber.org> 
Sent: Saturday, July 5, 2025 3:09 PM
To: Lombardo, Ed <Ed.Lombardo@netscout.com>
Cc: Ivan Malov <ivan.malov@arknetworks.am>; users <users@dpdk.org>
Subject: Re: dpdk Tx falling short

External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

On Sat, 5 Jul 2025 17:36:08 +0000
"Lombardo, Ed" <Ed.Lombardo@netscout.com> wrote:

> Hi Stephen,
> I saw your response to more mempools and cache behavior.
> 
> I have a goal to support 2x100G next, and if I can't get 10G with DPDK then something is seriously wrong.
> 
> Should I build the dpdk static libraries with LTO?
> 
> Thanks,
> Ed

Are you doing anything in the fast path that is an obvious cache miss.
at 10Gbit/sec and size of 84 bytes = 67.2ns CPU's haven't got that much faster 3G cpu that is 201 cycles.

Single cache miss is 32ns, so two cache misses means per-packet budget is gone.

Obvious cache misses.
 - passing packets to worker with ring
 - using spinlocks (cost 16ns)
 - fetching TSC
 - syscalls?

Also, never ever use floating point.

Kernel related and older but worth looking at:
https://urldefense.com/v3/__https://people.netfilter.org/hawk/presentations/LCA2015/net_stack_challenges_100G_LCA2015.pdf__;!!Nzg7nt7_!GXQ2fZd0SInSDkGmvq3j3Kk78iP6qOBV37umb2lNgU1Lo7VoBJ40ZTYOK0LfS3o3Lq64NXnFrTyRMJXxISJyNSqhy_c$