ConnectX-7 performance anomaly

ConnectX-7 performance anomaly

[Kamil Vojanec]

[-- Attachment #1: Type: text/plain, Size: 1393 bytes --]

Hello,

while benchmarking the Nvidia ConnectX-7 400Gbe NIC, I encountered some 
interesting behavior: when processing short packets (72 B), the maximum 
packet rate I can achieve is 230 Mpps. However, when I increase the 
packet size to 512 B briefly and then bring the packet size down to 72 B 
again, the maximum throughput is now 260 Mpps and seems to stabilize at 
that value. According to the Nvidia performance report 
(https://fast.dpdk.org/doc/perf/DPDK_24_07_NVIDIA_NIC_performance_report.pdf), 
the maximum achievable packet rate of ConnectX-7 NICs is 266 Mpps and 
that is what I am trying to reproduce. I have observed this behavior 
with dpdk-l3fwd as well as with my own DPDK application.

My setup is as follows:

CPU: AMD EPYC 9554P 64-Core Processor

Application commandline:

sudo dpdk_example_l3fwd -a0000:05:00.0 -l0-32 --main-lcore=32  -- -p0x1 
--config 
"(0,0,0),(0,1,1),(0,2,2),(0,3,3),(0,4,4),(0,5,5),(0,6,6),(0,7,7),(0,8,8),(0,9,9),(0,10,10),(0,11,11),(0,12,12),(0,13,13),(0,14,14),(0,15,15),(0,16,16),(0,17,17),(0,18,18),(0,19,19),(0,20,20),(0,21,21),(0,22,22),(0,23,23),(0,24,24),(0,25,25),(0,26,26),(0,27,27),(0,28,28),(0,29,29),(0,30,30),(0,31,31)" 
--eth-dest=0,00:10:94:00:00:02

Generator: Spirent 400G generator

Would anyone be able to share some insight as to what might be happening 
here?

Kamil Vojanec
CESNET z.s.p.o.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4495 bytes --]

RE: ConnectX-7 performance anomaly

[Maayan Kashani]

Hi, Kamil,
Thanks for reaching out.
After short sync with our performance team,
I can tell we did not observe such anomality and it 
seems to be some local server issue/configurations.

Regards,
Maayan Kashani

> -----Original Message-----
> From: Kamil Vojanec <vojanec@cesnet.cz>
> Sent: Thursday, 3 April 2025 14:28
> To: dev@dpdk.org
> Cc: Jan Viktorin <viktorin@cesnet.cz>
> Subject: ConnectX-7 performance anomaly
> 
> Hello,
> 
> while benchmarking the Nvidia ConnectX-7 400Gbe NIC, I encountered some
> interesting behavior: when processing short packets (72 B), the maximum
> packet rate I can achieve is 230 Mpps. However, when I increase the
> packet size to 512 B briefly and then bring the packet size down to 72 B
> again, the maximum throughput is now 260 Mpps and seems to stabilize at
> that value. According to the Nvidia performance report
> (https://fast.dpdk.org/doc/perf/DPDK_24_07_NVIDIA_NIC_performance_re
> port.pdf),
> the maximum achievable packet rate of ConnectX-7 NICs is 266 Mpps and
> that is what I am trying to reproduce. I have observed this behavior
> with dpdk-l3fwd as well as with my own DPDK application.
> 
> My setup is as follows:
> 
> CPU: AMD EPYC 9554P 64-Core Processor
> 
> Application commandline:
> 
> sudo dpdk_example_l3fwd -a0000:05:00.0 -l0-32 --main-lcore=32  -- -p0x1
> --config
> "(0,0,0),(0,1,1),(0,2,2),(0,3,3),(0,4,4),(0,5,5),(0,6,6),(0,7,7),(0,8,8),(0,9,9),(0,1
> 0,10),(0,11,11),(0,12,12),(0,13,13),(0,14,14),(0,15,15),(0,16,16),(0,17,17),(0,1
> 8,18),(0,19,19),(0,20,20),(0,21,21),(0,22,22),(0,23,23),(0,24,24),(0,25,25),(0,2
> 6,26),(0,27,27),(0,28,28),(0,29,29),(0,30,30),(0,31,31)"
> --eth-dest=0,00:10:94:00:00:02
> 
> Generator: Spirent 400G generator
> 
> Would anyone be able to share some insight as to what might be happening
> here?
> 
> Kamil Vojanec
> CESNET z.s.p.o.