Hi all,

       I was measuring the latency with the new Intel E810. I first used testpmd application with a single core and a single pair of queues and measured the latency on the generator side. The problem is that a latency bump occurs when the background traffic is above a certain threshold. I noticed that the threshold would move (at different rate of background traffic) depending on the speed of the recv and xmit function (i.e. bulk, SSE or AVX2)

       To identify where the bump occurs, I added hardware timestamp support to the application. I enabled rx hardware timestamp offload capability of E810, used rte_eth_timesync_read_time after rte_eth_rx_burst returns and rte_eth_timesync_read_tx_timestamp after rte_eth_tx_burst returns. I found the latency bump occurs between the packet arrives at the PHY Core and rte_eth_tx_burst returns. I also measures the CPU cycles before rte_eth_rx_burst is called and rte_eth_tx_burst returns in the user space. The gap in CPU cycles is stable regardless of the background traffic. This means the bump resides between the packet arrives the NIC and the packet is extracted from the main memory via rte_eth_rx_burst.

       Meanwhile I failed to find any DPDK latency report from Intel nor  mails from those who might experience the same problem. Does anyone meet the same problem and probably know what happens between the packet is in the PHY Core and the packet is in the memory? Maybe Intel Validation Team?

       I guess it may relate to packet discarding logic in the firmware or the DMA process. I saw this issue on different servers and different versions of firmware or DDP as well.

Configuration of the server:

CPU: Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz

RAM: DDR4 11 x 32 GB, 2933 MHz, 6 Channels

OS: Ubuntu 20.04.2 LTS

Kernel: 5.4.0-89-generic

Ice kernel driver version: 1.6.7

OS default DDP version: 1.3.26

Firmware version: 3.0

Traffic generator: MoonGen with two Mellanox ConnectX-5 EN 100G NICs

       Best Wishes

       Pan