From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1704CA034E for ; Wed, 22 Dec 2021 11:01:43 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D359A41140; Wed, 22 Dec 2021 11:01:39 +0100 (CET) Received: from smtp0.epfl.ch (smtp0.epfl.ch [128.178.224.218]) by mails.dpdk.org (Postfix) with ESMTP id 9B49240141 for ; Tue, 21 Dec 2021 13:47:50 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=epfl.ch; s=epfl; t=1640090870; h=Message-Id:MIME-Version:To:From:Subject:Date:Content-Type; bh=5OHYZNw7RwUuQHTwkDMOPTeQcUHx3WHlOh0NPoLo3v4=; b=AW8yE4TNNG8rJsLyfcuXo84aH+KtDyGTn2WE+RP4I3mCJA5WlEGwFrHXqRVrUIT3N kbQfjHAqAU4laA2kY5MISA9vAYB0QnfAaiw0k5mTQ+yIai3zszcq3P0DDNkVbSdmZ DIJNlkx2ECA8ru+57+/M3nFUFdgr/JujtPnMYsVIo= Received: (qmail 16395 invoked by uid 107); 21 Dec 2021 12:47:49 -0000 Received: from vpn-254-097.epfl.ch (HELO [IPv6:::ffff:10.252.0.48]) (128.179.254.97) (TLS, ECDHE-RSA-AES256-GCM-SHA384 (X25519 curve) cipher) (authenticated) by mail.epfl.ch (AngelmatoPhylax SMTP proxy) with ESMTPSA; Tue, 21 Dec 2021 13:47:49 +0100 X-EPFL-Auth: 5PeW6AcBFfSy9qwW8saEA215it1nKSPQ17Jh+r0DyKX7aTBbAXk= Message-Id: MIME-Version: 1.0 To: "users@dpdk.org" From: Yueyang Pan Subject: [ICE DPDK 21.11] Latency bump at certain rate Date: Tue, 21 Dec 2021 20:47:49 +0800 Importance: normal X-Priority: 3 Content-Type: multipart/alternative; boundary="_BD8D74E0-17CD-4131-8AD9-0F126232B666_" X-Mailman-Approved-At: Wed, 22 Dec 2021 11:01:37 +0100 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --_BD8D74E0-17CD-4131-8AD9-0F126232B666_ Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Hi all, I was measuring the latency with the new Intel E810. I first used testpmd = application with a single core and a single pair of queues and measured the= latency on the generator side. The problem is that a latency bump occurs w= hen the background traffic is above a certain threshold. I noticed that the= threshold would move (at different rate of background traffic) depending o= n the speed of the recv and xmit function (i.e. bulk, SSE or AVX2) To identify where the bump occurs, I added hardware timestamp support to t= he application. I enabled rx hardware timestamp offload capability of E810,= used rte_eth_timesync_read_time after rte_eth_rx_burst returns and rte_eth= _timesync_read_tx_timestamp after rte_eth_tx_burst returns. I found the lat= ency bump occurs between the packet arrives at the PHY Core and rte_eth_tx_= burst returns. I also measures the CPU cycles before rte_eth_rx_burst is ca= lled and rte_eth_tx_burst returns in the user space. The gap in CPU cycles = is stable regardless of the background traffic. This means the bump resides= between the packet arrives the NIC and the packet is extracted from the ma= in memory via rte_eth_rx_burst. Meanwhile I failed to find any DPDK latency report from Intel nor mails f= rom those who might experience the same problem. Does anyone meet the same = problem and probably know what happens between the packet is in the PHY Cor= e and the packet is in the memory? Maybe Intel Validation Team? I guess it may relate to packet discarding logic in the firmware or the DM= A process. I saw this issue on different servers and different versions of = firmware or DDP as well. Configuration of the server: CPU: Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz RAM: DDR4 11 x 32 GB, 2933 MHz, 6 Channels OS: Ubuntu 20.04.2 LTS Kernel: 5.4.0-89-generic Ice kernel driver version: 1.6.7 OS default DDP version: 1.3.26 Firmware version: 3.0 Traffic generator: MoonGen with two Mellanox ConnectX-5 EN 100G NICs Best Wishes Pan --_BD8D74E0-17CD-4131-8AD9-0F126232B666_ Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8"

Hi all,

=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 I was measuring the latency with the new Intel E810. I firs= t used testpmd application with a single core and a single pair of queues a= nd measured the latency on the generator side. The problem is that a latenc= y bump occurs when the background traffic is above a certain threshold. I n= oticed that the threshold would move (at different rate of background traff= ic) depending on the speed of the recv and xmit function (i.e. bulk, SSE or= AVX2)

=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 To identify where the bump occurs, I added h= ardware timestamp support to the application. I enabled rx hardware timesta= mp offload capability of E810, used rte_eth_timesync_read_time after= rte_eth_rx_burst returns and rte_eth_timesync_read_tx_timestamp = after rte_eth_tx_burst returns. I found the latency bump occurs = between the packet arrives at the PHY Core and rte_eth_tx_burst retu= rns. I also measures the CPU cycles before rte_eth_rx_burst is calle= d and rte_eth_tx_burst returns in the user space. The gap in CPU cyc= les is stable regardless of the background traffic. This means the bump res= ides between the packet arrives the NIC and the packet is extracted from th= e main memory via rte_eth_rx_burst.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Meanwhile= I failed to find any DPDK latency report from Intel nor =C2=A0mails from t= hose who might experience the same problem. Does anyone meet the same probl= em and probably know what happens between the packet is in the PHY Core and= the packet is in the memory? Maybe Intel Validation Team?

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 I guess it may relate to packet discarding logic in the firmware or = the DMA process. I saw this issue on different servers and different versio= ns of firmware or DDP as well.

 

Configuration of the server:

CPU: Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz

RAM: DDR4 11 x 32= GB, 2933 MHz, 6 Channels

OS: Ubuntu 20.04.2 LTS

Kernel: 5.4.0-89-generic

Ice kernel driver version: 1.6.7=

OS default DDP version: = 1.3.26

Firmwar= e version: 3.0

Traffic generator: MoonGen with two Mellanox ConnectX-5 EN 100G NICs<= /o:p>

 

=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 Best Wishes

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Pan

 

 

 

= --_BD8D74E0-17CD-4131-8AD9-0F126232B666_--