From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 629B246BF1 for ; Wed, 23 Jul 2025 15:51:15 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5981D406A2; Wed, 23 Jul 2025 15:51:15 +0200 (CEST) Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by mails.dpdk.org (Postfix) with ESMTP id 8EA2E4067C for ; Wed, 23 Jul 2025 15:51:14 +0200 (CEST) Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-31332cff2d5so5769211a91.1 for ; Wed, 23 Jul 2025 06:51:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1753278674; x=1753883474; darn=dpdk.org; h=mime-version:subject:references:in-reply-to:message-id:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=ja1KP8PtBu9L9oNVaoHCcPTDliOmmL1AVI3tJtc3Zdk=; b=hbIVEkvdY2Lq4QT1t8N6hOJVYc9gevb5kT08xtWhc0oMbZXPMZmTrUMty4b0LvsqjK t/ZDRBqtknClWyPYyNASXtU5GlIcxHy0/vn/fzyFgp1A+GsqfcFEzjwKm8CWWhauED8I 4FAONHpoobwWzu31vfIpX35GjGQTkL04G84TSbRwactBge1oXObivg1I+NGro00jGtBE q85zz5zc+OU46hXa3VtjlJ6Qp/9CattkRRF5noaZ/jIc6KF30qS+sj8Ruaodqq1YBnJ1 ccnLbjJV5NJkPSW1I67wb4VMMb/c/aSX7k5/Cnv7ylVZRC7xF1GA8ZSTmhOMoWBnuwRw KIJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1753278674; x=1753883474; h=mime-version:subject:references:in-reply-to:message-id:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ja1KP8PtBu9L9oNVaoHCcPTDliOmmL1AVI3tJtc3Zdk=; b=tnwAsf2XUQKsOOCiByNEyYD/rd0CcShoSUWUX5CGey8SCgtQmcmkbjeFKzGFV6racw /o6VQ5UZc2d7EDSpvh3zdgeSLiGzX9AIXCDINHooDnxWzvBwRT8xDeLgPTEyF+nbitx4 HBPDpdK33HQ372J/r/oC6OcRqiTT22EQW+gTCBjzLdEEDL1cKaPgtDN6SfEgEckXkGCA OF0CwpL0DhL4WaL2pmN7NpvtiBJ+C85owZD/t3MM1rFC3VOsdBmElheMIL/7qIKB2tTt 3DuZtWcYdsLt1qYFQbYQbfy/NxkCbGvcMHQJY2u+hv+b1nDuw/s8+y/sPMe7IjG77PKp q7fA== X-Gm-Message-State: AOJu0Yy9GfttxzkEE7y/xb37aWwgcfLXyTeBbG+pnFrSrSQwj9YVD68e Nx43CsG2TEdel0mbjCaY12rCSndhKEJ0AkwMuw1/JWsFOlvVzJFRxj/0 X-Gm-Gg: ASbGnct6SEVGmm/fYzpOmp3lb6Kw83A3MFXeaYj911NlSPx/41C+G4SJ3Uql5/UzU0Q 2K08eA4rO1GKIzaSpfnr/JHCRwQFz5ojqmr4d6Y0cVH5qEvsgFbIrSYYY8bDgWllldW+Nb2vsqp fpNRID6zk77vZA5YezBoRx0AG0ChfSJu+dEwNCOSp9/6pLOVO9di3/C047j+oOSKCau6yGzQ6aF hi6IcCFe35kbRqw+k6s2noNNpgPCsmYOr5cpUCDilO738bKVyPT5QG3yyDgZVg9sAiBPkHdd/jZ +QaXMQRMSOnVAJGFbdDm6BLWZF8UrMsg7oUy9zWMdBOGe2vkUyBRDGkyT0TYHmywNNXn8eOtPCT ds68sm1oV0i9tr8iEjQ== X-Google-Smtp-Source: AGHT+IGLTZ4Bbd45QAx0HVQQyeEoDhtRllc53d9I4wBLrXLPU9IuJ0l2ekwfPFT5XeT4GLF4I+cgNQ== X-Received: by 2002:a17:90b:3c90:b0:311:d3a5:572a with SMTP id 98e67ed59e1d1-31e5071cc94mr4958444a91.8.1753278673341; Wed, 23 Jul 2025 06:51:13 -0700 (PDT) Received: from localhost ([147.47.193.124]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-b3f2fe8b88bsm8765496a12.16.2025.07.23.06.51.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Jul 2025 06:51:12 -0700 (PDT) Date: Wed, 23 Jul 2025 22:50:53 +0900 From: Yoon Junghan To: Ivan Malov Cc: "=?utf-8?Q?users=40dpdk.org?=" Message-ID: <2c092b37-e8cd-4161-bebf-e5031b7c2990@Spark> In-Reply-To: <66490314-cb05-0018-4457-9bfab077048a@arknetworks.am> References: <3f7df93b-ee00-bcdb-337c-8faa7bd73f6c@arknetworks.am> <391481b9-1b18-5567-a8a0-563c5c5eb37c@arknetworks.am> <8f3a156a-2f2a-4418-a5e0-47a4450cd10a@Spark> <66490314-cb05-0018-4457-9bfab077048a@arknetworks.am> Subject: Re: HW RX timestamp with LRO enabled on ConnectX-7 (DPDK 20.11) X-Readdle-Message-ID: 2c092b37-e8cd-4161-bebf-e5031b7c2990@Spark MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="6880e8cc_749abb43_5997" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --6880e8cc_749abb43_5997 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline I isolated port 1 using -a option for EAL parameter and got the similar r= esult. Note that port 1 becomes port 0 in this time. =5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed) =5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed) =5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed) =5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed) =5Bport 0=5D RX HW timestamp: 0x00042819272fad (not LROed) =5Bport 0=5D RX HW timestamp: 0x000428192e6e77 (not LROed) =5Bport 0=5D RX HW timestamp: 0x000428192e7f01 (not LROed) =5Bport 0=5D RX HW timestamp: 0x000428192e833d (not LROed) =46YI, I have 4 CX-7 on the same machine. (eth0 =3D port 0, ... eth3 =3D = port 3 in DPDK) pci=400000:16:00.0 =C2=A0eth0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 n= etwork =C2=A0 =C2=A0 =C2=A0 =C2=A0MT2910 =46amily =5BConnectX-7=5D pci=400000:40:00.0 =C2=A0eth1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 n= etwork =C2=A0 =C2=A0 =C2=A0 =C2=A0MT2910 =46amily =5BConnectX-7=5D pci=400000:6a:00.0 =C2=A0eth2 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 n= etwork =C2=A0 =C2=A0 =C2=A0 =C2=A0MT2910 =46amily =5BConnectX-7=5D pci=400000:94:00.0 =C2=A0eth3 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 n= etwork =C2=A0 =C2=A0 =C2=A0 =C2=A0MT2910 =46amily =5BConnectX-7=5D Among them, only the first CX-7 shows consistent timestamp regardless of = LRO. Sincerely, Junghan Yoon On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 10:28 +0900, Ivan Malov , wrote: > On Wed, 23 Jul 2025, Yoon Junghan wrote: > > > Thank you for quick response. > > > > 1) They are different NICs. Not in the same board. Separate adapters = in different PCIe slots. > > 2) My DPDK app uses 4 separate ports; port 0, port 1, port 2, and por= t 3. They are all on different boards. Thus, they are running at the same= time. > > Excellent. I apologise for one more dumb question, but does isolating t= he very > specific NIC (so that DPDK does not grab the other ones) that is known = to give > strange timestamps, result in the same/unexpected behaviour=3F Just to = make sure. > > Thank you. > > > > > Sincerely, > > Junghan Yoon > > On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 10:09 +0900, Ivan Malov , wrote: > > > > Hello, > > > > On Wed, 23 Jul 2025, Yoon Junghan wrote: > > > > > > Hello, > > As advised, I tested hardware timestamps with LRO enabled on our Conn= ectX-7 NICs. However, the timestamps of LROed packets still show inconsis= tent and abnormally large > > gaps from normal > > packets. > > > > Interestingly, I found this issue does not appear on all CX-7 NICs. E= ven with identical DPDK code, firmware version (28.43.2566), and hardware= models from the same > > manufacturer, only > > specific NICs exhibit this inconsistency. > > I have confirmed that: > > * All NICs use the same driver and firmware version. > > * All NICs are of the same model (MCX75310AAS-NEA=5FAx). > > > > > > > > 1) Do the two =22NICs=22 ('port 0' and 'port 1' from below printout) = represent two > > different ports/P=46s of the same physical 'board'/'adapter card' in = fact=3F > > > > 2) If (1) is true, were the results obtained by running the applicati= on on both > > ports simultaneously (both managed by the DPDK at the same time)=3F > > > > (just to clarify, -- I'm confused by the fact that the NIC driver its= elf seems > > to invoke 'rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp=5Fregister' for each new= RxQ rather than call > > it once and then look-up and reuse the existing offsets for more port= s/queue ). > > > > Thank you. > > > > > > * The issue occurs only when LRO is enabled together with RX hardware= timestamping. > > * Disabling LRO eliminates the issue. > > I would appreciate any insight into how this behavior can occur on on= ly some ports despite same software and hardware setup. > > > > Below is my code snippet. > > > > =60=60=60c > > /*-------------------------------------------------------------------= ---------*/ > > static inline int > > is=5Ftimestamp=5Fenabled(struct rte=5Fmbuf *mbuf) > > =7B > > =C2=A0 =C2=A0static uint64=5Ft timestamp=5Frx=5Fdynflag =3D 0; > > =C2=A0 =C2=A0int timestamp=5Frx=5Fdynflag=5Foffset; > > > > =C2=A0 =C2=A0if (=21timestamp=5Frx=5Fdynflag) > > =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0timestamp=5Frx=5Fdynflag=5Foffset =3D > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rte=5Fmbuf=5Fdynflag=5Flooku= p(RTE=5FMBU=46=5FDYN=46LAG=5FRX=5FTIMESTAMP=5FNAME, NULL); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0if (timestamp=5Frx=5Fdynflag=5Foffset < 0)= > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return 0; > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=7D > > =C2=A0 =C2=A0 =C2=A0 =C2=A0timestamp=5Frx=5Fdynflag =3D RTE=5FBIT64(t= imestamp=5Frx=5Fdynflag=5Foffset); > > =C2=A0 =C2=A0=7D > > > > =C2=A0 =C2=A0return mbuf->ol=5Fflags & timestamp=5Frx=5Fdynflag; > > =7D > > /*-------------------------------------------------------------------= ---------*/ > > static inline rte=5Fmbuf=5Ftimestamp=5Ft * > > get=5Ftimestamp(struct rte=5Fmbuf *mbuf) > > =7B > > =C2=A0 =C2=A0static int timestamp=5Fdynfield=5Foffset =3D -1; > > > > =C2=A0 =C2=A0if (timestamp=5Fdynfield=5Foffset < 0) > > =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0timestamp=5Fdynfield=5Foffset =3D > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rte=5Fmbuf=5Fdynfield=5Flook= up(RTE=5FMBU=46=5FDYN=46IELD=5FTIMESTAMP=5FNAME, NULL); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0if (timestamp=5Fdynfield=5Foffset < 0) > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return 0; > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=7D > > =C2=A0 =C2=A0=7D > > > > =C2=A0 =C2=A0return RTE=5FMBU=46=5FDYN=46IELD(mbuf, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0timestamp=5Fdynfield=5Foffset, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rte=5Fmbuf=5Ftimestamp=5Ft *); > > =7D > > /*-------------------------------------------------------------------= ---------*/ > > static inline rte=5Fmbuf=5Ftimestamp=5Ft * > > get=5Frx=5Fhw=5Ftimestamp(struct rte=5Fmbuf *pkt) > > =7B > > =C2=A0 =C2=A0if (=21is=5Ftimestamp=5Fenabled(pkt)) > > =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0printf(=22rx=5Fhw=5Ftimestamp not enabled = in mbuf=21=5Cn=22); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0return NULL; > > =C2=A0 =C2=A0=7D > > > > =C2=A0 =C2=A0return get=5Ftimestamp(pkt); > > =7D > > =60=60=60 > > > > My DPDK application prints logs as below. > > > > =60=60=60c > > =C2=A0 =C2=A0/* parse HW timestamp */ > > =C2=A0 =C2=A0rte=5Fmbuf=5Ftimestamp=5Ft *rx=5Ftimestamp =3D get=5Frx=5F= hw=5Ftimestamp(pkt); > > =C2=A0 =C2=A0printf(=22=5Bport %d=5D RX HW timestamp: %=23016lx %s=5C= n=22, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pctx->port=5Fid, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 *rx=5Ftimestamp, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pkt->ol=5Fflags & PKT=5FRX=5FLRO =3F= =22(LROed)=22 : =22(not LROed)=22); > > =60=60=60 > > > > Below are observations from two CX-7 ports under identical conditions= . > > > > Normal NIC (port 0): > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d185b (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d1911 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d19c9 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d37ca (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d4cb3 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d4cb3 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd30e019 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd3280bb (not LROed) > > > > Erroneous NIC (port 1): > > Below is erroneous NIC's timestamp. > > =5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > =5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > =5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > =5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > =5Bport 1=5D RX HW timestamp: 0x000080691b7557 (not LROed) > > =5Bport 1=5D RX HW timestamp: 0x000080691e2311 (not LROed) > > =5Bport 1=5D RX HW timestamp: 0x00008069357553 (not LROed) > > =5Bport 1=5D RX HW timestamp: 0x0000806936e8c1 (not LROed) > > > > As shown above, non-LRO packets consistently have normal hardware tim= estamps on both NICs. However, on port 1, all LROed packets return a fixe= d, invalid timestamp > > (0x3e6eef91bc19f0fd), > > which is clearly inconsistent. > > I have also confirmed that other dynfields (rather than dynfield=5B1=5D= and dynfield=5B2=5D) are unused. > > > > > > Sincerely, > > Junghan Yoon > > On Jul 22, 2025, 5:31 PM +0900, Ivan Malov , wrote: > > Hello, > > > > On Tue, 22 Jul 2025, Yoon Junghan wrote: > > > > Hello, > > > > I'm currently using DPDK 20.11 with a ConnectX-7 NIC, and I'm trying = to retrieve RX hardware timestamps using =60rte=5Fmbuf=5Fdyn=5Frx=5Ftimes= tamp=5Fregister()=60. > > > > > > Does the application invoke 'rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp=5Fregi= ster' on its own=3F If > > yes, consider to replace this with invocations of APIs =5B1=5D (with = field name =5B2=5D) > > and =5B3=5D (with flag name =5B4=5D). =46or an example, please refer = to =5B5=5D and =5B6=5D. > > > > This is because, as per =5B7=5D, the driver in question might 'regist= er' the field > > and the flag on its own, in response to 'DEV=5FRX=5FO=46=46LOAD=5FTIM= ESTAMP' request, so, > > the user application should look up the field/flag, not 'register' it= afresh. > > > > If this does not help, then consider to clarify whether the timestamp= s are > > accurate (and whether the flag is seen in the mbufs) when LRO is not = enabled. > > > > =5B1=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.ht= ml=23a6adf9b352a83e7d521fd6aa04e305b1c > > =5B2=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.ht= ml=23a5159b2d34fa801d171ed0ccce451121b > > =5B3=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.ht= ml=23a89d835027034f76a27eb2afe7987ae35 > > =5B4=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.ht= ml=23a831d7066c7193788351797a65186848a > > =5B5=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b9659= 7469b7f6e6207/app/test-pmd/util.c=23L44 > > =5B6=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b9659= 7469b7f6e6207/app/test-pmd/util.c=23L60 > > =5B7=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b9659= 7469b7f6e6207/drivers/net/mlx5/mlx5=5Frxq.c=23L1743 > > > > Thank you. > > > > > > When LRO is enabled, I notice that LROed mbufs seem to share identica= l timestamp values, and the timestamps are unexpectedly large or inconsis= tent. This raises > > the question of whether > > LRO is interfering with the correctness of the RX HW timestamps. > > > > I=E2=80=99d appreciate any clarification on whether HW RX timestampin= g is reliable when LRO is enabled on this platform, or if LRO should be j= ust disabled for accurate > > per-packet timestamping. > > > > > > Sincerely, > > Junghan Yoon > > > > > > > > > > --6880e8cc_749abb43_5997 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
I isolated port 1 using -a optio= n for EAL parameter and got the similar result.
&=23160;
Note that port 1 becomes port 0 = in this time.
=5Bport 0=5D RX HW timestamp: 0x= 3eac4214bc574368 (LROed)
=5Bport 0=5D RX HW timestamp: 0x3eac4= 214bc574368 (LROed)
=5Bport 0=5D RX HW timestamp: 0x3eac4= 214bc574368 (LROed)
=5Bport 0=5D RX HW timestamp: 0x3eac4= 214bc574368 (LROed)
=5Bport 0=5D RX HW timestamp: 0x00042= 819272fad (not LROed)
=5Bport 0=5D RX HW timestamp: 0x00042= 8192e6e77 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x00042= 8192e7f01 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x00042= 8192e833d (not LROed)
&=23160;
=46YI, I have 4 CX-7 on the same= machine. (eth0 =3D port 0, ... eth3 =3D port 3 in DPDK)
pci=400000:16:00.0 &=23160;eth0 = &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; network &=23160; &=23= 160; &=23160; &=23160;MT2910 =46amily =5BConnectX-7=5D
pci=400000:40:00.0 &=23160;eth1 &=231= 60; &=23160; &=23160; &=23160; &=23160; &=23160; network &=23160; &=23160= ; &=23160; &=23160;MT2910 =46amily =5BConnectX-7=5D
pci=400000:6a:00.0 &=23160;eth2 &=231= 60; &=23160; &=23160; &=23160; &=23160; &=23160; network &=23160; &=23160= ; &=23160; &=23160;MT2910 =46amily =5BConnectX-7=5D
pci=400000:94:00.0 &=23160;eth3 &=231= 60; &=23160; &=23160; &=23160; &=23160; &=23160; network &=23160; &=23160= ; &=23160; &=23160;MT2910 =46amily =5BConnectX-7=5D
&=23160;
Among them, only the first CX-7 shows consistent timestamp regardles= s of LRO.

Sincerely,
Junghan Yoon
On 2025=EB=85=84 7=EC=9B=94 23=EC=9D= =BC PM 10:28 +0900, Ivan Malov <ivan.malov=40arknetworks.am>, wrote= :
On Wed, 23 Jul 2025, Yoon Junghan wrote:
Thank you for quick response.
&=23160;
1) They are different NICs. Not in the same board. Separate adapters in d= ifferent PCIe slots.
2) My DPDK app uses 4 separate ports; port 0, port 1, port 2, and port 3.= They are all on different boards. Thus, they are running at the same tim= e.

Excellent. I apologise for one more dumb question, but does isolating the= very
specific NIC (so that DPDK does not grab the other ones) that is known to= give
strange timestamps, result in the same/unexpected behaviour=3F Just to ma= ke sure.

Thank you.


Sincerely,
Junghan Yoon
On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 10:09 +0900, Ivan Malov <iv= an.malov=40arknetworks.am>, wrote:
&=23160;
Hello,

On Wed, 23 Jul 2025, Yoon Junghan wrote:

&=23160;
Hello,
As advised, I tested hardware timestamps with LRO enabled on our ConnectX= -7 NICs. However, the timestamps of LROed packets still show inconsistent= and abnormally large
gaps from normal
packets.
&=23160;
Interestingly, I found this issue does not appear on all CX-7 NICs. Even = with identical DPDK code, firmware version (28.43.2566), and hardware mod= els from the same
manufacturer, only
specific NICs exhibit this inconsistency.
I have confirmed that:
* All NICs use the same driver and firmware version.
* All NICs are of the same model (MCX75310AAS-NEA=5FAx).
&=23160;


1) Do the two =22NICs=22 ('port 0' and 'port 1' from below printout) repr= esent two
different ports/P=46s of the same physical 'board'/'adapter card' in fact= =3F

2) If (1) is true, were the results obtained by running the application o= n both
ports simultaneously (both managed by the DPDK at the same time)=3F
=
(just to clarify, -- I'm confused by the fact that the NIC driver itself = seems
to invoke 'rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp=5Fregister' for each new RxQ= rather than call
it once and then look-up and reuse the existing offsets for more ports/qu= eue ).

Thank you.

&=23160;
* The issue occurs only when LRO is enabled together with RX hardware tim= estamping.
* Disabling LRO eliminates the issue.
I would appreciate any insight into how this behavior can occur on only s= ome ports despite same software and hardware setup.
&=23160;
Below is my code snippet.
&=23160;
=60=60=60c
/*-----------------------------------------------------------------------= -----*/
static inline int
is=5Ftimestamp=5Fenabled(struct rte=5Fmbuf *mbuf)
=7B
&=23160; &=23160;static uint64=5Ft timestamp=5Frx=5Fdynflag =3D 0;
&=23160; &=23160;int timestamp=5Frx=5Fdynflag=5Foffset;
&=23160;
&=23160; &=23160;if (=21timestamp=5Frx=5Fdynflag)
&=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160;timestamp=5Frx=5Fdynflag=5Foffset =3D<= br /> &=23160; &=23160; &=23160; &=23160; &=23160; &=23160;rte=5Fmbuf=5Fdynflag= =5Flookup(RTE=5FMBU=46=5FDYN=46LAG=5FRX=5FTIMESTAMP=5FNAME, NULL);
&=23160; &=23160; &=23160; &=23160;if (timestamp=5Frx=5Fdynflag=5Foffset = < 0)
&=23160; &=23160; &=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160;return 0;
&=23160; &=23160; &=23160; &=23160;=7D
&=23160; &=23160; &=23160; &=23160;timestamp=5Frx=5Fdynflag =3D RTE=5FBIT= 64(timestamp=5Frx=5Fdynflag=5Foffset);
&=23160; &=23160;=7D
&=23160;
&=23160; &=23160;return mbuf->ol=5Fflags & timestamp=5Frx=5Fdynfla= g;
=7D
/*-----------------------------------------------------------------------= -----*/
static inline rte=5Fmbuf=5Ftimestamp=5Ft *
get=5Ftimestamp(struct rte=5Fmbuf *mbuf)
=7B
&=23160; &=23160;static int timestamp=5Fdynfield=5Foffset =3D -1;
&=23160;
&=23160; &=23160;if (timestamp=5Fdynfield=5Foffset < 0)
&=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160;timestamp=5Fdynfield=5Foffset =3D
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160;rte=5Fmbuf=5Fdynfiel= d=5Flookup(RTE=5FMBU=46=5FDYN=46IELD=5FTIMESTAMP=5FNAME, NULL);
&=23160; &=23160; &=23160; &=23160;if (timestamp=5Fdynfield=5Foffset <= 0)
&=23160; &=23160; &=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160;return 0;
&=23160; &=23160; &=23160; &=23160;=7D
&=23160; &=23160;=7D
&=23160;
&=23160; &=23160;return RTE=5FMBU=46=5FDYN=46IELD(mbuf,
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &= =23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160;timestamp=5F= dynfield=5Foffset,
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &= =23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160;rte=5Fmbuf=5F= timestamp=5Ft *);
=7D
/*-----------------------------------------------------------------------= -----*/
static inline rte=5Fmbuf=5Ftimestamp=5Ft *
get=5Frx=5Fhw=5Ftimestamp(struct rte=5Fmbuf *pkt)
=7B
&=23160; &=23160;if (=21is=5Ftimestamp=5Fenabled(pkt))
&=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160;printf(=22rx=5Fhw=5Ftimestamp not enab= led in mbuf=21=5Cn=22);
&=23160; &=23160; &=23160; &=23160;return NULL;
&=23160; &=23160;=7D
&=23160;
&=23160; &=23160;return get=5Ftimestamp(pkt);
=7D
=60=60=60
&=23160;
My DPDK application prints logs as below.
&=23160;
=60=60=60c
&=23160; &=23160;/* parse HW timestamp */
&=23160; &=23160;rte=5Fmbuf=5Ftimestamp=5Ft *rx=5Ftimestamp =3D get=5Frx=5F= hw=5Ftimestamp(pkt);
&=23160; &=23160;printf(=22=5Bport %d=5D RX HW timestamp: %=23016lx %s=5C= n=22,
&=23160; &=23160; &=23160; &=23160; &=23160; pctx->port=5Fid,
&=23160; &=23160; &=23160; &=23160; &=23160; *rx=5Ftimestamp,
&=23160; &=23160; &=23160; &=23160; &=23160; pkt->ol=5Fflags & PKT= =5FRX=5FLRO =3F =22(LROed)=22 : =22(not LROed)=22);
=60=60=60
&=23160;
Below are observations from two CX-7 ports under identical conditions. &=23160;
Normal NIC (port 0):
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d185b (LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d1911 (LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d19c9 (LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d37ca (LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd30e019 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd3280bb (not LROed)
&=23160;
Erroneous NIC (port 1):
Below is erroneous NIC's timestamp.
=5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
=5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
=5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
=5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
=5Bport 1=5D RX HW timestamp: 0x000080691b7557 (not LROed)
=5Bport 1=5D RX HW timestamp: 0x000080691e2311 (not LROed)
=5Bport 1=5D RX HW timestamp: 0x00008069357553 (not LROed)
=5Bport 1=5D RX HW timestamp: 0x0000806936e8c1 (not LROed)

As shown above, non-LRO packets consistently have normal hardware timesta= mps on both NICs. However, on port 1, all LROed packets return a fixed, i= nvalid timestamp
(0x3e6eef91bc19f0fd),
which is clearly inconsistent.
I have also confirmed that other dynfields (rather than dynfield=5B1=5D a= nd dynfield=5B2=5D) are unused.
&=23160;

Sincerely,
Junghan Yoon
On Jul 22, 2025, 5:31 PM +0900, Ivan Malov <ivan.malov=40arknetworks.a= m>, wrote:
Hello,

On Tue, 22 Jul 2025, Yoon Junghan wrote:

Hello,
&=23160;
I'm currently using DPDK 20.11 with a ConnectX-7 NIC, and I'm trying to r= etrieve RX hardware timestamps using =60rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp= =5Fregister()=60.


Does the application invoke 'rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp=5Fregister= ' on its own=3F If
yes, consider to replace this with invocations of APIs =5B1=5D (with fiel= d name =5B2=5D)
and =5B3=5D (with flag name =5B4=5D). =46or an example, please refer to =5B= 5=5D and =5B6=5D.

This is because, as per =5B7=5D, the driver in question might 'register' = the field
and the flag on its own, in response to 'DEV=5FRX=5FO=46=46LOAD=5FTIMESTA= MP' request, so,
the user application should look up the field/flag, not 'register' it afr= esh.

If this does not help, then consider to clarify whether the timestamps ar= e
accurate (and whether the flag is seen in the mbufs) when LRO is not enab= led.

=5B1=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.html=23= a6adf9b352a83e7d521fd6aa04e305b1c
=5B2=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.html=23= a5159b2d34fa801d171ed0ccce451121b
=5B3=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.html=23= a89d835027034f76a27eb2afe7987ae35
=5B4=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.html=23= a831d7066c7193788351797a65186848a
=5B5=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469= b7f6e6207/app/test-pmd/util.c=23L44
=5B6=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469= b7f6e6207/app/test-pmd/util.c=23L60
=5B7=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469= b7f6e6207/drivers/net/mlx5/mlx5=5Frxq.c=23L1743

Thank you.

&=23160;
When LRO is enabled, I notice that LROed mbufs seem to share identical ti= mestamp values, and the timestamps are unexpectedly large or inconsistent= . This raises
the question of whether
LRO is interfering with the correctness of the RX HW timestamps.
&=23160;
I=E2=80=99d appreciate any clarification on whether HW RX timestamping is= reliable when LRO is enabled on this platform, or if LRO should be just = disabled for accurate
per-packet timestamping.
&=23160;

Sincerely,
Junghan Yoon


&=23160;


--6880e8cc_749abb43_5997--