From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2FC3446BF2 for ; Wed, 23 Jul 2025 17:20:50 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 029EB40DD1; Wed, 23 Jul 2025 17:20:50 +0200 (CEST) Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by mails.dpdk.org (Postfix) with ESMTP id 73FB8402A9 for ; Wed, 23 Jul 2025 17:20:48 +0200 (CEST) Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-748feca4a61so42514b3a.3 for ; Wed, 23 Jul 2025 08:20:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1753284047; x=1753888847; darn=dpdk.org; h=mime-version:subject:references:in-reply-to:message-id:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=oU30XBSci2Q8syk1FVaEuuJtSsKMvze0a3Mw9kiE7aQ=; b=G17ml7lBpI0QbLdgiti/pno/LDjRQhMEvfW7INwR9zJcTBtLrvNzcUk0e/aNqGolSe BobJBUrAAFz79dEoBY8zX4c36XJTNzS7j67/NK0qaSvSdklsRoy452Y4DUSnHlUprbEh GkM0HCVuHaYRU9qP44ES7h1u6QT+N2E1WbpwWKFfOLwBN8TnsJsKYEXxHxzrIGvBZ+LH lMVXnqbugszyn2lj9PLMTL7AfwvYoHll1EChWsdSChj1k69ao1tkrdEjithxJq1KR+tM hdDmS750B2nB0ZISLLS9KjNwfqeC/Fm0njeKADL8w7GjZvwf0BrqT8f9fM619pNXnn1t mSzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1753284047; x=1753888847; h=mime-version:subject:references:in-reply-to:message-id:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=oU30XBSci2Q8syk1FVaEuuJtSsKMvze0a3Mw9kiE7aQ=; b=HjkxCatVRGc4mskZ/ttV2o4BVgYt/6ipJoqE+dFvu7dAS0EZtlZcS163pUQsbJcAwx ykBQVg/HAkp+CMR2/VEApitSUHOUMrbysfezyIS56Pd274cFy74qw8j4XnO2CJ2jCiZR cjwqJ14vb1Gqfn6sBDIKJt4M1rVfEko31qxGW2q+dxNSy6NsgXGtxqIVOI+kPi4PBNeY anQ89s+1H+3xpt+n5f3tXj83vILYGu0uuDOcoO15xU3TrJOYw3XQ2UbZKkDh3OYltjDT 605iEwcBaOK0TWaCHHqweF4T+bG2C4jX6VTGZWCuYUdrPaZnbqGbl/CJNyoFnMzSRoQG UQcQ== X-Gm-Message-State: AOJu0Yz72vnTstLMHEFeJbfhfjmXfA8TqtIRlR2NBRXiPf3Z6GBG68Ha Lx54vEiBgmHutx2g6W7T680C+E+7PnsltUoXpSEPbI3Joa3BdUh2SOTu X-Gm-Gg: ASbGnctGV2AXwR1rKciVzFfPGUH5xmm4Wk0YFwKfyt3wf30lAXAZrFCR/H8OP/X8SO3 Z558cBhNeY6ngYqdd5lzQnrxBcRRjWAUh6RaZSVDhbBuuk1t5Yzb9U/wflmII3jqQkwpnSYvJi4 7gfNx6YKF4QeLrCoueCxALjCnyl4dt5/jSgq0v3WBaxF8QSm5Gs+OvOTSt2FKtjst9LZfjmfcnY MiYBJpwd7E0TOjNfsaJBxhhHhmgNF9UgnzsBaqaIcYQM1wM31SHH0OnBsKoImg4xX0Z6UIN7tqz e4qJliQoYM4GlYR0mxlMKmaKKNYi0b78CvtrK1qjR/Uqd+RfITFsxfD+sTFNk5JPm56Axdfvu/d UPcGaCo/GiDldUA2H0TPeRRXX X-Google-Smtp-Source: AGHT+IHwJZGOjdJiNk0TKIG9ZNpiQrZX1EGti9BwMNed+skBCw8UoM5JuAs87PyVWqHLYbaBiSiokQ== X-Received: by 2002:a05:6a00:1989:b0:75f:7fc3:41ef with SMTP id d2e1a72fcca58-7603450bc55mr5050260b3a.6.1753284047118; Wed, 23 Jul 2025 08:20:47 -0700 (PDT) Received: from [192.168.0.12] ([147.47.193.124]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-761108e3e26sm720499b3a.81.2025.07.23.08.20.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Jul 2025 08:20:46 -0700 (PDT) Date: Thu, 24 Jul 2025 00:20:34 +0900 From: Yoon Junghan To: Ivan Malov Cc: "=?utf-8?Q?users=40dpdk.org?=" Message-ID: <8cbcf9f0-5889-4b11-a98c-8eac24d8343a@Spark> In-Reply-To: References: <3f7df93b-ee00-bcdb-337c-8faa7bd73f6c@arknetworks.am> <391481b9-1b18-5567-a8a0-563c5c5eb37c@arknetworks.am> <8f3a156a-2f2a-4418-a5e0-47a4450cd10a@Spark> <66490314-cb05-0018-4457-9bfab077048a@arknetworks.am> <2c092b37-e8cd-4161-bebf-e5031b7c2990@Spark> Subject: Re: HW RX timestamp with LRO enabled on ConnectX-7 (DPDK 20.11) X-Readdle-Message-ID: 8cbcf9f0-5889-4b11-a98c-8eac24d8343a@Spark MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="6880fdc7_235ba861_5997" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --6880fdc7_235ba861_5997 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline I'm not sure I did well. All interface show the same result. current settings: tx=5Ftype 0 rx=5Ffilter 0 Sincerely, Junghan Yoon On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 11:19 +0900, Ivan Malov , wrote: > On Wed, 23 Jul 2025, Yoon Junghan wrote: > > > I isolated port 1 using -a option for EAL parameter and got the simil= ar result. > > > > Note that port 1 becomes port 0 in this time. > > =5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00042819272fad (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x000428192e6e77 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x000428192e7f01 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x000428192e833d (not LROed) > > > > =46YI, I have 4 CX-7 on the same machine. (eth0 =3D port 0, ... eth3 = =3D port 3 in DPDK) > > pci=400000:16:00.0 =C2=A0eth0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= network =C2=A0 =C2=A0 =C2=A0 =C2=A0MT2910 =46amily =5BConnectX-7=5D > > pci=400000:40:00.0 =C2=A0eth1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= network =C2=A0 =C2=A0 =C2=A0 =C2=A0MT2910 =46amily =5BConnectX-7=5D > > pci=400000:6a:00.0 =C2=A0eth2 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= network =C2=A0 =C2=A0 =C2=A0 =C2=A0MT2910 =46amily =5BConnectX-7=5D > > pci=400000:94:00.0 =C2=A0eth3 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= network =C2=A0 =C2=A0 =C2=A0 =C2=A0MT2910 =46amily =5BConnectX-7=5D > > > > Among them, only the first CX-7 shows consistent timestamp regardless= of LRO. > > Does 'sudo hwstamp=5Fctl -i ' show consistent results across al= l the NICs=3F > > Thank you. > > > > > Sincerely, > > Junghan Yoon > > On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 10:28 +0900, Ivan Malov , wrote: > > On Wed, 23 Jul 2025, Yoon Junghan wrote: > > > > Thank you for quick response. > > > > 1) They are different NICs. Not in the same board. Separate adapters = in different PCIe slots. > > 2) My DPDK app uses 4 separate ports; port 0, port 1, port 2, and por= t 3. They are all on different boards. Thus, they are running at the same= time. > > > > > > Excellent. I apologise for one more dumb question, but does isolating= the very > > specific NIC (so that DPDK does not grab the other ones) that is know= n to give > > strange timestamps, result in the same/unexpected behaviour=3F Just t= o make sure. > > > > Thank you. > > > > > > Sincerely, > > Junghan Yoon > > On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 10:09 +0900, Ivan Malov , wrote: > > > > Hello, > > > > On Wed, 23 Jul 2025, Yoon Junghan wrote: > > > > > > Hello, > > As advised, I tested hardware timestamps with LRO enabled on our Conn= ectX-7 NICs. However, the timestamps of LROed packets still show inconsis= tent and abnormally > > large > > gaps from normal > > packets. > > > > Interestingly, I found this issue does not appear on all CX-7 NICs. E= ven with identical DPDK code, firmware version (28.43.2566), and hardware= models from the > > same > > manufacturer, only > > specific NICs exhibit this inconsistency. > > I have confirmed that: > > * All NICs use the same driver and firmware version. > > * All NICs are of the same model (MCX75310AAS-NEA=5FAx). > > > > > > > > 1) Do the two =22NICs=22 ('port 0' and 'port 1' from below printout) = represent two > > different ports/P=46s of the same physical 'board'/'adapter card' in = fact=3F > > > > 2) If (1) is true, were the results obtained by running the applicati= on on both > > ports simultaneously (both managed by the DPDK at the same time)=3F > > > > (just to clarify, -- I'm confused by the fact that the NIC driver its= elf seems > > to invoke 'rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp=5Fregister' for each new= RxQ rather than call > > it once and then look-up and reuse the existing offsets for more port= s/queue ). > > > > Thank you. > > > > > > * The issue occurs only when LRO is enabled together with RX hardware= timestamping. > > * Disabling LRO eliminates the issue. > > I would appreciate any insight into how this behavior can occur on on= ly some ports despite same software and hardware setup. > > > > Below is my code snippet. > > > > =60=60=60c > > /*-------------------------------------------------------------------= ---------*/ > > static inline int > > is=5Ftimestamp=5Fenabled(struct rte=5Fmbuf *mbuf) > > =7B > > =C2=A0 =C2=A0static uint64=5Ft timestamp=5Frx=5Fdynflag =3D 0; > > =C2=A0 =C2=A0int timestamp=5Frx=5Fdynflag=5Foffset; > > > > =C2=A0 =C2=A0if (=21timestamp=5Frx=5Fdynflag) > > =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0timestamp=5Frx=5Fdynflag=5Foffset =3D > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rte=5Fmbuf=5Fdynflag=5Flooku= p(RTE=5FMBU=46=5FDYN=46LAG=5FRX=5FTIMESTAMP=5FNAME, NULL); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0if (timestamp=5Frx=5Fdynflag=5Foffset < 0)= > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return 0; > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=7D > > =C2=A0 =C2=A0 =C2=A0 =C2=A0timestamp=5Frx=5Fdynflag =3D RTE=5FBIT64(t= imestamp=5Frx=5Fdynflag=5Foffset); > > =C2=A0 =C2=A0=7D > > > > =C2=A0 =C2=A0return mbuf->ol=5Fflags & timestamp=5Frx=5Fdynflag; > > =7D > > /*-------------------------------------------------------------------= ---------*/ > > static inline rte=5Fmbuf=5Ftimestamp=5Ft * > > get=5Ftimestamp(struct rte=5Fmbuf *mbuf) > > =7B > > =C2=A0 =C2=A0static int timestamp=5Fdynfield=5Foffset =3D -1; > > > > =C2=A0 =C2=A0if (timestamp=5Fdynfield=5Foffset < 0) > > =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0timestamp=5Fdynfield=5Foffset =3D > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rte=5Fmbuf=5Fdynfield=5Flook= up(RTE=5FMBU=46=5FDYN=46IELD=5FTIMESTAMP=5FNAME, NULL); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0if (timestamp=5Fdynfield=5Foffset < 0) > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return 0; > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=7D > > =C2=A0 =C2=A0=7D > > > > =C2=A0 =C2=A0return RTE=5FMBU=46=5FDYN=46IELD(mbuf, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0timestamp=5Fdynfield=5Foffset, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rte=5Fmbuf=5Ftimestamp=5Ft *); > > =7D > > /*-------------------------------------------------------------------= ---------*/ > > static inline rte=5Fmbuf=5Ftimestamp=5Ft * > > get=5Frx=5Fhw=5Ftimestamp(struct rte=5Fmbuf *pkt) > > =7B > > =C2=A0 =C2=A0if (=21is=5Ftimestamp=5Fenabled(pkt)) > > =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0printf(=22rx=5Fhw=5Ftimestamp not enabled = in mbuf=21=5Cn=22); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0return NULL; > > =C2=A0 =C2=A0=7D > > > > =C2=A0 =C2=A0return get=5Ftimestamp(pkt); > > =7D > > =60=60=60 > > > > My DPDK application prints logs as below. > > > > =60=60=60c > > =C2=A0 =C2=A0/* parse HW timestamp */ > > =C2=A0 =C2=A0rte=5Fmbuf=5Ftimestamp=5Ft *rx=5Ftimestamp =3D get=5Frx=5F= hw=5Ftimestamp(pkt); > > =C2=A0 =C2=A0printf(=22=5Bport %d=5D RX HW timestamp: %=23016lx %s=5C= n=22, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pctx->port=5Fid, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 *rx=5Ftimestamp, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pkt->ol=5Fflags & PKT=5FRX=5FLRO =3F= =22(LROed)=22 : =22(not LROed)=22); > > =60=60=60 > > > > Below are observations from two CX-7 ports under identical conditions= . > > > > Normal NIC (port 0): > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d185b (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d1911 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d19c9 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d37ca (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d4cb3 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d4cb3 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd30e019 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd3280bb (not LROed) > > > > Erroneous NIC (port 1): > > Below is erroneous NIC's timestamp. > > =5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > =5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > =5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > =5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > =5Bport 1=5D RX HW timestamp: 0x000080691b7557 (not LROed) > > =5Bport 1=5D RX HW timestamp: 0x000080691e2311 (not LROed) > > =5Bport 1=5D RX HW timestamp: 0x00008069357553 (not LROed) > > =5Bport 1=5D RX HW timestamp: 0x0000806936e8c1 (not LROed) > > > > As shown above, non-LRO packets consistently have normal hardware tim= estamps on both NICs. However, on port 1, all LROed packets return a fixe= d, invalid timestamp > > (0x3e6eef91bc19f0fd), > > which is clearly inconsistent. > > I have also confirmed that other dynfields (rather than dynfield=5B1=5D= and dynfield=5B2=5D) are unused. > > > > > > Sincerely, > > Junghan Yoon > > On Jul 22, 2025, 5:31 PM +0900, Ivan Malov , wrote: > > Hello, > > > > On Tue, 22 Jul 2025, Yoon Junghan wrote: > > > > Hello, > > > > I'm currently using DPDK 20.11 with a ConnectX-7 NIC, and I'm trying = to retrieve RX hardware timestamps using =60rte=5Fmbuf=5Fdyn=5Frx=5Ftimes= tamp=5Fregister()=60. > > > > > > Does the application invoke 'rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp=5Fregi= ster' on its own=3F If > > yes, consider to replace this with invocations of APIs =5B1=5D (with = field name =5B2=5D) > > and =5B3=5D (with flag name =5B4=5D). =46or an example, please refer = to =5B5=5D and =5B6=5D. > > > > This is because, as per =5B7=5D, the driver in question might 'regist= er' the field > > and the flag on its own, in response to 'DEV=5FRX=5FO=46=46LOAD=5FTIM= ESTAMP' request, so, > > the user application should look up the field/flag, not 'register' it= afresh. > > > > If this does not help, then consider to clarify whether the timestamp= s are > > accurate (and whether the flag is seen in the mbufs) when LRO is not = enabled. > > > > =5B1=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.ht= ml=23a6adf9b352a83e7d521fd6aa04e305b1c > > =5B2=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.ht= ml=23a5159b2d34fa801d171ed0ccce451121b > > =5B3=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.ht= ml=23a89d835027034f76a27eb2afe7987ae35 > > =5B4=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.ht= ml=23a831d7066c7193788351797a65186848a > > =5B5=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b9659= 7469b7f6e6207/app/test-pmd/util.c=23L44 > > =5B6=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b9659= 7469b7f6e6207/app/test-pmd/util.c=23L60 > > =5B7=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b9659= 7469b7f6e6207/drivers/net/mlx5/mlx5=5Frxq.c=23L1743 > > > > Thank you. > > > > > > When LRO is enabled, I notice that LROed mbufs seem to share identica= l timestamp values, and the timestamps are unexpectedly large or inconsis= tent. This raises > > the question of whether > > LRO is interfering with the correctness of the RX HW timestamps. > > > > I=E2=80=99d appreciate any clarification on whether HW RX timestampin= g is reliable when LRO is enabled on this platform, or if LRO should be j= ust disabled for accurate > > per-packet timestamping. > > > > > > Sincerely, > > Junghan Yoon > > > > > > > > > > > > --6880fdc7_235ba861_5997 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
I'm not sure I did well. All int= erface show the same result.
&=23160;
current settings:
tx=5Ftype 0
rx=5Ffilter 0

Sincerely,
Junghan Yoon
On 2025=EB=85=84 7=EC=9B=94 23=EC=9D= =BC PM 11:19 +0900, Ivan Malov <ivan.malov=40arknetworks.am>, wrote= :
On Wed, 23 Jul 2025, Yoon Junghan wrote:
I isolated port 1 using -a option for EAL parameter and got the= similar result.
&=23160;
Note that port 1 becomes port 0 in this time.
=5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed)
=5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed)
=5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed)
=5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed)
=5Bport 0=5D RX HW timestamp: 0x00042819272fad (not LROed)
=5Bport 0=5D RX HW timestamp: 0x000428192e6e77 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x000428192e7f01 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x000428192e833d (not LROed)
&=23160;
=46YI, I have 4 CX-7 on the same machine. (eth0 =3D port 0, ... eth3 =3D = port 3 in DPDK)
pci=400000:16:00.0 &=23160;eth0 &=23160; &=23160; &=23160; &=23160; &=231= 60; &=23160; network &=23160; &=23160; &=23160; &=23160;MT2910 =46amily =5B= ConnectX-7=5D
pci=400000:40:00.0 &=23160;eth1 &=23160; &=23160; &=23160; &=23160; &=231= 60; &=23160; network &=23160; &=23160; &=23160; &=23160;MT2910 =46amily =5B= ConnectX-7=5D
pci=400000:6a:00.0 &=23160;eth2 &=23160; &=23160; &=23160; &=23160; &=231= 60; &=23160; network &=23160; &=23160; &=23160; &=23160;MT2910 =46amily =5B= ConnectX-7=5D
pci=400000:94:00.0 &=23160;eth3 &=23160; &=23160; &=23160; &=23160; &=231= 60; &=23160; network &=23160; &=23160; &=23160; &=23160;MT2910 =46amily =5B= ConnectX-7=5D
&=23160;
Among them, only the first CX-7 shows consistent timestamp regardless of = LRO.

Does 'sudo hwstamp=5Fctl -i <ifname>' show consistent results acros= s all the NICs=3F

Thank you.


Sincerely,
Junghan Yoon
On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 10:28 +0900, Ivan Malov <iv= an.malov=40arknetworks.am>, wrote:
On Wed, 23 Jul 2025, Yoon Junghan wrote:

Thank you for quick response.
&=23160;
1) They are different NICs. Not in the same board. Separate adapters in d= ifferent PCIe slots.
2) My DPDK app uses 4 separate ports; port 0, port 1, port 2, and port 3.= They are all on different boards. Thus, they are running at the same tim= e.


Excellent. I apologise for one more dumb question, but does isolating the= very
specific NIC (so that DPDK does not grab the other ones) that is known to= give
strange timestamps, result in the same/unexpected behaviour=3F Just to ma= ke sure.

Thank you.


Sincerely,
Junghan Yoon
On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 10:09 +0900, Ivan Malov <iv= an.malov=40arknetworks.am>, wrote:
&=23160;
Hello,

On Wed, 23 Jul 2025, Yoon Junghan wrote:

&=23160;
Hello,
As advised, I tested hardware timestamps with LRO enabled on our ConnectX= -7 NICs. However, the timestamps of LROed packets still show inconsistent= and abnormally
large
gaps from normal
packets.
&=23160;
Interestingly, I found this issue does not appear on all CX-7 NICs. Even = with identical DPDK code, firmware version (28.43.2566), and hardware mod= els from the
same
manufacturer, only
specific NICs exhibit this inconsistency.
I have confirmed that:
* All NICs use the same driver and firmware version.
* All NICs are of the same model (MCX75310AAS-NEA=5FAx).
&=23160;


1) Do the two =22NICs=22 ('port 0' and 'port 1' from below printout) repr= esent two
different ports/P=46s of the same physical 'board'/'adapter card' in fact= =3F

2) If (1) is true, were the results obtained by running the application o= n both
ports simultaneously (both managed by the DPDK at the same time)=3F
=
(just to clarify, -- I'm confused by the fact that the NIC driver itself = seems
to invoke 'rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp=5Fregister' for each new RxQ= rather than call
it once and then look-up and reuse the existing offsets for more ports/qu= eue ).

Thank you.

&=23160;
* The issue occurs only when LRO is enabled together with RX hardware tim= estamping.
* Disabling LRO eliminates the issue.
I would appreciate any insight into how this behavior can occur on only s= ome ports despite same software and hardware setup.
&=23160;
Below is my code snippet.
&=23160;
=60=60=60c
/*-----------------------------------------------------------------------= -----*/
static inline int
is=5Ftimestamp=5Fenabled(struct rte=5Fmbuf *mbuf)
=7B
&=23160; &=23160;static uint64=5Ft timestamp=5Frx=5Fdynflag =3D 0;
&=23160; &=23160;int timestamp=5Frx=5Fdynflag=5Foffset;
&=23160;
&=23160; &=23160;if (=21timestamp=5Frx=5Fdynflag)
&=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160;timestamp=5Frx=5Fdynflag=5Foffset =3D<= br /> &=23160; &=23160; &=23160; &=23160; &=23160; &=23160;rte=5Fmbuf=5Fdynflag= =5Flookup(RTE=5FMBU=46=5FDYN=46LAG=5FRX=5FTIMESTAMP=5FNAME, NULL);
&=23160; &=23160; &=23160; &=23160;if (timestamp=5Frx=5Fdynflag=5Foffset = < 0)
&=23160; &=23160; &=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160;return 0;
&=23160; &=23160; &=23160; &=23160;=7D
&=23160; &=23160; &=23160; &=23160;timestamp=5Frx=5Fdynflag =3D RTE=5FBIT= 64(timestamp=5Frx=5Fdynflag=5Foffset);
&=23160; &=23160;=7D
&=23160;
&=23160; &=23160;return mbuf->ol=5Fflags & timestamp=5Frx=5Fdynfla= g;
=7D
/*-----------------------------------------------------------------------= -----*/
static inline rte=5Fmbuf=5Ftimestamp=5Ft *
get=5Ftimestamp(struct rte=5Fmbuf *mbuf)
=7B
&=23160; &=23160;static int timestamp=5Fdynfield=5Foffset =3D -1;
&=23160;
&=23160; &=23160;if (timestamp=5Fdynfield=5Foffset < 0)
&=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160;timestamp=5Fdynfield=5Foffset =3D
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160;rte=5Fmbuf=5Fdynfiel= d=5Flookup(RTE=5FMBU=46=5FDYN=46IELD=5FTIMESTAMP=5FNAME, NULL);
&=23160; &=23160; &=23160; &=23160;if (timestamp=5Fdynfield=5Foffset <= 0)
&=23160; &=23160; &=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160;return 0;
&=23160; &=23160; &=23160; &=23160;=7D
&=23160; &=23160;=7D
&=23160;
&=23160; &=23160;return RTE=5FMBU=46=5FDYN=46IELD(mbuf,
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &= =23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160;timestamp=5F= dynfield=5Foffset,
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &= =23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160;rte=5Fmbuf=5F= timestamp=5Ft *);
=7D
/*-----------------------------------------------------------------------= -----*/
static inline rte=5Fmbuf=5Ftimestamp=5Ft *
get=5Frx=5Fhw=5Ftimestamp(struct rte=5Fmbuf *pkt)
=7B
&=23160; &=23160;if (=21is=5Ftimestamp=5Fenabled(pkt))
&=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160;printf(=22rx=5Fhw=5Ftimestamp not enab= led in mbuf=21=5Cn=22);
&=23160; &=23160; &=23160; &=23160;return NULL;
&=23160; &=23160;=7D
&=23160;
&=23160; &=23160;return get=5Ftimestamp(pkt);
=7D
=60=60=60
&=23160;
My DPDK application prints logs as below.
&=23160;
=60=60=60c
&=23160; &=23160;/* parse HW timestamp */
&=23160; &=23160;rte=5Fmbuf=5Ftimestamp=5Ft *rx=5Ftimestamp =3D get=5Frx=5F= hw=5Ftimestamp(pkt);
&=23160; &=23160;printf(=22=5Bport %d=5D RX HW timestamp: %=23016lx %s=5C= n=22,
&=23160; &=23160; &=23160; &=23160; &=23160; pctx->port=5Fid,
&=23160; &=23160; &=23160; &=23160; &=23160; *rx=5Ftimestamp,
&=23160; &=23160; &=23160; &=23160; &=23160; pkt->ol=5Fflags & PKT= =5FRX=5FLRO =3F =22(LROed)=22 : =22(not LROed)=22);
=60=60=60
&=23160;
Below are observations from two CX-7 ports under identical conditions. &=23160;
Normal NIC (port 0):
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d185b (LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d1911 (LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d19c9 (LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d37ca (LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd30e019 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd3280bb (not LROed)
&=23160;
Erroneous NIC (port 1):
Below is erroneous NIC's timestamp.
=5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
=5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
=5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
=5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
=5Bport 1=5D RX HW timestamp: 0x000080691b7557 (not LROed)
=5Bport 1=5D RX HW timestamp: 0x000080691e2311 (not LROed)
=5Bport 1=5D RX HW timestamp: 0x00008069357553 (not LROed)
=5Bport 1=5D RX HW timestamp: 0x0000806936e8c1 (not LROed)

As shown above, non-LRO packets consistently have normal hardware timesta= mps on both NICs. However, on port 1, all LROed packets return a fixed, i= nvalid timestamp
(0x3e6eef91bc19f0fd),
which is clearly inconsistent.
I have also confirmed that other dynfields (rather than dynfield=5B1=5D a= nd dynfield=5B2=5D) are unused.
&=23160;

Sincerely,
Junghan Yoon
On Jul 22, 2025, 5:31 PM +0900, Ivan Malov <ivan.malov=40arknetworks.a= m>, wrote:
Hello,

On Tue, 22 Jul 2025, Yoon Junghan wrote:

Hello,
&=23160;
I'm currently using DPDK 20.11 with a ConnectX-7 NIC, and I'm trying to r= etrieve RX hardware timestamps using =60rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp= =5Fregister()=60.


Does the application invoke 'rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp=5Fregister= ' on its own=3F If
yes, consider to replace this with invocations of APIs =5B1=5D (with fiel= d name =5B2=5D)
and =5B3=5D (with flag name =5B4=5D). =46or an example, please refer to =5B= 5=5D and =5B6=5D.

This is because, as per =5B7=5D, the driver in question might 'register' = the field
and the flag on its own, in response to 'DEV=5FRX=5FO=46=46LOAD=5FTIMESTA= MP' request, so,
the user application should look up the field/flag, not 'register' it afr= esh.

If this does not help, then consider to clarify whether the timestamps ar= e
accurate (and whether the flag is seen in the mbufs) when LRO is not enab= led.

=5B1=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.html=23= a6adf9b352a83e7d521fd6aa04e305b1c
=5B2=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.html=23= a5159b2d34fa801d171ed0ccce451121b
=5B3=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.html=23= a89d835027034f76a27eb2afe7987ae35
=5B4=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.html=23= a831d7066c7193788351797a65186848a
=5B5=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469= b7f6e6207/app/test-pmd/util.c=23L44
=5B6=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469= b7f6e6207/app/test-pmd/util.c=23L60
=5B7=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469= b7f6e6207/drivers/net/mlx5/mlx5=5Frxq.c=23L1743

Thank you.

&=23160;
When LRO is enabled, I notice that LROed mbufs seem to share identical ti= mestamp values, and the timestamps are unexpectedly large or inconsistent= . This raises
the question of whether
LRO is interfering with the correctness of the RX HW timestamps.
&=23160;
I=E2=80=99d appreciate any clarification on whether HW RX timestamping is= reliable when LRO is enabled on this platform, or if LRO should be just = disabled for accurate
per-packet timestamping.
&=23160;

Sincerely,
Junghan Yoon


&=23160;



--6880fdc7_235ba861_5997--