From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DDB4F46BF8 for ; Thu, 24 Jul 2025 07:13:23 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7943640279; Thu, 24 Jul 2025 07:13:23 +0200 (CEST) Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by mails.dpdk.org (Postfix) with ESMTP id B330240267 for ; Thu, 24 Jul 2025 07:13:21 +0200 (CEST) Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-3135f3511bcso507487a91.0 for ; Wed, 23 Jul 2025 22:13:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1753334001; x=1753938801; darn=dpdk.org; h=mime-version:subject:references:in-reply-to:message-id:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=feETYR3CmzaOnp/wXMfk4/ZVJ6QmCv+R5n7u6l2DPmk=; b=M9rDP/sMWftn4GxxFeFxwujyEMaDxFS5TRy3lHbYE5MFqTzRtCF0VxhjsqJHE7Y8e2 orp62jAzdHsVo7cEDu8WLMfqbPvOk2JXzEd2XgyuzpTp0qNLWIFYlVGTn6VBO8RnhcfF +dAb+PUDxcXmsRMUaqtMlJnLgLdYTD5mvQPWB6w65tV0SP4GeGXv5d53XEw4Cj9siZq7 3WGSTT9YgO7Um6f4Kj/BCNIczhQEONoUHOAEzMBmRpaH9o4hSawiPXOBYZ02b7YRw6xF BbGCZ9vj0LqZpZ6XeiaHJAV2PIkLN0Wp7h5T/0sGziKyCoFuGO/OgF3lNBd0S+oYzxPy p0sQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1753334001; x=1753938801; h=mime-version:subject:references:in-reply-to:message-id:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=feETYR3CmzaOnp/wXMfk4/ZVJ6QmCv+R5n7u6l2DPmk=; b=RG1m8o+xU3ohBn5Njf4rbRUzYsEr2ks7meUiluYHWqL5XUQcqGNSzuRfz6Bm+hVKjl dkEPrJMWoiMB62O8/Ajj3oMtC+N9V+Mkq3RM8sS6D6FTfa5E0FUxaSXORBThiIRK+xwA 3q1R3yGNRUmXAt6o/nYyc5XedxyRAcKrmA1es7f71cmhVHkuDWwwoRy79jCU4Ds/DNd5 6yObJq4XLcG+nMUM862Q4GsCH2oc+4xp4Oks528yXXvn9Dn2PuGUFTDqut0EUV6cOTwg aNqZ0Lox0Ymg0U5YpDuR7yVoBJjS49R1JPY+FuDUt1Aw7jqpbGmKvDqWQwM6tZMK0S2H s7Sw== X-Gm-Message-State: AOJu0YxELfDXtIdZr+krSRPaE4UCbRBl6thdKq81IVTZJNO+Lm/nr/eb JZQI4mMa/T8W/BknvliH7MWEqoMmwXWJ1RRA9L0oe8kMqMpQujllSRGE X-Gm-Gg: ASbGncv3Xj441MkmsOogIGN5rnXE/q8BpfKBMw+1cYgKz4V4gwxr3WJ5j51FXLB8ygd SvGtc6byYyhvPZP28My+X69fOi4kdcsiN8NhFGCZTWvmTyJjl2vS8iEsvBWi9J7ulTupguiG004 9qWqb+Gy+byLER5VRJKF+qnYnNU5G/OxvdLalMvqLgNkjHfuCxJL0oplmpyPwzH8rGhVvM+W90V C/bxy7a8B+e5iODfOeGVCASnIL0oJ1pe/ptj+jVOY6s5zA3DykpVaz7NkqFdRT8AbkFqQs4Ui3H cxpo1XahHyIFqFAHIxJEzocta5j5chc/3MPlgdCkx/RTqamXisIufAAr4OGLEZ+y1/+Ix5lJB7B J+6bsSP63cFIX1pn94kMAWiuplWdzGy3KJ1sykBDAAA== X-Google-Smtp-Source: AGHT+IH6gw8F9GrHn8/l7Q8Zk3EwyVkzAlLW3wYmrfV8Yd37jTnnqu3o2oyTvWiD+2yjSn8mBCFo8Q== X-Received: by 2002:a17:90b:3904:b0:313:d79e:1f48 with SMTP id 98e67ed59e1d1-31e507b3b7amr8743764a91.16.1753334000075; Wed, 23 Jul 2025 22:13:20 -0700 (PDT) Received: from [147.46.248.138] ([147.46.248.138]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-31e63a766d9sm439660a91.2.2025.07.23.22.13.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Jul 2025 22:13:19 -0700 (PDT) Date: Thu, 24 Jul 2025 14:12:39 +0900 From: Yoon Junghan To: Ivan Malov Cc: "=?utf-8?Q?users=40dpdk.org?=" Message-ID: <72d04ff6-9018-421b-9cd6-4260dd7a3ee7@Spark> In-Reply-To: <6888c20e-d18d-d4fc-fe5f-b4dbbb5fdb30@arknetworks.am> References: <3f7df93b-ee00-bcdb-337c-8faa7bd73f6c@arknetworks.am> <391481b9-1b18-5567-a8a0-563c5c5eb37c@arknetworks.am> <8f3a156a-2f2a-4418-a5e0-47a4450cd10a@Spark> <66490314-cb05-0018-4457-9bfab077048a@arknetworks.am> <2c092b37-e8cd-4161-bebf-e5031b7c2990@Spark> <8cbcf9f0-5889-4b11-a98c-8eac24d8343a@Spark> <6888c20e-d18d-d4fc-fe5f-b4dbbb5fdb30@arknetworks.am> Subject: Re: HW RX timestamp with LRO enabled on ConnectX-7 (DPDK 20.11) X-Readdle-Message-ID: 72d04ff6-9018-421b-9cd6-4260dd7a3ee7@Spark MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="6881c0cc_17e8da44_bfe4" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --6881c0cc_17e8da44_bfe4 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline I found the key difference: when TCP timestamps (R=46C 7323) are enabled = on the TCP sender, the RX HW timestamp of LROed packets on the DPDK PMD o= n middlebox machine becomes inconsistent or invalid. Is there a known lim= itation or erratum in the mlx5 driver or CX7 firmware regarding this=3F Sincerely, Junghan Yoon On Jul 24, 2025, 12:26 AM +0900, Ivan Malov = , wrote: > On Thu, 24 Jul 2025, Yoon Junghan wrote: > > > I'm not sure I did well. All interface show the same result. > > > > current settings: > > tx=5Ftype 0 > > rx=5Ffilter 0 > > But that should mean.. no timestamping=3F > > 1) May be also check 'sudo ethtool -T '. > 2) May be try to enable 'sudo hwstamp=5Fctl -i -r 1'. > > Thank you. > > > > > Sincerely, > > Junghan Yoon > > On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 11:19 +0900, Ivan Malov , wrote: > > On Wed, 23 Jul 2025, Yoon Junghan wrote: > > > > I isolated port 1 using -a option for EAL parameter and got the simil= ar result. > > > > Note that port 1 becomes port 0 in this time. > > =5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00042819272fad (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x000428192e6e77 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x000428192e7f01 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x000428192e833d (not LROed) > > > > =46YI, I have 4 CX-7 on the same machine. (eth0 =3D port 0, ... eth3 = =3D port 3 in DPDK) > > pci=400000:16:00.0 =C2=A0eth0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= network =C2=A0 =C2=A0 =C2=A0 =C2=A0MT2910 =46amily =5BConnectX-7=5D > > pci=400000:40:00.0 =C2=A0eth1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= network =C2=A0 =C2=A0 =C2=A0 =C2=A0MT2910 =46amily =5BConnectX-7=5D > > pci=400000:6a:00.0 =C2=A0eth2 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= network =C2=A0 =C2=A0 =C2=A0 =C2=A0MT2910 =46amily =5BConnectX-7=5D > > pci=400000:94:00.0 =C2=A0eth3 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= network =C2=A0 =C2=A0 =C2=A0 =C2=A0MT2910 =46amily =5BConnectX-7=5D > > > > Among them, only the first CX-7 shows consistent timestamp regardless= of LRO. > > > > > > Does 'sudo hwstamp=5Fctl -i ' show consistent results across = all the NICs=3F > > > > Thank you. > > > > > > Sincerely, > > Junghan Yoon > > On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 10:28 +0900, Ivan Malov , wrote: > > On Wed, 23 Jul 2025, Yoon Junghan wrote: > > > > Thank you for quick response. > > > > 1) They are different NICs. Not in the same board. Separate adapters = in different PCIe slots. > > 2) My DPDK app uses 4 separate ports; port 0, port 1, port 2, and por= t 3. They are all on different boards. Thus, they are running at the same= time. > > > > > > Excellent. I apologise for one more dumb question, but does isolating= the very > > specific NIC (so that DPDK does not grab the other ones) that is know= n to give > > strange timestamps, result in the same/unexpected behaviour=3F Just t= o make sure. > > > > Thank you. > > > > > > Sincerely, > > Junghan Yoon > > On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 10:09 +0900, Ivan Malov , wrote: > > > > Hello, > > > > On Wed, 23 Jul 2025, Yoon Junghan wrote: > > > > > > Hello, > > As advised, I tested hardware timestamps with LRO enabled on our Conn= ectX-7 NICs. However, the timestamps of LROed packets still show inconsis= tent and abnormally > > large > > gaps from normal > > packets. > > > > Interestingly, I found this issue does not appear on all CX-7 NICs. E= ven with identical DPDK code, firmware version (28.43.2566), and hardware= models from the > > same > > manufacturer, only > > specific NICs exhibit this inconsistency. > > I have confirmed that: > > * All NICs use the same driver and firmware version. > > * All NICs are of the same model (MCX75310AAS-NEA=5FAx). > > > > > > > > 1) Do the two =22NICs=22 ('port 0' and 'port 1' from below printout) = represent two > > different ports/P=46s of the same physical 'board'/'adapter card' in = fact=3F > > > > 2) If (1) is true, were the results obtained by running the applicati= on on both > > ports simultaneously (both managed by the DPDK at the same time)=3F > > > > (just to clarify, -- I'm confused by the fact that the NIC driver its= elf seems > > to invoke 'rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp=5Fregister' for each new= RxQ rather than call > > it once and then look-up and reuse the existing offsets for more port= s/queue ). > > > > Thank you. > > > > > > * The issue occurs only when LRO is enabled together with RX hardware= timestamping. > > * Disabling LRO eliminates the issue. > > I would appreciate any insight into how this behavior can occur on on= ly some ports despite same software and hardware setup. > > > > Below is my code snippet. > > > > =60=60=60c > > /*-------------------------------------------------------------------= ---------*/ > > static inline int > > is=5Ftimestamp=5Fenabled(struct rte=5Fmbuf *mbuf) > > =7B > > =C2=A0 =C2=A0static uint64=5Ft timestamp=5Frx=5Fdynflag =3D 0; > > =C2=A0 =C2=A0int timestamp=5Frx=5Fdynflag=5Foffset; > > > > =C2=A0 =C2=A0if (=21timestamp=5Frx=5Fdynflag) > > =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0timestamp=5Frx=5Fdynflag=5Foffset =3D > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rte=5Fmbuf=5Fdynflag=5Flooku= p(RTE=5FMBU=46=5FDYN=46LAG=5FRX=5FTIMESTAMP=5FNAME, NULL); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0if (timestamp=5Frx=5Fdynflag=5Foffset < 0)= > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return 0; > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=7D > > =C2=A0 =C2=A0 =C2=A0 =C2=A0timestamp=5Frx=5Fdynflag =3D RTE=5FBIT64(t= imestamp=5Frx=5Fdynflag=5Foffset); > > =C2=A0 =C2=A0=7D > > > > =C2=A0 =C2=A0return mbuf->ol=5Fflags & timestamp=5Frx=5Fdynflag; > > =7D > > /*-------------------------------------------------------------------= ---------*/ > > static inline rte=5Fmbuf=5Ftimestamp=5Ft * > > get=5Ftimestamp(struct rte=5Fmbuf *mbuf) > > =7B > > =C2=A0 =C2=A0static int timestamp=5Fdynfield=5Foffset =3D -1; > > > > =C2=A0 =C2=A0if (timestamp=5Fdynfield=5Foffset < 0) > > =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0timestamp=5Fdynfield=5Foffset =3D > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rte=5Fmbuf=5Fdynfield=5Flook= up(RTE=5FMBU=46=5FDYN=46IELD=5FTIMESTAMP=5FNAME, NULL); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0if (timestamp=5Fdynfield=5Foffset < 0) > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return 0; > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=7D > > =C2=A0 =C2=A0=7D > > > > =C2=A0 =C2=A0return RTE=5FMBU=46=5FDYN=46IELD(mbuf, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0timestamp=5Fdynfield=5Foffset, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rte=5Fmbuf=5Ftimestamp=5Ft *); > > =7D > > /*-------------------------------------------------------------------= ---------*/ > > static inline rte=5Fmbuf=5Ftimestamp=5Ft * > > get=5Frx=5Fhw=5Ftimestamp(struct rte=5Fmbuf *pkt) > > =7B > > =C2=A0 =C2=A0if (=21is=5Ftimestamp=5Fenabled(pkt)) > > =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0printf(=22rx=5Fhw=5Ftimestamp not enabled = in mbuf=21=5Cn=22); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0return NULL; > > =C2=A0 =C2=A0=7D > > > > =C2=A0 =C2=A0return get=5Ftimestamp(pkt); > > =7D > > =60=60=60 > > > > My DPDK application prints logs as below. > > > > =60=60=60c > > =C2=A0 =C2=A0/* parse HW timestamp */ > > =C2=A0 =C2=A0rte=5Fmbuf=5Ftimestamp=5Ft *rx=5Ftimestamp =3D get=5Frx=5F= hw=5Ftimestamp(pkt); > > =C2=A0 =C2=A0printf(=22=5Bport %d=5D RX HW timestamp: %=23016lx %s=5C= n=22, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pctx->port=5Fid, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 *rx=5Ftimestamp, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pkt->ol=5Fflags & PKT=5FRX=5FLRO =3F= =22(LROed)=22 : =22(not LROed)=22); > > =60=60=60 > > > > Below are observations from two CX-7 ports under identical conditions= . > > > > Normal NIC (port 0): > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d185b (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d1911 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d19c9 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d37ca (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d4cb3 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d4cb3 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd30e019 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd3280bb (not LROed) > > > > Erroneous NIC (port 1): > > Below is erroneous NIC's timestamp. > > =5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > =5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > =5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > =5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > =5Bport 1=5D RX HW timestamp: 0x000080691b7557 (not LROed) > > =5Bport 1=5D RX HW timestamp: 0x000080691e2311 (not LROed) > > =5Bport 1=5D RX HW timestamp: 0x00008069357553 (not LROed) > > =5Bport 1=5D RX HW timestamp: 0x0000806936e8c1 (not LROed) > > > > As shown above, non-LRO packets consistently have normal hardware tim= estamps on both NICs. However, on port 1, all LROed packets return a fixe= d, invalid timestamp > > (0x3e6eef91bc19f0fd), > > which is clearly inconsistent. > > I have also confirmed that other dynfields (rather than dynfield=5B1=5D= and dynfield=5B2=5D) are unused. > > > > > > Sincerely, > > Junghan Yoon > > On Jul 22, 2025, 5:31 PM +0900, Ivan Malov , wrote: > > Hello, > > > > On Tue, 22 Jul 2025, Yoon Junghan wrote: > > > > Hello, > > > > I'm currently using DPDK 20.11 with a ConnectX-7 NIC, and I'm trying = to retrieve RX hardware timestamps using =60rte=5Fmbuf=5Fdyn=5Frx=5Ftimes= tamp=5Fregister()=60. > > > > > > Does the application invoke 'rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp=5Fregi= ster' on its own=3F If > > yes, consider to replace this with invocations of APIs =5B1=5D (with = field name =5B2=5D) > > and =5B3=5D (with flag name =5B4=5D). =46or an example, please refer = to =5B5=5D and =5B6=5D. > > > > This is because, as per =5B7=5D, the driver in question might 'regist= er' the field > > and the flag on its own, in response to 'DEV=5FRX=5FO=46=46LOAD=5FTIM= ESTAMP' request, so, > > the user application should look up the field/flag, not 'register' it= afresh. > > > > If this does not help, then consider to clarify whether the timestamp= s are > > accurate (and whether the flag is seen in the mbufs) when LRO is not = enabled. > > > > =5B1=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.ht= ml=23a6adf9b352a83e7d521fd6aa04e305b1c > > =5B2=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.ht= ml=23a5159b2d34fa801d171ed0ccce451121b > > =5B3=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.ht= ml=23a89d835027034f76a27eb2afe7987ae35 > > =5B4=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.ht= ml=23a831d7066c7193788351797a65186848a > > =5B5=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b9659= 7469b7f6e6207/app/test-pmd/util.c=23L44 > > =5B6=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b9659= 7469b7f6e6207/app/test-pmd/util.c=23L60 > > =5B7=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b9659= 7469b7f6e6207/drivers/net/mlx5/mlx5=5Frxq.c=23L1743 > > > > Thank you. > > > > > > When LRO is enabled, I notice that LROed mbufs seem to share identica= l timestamp values, and the timestamps are unexpectedly large or inconsis= tent. This raises > > the question of whether > > LRO is interfering with the correctness of the RX HW timestamps. > > > > I=E2=80=99d appreciate any clarification on whether HW RX timestampin= g is reliable when LRO is enabled on this platform, or if LRO should be j= ust disabled for accurate > > per-packet timestamping. > > > > > > Sincerely, > > Junghan Yoon > > > > > > > > > > > > > > --6881c0cc_17e8da44_bfe4 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
I found the key difference: when TCP timestamps (R=46C 7323)= are enabled on the TCP sender, the RX HW timestamp of LROed pac= kets on the DPDK PMD on middlebox machine becomes inconsistent or invalid= . Is there a known limitation or erratum in the mlx5 driver = or CX7 firmware regarding this=3F

Sincerely,
Junghan Yoon
On Jul 24, 2025, 12:26 AM +0900, Iv= an Malov <ivan.malov=40arknetworks.am>, wrote:
On Thu, 24 Jul 2025, Yoon Junghan wrote:
I'm not sure I did well. All interface show the same result. &=23160;
current settings:
tx=5Ftype 0
rx=5Ffilter 0

But that should mean.. no timestamping=3F

1) May be also check 'sudo ethtool -T <ifname>'.
2) May be try to enable 'sudo hwstamp=5Fctl -i <ifname> -r 1'.

Thank you.


Sincerely,
Junghan Yoon
On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 11:19 +0900, Ivan Malov <iv= an.malov=40arknetworks.am>, wrote:
On Wed, 23 Jul 2025, Yoon Junghan wrote:

I isolated port 1 using -a option for EAL parameter and got the similar r= esult.
&=23160;
Note that port 1 becomes port 0 in this time.
=5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed)
=5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed)
=5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed)
=5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed)
=5Bport 0=5D RX HW timestamp: 0x00042819272fad (not LROed)
=5Bport 0=5D RX HW timestamp: 0x000428192e6e77 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x000428192e7f01 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x000428192e833d (not LROed)
&=23160;
=46YI, I have 4 CX-7 on the same machine. (eth0 =3D port 0, ... eth3 =3D = port 3 in DPDK)
pci=400000:16:00.0 &=23160;eth0 &=23160; &=23160; &=23160; &=23160; &=231= 60; &=23160; network &=23160; &=23160; &=23160; &=23160;MT2910 =46amily =5B= ConnectX-7=5D
pci=400000:40:00.0 &=23160;eth1 &=23160; &=23160; &=23160; &=23160; &=231= 60; &=23160; network &=23160; &=23160; &=23160; &=23160;MT2910 =46amily =5B= ConnectX-7=5D
pci=400000:6a:00.0 &=23160;eth2 &=23160; &=23160; &=23160; &=23160; &=231= 60; &=23160; network &=23160; &=23160; &=23160; &=23160;MT2910 =46amily =5B= ConnectX-7=5D
pci=400000:94:00.0 &=23160;eth3 &=23160; &=23160; &=23160; &=23160; &=231= 60; &=23160; network &=23160; &=23160; &=23160; &=23160;MT2910 =46amily =5B= ConnectX-7=5D
&=23160;
Among them, only the first CX-7 shows consistent timestamp regardless of = LRO.


Does 'sudo hwstamp=5Fctl -i <ifname>' show consistent results acros= s all the NICs=3F

Thank you.


Sincerely,
Junghan Yoon
On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 10:28 +0900, Ivan Malov <iv= an.malov=40arknetworks.am>, wrote:
On Wed, 23 Jul 2025, Yoon Junghan wrote:

Thank you for quick response.
&=23160;
1) They are different NICs. Not in the same board. Separate adapters in d= ifferent PCIe slots.
2) My DPDK app uses 4 separate ports; port 0, port 1, port 2, and port 3.= They are all on different boards. Thus, they are running at the same tim= e.


Excellent. I apologise for one more dumb question, but does isolating the= very
specific NIC (so that DPDK does not grab the other ones) that is known to= give
strange timestamps, result in the same/unexpected behaviour=3F Just to ma= ke sure.

Thank you.


Sincerely,
Junghan Yoon
On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 10:09 +0900, Ivan Malov <iv= an.malov=40arknetworks.am>, wrote:
&=23160;
Hello,

On Wed, 23 Jul 2025, Yoon Junghan wrote:

&=23160;
Hello,
As advised, I tested hardware timestamps with LRO enabled on our ConnectX= -7 NICs. However, the timestamps of LROed packets still show inconsistent= and abnormally
large
gaps from normal
packets.
&=23160;
Interestingly, I found this issue does not appear on all CX-7 NICs. Even = with identical DPDK code, firmware version (28.43.2566), and hardware mod= els from the
same
manufacturer, only
specific NICs exhibit this inconsistency.
I have confirmed that:
* All NICs use the same driver and firmware version.
* All NICs are of the same model (MCX75310AAS-NEA=5FAx).
&=23160;


1) Do the two =22NICs=22 ('port 0' and 'port 1' from below printout) repr= esent two
different ports/P=46s of the same physical 'board'/'adapter card' in fact= =3F

2) If (1) is true, were the results obtained by running the application o= n both
ports simultaneously (both managed by the DPDK at the same time)=3F
=
(just to clarify, -- I'm confused by the fact that the NIC driver itself = seems
to invoke 'rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp=5Fregister' for each new RxQ= rather than call
it once and then look-up and reuse the existing offsets for more ports/qu= eue ).

Thank you.

&=23160;
* The issue occurs only when LRO is enabled together with RX hardware tim= estamping.
* Disabling LRO eliminates the issue.
I would appreciate any insight into how this behavior can occur on only s= ome ports despite same software and hardware setup.
&=23160;
Below is my code snippet.
&=23160;
=60=60=60c
/*-----------------------------------------------------------------------= -----*/
static inline int
is=5Ftimestamp=5Fenabled(struct rte=5Fmbuf *mbuf)
=7B
&=23160; &=23160;static uint64=5Ft timestamp=5Frx=5Fdynflag =3D 0;
&=23160; &=23160;int timestamp=5Frx=5Fdynflag=5Foffset;
&=23160;
&=23160; &=23160;if (=21timestamp=5Frx=5Fdynflag)
&=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160;timestamp=5Frx=5Fdynflag=5Foffset =3D<= br /> &=23160; &=23160; &=23160; &=23160; &=23160; &=23160;rte=5Fmbuf=5Fdynflag= =5Flookup(RTE=5FMBU=46=5FDYN=46LAG=5FRX=5FTIMESTAMP=5FNAME, NULL);
&=23160; &=23160; &=23160; &=23160;if (timestamp=5Frx=5Fdynflag=5Foffset = < 0)
&=23160; &=23160; &=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160;return 0;
&=23160; &=23160; &=23160; &=23160;=7D
&=23160; &=23160; &=23160; &=23160;timestamp=5Frx=5Fdynflag =3D RTE=5FBIT= 64(timestamp=5Frx=5Fdynflag=5Foffset);
&=23160; &=23160;=7D
&=23160;
&=23160; &=23160;return mbuf->ol=5Fflags & timestamp=5Frx=5Fdynfla= g;
=7D
/*-----------------------------------------------------------------------= -----*/
static inline rte=5Fmbuf=5Ftimestamp=5Ft *
get=5Ftimestamp(struct rte=5Fmbuf *mbuf)
=7B
&=23160; &=23160;static int timestamp=5Fdynfield=5Foffset =3D -1;
&=23160;
&=23160; &=23160;if (timestamp=5Fdynfield=5Foffset < 0)
&=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160;timestamp=5Fdynfield=5Foffset =3D
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160;rte=5Fmbuf=5Fdynfiel= d=5Flookup(RTE=5FMBU=46=5FDYN=46IELD=5FTIMESTAMP=5FNAME, NULL);
&=23160; &=23160; &=23160; &=23160;if (timestamp=5Fdynfield=5Foffset <= 0)
&=23160; &=23160; &=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160;return 0;
&=23160; &=23160; &=23160; &=23160;=7D
&=23160; &=23160;=7D
&=23160;
&=23160; &=23160;return RTE=5FMBU=46=5FDYN=46IELD(mbuf,
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &= =23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160;timestamp=5F= dynfield=5Foffset,
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &= =23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160;rte=5Fmbuf=5F= timestamp=5Ft *);
=7D
/*-----------------------------------------------------------------------= -----*/
static inline rte=5Fmbuf=5Ftimestamp=5Ft *
get=5Frx=5Fhw=5Ftimestamp(struct rte=5Fmbuf *pkt)
=7B
&=23160; &=23160;if (=21is=5Ftimestamp=5Fenabled(pkt))
&=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160;printf(=22rx=5Fhw=5Ftimestamp not enab= led in mbuf=21=5Cn=22);
&=23160; &=23160; &=23160; &=23160;return NULL;
&=23160; &=23160;=7D
&=23160;
&=23160; &=23160;return get=5Ftimestamp(pkt);
=7D
=60=60=60
&=23160;
My DPDK application prints logs as below.
&=23160;
=60=60=60c
&=23160; &=23160;/* parse HW timestamp */
&=23160; &=23160;rte=5Fmbuf=5Ftimestamp=5Ft *rx=5Ftimestamp =3D get=5Frx=5F= hw=5Ftimestamp(pkt);
&=23160; &=23160;printf(=22=5Bport %d=5D RX HW timestamp: %=23016lx %s=5C= n=22,
&=23160; &=23160; &=23160; &=23160; &=23160; pctx->port=5Fid,
&=23160; &=23160; &=23160; &=23160; &=23160; *rx=5Ftimestamp,
&=23160; &=23160; &=23160; &=23160; &=23160; pkt->ol=5Fflags & PKT= =5FRX=5FLRO =3F =22(LROed)=22 : =22(not LROed)=22);
=60=60=60
&=23160;
Below are observations from two CX-7 ports under identical conditions. &=23160;
Normal NIC (port 0):
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d185b (LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d1911 (LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d19c9 (LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d37ca (LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd30e019 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd3280bb (not LROed)
&=23160;
Erroneous NIC (port 1):
Below is erroneous NIC's timestamp.
=5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
=5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
=5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
=5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
=5Bport 1=5D RX HW timestamp: 0x000080691b7557 (not LROed)
=5Bport 1=5D RX HW timestamp: 0x000080691e2311 (not LROed)
=5Bport 1=5D RX HW timestamp: 0x00008069357553 (not LROed)
=5Bport 1=5D RX HW timestamp: 0x0000806936e8c1 (not LROed)

As shown above, non-LRO packets consistently have normal hardware timesta= mps on both NICs. However, on port 1, all LROed packets return a fixed, i= nvalid timestamp
(0x3e6eef91bc19f0fd),
which is clearly inconsistent.
I have also confirmed that other dynfields (rather than dynfield=5B1=5D a= nd dynfield=5B2=5D) are unused.
&=23160;

Sincerely,
Junghan Yoon
On Jul 22, 2025, 5:31 PM +0900, Ivan Malov <ivan.malov=40arknetworks.a= m>, wrote:
Hello,

On Tue, 22 Jul 2025, Yoon Junghan wrote:

Hello,
&=23160;
I'm currently using DPDK 20.11 with a ConnectX-7 NIC, and I'm trying to r= etrieve RX hardware timestamps using =60rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp= =5Fregister()=60.


Does the application invoke 'rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp=5Fregister= ' on its own=3F If
yes, consider to replace this with invocations of APIs =5B1=5D (with fiel= d name =5B2=5D)
and =5B3=5D (with flag name =5B4=5D). =46or an example, please refer to =5B= 5=5D and =5B6=5D.

This is because, as per =5B7=5D, the driver in question might 'register' = the field
and the flag on its own, in response to 'DEV=5FRX=5FO=46=46LOAD=5FTIMESTA= MP' request, so,
the user application should look up the field/flag, not 'register' it afr= esh.

If this does not help, then consider to clarify whether the timestamps ar= e
accurate (and whether the flag is seen in the mbufs) when LRO is not enab= led.

=5B1=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.html=23= a6adf9b352a83e7d521fd6aa04e305b1c
=5B2=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.html=23= a5159b2d34fa801d171ed0ccce451121b
=5B3=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.html=23= a89d835027034f76a27eb2afe7987ae35
=5B4=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.html=23= a831d7066c7193788351797a65186848a
=5B5=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469= b7f6e6207/app/test-pmd/util.c=23L44
=5B6=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469= b7f6e6207/app/test-pmd/util.c=23L60
=5B7=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469= b7f6e6207/drivers/net/mlx5/mlx5=5Frxq.c=23L1743

Thank you.

&=23160;
When LRO is enabled, I notice that LROed mbufs seem to share identical ti= mestamp values, and the timestamps are unexpectedly large or inconsistent= . This raises
the question of whether
LRO is interfering with the correctness of the RX HW timestamps.
&=23160;
I=E2=80=99d appreciate any clarification on whether HW RX timestamping is= reliable when LRO is enabled on this platform, or if LRO should be just = disabled for accurate
per-packet timestamping.
&=23160;

Sincerely,
Junghan Yoon


&=23160;




--6881c0cc_17e8da44_bfe4--