From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B714046BFE for ; Thu, 24 Jul 2025 12:20:54 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3D51C40B9D; Thu, 24 Jul 2025 12:20:54 +0200 (CEST) Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by mails.dpdk.org (Postfix) with ESMTP id 7D41F40279 for ; Thu, 24 Jul 2025 12:20:52 +0200 (CEST) Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-748e63d4b05so544399b3a.2 for ; Thu, 24 Jul 2025 03:20:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1753352451; x=1753957251; darn=dpdk.org; h=mime-version:subject:references:in-reply-to:message-id:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=QxOD8TVtDzyTF/ErmKXNM4ty0gFilqYB5VYTSBxkNis=; b=kxS1jEUYY0jO9TVEWkNM0Acue8WyD6+Ioaf2d2VcRn3EML+hMFb6erVHfQdJ+fd2fb fhrL8VdFtxJ3We4brT3VXBKyrDduszDkNGGlaHrsiIvXbePYg/mQPsDwY5lwZH+eyhGM dxPJ+b70W+1PQQpHNcfL1mGW0zs9f6uqmBwaBMq30vai6attkAV9XAlIjGV1uKaOU1H0 baTZED8mLiePU8JiCP+D5Cwy8jeHvxV9FXchSi9+O56RC7tSX2SXLa8QoIj886CvqSOR cU7aGHEAYkB3sPmiNmypddGJ7heYANLO+q+GBWH/gBaXZUjncGD4kHtKaGdhf9aIW+2P c9fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1753352451; x=1753957251; h=mime-version:subject:references:in-reply-to:message-id:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QxOD8TVtDzyTF/ErmKXNM4ty0gFilqYB5VYTSBxkNis=; b=iAPMqQ/WIuc5Ix30ijh2FuDSVdBA/ImfTDnRwD5vIeDDEi9KRFZ22AxmV6pugZG3Uk EWVEyCeLGYefwp/QXAe4iopNMp5BPc880oP7AO7S/Ft55NmN0L6P52DUvtL9APk9dMUw QJMA8qmR4IC5J0Fsjwix7HlA89JKmKKZzkVa1h/Qy97M3W+ShNnKBwLdCNq09JJJlCX3 8h73lrRBAdwrNex8fp7Y9lyU38HiZ2h5u5BcsW4Q3wOhTvms1qKOEPmFy5z/rMPT/+eV jqSS9i0wjJVG3YWSOHkyjXUQLR119YRQXZW31aCBDKRsmRmGZeRvthMRDHdjgcwR8UaI K2IQ== X-Gm-Message-State: AOJu0YyNBtg54j6aoHbaVX2acg0BSgJR91WxJbq/ppjC4wGlulKGHN96 9JQIJK9PQn0sVWK/6Oe07SRwnbNX5bFZlc9DFQS+EW1dCKz4UuLwVPvUDyjW8AS7 X-Gm-Gg: ASbGncs+q9PyVvhvgAt69k75bPcwl+pnVIbf1arHtp49E4rGO2wCdIW1Irw2HczWu/U 8eJseu6TWR/me3ui8dI3fIDn9ZM91xRvorfk5ZG5dnNLwPx98WYhMYzvFROb2ghWXVxT8AjPs0+ pUrmim8pEz97o1KqD8qVGZfp7F9ynai0KJUHBLZ1KxVnLBtD/UtRTPObq1M1ZOLR1xaNJy1tV+i 40oB681zise6PYhQ+8XPbd1SBMv3PRolZx/Yr3al8xY5Y4UfjXkLI7PpzIyEx35cqsDJKPizWHp +BnUu4+79L3fLnCi/51KXgcKdd5kGHXBVmcCrFwSCRqSEALH1dn3lbaTZyEEIh8utj5CKlg7bPb BjXWISPgFUrfj+5DMXp6ctNGJtAMWSOb5uJPgMLSf6A== X-Google-Smtp-Source: AGHT+IGCGnMrBuEwD8K4V1tj00UGVXAHr+hdlEmzPan6/qKNXmsQMRkg4YOP4oVHcv7lPFo2bAQ0SA== X-Received: by 2002:a05:6a00:178a:b0:748:f1ba:9b0d with SMTP id d2e1a72fcca58-76034cececamr8994215b3a.11.1753352451270; Thu, 24 Jul 2025 03:20:51 -0700 (PDT) Received: from [147.46.248.138] ([147.46.248.138]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-761b061b14dsm1329529b3a.117.2025.07.24.03.20.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Jul 2025 03:20:50 -0700 (PDT) Date: Thu, 24 Jul 2025 19:20:11 +0900 From: Yoon Junghan To: Ivan Malov Cc: "=?utf-8?Q?users=40dpdk.org?=" Message-ID: <0e8116cd-0572-4453-82bf-c3b74f52b33f@Spark> In-Reply-To: References: <3f7df93b-ee00-bcdb-337c-8faa7bd73f6c@arknetworks.am> <391481b9-1b18-5567-a8a0-563c5c5eb37c@arknetworks.am> <8f3a156a-2f2a-4418-a5e0-47a4450cd10a@Spark> <66490314-cb05-0018-4457-9bfab077048a@arknetworks.am> <2c092b37-e8cd-4161-bebf-e5031b7c2990@Spark> <8cbcf9f0-5889-4b11-a98c-8eac24d8343a@Spark> <6888c20e-d18d-d4fc-fe5f-b4dbbb5fdb30@arknetworks.am> <72d04ff6-9018-421b-9cd6-4260dd7a3ee7@Spark> Subject: Re: HW RX timestamp with LRO enabled on ConnectX-7 (DPDK 20.11) X-Readdle-Message-ID: 0e8116cd-0572-4453-82bf-c3b74f52b33f@Spark MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="688208e0_fad28df3_bfe4" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --688208e0_fad28df3_bfe4 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hello, As advised, I checked the TCP timestamp and found that the RX HW timestam= p appears to be replaced by the TCP timestamp in LROed packets. This is a= very weird bug... =5Bport 0=5D RX HW timestamp: 0x00001208f2a8d6c7, TCP timestamp: 0x669efc= 3b6a269469 (tsval: 1721695291, tsecr: 1780913257) (not LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3b6a26946a, TCP timestamp: 0x669efc= 3b6a26946a (tsval: 1721695291, tsecr: 1780913258) (LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3b6a26946a, TCP timestamp: 0x669efc= 3b6a26946a (tsval: 1721695291, tsecr: 1780913258) (LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3b6a26946a, TCP timestamp: 0x669efc= 3b6a26946a (tsval: 1721695291, tsecr: 1780913258) (LROed) =5Bport 0=5D RX HW timestamp: 0x00001208f2b05907, TCP timestamp: 0x669efc= 3b6a26946a (tsval: 1721695291, tsecr: 1780913258) (not LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3c6a26946a, TCP timestamp: 0x669efc= 3c6a26946a (tsval: 1721695292, tsecr: 1780913258) (LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3c6a26946a, TCP timestamp: 0x669efc= 3c6a26946a (tsval: 1721695292, tsecr: 1780913258) (LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3c6a26946a, TCP timestamp: 0x669efc= 3c6a26946a (tsval: 1721695292, tsecr: 1780913258) (LROed) =5Bport 0=5D RX HW timestamp: 0x00001208f2b8f191, TCP timestamp: 0x669efc= 3c6a26946a (tsval: 1721695292, tsecr: 1780913258) (not LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3c6a26946b, TCP timestamp: 0x669efc= 3c6a26946b (tsval: 1721695292, tsecr: 1780913259) (LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3c6a26946b, TCP timestamp: 0x669efc= 3c6a26946b (tsval: 1721695292, tsecr: 1780913259) (LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3c6a26946b, TCP timestamp: 0x669efc= 3c6a26946b (tsval: 1721695292, tsecr: 1780913259) (LROed) =5Bport 0=5D RX HW timestamp: 0x00001208f2c0689b, TCP timestamp: 0x669efc= 3c6a26946b (tsval: 1721695292, tsecr: 1780913259) (not LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3d6a26946b, TCP timestamp: 0x669efc= 3d6a26946b (tsval: 1721695293, tsecr: 1780913259) (LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3d6a26946b, TCP timestamp: 0x669efc= 3d6a26946b (tsval: 1721695293, tsecr: 1780913259) (LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3d6a26946b, TCP timestamp: 0x669efc= 3d6a26946b (tsval: 1721695293, tsecr: 1780913259) (LROed) =5Bport 0=5D RX HW timestamp: 0x00001208f2c83627, TCP timestamp: 0x669efc= 3d6a26946b (tsval: 1721695293, tsecr: 1780913259) (not LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3d6a26946c, TCP timestamp: 0x669efc= 3d6a26946c (tsval: 1721695293, tsecr: 1780913260) (LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3d6a26946c, TCP timestamp: 0x669efc= 3d6a26946c (tsval: 1721695293, tsecr: 1780913260) (LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3d6a26946c, TCP timestamp: 0x669efc= 3d6a26946c (tsval: 1721695293, tsecr: 1780913260) (LROed) =5Bport 0=5D RX HW timestamp: 0x00001208f2cfa33b, TCP timestamp: 0x669efc= 3d6a26946c (tsval: 1721695293, tsecr: 1780913260) (not LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3e6a26946c, TCP timestamp: 0x669efc= 3e6a26946c (tsval: 1721695294, tsecr: 1780913260) (LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3e6a26946c, TCP timestamp: 0x669efc= 3e6a26946c (tsval: 1721695294, tsecr: 1780913260) (LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3e6a26946c, TCP timestamp: 0x669efc= 3e6a26946c (tsval: 1721695294, tsecr: 1780913260) (LROed) =5Bport 0=5D RX HW timestamp: 0x00001208f2d8605d, TCP timestamp: 0x669efc= 3e6a26946c (tsval: 1721695294, tsecr: 1780913260) (not LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3e6a26946d, TCP timestamp: 0x669efc= 3e6a26946d (tsval: 1721695294, tsecr: 1780913261) (LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3e6a26946d, TCP timestamp: 0x669efc= 3e6a26946d (tsval: 1721695294, tsecr: 1780913261) (LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3e6a26946d, TCP timestamp: 0x669efc= 3e6a26946d (tsval: 1721695294, tsecr: 1780913261) (LROed) =5Bport 0=5D RX HW timestamp: 0x00001208f2dfd977, TCP timestamp: 0x669efc= 3f6a26946d (tsval: 1721695295, tsecr: 1780913261) (not LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3f6a26946d, TCP timestamp: 0x669efc= 3f6a26946d (tsval: 1721695295, tsecr: 1780913261) (LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3f6a26946d, TCP timestamp: 0x669efc= 3f6a26946d (tsval: 1721695295, tsecr: 1780913261) (LROed) =5Bport 0=5D RX HW timestamp: 0x669efc3f6a26946d, TCP timestamp: 0x669efc= 3f6a26946d (tsval: 1721695295, tsecr: 1780913261) (LROed) =5Bport 0=5D RX HW timestamp: 0x00001208f2e78fab, TCP timestamp: 0x669efc= 3f6a26946d (tsval: 1721695295, tsecr: 1780913261) (not LROed) Sincerely, Junghan Yoon On Jul 24, 2025, 5:40 PM +0900, Ivan Malov ,= wrote: > On Thu, 24 Jul 2025, Yoon Junghan wrote: > > > I found the key difference: when TCP timestamps (R=46C 7323) are enab= led on the TCP sender, the RX HW timestamp of LROed packets on the DPDK P= MD on middlebox machine becomes inconsistent > > or invalid. Is there a known limitation or erratum in the mlx5 driver= or CX7 firmware regarding this=3F > > Very interesting observation. So did you have TCP timestamps enabled on= those > sender NICs that match the receiver NICs that would fail to do HW times= tamp > and did you have it disabled on links where HW timestamp was OK on rece= iver=3F > > But in the case of LRO on such NICs that have wrong HW timestamp, is th= e TCP > timestamp option present in a LROed packet and does it have accurate va= lue=3F > > Regarding erratum, - I have to confess I'm not an expert in this partic= ular > driver. May be they have some official documentation regarding this som= ewhere. > > Thank you. > > > > > Sincerely, > > Junghan Yoon > > On Jul 24, 2025, 12:26 AM +0900, Ivan Malov , wrote: > > On Thu, 24 Jul 2025, Yoon Junghan wrote: > > > > I'm not sure I did well. All interface show the same result. > > > > current settings: > > tx=5Ftype 0 > > rx=5Ffilter 0 > > > > > > But that should mean.. no timestamping=3F > > > > 1) May be also check 'sudo ethtool -T '. > > 2) May be try to enable 'sudo hwstamp=5Fctl -i -r 1'. > > > > Thank you. > > > > > > Sincerely, > > Junghan Yoon > > On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 11:19 +0900, Ivan Malov , wrote: > > On Wed, 23 Jul 2025, Yoon Junghan wrote: > > > > I isolated port 1 using -a option for EAL parameter and got the simil= ar result. > > > > Note that port 1 becomes port 0 in this time. > > =5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00042819272fad (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x000428192e6e77 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x000428192e7f01 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x000428192e833d (not LROed) > > > > =46YI, I have 4 CX-7 on the same machine. (eth0 =3D port 0, ... eth3 = =3D port 3 in DPDK) > > pci=400000:16:00.0 =C2=A0eth0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= network =C2=A0 =C2=A0 =C2=A0 =C2=A0MT2910 =46amily =5BConnectX-7=5D > > pci=400000:40:00.0 =C2=A0eth1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= network =C2=A0 =C2=A0 =C2=A0 =C2=A0MT2910 =46amily =5BConnectX-7=5D > > pci=400000:6a:00.0 =C2=A0eth2 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= network =C2=A0 =C2=A0 =C2=A0 =C2=A0MT2910 =46amily =5BConnectX-7=5D > > pci=400000:94:00.0 =C2=A0eth3 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= network =C2=A0 =C2=A0 =C2=A0 =C2=A0MT2910 =46amily =5BConnectX-7=5D > > > > Among them, only the first CX-7 shows consistent timestamp regardless= of LRO. > > > > > > Does 'sudo hwstamp=5Fctl -i ' show consistent results across = all the NICs=3F > > > > Thank you. > > > > > > Sincerely, > > Junghan Yoon > > On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 10:28 +0900, Ivan Malov , wrote: > > On Wed, 23 Jul 2025, Yoon Junghan wrote: > > > > Thank you for quick response. > > > > 1) They are different NICs. Not in the same board. Separate adapters = in different PCIe slots. > > 2) My DPDK app uses 4 separate ports; port 0, port 1, port 2, and por= t 3. They are all on different boards. Thus, they are running at the same= time. > > > > > > Excellent. I apologise for one more dumb question, but does isolating= the very > > specific NIC (so that DPDK does not grab the other ones) that is know= n to give > > strange timestamps, result in the same/unexpected behaviour=3F Just t= o make sure. > > > > Thank you. > > > > > > Sincerely, > > Junghan Yoon > > On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 10:09 +0900, Ivan Malov , wrote: > > > > Hello, > > > > On Wed, 23 Jul 2025, Yoon Junghan wrote: > > > > > > Hello, > > As advised, I tested hardware timestamps with LRO enabled on our Conn= ectX-7 NICs. However, the timestamps of LROed packets still show inconsis= tent and abnormally > > large > > gaps from normal > > packets. > > > > Interestingly, I found this issue does not appear on all CX-7 NICs. E= ven with identical DPDK code, firmware version (28.43.2566), and hardware= models from the > > same > > manufacturer, only > > specific NICs exhibit this inconsistency. > > I have confirmed that: > > * All NICs use the same driver and firmware version. > > * All NICs are of the same model (MCX75310AAS-NEA=5FAx). > > > > > > > > 1) Do the two =22NICs=22 ('port 0' and 'port 1' from below printout) = represent two > > different ports/P=46s of the same physical 'board'/'adapter card' in = fact=3F > > > > 2) If (1) is true, were the results obtained by running the applicati= on on both > > ports simultaneously (both managed by the DPDK at the same time)=3F > > > > (just to clarify, -- I'm confused by the fact that the NIC driver its= elf seems > > to invoke 'rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp=5Fregister' for each new= RxQ rather than call > > it once and then look-up and reuse the existing offsets for more port= s/queue ). > > > > Thank you. > > > > > > * The issue occurs only when LRO is enabled together with RX hardware= timestamping. > > * Disabling LRO eliminates the issue. > > I would appreciate any insight into how this behavior can occur on on= ly some ports despite same software and hardware setup. > > > > Below is my code snippet. > > > > =60=60=60c > > /*-------------------------------------------------------------------= ---------*/ > > static inline int > > is=5Ftimestamp=5Fenabled(struct rte=5Fmbuf *mbuf) > > =7B > > =C2=A0 =C2=A0static uint64=5Ft timestamp=5Frx=5Fdynflag =3D 0; > > =C2=A0 =C2=A0int timestamp=5Frx=5Fdynflag=5Foffset; > > > > =C2=A0 =C2=A0if (=21timestamp=5Frx=5Fdynflag) > > =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0timestamp=5Frx=5Fdynflag=5Foffset =3D > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rte=5Fmbuf=5Fdynflag=5Flooku= p(RTE=5FMBU=46=5FDYN=46LAG=5FRX=5FTIMESTAMP=5FNAME, NULL); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0if (timestamp=5Frx=5Fdynflag=5Foffset < 0)= > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return 0; > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=7D > > =C2=A0 =C2=A0 =C2=A0 =C2=A0timestamp=5Frx=5Fdynflag =3D RTE=5FBIT64(t= imestamp=5Frx=5Fdynflag=5Foffset); > > =C2=A0 =C2=A0=7D > > > > =C2=A0 =C2=A0return mbuf->ol=5Fflags & timestamp=5Frx=5Fdynflag; > > =7D > > /*-------------------------------------------------------------------= ---------*/ > > static inline rte=5Fmbuf=5Ftimestamp=5Ft * > > get=5Ftimestamp(struct rte=5Fmbuf *mbuf) > > =7B > > =C2=A0 =C2=A0static int timestamp=5Fdynfield=5Foffset =3D -1; > > > > =C2=A0 =C2=A0if (timestamp=5Fdynfield=5Foffset < 0) > > =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0timestamp=5Fdynfield=5Foffset =3D > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rte=5Fmbuf=5Fdynfield=5Flook= up(RTE=5FMBU=46=5FDYN=46IELD=5FTIMESTAMP=5FNAME, NULL); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0if (timestamp=5Fdynfield=5Foffset < 0) > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return 0; > > =C2=A0 =C2=A0 =C2=A0 =C2=A0=7D > > =C2=A0 =C2=A0=7D > > > > =C2=A0 =C2=A0return RTE=5FMBU=46=5FDYN=46IELD(mbuf, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0timestamp=5Fdynfield=5Foffset, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rte=5Fmbuf=5Ftimestamp=5Ft *); > > =7D > > /*-------------------------------------------------------------------= ---------*/ > > static inline rte=5Fmbuf=5Ftimestamp=5Ft * > > get=5Frx=5Fhw=5Ftimestamp(struct rte=5Fmbuf *pkt) > > =7B > > =C2=A0 =C2=A0if (=21is=5Ftimestamp=5Fenabled(pkt)) > > =C2=A0 =C2=A0=7B > > =C2=A0 =C2=A0 =C2=A0 =C2=A0printf(=22rx=5Fhw=5Ftimestamp not enabled = in mbuf=21=5Cn=22); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0return NULL; > > =C2=A0 =C2=A0=7D > > > > =C2=A0 =C2=A0return get=5Ftimestamp(pkt); > > =7D > > =60=60=60 > > > > My DPDK application prints logs as below. > > > > =60=60=60c > > =C2=A0 =C2=A0/* parse HW timestamp */ > > =C2=A0 =C2=A0rte=5Fmbuf=5Ftimestamp=5Ft *rx=5Ftimestamp =3D get=5Frx=5F= hw=5Ftimestamp(pkt); > > =C2=A0 =C2=A0printf(=22=5Bport %d=5D RX HW timestamp: %=23016lx %s=5C= n=22, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pctx->port=5Fid, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 *rx=5Ftimestamp, > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pkt->ol=5Fflags & PKT=5FRX=5FLRO =3F= =22(LROed)=22 : =22(not LROed)=22); > > =60=60=60 > > > > Below are observations from two CX-7 ports under identical conditions= . > > > > Normal NIC (port 0): > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d185b (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d1911 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d19c9 (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d37ca (LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d4cb3 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd2d4cb3 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd30e019 (not LROed) > > =5Bport 0=5D RX HW timestamp: 0x00007dcd3280bb (not LROed) > > > > Erroneous NIC (port 1): > > Below is erroneous NIC's timestamp. > > =5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > =5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > =5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > =5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > =5Bport 1=5D RX HW timestamp: 0x000080691b7557 (not LROed) > > =5Bport 1=5D RX HW timestamp: 0x000080691e2311 (not LROed) > > =5Bport 1=5D RX HW timestamp: 0x00008069357553 (not LROed) > > =5Bport 1=5D RX HW timestamp: 0x0000806936e8c1 (not LROed) > > > > As shown above, non-LRO packets consistently have normal hardware tim= estamps on both NICs. However, on port 1, all LROed packets return a fixe= d, invalid timestamp > > (0x3e6eef91bc19f0fd), > > which is clearly inconsistent. > > I have also confirmed that other dynfields (rather than dynfield=5B1=5D= and dynfield=5B2=5D) are unused. > > > > > > Sincerely, > > Junghan Yoon > > On Jul 22, 2025, 5:31 PM +0900, Ivan Malov , wrote: > > Hello, > > > > On Tue, 22 Jul 2025, Yoon Junghan wrote: > > > > Hello, > > > > I'm currently using DPDK 20.11 with a ConnectX-7 NIC, and I'm trying = to retrieve RX hardware timestamps using =60rte=5Fmbuf=5Fdyn=5Frx=5Ftimes= tamp=5Fregister()=60. > > > > > > Does the application invoke 'rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp=5Fregi= ster' on its own=3F If > > yes, consider to replace this with invocations of APIs =5B1=5D (with = field name =5B2=5D) > > and =5B3=5D (with flag name =5B4=5D). =46or an example, please refer = to =5B5=5D and =5B6=5D. > > > > This is because, as per =5B7=5D, the driver in question might 'regist= er' the field > > and the flag on its own, in response to 'DEV=5FRX=5FO=46=46LOAD=5FTIM= ESTAMP' request, so, > > the user application should look up the field/flag, not 'register' it= afresh. > > > > If this does not help, then consider to clarify whether the timestamp= s are > > accurate (and whether the flag is seen in the mbufs) when LRO is not = enabled. > > > > =5B1=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.ht= ml=23a6adf9b352a83e7d521fd6aa04e305b1c > > =5B2=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.ht= ml=23a5159b2d34fa801d171ed0ccce451121b > > =5B3=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.ht= ml=23a89d835027034f76a27eb2afe7987ae35 > > =5B4=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.ht= ml=23a831d7066c7193788351797a65186848a > > =5B5=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b9659= 7469b7f6e6207/app/test-pmd/util.c=23L44 > > =5B6=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b9659= 7469b7f6e6207/app/test-pmd/util.c=23L60 > > =5B7=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b9659= 7469b7f6e6207/drivers/net/mlx5/mlx5=5Frxq.c=23L1743 > > > > Thank you. > > > > > > When LRO is enabled, I notice that LROed mbufs seem to share identica= l timestamp values, and the timestamps are unexpectedly large or inconsis= tent. This raises > > the question of whether > > LRO is interfering with the correctness of the RX HW timestamps. > > > > I=E2=80=99d appreciate any clarification on whether HW RX timestampin= g is reliable when LRO is enabled on this platform, or if LRO should be j= ust disabled for accurate > > per-packet timestamping. > > > > > > Sincerely, > > Junghan Yoon > > > > > > > > > > > > > > > > --688208e0_fad28df3_bfe4 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
Hello,
As advised, I checked the TCP timestamp and found that the RX HW tim= estamp appears to be replaced by the TCP timestamp in LROed packets. This= is a very weird bug...
&=23160;
=5Bport 0=5D RX HW timestamp: 0x00001208f2a8d6c7, TCP timestamp: 0x6= 69efc3b6a269469 (tsval: 1721695291, tsecr: 1780913257) (not LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3b6a26946a, TCP timestamp: 0x669efc= 3b6a26946a (tsval: 1721695291, tsecr: 1780913258) (LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3b6a26946a, TCP timestamp: 0x669efc= 3b6a26946a (tsval: 1721695291, tsecr: 1780913258) (LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3b6a26946a, TCP timestamp: 0x669efc= 3b6a26946a (tsval: 1721695291, tsecr: 1780913258) (LROed)
=5Bport 0=5D RX HW timestamp: 0x00001208f2b05907, TCP timestamp: 0x669efc= 3b6a26946a (tsval: 1721695291, tsecr: 1780913258) (not LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3c6a26946a, TCP timestamp: 0x669efc= 3c6a26946a (tsval: 1721695292, tsecr: 1780913258) (LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3c6a26946a, TCP timestamp: 0x669efc= 3c6a26946a (tsval: 1721695292, tsecr: 1780913258) (LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3c6a26946a, TCP timestamp: 0x669efc= 3c6a26946a (tsval: 1721695292, tsecr: 1780913258) (LROed)
=5Bport 0=5D RX HW timestamp: 0x00001208f2b8f191, TCP timestamp: 0x669efc= 3c6a26946a (tsval: 1721695292, tsecr: 1780913258) (not LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3c6a26946b, TCP timestamp: 0x669efc= 3c6a26946b (tsval: 1721695292, tsecr: 1780913259) (LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3c6a26946b, TCP timestamp: 0x669efc= 3c6a26946b (tsval: 1721695292, tsecr: 1780913259) (LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3c6a26946b, TCP timestamp: 0x669efc= 3c6a26946b (tsval: 1721695292, tsecr: 1780913259) (LROed)
=5Bport 0=5D RX HW timestamp: 0x00001208f2c0689b, TCP timestamp: 0x669efc= 3c6a26946b (tsval: 1721695292, tsecr: 1780913259) (not LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3d6a26946b, TCP timestamp: 0x669efc= 3d6a26946b (tsval: 1721695293, tsecr: 1780913259) (LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3d6a26946b, TCP timestamp: 0x669efc= 3d6a26946b (tsval: 1721695293, tsecr: 1780913259) (LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3d6a26946b, TCP timestamp: 0x669efc= 3d6a26946b (tsval: 1721695293, tsecr: 1780913259) (LROed)
=5Bport 0=5D RX HW timestamp: 0x00001208f2c83627, TCP timestamp: 0x669efc= 3d6a26946b (tsval: 1721695293, tsecr: 1780913259) (not LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3d6a26946c, TCP timestamp: 0x669efc= 3d6a26946c (tsval: 1721695293, tsecr: 1780913260) (LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3d6a26946c, TCP timestamp: 0x669efc= 3d6a26946c (tsval: 1721695293, tsecr: 1780913260) (LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3d6a26946c, TCP timestamp: 0x669efc= 3d6a26946c (tsval: 1721695293, tsecr: 1780913260) (LROed)
=5Bport 0=5D RX HW timestamp: 0x00001208f2cfa33b, TCP timestamp: 0x669efc= 3d6a26946c (tsval: 1721695293, tsecr: 1780913260) (not LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3e6a26946c, TCP timestamp: 0x669efc= 3e6a26946c (tsval: 1721695294, tsecr: 1780913260) (LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3e6a26946c, TCP timestamp: 0x669efc= 3e6a26946c (tsval: 1721695294, tsecr: 1780913260) (LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3e6a26946c, TCP timestamp: 0x669efc= 3e6a26946c (tsval: 1721695294, tsecr: 1780913260) (LROed)
=5Bport 0=5D RX HW timestamp: 0x00001208f2d8605d, TCP timestamp: 0x669efc= 3e6a26946c (tsval: 1721695294, tsecr: 1780913260) (not LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3e6a26946d, TCP timestamp: 0x669efc= 3e6a26946d (tsval: 1721695294, tsecr: 1780913261) (LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3e6a26946d, TCP timestamp: 0x669efc= 3e6a26946d (tsval: 1721695294, tsecr: 1780913261) (LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3e6a26946d, TCP timestamp: 0x669efc= 3e6a26946d (tsval: 1721695294, tsecr: 1780913261) (LROed)
=5Bport 0=5D RX HW timestamp: 0x00001208f2dfd977, TCP timestamp: 0x669efc= 3f6a26946d (tsval: 1721695295, tsecr: 1780913261) (not LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3f6a26946d, TCP timestamp: 0x669efc= 3f6a26946d (tsval: 1721695295, tsecr: 1780913261) (LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3f6a26946d, TCP timestamp: 0x669efc= 3f6a26946d (tsval: 1721695295, tsecr: 1780913261) (LROed)
=5Bport 0=5D RX HW timestamp: 0x669efc3f6a26946d, TCP timestamp: 0x669efc= 3f6a26946d (tsval: 1721695295, tsecr: 1780913261) (LROed)
=5Bport 0=5D RX HW timestamp: 0x00001208f2e78fab, TCP timestamp: 0x669efc= 3f6a26946d (tsval: 1721695295, tsecr: 1780913261) (not LROed)

Sincerely,
Junghan Yoon
On Jul 24, 2025, 5:40 PM +0900, Iva= n Malov <ivan.malov=40arknetworks.am>, wrote:
On Thu, 24 Jul 2025, Yoon Junghan wrote:
I found the key difference: when TCP timestamps (R=46C 7323) ar= e enabled on the TCP sender, the RX HW timestamp of LROed packets on the = DPDK PMD on middlebox machine becomes inconsistent
or invalid. Is there a known limitation or erratum in the mlx5 driver or = CX7 firmware regarding this=3F

Very interesting observation. So did you have TCP timestamps enabled on t= hose
sender NICs that match the receiver NICs that would fail to do HW timesta= mp
and did you have it disabled on links where HW timestamp was OK on receiv= er=3F

But in the case of LRO on such NICs that have wrong HW timestamp, is the = TCP
timestamp option present in a LROed packet and does it have accurate valu= e=3F

Regarding erratum, - I have to confess I'm not an expert in this particul= ar
driver. May be they have some official documentation regarding this somew= here.

Thank you.


Sincerely,
Junghan Yoon
On Jul 24, 2025, 12:26 AM +0900, Ivan Malov <ivan.malov=40arknetworks.= am>, wrote:
On Thu, 24 Jul 2025, Yoon Junghan wrote:

I'm not sure I did well. All interface show the same result.
&=23160;
current settings:
tx=5Ftype 0
rx=5Ffilter 0


But that should mean.. no timestamping=3F

1) May be also check 'sudo ethtool -T <ifname>'.
2) May be try to enable 'sudo hwstamp=5Fctl -i <ifname> -r 1'.

Thank you.


Sincerely,
Junghan Yoon
On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 11:19 +0900, Ivan Malov <iv= an.malov=40arknetworks.am>, wrote:
On Wed, 23 Jul 2025, Yoon Junghan wrote:

I isolated port 1 using -a option for EAL parameter and got the similar r= esult.
&=23160;
Note that port 1 becomes port 0 in this time.
=5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed)
=5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed)
=5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed)
=5Bport 0=5D RX HW timestamp: 0x3eac4214bc574368 (LROed)
=5Bport 0=5D RX HW timestamp: 0x00042819272fad (not LROed)
=5Bport 0=5D RX HW timestamp: 0x000428192e6e77 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x000428192e7f01 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x000428192e833d (not LROed)
&=23160;
=46YI, I have 4 CX-7 on the same machine. (eth0 =3D port 0, ... eth3 =3D = port 3 in DPDK)
pci=400000:16:00.0 &=23160;eth0 &=23160; &=23160; &=23160; &=23160; &=231= 60; &=23160; network &=23160; &=23160; &=23160; &=23160;MT2910 =46amily =5B= ConnectX-7=5D
pci=400000:40:00.0 &=23160;eth1 &=23160; &=23160; &=23160; &=23160; &=231= 60; &=23160; network &=23160; &=23160; &=23160; &=23160;MT2910 =46amily =5B= ConnectX-7=5D
pci=400000:6a:00.0 &=23160;eth2 &=23160; &=23160; &=23160; &=23160; &=231= 60; &=23160; network &=23160; &=23160; &=23160; &=23160;MT2910 =46amily =5B= ConnectX-7=5D
pci=400000:94:00.0 &=23160;eth3 &=23160; &=23160; &=23160; &=23160; &=231= 60; &=23160; network &=23160; &=23160; &=23160; &=23160;MT2910 =46amily =5B= ConnectX-7=5D
&=23160;
Among them, only the first CX-7 shows consistent timestamp regardless of = LRO.


Does 'sudo hwstamp=5Fctl -i <ifname>' show consistent results acros= s all the NICs=3F

Thank you.


Sincerely,
Junghan Yoon
On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 10:28 +0900, Ivan Malov <iv= an.malov=40arknetworks.am>, wrote:
On Wed, 23 Jul 2025, Yoon Junghan wrote:

Thank you for quick response.
&=23160;
1) They are different NICs. Not in the same board. Separate adapters in d= ifferent PCIe slots.
2) My DPDK app uses 4 separate ports; port 0, port 1, port 2, and port 3.= They are all on different boards. Thus, they are running at the same tim= e.


Excellent. I apologise for one more dumb question, but does isolating the= very
specific NIC (so that DPDK does not grab the other ones) that is known to= give
strange timestamps, result in the same/unexpected behaviour=3F Just to ma= ke sure.

Thank you.


Sincerely,
Junghan Yoon
On 2025=EB=85=84 7=EC=9B=94 23=EC=9D=BC PM 10:09 +0900, Ivan Malov <iv= an.malov=40arknetworks.am>, wrote:
&=23160;
Hello,

On Wed, 23 Jul 2025, Yoon Junghan wrote:

&=23160;
Hello,
As advised, I tested hardware timestamps with LRO enabled on our ConnectX= -7 NICs. However, the timestamps of LROed packets still show inconsistent= and abnormally
large
gaps from normal
packets.
&=23160;
Interestingly, I found this issue does not appear on all CX-7 NICs. Even = with identical DPDK code, firmware version (28.43.2566), and hardware mod= els from the
same
manufacturer, only
specific NICs exhibit this inconsistency.
I have confirmed that:
* All NICs use the same driver and firmware version.
* All NICs are of the same model (MCX75310AAS-NEA=5FAx).
&=23160;


1) Do the two =22NICs=22 ('port 0' and 'port 1' from below printout) repr= esent two
different ports/P=46s of the same physical 'board'/'adapter card' in fact= =3F

2) If (1) is true, were the results obtained by running the application o= n both
ports simultaneously (both managed by the DPDK at the same time)=3F
=
(just to clarify, -- I'm confused by the fact that the NIC driver itself = seems
to invoke 'rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp=5Fregister' for each new RxQ= rather than call
it once and then look-up and reuse the existing offsets for more ports/qu= eue ).

Thank you.

&=23160;
* The issue occurs only when LRO is enabled together with RX hardware tim= estamping.
* Disabling LRO eliminates the issue.
I would appreciate any insight into how this behavior can occur on only s= ome ports despite same software and hardware setup.
&=23160;
Below is my code snippet.
&=23160;
=60=60=60c
/*-----------------------------------------------------------------------= -----*/
static inline int
is=5Ftimestamp=5Fenabled(struct rte=5Fmbuf *mbuf)
=7B
&=23160; &=23160;static uint64=5Ft timestamp=5Frx=5Fdynflag =3D 0;
&=23160; &=23160;int timestamp=5Frx=5Fdynflag=5Foffset;
&=23160;
&=23160; &=23160;if (=21timestamp=5Frx=5Fdynflag)
&=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160;timestamp=5Frx=5Fdynflag=5Foffset =3D<= br /> &=23160; &=23160; &=23160; &=23160; &=23160; &=23160;rte=5Fmbuf=5Fdynflag= =5Flookup(RTE=5FMBU=46=5FDYN=46LAG=5FRX=5FTIMESTAMP=5FNAME, NULL);
&=23160; &=23160; &=23160; &=23160;if (timestamp=5Frx=5Fdynflag=5Foffset = < 0)
&=23160; &=23160; &=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160;return 0;
&=23160; &=23160; &=23160; &=23160;=7D
&=23160; &=23160; &=23160; &=23160;timestamp=5Frx=5Fdynflag =3D RTE=5FBIT= 64(timestamp=5Frx=5Fdynflag=5Foffset);
&=23160; &=23160;=7D
&=23160;
&=23160; &=23160;return mbuf->ol=5Fflags & timestamp=5Frx=5Fdynfla= g;
=7D
/*-----------------------------------------------------------------------= -----*/
static inline rte=5Fmbuf=5Ftimestamp=5Ft *
get=5Ftimestamp(struct rte=5Fmbuf *mbuf)
=7B
&=23160; &=23160;static int timestamp=5Fdynfield=5Foffset =3D -1;
&=23160;
&=23160; &=23160;if (timestamp=5Fdynfield=5Foffset < 0)
&=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160;timestamp=5Fdynfield=5Foffset =3D
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160;rte=5Fmbuf=5Fdynfiel= d=5Flookup(RTE=5FMBU=46=5FDYN=46IELD=5FTIMESTAMP=5FNAME, NULL);
&=23160; &=23160; &=23160; &=23160;if (timestamp=5Fdynfield=5Foffset <= 0)
&=23160; &=23160; &=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160;return 0;
&=23160; &=23160; &=23160; &=23160;=7D
&=23160; &=23160;=7D
&=23160;
&=23160; &=23160;return RTE=5FMBU=46=5FDYN=46IELD(mbuf,
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &= =23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160;timestamp=5F= dynfield=5Foffset,
&=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160; &= =23160; &=23160; &=23160; &=23160; &=23160; &=23160; &=23160;rte=5Fmbuf=5F= timestamp=5Ft *);
=7D
/*-----------------------------------------------------------------------= -----*/
static inline rte=5Fmbuf=5Ftimestamp=5Ft *
get=5Frx=5Fhw=5Ftimestamp(struct rte=5Fmbuf *pkt)
=7B
&=23160; &=23160;if (=21is=5Ftimestamp=5Fenabled(pkt))
&=23160; &=23160;=7B
&=23160; &=23160; &=23160; &=23160;printf(=22rx=5Fhw=5Ftimestamp not enab= led in mbuf=21=5Cn=22);
&=23160; &=23160; &=23160; &=23160;return NULL;
&=23160; &=23160;=7D
&=23160;
&=23160; &=23160;return get=5Ftimestamp(pkt);
=7D
=60=60=60
&=23160;
My DPDK application prints logs as below.
&=23160;
=60=60=60c
&=23160; &=23160;/* parse HW timestamp */
&=23160; &=23160;rte=5Fmbuf=5Ftimestamp=5Ft *rx=5Ftimestamp =3D get=5Frx=5F= hw=5Ftimestamp(pkt);
&=23160; &=23160;printf(=22=5Bport %d=5D RX HW timestamp: %=23016lx %s=5C= n=22,
&=23160; &=23160; &=23160; &=23160; &=23160; pctx->port=5Fid,
&=23160; &=23160; &=23160; &=23160; &=23160; *rx=5Ftimestamp,
&=23160; &=23160; &=23160; &=23160; &=23160; pkt->ol=5Fflags & PKT= =5FRX=5FLRO =3F =22(LROed)=22 : =22(not LROed)=22);
=60=60=60
&=23160;
Below are observations from two CX-7 ports under identical conditions. &=23160;
Normal NIC (port 0):
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d185b (LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d1911 (LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d19c9 (LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d37ca (LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd30e019 (not LROed)
=5Bport 0=5D RX HW timestamp: 0x00007dcd3280bb (not LROed)
&=23160;
Erroneous NIC (port 1):
Below is erroneous NIC's timestamp.
=5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
=5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
=5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
=5Bport 1=5D RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
=5Bport 1=5D RX HW timestamp: 0x000080691b7557 (not LROed)
=5Bport 1=5D RX HW timestamp: 0x000080691e2311 (not LROed)
=5Bport 1=5D RX HW timestamp: 0x00008069357553 (not LROed)
=5Bport 1=5D RX HW timestamp: 0x0000806936e8c1 (not LROed)

As shown above, non-LRO packets consistently have normal hardware timesta= mps on both NICs. However, on port 1, all LROed packets return a fixed, i= nvalid timestamp
(0x3e6eef91bc19f0fd),
which is clearly inconsistent.
I have also confirmed that other dynfields (rather than dynfield=5B1=5D a= nd dynfield=5B2=5D) are unused.
&=23160;

Sincerely,
Junghan Yoon
On Jul 22, 2025, 5:31 PM +0900, Ivan Malov <ivan.malov=40arknetworks.a= m>, wrote:
Hello,

On Tue, 22 Jul 2025, Yoon Junghan wrote:

Hello,
&=23160;
I'm currently using DPDK 20.11 with a ConnectX-7 NIC, and I'm trying to r= etrieve RX hardware timestamps using =60rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp= =5Fregister()=60.


Does the application invoke 'rte=5Fmbuf=5Fdyn=5Frx=5Ftimestamp=5Fregister= ' on its own=3F If
yes, consider to replace this with invocations of APIs =5B1=5D (with fiel= d name =5B2=5D)
and =5B3=5D (with flag name =5B4=5D). =46or an example, please refer to =5B= 5=5D and =5B6=5D.

This is because, as per =5B7=5D, the driver in question might 'register' = the field
and the flag on its own, in response to 'DEV=5FRX=5FO=46=46LOAD=5FTIMESTA= MP' request, so,
the user application should look up the field/flag, not 'register' it afr= esh.

If this does not help, then consider to clarify whether the timestamps ar= e
accurate (and whether the flag is seen in the mbufs) when LRO is not enab= led.

=5B1=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.html=23= a6adf9b352a83e7d521fd6aa04e305b1c
=5B2=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.html=23= a5159b2d34fa801d171ed0ccce451121b
=5B3=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.html=23= a89d835027034f76a27eb2afe7987ae35
=5B4=5D https://doc.dpdk.org/api-20.11/rte=5F=5Fmbuf=5F=5Fdyn=5F8h.html=23= a831d7066c7193788351797a65186848a
=5B5=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469= b7f6e6207/app/test-pmd/util.c=23L44
=5B6=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469= b7f6e6207/app/test-pmd/util.c=23L60
=5B7=5D https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469= b7f6e6207/drivers/net/mlx5/mlx5=5Frxq.c=23L1743

Thank you.

&=23160;
When LRO is enabled, I notice that LROed mbufs seem to share identical ti= mestamp values, and the timestamps are unexpectedly large or inconsistent= . This raises
the question of whether
LRO is interfering with the correctness of the RX HW timestamps.
&=23160;
I=E2=80=99d appreciate any clarification on whether HW RX timestamping is= reliable when LRO is enabled on this platform, or if LRO should be just = disabled for accurate
per-packet timestamping.
&=23160;

Sincerely,
Junghan Yoon


&=23160;





--688208e0_fad28df3_bfe4--