I isolated port 1 using -a option for EAL parameter and got the similar result. Note that port 1 becomes port 0 in this time. [port 0] RX HW timestamp: 0x3eac4214bc574368 (LROed) [port 0] RX HW timestamp: 0x3eac4214bc574368 (LROed) [port 0] RX HW timestamp: 0x3eac4214bc574368 (LROed) [port 0] RX HW timestamp: 0x3eac4214bc574368 (LROed) [port 0] RX HW timestamp: 0x00042819272fad (not LROed) [port 0] RX HW timestamp: 0x000428192e6e77 (not LROed) [port 0] RX HW timestamp: 0x000428192e7f01 (not LROed) [port 0] RX HW timestamp: 0x000428192e833d (not LROed) FYI, I have 4 CX-7 on the same machine. (eth0 = port 0, ... eth3 = port 3 in DPDK) pci@0000:16:00.0  eth0             network        MT2910 Family [ConnectX-7] pci@0000:40:00.0  eth1             network        MT2910 Family [ConnectX-7] pci@0000:6a:00.0  eth2             network        MT2910 Family [ConnectX-7] pci@0000:94:00.0  eth3             network        MT2910 Family [ConnectX-7] Among them, only the first CX-7 shows consistent timestamp regardless of LRO. Sincerely, Junghan Yoon On 2025년 7월 23일 PM 10:28 +0900, Ivan Malov , wrote: > On Wed, 23 Jul 2025, Yoon Junghan wrote: > > > Thank you for quick response. > > > > 1) They are different NICs. Not in the same board. Separate adapters in different PCIe slots. > > 2) My DPDK app uses 4 separate ports; port 0, port 1, port 2, and port 3. They are all on different boards. Thus, they are running at the same time. > > Excellent. I apologise for one more dumb question, but does isolating the very > specific NIC (so that DPDK does not grab the other ones) that is known to give > strange timestamps, result in the same/unexpected behaviour? Just to make sure. > > Thank you. > > > > > Sincerely, > > Junghan Yoon > > On 2025년 7월 23일 PM 10:09 +0900, Ivan Malov , wrote: > > > > Hello, > > > > On Wed, 23 Jul 2025, Yoon Junghan wrote: > > > > > > Hello, > > As advised, I tested hardware timestamps with LRO enabled on our ConnectX-7 NICs. However, the timestamps of LROed packets still show inconsistent and abnormally large > > gaps from normal > > packets. > > > > Interestingly, I found this issue does not appear on all CX-7 NICs. Even with identical DPDK code, firmware version (28.43.2566), and hardware models from the same > > manufacturer, only > > specific NICs exhibit this inconsistency. > > I have confirmed that: > > * All NICs use the same driver and firmware version. > > * All NICs are of the same model (MCX75310AAS-NEA_Ax). > > > > > > > > 1) Do the two "NICs" ('port 0' and 'port 1' from below printout) represent two > > different ports/PFs of the same physical 'board'/'adapter card' in fact? > > > > 2) If (1) is true, were the results obtained by running the application on both > > ports simultaneously (both managed by the DPDK at the same time)? > > > > (just to clarify, -- I'm confused by the fact that the NIC driver itself seems > > to invoke 'rte_mbuf_dyn_rx_timestamp_register' for each new RxQ rather than call > > it once and then look-up and reuse the existing offsets for more ports/queue ). > > > > Thank you. > > > > > > * The issue occurs only when LRO is enabled together with RX hardware timestamping. > > * Disabling LRO eliminates the issue. > > I would appreciate any insight into how this behavior can occur on only some ports despite same software and hardware setup. > > > > Below is my code snippet. > > > > ```c > > /*----------------------------------------------------------------------------*/ > > static inline int > > is_timestamp_enabled(struct rte_mbuf *mbuf) > > { > >    static uint64_t timestamp_rx_dynflag = 0; > >    int timestamp_rx_dynflag_offset; > > > >    if (!timestamp_rx_dynflag) > >    { > >        timestamp_rx_dynflag_offset = > >            rte_mbuf_dynflag_lookup(RTE_MBUF_DYNFLAG_RX_TIMESTAMP_NAME, NULL); > >        if (timestamp_rx_dynflag_offset < 0) > >        { > >            return 0; > >        } > >        timestamp_rx_dynflag = RTE_BIT64(timestamp_rx_dynflag_offset); > >    } > > > >    return mbuf->ol_flags & timestamp_rx_dynflag; > > } > > /*----------------------------------------------------------------------------*/ > > static inline rte_mbuf_timestamp_t * > > get_timestamp(struct rte_mbuf *mbuf) > > { > >    static int timestamp_dynfield_offset = -1; > > > >    if (timestamp_dynfield_offset < 0) > >    { > >        timestamp_dynfield_offset = > >            rte_mbuf_dynfield_lookup(RTE_MBUF_DYNFIELD_TIMESTAMP_NAME, NULL); > >        if (timestamp_dynfield_offset < 0) > >        { > >            return 0; > >        } > >    } > > > >    return RTE_MBUF_DYNFIELD(mbuf, > >                              timestamp_dynfield_offset, > >                              rte_mbuf_timestamp_t *); > > } > > /*----------------------------------------------------------------------------*/ > > static inline rte_mbuf_timestamp_t * > > get_rx_hw_timestamp(struct rte_mbuf *pkt) > > { > >    if (!is_timestamp_enabled(pkt)) > >    { > >        printf("rx_hw_timestamp not enabled in mbuf!\n"); > >        return NULL; > >    } > > > >    return get_timestamp(pkt); > > } > > ``` > > > > My DPDK application prints logs as below. > > > > ```c > >    /* parse HW timestamp */ > >    rte_mbuf_timestamp_t *rx_timestamp = get_rx_hw_timestamp(pkt); > >    printf("[port %d] RX HW timestamp: %#016lx %s\n", > >           pctx->port_id, > >           *rx_timestamp, > >           pkt->ol_flags & PKT_RX_LRO ? "(LROed)" : "(not LROed)"); > > ``` > > > > Below are observations from two CX-7 ports under identical conditions. > > > > Normal NIC (port 0): > > [port 0] RX HW timestamp: 0x00007dcd2d185b (LROed) > > [port 0] RX HW timestamp: 0x00007dcd2d1911 (LROed) > > [port 0] RX HW timestamp: 0x00007dcd2d19c9 (LROed) > > [port 0] RX HW timestamp: 0x00007dcd2d37ca (LROed) > > [port 0] RX HW timestamp: 0x00007dcd2d4cb3 (not LROed) > > [port 0] RX HW timestamp: 0x00007dcd2d4cb3 (not LROed) > > [port 0] RX HW timestamp: 0x00007dcd30e019 (not LROed) > > [port 0] RX HW timestamp: 0x00007dcd3280bb (not LROed) > > > > Erroneous NIC (port 1): > > Below is erroneous NIC's timestamp. > > [port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > [port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > [port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > [port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > > [port 1] RX HW timestamp: 0x000080691b7557 (not LROed) > > [port 1] RX HW timestamp: 0x000080691e2311 (not LROed) > > [port 1] RX HW timestamp: 0x00008069357553 (not LROed) > > [port 1] RX HW timestamp: 0x0000806936e8c1 (not LROed) > > > > As shown above, non-LRO packets consistently have normal hardware timestamps on both NICs. However, on port 1, all LROed packets return a fixed, invalid timestamp > > (0x3e6eef91bc19f0fd), > > which is clearly inconsistent. > > I have also confirmed that other dynfields (rather than dynfield[1] and dynfield[2]) are unused. > > > > > > Sincerely, > > Junghan Yoon > > On Jul 22, 2025, 5:31 PM +0900, Ivan Malov , wrote: > > Hello, > > > > On Tue, 22 Jul 2025, Yoon Junghan wrote: > > > > Hello, > > > > I'm currently using DPDK 20.11 with a ConnectX-7 NIC, and I'm trying to retrieve RX hardware timestamps using `rte_mbuf_dyn_rx_timestamp_register()`. > > > > > > Does the application invoke 'rte_mbuf_dyn_rx_timestamp_register' on its own? If > > yes, consider to replace this with invocations of APIs [1] (with field name [2]) > > and [3] (with flag name [4]). For an example, please refer to [5] and [6]. > > > > This is because, as per [7], the driver in question might 'register' the field > > and the flag on its own, in response to 'DEV_RX_OFFLOAD_TIMESTAMP' request, so, > > the user application should look up the field/flag, not 'register' it afresh. > > > > If this does not help, then consider to clarify whether the timestamps are > > accurate (and whether the flag is seen in the mbufs) when LRO is not enabled. > > > > [1] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a6adf9b352a83e7d521fd6aa04e305b1c > > [2] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a5159b2d34fa801d171ed0ccce451121b > > [3] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a89d835027034f76a27eb2afe7987ae35 > > [4] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a831d7066c7193788351797a65186848a > > [5] https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/app/test-pmd/util.c#L44 > > [6] https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/app/test-pmd/util.c#L60 > > [7] https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/drivers/net/mlx5/mlx5_rxq.c#L1743 > > > > Thank you. > > > > > > When LRO is enabled, I notice that LROed mbufs seem to share identical timestamp values, and the timestamps are unexpectedly large or inconsistent. This raises > > the question of whether > > LRO is interfering with the correctness of the RX HW timestamps. > > > > I’d appreciate any clarification on whether HW RX timestamping is reliable when LRO is enabled on this platform, or if LRO should be just disabled for accurate > > per-packet timestamping. > > > > > > Sincerely, > > Junghan Yoon > > > > > > > > > >