On Wed, 23 Jul 2025, Yoon Junghan wrote: > I isolated port 1 using -a option for EAL parameter and got the similar result. >   > Note that port 1 becomes port 0 in this time. > [port 0] RX HW timestamp: 0x3eac4214bc574368 (LROed) > [port 0] RX HW timestamp: 0x3eac4214bc574368 (LROed) > [port 0] RX HW timestamp: 0x3eac4214bc574368 (LROed) > [port 0] RX HW timestamp: 0x3eac4214bc574368 (LROed) > [port 0] RX HW timestamp: 0x00042819272fad (not LROed) > [port 0] RX HW timestamp: 0x000428192e6e77 (not LROed) > [port 0] RX HW timestamp: 0x000428192e7f01 (not LROed) > [port 0] RX HW timestamp: 0x000428192e833d (not LROed) >   > FYI, I have 4 CX-7 on the same machine. (eth0 = port 0, ... eth3 = port 3 in DPDK) > pci@0000:16:00.0  eth0             network        MT2910 Family [ConnectX-7] > pci@0000:40:00.0  eth1             network        MT2910 Family [ConnectX-7] > pci@0000:6a:00.0  eth2             network        MT2910 Family [ConnectX-7] > pci@0000:94:00.0  eth3             network        MT2910 Family [ConnectX-7] >   > Among them, only the first CX-7 shows consistent timestamp regardless of LRO. Does 'sudo hwstamp_ctl -i ' show consistent results across all the NICs? Thank you. > > Sincerely, > Junghan Yoon > On 2025년 7월 23일 PM 10:28 +0900, Ivan Malov , wrote: > On Wed, 23 Jul 2025, Yoon Junghan wrote: > > Thank you for quick response. >   > 1) They are different NICs. Not in the same board. Separate adapters in different PCIe slots. > 2) My DPDK app uses 4 separate ports; port 0, port 1, port 2, and port 3. They are all on different boards. Thus, they are running at the same time. > > > Excellent. I apologise for one more dumb question, but does isolating the very > specific NIC (so that DPDK does not grab the other ones) that is known to give > strange timestamps, result in the same/unexpected behaviour? Just to make sure. > > Thank you. > > > Sincerely, > Junghan Yoon > On 2025년 7월 23일 PM 10:09 +0900, Ivan Malov , wrote: >   > Hello, > > On Wed, 23 Jul 2025, Yoon Junghan wrote: > >   > Hello, > As advised, I tested hardware timestamps with LRO enabled on our ConnectX-7 NICs. However, the timestamps of LROed packets still show inconsistent and abnormally > large > gaps from normal > packets. >   > Interestingly, I found this issue does not appear on all CX-7 NICs. Even with identical DPDK code, firmware version (28.43.2566), and hardware models from the > same > manufacturer, only > specific NICs exhibit this inconsistency. > I have confirmed that: > * All NICs use the same driver and firmware version. > * All NICs are of the same model (MCX75310AAS-NEA_Ax). >   > > > 1) Do the two "NICs" ('port 0' and 'port 1' from below printout) represent two > different ports/PFs of the same physical 'board'/'adapter card' in fact? > > 2) If (1) is true, were the results obtained by running the application on both > ports simultaneously (both managed by the DPDK at the same time)? > > (just to clarify, -- I'm confused by the fact that the NIC driver itself seems > to invoke 'rte_mbuf_dyn_rx_timestamp_register' for each new RxQ rather than call > it once and then look-up and reuse the existing offsets for more ports/queue ). > > Thank you. > >   > * The issue occurs only when LRO is enabled together with RX hardware timestamping. > * Disabling LRO eliminates the issue. > I would appreciate any insight into how this behavior can occur on only some ports despite same software and hardware setup. >   > Below is my code snippet. >   > ```c > /*----------------------------------------------------------------------------*/ > static inline int > is_timestamp_enabled(struct rte_mbuf *mbuf) > { >    static uint64_t timestamp_rx_dynflag = 0; >    int timestamp_rx_dynflag_offset; >   >    if (!timestamp_rx_dynflag) >    { >        timestamp_rx_dynflag_offset = >            rte_mbuf_dynflag_lookup(RTE_MBUF_DYNFLAG_RX_TIMESTAMP_NAME, NULL); >        if (timestamp_rx_dynflag_offset < 0) >        { >            return 0; >        } >        timestamp_rx_dynflag = RTE_BIT64(timestamp_rx_dynflag_offset); >    } >   >    return mbuf->ol_flags & timestamp_rx_dynflag; > } > /*----------------------------------------------------------------------------*/ > static inline rte_mbuf_timestamp_t * > get_timestamp(struct rte_mbuf *mbuf) > { >    static int timestamp_dynfield_offset = -1; >   >    if (timestamp_dynfield_offset < 0) >    { >        timestamp_dynfield_offset = >            rte_mbuf_dynfield_lookup(RTE_MBUF_DYNFIELD_TIMESTAMP_NAME, NULL); >        if (timestamp_dynfield_offset < 0) >        { >            return 0; >        } >    } >   >    return RTE_MBUF_DYNFIELD(mbuf, >                              timestamp_dynfield_offset, >                              rte_mbuf_timestamp_t *); > } > /*----------------------------------------------------------------------------*/ > static inline rte_mbuf_timestamp_t * > get_rx_hw_timestamp(struct rte_mbuf *pkt) > { >    if (!is_timestamp_enabled(pkt)) >    { >        printf("rx_hw_timestamp not enabled in mbuf!\n"); >        return NULL; >    } >   >    return get_timestamp(pkt); > } > ``` >   > My DPDK application prints logs as below. >   > ```c >    /* parse HW timestamp */ >    rte_mbuf_timestamp_t *rx_timestamp = get_rx_hw_timestamp(pkt); >    printf("[port %d] RX HW timestamp: %#016lx %s\n", >           pctx->port_id, >           *rx_timestamp, >           pkt->ol_flags & PKT_RX_LRO ? "(LROed)" : "(not LROed)"); > ``` >   > Below are observations from two CX-7 ports under identical conditions. >   > Normal NIC (port 0): > [port 0] RX HW timestamp: 0x00007dcd2d185b (LROed) > [port 0] RX HW timestamp: 0x00007dcd2d1911 (LROed) > [port 0] RX HW timestamp: 0x00007dcd2d19c9 (LROed) > [port 0] RX HW timestamp: 0x00007dcd2d37ca (LROed) > [port 0] RX HW timestamp: 0x00007dcd2d4cb3 (not LROed) > [port 0] RX HW timestamp: 0x00007dcd2d4cb3 (not LROed) > [port 0] RX HW timestamp: 0x00007dcd30e019 (not LROed) > [port 0] RX HW timestamp: 0x00007dcd3280bb (not LROed) >   > Erroneous NIC (port 1): > Below is erroneous NIC's timestamp. > [port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > [port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > [port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > [port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > [port 1] RX HW timestamp: 0x000080691b7557 (not LROed) > [port 1] RX HW timestamp: 0x000080691e2311 (not LROed) > [port 1] RX HW timestamp: 0x00008069357553 (not LROed) > [port 1] RX HW timestamp: 0x0000806936e8c1 (not LROed) > > As shown above, non-LRO packets consistently have normal hardware timestamps on both NICs. However, on port 1, all LROed packets return a fixed, invalid timestamp > (0x3e6eef91bc19f0fd), > which is clearly inconsistent. > I have also confirmed that other dynfields (rather than dynfield[1] and dynfield[2]) are unused. >   > > Sincerely, > Junghan Yoon > On Jul 22, 2025, 5:31 PM +0900, Ivan Malov , wrote: > Hello, > > On Tue, 22 Jul 2025, Yoon Junghan wrote: > > Hello, >   > I'm currently using DPDK 20.11 with a ConnectX-7 NIC, and I'm trying to retrieve RX hardware timestamps using `rte_mbuf_dyn_rx_timestamp_register()`. > > > Does the application invoke 'rte_mbuf_dyn_rx_timestamp_register' on its own? If > yes, consider to replace this with invocations of APIs [1] (with field name [2]) > and [3] (with flag name [4]). For an example, please refer to [5] and [6]. > > This is because, as per [7], the driver in question might 'register' the field > and the flag on its own, in response to 'DEV_RX_OFFLOAD_TIMESTAMP' request, so, > the user application should look up the field/flag, not 'register' it afresh. > > If this does not help, then consider to clarify whether the timestamps are > accurate (and whether the flag is seen in the mbufs) when LRO is not enabled. > > [1] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a6adf9b352a83e7d521fd6aa04e305b1c > [2] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a5159b2d34fa801d171ed0ccce451121b > [3] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a89d835027034f76a27eb2afe7987ae35 > [4] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a831d7066c7193788351797a65186848a > [5] https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/app/test-pmd/util.c#L44 > [6] https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/app/test-pmd/util.c#L60 > [7] https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/drivers/net/mlx5/mlx5_rxq.c#L1743 > > Thank you. > >   > When LRO is enabled, I notice that LROed mbufs seem to share identical timestamp values, and the timestamps are unexpectedly large or inconsistent. This raises > the question of whether > LRO is interfering with the correctness of the RX HW timestamps. >   > I’d appreciate any clarification on whether HW RX timestamping is reliable when LRO is enabled on this platform, or if LRO should be just disabled for accurate > per-packet timestamping. >   > > Sincerely, > Junghan Yoon > > >   > > > >