Re: HW RX timestamp with LRO enabled on ConnectX-7 (DPDK 20.11)

[Yoon Junghan <cerotyki@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 9665 bytes --]

I isolated port 1 using -a option for EAL parameter and got the similar result.

Note that port 1 becomes port 0 in this time.
[port 0] RX HW timestamp: 0x3eac4214bc574368 (LROed)
[port 0] RX HW timestamp: 0x3eac4214bc574368 (LROed)
[port 0] RX HW timestamp: 0x3eac4214bc574368 (LROed)
[port 0] RX HW timestamp: 0x3eac4214bc574368 (LROed)
[port 0] RX HW timestamp: 0x00042819272fad (not LROed)
[port 0] RX HW timestamp: 0x000428192e6e77 (not LROed)
[port 0] RX HW timestamp: 0x000428192e7f01 (not LROed)
[port 0] RX HW timestamp: 0x000428192e833d (not LROed)

FYI, I have 4 CX-7 on the same machine. (eth0 = port 0, ... eth3 = port 3 in DPDK)
pci@0000:16:00.0  eth0             network        MT2910 Family [ConnectX-7]
pci@0000:40:00.0  eth1             network        MT2910 Family [ConnectX-7]
pci@0000:6a:00.0  eth2             network        MT2910 Family [ConnectX-7]
pci@0000:94:00.0  eth3             network        MT2910 Family [ConnectX-7]

Among them, only the first CX-7 shows consistent timestamp regardless of LRO.

Sincerely,
Junghan Yoon
On 2025년 7월 23일 PM 10:28 +0900, Ivan Malov <ivan.malov@arknetworks.am>, wrote:
> On Wed, 23 Jul 2025, Yoon Junghan wrote:
>
> > Thank you for quick response.
> >
> > 1) They are different NICs. Not in the same board. Separate adapters in different PCIe slots.
> > 2) My DPDK app uses 4 separate ports; port 0, port 1, port 2, and port 3. They are all on different boards. Thus, they are running at the same time.
>
> Excellent. I apologise for one more dumb question, but does isolating the very
> specific NIC (so that DPDK does not grab the other ones) that is known to give
> strange timestamps, result in the same/unexpected behaviour? Just to make sure.
>
> Thank you.
>
> >
> > Sincerely,
> > Junghan Yoon
> > On 2025년 7월 23일 PM 10:09 +0900, Ivan Malov <ivan.malov@arknetworks.am>, wrote:
> >
> > Hello,
> >
> > On Wed, 23 Jul 2025, Yoon Junghan wrote:
> >
> >
> > Hello,
> > As advised, I tested hardware timestamps with LRO enabled on our ConnectX-7 NICs. However, the timestamps of LROed packets still show inconsistent and abnormally large
> > gaps from normal
> > packets.
> >
> > Interestingly, I found this issue does not appear on all CX-7 NICs. Even with identical DPDK code, firmware version (28.43.2566), and hardware models from the same
> > manufacturer, only
> > specific NICs exhibit this inconsistency.
> > I have confirmed that:
> > * All NICs use the same driver and firmware version.
> > * All NICs are of the same model (MCX75310AAS-NEA_Ax).
> >
> >
> >
> > 1) Do the two "NICs" ('port 0' and 'port 1' from below printout) represent two
> > different ports/PFs of the same physical 'board'/'adapter card' in fact?
> >
> > 2) If (1) is true, were the results obtained by running the application on both
> > ports simultaneously (both managed by the DPDK at the same time)?
> >
> > (just to clarify, -- I'm confused by the fact that the NIC driver itself seems
> > to invoke 'rte_mbuf_dyn_rx_timestamp_register' for each new RxQ rather than call
> > it once and then look-up and reuse the existing offsets for more ports/queue ).
> >
> > Thank you.
> >
> >
> > * The issue occurs only when LRO is enabled together with RX hardware timestamping.
> > * Disabling LRO eliminates the issue.
> > I would appreciate any insight into how this behavior can occur on only some ports despite same software and hardware setup.
> >
> > Below is my code snippet.
> >
> > ```c
> > /*----------------------------------------------------------------------------*/
> > static inline int
> > is_timestamp_enabled(struct rte_mbuf *mbuf)
> > {
> >    static uint64_t timestamp_rx_dynflag = 0;
> >    int timestamp_rx_dynflag_offset;
> >
> >    if (!timestamp_rx_dynflag)
> >    {
> >        timestamp_rx_dynflag_offset =
> >            rte_mbuf_dynflag_lookup(RTE_MBUF_DYNFLAG_RX_TIMESTAMP_NAME, NULL);
> >        if (timestamp_rx_dynflag_offset < 0)
> >        {
> >            return 0;
> >        }
> >        timestamp_rx_dynflag = RTE_BIT64(timestamp_rx_dynflag_offset);
> >    }
> >
> >    return mbuf->ol_flags & timestamp_rx_dynflag;
> > }
> > /*----------------------------------------------------------------------------*/
> > static inline rte_mbuf_timestamp_t *
> > get_timestamp(struct rte_mbuf *mbuf)
> > {
> >    static int timestamp_dynfield_offset = -1;
> >
> >    if (timestamp_dynfield_offset < 0)
> >    {
> >        timestamp_dynfield_offset =
> >            rte_mbuf_dynfield_lookup(RTE_MBUF_DYNFIELD_TIMESTAMP_NAME, NULL);
> >        if (timestamp_dynfield_offset < 0)
> >        {
> >            return 0;
> >        }
> >    }
> >
> >    return RTE_MBUF_DYNFIELD(mbuf,
> >                              timestamp_dynfield_offset,
> >                              rte_mbuf_timestamp_t *);
> > }
> > /*----------------------------------------------------------------------------*/
> > static inline rte_mbuf_timestamp_t *
> > get_rx_hw_timestamp(struct rte_mbuf *pkt)
> > {
> >    if (!is_timestamp_enabled(pkt))
> >    {
> >        printf("rx_hw_timestamp not enabled in mbuf!\n");
> >        return NULL;
> >    }
> >
> >    return get_timestamp(pkt);
> > }
> > ```
> >
> > My DPDK application prints logs as below.
> >
> > ```c
> >    /* parse HW timestamp */
> >    rte_mbuf_timestamp_t *rx_timestamp = get_rx_hw_timestamp(pkt);
> >    printf("[port %d] RX HW timestamp: %#016lx %s\n",
> >           pctx->port_id,
> >           *rx_timestamp,
> >           pkt->ol_flags & PKT_RX_LRO ? "(LROed)" : "(not LROed)");
> > ```
> >
> > Below are observations from two CX-7 ports under identical conditions.
> >
> > Normal NIC (port 0):
> > [port 0] RX HW timestamp: 0x00007dcd2d185b (LROed)
> > [port 0] RX HW timestamp: 0x00007dcd2d1911 (LROed)
> > [port 0] RX HW timestamp: 0x00007dcd2d19c9 (LROed)
> > [port 0] RX HW timestamp: 0x00007dcd2d37ca (LROed)
> > [port 0] RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
> > [port 0] RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
> > [port 0] RX HW timestamp: 0x00007dcd30e019 (not LROed)
> > [port 0] RX HW timestamp: 0x00007dcd3280bb (not LROed)
> >
> > Erroneous NIC (port 1):
> > Below is erroneous NIC's timestamp.
> > [port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
> > [port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
> > [port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
> > [port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
> > [port 1] RX HW timestamp: 0x000080691b7557 (not LROed)
> > [port 1] RX HW timestamp: 0x000080691e2311 (not LROed)
> > [port 1] RX HW timestamp: 0x00008069357553 (not LROed)
> > [port 1] RX HW timestamp: 0x0000806936e8c1 (not LROed)
> >
> > As shown above, non-LRO packets consistently have normal hardware timestamps on both NICs. However, on port 1, all LROed packets return a fixed, invalid timestamp
> > (0x3e6eef91bc19f0fd),
> > which is clearly inconsistent.
> > I have also confirmed that other dynfields (rather than dynfield[1] and dynfield[2]) are unused.
> >
> >
> > Sincerely,
> > Junghan Yoon
> > On Jul 22, 2025, 5:31 PM +0900, Ivan Malov <ivan.malov@arknetworks.am>, wrote:
> > Hello,
> >
> > On Tue, 22 Jul 2025, Yoon Junghan wrote:
> >
> > Hello,
> >
> > I'm currently using DPDK 20.11 with a ConnectX-7 NIC, and I'm trying to retrieve RX hardware timestamps using `rte_mbuf_dyn_rx_timestamp_register()`.
> >
> >
> > Does the application invoke 'rte_mbuf_dyn_rx_timestamp_register' on its own? If
> > yes, consider to replace this with invocations of APIs [1] (with field name [2])
> > and [3] (with flag name [4]). For an example, please refer to [5] and [6].
> >
> > This is because, as per [7], the driver in question might 'register' the field
> > and the flag on its own, in response to 'DEV_RX_OFFLOAD_TIMESTAMP' request, so,
> > the user application should look up the field/flag, not 'register' it afresh.
> >
> > If this does not help, then consider to clarify whether the timestamps are
> > accurate (and whether the flag is seen in the mbufs) when LRO is not enabled.
> >
> > [1] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a6adf9b352a83e7d521fd6aa04e305b1c
> > [2] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a5159b2d34fa801d171ed0ccce451121b
> > [3] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a89d835027034f76a27eb2afe7987ae35
> > [4] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a831d7066c7193788351797a65186848a
> > [5] https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/app/test-pmd/util.c#L44
> > [6] https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/app/test-pmd/util.c#L60
> > [7] https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/drivers/net/mlx5/mlx5_rxq.c#L1743
> >
> > Thank you.
> >
> >
> > When LRO is enabled, I notice that LROed mbufs seem to share identical timestamp values, and the timestamps are unexpectedly large or inconsistent. This raises
> > the question of whether
> > LRO is interfering with the correctness of the RX HW timestamps.
> >
> > I’d appreciate any clarification on whether HW RX timestamping is reliable when LRO is enabled on this platform, or if LRO should be just disabled for accurate
> > per-packet timestamping.
> >
> >
> > Sincerely,
> > Junghan Yoon
> >
> >
> >
> >
> >

[-- Attachment #2: Type: text/html, Size: 12445 bytes --]