Re: HW RX timestamp with LRO enabled on ConnectX-7 (DPDK 20.11)

[Yoon Junghan <cerotyki@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 6487 bytes --]

Hello,
As advised, I tested hardware timestamps with LRO enabled on our ConnectX-7 NICs. However, the timestamps of LROed packets still show inconsistent and abnormally large gaps from normal packets.

Interestingly, I found this issue does not appear on all CX-7 NICs. Even with identical DPDK code, firmware version (28.43.2566), and hardware models from the same manufacturer, only specific NICs exhibit this inconsistency.
I have confirmed that:

• All NICs use the same driver and firmware version.
• All NICs are of the same model (MCX75310AAS-NEA_Ax).
• The issue occurs only when LRO is enabled together with RX hardware timestamping.
• Disabling LRO eliminates the issue.

I would appreciate any insight into how this behavior can occur on only some ports despite same software and hardware setup.

Below is my code snippet.

```c
/*----------------------------------------------------------------------------*/
static inline int
is_timestamp_enabled(struct rte_mbuf *mbuf)
{
   static uint64_t timestamp_rx_dynflag = 0;
   int timestamp_rx_dynflag_offset;

   if (!timestamp_rx_dynflag)
   {
       timestamp_rx_dynflag_offset =
           rte_mbuf_dynflag_lookup(RTE_MBUF_DYNFLAG_RX_TIMESTAMP_NAME, NULL);
       if (timestamp_rx_dynflag_offset < 0)
       {
           return 0;
       }
       timestamp_rx_dynflag = RTE_BIT64(timestamp_rx_dynflag_offset);
   }

   return mbuf->ol_flags & timestamp_rx_dynflag;
}
/*----------------------------------------------------------------------------*/
static inline rte_mbuf_timestamp_t *
get_timestamp(struct rte_mbuf *mbuf)
{
   static int timestamp_dynfield_offset = -1;

   if (timestamp_dynfield_offset < 0)
   {
       timestamp_dynfield_offset =
           rte_mbuf_dynfield_lookup(RTE_MBUF_DYNFIELD_TIMESTAMP_NAME, NULL);
       if (timestamp_dynfield_offset < 0)
       {
           return 0;
       }
   }

   return RTE_MBUF_DYNFIELD(mbuf,
                             timestamp_dynfield_offset,
                             rte_mbuf_timestamp_t *);
}
/*----------------------------------------------------------------------------*/
static inline rte_mbuf_timestamp_t *
get_rx_hw_timestamp(struct rte_mbuf *pkt)
{
   if (!is_timestamp_enabled(pkt))
   {
       printf("rx_hw_timestamp not enabled in mbuf!\n");
       return NULL;
   }

   return get_timestamp(pkt);
}
```

My DPDK application prints logs as below.

```c
   /* parse HW timestamp */
   rte_mbuf_timestamp_t *rx_timestamp = get_rx_hw_timestamp(pkt);
   printf("[port %d] RX HW timestamp: %#016lx %s\n",
          pctx->port_id,
          *rx_timestamp,
          pkt->ol_flags & PKT_RX_LRO ? "(LROed)" : "(not LROed)");
```

Below are observations from two CX-7 ports under identical conditions.

Normal NIC (port 0):
[port 0] RX HW timestamp: 0x00007dcd2d185b (LROed)
[port 0] RX HW timestamp: 0x00007dcd2d1911 (LROed)
[port 0] RX HW timestamp: 0x00007dcd2d19c9 (LROed)
[port 0] RX HW timestamp: 0x00007dcd2d37ca (LROed)
[port 0] RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
[port 0] RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
[port 0] RX HW timestamp: 0x00007dcd30e019 (not LROed)
[port 0] RX HW timestamp: 0x00007dcd3280bb (not LROed)

Erroneous NIC (port 1):
Below is erroneous NIC's timestamp.
[port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
[port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
[port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
[port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
[port 1] RX HW timestamp: 0x000080691b7557 (not LROed)
[port 1] RX HW timestamp: 0x000080691e2311 (not LROed)
[port 1] RX HW timestamp: 0x00008069357553 (not LROed)
[port 1] RX HW timestamp: 0x0000806936e8c1 (not LROed)

As shown above, non-LRO packets consistently have normal hardware timestamps on both NICs. However, on port 1, all LROed packets return a fixed, invalid timestamp (0x3e6eef91bc19f0fd), which is clearly inconsistent.
I have also confirmed that other dynfields (rather than dynfield[1] and dynfield[2]) are unused.


Sincerely,
Junghan Yoon
On Jul 22, 2025, 5:31 PM +0900, Ivan Malov <ivan.malov@arknetworks.am>, wrote:
> Hello,
>
> On Tue, 22 Jul 2025, Yoon Junghan wrote:
>
> > Hello,
> >
> > I'm currently using DPDK 20.11 with a ConnectX-7 NIC, and I'm trying to retrieve RX hardware timestamps using `rte_mbuf_dyn_rx_timestamp_register()`.
>
> Does the application invoke 'rte_mbuf_dyn_rx_timestamp_register' on its own? If
> yes, consider to replace this with invocations of APIs [1] (with field name [2])
> and [3] (with flag name [4]). For an example, please refer to [5] and [6].
>
> This is because, as per [7], the driver in question might 'register' the field
> and the flag on its own, in response to 'DEV_RX_OFFLOAD_TIMESTAMP' request, so,
> the user application should look up the field/flag, not 'register' it afresh.
>
> If this does not help, then consider to clarify whether the timestamps are
> accurate (and whether the flag is seen in the mbufs) when LRO is not enabled.
>
> [1] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a6adf9b352a83e7d521fd6aa04e305b1c
> [2] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a5159b2d34fa801d171ed0ccce451121b
> [3] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a89d835027034f76a27eb2afe7987ae35
> [4] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a831d7066c7193788351797a65186848a
> [5] https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/app/test-pmd/util.c#L44
> [6] https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/app/test-pmd/util.c#L60
> [7] https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/drivers/net/mlx5/mlx5_rxq.c#L1743
>
> Thank you.
>
> >
> > When LRO is enabled, I notice that LROed mbufs seem to share identical timestamp values, and the timestamps are unexpectedly large or inconsistent. This raises the question of whether
> > LRO is interfering with the correctness of the RX HW timestamps.
> >
> > I’d appreciate any clarification on whether HW RX timestamping is reliable when LRO is enabled on this platform, or if LRO should be just disabled for accurate per-packet timestamping.
> >
> >
> > Sincerely,
> > Junghan Yoon
> >

[-- Attachment #2: Type: text/html, Size: 8810 bytes --]