Hello,
As advised, I tested hardware timestamps with LRO enabled on our ConnectX-7 NICs. However, the timestamps of LROed packets still show inconsistent and abnormally large gaps from normal packets.
Interestingly, I found this issue does not appear on all CX-7 NICs. Even with identical DPDK code, firmware version (28.43.2566), and hardware models from the same manufacturer, only specific NICs exhibit this inconsistency.
I have confirmed that:
- All NICs use the same driver and firmware version.
- All NICs are of the same model (
MCX75310AAS-NEA_Ax).
- The issue occurs only when LRO is enabled together with RX hardware timestamping.
- Disabling LRO eliminates the issue.
I would appreciate any insight into how this behavior can occur on only some ports despite same software and hardware setup.
Below is my code snippet.
```c
/*----------------------------------------------------------------------------*/
static inline int
is_timestamp_enabled(struct rte_mbuf *mbuf)
{
static uint64_t timestamp_rx_dynflag = 0;
int timestamp_rx_dynflag_offset;
if (!timestamp_rx_dynflag)
{
timestamp_rx_dynflag_offset =
rte_mbuf_dynflag_lookup(RTE_MBUF_DYNFLAG_RX_TIMESTAMP_NAME, NULL);
if (timestamp_rx_dynflag_offset < 0)
{
return 0;
}
timestamp_rx_dynflag = RTE_BIT64(timestamp_rx_dynflag_offset);
}
return mbuf->ol_flags & timestamp_rx_dynflag;
}
/*----------------------------------------------------------------------------*/
static inline rte_mbuf_timestamp_t *
get_timestamp(struct rte_mbuf *mbuf)
{
static int timestamp_dynfield_offset = -1;
if (timestamp_dynfield_offset < 0)
{
timestamp_dynfield_offset =
rte_mbuf_dynfield_lookup(RTE_MBUF_DYNFIELD_TIMESTAMP_NAME, NULL);
if (timestamp_dynfield_offset < 0)
{
return 0;
}
}
return RTE_MBUF_DYNFIELD(mbuf,
timestamp_dynfield_offset,
rte_mbuf_timestamp_t *);
}
/*----------------------------------------------------------------------------*/
static inline rte_mbuf_timestamp_t *
get_rx_hw_timestamp(struct rte_mbuf *pkt)
{
if (!is_timestamp_enabled(pkt))
{
printf("rx_hw_timestamp not enabled in mbuf!\n");
return NULL;
}
return get_timestamp(pkt);
}
```
My DPDK application prints logs as below.
```c
/* parse HW timestamp */
rte_mbuf_timestamp_t *rx_timestamp = get_rx_hw_timestamp(pkt);
printf("[port %d] RX HW timestamp: %#016lx %s\n",
pctx->port_id,
*rx_timestamp,
pkt->ol_flags & PKT_RX_LRO ? "(LROed)" : "(not LROed)");
```
Below are observations from two CX-7 ports under identical conditions.
Normal NIC (port 0):
[port 0] RX HW timestamp: 0x00007dcd2d185b (LROed)
[port 0] RX HW timestamp: 0x00007dcd2d1911 (LROed)
[port 0] RX HW timestamp: 0x00007dcd2d19c9 (LROed)
[port 0] RX HW timestamp: 0x00007dcd2d37ca (LROed)
[port 0] RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
[port 0] RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
[port 0] RX HW timestamp: 0x00007dcd30e019 (not LROed)
[port 0] RX HW timestamp: 0x00007dcd3280bb (not LROed)
Erroneous NIC (port 1):
Below is erroneous NIC's timestamp.
[port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
[port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
[port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
[port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
[port 1] RX HW timestamp: 0x000080691b7557 (not LROed)
[port 1] RX HW timestamp: 0x000080691e2311 (not LROed)
[port 1] RX HW timestamp: 0x00008069357553 (not LROed)
[port 1] RX HW timestamp: 0x0000806936e8c1 (not LROed)
As shown above, non-LRO packets consistently have normal hardware timestamps on both NICs. However, on port 1, all LROed packets return a fixed, invalid timestamp (0x3e6eef91bc19f0fd), which is clearly inconsistent.
I have also confirmed that other dynfields (rather than dynfield[1] and dynfield[2]) are unused.