From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EC5D846BF1 for ; Wed, 23 Jul 2025 15:09:47 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E219A40B99; Wed, 23 Jul 2025 15:09:47 +0200 (CEST) Received: from agw.arknetworks.am (agw.arknetworks.am [79.141.165.80]) by mails.dpdk.org (Postfix) with ESMTP id 058A240B98 for ; Wed, 23 Jul 2025 15:09:47 +0200 (CEST) Received: from debian (unknown [78.109.70.60]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by agw.arknetworks.am (Postfix) with ESMTPSA id 5F08CE0568; Wed, 23 Jul 2025 17:09:46 +0400 (+04) DKIM-Filter: OpenDKIM Filter v2.11.0 agw.arknetworks.am 5F08CE0568 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arknetworks.am; s=default; t=1753276186; bh=mk7e5Wz1PQcc0nVVPGigqMjokYGvzYZginRq3JCAEHs=; h=Date:From:To:cc:Subject:In-Reply-To:References:From; b=hWLS3FkJq14gCdACpldR5ygqbloz/YvKbZCYAxqzhISWAMVztEjY3r7T1yLKIWNrk 6bvhqkDJqAKHT8zXFLJWw3ZppzyuFXyY8sFEKx0IloDKtZBSO1V1/zpy29YBaRaDhH 1Yecqrbw8WeaNULc3H67yUBvPqBFKKaSRcmZZb9b8fznlmR62+MwS/NRCGju+sh8Xa 5JYxO1dDSCdF0IEyRy2h5eY7AvwQxvaEl6tm2VUz3c+LKV4makGQ0cGzTSyuqXUYXQ kT5kKV0kW9LlaIT1erZiyYLAT1h2qqZhpLxcxRgkkeeQ0Wth03dFhi8+7nur+3kCon FanHfcXLZpU9Q== Date: Wed, 23 Jul 2025 17:09:45 +0400 (+04) From: Ivan Malov To: Yoon Junghan cc: users@dpdk.org Subject: Re: HW RX timestamp with LRO enabled on ConnectX-7 (DPDK 20.11) In-Reply-To: Message-ID: <391481b9-1b18-5567-a8a0-563c5c5eb37c@arknetworks.am> References: <3f7df93b-ee00-bcdb-337c-8faa7bd73f6c@arknetworks.am> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="8323328-1234223104-1753276186=:17938" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --8323328-1234223104-1753276186=:17938 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8BIT Hello, On Wed, 23 Jul 2025, Yoon Junghan wrote: > Hello, > As advised, I tested hardware timestamps with LRO enabled on our ConnectX-7 NICs. However, the timestamps of LROed packets still show inconsistent and abnormally large gaps from normal > packets. >   > Interestingly, I found this issue does not appear on all CX-7 NICs. Even with identical DPDK code, firmware version (28.43.2566), and hardware models from the same manufacturer, only > specific NICs exhibit this inconsistency. > I have confirmed that: > * All NICs use the same driver and firmware version. > * All NICs are of the same model (MCX75310AAS-NEA_Ax). 1) Do the two "NICs" ('port 0' and 'port 1' from below printout) represent two different ports/PFs of the same physical 'board'/'adapter card' in fact? 2) If (1) is true, were the results obtained by running the application on both ports simultaneously (both managed by the DPDK at the same time)? (just to clarify, -- I'm confused by the fact that the NIC driver itself seems to invoke 'rte_mbuf_dyn_rx_timestamp_register' for each new RxQ rather than call it once and then look-up and reuse the existing offsets for more ports/queue ). Thank you. > * The issue occurs only when LRO is enabled together with RX hardware timestamping. > * Disabling LRO eliminates the issue. > I would appreciate any insight into how this behavior can occur on only some ports despite same software and hardware setup. >   > Below is my code snippet. >   > ```c > /*----------------------------------------------------------------------------*/ > static inline int > is_timestamp_enabled(struct rte_mbuf *mbuf) > { >    static uint64_t timestamp_rx_dynflag = 0; >    int timestamp_rx_dynflag_offset; >   >    if (!timestamp_rx_dynflag) >    { >        timestamp_rx_dynflag_offset = >            rte_mbuf_dynflag_lookup(RTE_MBUF_DYNFLAG_RX_TIMESTAMP_NAME, NULL); >        if (timestamp_rx_dynflag_offset < 0) >        { >            return 0; >        } >        timestamp_rx_dynflag = RTE_BIT64(timestamp_rx_dynflag_offset); >    } >   >    return mbuf->ol_flags & timestamp_rx_dynflag; > } > /*----------------------------------------------------------------------------*/ > static inline rte_mbuf_timestamp_t * > get_timestamp(struct rte_mbuf *mbuf) > { >    static int timestamp_dynfield_offset = -1; >   >    if (timestamp_dynfield_offset < 0) >    { >        timestamp_dynfield_offset = >            rte_mbuf_dynfield_lookup(RTE_MBUF_DYNFIELD_TIMESTAMP_NAME, NULL); >        if (timestamp_dynfield_offset < 0) >        { >            return 0; >        } >    } >   >    return RTE_MBUF_DYNFIELD(mbuf, >                              timestamp_dynfield_offset, >                              rte_mbuf_timestamp_t *); > } > /*----------------------------------------------------------------------------*/ > static inline rte_mbuf_timestamp_t * > get_rx_hw_timestamp(struct rte_mbuf *pkt) > { >    if (!is_timestamp_enabled(pkt)) >    { >        printf("rx_hw_timestamp not enabled in mbuf!\n"); >        return NULL; >    } >   >    return get_timestamp(pkt); > } > ``` >   > My DPDK application prints logs as below. >   > ```c >    /* parse HW timestamp */ >    rte_mbuf_timestamp_t *rx_timestamp = get_rx_hw_timestamp(pkt); >    printf("[port %d] RX HW timestamp: %#016lx %s\n", >           pctx->port_id, >           *rx_timestamp, >           pkt->ol_flags & PKT_RX_LRO ? "(LROed)" : "(not LROed)"); > ``` >   > Below are observations from two CX-7 ports under identical conditions. >   > Normal NIC (port 0): > [port 0] RX HW timestamp: 0x00007dcd2d185b (LROed) > [port 0] RX HW timestamp: 0x00007dcd2d1911 (LROed) > [port 0] RX HW timestamp: 0x00007dcd2d19c9 (LROed) > [port 0] RX HW timestamp: 0x00007dcd2d37ca (LROed) > [port 0] RX HW timestamp: 0x00007dcd2d4cb3 (not LROed) > [port 0] RX HW timestamp: 0x00007dcd2d4cb3 (not LROed) > [port 0] RX HW timestamp: 0x00007dcd30e019 (not LROed) > [port 0] RX HW timestamp: 0x00007dcd3280bb (not LROed) >   > Erroneous NIC (port 1): > Below is erroneous NIC's timestamp. > [port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > [port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > [port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > [port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed) > [port 1] RX HW timestamp: 0x000080691b7557 (not LROed) > [port 1] RX HW timestamp: 0x000080691e2311 (not LROed) > [port 1] RX HW timestamp: 0x00008069357553 (not LROed) > [port 1] RX HW timestamp: 0x0000806936e8c1 (not LROed) > > As shown above, non-LRO packets consistently have normal hardware timestamps on both NICs. However, on port 1, all LROed packets return a fixed, invalid timestamp (0x3e6eef91bc19f0fd), > which is clearly inconsistent. > I have also confirmed that other dynfields (rather than dynfield[1] and dynfield[2]) are unused. >   > > Sincerely, > Junghan Yoon > On Jul 22, 2025, 5:31 PM +0900, Ivan Malov , wrote: > Hello, > > On Tue, 22 Jul 2025, Yoon Junghan wrote: > > Hello, >   > I'm currently using DPDK 20.11 with a ConnectX-7 NIC, and I'm trying to retrieve RX hardware timestamps using `rte_mbuf_dyn_rx_timestamp_register()`. > > > Does the application invoke 'rte_mbuf_dyn_rx_timestamp_register' on its own? If > yes, consider to replace this with invocations of APIs [1] (with field name [2]) > and [3] (with flag name [4]). For an example, please refer to [5] and [6]. > > This is because, as per [7], the driver in question might 'register' the field > and the flag on its own, in response to 'DEV_RX_OFFLOAD_TIMESTAMP' request, so, > the user application should look up the field/flag, not 'register' it afresh. > > If this does not help, then consider to clarify whether the timestamps are > accurate (and whether the flag is seen in the mbufs) when LRO is not enabled. > > [1] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a6adf9b352a83e7d521fd6aa04e305b1c > [2] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a5159b2d34fa801d171ed0ccce451121b > [3] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a89d835027034f76a27eb2afe7987ae35 > [4] https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a831d7066c7193788351797a65186848a > [5] https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/app/test-pmd/util.c#L44 > [6] https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/app/test-pmd/util.c#L60 > [7] https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/drivers/net/mlx5/mlx5_rxq.c#L1743 > > Thank you. > >   > When LRO is enabled, I notice that LROed mbufs seem to share identical timestamp values, and the timestamps are unexpectedly large or inconsistent. This raises > the question of whether > LRO is interfering with the correctness of the RX HW timestamps. >   > I’d appreciate any clarification on whether HW RX timestamping is reliable when LRO is enabled on this platform, or if LRO should be just disabled for accurate > per-packet timestamping. >   > > Sincerely, > Junghan Yoon > > > --8323328-1234223104-1753276186=:17938--