From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 57FC046506 for ; Sat, 5 Apr 2025 00:29:11 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F3CC340144; Sat, 5 Apr 2025 00:29:10 +0200 (CEST) Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) by mails.dpdk.org (Postfix) with ESMTP id 0158A400EF for ; Sat, 5 Apr 2025 00:29:08 +0200 (CEST) Received: by mail-lf1-f46.google.com with SMTP id 2adb3069b0e04-54acc04516fso2348244e87.0 for ; Fri, 04 Apr 2025 15:29:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1743805748; x=1744410548; darn=dpdk.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:to:subject:user-agent:mime-version:date:message-id:from :to:cc:subject:date:message-id:reply-to; bh=GoFyJ7LqTcw5K8O5ITyPa2dSseCWkffKGWru1eeezPQ=; b=cUfdxw3b3CUl7rrfdU/Z6/CfBpLgUm0SnsqvJw/XdPdQa99nsTYcCLoLZcwlbCpAch Sd1YqDZzMJDSXn1FBIkJpm01543U8I6TVHrChANrhiCEIrc69lbbqtVShx9XSpqI521x 6isY+PaxLv2CNE5X9sx8GDDLcMcNWE0XmxZnK3S5dQtwAQXI2L4kcap5DnhUwA6ApQ5H 3OBCYHj3XvDzh4PYjFZEV05BRLUCt35keng95tLGhWtDM6UoaQBL1p9WajIcmaowXfR4 1o2VNQjAYU/4Ttuxllxut4AAT8+3kCYsMtFJhLD1npNv4phcgZnp/39f27fx/DMJPgFE t/1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743805748; x=1744410548; h=content-transfer-encoding:in-reply-to:from:content-language :references:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GoFyJ7LqTcw5K8O5ITyPa2dSseCWkffKGWru1eeezPQ=; b=sgpwROwSufHL/qK9IfaicW4jLl2K49MD1509VkW38eN84zOH9FzV6YUDAlsh4s7v2S fIMTrFgUVb5vRZ0DrhXCwGMt0uhoBm+M3WAVClCOIFXchb8SULr05AbrO4ffRFQXnbxU lXkwdZmghyEVudxbqheyGkR9aMM0rN8scicBLCASK+mv2GSpCkWKLjVMjsvFT9uoxgGv cngsBd5WtDqkYUbDavnL3OV5jLKqZO7e85Vrusx+eyzcAt1UIFsjvAZe0aRNPDk7FUii K6Wo9Wy48qeuVMO3QFJRDN6HdLkoJONE0xAmE249oaUE6KevqEjCoa/OueyQ1WJO4bSA /DKA== X-Forwarded-Encrypted: i=1; AJvYcCWVUXBy+9NiHK0t0a1RJZhiG1uV+kj3vuHZ2anUC9TyLgZiLtlqrnRS86MtxYzU/NrikUXCqg==@dpdk.org X-Gm-Message-State: AOJu0Yw3DuTd/D7cqjIUR7Ep2nGpWZh11BV3op3bZRdAt4dLUdCwqbmv HOOjdwMCd1at8+MbujsVx7MuQt0MPtqie9EtjY856NkxVphvASwl X-Gm-Gg: ASbGncuEehf3XH22n0JZNS4sR4RY/QNf7oT41H3wgq4BpLQpI8wtSsj11Rt5KzEQoal D74dC5Cj7YkXY3JksdiOgSmPy2iu7pMYk3EsV/q+U7UD7NTePrVzqPrUS6+6n5jsOMrH1YfLZPm tIkQjaOggd7YvUhLBHA1K977KTHJ7kck/99BwysuHep3cJw9/ivlCq/LA4rMdAzUWo0de09cyvU g5O0zjs+jh6oGvb90ZEGtO20cxx5mdsKugvMagiAYySH0HQtz/xjKDcSrZce5y+VvkRZoldlmOo fQJshuQ7BinPyFPYBI5SKkUfsPHmLG5cNix1biBXdrpU1ZXRYPl6DKs3DM1EOCwT2vMDMkh3Dp3 P0tUYGquYydd68Bj8U1Vl+nbgw0eN20tN/9iW X-Google-Smtp-Source: AGHT+IHiLUkIRjXijDy583ICRqPb6SD2x93alc+lF6VrIrnSEV0wYMe45Kd84f7qWWJG5XuSRRyCfg== X-Received: by 2002:a05:6512:230c:b0:549:78bd:6b9f with SMTP id 2adb3069b0e04-54c232f949bmr1260073e87.30.1743805747916; Fri, 04 Apr 2025 15:29:07 -0700 (PDT) Received: from [192.168.88.232] (broadband-109-173-43-194.ip.moscow.rt.ru. [109.173.43.194]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-54c1e65d34fsm538942e87.174.2025.04.04.15.29.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 04 Apr 2025 15:29:07 -0700 (PDT) Message-ID: Date: Sat, 5 Apr 2025 01:29:05 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: mbuf refcnt issue To: "Lombardo, Ed" , "users@dpdk.org" References: Content-Language: en-US From: Dmitry Kozlyuk In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi Ed, On 05.04.2025 01:00, Lombardo, Ed wrote: > > Hi, > > I have an application where we receive packets and transmit them.  The > packet data is inspected and later mbuf is freed to mempool. > > The pipeline is such that the Rx packet mbuf is saved to rx worker > ring, then the application threads process the packets and decides if > to transmit the packet and if true then increments the mbuf to a value > of 2. > Do I understand the pipeline correctly? Rx thread:     receive mbuf     put mbuf into the ring     inspect mbuf     free mbuf Worker thread:     take mbuf from the ring     if decided to transmit it,         increment refcnt         transmit mbuf If so, there's a problem that after Rx thread puts mbuf into the ring, mbuf is owned by Rx thread and the ring, so its refcnt must be 2 when it enters the ring: Rx thread:     receive mbuf     increment refcnt     put mbuf into the ring     inspect mbuf     free mbuf (just decrements refcnt if > 1) Worker thread:     take mbuf from the ring     if decided to transmit it,         transmit (or put into the bulk transmitted later)     else         free mbuf (just decrements refcnt if > 1) > The batch of mbufs to transmit are put in a Tx ring queue for the Tx > thread to pull from and call the DPDK rte_eth_tx_burst() with the > batch of mbufs (limited to 400 mbufs).  In theory the transmit > operation will decrement the mbuf refcnt.  In our application we could > see the tx of the mbuf followed by another application thread that > calls to free the mbufs, or vice versa.  We have no way to synchronize > these threads. > > Is the mbuf refcnt updates thread safe to allow un-deterministic > handling of the mbufs among multiple threads?  The decision to > transmit the mbuf and increment the mbuf refcnt and load in the tx > ring is completed before the application says it is finished and frees > the mbufs. > Have you validated this assumption? If my understanding above is correct, there's no synchronization and thus no guarantees. > > I am seeing in my error checking code the mbuf refcnt contains large > values like 65520, 65529, 65530, 65534, 65535 in the early pipeline > stage refcnt checks. > > I read online and in the DPDK code that the mbuf refcnt update is > atomic, and is thread safe; so, this is good. > > Now this part is unclear to me and that is when the rte_eth_tx_burst() > is called and returns the number of packets transmitted , does this >  mean that transmit of the packets are completed and mbuf refcnt is > decremented by 1 on return, or maybe the Tx engine queue is populated > and mbuf refcnt is not decremented until it is actually transmitted, > or much worse later in time. > > Is the DPDK Tx operation intended to be the last stage of any pipeline > that will free the mbuf if successfully transmitted? > Return from rte_eth_tx_burst() means that mbufs are queued for transmission. Hardware completes transmission asynchronously. The next call to rte_eth_tx_burst() will poll HW, learn status of mbufs *previously* queued, and calls rte_pktmbuf_free() for those that are transmitted. The latter will free mbufs to mempool if and only if refcnt == 1.