From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 60D7F4654A for ; Thu, 10 Apr 2025 05:25:46 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 187D5402E1; Thu, 10 Apr 2025 05:25:45 +0200 (CEST) Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) by mails.dpdk.org (Postfix) with ESMTP id 3EE3740288 for ; Thu, 10 Apr 2025 05:25:43 +0200 (CEST) Received: by mail-qk1-f177.google.com with SMTP id af79cd13be357-7c56321b22cso42785985a.1 for ; Wed, 09 Apr 2025 20:25:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1744255542; x=1744860342; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=HnH473QKQ1ireEqgTfyGTLypGCOXF339xELJjEdC3Z4=; b=V7rGmWSX7nkPaHDkJnufIImr3Lmx5fv+tXo5vSP38pu6OurxL4Y9auILdMADwDPFr1 uXo8JkBo/BxVlpzHi+F+ThjGOx6GTU+OmSV+IvKmcS9rWCKa2tE3RxGqSawCzbiovxHd A1T3Lt5UM5QyLE4UhfEvcu9z5EbyBKVF73oYSzshMyTPpjVgIkH+s+izcHRPJCdaQFyD 25xqvn5iWaNUY3HHUvCs2Mobrn+vqcJpxqt2Y5Uiisw6ATAciolaycYpvbQO6HNpIHaP VEl/rGG6uSPkIA7CncV2h0Go9jmsTFoK8pG1VHkhSdsiMk49vEFN/AlTxzFMZBYZ433T 4x6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744255542; x=1744860342; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HnH473QKQ1ireEqgTfyGTLypGCOXF339xELJjEdC3Z4=; b=N6A28dVkiI0sU7KkZmmDtF2mp7DxMTSmFC2OorgpPhhml4BT98kOjPxcoY/Kk9Tspu kbDt7ch2+qnuFZikR3dC0rfv64uAsnMRt/1ybabJf5g94/wVP7az9vvqhAvHHHLxkV6x b3pT7/6A2NK0xBhj60NaH39DWj4CILCS2QMVKcqRZaE9qsFjbBgYUDbWwtfyes1GfwVS Dw0DIxZN/jHKGqYez1CgtcFOS0+yOrDQdUzweRIYghJGGNwTJX7tbrYlhOtoniYezRFR WiGe5+NO8qNAbM771Gt4/Wc8ShoiqN+PcN5l6GWJpJ8XWDxsDscXhe6PI3e+gtyp5jqj kSjg== X-Forwarded-Encrypted: i=1; AJvYcCXfIiFRJ54qIXKXo98HMwcHzdVPS7l+Tmurqgm6f6z+Jhc32Jvp5/d8C/OrVav+VeX/1GVWSA==@dpdk.org X-Gm-Message-State: AOJu0YxkB8XxdwKdEjsiWXOQwLuqfL5IyhgNtERfj2IZhdSGzMJeMcsL ygn9mH61T+GqxIO08CsSbSqLTB5+Py3VwPhPRMZfGwjCDwCh+eXQKt3y8+zLr48= X-Gm-Gg: ASbGncsrt6sWxfpZihCWekbunzjqeCBKtHwSOehP1NPlEx6YnjZCjxrOYP/ndzOkxPL Q2wLvnZsibWK822b4QRo4s7DaHTzCBSGg0AttJ4mBp47D98xjbNu1DMZiLMFgC5UXL85Hf+3z99 bpqflatybHZkSjfWeX1PumIH/ajwWcC/seuNVRSLKwY/74CZy4uXwcpmKDsFq8EN8ybxgYcke3R PL5nGcxsPt2yvPOAqFj9UMd4AalWYEu5RN6tbiYzU0ZptOk6Lf023vohxjVCsUIu1GqVKZGt21U KTc1Qq2AWKGJbQSX4ijDiFUKN0lER3clHjxIp3CcHuVGXwKma0EkSbfXEfzNPcMYbp2YvkYxhmz Xn0aDMqhLaZwWNPxD X-Google-Smtp-Source: AGHT+IGHFIKoBBrPECJiLdVzCdoFqUZRmnx+J5REhiErwNtfyDxW6gvPuBZ2kiwQj3XjH7YEjGAyeQ== X-Received: by 2002:a05:620a:c51:b0:7c5:952f:5024 with SMTP id af79cd13be357-7c7a7811261mr168203185a.22.1744255542340; Wed, 09 Apr 2025 20:25:42 -0700 (PDT) Received: from hermes.local (204-195-96-226.wavecable.com. [204.195.96.226]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c7a89faaeasm24072385a.77.2025.04.09.20.25.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Apr 2025 20:25:42 -0700 (PDT) Date: Wed, 9 Apr 2025 20:25:38 -0700 From: Stephen Hemminger To: "Lombardo, Ed" Cc: Dmitry Kozlyuk , "users@dpdk.org" Subject: Re: mbuf refcnt issue Message-ID: <20250409202538.46a08ba8@hermes.local> In-Reply-To: References: <20250408205341.68d37d0a@hermes.local> <20250409092418.70c5eb3d@hermes.local> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Wed, 9 Apr 2025 23:22:50 +0000 "Lombardo, Ed" wrote: > Hi, > I just finished modifying and testing our application to just do transmit of packets received on an NIC interface and let the rte_eth_tx_burst () free the mbuf and all works fine for both traffic types. This proves to me that my implementation of processing the packets and queueing them to tx ring and transmit from the tx ring is not buggy, which I had carefully verified in gdb early on. I still believe there is a problem with our application with many threads that can do rte_pktmbuf_free() on the same mbuf. > > I added these lines in my driver source file: > #define RTE_LITRTE_MBUF_DEBUG 1 > #define RTE_LIBRTE_MEMPOOL_DEBUG 1 > #define RTE_ENABLE_ASSERT 1 > > I don't see any asserts occur during my tx packet testing. > > The dpdk header files show the Atomic ifdef checks > rte_build_config.h:#define RTE_MBUF_REFCNT_ATOMIC > rte_mbuf_core.h: * or non-atomic) is controlled by the RTE_MBUF_REFCNT_ATOMIC flag. > rte_mbuf.h:#ifdef RTE_MBUF_REFCNT_ATOMIC > rte_mbuf.h:#else /* ! RTE_MBUF_REFCNT_ATOMIC */ > rte_mbuf.h:#endif /* RTE_MBUF_REFCNT_ATOMIC */ > > I verified in building our application with DPDK rte_mbuf.h header file that the atomic functions for mbuf refcnt read/writes are turned ON. I added junk characters, and the compiler spotted syntax errors. > > So, I am back to the question as to why I get mbuf issue with multiple threads processing the same mbuf? > > Any more suggestions. > > Thanks, > Ed > > -----Original Message----- > From: Stephen Hemminger > Sent: Wednesday, April 9, 2025 12:24 PM > To: Lombardo, Ed > Cc: Dmitry Kozlyuk ; users@dpdk.org > Subject: Re: mbuf refcnt issue > > External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe. > > On Wed, 9 Apr 2025 04:46:09 +0000 > "Lombardo, Ed" wrote: > > > Hi Stephen, > > I am looking a the rte_mbuf.h file for rte_pktmbuf_free() and it is not clear to me that it checks if the mbuf refcnt is 1 before decrementing it and allowing the mbuf and segments (if any) to be returned to free pool. > > > > Could my application issue be I have tx threads that transmit packets and does rte_pktmbuf_free(), while one other thread will perform rte_pktmbuf_free() on the same mbuf? I ensured I bump the mbuf refcnt to 2 before other threads can process the same mbuf. > > > > Thanks, > > Ed > > It doesn't need to check refcnt there. The check is done later (since mbuf can be multi segment). > > rte_pktmbuf_free > -> rte_pktmbuf_free_seg > -> rte_pktmbuf_prefree_seg > > static __rte_always_inline struct rte_mbuf * rte_pktmbuf_prefree_seg(struct rte_mbuf *m) { > __rte_mbuf_sanity_check(m, 0); > > if (likely(rte_mbuf_refcnt_read(m) == 1)) { > normal fast path. breaks the chain. > } else if (__rte_mbuf_refcnt_update(m, -1) == 0) { > refcnt > 1 logic > > Note, the refcnt doesn't always go to zero when the mbuf is put back in the pool. > The refcnt for a freed mbuf (in the pool) doesn't matter, it is free, it is dead. > The refcnt is reset to 1 when mbuf is extracted from the pool. > > > You might find something by poisoning the mbuf when it is freed, so that any attempt to use the data would get junk. Also, add check at start of rte_pktmbuf_free_seg() to catch dup free. Something like this, but only compile tested. It will break some of the functional tests, because they tend to make bogus dummy mbufs. diff --git a/lib/mbuf/rte_mbuf.h b/lib/mbuf/rte_mbuf.h index 06ab7502a5..6088b34506 100644 --- a/lib/mbuf/rte_mbuf.h +++ b/lib/mbuf/rte_mbuf.h @@ -1423,10 +1423,11 @@ static inline int __rte_pktmbuf_pinned_extbuf_decref(struct rte_mbuf *m) static __rte_always_inline struct rte_mbuf * rte_pktmbuf_prefree_seg(struct rte_mbuf *m) { - __rte_mbuf_sanity_check(m, 0); - if (likely(rte_mbuf_refcnt_read(m) == 1)) { + __rte_mbuf_sanity_check(m, 0); + unsigned int refcnt = rte_mbuf_refcnt_read(m); + if (likely(refcnt == 1)) { if (!RTE_MBUF_DIRECT(m)) { rte_pktmbuf_detach(m); if (RTE_MBUF_HAS_EXTBUF(m) && @@ -1435,13 +1436,15 @@ rte_pktmbuf_prefree_seg(struct rte_mbuf *m) return NULL; } + m->refcnt = 0; if (m->next != NULL) m->next = NULL; if (m->nb_segs != 1) m->nb_segs = 1; return m; - + } else if (unlikely(refcnt == 0 || refcnt >= UINT16_MAX - 1)) { + rte_panic("mbuf refcnt underflow %u\n", refcnt); } else if (__rte_mbuf_refcnt_update(m, -1) == 0) { if (!RTE_MBUF_DIRECT(m)) { @@ -1452,6 +1455,7 @@ rte_pktmbuf_prefree_seg(struct rte_mbuf *m) return NULL; } + m->refcnt = 0; if (m->next != NULL) m->next = NULL; if (m->nb_segs != 1)