From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 76B9945D06 for ; Thu, 14 Nov 2024 17:14:31 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 449334025D; Thu, 14 Nov 2024 17:14:31 +0100 (CET) Received: from mail-oo1-f50.google.com (mail-oo1-f50.google.com [209.85.161.50]) by mails.dpdk.org (Postfix) with ESMTP id AE6F34021F for ; Thu, 14 Nov 2024 17:14:29 +0100 (CET) Received: by mail-oo1-f50.google.com with SMTP id 006d021491bc7-5ebc05007daso355954eaf.1 for ; Thu, 14 Nov 2024 08:14:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1731600869; x=1732205669; darn=dpdk.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=wMv2t3SlB5OCNktFVB/p3q029f6Ln9WVGzUx6Ta6QVw=; b=QGP02eTdF9XERjFQDOkfgrpZwb84DinRLLSIcgih6Anv+1niH30ASA0PDkIwRSUWQs Y5iPXnUYTTiLB/4w/2eu5PE8mpEscdQ4GvrNwDmOxGWQt/jZ0L+TRgE1ZtxA3Ntw3lds V9ZkQcqLVEi59xQBWT3nSa4rskELLmF4WqHO8jusL7IJ1q1SzJmxUXlrBrJyES/x9R9p PbevuGZMeY8T7XoYOefaBU0Eib4yBijZDR9fq+ZepWMcr79X86SiehT+h3Z6qO/V2mlQ Cu4R/Q41UKDih0EzL2Zb4wNbI3gLblvXSCQp5QP+Iynw2vTOsmsTN3K/MSsval3SGJpX +OuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731600869; x=1732205669; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wMv2t3SlB5OCNktFVB/p3q029f6Ln9WVGzUx6Ta6QVw=; b=JHh2JZmXPYECJZ6Stagda/20/GANHF9jAtau7dDbWcWUNem0CNQfKLHGpFlbw6E72P UwJc9SEIkux0t+n/nt8QjWSYCa9uNT/OCpO+6dQqR2KSv4Y+zjK5ZLp+pgE1/WXEhT8J Bgv8/5IjV0qINDkJ2g9Z/pdfP5/6hxKAPDMULaIrtayMmtID2jNy+WN2k7qsMiSqkccR TbS3QZO4yFguxZ4zoBbm0S/RAUWf4/9MV2opFZDfuIXIIFZqxJp/xPEyStYfBrn/N2/a LP9NF3TnXjvc5RTY2quZb8zOzCpYlMtZeZByGwYy7EtqdD4c+jHb+0unRU5ZqDNlK4Ii Uafw== X-Gm-Message-State: AOJu0YzKofhJYvmrRLsaGeVkuVRh5V97AVfrZCHMJDdb27gGU5FkROxr 4SAf9ONnP3dSmj5u/I5g032u4Zam5gca1opu3mDPLoUUWi8y5NhObW1BpxPY3xjKDvXnL+/5Jtx sGfrDJc0UOgvtY9RMg1O2K/0pFua8o/H7hHg= X-Google-Smtp-Source: AGHT+IFlT9HkJQtLoz/KIH075t7KiVgb7Y5pZ+TcFIJ7R/ZQppcsu/0fldvH07ipEBMUgiiHZ+n0S8+bMjCOFAFI97Q= X-Received: by 2002:a05:6871:a982:b0:287:b2b:c343 with SMTP id 586e51a60fabf-295e8cc5359mr8482579fac.6.1731600868652; Thu, 14 Nov 2024 08:14:28 -0800 (PST) MIME-Version: 1.0 References: <20241110091245.677a3983@hermes.local> In-Reply-To: From: Alan Beadle Date: Thu, 14 Nov 2024 11:14:17 -0500 Message-ID: Subject: Re: mbufs getting reused despite nonzero refcnt To: Stephen Hemminger Cc: users@dpdk.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org I think I have a theory about what might be going wrong now, but I have a question. Is there any circumstance under which rte_eth_rx_burst() will return mbufs which were previously sent by the same machine? That is, might my application be "receiving" it's own packets? It looks like this may be the case, and since dropped packets get freed, and packets not addressed to the machine are dropped, I might be accidentally freeing mbufs when they are "received" on the same machine that sent them. On Tue, Nov 12, 2024 at 8:02=E2=80=AFAM Alan Beadle w= rote: > > Is there anything in the usage I described in my previous email which > might explain this problem? Is there anything else wrong with what I'm > doing driver-wise? > > On Sun, Nov 10, 2024 at 12:31=E2=80=AFPM Alan Beadle wrote: > > > > I'm using the vfio-pci module with Intel X550-T2 NICs. I believe this > > means it will use the ixgbe driver? To be honest, I am a bit confused > > about the use of drivers in DPDK. I am using the first setup that I > > got to work and send/receive packets. Additional tips would be greatly > > appreciated. After loading the vfio-pci module I run dpdk-devbind.py > > --bind vfio-pci 65:00.1 and then I just use the standard DPDK API > > calls in my app. I was meaning to revisit this once my app was more > > complete. > > > > On Sun, Nov 10, 2024 at 12:12=E2=80=AFPM Stephen Hemminger > > wrote: > > > > > > On Sun, 10 Nov 2024 11:23:29 -0500 > > > Alan Beadle wrote: > > > > > > > Hi everyone, > > > > > > > > I am using DPDK to send two-way traffic between a pair of machines.= My > > > > application has local readers, remote acknowledgments, as well as > > > > automatic retries when a packet is lost. For these reasons I am usi= ng > > > > rte_mbuf_refcnt_update() to prevent the NIC from freeing the packet > > > > and recycling the mbuf before my local readers are done and the rem= ote > > > > reader has acknowledged the message. I was advised to do this in an > > > > earlier thread on this mailing list. > > > > > > > > However, this does not seem to be working. After running my app for > > > > awhile and exchanging about 1000 messages in this way, my queue of > > > > unacknowledged mbufs is getting corrupted. The mbufs attached to my > > > > queue seem to contain data for newer messages than what is supposed= to > > > > be in them, and in some cases contains a totally different type of > > > > packet (an acknack for instance). Obviously this results in retries= of > > > > those messages failing to send the correct data and my application > > > > gets stuck. > > > > > > > > I have ensured that the refcount is not reaching 0. Each new mbuf > > > > immediately has the refcnt incremented by 1. I was concerned that > > > > retries might need the refcnt bumped again, but if I bump the refco= unt > > > > every time I resend a specific mbuf to the NIC, the refcounts just > > > > keep getting higher. So it looks like re-bumping it on a resend is = not > > > > necessary. > > > > > > > > I have ruled out other possible explanations. The mbufs are being > > > > reused by rte_pktmbuf_alloc. I even tried playing with the EAL > > > > settings related to the number of mbuf descriptors and saw my chang= es > > > > directly correlate with how long it takes this problem to occur. Ho= w > > > > do I really prevent the driver from reusing packets that I still mi= ght > > > > need to resend? > > > > > > > > Thanks in advance, > > > > -Alan > > > > > > Which driver, could be a driver bug. > > > > > > Also, you should be able to trace mbuf functions, either with rte_tra= ce > > > or by other facility.