From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E797445CF5 for ; Tue, 12 Nov 2024 14:02:20 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CE4D742D91; Tue, 12 Nov 2024 14:02:20 +0100 (CET) Received: from mail-oa1-f46.google.com (mail-oa1-f46.google.com [209.85.160.46]) by mails.dpdk.org (Postfix) with ESMTP id 32ECD427AC for ; Tue, 12 Nov 2024 14:02:19 +0100 (CET) Received: by mail-oa1-f46.google.com with SMTP id 586e51a60fabf-2890f273296so2494933fac.2 for ; Tue, 12 Nov 2024 05:02:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1731416538; x=1732021338; darn=dpdk.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=vG7ykJs+dAnaF1krwEsrx4PsALLtq0+bQGMHS5ajDxI=; b=j3l8yux6cwvLnq7pA6iIZ20s0vKucXaWFvecMFKTYhnpXCpJPwh7t6dJVpx+ONUjL2 C9G9RomT1Pglb0gK13wrhtGUTkc733e6hgKcgThkB6daC+FdidMLq347I17s1wJ7ZLpo y+iebK+VpjwyjYeexNIUhQFBkehKFx5ZQuwTIiWR2QBAIZnQu47nW/NKnuIEp8Fzx7xe QWcFFpAY3NHoUhdlrmTJD75YnvTi5UfYGWBzhS2FXqC0SSkGXHpVdiXPgz0SPHQbLjZi h1y8IJ0E3MAsXRozOryKY4seH+BnE9HrSwZEVcOlJecnQX0SiITowl5RpUJ4EDbZBggv PZ2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731416538; x=1732021338; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vG7ykJs+dAnaF1krwEsrx4PsALLtq0+bQGMHS5ajDxI=; b=tTAXDCDeo+4nm2Zu7zeqSf0lrSbhpnXjf1XVf/UP9bWjDb8/qC1s080ofDQ9SPQz5a +XQ+bQsf4vGluKDmMRCA1/vQHzrJk0XGL+c3WoHZgd8d9YQifFErQUjZRmC0EN/bfTw7 78wjKMNPInpxO4eJeBEJc15v2uDGwvk4/bHEFgB6fKircVAbhsk2M25g14+H5AkqnkMn VBx2SG2F0dc4z2SOPj9Bg2CjyzOIgbgDrZISs2kL8JQz9wrotnKw9XxhL4RAk2Q2g5R1 KXX6VFYossag5VDP0prUCXbgiTOR1pH8r1DWYfhk0Nw31XwWFQ67+LU1cx6aHuegqyta dU+w== X-Gm-Message-State: AOJu0Yxhnj3I1UJf7WoVh93ANf0CS7Atzg1kl6a6UNb0apqFgqWFDiRx NYk0waTFd+riWkYEng9w47G0hEkmVRxDY4CE8IniYzmUae0exyCFW++e3b2A9GDCMPwXY970q5X wbjM/6uoEmifuNk/ByObzO/PQE0s= X-Google-Smtp-Source: AGHT+IGOdfGIR57Llxc0m74G9+H/3kWtHCgD6NkD3N+Fp/MwJ9wOdrODM64LX2m9IMc12g/R2nMwlt6VCVcI+2vFOIs= X-Received: by 2002:a05:6870:44cf:b0:270:1884:9db1 with SMTP id 586e51a60fabf-29560001556mr13288746fac.7.1731416538155; Tue, 12 Nov 2024 05:02:18 -0800 (PST) MIME-Version: 1.0 References: <20241110091245.677a3983@hermes.local> In-Reply-To: From: Alan Beadle Date: Tue, 12 Nov 2024 08:02:06 -0500 Message-ID: Subject: Re: mbufs getting reused despite nonzero refcnt To: Stephen Hemminger Cc: users@dpdk.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Is there anything in the usage I described in my previous email which might explain this problem? Is there anything else wrong with what I'm doing driver-wise? On Sun, Nov 10, 2024 at 12:31=E2=80=AFPM Alan Beadle = wrote: > > I'm using the vfio-pci module with Intel X550-T2 NICs. I believe this > means it will use the ixgbe driver? To be honest, I am a bit confused > about the use of drivers in DPDK. I am using the first setup that I > got to work and send/receive packets. Additional tips would be greatly > appreciated. After loading the vfio-pci module I run dpdk-devbind.py > --bind vfio-pci 65:00.1 and then I just use the standard DPDK API > calls in my app. I was meaning to revisit this once my app was more > complete. > > On Sun, Nov 10, 2024 at 12:12=E2=80=AFPM Stephen Hemminger > wrote: > > > > On Sun, 10 Nov 2024 11:23:29 -0500 > > Alan Beadle wrote: > > > > > Hi everyone, > > > > > > I am using DPDK to send two-way traffic between a pair of machines. M= y > > > application has local readers, remote acknowledgments, as well as > > > automatic retries when a packet is lost. For these reasons I am using > > > rte_mbuf_refcnt_update() to prevent the NIC from freeing the packet > > > and recycling the mbuf before my local readers are done and the remot= e > > > reader has acknowledged the message. I was advised to do this in an > > > earlier thread on this mailing list. > > > > > > However, this does not seem to be working. After running my app for > > > awhile and exchanging about 1000 messages in this way, my queue of > > > unacknowledged mbufs is getting corrupted. The mbufs attached to my > > > queue seem to contain data for newer messages than what is supposed t= o > > > be in them, and in some cases contains a totally different type of > > > packet (an acknack for instance). Obviously this results in retries o= f > > > those messages failing to send the correct data and my application > > > gets stuck. > > > > > > I have ensured that the refcount is not reaching 0. Each new mbuf > > > immediately has the refcnt incremented by 1. I was concerned that > > > retries might need the refcnt bumped again, but if I bump the refcoun= t > > > every time I resend a specific mbuf to the NIC, the refcounts just > > > keep getting higher. So it looks like re-bumping it on a resend is no= t > > > necessary. > > > > > > I have ruled out other possible explanations. The mbufs are being > > > reused by rte_pktmbuf_alloc. I even tried playing with the EAL > > > settings related to the number of mbuf descriptors and saw my changes > > > directly correlate with how long it takes this problem to occur. How > > > do I really prevent the driver from reusing packets that I still migh= t > > > need to resend? > > > > > > Thanks in advance, > > > -Alan > > > > Which driver, could be a driver bug. > > > > Also, you should be able to trace mbuf functions, either with rte_trace > > or by other facility.