From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 11E2145CDD for ; Sun, 10 Nov 2024 18:31:51 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0752E40151; Sun, 10 Nov 2024 18:31:50 +0100 (CET) Received: from mail-oo1-f44.google.com (mail-oo1-f44.google.com [209.85.161.44]) by mails.dpdk.org (Postfix) with ESMTP id C8A44400EF for ; Sun, 10 Nov 2024 18:31:48 +0100 (CET) Received: by mail-oo1-f44.google.com with SMTP id 006d021491bc7-5ee55fa4b31so1391615eaf.0 for ; Sun, 10 Nov 2024 09:31:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1731259907; x=1731864707; darn=dpdk.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=bvJTsLPq2Kn8vQAMd4MNWBiaOE7KOEXb9MHI98iXf2Q=; b=P9RKvyI8fzS+iTl5SuKOK96LpVGDVSjWDkg+dxwCYrKkL0Yiygvo9QD5HNSlob2zWP 6/8F5OiGGHSBAqSgVdFgksfWQgWG5KHXBs/pn9jDbKY0m9DoNRDAhyHQf+pLacdQofh+ 2fMvPLHcA+cpXE88P0GPka7U7dgXvax0nJs563rtLtYdajmsv/gcuWOJTnyPZFPxFdef F8MSiwsnZtOmMop6KwpHKIVigo0H7O65uvykLmYzBxQJRHmG8x0OM89js7hOAKnk7+Tg NPoMuZrbxOPOblN23McmQ4j3M44DWYiZb895XvLDM+LQWpWJfk6ZFmDYjZSVnSlmGWt5 utYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731259907; x=1731864707; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bvJTsLPq2Kn8vQAMd4MNWBiaOE7KOEXb9MHI98iXf2Q=; b=LpnkW3dD5TIdpybTX2UD7/XoGU4xwr2CvvlgZjFDHjlxc81ISSqCt1kNyB3eUGQy8I BO5PPYrdluJKiY0dLbjSSnLFy/cJNYrICGhrLsOQ/zcn5Vb3sTirkwiTq/kijHsA3UBp 0NhsL6+FHzBppXQJTqfcIUxsTXoZXlH1OmrRWRSw27SXnCxjSJiCstnYIAFl46Sn+ODo GS+FOIUydoVb4WoHEALUzaiV54ayVJZL5kWTxw9ahHol+8r+XfFSgKWB6hHWtzI1ri4O OzLrTp86l2wa7768EdUarWb4EG1TMLlZUrU36k91bNzIUpALsTpObgV1h1G+Y/gEFVEp 3qLg== X-Gm-Message-State: AOJu0YzL1KXyfGjSkxOfcg9IojJgA2wEQFKkb2o5VBv7zoLuzjY6q3NS h1zGagLW+kyjhE3660vNzvYiDqzh4O+yDexkTuRuG5ISAcdV6j+Zv7YEZaq8d7nzqxa5m75XMR/ VgmX+dFeOLwygDHFhoh4amOwOKD8= X-Google-Smtp-Source: AGHT+IEn4fBRcTTGjenlNubSsO59zeM0dCwJbU8F7eDg1MDVzaDPRCfhBOEvwlailvXhW3+u+RB7OM69k6QKaehdxLY= X-Received: by 2002:a05:6871:4b06:b0:250:70bb:1a97 with SMTP id 586e51a60fabf-295600e3b4amr8710063fac.24.1731259907625; Sun, 10 Nov 2024 09:31:47 -0800 (PST) MIME-Version: 1.0 References: <20241110091245.677a3983@hermes.local> In-Reply-To: <20241110091245.677a3983@hermes.local> From: Alan Beadle Date: Sun, 10 Nov 2024 12:31:36 -0500 Message-ID: Subject: Re: mbufs getting reused despite nonzero refcnt To: Stephen Hemminger Cc: users@dpdk.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org I'm using the vfio-pci module with Intel X550-T2 NICs. I believe this means it will use the ixgbe driver? To be honest, I am a bit confused about the use of drivers in DPDK. I am using the first setup that I got to work and send/receive packets. Additional tips would be greatly appreciated. After loading the vfio-pci module I run dpdk-devbind.py --bind vfio-pci 65:00.1 and then I just use the standard DPDK API calls in my app. I was meaning to revisit this once my app was more complete. On Sun, Nov 10, 2024 at 12:12=E2=80=AFPM Stephen Hemminger wrote: > > On Sun, 10 Nov 2024 11:23:29 -0500 > Alan Beadle wrote: > > > Hi everyone, > > > > I am using DPDK to send two-way traffic between a pair of machines. My > > application has local readers, remote acknowledgments, as well as > > automatic retries when a packet is lost. For these reasons I am using > > rte_mbuf_refcnt_update() to prevent the NIC from freeing the packet > > and recycling the mbuf before my local readers are done and the remote > > reader has acknowledged the message. I was advised to do this in an > > earlier thread on this mailing list. > > > > However, this does not seem to be working. After running my app for > > awhile and exchanging about 1000 messages in this way, my queue of > > unacknowledged mbufs is getting corrupted. The mbufs attached to my > > queue seem to contain data for newer messages than what is supposed to > > be in them, and in some cases contains a totally different type of > > packet (an acknack for instance). Obviously this results in retries of > > those messages failing to send the correct data and my application > > gets stuck. > > > > I have ensured that the refcount is not reaching 0. Each new mbuf > > immediately has the refcnt incremented by 1. I was concerned that > > retries might need the refcnt bumped again, but if I bump the refcount > > every time I resend a specific mbuf to the NIC, the refcounts just > > keep getting higher. So it looks like re-bumping it on a resend is not > > necessary. > > > > I have ruled out other possible explanations. The mbufs are being > > reused by rte_pktmbuf_alloc. I even tried playing with the EAL > > settings related to the number of mbuf descriptors and saw my changes > > directly correlate with how long it takes this problem to occur. How > > do I really prevent the driver from reusing packets that I still might > > need to resend? > > > > Thanks in advance, > > -Alan > > Which driver, could be a driver bug. > > Also, you should be able to trace mbuf functions, either with rte_trace > or by other facility.