From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7E22BA0032 for ; Tue, 15 Mar 2022 21:15:39 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0A65340395; Tue, 15 Mar 2022 21:15:39 +0100 (CET) Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by mails.dpdk.org (Postfix) with ESMTP id DD7704014F for ; Tue, 15 Mar 2022 21:15:37 +0100 (CET) Received: by mail-pl1-f182.google.com with SMTP id d18so72966plr.6 for ; Tue, 15 Mar 2022 13:15:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=w8+RqFLrV1QMTbuH+VclEnr7+BVugsGJ01KSQ9jlxkw=; b=nMskVLnkcsPsif4/v6LdpzbpzszuhCIqSbkSk5BrHZAbiT3AXS2ipUr57tekat+pja xnSeL6IJPXbS6wWzxJa5eXKETRn9n/7zMRxXx8IB/x2t1R6kJQUwdtj0qeI1+ZXiknAX nwZgtbqVxaHo9tyOfHQ64F68NuFn3l6cAIcbY2Kua0L3hWx2CDLloBb/4KJQ91tj4kxu airgvxPbO1y2yuA06xQKj2gjpcMA9FpPkhUBdR2Kd4U1xoqDqwFbxAUhuGcNapMqm/s7 yQn+c4Gyute5ZJ4KDhDTNctJaCgEDr6Fz7uOXVK2eA4ECwDvaqfpovRG/i6q1k2WXsvN gI+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=w8+RqFLrV1QMTbuH+VclEnr7+BVugsGJ01KSQ9jlxkw=; b=vvCA/r++++8Z6Mtm7IW411h5ZZOHHOE6m5PaXJFLSzN+tqJq/vn7EnKt4Lumh2azv7 b9ot7BiMmD49EjNApk8hfy7nlQoi0ya9e5Wy6d0CJbV6tRrSRfN322qrSSSOJVbBC8Bx Pxy7T56cRbTICSVbRkkZ0FqWbXWyH7RoFfimEvxOKf+iwukzrAfcRVbpxPI0NkVU/Jjm LEwT9fzqOnJ//TIGrcHcNEJriAwdYAf6DV9NbY0jXyRC0gPBSkryjk3gpr+nAZpP/G96 7FWmZpTFY3NB5cR9S1iHFxt5G1ivZFreKVS5+y5UEJ6bdA2ySOVflAMUQKIs8ExgjsAR eV7A== X-Gm-Message-State: AOAM533xYmF5Lxz4PTmhhZUbPpw/mCCuZALZucF+VbC+WAaKNpXCoTOw vcIYnJJeMh9sSpcY/bVmCODGVw== X-Google-Smtp-Source: ABdhPJzQhrM4Jz7CvF1DJqLSnMN6eOsb+73A3ln8tP/GWTEfBqoxMrI+zdqNDqs7TiibkSGLsamdWQ== X-Received: by 2002:a17:903:2c5:b0:14f:4a29:1f64 with SMTP id s5-20020a17090302c500b0014f4a291f64mr29993384plk.90.1647375336777; Tue, 15 Mar 2022 13:15:36 -0700 (PDT) Received: from hermes.local (204-195-112-199.wavecable.com. [204.195.112.199]) by smtp.gmail.com with ESMTPSA id c34-20020a630d22000000b0034cb89e4695sm91033pgl.28.2022.03.15.13.15.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Mar 2022 13:15:36 -0700 (PDT) Date: Tue, 15 Mar 2022 13:15:34 -0700 From: Stephen Hemminger To: fwefew 4t4tg <7532yahoo@gmail.com> Cc: users@dpdk.org Subject: Re: (AWS ENA NIC) queued v. out packets reported by any of the DPDK stats functions Message-ID: <20220315131534.1e717ff2@hermes.local> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Tue, 15 Mar 2022 15:28:55 -0400 fwefew 4t4tg <7532yahoo@gmail.com> wrote: > On DPDK head commit (917229) I have a simple program creating two TXQs each > sending 15000 IPV4 UDP packets. > > As far as I can see there are no errors anymore. I have looked. I burst TX > them in blocks of 15; rte_eth_tx_burst always returns 15. > > The lcores are launched with rte_eal_mp_remote_launch, and main() blocks > until rte_eal_mp_wait_lcore() returns. > > However, at any time after rte_eal_mp_wait_lcore returns showing the number > of out packets as per rte_eth_stats_get or xstats per rte_eth_xstats_get I > see two things: > > - the reported number of queued packets per TXQ is always 15000 > - the number of out packets or "good packets" is close to 15000*2 but never > 30000 > > It appears like the TXQs still have work in-queue even after the lcore's > thread returns and rte_eal_mp_wait_lcore returns. I've tried to > rte_eth_tx_done_cleanup and/or rte_eth_dev_tx_queue_stop before I get the > stats. That doesn't seem to help. > > The TXQ config is default except for checksum offloads: > > 000000.336378246 DEBUG reinvent_dpdk_initaws.cpp:1341 TXQ conf: > {"tx_thresh": { "pthresh": 0, "hthresh": 0, "wthresh": 0 }, "tx_rs_thresh": > 0, "tx_free_thresh": 0, "tx_deferred_start": 0, "tx_offloads": 00008006, } > > Is this expected behavior? I've look through code; I haven't seen clear > signs of a TXQ oriented lcore flushed or waiting for the output queue to be > actually written onto the wire before exiting. > No part of lcore_wait is related directly to TxQ. The TxQ runs asynchronously. If you have to check if packet was sent, the you would need to use rte_eth_tx_descriptor_status(). But not all hardware supports this API; AWS ENA does not.