From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 10A17A0542; Mon, 6 Jun 2022 23:30:50 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B273B4069C; Mon, 6 Jun 2022 23:30:49 +0200 (CEST) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by mails.dpdk.org (Postfix) with ESMTP id 53F0D4003F for ; Mon, 6 Jun 2022 23:30:48 +0200 (CEST) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id A600A5C0032; Mon, 6 Jun 2022 17:30:47 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 06 Jun 2022 17:30:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= cc:cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm2; t=1654551047; x= 1654637447; bh=5l4efxdMc5fTRXgtDxwyoF7lOaK2Bx/VEnjfuDw4NFk=; b=g zDbOzvAtsBr3MLoT6c3w4lA2a/0MI6rjhJbVWnMRk6HgqILCjWNcbMZZuSWeUe6O USRshGmYOSvOZDAIsY7qYXwjURbPe1IMA2saOb+VctOz0d78OC4Q4WKz48cx7iVp Vc/vGm407G+CYU2RP9ohwiJxyFxGeiE74MtMqHngYWD51bVVKPsDvQ7Q5CWuRn79 WoTbrdnp035UEmsn6f6EE7MJyFziWYOdv1bXq75xXPQoPLGyPLc8xzB1lTEtmGLo 7mQRkIGYimGPkWWsVqE1ny5qq9J09sVDxPST7GLEhbzmYBbrTdjqQrwD3fRLFPkN xHEWRh2bTXG2Yti1x2sOQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1654551047; x= 1654637447; bh=5l4efxdMc5fTRXgtDxwyoF7lOaK2Bx/VEnjfuDw4NFk=; b=l a3aDumycW8ZTMkAbNvNrMCwqfxFo4feZEvetPk7gAOVfn00QZf0cgte23AODv6X5 x5tsC/Pys3g7sYWHTmQX+R5vMDPUe8rXFI9VCj4nvfNk+XrC6jE3whOrOzicmtFi B4kAMG9ybtdQUS0nHMh13yJa+fjuh1HIVjk+Cio1d5TsPIX5rGskt5aGvIysSEfd FUIzIotzaoJAMX9bND/+ddcIOsIyQPUgzGf6CexfgfyIkrIor2pZuvGvGrVKB270 os+wPNIM49pltFPd1ocA9U8CFz7uHN+oAp1lNhLZ0U8qOoKvLoEVmLxgEiypPTb/ xHgMwuU2wbA8Ju0SmINlA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedruddtfedgjedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkjghfggfgtgesthfuredttddtvdenucfhrhhomhepvfhhohhm rghsucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenuc ggtffrrghtthgvrhhnpedtjeeiieefhedtfffgvdelteeufeefheeujefgueetfedttdei kefgkeduhedtgfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfh hrohhmpehthhhomhgrshesmhhonhhjrghlohhnrdhnvght X-ME-Proxy: Feedback-ID: i47234305:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 6 Jun 2022 17:30:45 -0400 (EDT) From: Thomas Monjalon To: Spike Du , Andrew Rybchenko Cc: Matan Azrad , Slava Ovsiienko , Ori Kam , Wenzhuo Lu , Beilei Xing , Bernard Iremonger , Ray Kinsella , dev@dpdk.org, "stephen@networkplumber.org" , "mb@smartsharesystems.com" , Raslan Darawsheh , ferruh.yigit@amd.com Subject: Re: [PATCH v4 3/7] ethdev: introduce Rx queue based fill threshold Date: Mon, 06 Jun 2022 23:30:41 +0200 Message-ID: <7144473.aeNJFYEL58@thomas> In-Reply-To: References: <20220524152041.737154-1-spiked@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org It seems we share a common understanding and we need to agree on a good wording for the most meaningful API. Questions inline below: 06/06/2022 19:15, Andrew Rybchenko: > On 6/6/22 16:16, Spike Du wrote: > > Hi Andrew, > > Please see below for "fill threshold" concept, I'm ok with other comments about code. > > > > Regards, > > Spike. > > > > > > From: Andrew Rybchenko > >> On 6/3/22 15:48, Spike Du wrote: > >>> Fill threshold describes the fullness of a Rx queue. If the Rx queue > >>> fullness is above the threshold, the device will trigger the event > >>> RTE_ETH_EVENT_RX_FILL_THRESH. > >> > >> Sorry, I'm not sure that I understand. As far as I know the process to add > >> more Rx buffers to Rx queue is called 'refill' in many drivers. So fill level is a > >> number (or percentage) of free buffers in an Rx queue. > >> If so, fill threashold should be a minimum fill level and below the level we > >> should generate an event. > >> > >> However reading the first paragraph of the descrition it looks like you mean > >> oposite thing - a number (or percentage) of ready Rx buffers with received > >> packets. > >> > >> I think that the term "fill threshold" is suggested by me, but I did it with mine > >> understanding of the added feature. Now I'm confused. > >> > >> Moreover, I don't understand how "fill threshold" could be in terms of ready > >> Rx buffers. HW simply don't really know when ready Rx buffers are > >> processed by SW. So, HW can't say for sure how many ready Rx buffers are > >> pending. It could be calculated as Rx queue size minus number of free Rx > >> buffers, but it is imprecise. First of all not all Rx descriptors could be used. > >> Second, HW ring size could differ queue size specified in SW. > >> Queue size specified in SW could just limit maximum nubmer of free Rx > >> buffers provided by the driver. > >> > > > > Let me use other terms because "fill"/"refill" is also ambiguous to me. > > In a RX ring, there are Rx buffers with received packets, you call it "ready Rx buffers", there is a RTE api rte_eth_rx_queue_count() to get the number, > > It's also called "used descriptors" in the code. > > Also there are Rx buffers provided by SW to allow HW "fill in" received packets, we can call it "usable Rx buffers" (here "usable" means usable for HW). > > May be it is better to stick to Rx descriptor status terminology? > Available - Rx descriptor available to HW to put received packet to > Done - Rx descriptor with received packet reported to Sw > Unavailable - other (e.g. gap which cannot be used or just processed > Done, but not refilled (made available to HW). > > > Let's define Rx queue "fullness": > > Fullness = ready-Rx-buffers/Rxq-size > > i.e. number of DONE descriptors divided by RxQ size > > > On the opposite, we have "emptiness" > > Emptiness = usable-Rx-buffers/Rxq-size > > i.e. number of AVAIL descriptors divided by RxQ size > Note, that AVAIL != RxQ-size - DONE > > HW really knows number of available descriptors by its nature. > It is a space between latest done and latest received on refill. > > HW does not know which descriptors are DONE, since some which > are DONE before could be already processed by SW, but not yet > made available again. > > > > Here "fill threshold" describes "fullness", it's not "refill" described in you above words. Because in your words, "refill" is the opposite, it's filling "usable/free Rx buffers", or "emptiness". > > > > I can only briefly explain how mlx5 works to get LWM, because I'm not a Firmware guy. > > Mlx5 Rx queue is basically RDMA queue. It has two indexes: producer index which increases when HW fills in packet, consumer index which increases when SW consumes the packet. > > The queue size is known when it's created. The fullness is something like (producer_index - consumer_index) (I don't consider in wrap-around here). > > So mlx5 has the way to get the fullness or emptiness in HW or FW. > > Another detail is mlx5 uses the term "LWM"(limit watermark), which describes "emptiness". When usable-Rx-buffers is below LWM, we trigger an event. > > But Thomas think "fullness" is easier to understand, so we use "fullness" in rte APIs and we'll translate it to LWM in mlx5 PMD. I may be wrong :) > HW simply does now know fullness and there can't generate any events > based on it. It is a problem on Rx when there is now available > descriptors. I.e. emptiness. So you think "empty_thresh" would be better? Or "avail_thresh"? > >>> Fill threshold is defined as a percentage of Rx queue size with valid > >>> value of [0,99]. > >>> Setting fill threshold to 0 means disable it, which is the default. > >>> Add fill threshold configuration and query driver callbacks in eth_dev_ops. > >>> Add command line options to support fill_thresh per-rxq configure. > >>> - Command syntax: > >>> set port rxq fill_thresh > >>> > >>> - Example commands: > >>> To configure fill_thresh as 30% of rxq size on port 1 rxq 0: > >>> testpmd> set port 1 rxq 0 fill_thresh 30 > >>> > >>> To disable fill_thresh on port 1 rxq 0: > >>> testpmd> set port 1 rxq 0 fill_thresh 0 > >>> > >>> Signed-off-by: Spike Du