From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 745A3A0543; Tue, 7 Jun 2022 10:02:46 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1B7C24021D; Tue, 7 Jun 2022 10:02:46 +0200 (CEST) Received: from shelob.oktetlabs.ru (shelob.oktetlabs.ru [91.220.146.113]) by mails.dpdk.org (Postfix) with ESMTP id 1CDF340156 for ; Tue, 7 Jun 2022 10:02:45 +0200 (CEST) Received: from [192.168.1.40] (unknown [188.170.81.145]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by shelob.oktetlabs.ru (Postfix) with ESMTPSA id CF6D62C7; Tue, 7 Jun 2022 11:02:43 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 shelob.oktetlabs.ru CF6D62C7 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=oktetlabs.ru; s=default; t=1654588964; bh=/drpVnT9uVEOsglqQg+fP5vByqBsMSAsEtlAMG5VlgM=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=DQbzDtjNjpQOeiA8PdFSgbk+qcAp68P43221tM1uJbeQDriIAJw8CkHXKMybja9iH jFYrYv8ANqokXFA7qmNpe5D9fnxw0WyNhs97xCWHFLFWW/2sJabXJAp+gXUm7GoHes ZRlys5/3nmDJB6bcQJU8jnaieh3tOubz8tv7lc6I= Message-ID: <420fe31a-6fd3-b10e-d13e-eb27b40af3f8@oktetlabs.ru> Date: Tue, 7 Jun 2022 11:02:43 +0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.9.0 Subject: Re: [PATCH v4 3/7] ethdev: introduce Rx queue based fill threshold Content-Language: en-US To: Thomas Monjalon , Spike Du Cc: Matan Azrad , Slava Ovsiienko , Ori Kam , Wenzhuo Lu , Beilei Xing , Bernard Iremonger , Ray Kinsella , dev@dpdk.org, "stephen@networkplumber.org" , "mb@smartsharesystems.com" , Raslan Darawsheh , ferruh.yigit@amd.com References: <20220524152041.737154-1-spiked@nvidia.com> <7144473.aeNJFYEL58@thomas> From: Andrew Rybchenko In-Reply-To: <7144473.aeNJFYEL58@thomas> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 6/7/22 00:30, Thomas Monjalon wrote: > It seems we share a common understanding > and we need to agree on a good wording > for the most meaningful API. > Questions inline below: > > 06/06/2022 19:15, Andrew Rybchenko: >> On 6/6/22 16:16, Spike Du wrote: >>> Hi Andrew, >>> Please see below for "fill threshold" concept, I'm ok with other comments about code. >>> >>> Regards, >>> Spike. >>> >>> >>> From: Andrew Rybchenko >>>> On 6/3/22 15:48, Spike Du wrote: >>>>> Fill threshold describes the fullness of a Rx queue. If the Rx queue >>>>> fullness is above the threshold, the device will trigger the event >>>>> RTE_ETH_EVENT_RX_FILL_THRESH. >>>> >>>> Sorry, I'm not sure that I understand. As far as I know the process to add >>>> more Rx buffers to Rx queue is called 'refill' in many drivers. So fill level is a >>>> number (or percentage) of free buffers in an Rx queue. >>>> If so, fill threashold should be a minimum fill level and below the level we >>>> should generate an event. >>>> >>>> However reading the first paragraph of the descrition it looks like you mean >>>> oposite thing - a number (or percentage) of ready Rx buffers with received >>>> packets. >>>> >>>> I think that the term "fill threshold" is suggested by me, but I did it with mine >>>> understanding of the added feature. Now I'm confused. >>>> >>>> Moreover, I don't understand how "fill threshold" could be in terms of ready >>>> Rx buffers. HW simply don't really know when ready Rx buffers are >>>> processed by SW. So, HW can't say for sure how many ready Rx buffers are >>>> pending. It could be calculated as Rx queue size minus number of free Rx >>>> buffers, but it is imprecise. First of all not all Rx descriptors could be used. >>>> Second, HW ring size could differ queue size specified in SW. >>>> Queue size specified in SW could just limit maximum nubmer of free Rx >>>> buffers provided by the driver. >>>> >>> >>> Let me use other terms because "fill"/"refill" is also ambiguous to me. >>> In a RX ring, there are Rx buffers with received packets, you call it "ready Rx buffers", there is a RTE api rte_eth_rx_queue_count() to get the number, >>> It's also called "used descriptors" in the code. >>> Also there are Rx buffers provided by SW to allow HW "fill in" received packets, we can call it "usable Rx buffers" (here "usable" means usable for HW). >> >> May be it is better to stick to Rx descriptor status terminology? >> Available - Rx descriptor available to HW to put received packet to >> Done - Rx descriptor with received packet reported to Sw >> Unavailable - other (e.g. gap which cannot be used or just processed >> Done, but not refilled (made available to HW). >> >>> Let's define Rx queue "fullness": >>> Fullness = ready-Rx-buffers/Rxq-size >> >> i.e. number of DONE descriptors divided by RxQ size >> >>> On the opposite, we have "emptiness" >>> Emptiness = usable-Rx-buffers/Rxq-size >> >> i.e. number of AVAIL descriptors divided by RxQ size >> Note, that AVAIL != RxQ-size - DONE >> >> HW really knows number of available descriptors by its nature. >> It is a space between latest done and latest received on refill. >> >> HW does not know which descriptors are DONE, since some which >> are DONE before could be already processed by SW, but not yet >> made available again. >> >> >>> Here "fill threshold" describes "fullness", it's not "refill" described in you above words. Because in your words, "refill" is the opposite, it's filling "usable/free Rx buffers", or "emptiness". >>> >>> I can only briefly explain how mlx5 works to get LWM, because I'm not a Firmware guy. >>> Mlx5 Rx queue is basically RDMA queue. It has two indexes: producer index which increases when HW fills in packet, consumer index which increases when SW consumes the packet. >>> The queue size is known when it's created. The fullness is something like (producer_index - consumer_index) (I don't consider in wrap-around here). >>> So mlx5 has the way to get the fullness or emptiness in HW or FW. >>> Another detail is mlx5 uses the term "LWM"(limit watermark), which describes "emptiness". When usable-Rx-buffers is below LWM, we trigger an event. >>> But Thomas think "fullness" is easier to understand, so we use "fullness" in rte APIs and we'll translate it to LWM in mlx5 PMD. > > I may be wrong :) > >> HW simply does now know fullness and there can't generate any events >> based on it. It is a problem on Rx when there is now available >> descriptors. I.e. emptiness. > > So you think "empty_thresh" would be better? > Or "avail_thresh"? I'd go for "avail_thresh" since it is consistent with descriptor status API terminology. One moment I thought that there is a problem with rx_free_threshold, but finally realize that there is no problem, since it is in terms of free descriptors which could be made available (refilled). >>>>> Fill threshold is defined as a percentage of Rx queue size with valid >>>>> value of [0,99]. >>>>> Setting fill threshold to 0 means disable it, which is the default. >>>>> Add fill threshold configuration and query driver callbacks in eth_dev_ops. >>>>> Add command line options to support fill_thresh per-rxq configure. >>>>> - Command syntax: >>>>> set port rxq fill_thresh >>>>> >>>>> - Example commands: >>>>> To configure fill_thresh as 30% of rxq size on port 1 rxq 0: >>>>> testpmd> set port 1 rxq 0 fill_thresh 30 >>>>> >>>>> To disable fill_thresh on port 1 rxq 0: >>>>> testpmd> set port 1 rxq 0 fill_thresh 0 >>>>> >>>>> Signed-off-by: Spike Du > > >