From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AA573A0542; Mon, 6 Jun 2022 19:15:41 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 50048406B4; Mon, 6 Jun 2022 19:15:41 +0200 (CEST) Received: from shelob.oktetlabs.ru (shelob.oktetlabs.ru [91.220.146.113]) by mails.dpdk.org (Postfix) with ESMTP id C545B4069C for ; Mon, 6 Jun 2022 19:15:39 +0200 (CEST) Received: from [192.168.1.40] (unknown [188.170.81.145]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by shelob.oktetlabs.ru (Postfix) with ESMTPSA id A3C8028D; Mon, 6 Jun 2022 20:15:38 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 shelob.oktetlabs.ru A3C8028D DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=oktetlabs.ru; s=default; t=1654535739; bh=9e1Uf1a4xyBQj2yoAdX/BOUGj30v3GK6ypD2kW9EmYE=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=Jl5fp6hR4hjnv/mwcavEKvCocDEEysOIrJVPAZa9CPlESwMzgZW+ahV4DHwbn3pr6 qIKcZUMe57CrgA39UVNf8hycyXLoaP3sRmbN3mH4951TvGCbu8kpPS+amE1GL7yGZj F3MZ3+v+wZbiucbaspKiPuilF3K8LfB/CMu/5dBQ= Message-ID: Date: Mon, 6 Jun 2022 20:15:37 +0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.9.0 Subject: Re: [PATCH v4 3/7] ethdev: introduce Rx queue based fill threshold Content-Language: en-US To: Spike Du , Matan Azrad , Slava Ovsiienko , Ori Kam , "NBU-Contact-Thomas Monjalon (EXTERNAL)" , Wenzhuo Lu , Beilei Xing , Bernard Iremonger , Ray Kinsella , Neil Horman Cc: "stephen@networkplumber.org" , "mb@smartsharesystems.com" , "dev@dpdk.org" , Raslan Darawsheh References: <20220524152041.737154-1-spiked@nvidia.com> <20220603124821.1148119-1-spiked@nvidia.com> <20220603124821.1148119-4-spiked@nvidia.com> From: Andrew Rybchenko In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 6/6/22 16:16, Spike Du wrote: > Hi Andrew, > Please see below for "fill threshold" concept, I'm ok with other comments about code. > > Regards, > Spike. > > >> -----Original Message----- >> From: Andrew Rybchenko >> Sent: Saturday, June 4, 2022 8:46 PM >> To: Spike Du ; Matan Azrad ; >> Slava Ovsiienko ; Ori Kam ; >> NBU-Contact-Thomas Monjalon (EXTERNAL) ; >> Wenzhuo Lu ; Beilei Xing ; >> Bernard Iremonger ; Ray Kinsella >> ; Neil Horman >> Cc: stephen@networkplumber.org; mb@smartsharesystems.com; >> dev@dpdk.org; Raslan Darawsheh >> Subject: Re: [PATCH v4 3/7] ethdev: introduce Rx queue based fill threshold >> >> External email: Use caution opening links or attachments >> >> >> On 6/3/22 15:48, Spike Du wrote: >>> Fill threshold describes the fullness of a Rx queue. If the Rx queue >>> fullness is above the threshold, the device will trigger the event >>> RTE_ETH_EVENT_RX_FILL_THRESH. >> >> Sorry, I'm not sure that I understand. As far as I know the process to add >> more Rx buffers to Rx queue is called 'refill' in many drivers. So fill level is a >> number (or percentage) of free buffers in an Rx queue. >> If so, fill threashold should be a minimum fill level and below the level we >> should generate an event. >> >> However reading the first paragraph of the descrition it looks like you mean >> oposite thing - a number (or percentage) of ready Rx buffers with received >> packets. >> >> I think that the term "fill threshold" is suggested by me, but I did it with mine >> understanding of the added feature. Now I'm confused. >> >> Moreover, I don't understand how "fill threshold" could be in terms of ready >> Rx buffers. HW simply don't really know when ready Rx buffers are >> processed by SW. So, HW can't say for sure how many ready Rx buffers are >> pending. It could be calculated as Rx queue size minus number of free Rx >> buffers, but it is imprecise. First of all not all Rx descriptors could be used. >> Second, HW ring size could differ queue size specified in SW. >> Queue size specified in SW could just limit maximum nubmer of free Rx >> buffers provided by the driver. >> > > Let me use other terms because "fill"/"refill" is also ambiguous to me. > In a RX ring, there are Rx buffers with received packets, you call it "ready Rx buffers", there is a RTE api rte_eth_rx_queue_count() to get the number, > It's also called "used descriptors" in the code. > Also there are Rx buffers provided by SW to allow HW "fill in" received packets, we can call it "usable Rx buffers" (here "usable" means usable for HW). May be it is better to stick to Rx descriptor status terminology? Available - Rx descriptor available to HW to put received packet to Done - Rx descriptor with received packet reported to Sw Unavailable - other (e.g. gap which cannot be used or just processed Done, but not refilled (made available to HW). > Let's define Rx queue "fullness": > Fullness = ready-Rx-buffers/Rxq-size i.e. number of DONE descriptors divided by RxQ size > On the opposite, we have "emptiness" > Emptiness = usable-Rx-buffers/Rxq-size i.e. number of AVAIL descriptors divided by RxQ size Note, that AVAIL != RxQ-size - DONE HW really knows number of available descriptors by its nature. It is a space between latest done and latest received on refill. HW does not know which descriptors are DONE, since some which are DONE before could be already processed by SW, but not yet made available again. > Here "fill threshold" describes "fullness", it's not "refill" described in you above words. Because in your words, "refill" is the opposite, it's filling "usable/free Rx buffers", or "emptiness". > > I can only briefly explain how mlx5 works to get LWM, because I'm not a Firmware guy. > Mlx5 Rx queue is basically RDMA queue. It has two indexes: producer index which increases when HW fills in packet, consumer index which increases when SW consumes the packet. > The queue size is known when it's created. The fullness is something like (producer_index - consumer_index) (I don't consider in wrap-around here). > So mlx5 has the way to get the fullness or emptiness in HW or FW. > Another detail is mlx5 uses the term "LWM"(limit watermark), which describes "emptiness". When usable-Rx-buffers is below LWM, we trigger an event. > But Thomas think "fullness" is easier to understand, so we use "fullness" in rte APIs and we'll translate it to LWM in mlx5 PMD. HW simply does now know fullness and there can't generate any events based on it. It is a problem on Rx when there is now available descriptors. I.e. emptiness. >>> Fill threshold is defined as a percentage of Rx queue size with valid >>> value of [0,99]. >>> Setting fill threshold to 0 means disable it, which is the default. >>> Add fill threshold configuration and query driver callbacks in eth_dev_ops. >>> Add command line options to support fill_thresh per-rxq configure. >>> - Command syntax: >>> set port rxq fill_thresh >>> >>> - Example commands: >>> To configure fill_thresh as 30% of rxq size on port 1 rxq 0: >>> testpmd> set port 1 rxq 0 fill_thresh 30 >>> >>> To disable fill_thresh on port 1 rxq 0: >>> testpmd> set port 1 rxq 0 fill_thresh 0 >>> >>> Signed-off-by: Spike Du [snip]