Re: [PATCH v4 3/7] ethdev: introduce Rx queue based fill threshold

[Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>]

On 6/6/22 16:16, Spike Du wrote:
> Hi Andrew,
> 	Please see below for "fill threshold" concept, I'm ok with other comments about code.
> 
> Regards,
> Spike.
> 
> 
>> -----Original Message-----
>> From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
>> Sent: Saturday, June 4, 2022 8:46 PM
>> To: Spike Du <spiked@nvidia.com>; Matan Azrad <matan@nvidia.com>;
>> Slava Ovsiienko <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>;
>> NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
>> Wenzhuo Lu <wenzhuo.lu@intel.com>; Beilei Xing <beilei.xing@intel.com>;
>> Bernard Iremonger <bernard.iremonger@intel.com>; Ray Kinsella
>> <mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>
>> Cc: stephen@networkplumber.org; mb@smartsharesystems.com;
>> dev@dpdk.org; Raslan Darawsheh <rasland@nvidia.com>
>> Subject: Re: [PATCH v4 3/7] ethdev: introduce Rx queue based fill threshold
>>
>> External email: Use caution opening links or attachments
>>
>>
>> On 6/3/22 15:48, Spike Du wrote:
>>> Fill threshold describes the fullness of a Rx queue. If the Rx queue
>>> fullness is above the threshold, the device will trigger the event
>>> RTE_ETH_EVENT_RX_FILL_THRESH.
>>
>> Sorry, I'm not sure that I understand. As far as I know the process to add
>> more Rx buffers to Rx queue is called 'refill' in many drivers. So fill level is a
>> number (or percentage) of free buffers in an Rx queue.
>> If so, fill threashold should be a minimum fill level and below the level we
>> should generate an event.
>>
>> However reading the first paragraph of the descrition it looks like you mean
>> oposite thing - a number (or percentage) of ready Rx buffers with received
>> packets.
>>
>> I think that the term "fill threshold" is suggested by me, but I did it with mine
>> understanding of the added feature. Now I'm confused.
>>
>> Moreover, I don't understand how "fill threshold" could be in terms of ready
>> Rx buffers. HW simply don't really know when ready Rx buffers are
>> processed by SW. So, HW can't say for sure how many ready Rx buffers are
>> pending. It could be calculated as Rx queue size minus number of free Rx
>> buffers, but it is imprecise. First of all not all Rx descriptors could be used.
>> Second, HW ring size could differ queue size specified in SW.
>> Queue size specified in SW could just limit maximum nubmer of free Rx
>> buffers provided by the driver.
>>
> 
> Let me use other terms because "fill"/"refill" is also ambiguous to me.
> In a RX ring, there are Rx buffers with received packets, you call it "ready Rx buffers", there is a RTE api rte_eth_rx_queue_count() to get the number,
> It's also called "used descriptors" in the code.
> Also there are Rx buffers provided by SW to allow HW "fill in" received packets, we can call it "usable Rx buffers" (here "usable" means usable for HW).

May be it is better to stick to Rx descriptor status terminology?
Available - Rx descriptor available to HW to put received packet to
Done - Rx descriptor with received packet reported to Sw
Unavailable - other (e.g. gap which cannot be used or just processed
Done, but not refilled (made available to HW).

> Let's define Rx queue "fullness":
> 	Fullness = ready-Rx-buffers/Rxq-size

i.e. number of DONE descriptors divided by RxQ size

> On the opposite, we have "emptiness"
> 	Emptiness = usable-Rx-buffers/Rxq-size

i.e. number of AVAIL descriptors divided by RxQ size
Note, that AVAIL != RxQ-size - DONE

HW really knows number of available descriptors by its nature.
It is a space between latest done and latest received on refill.

HW does not know which descriptors are DONE, since some which
are DONE before could be already processed by SW, but not yet
made available again.


> Here "fill threshold" describes "fullness", it's not "refill" described in you above words. Because in your words, "refill" is the opposite, it's filling "usable/free Rx buffers", or "emptiness".
> 
> I can only briefly explain how mlx5 works to get LWM, because I'm not a Firmware guy.
> Mlx5 Rx queue is basically RDMA queue. It has two indexes: producer index which increases when HW fills in packet, consumer index which increases when SW consumes the packet.
> The queue size is known when it's created. The fullness is something like (producer_index - consumer_index) (I don't consider in wrap-around here).
> So mlx5 has the way to get the fullness or emptiness in HW or FW.
> Another detail is mlx5 uses the term "LWM"(limit watermark), which describes "emptiness". When usable-Rx-buffers is below LWM, we trigger an event.
> But Thomas think "fullness" is easier to understand, so we use "fullness" in rte APIs and we'll translate it to LWM in mlx5 PMD.

HW simply does now know fullness and there can't generate any events
based on it. It is a problem on Rx when there is now available
descriptors.  I.e. emptiness.

>>> Fill threshold is defined as a percentage of Rx queue size with valid
>>> value of [0,99].
>>> Setting fill threshold to 0 means disable it, which is the default.
>>> Add fill threshold configuration and query driver callbacks in eth_dev_ops.
>>> Add command line options to support fill_thresh per-rxq configure.
>>> - Command syntax:
>>>     set port <port_id> rxq <rxq_id> fill_thresh <fill_thresh_num>
>>>
>>> - Example commands:
>>> To configure fill_thresh as 30% of rxq size on port 1 rxq 0:
>>> testpmd> set port 1 rxq 0 fill_thresh 30
>>>
>>> To disable fill_thresh on port 1 rxq 0:
>>> testpmd> set port 1 rxq 0 fill_thresh 0
>>>
>>> Signed-off-by: Spike Du <spiked@nvidia.com>

[snip]