Re: [PATCH v4 3/7] ethdev: introduce Rx queue based fill threshold

[Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>]

On 6/7/22 00:30, Thomas Monjalon wrote:
> It seems we share a common understanding
> and we need to agree on a good wording
> for the most meaningful API.
> Questions inline below:
> 
> 06/06/2022 19:15, Andrew Rybchenko:
>> On 6/6/22 16:16, Spike Du wrote:
>>> Hi Andrew,
>>> 	Please see below for "fill threshold" concept, I'm ok with other comments about code.
>>>
>>> Regards,
>>> Spike.
>>>
>>>
>>> From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
>>>> On 6/3/22 15:48, Spike Du wrote:
>>>>> Fill threshold describes the fullness of a Rx queue. If the Rx queue
>>>>> fullness is above the threshold, the device will trigger the event
>>>>> RTE_ETH_EVENT_RX_FILL_THRESH.
>>>>
>>>> Sorry, I'm not sure that I understand. As far as I know the process to add
>>>> more Rx buffers to Rx queue is called 'refill' in many drivers. So fill level is a
>>>> number (or percentage) of free buffers in an Rx queue.
>>>> If so, fill threashold should be a minimum fill level and below the level we
>>>> should generate an event.
>>>>
>>>> However reading the first paragraph of the descrition it looks like you mean
>>>> oposite thing - a number (or percentage) of ready Rx buffers with received
>>>> packets.
>>>>
>>>> I think that the term "fill threshold" is suggested by me, but I did it with mine
>>>> understanding of the added feature. Now I'm confused.
>>>>
>>>> Moreover, I don't understand how "fill threshold" could be in terms of ready
>>>> Rx buffers. HW simply don't really know when ready Rx buffers are
>>>> processed by SW. So, HW can't say for sure how many ready Rx buffers are
>>>> pending. It could be calculated as Rx queue size minus number of free Rx
>>>> buffers, but it is imprecise. First of all not all Rx descriptors could be used.
>>>> Second, HW ring size could differ queue size specified in SW.
>>>> Queue size specified in SW could just limit maximum nubmer of free Rx
>>>> buffers provided by the driver.
>>>>
>>>
>>> Let me use other terms because "fill"/"refill" is also ambiguous to me.
>>> In a RX ring, there are Rx buffers with received packets, you call it "ready Rx buffers", there is a RTE api rte_eth_rx_queue_count() to get the number,
>>> It's also called "used descriptors" in the code.
>>> Also there are Rx buffers provided by SW to allow HW "fill in" received packets, we can call it "usable Rx buffers" (here "usable" means usable for HW).
>>
>> May be it is better to stick to Rx descriptor status terminology?
>> Available - Rx descriptor available to HW to put received packet to
>> Done - Rx descriptor with received packet reported to Sw
>> Unavailable - other (e.g. gap which cannot be used or just processed
>> Done, but not refilled (made available to HW).
>>
>>> Let's define Rx queue "fullness":
>>> 	Fullness = ready-Rx-buffers/Rxq-size
>>
>> i.e. number of DONE descriptors divided by RxQ size
>>
>>> On the opposite, we have "emptiness"
>>> 	Emptiness = usable-Rx-buffers/Rxq-size
>>
>> i.e. number of AVAIL descriptors divided by RxQ size
>> Note, that AVAIL != RxQ-size - DONE
>>
>> HW really knows number of available descriptors by its nature.
>> It is a space between latest done and latest received on refill.
>>
>> HW does not know which descriptors are DONE, since some which
>> are DONE before could be already processed by SW, but not yet
>> made available again.
>>
>>
>>> Here "fill threshold" describes "fullness", it's not "refill" described in you above words. Because in your words, "refill" is the opposite, it's filling "usable/free Rx buffers", or "emptiness".
>>>
>>> I can only briefly explain how mlx5 works to get LWM, because I'm not a Firmware guy.
>>> Mlx5 Rx queue is basically RDMA queue. It has two indexes: producer index which increases when HW fills in packet, consumer index which increases when SW consumes the packet.
>>> The queue size is known when it's created. The fullness is something like (producer_index - consumer_index) (I don't consider in wrap-around here).
>>> So mlx5 has the way to get the fullness or emptiness in HW or FW.
>>> Another detail is mlx5 uses the term "LWM"(limit watermark), which describes "emptiness". When usable-Rx-buffers is below LWM, we trigger an event.
>>> But Thomas think "fullness" is easier to understand, so we use "fullness" in rte APIs and we'll translate it to LWM in mlx5 PMD.
> 
> I may be wrong :)
> 
>> HW simply does now know fullness and there can't generate any events
>> based on it. It is a problem on Rx when there is now available
>> descriptors.  I.e. emptiness.
> 
> So you think "empty_thresh" would be better?
> Or "avail_thresh"?

I'd go for "avail_thresh" since it is consistent with descriptor status
API terminology.

One moment I thought that there is a problem with rx_free_threshold, but
finally realize that there is no problem, since it is in terms of free
descriptors which could be made available (refilled).

>>>>> Fill threshold is defined as a percentage of Rx queue size with valid
>>>>> value of [0,99].
>>>>> Setting fill threshold to 0 means disable it, which is the default.
>>>>> Add fill threshold configuration and query driver callbacks in eth_dev_ops.
>>>>> Add command line options to support fill_thresh per-rxq configure.
>>>>> - Command syntax:
>>>>>      set port <port_id> rxq <rxq_id> fill_thresh <fill_thresh_num>
>>>>>
>>>>> - Example commands:
>>>>> To configure fill_thresh as 30% of rxq size on port 1 rxq 0:
>>>>> testpmd> set port 1 rxq 0 fill_thresh 30
>>>>>
>>>>> To disable fill_thresh on port 1 rxq 0:
>>>>> testpmd> set port 1 rxq 0 fill_thresh 0
>>>>>
>>>>> Signed-off-by: Spike Du <spiked@nvidia.com>
> 
> 
>