Re: [PATCH v3 0/7] introduce per-queue limit watermark and host shaper

[Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>]

On 5/25/22 17:16, Morten Brørup wrote:
>> From: Spike Du [mailto:spiked@nvidia.com]
>> Sent: Wednesday, 25 May 2022 15.59
>>
>>> From: Morten Brørup <mb@smartsharesystems.com>
>>> Sent: Wednesday, May 25, 2022 9:40 PM
>>>
>>>> From: Spike Du [mailto:spiked@nvidia.com]
>>>> Sent: Wednesday, 25 May 2022 15.15
>>>>
>>>>> From: Morten Brørup <mb@smartsharesystems.com>
>>>>> Sent: Wednesday, May 25, 2022 3:00 AM
>>>>>
>>>>>> From: Thomas Monjalon [mailto:thomas@monjalon.net]
>>>>>> Sent: Tuesday, 24 May 2022 17.59
>>>>>>
>>>>>> +Cc people involved in previous versions
>>>>>>
>>>>>> 24/05/2022 17:20, Spike Du:
>>>>>>> LWM(limit watermark) is per RX queue attribute, when RX queue
>>>>>> fullness reach the LWM limit, HW sends an event to dpdk
>>>> application.
>>>>>>> Host shaper can configure shaper rate and lwm-triggered for a
>>>> host
>>>>>> port.
>>>>>
>>>>>
>>>>>>> The shaper limits the rate of traffic from host port to wire
>>>> port.
>>>>>
>>>>>  From host to wire? It is RX, so you must mean from wire to host.
>>>>
>>>> The host shaper is quite private to Nvidia's BlueField 2 NIC. The
>> NIC
>>>> is inserted In a server which we call it host-system, and the NIC
>> has
>>>> an embedded Arm-system Which does the forwarding.
>>>> The traffic flows from host-system to wire like this:
>>>> Host-system generates traffic, send it to Arm-system, Arm sends it
>> to
>>>> physical/wire port.
>>>> So the RX happens between host-system and Arm-system, and the
>> traffic
>>>> is host to wire.
>>>> The shaper also works in a special way: you configure it on
>>>> Arm-system, but it takes effect On host-sysmem's TX side.
>>>>
>>>>>
>>>>>>> If lwm-triggered is enabled, a 100Mbps shaper is enabled
>>>>>> automatically when one of the host port's Rx queues receives
>> LWM
>>>> event.
>>>>>>>
>>>>>>> These two features can combine to control traffic from host
>> port
>>>> to
>>>>>> wire port.
>>>>>
>>>>> Again, you mean from wire to host?
>>>>
>>>> Pls see above.
>>>>
>>>>>
>>>>>>> The work flow is configure LWM to RX queue and enable lwm-
>>>> triggered
>>>>>> flag in host shaper, after receiving LWM event, delay a while
>>>>>> until
>>>> RX
>>>>>> queue is empty , then disable the shaper. We recycle this work
>>>>>> flow
>>>> to
>>>>>> reduce RX queue drops.
>>>>>
>>>>> You delay while RX queue gets drained by some other threads, I
>>>> assume.
>>>>
>>>> The PMD thread drains the Rx queue, the PMD receiving  as normal,
>> as
>>>> the PMD Implementation uses rte interrupt thread to handle LWM
>> event.
>>>>
>>>
>>> Thank you for the explanation, Spike. It really clarifies a lot!
>>>
>>> If this patch is intended for DPDK running on the host-system, then
>> the LWM
>>> attribute is associated with a TX queue, not an RX queue. The packets
>> are
>>> egressing from the host-system, so TX from the host-system's
>> perspective.
>>>
>>> Otherwise, if this patch is for DPDK running on the embedded ARM-
>> system,
>>> it should be highlighted somewhere.
>>
>> The host-shaper patch is running on ARM-system, I think in that patch I
>> have some explanation in mlx5.rst.
>> The LWM patch is common and should work on any Rx queue(right now mlx5
>> doesn't support Hairpin Rx queue and shared Rx queue).
>> On ARM-system, we can use it to monitor traffic from host(representor
>> port) or from wire(physical port).
>> LWM can also work on host-system if there is DPDK running, for example
>> it can monitor traffic from Arm-system to host-system.
> 
> OK. Then I get it! I was reading the patch description wearing my host-system glasses, and thus got very confused. :-)

The description in cover letter is very misleading for me as
well. It is not a problem right now after long detailed
explanations. Hopefully there is no such problem in suggested
ethdev documentation. I'll reread it carefully before applying
when time comes.

> 
>>
>>>
>>>>>
>>>>> Surely, the excess packets must be dropped somewhere, e.g. by the
>>>> shaper?
>>>
>>> I guess the shaper doesn't have to drop any packets, but the host-
>> system will
>>> simply be unable to put more packets into the queue if it runs full.
>>>
>>
>> When LWM event happens, the host-shaper throttles traffic from host-
>> system to Arm-system. Yes, the shaper doesn't drop pkts.
>> Normally the shaper is small and if PMD thread on Arm keeps working, Rx
>> queue is dropless.
>> But if PMD thread doesn't receive fast enough, or even with a small
>> shaper but host-system is sending some burst,  Rx queue may still drop
>> on Arm.
>> Anyway even sometimes drop still happens, the cooperation of host-
>> shaper and LWM greatly reduce the Rx drop on Arm.
> 
> Thanks for elaborating. And yes, shapers are excellent for many scenarios.
>