From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E355AA0558; Wed, 25 May 2022 16:16:17 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8F50840146; Wed, 25 May 2022 16:16:17 +0200 (CEST) Received: from smartserver.smartsharesystems.com (smartserver.smartsharesystems.com [77.243.40.215]) by mails.dpdk.org (Postfix) with ESMTP id B0368400EF for ; Wed, 25 May 2022 16:16:15 +0200 (CEST) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Subject: RE: [PATCH v3 0/7] introduce per-queue limit watermark and host shaper Date: Wed, 25 May 2022 16:16:14 +0200 Message-ID: <98CBD80474FA8B44BF855DF32C47DC35D870B2@smartserver.smartshare.dk> In-Reply-To: X-MimeOLE: Produced By Microsoft Exchange V6.5 X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [PATCH v3 0/7] introduce per-queue limit watermark and host shaper Thread-Index: AQHYb4dFoMSbjtMvK0eEvLO+qIfZQK0uYgCAgAEuw4CAAAWEQIAABPlggAAHuAA= References: <20220522055900.417282-1-spiked@nvidia.com> <20220524152041.737154-1-spiked@nvidia.com> <6057836.17fYzF0512@thomas> <98CBD80474FA8B44BF855DF32C47DC35D870AA@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35D870B1@smartserver.smartshare.dk> From: =?iso-8859-1?Q?Morten_Br=F8rup?= To: "Spike Du" , "NBU-Contact-Thomas Monjalon (EXTERNAL)" Cc: "Matan Azrad" , "Slava Ovsiienko" , "Ori Kam" , , "Raslan Darawsheh" , , , , X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > From: Spike Du [mailto:spiked@nvidia.com] > Sent: Wednesday, 25 May 2022 15.59 >=20 > > From: Morten Br=F8rup > > Sent: Wednesday, May 25, 2022 9:40 PM > > > > > From: Spike Du [mailto:spiked@nvidia.com] > > > Sent: Wednesday, 25 May 2022 15.15 > > > > > > > From: Morten Br=F8rup > > > > Sent: Wednesday, May 25, 2022 3:00 AM > > > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net] > > > > > Sent: Tuesday, 24 May 2022 17.59 > > > > > > > > > > +Cc people involved in previous versions > > > > > > > > > > 24/05/2022 17:20, Spike Du: > > > > > > LWM(limit watermark) is per RX queue attribute, when RX = queue > > > > > fullness reach the LWM limit, HW sends an event to dpdk > > > application. > > > > > > Host shaper can configure shaper rate and lwm-triggered for = a > > > host > > > > > port. > > > > > > > > > > > > > > The shaper limits the rate of traffic from host port to wire > > > port. > > > > > > > > From host to wire? It is RX, so you must mean from wire to host. > > > > > > The host shaper is quite private to Nvidia's BlueField 2 NIC. The > NIC > > > is inserted In a server which we call it host-system, and the NIC > has > > > an embedded Arm-system Which does the forwarding. > > > The traffic flows from host-system to wire like this: > > > Host-system generates traffic, send it to Arm-system, Arm sends it > to > > > physical/wire port. > > > So the RX happens between host-system and Arm-system, and the > traffic > > > is host to wire. > > > The shaper also works in a special way: you configure it on > > > Arm-system, but it takes effect On host-sysmem's TX side. > > > > > > > > > > > > > If lwm-triggered is enabled, a 100Mbps shaper is enabled > > > > > automatically when one of the host port's Rx queues receives > LWM > > > event. > > > > > > > > > > > > These two features can combine to control traffic from host > port > > > to > > > > > wire port. > > > > > > > > Again, you mean from wire to host? > > > > > > Pls see above. > > > > > > > > > > > > > The work flow is configure LWM to RX queue and enable lwm- > > > triggered > > > > > flag in host shaper, after receiving LWM event, delay a while > > > > > until > > > RX > > > > > queue is empty , then disable the shaper. We recycle this work > > > > > flow > > > to > > > > > reduce RX queue drops. > > > > > > > > You delay while RX queue gets drained by some other threads, I > > > assume. > > > > > > The PMD thread drains the Rx queue, the PMD receiving as normal, > as > > > the PMD Implementation uses rte interrupt thread to handle LWM > event. > > > > > > > Thank you for the explanation, Spike. It really clarifies a lot! > > > > If this patch is intended for DPDK running on the host-system, then > the LWM > > attribute is associated with a TX queue, not an RX queue. The = packets > are > > egressing from the host-system, so TX from the host-system's > perspective. > > > > Otherwise, if this patch is for DPDK running on the embedded ARM- > system, > > it should be highlighted somewhere. >=20 > The host-shaper patch is running on ARM-system, I think in that patch = I > have some explanation in mlx5.rst. > The LWM patch is common and should work on any Rx queue(right now mlx5 > doesn't support Hairpin Rx queue and shared Rx queue). > On ARM-system, we can use it to monitor traffic from host(representor > port) or from wire(physical port). > LWM can also work on host-system if there is DPDK running, for example > it can monitor traffic from Arm-system to host-system. OK. Then I get it! I was reading the patch description wearing my = host-system glasses, and thus got very confused. :-) >=20 > > > > > > > > > > Surely, the excess packets must be dropped somewhere, e.g. by = the > > > shaper? > > > > I guess the shaper doesn't have to drop any packets, but the host- > system will > > simply be unable to put more packets into the queue if it runs full. > > >=20 > When LWM event happens, the host-shaper throttles traffic from host- > system to Arm-system. Yes, the shaper doesn't drop pkts. > Normally the shaper is small and if PMD thread on Arm keeps working, = Rx > queue is dropless. > But if PMD thread doesn't receive fast enough, or even with a small > shaper but host-system is sending some burst, Rx queue may still drop > on Arm. > Anyway even sometimes drop still happens, the cooperation of host- > shaper and LWM greatly reduce the Rx drop on Arm. Thanks for elaborating. And yes, shapers are excellent for many = scenarios.