From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3D2D6A0558; Wed, 25 May 2022 16:30:15 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 25BD540150; Wed, 25 May 2022 16:30:15 +0200 (CEST) Received: from shelob.oktetlabs.ru (shelob.oktetlabs.ru [91.220.146.113]) by mails.dpdk.org (Postfix) with ESMTP id 606C8400EF for ; Wed, 25 May 2022 16:30:13 +0200 (CEST) Received: from [192.168.38.17] (aros.oktetlabs.ru [192.168.38.17]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by shelob.oktetlabs.ru (Postfix) with ESMTPSA id E10AD89; Wed, 25 May 2022 17:30:12 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 shelob.oktetlabs.ru E10AD89 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=oktetlabs.ru; s=default; t=1653489012; bh=Pog8Er5e+GxSRj9fxFtyRfz+/5q3d72pwhuLxTsKuzA=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=IZYXahlUaSOoIOQQnBzaszY9uB80rHJxJv5j3Y5jJYxfG8xc0e5NKuVyJ9Po2oA2f r78d48lkzO3QEx01mXwiZJpsJgd5krJyrSaOjRMne/izlLqXAapB19W1AwWTgTnrAm iYvxy7DXVMYxhxqLf9V0Ip03K6E3n6V/Dr6Zmssc= Message-ID: Date: Wed, 25 May 2022 17:30:12 +0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.9.0 Subject: Re: [PATCH v3 0/7] introduce per-queue limit watermark and host shaper Content-Language: en-US To: =?UTF-8?Q?Morten_Br=c3=b8rup?= , Spike Du , "NBU-Contact-Thomas Monjalon (EXTERNAL)" Cc: Matan Azrad , Slava Ovsiienko , Ori Kam , dev@dpdk.org, Raslan Darawsheh , stephen@networkplumber.org, ferruh.yigit@amd.com, david.marchand@redhat.com References: <20220522055900.417282-1-spiked@nvidia.com> <20220524152041.737154-1-spiked@nvidia.com> <6057836.17fYzF0512@thomas> <98CBD80474FA8B44BF855DF32C47DC35D870AA@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35D870B1@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35D870B2@smartserver.smartshare.dk> From: Andrew Rybchenko Organization: OKTET Labs In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D870B2@smartserver.smartshare.dk> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 5/25/22 17:16, Morten Brørup wrote: >> From: Spike Du [mailto:spiked@nvidia.com] >> Sent: Wednesday, 25 May 2022 15.59 >> >>> From: Morten Brørup >>> Sent: Wednesday, May 25, 2022 9:40 PM >>> >>>> From: Spike Du [mailto:spiked@nvidia.com] >>>> Sent: Wednesday, 25 May 2022 15.15 >>>> >>>>> From: Morten Brørup >>>>> Sent: Wednesday, May 25, 2022 3:00 AM >>>>> >>>>>> From: Thomas Monjalon [mailto:thomas@monjalon.net] >>>>>> Sent: Tuesday, 24 May 2022 17.59 >>>>>> >>>>>> +Cc people involved in previous versions >>>>>> >>>>>> 24/05/2022 17:20, Spike Du: >>>>>>> LWM(limit watermark) is per RX queue attribute, when RX queue >>>>>> fullness reach the LWM limit, HW sends an event to dpdk >>>> application. >>>>>>> Host shaper can configure shaper rate and lwm-triggered for a >>>> host >>>>>> port. >>>>> >>>>> >>>>>>> The shaper limits the rate of traffic from host port to wire >>>> port. >>>>> >>>>> From host to wire? It is RX, so you must mean from wire to host. >>>> >>>> The host shaper is quite private to Nvidia's BlueField 2 NIC. The >> NIC >>>> is inserted In a server which we call it host-system, and the NIC >> has >>>> an embedded Arm-system Which does the forwarding. >>>> The traffic flows from host-system to wire like this: >>>> Host-system generates traffic, send it to Arm-system, Arm sends it >> to >>>> physical/wire port. >>>> So the RX happens between host-system and Arm-system, and the >> traffic >>>> is host to wire. >>>> The shaper also works in a special way: you configure it on >>>> Arm-system, but it takes effect On host-sysmem's TX side. >>>> >>>>> >>>>>>> If lwm-triggered is enabled, a 100Mbps shaper is enabled >>>>>> automatically when one of the host port's Rx queues receives >> LWM >>>> event. >>>>>>> >>>>>>> These two features can combine to control traffic from host >> port >>>> to >>>>>> wire port. >>>>> >>>>> Again, you mean from wire to host? >>>> >>>> Pls see above. >>>> >>>>> >>>>>>> The work flow is configure LWM to RX queue and enable lwm- >>>> triggered >>>>>> flag in host shaper, after receiving LWM event, delay a while >>>>>> until >>>> RX >>>>>> queue is empty , then disable the shaper. We recycle this work >>>>>> flow >>>> to >>>>>> reduce RX queue drops. >>>>> >>>>> You delay while RX queue gets drained by some other threads, I >>>> assume. >>>> >>>> The PMD thread drains the Rx queue, the PMD receiving as normal, >> as >>>> the PMD Implementation uses rte interrupt thread to handle LWM >> event. >>>> >>> >>> Thank you for the explanation, Spike. It really clarifies a lot! >>> >>> If this patch is intended for DPDK running on the host-system, then >> the LWM >>> attribute is associated with a TX queue, not an RX queue. The packets >> are >>> egressing from the host-system, so TX from the host-system's >> perspective. >>> >>> Otherwise, if this patch is for DPDK running on the embedded ARM- >> system, >>> it should be highlighted somewhere. >> >> The host-shaper patch is running on ARM-system, I think in that patch I >> have some explanation in mlx5.rst. >> The LWM patch is common and should work on any Rx queue(right now mlx5 >> doesn't support Hairpin Rx queue and shared Rx queue). >> On ARM-system, we can use it to monitor traffic from host(representor >> port) or from wire(physical port). >> LWM can also work on host-system if there is DPDK running, for example >> it can monitor traffic from Arm-system to host-system. > > OK. Then I get it! I was reading the patch description wearing my host-system glasses, and thus got very confused. :-) The description in cover letter is very misleading for me as well. It is not a problem right now after long detailed explanations. Hopefully there is no such problem in suggested ethdev documentation. I'll reread it carefully before applying when time comes. > >> >>> >>>>> >>>>> Surely, the excess packets must be dropped somewhere, e.g. by the >>>> shaper? >>> >>> I guess the shaper doesn't have to drop any packets, but the host- >> system will >>> simply be unable to put more packets into the queue if it runs full. >>> >> >> When LWM event happens, the host-shaper throttles traffic from host- >> system to Arm-system. Yes, the shaper doesn't drop pkts. >> Normally the shaper is small and if PMD thread on Arm keeps working, Rx >> queue is dropless. >> But if PMD thread doesn't receive fast enough, or even with a small >> shaper but host-system is sending some burst, Rx queue may still drop >> on Arm. >> Anyway even sometimes drop still happens, the cooperation of host- >> shaper and LWM greatly reduce the Rx drop on Arm. > > Thanks for elaborating. And yes, shapers are excellent for many scenarios. >