From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B312EA0558; Wed, 25 May 2022 15:59:40 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 36D9D42B7A; Wed, 25 May 2022 15:59:33 +0200 (CEST) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2069.outbound.protection.outlook.com [40.107.243.69]) by mails.dpdk.org (Postfix) with ESMTP id 2546242B76 for ; Wed, 25 May 2022 15:59:31 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ptb9ZhC+7NEv26kXyROiR5x77proXBa+MzlANQkC17pNhzzo8LxyFSQfKkh6PCckv5v8K/98afBZc4HnlLeWAlohxWjfITQYYyv2e+jJ8tqCk4PSZk378OwzLJsg52dbkob9QJkr/WqWajJnqPinFLsgxTU0sQOHsCifcIgIrlBrCXSEC3pTmHTP0bd8KsalrRTkwvXXaERdnSjw94fPTiCrt9uqOH0QBT/Kh1HKpNykC0QbPhTUR74RKR8X/p0FaP6uM2LAZhQ+8xSQr/CxgoXv/ZZJ/LrZQlDkd65vVWnq4gFi2NAuOHe3JeE/nLHa++9ss0SRxU1geQpyjnZJBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=94r6kvnIkjZoCWf2MwAvRQMsJiAMCe74drYztX7t00A=; b=Ce7spxtjywfFZIle9zMaHE0LfCNy4BQGPiK0ahs3e3vtHNr+WCyK0E5qUjHTCl4OKVOc/Ujp9G95526cdlY2Wn3H+caBd/MH1heOHuQuY/GCP0XPH2lztRJKxhqz26iuw5E4Rv9sqrv/qyv6J4vaiKgHue5l/UwTq04Sgd/NLtulz+8li6mOZJcVEgZIknB7o+Zyj2l7X2zKk4l06hBe1MmlsVCnUU03JaKHc4lUFmxDH7H2cm8AXLMd2oflQHQ8iL7Se87cy2LzXa5WaLmFdape/ErV/1uw2aA1lfzD1vhP6NTYd0NwDbeO9Yn7ESBVcY6Q0G/lgyFgx8y+0JboOw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=94r6kvnIkjZoCWf2MwAvRQMsJiAMCe74drYztX7t00A=; b=mrKevb19GRrOPL+hXj1Dupmf4Dr0hTYHPZKD7fPw2ECZIdqX2lFTgoOpt2ItPVUX08L/MMsl32ByxUZE/hcfbKvKsrvnwlIVWHsS9PpnqlWxgVN2g4Iri3AmOojD3qC8Up8COqzsDL1ux0UUu9/DhJahjJr26ghAE43mGYlwjjdgki3hmvnK+f3IHFXSkfs6p7IDF40KpOBEz3utwF9idgTcaVelZ1JQLptyvLQ+9QYnqTA+uVjFfDAegcOZMUGrn2lz9IlTvr1oHkTl8bE3AL7CplSC0Vdgccs7EYi9BlKVlJMmv8xI2dHibwMremXVQoupFBhyHG9BGfPsWotd+g== Received: from MN2PR12MB3647.namprd12.prod.outlook.com (2603:10b6:208:c4::17) by LV2PR12MB5941.namprd12.prod.outlook.com (2603:10b6:408:172::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Wed, 25 May 2022 13:59:29 +0000 Received: from MN2PR12MB3647.namprd12.prod.outlook.com ([fe80::f831:cec1:9c49:5988]) by MN2PR12MB3647.namprd12.prod.outlook.com ([fe80::f831:cec1:9c49:5988%6]) with mapi id 15.20.5273.022; Wed, 25 May 2022 13:59:29 +0000 From: Spike Du To: =?iso-8859-1?Q?Morten_Br=F8rup?= , "NBU-Contact-Thomas Monjalon (EXTERNAL)" CC: Matan Azrad , Slava Ovsiienko , Ori Kam , "dev@dpdk.org" , Raslan Darawsheh , "stephen@networkplumber.org" , "andrew.rybchenko@oktetlabs.ru" , "ferruh.yigit@amd.com" , "david.marchand@redhat.com" Subject: RE: [PATCH v3 0/7] introduce per-queue limit watermark and host shaper Thread-Topic: [PATCH v3 0/7] introduce per-queue limit watermark and host shaper Thread-Index: AQHYb4dFoMSbjtMvK0eEvLO+qIfZQK0uYgCAgAEuw4CAAAWEQIAABPlg Date: Wed, 25 May 2022 13:59:28 +0000 Message-ID: References: <20220522055900.417282-1-spiked@nvidia.com> <20220524152041.737154-1-spiked@nvidia.com> <6057836.17fYzF0512@thomas> <98CBD80474FA8B44BF855DF32C47DC35D870AA@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35D870B1@smartserver.smartshare.dk> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D870B1@smartserver.smartshare.dk> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: e9e8dc5f-2937-4edf-f9a5-08da3e56c934 x-ms-traffictypediagnostic: LV2PR12MB5941:EE_ x-ld-processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Tj+qjgUMLEcRZSmB+G2HKgEU5bfoQdfWELFncYlGN2LXG/m5c+agl/2NlhqXbsVl4iUYkoE0/KGWhV0u7kkdVmeSvN40KlPqSZSRsBftohAD0N7MM6h6VWkHXkWksRPxDxtLnu2yYftjlg9FgCfyGdPBVcik9gtuOXHz7gfXE+XyfknQZy/0da4128th5d06jiS7Yhkw9vBi+7RBv//59IZeR21VuEP3MmSORO73VkJA58PBdukw8JRMPh3XqL2d8CW+LM6gfqRNZlAu7yyvOn1xkftTWO1qYysY2/QbsAcFJ1St44Dc890+OxOu7foBuDT54ZSe40wV3kM3TFIxUPScUtqdA2jdzE5u7Gujmcv1n+Y40iiaeB6LGvCSiH6XM5KgcualMpKdl2+gURjEEQAOE+7s4Flhit+wzhyni5D97fX8REJwD6NI0bkXZvJRpQtd6f04fdbSHrvXWshNL77Lf3UmLq5lH3lma44KZLuHr2BsueOdFocBwlea5f87y/3KQ2CXjX/wYYKlTdQ3oK5uUYeqSr5yweYsRfI5Kw9o5YVZ2Eul1pe2MVXz4pzDjQiPBrP66DrYG5xtMIQovwDlL4ly4oCm86pLF6gEXhgUpvypF+SVa1JPlDzq9ZPqHQsqduBwZaYsWUK/yxkWJo2uNrbK5rHUNgjwP4ypm/9Af5R4lxvcTKN4YyKNtvRouJ7Ke4xOnbQwN9qEDcZr/Q== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MN2PR12MB3647.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(4636009)(366004)(2906002)(52536014)(38070700005)(38100700002)(5660300002)(8676002)(64756008)(76116006)(122000001)(8936002)(4326008)(66446008)(66476007)(54906003)(86362001)(66574015)(110136005)(6506007)(7696005)(9686003)(316002)(186003)(26005)(71200400001)(53546011)(55016003)(83380400001)(33656002)(508600001)(66556008)(66946007); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?M/j5CcFpEPYonEUjStIRnQNGDO2d1JEPH9qz9ot766Yu5aPfDMaU9sOkJT?= =?iso-8859-1?Q?SYxXE5Ny+LDw4JbZGD9wz5dCK6acaih+HSkhe09GuspLQq7Vnby5cKVOwM?= =?iso-8859-1?Q?7pgwrx55rHVMVlbTkyZ6EYTIDFgbGal3jGKxuTTSXeDGC2V8argOlD/w62?= =?iso-8859-1?Q?MFUaxjefzGf0+gI5KFwm5lUaYK3sCrMz3II/e6vWdkOYfpRGk79qMOna3z?= =?iso-8859-1?Q?lvibpvaiFJNXA4qn5Oo1Lj0RKfzEheF4Ekh/i9scXF2bhtA11uM2gOPrs7?= =?iso-8859-1?Q?DdExr+jaP6rkD8GhGI9Hv3JwBZQxKlUBfcsYNh7F2jjoErf7l0hZi2rRE0?= =?iso-8859-1?Q?H5MAT0r8Uzp0jrLwC7OzQSSOS8P8rUeyxddYZEtd7zcUwZJ0ZNEE7O6Wjt?= =?iso-8859-1?Q?EiclqMiEQOplVYGB0ygjs32udqTOGyuKBh6Cfozey1tBtPmIAteQuPIifF?= =?iso-8859-1?Q?SaGF9PKBn9D6GAsV1z/3z1fDEb309iMwhpMhRsD9MAgSL3+/ToWPPSvqQA?= =?iso-8859-1?Q?for+czL05937RX75cA5a8zCV6Ru8JX3QYAcKm8nyC+jrXmzHtg6c8/kD2b?= =?iso-8859-1?Q?BldVJ7MkXSIiukhvcslAdGyJ5p5E2Bv4nBRdP3wqP0x4MdVvPsoXxz70It?= =?iso-8859-1?Q?grwOL4w4+SL2xFm0EqDQJXQKGLVVWjpW8UsY7dwYkcpvh2OenpDDtymIh+?= =?iso-8859-1?Q?84dthyOVgD/Mk8q8aveLbVIVu6BBWzY3Qj1lcaXkDwjrorOwboLs3CkT6k?= =?iso-8859-1?Q?eyGdmuH/A1tNB1IloHgXTagUalHmNxWbeEouRvcMQEhPRL78YeDSaa+uGJ?= =?iso-8859-1?Q?BaUQadb8g7Ae0tKPJclFsV+GsmAMkh4vVTiFCuQT1r/051BGaIiTH1HR8V?= =?iso-8859-1?Q?lS7T18kI5LZXUWBzabLoy3YZ7MrzyrWMBsyB+9DyN1nJhkAKG8e6mHgZoY?= =?iso-8859-1?Q?tLw1ia48ysav9u0idPB2Nm4U5RKGtJruCVHlVbLbQzm/4hM0CkiBelIzyk?= =?iso-8859-1?Q?+vKl8ZzHJQhGnWrPV1XMPmDAc7/dVP3IcBfh7kLC78eG9TFiuqUcZQy1OL?= =?iso-8859-1?Q?yuJK33ZWuqFOY7kHhQAyatTN6hgmRXfhC0jGEQvCmkDZMwA6veGNK/cwLA?= =?iso-8859-1?Q?wwZghUTJUpkiqMdaYg7u/yWldi+Zkulfw5NErjox0j0+V/Rd1ZZt50TVCf?= =?iso-8859-1?Q?ifepKILwuDnkMjWlGSILMpyLUawjtFJL+AbhkK8Er1rfxCT9dt0M0nK2Ba?= =?iso-8859-1?Q?RbBOTKtc09B4vwtUNTOFO53zJNGBE26U+AFlcKnkeXX9hoxH6lGkl/SWrJ?= =?iso-8859-1?Q?ShEZ2VCNnLcjOumbmIkWdGvlzyVggS458oCgK1Ln2iWCOqmxxuxCYChMkb?= =?iso-8859-1?Q?lA1eGUJVKwP5DyenAuvMOxDe74WZXV/LGfr7IVzyiAQkw0Ifw11N/POMwB?= =?iso-8859-1?Q?U9KGZk0hQ2zlwCKNX5cAjtF0MzTJI+0FqbrakduNA2RtzoguQUYmkavxBr?= =?iso-8859-1?Q?24xKBx4mHLlCmaLmY0wZ+8ImL5/YRFNlLTQOmeLRbAeVVu7BF/KCeZwkf3?= =?iso-8859-1?Q?BTw+X1q9viBvX2B9zp2WJQMiZVZ1qtuw1cmJcgT3x889lk5tQXpvu0dX8x?= =?iso-8859-1?Q?wHyjschl9+84jDpE4KuXWFeBkDTSCSvB3t9tY7YWojQVpSWi0w8W+e6F8b?= =?iso-8859-1?Q?dLxoVQrFs+EQN+pcDGkCjxIEUmfmAmfWpJHujAFx4ITA/lP28LEMdydAZ+?= =?iso-8859-1?Q?Crg/BSMsxbLqjkHLCusJGoB7a3GgZmeeWaiMHznYVp2Iyz?= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB3647.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: e9e8dc5f-2937-4edf-f9a5-08da3e56c934 X-MS-Exchange-CrossTenant-originalarrivaltime: 25 May 2022 13:59:28.9992 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: R3lb+UeK71C++1vvUno5e08QPPKUZufz23f3YIPe7lLSTc8Yny/ZK40t1tyzfup/4o20BJDMkgZnKwXrfRDUVA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5941 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Morten Br=F8rup > Sent: Wednesday, May 25, 2022 9:40 PM > To: Spike Du ; NBU-Contact-Thomas Monjalon > (EXTERNAL) > Cc: Matan Azrad ; Slava Ovsiienko > ; Ori Kam ; dev@dpdk.org; > Raslan Darawsheh ; stephen@networkplumber.org; > andrew.rybchenko@oktetlabs.ru; ferruh.yigit@amd.com; > david.marchand@redhat.com > Subject: RE: [PATCH v3 0/7] introduce per-queue limit watermark and host > shaper >=20 > External email: Use caution opening links or attachments >=20 >=20 > > From: Spike Du [mailto:spiked@nvidia.com] > > Sent: Wednesday, 25 May 2022 15.15 > > > > > From: Morten Br=F8rup > > > Sent: Wednesday, May 25, 2022 3:00 AM > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net] > > > > Sent: Tuesday, 24 May 2022 17.59 > > > > > > > > +Cc people involved in previous versions > > > > > > > > 24/05/2022 17:20, Spike Du: > > > > > LWM(limit watermark) is per RX queue attribute, when RX queue > > > > fullness reach the LWM limit, HW sends an event to dpdk > > application. > > > > > Host shaper can configure shaper rate and lwm-triggered for a > > host > > > > port. > > > > > > Please ignore this comment, it is not important, but I had to get it > > out of my > > > system: I assume that the "LWM" name is from the NIC datasheet; > > otherwise > > > I would probably prefer something with "threshold"... LWM is easily > > > confused with "low water mark", which is the opposite of what the > > > LWM does. Names are always open for discussion, so I won't object to = it. > > > > > > > > The shaper limits the rate of traffic from host port to wire > > port. > > > > > > From host to wire? It is RX, so you must mean from wire to host. > > > > The host shaper is quite private to Nvidia's BlueField 2 NIC. The NIC > > is inserted In a server which we call it host-system, and the NIC has > > an embedded Arm-system Which does the forwarding. > > The traffic flows from host-system to wire like this: > > Host-system generates traffic, send it to Arm-system, Arm sends it to > > physical/wire port. > > So the RX happens between host-system and Arm-system, and the traffic > > is host to wire. > > The shaper also works in a special way: you configure it on > > Arm-system, but it takes effect On host-sysmem's TX side. > > > > > > > > > > If lwm-triggered is enabled, a 100Mbps shaper is enabled > > > > automatically when one of the host port's Rx queues receives LWM > > event. > > > > > > > > > > These two features can combine to control traffic from host port > > to > > > > wire port. > > > > > > Again, you mean from wire to host? > > > > Pls see above. > > > > > > > > > > The work flow is configure LWM to RX queue and enable lwm- > > triggered > > > > flag in host shaper, after receiving LWM event, delay a while > > > > until > > RX > > > > queue is empty , then disable the shaper. We recycle this work > > > > flow > > to > > > > reduce RX queue drops. > > > > > > You delay while RX queue gets drained by some other threads, I > > assume. > > > > The PMD thread drains the Rx queue, the PMD receiving as normal, as > > the PMD Implementation uses rte interrupt thread to handle LWM event. > > >=20 > Thank you for the explanation, Spike. It really clarifies a lot! >=20 > If this patch is intended for DPDK running on the host-system, then the L= WM > attribute is associated with a TX queue, not an RX queue. The packets are > egressing from the host-system, so TX from the host-system's perspective. >=20 > Otherwise, if this patch is for DPDK running on the embedded ARM-system, > it should be highlighted somewhere. The host-shaper patch is running on ARM-system, I think in that patch I hav= e some explanation in mlx5.rst. The LWM patch is common and should work on any Rx queue(right now mlx5 does= n't support Hairpin Rx queue and shared Rx queue). On ARM-system, we can use it to monitor traffic from host(representor port)= or from wire(physical port). LWM can also work on host-system if there is DPDK running, for example it c= an monitor traffic from Arm-system to host-system. >=20 > > > > > > Surely, the excess packets must be dropped somewhere, e.g. by the > > shaper? >=20 > I guess the shaper doesn't have to drop any packets, but the host-system = will > simply be unable to put more packets into the queue if it runs full. >=20 When LWM event happens, the host-shaper throttles traffic from host-system = to Arm-system. Yes, the shaper doesn't drop pkts. Normally the shaper is small and if PMD thread on Arm keeps working, Rx que= ue is dropless. But if PMD thread doesn't receive fast enough, or even with a small shaper = but host-system is sending some burst, Rx queue may still drop on Arm. Anyway even sometimes drop still happens, the cooperation of host-shaper an= d LWM greatly reduce the Rx drop on Arm.