From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E8D2BA0556; Wed, 25 May 2022 15:14:53 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9366440146; Wed, 25 May 2022 15:14:53 +0200 (CEST) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2085.outbound.protection.outlook.com [40.107.236.85]) by mails.dpdk.org (Postfix) with ESMTP id 83D1D400EF for ; Wed, 25 May 2022 15:14:52 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YLWb4Jvp+Hqe3vbxeeiNq9brmxKJUx8+xd3zHAzCPfvuebMAoJ67jQZAIw2e84zL3HFLTTyZm6GgEEHBG5nQS/o71z2EPdCfAuXWqb8gcWhiDKq3ogsjIPccYxQ+r3ZtkKHgk2LAV2xiaeLogq+7T30nPlN6bdGDPMqJQDpubHvBMk0yDPm/mn5nTw7/mZBoeaExoiMv//EnqjArvdUWlSFkC69bEs8hDqTQqtA9SJsW5dO8VwJy3tMPunyYvjSRjr5M97Dr0z7wHeBOyGrd3VAo8gMq/Z6aiL76U8vkudWxCAdWVqE3P/jPaENT3OBVPTsztowu+ZfT6/zjFOX2sQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rEHEL3vUaJF89Q3juxE9QS2f5GoTOqhQLQcbCi30ByA=; b=OcOhh21XYXDOBdetWszvWPbcuyhIzoft8glacESt3k7gq8efFhsM1lcN4qo3kfQw0JQ2YcMqvjdVhxLJZbc9NE/3CC75lJiBuq2qi8+y4cTBxtEI2ILevfw9ChfQ3nSh9xyMEHMMRzukrPACveXufSulWuO+Urq7F4FszfJUmVgROZV4nOiF6LkzrCkSSAqbvq25F6scpAls0+PFG0H1CdB8LvBE8h9D4pw1RGbuEQwd6GoMJOdAnKlsxdyKODm9KNiOKKpg9WT/Q2+buGpY98+LxncuUbBO6ez4dFrHkcGf47K0oJjnedKLwjvoYNZwD+JkOW/q71Y0J595QDNZEw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rEHEL3vUaJF89Q3juxE9QS2f5GoTOqhQLQcbCi30ByA=; b=CFjCadUVwF55PRAV5GNGABv8z8mgcrePe1k+2l1Q76m5n+qDVyD0CcmHhFfwE6FigtuF442zefxV1+H8M2g83AgRZEZvijEkLGpFsHNJ5SDm85wFSEo+bRNitZQZY+IOlaBp+BsHb0bBO9TpeGUQlRKTA9EHnXO1dt1wV0s+yg4azbKJJOQCNe3n/Yw3ZiRtdoDS77NnxpC+fzG5Ns+bM7kpiv8npuU9hqmXKY+tih7prVxD/zL5b6VidZi80A/tU83wwc+bqlbc1IlfPXBpfgZ9b2YvV0q+u/xGPTGfKcMsBmu5cPXEh0jGl+moHLb3HG/c8rqpnIXtLdOyy14Rvw== Received: from MN2PR12MB3647.namprd12.prod.outlook.com (2603:10b6:208:c4::17) by DM6PR12MB3338.namprd12.prod.outlook.com (2603:10b6:5:11f::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5273.21; Wed, 25 May 2022 13:14:49 +0000 Received: from MN2PR12MB3647.namprd12.prod.outlook.com ([fe80::f831:cec1:9c49:5988]) by MN2PR12MB3647.namprd12.prod.outlook.com ([fe80::f831:cec1:9c49:5988%6]) with mapi id 15.20.5273.022; Wed, 25 May 2022 13:14:49 +0000 From: Spike Du To: =?iso-8859-1?Q?Morten_Br=F8rup?= , "NBU-Contact-Thomas Monjalon (EXTERNAL)" CC: Matan Azrad , Slava Ovsiienko , Ori Kam , "dev@dpdk.org" , Raslan Darawsheh , "stephen@networkplumber.org" , "andrew.rybchenko@oktetlabs.ru" , "ferruh.yigit@amd.com" , "david.marchand@redhat.com" Subject: RE: [PATCH v3 0/7] introduce per-queue limit watermark and host shaper Thread-Topic: [PATCH v3 0/7] introduce per-queue limit watermark and host shaper Thread-Index: AQHYb4dFoMSbjtMvK0eEvLO+qIfZQK0uYgCAgAEuw4A= Date: Wed, 25 May 2022 13:14:48 +0000 Message-ID: References: <20220522055900.417282-1-spiked@nvidia.com> <20220524152041.737154-1-spiked@nvidia.com> <6057836.17fYzF0512@thomas> <98CBD80474FA8B44BF855DF32C47DC35D870AA@smartserver.smartshare.dk> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D870AA@smartserver.smartshare.dk> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 1d0ee5a3-a4bd-4112-865b-08da3e508bc2 x-ms-traffictypediagnostic: DM6PR12MB3338:EE_ x-ld-processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 4Lo1QKK8dpqA0Nicc7Qe0bGgSX1JrqsMkZ3IXC/h49nhFOkx4zBgIS6eh1jOWWSF8EujDdOMH8SoUSxQMATQYJQisXLobgAR6gE8Fl/46omUoGkJENThZ/+XiQ1JqJA1LU9NXOnWDX4xtXt5xoXateePIiQ+o9QND5FB2wDKKHpkDj0CIBnPai6yLOcJ+xm+saeWJDwv+hQv4ATcs4Tu7FgA311lgYjjLR2+AsQY5Wqv5z2WQ1OrT3QZPBCueIIrdMOdTp1cvtufhEJiMx8a21Q5kmAVkWq8ugxn9oCADhozt9nt5RU0TxKZy7gnqIKbC6aaWu+cc/yml/MU4R7JEKPzwN93XbgF5ECkX4clwIDHV+XrtgkbuVY3PKxPimwU14QxCIG+Z1pBRbqVGWyU0kDaPnG+0RhVWaLGYmvNfuXn4vzoujQRXcKGfeAB1HHFKOuOgyqxHyAffGp0Mr+fL7xwX8rnAjdfboJ4YhT9T1lzjMp6C/AsrO92FDIATdJ06Sy6h1CD3CrcHT+HFLdtcvpQtc6iE6yHOCEwc9xARl5CSnzaxS6tXVdq4pv6qEmrbh1KQBHmGZRX+lek0NXBSDLnlYXDRHvN+NzQQ+hasrUHo/2QpLZuhUAs6X47VaZUbd9d/ZyYcG7KYoGiH/PEM1JaHNWCsGSU178iJciolWYqn3AB9Jb21dEsAnQk1T0aZltpT1dYay4PGrFRL5abyQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MN2PR12MB3647.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(4636009)(366004)(66556008)(8676002)(52536014)(66476007)(64756008)(66946007)(33656002)(76116006)(4326008)(54906003)(55016003)(110136005)(8936002)(53546011)(2906002)(26005)(71200400001)(9686003)(7696005)(508600001)(186003)(66446008)(122000001)(6506007)(83380400001)(66574015)(5660300002)(86362001)(316002)(38070700005)(38100700002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?eVFe75fHv9GGnOPqIokdpsttPue5VRKa9DdUNZyP4Y+MnFB7mz8iK+JlND?= =?iso-8859-1?Q?POieSVWiXkoCeX6qQl0TxEP5q4no3Lo2GWgEooIoSEaIDclC4vURg1YfDX?= =?iso-8859-1?Q?ic9gFTox7yG7+L1zL6N8PX2g4mRxC+O8d6txjBY1vhlPtfjAsM5vm6YL1j?= =?iso-8859-1?Q?BbeNJEzD4CF7MrqyU18VG7WGqpUjQmiXtyyqTlJOSJIBS1TB9stujS3esT?= =?iso-8859-1?Q?wIDCKBpui/+CpFAhPrWbeWcfKOaYzSV9CEfx/f29gpzbg6T7Ubrv5zJqhQ?= =?iso-8859-1?Q?5VdB8XzX5vf+mwbDTwPIbhiE0M5WK80nZtjUCU8f7MF2OPtINce/L2/ggk?= =?iso-8859-1?Q?47N+n839ZeJLixat2RmJC/xYmSg9W1W/BrUVsCZRSNAwfnefuABlLOMwLc?= =?iso-8859-1?Q?4H2tqWUbkvArDhnloYjwM5aDX0H/BQAZBLUkGH73wZyroT24s3dLkqeZAf?= =?iso-8859-1?Q?DpjEze0yFyJhd8Z9Xo4xaLvQkcA8VkP8xKQIn8p1VYdYaw/oigqxFY+2PD?= =?iso-8859-1?Q?qwPU8/LebUUw6b8cJePKRDJAj9Z4BdWUwA3fhhY5voFrOZXC0oMDEl6/nP?= =?iso-8859-1?Q?Qr+Hoip0qfzVHuKbW//CurZhfXtjydEinj9xR3FiSSJJWf9SpuC7yM1kPS?= =?iso-8859-1?Q?sv23qkVdhiB75rXDYIK1Ymj18jmXAGDm2O2gSNxnE8k6Dv8g+2yU7lZibc?= =?iso-8859-1?Q?SlF/V6umPDxGld95nHH6Yv489uFxIpRVytDdNSAGtEyPcbZsNp8o6pe3v1?= =?iso-8859-1?Q?4IxdB1RRYeMmXHY7eKo8jhgmH93wRxjmTBuJ6LrDJW8VJrki/zNgiBD7Hr?= =?iso-8859-1?Q?ke2nZVkhnoyQPbis9elk6XFl30+XsJMAHbvHbZs+xSFB1/P4M/bRUsk0Jm?= =?iso-8859-1?Q?isoLJX3bOTQvCEVnoNaxSZFGqs7aqU1fxQNPKuJFChHlxkkUPK2xIcxN2/?= =?iso-8859-1?Q?UlCb4M09MCm7VcNNjoEYpJiDJGUeWY2HdWdekOirizSAckimrALpaRmaqN?= =?iso-8859-1?Q?1DgiwmBn7QysX/OJrupcQkaYdg7RSxbWKNwuq9D/gbz8g5tnfRJojhixVj?= =?iso-8859-1?Q?RthZlmuD2qxBFmWMQ5kTRokgGB4/hYjex+fscxbxUa/P/Xe6HD1uNvdgG8?= =?iso-8859-1?Q?/CfSvTl2gj/BgtPk8ORt0itwmq8uW2H2qH4EHuMxGIk9kRpqHWIQawXrLi?= =?iso-8859-1?Q?wBt+4aiJyGTbXo5j7yhTMUO58I6pDm7Llh7k/AM5HGKQwFG/JmZwnXmz0F?= =?iso-8859-1?Q?1iP7+dTCBI1UwGCDJ97wM0evH3m/+Kq2jiLb1xjpQeXFPuzmjqf3nWNJpg?= =?iso-8859-1?Q?m2slHsc5w6DYZKX84KROQv2ZgMX4Kj0pET9fvKX5Qxwq1EEmjaa3fdbZt+?= =?iso-8859-1?Q?gStKVZXveWiU6RtcT11ND/I05d3cEMztfjKWq/Ppq9BtZA6hmCHiXGlt3y?= =?iso-8859-1?Q?BtAWMmMCjjwEjIjqmkjWuH3dTo31GbBefyhTwkF+tzO+kQdUvPKrY49GDa?= =?iso-8859-1?Q?ZS8Ew/InFSm3Kcitjok3q4UPxypzXhiuUdO/BaWyC8JZGhuzxqLFB0TTVm?= =?iso-8859-1?Q?QdVQMaIcOgNL1fYgUmRx8hBcuS3f3B92oyqpGOZeaFYae3CSpRMKhz7fzL?= =?iso-8859-1?Q?UtotebunKP0G5QsTU+iALpKGn3Rg2QDD82Op9+y4FNGSbtsz9wzniBMcVZ?= =?iso-8859-1?Q?cgbQleO6mG40d8yxPq4zqCu3qm8D/8KTp1FxHTRoH9dcAh2oIq+7gzEV2u?= =?iso-8859-1?Q?fWfTBjxwEBf1BXxpHNt7D/+FSVOI+OssWMjumKkvmbCVQ8?= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB3647.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1d0ee5a3-a4bd-4112-865b-08da3e508bc2 X-MS-Exchange-CrossTenant-originalarrivaltime: 25 May 2022 13:14:48.9277 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: YvQ7exoutauftp1kLk+d32cKwCc0dKykpRj8k8Oy7eNoJkRXgFV3FGhBTs2pj8BGV19A39ZTO/9eAvylRN4vig== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB3338 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Morten Br=F8rup > Sent: Wednesday, May 25, 2022 3:00 AM > To: NBU-Contact-Thomas Monjalon (EXTERNAL) ; > Spike Du > Cc: Matan Azrad ; Slava Ovsiienko > ; Ori Kam ; dev@dpdk.org; > Raslan Darawsheh ; stephen@networkplumber.org; > andrew.rybchenko@oktetlabs.ru; ferruh.yigit@amd.com; > david.marchand@redhat.com > Subject: RE: [PATCH v3 0/7] introduce per-queue limit watermark and host > shaper >=20 > External email: Use caution opening links or attachments >=20 >=20 > > From: Thomas Monjalon [mailto:thomas@monjalon.net] > > Sent: Tuesday, 24 May 2022 17.59 > > > > +Cc people involved in previous versions > > > > 24/05/2022 17:20, Spike Du: > > > LWM(limit watermark) is per RX queue attribute, when RX queue > > fullness reach the LWM limit, HW sends an event to dpdk application. > > > Host shaper can configure shaper rate and lwm-triggered for a host > > port. >=20 > Please ignore this comment, it is not important, but I had to get it out = of my > system: I assume that the "LWM" name is from the NIC datasheet; otherwise > I would probably prefer something with "threshold"... LWM is easily > confused with "low water mark", which is the opposite of what the LWM > does. Names are always open for discussion, so I won't object to it. >=20 > > > The shaper limits the rate of traffic from host port to wire port. >=20 > From host to wire? It is RX, so you must mean from wire to host. The host shaper is quite private to Nvidia's BlueField 2 NIC. The NIC is in= serted In a server which we call it host-system, and the NIC has an embedded Arm-s= ystem Which does the forwarding. The traffic flows from host-system to wire like this: Host-system generates traffic, send it to Arm-system, Arm sends it to physi= cal/wire port. So the RX happens between host-system and Arm-system, and the traffic is ho= st to wire. The shaper also works in a special way: you configure it on Arm-system, but= it takes effect On host-sysmem's TX side.=20 >=20 > > > If lwm-triggered is enabled, a 100Mbps shaper is enabled > > automatically when one of the host port's Rx queues receives LWM event. > > > > > > These two features can combine to control traffic from host port to > > wire port. >=20 > Again, you mean from wire to host? Pls see above. >=20 > > > The work flow is configure LWM to RX queue and enable lwm-triggered > > flag in host shaper, after receiving LWM event, delay a while until RX > > queue is empty , then disable the shaper. We recycle this work flow to > > reduce RX queue drops. >=20 > You delay while RX queue gets drained by some other threads, I assume. The PMD thread drains the Rx queue, the PMD receiving as normal, as the PM= D Implementation uses rte interrupt thread to handle LWM event. >=20 > Surely, the excess packets must be dropped somewhere, e.g. by the shaper? >=20 > > > > > > Add new libethdev API to set LWM, add rte event > > RTE_ETH_EVENT_RXQ_LIMIT_REACHED to handle LWM event. >=20 > Makes sense to make it public; could be usable for other purposes, simila= r to > interrupt coalescing, as mentioned by Stephen. >=20 > > > For host shaper, > > because it doesn't align to existing DPDK framework and is specific to > > Nvidia NIC, use PMD private API. >=20 > Makes sense to keep it private. >=20 > > > > > > For integration with testpmd, put the private cmdline function and > > LWM event handler in mlx5 PMD directory by adding a new file > > mlx5_test.c. Only add minimal code in testpmd to invoke interfaces > > from mlx5_test.c. > > > > > > Spike Du (7): > > > net/mlx5: add LWM support for Rxq > > > common/mlx5: share interrupt management > > > ethdev: introduce Rx queue based limit watermark > > > net/mlx5: add LWM event handling support > > > net/mlx5: support Rx queue based limit watermark > > > net/mlx5: add private API to config host port shaper > > > app/testpmd: add LWM and Host Shaper command > > >