From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2E41B439FE; Mon, 29 Jan 2024 14:39:06 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EA12640A7F; Mon, 29 Jan 2024 14:39:05 +0100 (CET) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2069.outbound.protection.outlook.com [40.107.93.69]) by mails.dpdk.org (Postfix) with ESMTP id A13FC402C2 for ; Mon, 29 Jan 2024 14:39:03 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SkuwT92Ot26bJ3Y1yMTcrsL5xQXA0QCp9gZOWe+cSaWXK87vsEzgEkjlClBphqxiSB8+xPpOCLcTtgv/TpGP9qAmcULhYhcDCk55eOpWmNtBIwCCnWdJ/CIVpc4C5qnf48xHGaefHYJmp+uoND/PlLWuaUtRd19kN8c3LVnQUqNDsiOGt58vpbQHw93gfcIAa65si7IhRlS7EDw5vfY9IsbUUbdbf28OBxi52c1v5DPmRgfZsDz/7RCdLfKLoiN3iOIvYyZ7TCvO2egxfKSxwl7ThYVaL6WvvmRNaHHJmtoTHKSy0Ertl5QAD8T4MleRuOdU1KZoponqcoR3613sFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KN13mxi3Ki10YdrsnwnEFrTFIX7CoqzmgtYdE4weqLk=; b=KBx1tyOJtwcYAl8k7hDSjkDRb8vrssX9SXnHC4ebhgVIR6eAwddBUXN1m5iEFtfhhXhgyfHugrJdSf09oxjKowuulNRKkZA1g3PrAI6lcHaS9cJvwxXQ/EHqQojLLuFnCY61Xu3PGTJVQ3iejagNL/TzTBfAzoME7jgCk0BYc4eLQML/yA9tC9yahX+YutGFf5BetYx9umPWEcHZMq7ptRUkn7gCqZuWemDvV4Gm+jI5jx4/BT1kyw20c4pq9/5nhPZ7rvjuOMWtI+i1ZUJpvhkvgHjLkGzzhwsi6FWVTdbJAaW7GBmN/yc+VnT58RHB1ULk8DxmSVz3Ogn5mY9qWQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KN13mxi3Ki10YdrsnwnEFrTFIX7CoqzmgtYdE4weqLk=; b=mfi3AUUPETSXXPxAkmpCV5qlZa5Cqj8u2ehTcibOQze1h1qxoSMa+MB+ZYvB4ZLzsZYFWNXlgqdIoN3qQCMbA+fuve0hFPR7euQ6zz5Zge+r3H5ocXvWWQUeZcCeRsMkSQfSppRrdoK1VQnFrgGx4fAh5OQZSj8wyFJyTyRst8rlRNuQ5sq0L8zVTWAQ0geTKCgzbJbfLepFC/rTu77IK2imNuvgfP6cVw2ijByZPxkOfAHvgiPD1Mz/yn6hmLX/jOKT1DEdcIGgYD06gqniuQ/Erf4Ewy55PgVPFKjZpVDqACgA7PqmrJNLo3UdCKmloOSjYPXaEmT3Uew+yCGPjA== Received: from IA1PR12MB8311.namprd12.prod.outlook.com (2603:10b6:208:3fa::12) by LV8PR12MB9206.namprd12.prod.outlook.com (2603:10b6:408:186::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.32; Mon, 29 Jan 2024 13:38:58 +0000 Received: from IA1PR12MB8311.namprd12.prod.outlook.com ([fe80::708b:e498:d9c4:94]) by IA1PR12MB8311.namprd12.prod.outlook.com ([fe80::708b:e498:d9c4:94%4]) with mapi id 15.20.7228.029; Mon, 29 Jan 2024 13:38:58 +0000 From: Dariusz Sosnowski To: Stephen Hemminger CC: "NBU-Contact-Thomas Monjalon (EXTERNAL)" , Ferruh Yigit , Andrew Rybchenko , Ori Kam , "dev@dpdk.org" Subject: RE: [RFC] ethdev: fast path async flow API Thread-Topic: [RFC] ethdev: fast path async flow API Thread-Index: AQHaOLOJMBiGV3bi0EOK9su1qim887C+8euAgAl5pJCAAHfyAIAeiClggAl+HjA= Date: Mon, 29 Jan 2024 13:38:58 +0000 Message-ID: References: <20231227105709.1951231-1-dsosnowski@nvidia.com> <20231228091657.14769682@hermes.local> <20240103170758.119e99d4@hermes.local> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: IA1PR12MB8311:EE_|LV8PR12MB9206:EE_ x-ms-office365-filtering-correlation-id: f6ab6059-b952-41b5-e918-08dc20cfa537 x-ld-processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: XlGGb1e0zMfwtVhfey+T/9zqZh5p2vBk3bG1cTHfH8eO/WO8JGiXewj/T2EHVifHvkYLwZ8GYWHIXv+/RlkUxh3UITXCxGALm74TPBH5VmgbiB9trJN8LVR1NTmYGooWJYbPVlGwZ1GVI/ccIPoAK8dQ8PoUPWBt34+Nto5AW1BCpsrGtsuzDMyWXt2/zMb5DOcuF+MLqw6hjoGpogwVNZH5hnQRoAwOMnFPVMCsvKdebgXDNIAiV8uTmctqcSNk2kZFRUNpF6fCZT+nLpVaLWugzyPDcdPUp69olfZpUFo1Ot2yUo7CdasJlKUqpAbvkbLiQYZeFy8iP4zGiZNPTh9mvog0rLkQYm1khuwMsrso5udBZibe3LChI6LChj19MO6rEL4qUhwUdjD22jcSoarpUvG58YxoubEePltoGX76rmwmFOSC7G2vepBghnOTtzPwgfQgVQcGhu3gdtn2N1TN5m+f+EdeWGqlc5+OUHJA4Gpl7GTqpR6FZn3VhsRk5C8ZIQuOVwbI3GJeJwblf/B9QhZDGtmbEYoct+scokKye2CkvWr/sV/ePxKVwYM5dzbgFHBfdyp45HMAlS+CzCqndpRNG+PKc4ykpsVMpYGR7llH8QLgGHqNmOAEq+ODT3UaoPMCx3ByHNaTbk3RNyGVDqV86HXsHaiTyes4FbI= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:IA1PR12MB8311.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(136003)(376002)(396003)(39860400002)(366004)(346002)(230922051799003)(230273577357003)(230173577357003)(64100799003)(451199024)(1800799012)(186009)(38070700009)(41300700001)(2906002)(5660300002)(33656002)(66476007)(64756008)(66946007)(66556008)(66446008)(6916009)(54906003)(316002)(76116006)(86362001)(9686003)(83380400001)(6506007)(55016003)(26005)(478600001)(71200400001)(7696005)(53546011)(8936002)(52536014)(4326008)(8676002)(122000001)(38100700002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?AZMi59sPF/TWx34mnoJ+0AGE10B7Z89UMAejZsqnYX9nhp0ajwYl4ApMx8ZT?= =?us-ascii?Q?0nlSfo9DEp7qqOwFHhu+m7bbockuELdpvQ90HVguEF4xit9HGDfcnoYrp7wR?= =?us-ascii?Q?JdmLsbPPyspKfEGHfhoY5uxHwKVrML4vT/Od4AdvgsPuwLNB0LYYDrUt+DAo?= =?us-ascii?Q?G1uR5GUTXh/4NPLGa39N8hX8y4EmjVNjjIGH0t6sS6bcz2HLnVvSAkomcG1O?= =?us-ascii?Q?5x2lvwNMZU5bwY+mTaash8bDoP6B7Hn2GAyAfQU7f48ZCaPuDvvnKQWO0vD8?= =?us-ascii?Q?lujbFPBkLLtF5/mrolJMWZ6MN+kQhn7dtqS+qJIh+drAp9XYViF6IRI9hLAt?= =?us-ascii?Q?e4yoEsH62syw7q/6WzOgs/SExGo8nujj0OXGwNG76yrkqqbqweNnfTHFddQQ?= =?us-ascii?Q?7S6f3N8gzIvX3lpKukUW6s8FUiy12pKlLErOYGSaq64ykszjxBrOkg5e63EJ?= =?us-ascii?Q?kn9ge/biCuAZeLcoyJUc378i8eMJjgqs5yW5YA4cWmgM0zdmmxgT1RacrHmC?= =?us-ascii?Q?I4qQ1k7RRySNkmU8MsNFicjnTu3aog2gPBCpj/hx/0+v06EmDmVv3imIQZqa?= =?us-ascii?Q?OycjD2V9G+rKtBRz9+ffd/cKaw5FhBm4be2+2RNGiSuivALEvLpKFLcYrP3r?= =?us-ascii?Q?jfAiX9SWcVgYKeikbZXvtZmIqBf0oblbAOAY6sDnhWQQyV+fx+BlaZ5n+Ofo?= =?us-ascii?Q?TGtww32xjMb4aujNpRsKXLOC4LzWz5K7C+Z0XUXm4zXtgJObdp73YxIE2F67?= =?us-ascii?Q?NEwfzKIk5+X84E05Z2c1nDZB+PBjQjSJNQpuDBaB1348dXBeznRFIlr0V/Pv?= =?us-ascii?Q?hu+p2QMa7ITD9RqsKuiQPV9NcsvvbFeAZj7vY/CzK6ZTiz7l/s04pnLClg6t?= =?us-ascii?Q?DSY4WPOg0p/ctwtR+EjDF3Mvymrw5ucCCBxBXcqBBx0DGN29r6ArvnRIzJA2?= =?us-ascii?Q?TcDhA8tRWAvNFmeSJzTk3f0vRT0hQLT8XTSHDGiAVMaC82hZYJM2EbxAGyds?= =?us-ascii?Q?kmaCoqCNMTdtxs8njFDoLjpVMAlPwiTDxcxOS4Z5l+uzTu4LyDsuiUCrl4o4?= =?us-ascii?Q?Kpfemy/MUGXSx+3LQ4GXkXD2hdaLAf84I16GBYwQQhWtaudPbG3ENpovFgj/?= =?us-ascii?Q?bYcTOHmN7a//9HbfzsUas86SWa0yZKhd0SITJImpBKKjFL4rxRxGAMS2UBsj?= =?us-ascii?Q?6HjpofHLB0aE8e2cBzOMF5uSdi2496xIRBJ9x4eVTnsaSxDxgDrKGRdii8P/?= =?us-ascii?Q?NxhazOQPQ3caM5pj4968uY948EoL4XTAqeLJxJhDOemxR8LBQFdmCawM/Luu?= =?us-ascii?Q?KF/DAM9G6i2voOcgk7YV4HuEEZACdc0Lq3dyTLB37F/onzBcuXhTXB3SLmI+?= =?us-ascii?Q?bMVFWahsMp9wxYKs0FlW8Av1UW26avyG4PWebqVHM5XlEebwuJelyXjEfeS1?= =?us-ascii?Q?FNfiW8e3cu0RozfM/Gej1rF1RbAyv7gijdUEqGL5XVbIFq2lyy8f38INkzub?= =?us-ascii?Q?vWzrKcWRgvTqnACNRZr8x2Cpps4tLeCmb5lqv2cP0+jHoJwqHd3DERuODJs1?= =?us-ascii?Q?fse0cwdtLc1lTsYNGYLSLUr7BK1IG97GeLEMCebH?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB8311.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: f6ab6059-b952-41b5-e918-08dc20cfa537 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Jan 2024 13:38:58.1948 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: JXjTm7bMo5ufEHT9E9xqz2CFjoxs5bNEuc+GvOXFPYO7dCwQg0LUKmo1yXaO12FMr8dPYrYsPJ7z7+PxjZVKQw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR12MB9206 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi all, > -----Original Message----- > From: Dariusz Sosnowski > Sent: Tuesday, January 23, 2024 12:37 > To: Stephen Hemminger > Cc: NBU-Contact-Thomas Monjalon (EXTERNAL) ; > Ferruh Yigit ; Andrew Rybchenko > ; Ori Kam ; > dev@dpdk.org > Subject: RE: [RFC] ethdev: fast path async flow API >=20 > Hi Stephen, >=20 > Sorry for such a late response. >=20 > From: Stephen Hemminger > Sent: Thursday, January 4, 2024 02:08 > > On Wed, 3 Jan 2024 19:14:49 +0000 > > Dariusz Sosnowski wrote: > > > In summary, in my opinion extending the async flow API with bulking > > capabilities or exposing the queue directly to the application is not d= esirable. > > > This proposal aims to reduce the I-cache overhead in async flow API > > > by > > reusing the existing design pattern in DPDK - fast path functions are > > inlined to the application code and they call cached PMD callbacks. > > > > Inline needs to more discouraged in DPDK, because it only works if > > application ends up building with DPDK from source. > > It doesn't work for the Linux distro packaging model and symbol > > versioning, etc. >=20 > I understand the problem. In that case, I have a proposal. > I had a chat with Thomas regarding this RFC, and he noticed that there ar= e 2 > separate changes proposed here: >=20 > 1. Per-port callbacks for async flow API. > - Moves specified callbacks to rte_flow_fp_ops struct and allow PMDs = to > register them dynamically. > - Removes indirection at API level - no need to call rte_flow_ops_get= (). > - Removes checking if callbacks are defined - either the default DPDK= callback > is used or the one provided by PMD. > 2. Make async flow API functions inlineable. >=20 > Change (1) won't break ABI (existing callbacks in rte_flow_ops can be mar= ked > as deprecated for now and phased out later) and in my opinion is less > controversial compared to change (2). >=20 > I'll rerun the benchmarks for both changes separately to compare their > benefits in isolation. > This would allow us to decide if change (2) is worth the drawbacks it > introduces. >=20 > What do you think? I split the RFC into 2 parts: 1. Introduce per-port callbacks: - Introduce rte_flow_fp_ops struct - holds callbacks for fast path call= s, for each port. PMD registers callbacks through rte_flow_fp_ops_register(= ). - Relevant rte_flow_async_* functions just pass arguments to fast path = callbacks. Validation checks are done only if RTE_FLOW_DEBUG macro is defin= ed. - Biggest difference is the removal of callback access through rte_flow= _get_ops(). 2. Inline async flow API functions. - Relevant rte_flow_async_* functions definitions are moved to rte_flow= .h and made inlineable. Here are the results of the benchmark: | | Avg Insertion | Diff over baseline | Avg Deletion= | Diff over baseline | |-----------------------|---------------|--------------------|-------------= -|--------------------| | baseline (v24.03-rc0) | 5855.4 | | 9026.8 = | | | applied (1) | 6384.2 | +528.8 (+9%) | 10054.2 = | +1027.4 (+11.4%) | | applied (2) | 6434.6 | +579.2 (+9.9%) | 10011.4 = | +984.6 (+10.9%) | Results are in KFlows/sec. The benchmark is continuously inserting and deleting 2M flow rules. These rules match on IPv4 destination address and with a single action DROP= . Flow rules are inserted and deleted using a single flow queue. Change (2) improves insertion rate performance by ~1% compared to (1), but = decreases deletion rate by ~0.5%. Based on these results, I think we can say that making rte_flow_async_*() c= alls inlineable may not be worth it compared to the issues it causes. What do you all think about the results? Best regards, Dariusz Sosnowski =20