From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9BA8C43818; Thu, 4 Jan 2024 17:08:53 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 61136402DF; Thu, 4 Jan 2024 17:08:53 +0100 (CET) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2075.outbound.protection.outlook.com [40.107.220.75]) by mails.dpdk.org (Postfix) with ESMTP id C1D7C4029A for ; Thu, 4 Jan 2024 17:08:51 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=I8eHmdZ279xmF5x0SpER8OI8+6YI7GKJbycZakN5++KFCUM0Q/6Jz3Ieo4dG5g5YD4M3iaqJpW5AmTDkNt1/Ihmzrf4I3LR8kBKeihvgMUVsEvZA74/0XWzM1wZL0tjhUNxN648QfOW9yxGtgrOTCcCj1fEiTntbuu0tOVysR7AvpdIDiVu6ajF50uE2NPWZNveoIQOT2YH/ZZEXVlBnmf/E0t4hND5OCT+48SOT3QeTNgte22bhUg7ZeguBwaMhG5uuPYobhluaSvA/DJFlcu0gmMo/XuDL9b+tdK3XpXzHy2vnJlRXJHlCplDOBc6dWxyBtO3MpJLZzMJBSwYagg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=RJECiVu5DQ+GhicIKdSPWjXDvMSR4YKGJ4AiG0ANGmA=; b=DWemJz5RqIC24d2f7+jK4LVGdCH80/v5HAIw/T+kvZEzz48eUW6dQ2vLNpEabGYROS8JUtJV2WulT/olWgiwpCIi+o6DwLrHwaPzL3FIaF1XLRCYLKL5X38Luk43ct/qqZzMeqXVcrexv3TUIaQISs54bE8QUfEy8o93akdQKACAtKaYFnx0Ele8Y8Nswwb1FTrMMkuZ3WgwnxkrYRQ18JmnXcmz4wQTftze8G/R95lKIGkbSAAAl2YInK/dRAh/U81Gf549/GtVDwHtKaLEVW0DIo969o5eO5Qx8J1jle3DlXS6GCuP0bHIJZiJ1eoJ0wd9C7Hp97btsioXnxUuaQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=RJECiVu5DQ+GhicIKdSPWjXDvMSR4YKGJ4AiG0ANGmA=; b=lnyzyAbVyM+I7Rficd/xdjaVPK3n7uRaPSQlTWNyxdIi9+56xQlKF6AXFdloK3oYmMeg+IN/ECBeDbKBooyMNskzCOsfnbcPHQBAjCuAyrNBifMKPO3Ap8+R8TDqjD/d+uyMpnztm2cFNGtdmq7+wZCWQIMJVjrX2hb9lH6wEFWa7+pMdmIawqmyJ29RyPh+bt47AhhLIw6jz/IJdMTyurXApJ4yG3zAA0+eE47VWvFTHHOAoEIZH1kyfXIUHE47RRmLlZLeidcHP4ZnAxaRLR21mARCy81PJfcFuMlIDALqUe3QrusBVGMB6hOLW8IIZzU1TxF3DoG95zm4HtqCOw== Received: from IA1PR12MB8311.namprd12.prod.outlook.com (2603:10b6:208:3fa::12) by BL0PR12MB5508.namprd12.prod.outlook.com (2603:10b6:208:1c1::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7159.15; Thu, 4 Jan 2024 16:08:48 +0000 Received: from IA1PR12MB8311.namprd12.prod.outlook.com ([fe80::2d14:dd8e:f91d:3175]) by IA1PR12MB8311.namprd12.prod.outlook.com ([fe80::2d14:dd8e:f91d:3175%3]) with mapi id 15.20.7159.013; Thu, 4 Jan 2024 16:08:48 +0000 From: Dariusz Sosnowski To: Konstantin Ananyev , Stephen Hemminger CC: "NBU-Contact-Thomas Monjalon (EXTERNAL)" , Ferruh Yigit , Andrew Rybchenko , Ori Kam , "dev@dpdk.org" Subject: RE: [RFC] ethdev: fast path async flow API Thread-Topic: [RFC] ethdev: fast path async flow API Thread-Index: AQHaOLOJMBiGV3bi0EOK9su1qim887C+8euAgAl5pJCAAPg1AIAAZzYQ Date: Thu, 4 Jan 2024 16:08:47 +0000 Message-ID: References: <20231227105709.1951231-1-dsosnowski@nvidia.com> <20231228091657.14769682@hermes.local> <4efb00a7f6f3406ab819424ac7a25542@huawei.com> In-Reply-To: <4efb00a7f6f3406ab819424ac7a25542@huawei.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: IA1PR12MB8311:EE_|BL0PR12MB5508:EE_ x-ms-office365-filtering-correlation-id: ddf63837-ae1f-4098-100c-08dc0d3f6f37 x-ld-processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: jVgRIGMIGFzbi/s1LTVjCeH2HGdCtggd/EuNSKNMyTTPS8YhPfPcaVVnOF4TSMyOn6PL6MIAU9erFgm6ANyAJ0TY0z0EQjuJ7+bhde6OHCWuGGKQYcF2QH+qyoTL7Kmz97zb9PdStgJxALRHQLMKQSV8DYTlot5UW2nwdTHcqUheB4IdY5YSgAm6KmTezBNaVWeAekbAIIaSde80CK2QHCA2fPV8jaNlFvCSOpzbtpCtWshLKZh+slf0wEbY9hwH+K2TyHzSH3qN2WzII00/2ThgICEsxZZcZ6iGPXdrATiixuznlHbrDASHqxQWmK9qHCAfXDq5Yde4eb+HKqp1+if0m4U5leIsb+/PBBAx8NRCUI33hcFYK8Xj/wuGICgZAAQB3OA6nGG5UVMMIXlRe3QfB7EoADngtKpDYW9o7DIKqwHlc6x/pAJRIhduQKm8wa3mTMCc3/XDu2QlVTmjkCVksHtW1AerOy0MwyYxnMl+olBY555lO4H3+kzxAteYAOd94QHu+sdZYc3Ea3NmWCR3on2xuX2IiV/SxWhxLNA8OoJj++9YnqGcNKvj3iQ6WkPsuIMN5DRrz8bBzNOUrMBhO2l7I1sE9ip7/bMhtPiPHPDFdT9AwykzDHDah+aB x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:IA1PR12MB8311.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(136003)(366004)(346002)(376002)(396003)(39860400002)(230922051799003)(64100799003)(1800799012)(451199024)(186009)(38070700009)(4326008)(52536014)(5660300002)(8676002)(8936002)(2906002)(41300700001)(55236004)(478600001)(33656002)(71200400001)(122000001)(9686003)(6506007)(26005)(7696005)(83380400001)(316002)(54906003)(110136005)(66446008)(66476007)(66946007)(76116006)(66556008)(64756008)(86362001)(38100700002)(55016003); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?+Ub6M+AKItrmuA269nye+QLbScDdz7BsrVldXPpBguGHU5oWWZ9/ijwmGhug?= =?us-ascii?Q?zZB7dO5VvKEKvWX/OZDYT0VBYtJZDnrUjHdlA4f2zULYX+sDMhx61GKYeOVn?= =?us-ascii?Q?z6Os0MTKGVN8arDVSrgu5L4VrraijBlwP81EEWquP90JLu37sxO6cnXF9DCL?= =?us-ascii?Q?0/BSFfY1iCBxxA/43IwTS7Cy7rOjGtuI5JxJU9F8asmqPQSiC1Ba/BrqZusa?= =?us-ascii?Q?2GQtym7p9EjLiL3ybiyjtyFmAFyW7FfWOAGEOrn/bEpJ4DcpxwVVWcnzB1pk?= =?us-ascii?Q?0z2TRHTSBI7qbyMTS4C3sIfSSk5B+d/BkdhUlBAUHicclNWcw4cbiV5NofJV?= =?us-ascii?Q?DuMOgi53iDV2mHlwvISNql/iaFrqqOBYWUtDzB0Gk64LQ4Z+2RCGYdknRm6+?= =?us-ascii?Q?TPZ1RAQVI/giFa23MUH8tGb3jF6HqfPA7MnfPlojojFTlYZpJiBoO5KJj4zA?= =?us-ascii?Q?tkXI7+3yxvZbFs6TEKsLH5vvF/lJgSyhQEIv2pEhcu69oIuHUpkARWkDz4Gh?= =?us-ascii?Q?Dnpg6D4jGTo9c6+wqUdnI+9522yUg17DSVedAK0pJ6Oq+SZXTWjY1Tga+8g3?= =?us-ascii?Q?YAdWsZr3Qbj3yzgXFO+GeAABKu//nZxSXPuGmClsf5Obe4Buz8z40IhQQFwV?= =?us-ascii?Q?WjQbcAvvk9g+UI+65vwXP4FOQU6iJKX8iQs+yW1sAfD89DtpNapdAJBqtlCc?= =?us-ascii?Q?0BGKXMWvFeBkyQu5Z5g9L1yJE0EwPQtOoQGnhTbc4LSEMxP1Eq9pBoIGaYyI?= =?us-ascii?Q?8et87tQXmPaWdny5e7kmCi/ejov5yUH4z7cq1loGTVsxT7GDMmEdz9AzPsS/?= =?us-ascii?Q?Rs4wuw1YvyVJ4Ia5XWCNKukzJzJ+TbPLcDNNWQY0r181C7lKzRm5xB1BVkU+?= =?us-ascii?Q?Vt3zitBONU8JNiIqFx3kxmb0KhnBH+7N+DZEDlCuX5j5sC/dWJoZvDNVySin?= =?us-ascii?Q?n3PydD3TJ3ppFYVIS9V9C9beOm30ohM2qSuDVlyvG72QbrmKganOxwm1Yn5g?= =?us-ascii?Q?nrN9fUCz94rwEUq5CjaalViuTzk0q2dhedPIeabQituanAJkXCeQ22koMm6G?= =?us-ascii?Q?/exZtsjsvxQOR7XKAy+tE1yr2i7QUGO1vcjL9Jworz4xnh5Bnh8sR80sEoEE?= =?us-ascii?Q?Hp54+6913cgLiUORuTDIkdM/Bhnend2fb+9K5X5nVlvEpBd9RzbpKMp1BqIT?= =?us-ascii?Q?U8m5jW2RAqr7AdwM5e97mD7QCYG+WtKOlcN1c81/lnBD2WTN662kGVudssai?= =?us-ascii?Q?k7uTEUsXhTbmzzreLHL0reToYPA3TlfLkk3W96IT8QMJiwhr01Tp5dbwCxv/?= =?us-ascii?Q?rdL67MQQoHC3N7KQiDeMn5FXI9I7ZVQi/sUPTo6FwO6yx4qq7OWbf+34rSst?= =?us-ascii?Q?x91kyGQSL1QjUwqemVceYLxetTItlkHxjNP20Ru6X6/EquGSw7fYyZ5IwTsh?= =?us-ascii?Q?eZMajDg2AmocS4gZa6yu2S3seX72YwFTf0q+ZaJuJoDGubgNEaXoiyf/nDs7?= =?us-ascii?Q?SNzfCpkp7LKK2+f5j4ek0jT4WXTYo2AJJ9KSV7cNzQtec87aSlhPeM1br/Gp?= =?us-ascii?Q?gXsCElWW9UtQ6bqJhXjTrUpEF4Y5Lo50mZpBB0/v?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB8311.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: ddf63837-ae1f-4098-100c-08dc0d3f6f37 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Jan 2024 16:08:47.9389 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Zc8x1LBESaht5I5KSW/SH5g3KvgqT95syvI8mU4/S13A5mhCYd2qwDvgycQQQSejkCND7K/Qs+VzB1eKAjCLTw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB5508 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Konstantin, > -----Original Message----- > From: Konstantin Ananyev > Sent: Thursday, January 4, 2024 09:47 > > > This is a blocker, showstopper for me. > > +1 > > > > > Have you considered having something like > > > rte_flow_create_bulk() > > > > > > or better yet a Linux iouring style API? > > > > > > A ring style API would allow for better mixed operations across the > > > board and get rid of the I-cache overhead which is the root cause of = the > needing inline. > > Existing async flow API is somewhat close to the io_uring interface. > > The difference being that queue is not directly exposed to the applicat= ion. > > Application interacts with the queue using rte_flow_async_* APIs (e.g., > places operations in the queue, pushes them to the HW). > > Such design has some benefits over a flow API which exposes the queue t= o > the user: > > - Easier to use - Applications do not manage the queue directly, they d= o it > through exposed APIs. > > - Consistent with other DPDK APIs - In other libraries, queues are > manipulated through API, not directly by an application. > > - Lower memory usage - only HW primitives are needed (e.g., HW queue > > on PMD side), no need to allocate separate application queues. > > > > Bulking of flow operations is a tricky subject. > > Compared to packet processing, where it is desired to keep the > > manipulation of raw packet data to the minimum (e.g., only packet > > headers are accessed), during flow rule creation all items and actions = must > be processed by PMD to create a flow rule. > > The amount of memory consumed by items and actions themselves during > this process might be nonnegligible. > > If flow rule operations were bulked, the size of working set of memory > > would increase, which could have negative consequences on the cache > behavior. > > So, it might be the case that by utilizing bulking the I-cache overhead= is > removed, but the D-cache overhead is added. >=20 > Is rte_flow struct really that big? > We do bulk processing for mbufs, crypto_ops, etc., and usually bulk > processing improves performance not degrades it. > Of course bulk size has to be somewhat reasonable. It does not really depend on rte_flow struct size itself (it's opaque to th= e user), but on sizes of items and actions which are the parameters for flo= w operations. To create a flow through async flow API the following is needed: - array of items and their spec, - array of actions and their configuration, - pointer to template table, - indexes of pattern and actions templates to be used. If we assume a simple case of ETH/IPV4/TCP/END match and COUNT/RSS/END acti= ons, then we have at most: - 4 items (32B each) + 3 specs (20B each) =3D 188B - 3 actions (16B each) + 2 configurations (4B and 40B) =3D 92B - 8B for table pointer - 2B for template indexes In total =3D 290B. Bulk API can be designed in a way that single bulk operates on a single set= of tables and templates - this would remove a few bytes. Flow actions can be based on actions templates (so no need for conf), but i= tems' specs are still needed. This would leave us at 236B, so at least 4 cache lines (assuming everything= is tightly packed) for a single flow and almost twice the size of the mbuf= . Depending on the bulk size it might be a much more significant chunk of the= cache. I don't want to dismiss the idea. I think it's worth of evaluation. However, I'm not entirely confident if bulking API would introduce performa= nce benefits. Best regards, Dariusz Sosnowski