From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3D6134380C; Wed, 3 Jan 2024 20:14:54 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C39B2402BC; Wed, 3 Jan 2024 20:14:53 +0100 (CET) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2048.outbound.protection.outlook.com [40.107.223.48]) by mails.dpdk.org (Postfix) with ESMTP id 70F254013F for ; Wed, 3 Jan 2024 20:14:52 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=n6IBxkGph15V1s08qa7Gf3bVUo72dKmfEiGL5uH0AobLTxgA2MGD15xspv+q7OrXyMGA4mWo3DDs1ZY0tsUS1iiySD7tyq/Hs3KXyKXDs1ZHba8x4b+jTzenJ3KPzthuDxeJVWX4EjzQA4hZIuCpVcYUtbZxDuJfF9difGik4AdGwZ+x/jLfnYh6rsCyhVzt2m39+fPemWBy022NREkXRSBAXxhRbY4KLyXQKk2yTA+Ju04rv/beLy0jpSfpjWMKXBByXq8FmDs59xPy/V+btOETHEi3D4Bb37+6NExMzG/yQOeDMQXAUA4Z8gv3QHFSSy3tWqej+ywUGYONUCr/Sw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5ednt/+QnK5jaDns1hsIfzefoUtyOPXB6TnqfM5OWTQ=; b=T094sg6RB8coFG3y0uWGX7LTjqhnkHzDrJL1fnAFppwENcyKrdRgERxPBaQuKcCdr9tOik2GRCOr1bLmSrSX7B/QK06jzUhqAcCcpOFuo1JhPCoh0J5+UFT7zcSSCazNNFbqosQfrzsz2f4ydB3MJmhplmu7A/ry7W/bSrF7PSP0ROhweXLk7tQ8kc7NNACAbYOJX7r6vcgIa3FiNlAJLZvt4o1i8GjL7+gXzaTqSyMTcSrSlMrIDc2rnfMwFL4t8V31Nl5LPGgtF/5mPpG2+fBT9AmivbK3owoy2he1GvOpnkFsCHhjOrYYcWetqJ/55GNZ3qpErBEwSsckTeJcQQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5ednt/+QnK5jaDns1hsIfzefoUtyOPXB6TnqfM5OWTQ=; b=Gd8XcfhdZP7Uiy2B9WX3Sey5zS1o1IbU03AqdVJkLBXp8SGxM2rNxGiSauiea0+UxSOpYYY3gCVE8jE0MVKM0cVPIxv4F5zlmSD+iDrxrC2X9hnedY0zp6FjTrQyxRTSXkVsEoJSFL7Fn3d4NvzWCpwe5id+8UMFTrTKoe6Rco56jMN6MaNrIPAMLKVM1nFxd/AgKJ2ZV5SNnpa8YHRKF3RXAYxrCwV24NeOxoN/BN/FhgF0dkkdzhAUY6MAEzsvk+Mh7nKwAZhmS0BoBdASgdKp2kyzSqyxl6p7YFskKb37+jjvf82zKunAOurq4xaxt4R9PCggPYLuenorh/BwmQ== Received: from IA1PR12MB8311.namprd12.prod.outlook.com (2603:10b6:208:3fa::12) by SJ0PR12MB6965.namprd12.prod.outlook.com (2603:10b6:a03:448::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7135.25; Wed, 3 Jan 2024 19:14:49 +0000 Received: from IA1PR12MB8311.namprd12.prod.outlook.com ([fe80::2d14:dd8e:f91d:3175]) by IA1PR12MB8311.namprd12.prod.outlook.com ([fe80::2d14:dd8e:f91d:3175%3]) with mapi id 15.20.7159.013; Wed, 3 Jan 2024 19:14:49 +0000 From: Dariusz Sosnowski To: Stephen Hemminger CC: "NBU-Contact-Thomas Monjalon (EXTERNAL)" , Ferruh Yigit , Andrew Rybchenko , Ori Kam , "dev@dpdk.org" Subject: RE: [RFC] ethdev: fast path async flow API Thread-Topic: [RFC] ethdev: fast path async flow API Thread-Index: AQHaOLOJMBiGV3bi0EOK9su1qim887C+8euAgAl5pJA= Date: Wed, 3 Jan 2024 19:14:49 +0000 Message-ID: References: <20231227105709.1951231-1-dsosnowski@nvidia.com> <20231228091657.14769682@hermes.local> In-Reply-To: <20231228091657.14769682@hermes.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: IA1PR12MB8311:EE_|SJ0PR12MB6965:EE_ x-ms-office365-filtering-correlation-id: 7f218a3a-fdd2-4a85-b6be-08dc0c904180 x-ld-processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: aLIN+eLyphEP4PaYeNaO+FX3mmeOUC0UM8T3vjxxOQHyCfdHZF5G67BeNfYZJ9yzW78/cslTRdPBFQJGgaOazKGTz+lNmL3IvVljJCl0nqiXiKXUJb7bCiN/8rDuYQ3OnimSddSk/bCEnwmKYaCPFBMdMKyul6upbzZLlvSTI9rSdfm0wKXt4mqETFfJgsMkbC/EE+iYoedLp0vQrS7EaZpWDEYZkeI1IFih96x8dIwbWZppEPI4SeJeYQd/9DZ8y1DnJ9b20Eb6KpS/hTmivuj9Yentc3AsnkV9Y0UJZKHaYucVh2fIhDFuMsQlEQTYF5fmexgmglyzgDJQpclizI5KR0PHchELQSTPXPHMoj3ewTiF+83rUFDQWSRLSCCRPRQKnMfmas/3k9Y5ZEdmXGdaIaSoH3/37YZzE9EzUQ/MyNU7KmT5tmiIoCnHLG8V0b28oBmOy/5Wsa+CtKweuumWTo/PibKLBSyXfGUphrDFtrh1IeYUbFZCK3iRMVIl2fquE7nr6ro4/3O+Kf8qswnsyH365NGWLWtJCUvOXS+XH/7UH81SQwf2URexXRmmtB+wKiSjkArQx2yt6JB418reLENIqMNxOnU6GXPYx5a/D9IAWlcx09X2Fdc1QT0L x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:IA1PR12MB8311.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(136003)(366004)(396003)(39860400002)(376002)(346002)(230922051799003)(1800799012)(186009)(451199024)(64100799003)(26005)(38100700002)(122000001)(83380400001)(8936002)(54906003)(8676002)(316002)(41300700001)(5660300002)(71200400001)(4326008)(2906002)(52536014)(478600001)(6916009)(66446008)(66556008)(7696005)(64756008)(6506007)(76116006)(9686003)(66946007)(55236004)(66476007)(38070700009)(86362001)(55016003)(33656002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?2N/O6Wua1F9343af3vRPAM3DXXxeV70wH58+lQZjHIRZaEhKKeBooqbzrb8+?= =?us-ascii?Q?xIaLEIcT3BFUWjESNlSwKOodXFaeFgcF7vuUyEuOGtV+cWHwyoyEyrNPDsox?= =?us-ascii?Q?NGmoWrsBoNsaPy62XFsqZheTw8KxhF9kpQEKeqDFguGhiD0uwrxeLWbhqrli?= =?us-ascii?Q?78T11R77oSy66xcCM2tXERcZx+mPwWQQbA2IxKp6KPzPTKBPfH8ZxQLUvy1y?= =?us-ascii?Q?xPDu61hLaq4h6eCylmvoKqK+3FKyQyKVTfdxORy9QMvw1x1fIkezVHXnSPo0?= =?us-ascii?Q?B5mtrK37v+F37DG/BwBmVIoHd6eISadB3PSQgpBp7i2ef9zZLjPEKe0eocIC?= =?us-ascii?Q?hB+NYAQypsJTT4kYKymnJTOV33HHWLsDn3paGuRmKsTCDR39s71vA7B5cEjh?= =?us-ascii?Q?S1q2xqH7YwmsATCtJPHMqs1Th/8bjtBI8rPsyzhDjTg2CNjodhRLAx0wti8G?= =?us-ascii?Q?OpLhkxg1ryENDRtWHTSgLpZ6pN0f1Fj7TJsjzZOOsDjMqD95Gsz3670bwML+?= =?us-ascii?Q?sQa8eecSRCwCubOJANQrkv8DSW/F7a32bx6UGy9AdHfU4+Jpr4nzI5R5OChY?= =?us-ascii?Q?/qIY19uJ5UzWQt/574YjzDm2rDRY8FqgnRvy6sm6Xk92ptG79sbcXySqD8nN?= =?us-ascii?Q?o8Iz5PJYdhf1qsIRmfPzZbzA+MKHLUrCEPXv+LLZBPoy1Jkw2HJ+DfT6PIxR?= =?us-ascii?Q?CC3NuBcFUxQ1Ha5DTXgP0fPWxP4jIumaEqUR9dvTAqHxn3p7nLWat5N+J79j?= =?us-ascii?Q?lVVz0J5ZPqeIjJgtjQWDMIJVvQDWh1qqZaoeD6pP7zsVPQU84fQ3ZLIPYmRx?= =?us-ascii?Q?J8O6cyxRXZm3GLcVxyXT/SXboDErVYujWn05Y6swJ43pD76GAxquNkeLfjzo?= =?us-ascii?Q?sJfsoqfJ3goD/2jETGZnq1xIuX1+J+aWvniFFnJqXOqPkbT/cwEl4x71M032?= =?us-ascii?Q?uY9//CxM7qXZ7RJw8vEVUFq2iP6+2SfvCep7f0dHipaiGZVrPxKTEm9sBQw6?= =?us-ascii?Q?WeQAQarNpbkV1tEFA8GGJeD/KBR/TcTEhyVainxplHGV3I0lsMrGHBZH/tld?= =?us-ascii?Q?jSekpQL8OB1ClGBkM7UGAwhCOslV7SmeTgjhQqHaaj4Rq7jPWM2hpa+GojuP?= =?us-ascii?Q?dXvLhygymppL6j+z97Yky9DO3FMSimAN1YHEDVRETAN9gpAc7OWCb1QQ8qGR?= =?us-ascii?Q?TM8kgGBYsI/oyzGmEeNjCi8J/d14ykTguiQdubo4gxfZ8JPrpAyhNAMV5TVq?= =?us-ascii?Q?b7tW27HShho7G99vJ2jGwYUH/nnN3KLaHieqLnrHZEHTzNlnuzlfRKID+4B3?= =?us-ascii?Q?g+mJjLXB+RRDCj06E+5W85rRtviWfeohiwDwM2pLhMtANqL/yLWyw8OrCxAe?= =?us-ascii?Q?p7KmryiepN66uwYOVSNLW++ov7/ExyXJFhzoloZ+ZWy4s9bRFzY4LNvZxGzF?= =?us-ascii?Q?9oY7ezMvWu+++6E6WkltP2FgrfLtk/XW1WvS//yB9Uc5g+Wopd678JzaddZr?= =?us-ascii?Q?0Zco5Vo570WkuYlBO3tfHD5OIjXfhRRbAk6pERJczWYkTiqLUvs3KKsAIDTT?= =?us-ascii?Q?slWJoaI0owfD8tLnNrL+D/6Rg/ibZjFpz7LcmLxl?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB8311.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7f218a3a-fdd2-4a85-b6be-08dc0c904180 X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Jan 2024 19:14:49.3070 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: giC6XVJP1tqQNUuAH7ZwGB6vFbNnU/DG8f/3r+CWcmA7ovQm47KDp75zQ4nzvreb/zsbhF4s5G7ybYXYUxAWnA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB6965 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Stephen Hemminger > Sent: Thursday, December 28, 2023 18:17 > > However, at the moment I see one problem with this approach. > > It would require DPDK to expose the rte_eth_dev struct definition, > > because of implied locking implemented in the flow API. >=20 > This is a blocker, showstopper for me. +1 > Have you considered having something like > rte_flow_create_bulk() >=20 > or better yet a Linux iouring style API? >=20 > A ring style API would allow for better mixed operations across the board= and > get rid of the I-cache overhead which is the root cause of the needing in= line. Existing async flow API is somewhat close to the io_uring interface. The difference being that queue is not directly exposed to the application. Application interacts with the queue using rte_flow_async_* APIs (e.g., pla= ces operations in the queue, pushes them to the HW). Such design has some benefits over a flow API which exposes the queue to th= e user: - Easier to use - Applications do not manage the queue directly, they do it= through exposed APIs. - Consistent with other DPDK APIs - In other libraries, queues are manipula= ted through API, not directly by an application. - Lower memory usage - only HW primitives are needed (e.g., HW queue on PMD= side), no need to allocate separate application queues. Bulking of flow operations is a tricky subject. Compared to packet processing, where it is desired to keep the manipulation= of raw packet data to the minimum (e.g., only packet headers are accessed)= , during flow rule creation all items and actions must be processed by PMD to= create a flow rule. The amount of memory consumed by items and actions themselves during this p= rocess might be nonnegligible. If flow rule operations were bulked, the size of working set of memory woul= d increase, which could have negative consequences on the cache behavior. So, it might be the case that by utilizing bulking the I-cache overhead is = removed, but the D-cache overhead is added. On the other hand, creating flow rule operations (or enqueuing flow rule op= erations) one by one enables applications to reuse the same memory for diff= erent flow rules. In summary, in my opinion extending the async flow API with bulking capabil= ities or exposing the queue directly to the application is not desirable. This proposal aims to reduce the I-cache overhead in async flow API by reus= ing the existing design pattern in DPDK - fast path functions are inlined t= o the application code and they call cached PMD callbacks. Best regards, Dariusz Sosnowski