From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 27A8A43ABE; Fri, 9 Feb 2024 06:56:10 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C7441402AF; Fri, 9 Feb 2024 06:56:09 +0100 (CET) Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2084.outbound.protection.outlook.com [40.107.212.84]) by mails.dpdk.org (Postfix) with ESMTP id 15F4640293 for ; Fri, 9 Feb 2024 06:56:08 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=R+I/pvorHxDeUGjFOOudDhMrLxSbB2XOKkI2qh1BPBHE5vVBk5gX7uXvUUpFwEzvX+d+QqKmx5c5xRJYVM2itgdhdX+8aU6nUJbrlgtLTCI03RGmd9Yz4xf2RBp3cmwRymDbGDdNBG2wIV2/PCF5IxHILIv8xR6AhkRxtfgNNNmHWq3eK9KL92qVHxklOOC1URxZ2T+gVEkk5x/aH2hXIDS889DgAKEdJaXg6VlcVRiHBrgegnaT1shQg28JDHA3i+OBLPZ65EWV4JBlVAI/T3boolr9dpg4//7DU9NKimaYr67SgA6G00apfPwQFTRR8s6BYo8SWkRWHus8MrG4xw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9SxR9O/nF81EpUo+ld/yfnd+ceE84KSYdyC4mRNseaE=; b=ZQUoTds1wJ4NmDfqhyWgiXRmif9rh7/Ie3E3Dl1/XQEPHfVd/XBpOHAT9VYwFot2Gy+7mQBIH+/SB9b0mo07N0b7i9/qOlMA+hg0R7J8+/sKCYuU5aKEc/fsteRvOiox2JxVj20OQtBngLnZxHgQ2vMLXEq1TEhLXcfklRxc2T15lkeOXSvc1zjLNT/BaV4VuUm6fBe2/mUCyevOocVTVHxcTNz8s8djTNbtLHJ5Uw91kqgw+4YscEOJv6b5y4N0AcUZmO47TRciktBhUdnVzV8bpEANDdAmPaVwvM+rzWIeXRF+JqFvbFmJyQZszJXuh15Dd6kkoZDf955ZmgCm0A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9SxR9O/nF81EpUo+ld/yfnd+ceE84KSYdyC4mRNseaE=; b=ISuhAOUjs1BiIf8O3swKt4V3bhTpJL6V8K5tYdVVY/EXiwrag2tHLdzgkZrN+NWE5PkVV4mvnIhNDkG9uknRh45ysa36NOFSEnpBFjuGaUHWejX6ls5tMZjvZ1haR59SwjIcip0Lip9qEDzb5JdecGRFkw4I5GN6FCRKGDmSLVZ9IGTHReY5O7S0slub+QQSMhr8UR7r5w+WDs9PU4wpsPKJqulNvaMTHkQjVy+3SdByY+GlfO9hK3abMDTHUSQ1vI6vB206wr8m8fuo6rrb19i2pqfOm+dA3WLFS+PjtXuEoB76/x0IyuSMwHnPO0JsnlJpdtb32EH/1ldQ9a+tKQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from IA1PR12MB6332.namprd12.prod.outlook.com (2603:10b6:208:3e2::13) by SA1PR12MB7341.namprd12.prod.outlook.com (2603:10b6:806:2ba::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7292.11; Fri, 9 Feb 2024 05:56:04 +0000 Received: from IA1PR12MB6332.namprd12.prod.outlook.com ([fe80::5610:835d:9e24:6cfd]) by IA1PR12MB6332.namprd12.prod.outlook.com ([fe80::5610:835d:9e24:6cfd%4]) with mapi id 15.20.7270.015; Fri, 9 Feb 2024 05:56:04 +0000 Date: Fri, 9 Feb 2024 07:55:59 +0200 (IST) From: "Etelson, Gregory" To: Ferruh Yigit cc: Gregory Etelson , "dev@dpdk.org" , Maayan Kashani , Ori Kam , Aman Singh , Yuying Zhang , "NBU-Contact-Thomas Monjalon (EXTERNAL)" , Andrew Rybchenko Subject: Re: [PATCH] ethdev: add template table resize API In-Reply-To: <5bafdaf7-2d8c-41dd-90bd-043c54ff1d7f@amd.com> Message-ID: <4bd8d9d2-4c52-2af2-27f6-925000444c4d@nvidia.com> References: <20231217093205.321082-1-getelson@nvidia.com> <56337607-a330-0967-e99c-32a6f6e9f058@nvidia.com> <62e78a2b-4827-030e-713b-28d671b2dac3@nvidia.com> <8e25176d-9438-3ca5-e862-6d3cab7ef377@nvidia.com> <5bafdaf7-2d8c-41dd-90bd-043c54ff1d7f@amd.com> Content-Type: multipart/mixed; boundary="8323329-2131945933-1707458164=:115611" X-ClientProxiedBy: TL2P290CA0005.ISRP290.PROD.OUTLOOK.COM (2603:1096:950:2::15) To IA1PR12MB6332.namprd12.prod.outlook.com (2603:10b6:208:3e2::13) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: IA1PR12MB6332:EE_|SA1PR12MB7341:EE_ X-MS-Office365-Filtering-Correlation-Id: 786c9856-0826-4f4f-2a92-08dc2933cd44 X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: a6Zemp9JSkTH2tJc+jvZz0B7cvg4eFxlWdFdfRHFSEuCyOJ8pFr4PwcAeGDbsAH0Ir3vFR+Q/neDk/5ZlgfZQJbc30ES3621Lj3jOHWz3u+JCurA20j2GjglIX1okl+NsdDguB37BaYL11ddLm5ZnzdS0bQbV5nbds0e0TmZzRE/PesElisjqXD2k+FWOsumHw70kmu1Bis4bVqCplOgsTfYpkEYhcgUpS2v6pwbm2EokftoJipnRprqofc3sc/4SRTyRnDNSPQ7onTkNCaSAhdcRPEKWpRV1Tt/MM3BGSNud80xYi+7b8Egw4Wl0x/Iq8KaD58RdxsuTlC0TrB7vH98K71jPTpixaQwEyYBweicwtYH8Py1xmeiqFxo4qz6WZ7/78mKTEXhCrrfKPw/d1HgXs2AxUuE7RnZ21xIyIHLMkHh6IfU3TZ01d4+3QgJUXWyJIhsTZq7INn6qifriIVTiGwb/4E1p3aZ0KATyq7aSXJpXNQTb53mLAEItwPnbkkCYLzTvgi0wb9B2WujoBqgIX++K9aB4iOAKGJm/5g0oPVqfTnob74tmS2FyPmn X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:IA1PR12MB6332.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(39860400002)(136003)(346002)(366004)(396003)(376002)(230922051799003)(451199024)(1800799012)(186009)(64100799003)(2906002)(5660300002)(38100700002)(31696002)(86362001)(6512007)(2616005)(6666004)(6506007)(33964004)(83380400001)(8676002)(66946007)(316002)(54906003)(6916009)(66556008)(4326008)(36756003)(66476007)(6486002)(478600001)(8936002)(31686004)(41300700001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?d1VoaUk0M3Y0UFBIRjRwYjB4TGN1WjVvcE9ES2hYL0tQU0o0OEpkNEdvaC9U?= =?utf-8?B?aWRjYVRpQXNndXdEQVpPbUZIMUd2dGZwYlRvOTR1Qm56QXpYTlRCV0g1SmxF?= =?utf-8?B?YlM1MzZDTkVTM3M4QXd4NUpQRGJxemU1SnBOWFFzYmFodGFCRnpaYUFtS3Fr?= =?utf-8?B?dXNkUGlrU3NvS01QLzE5cHlGUXkwTWR2c1RNN1huL1ZRUjgxV1dvOHJVTXYv?= =?utf-8?B?U2VadVVlSmszL0pUdDRPa0t5WG9jazhCUlZvaDZRZ0IvQmc3YWF4cmZ2K0Vt?= =?utf-8?B?L05qc0JaUWNMREFlNlp4MTNnaksyUlBIaEdKc1ppZ3VRU3M3UTRRY1lYbTRm?= =?utf-8?B?V09wTDRsaHZyZWlRTHRJNGtmajJkT3JwS2ZqV3Q3eTF1Ym9qY1BPQWR3RFMv?= =?utf-8?B?a0t2MUMvR0dnNUxRYk4yTzVZVGtmR2s3TzVoTWdRRGQ2SGY3cEhtMVpLY2hx?= =?utf-8?B?ZlJnQk10endYNzlMY2pTV083Z2pXSFg2SWpya2tVQWJKd2hsYnUzaS8zQ255?= =?utf-8?B?TlovYm5VcVprRXEvc251eE53eW1sMG9CeTVvTjd4SVpxVzVKVC90clBSUmhF?= =?utf-8?B?VkhrZkJIeEFNYS94cktHZnVWZkpYN2l1ZDBHSkRXWEZsNjQ5Q1JvSld1NTZI?= =?utf-8?B?MUVKYUJMU0R6M0JRUzBTWkFITXp6ZUZFQ2NYbU5IMlFBdG9nQlJ4R3p0UE52?= =?utf-8?B?OE9mOFZ5djl1QWg5Ym5CNncxNDBCQVNiakZRNGdybDhwNUdpaTI5ejVhVFB4?= =?utf-8?B?cTFDVFpmUjk5RUNuWFNUL0xCTUQ5U043dXdPSG9jUlFHWFVXdlhXTUJ1TlFQ?= =?utf-8?B?bUkxbW52bDhhMEpBdy9vRENTekRDSVRTVUNMUDI1QmdtSTVYZ0JIZEp6NVJz?= =?utf-8?B?QTlOakFqWHRJY3BSNngwbER4SUdGL042bHIvbVJEaDl4aGFPamswV2hwaHIx?= =?utf-8?B?SExtKzFLcDlkbTFLZ0dXQzdQZzlDUzNrdVprKzYvbk1JTTloakhuMDQ0N3VF?= =?utf-8?B?NTI4ZmJkY0tsS1JvcFc1QXcwelVuUGxDdFFNcXc1R0krL09BRGVqUThBdlVl?= =?utf-8?B?MGVyRW5sZ2pjRUZXMXBRQmpPbGhvd3FrVjdMNGZibjk4a0dEN01TU3EvR2pn?= =?utf-8?B?YmptK2RxWnhpaFFXSWEzTnE1cWtkQTBjWUZsWndURGZiOTB5b3V3dDdKdWlh?= =?utf-8?B?VW94SWpTSHpDcXpVM0lVdGEzWkFBMnZ1RldoOE4xVlhTemE1TU0wY0JXRFhD?= =?utf-8?B?NWd2NVZkNDhUNURpRnBSaGhLeWd0MFNwQzhra2d2ZmpId3JBNnVlRml4bHZo?= =?utf-8?B?UHBZRyt6Rm0vQ1BqMnphZ21XTE9HelJTZDZRYWFYa2YxZUcxNFBqWG10aXY1?= =?utf-8?B?amhENXJuSjcwd29OeVhUREk5N2dLQkZxMEFoMXZjSi9kZ1BuZ1JhTU5WTUUv?= =?utf-8?B?ZXppSVNsSEVMTDg0VVhLVXd0RHdTY1JEMGNjbWdLQlBSNGFPUmZhVmdId2cy?= =?utf-8?B?aCtKNEUvU2hkYjFiaFJ2dFBCUlVrTDFOQnYrOU9GU0VvcnFRd1ZHSmU5Y1pq?= =?utf-8?B?SkhMRXNPUUk0eno3QmlMVG14UGpFWWw2a04rTlFCcVY4UHBWZngrZ1dRb1J6?= =?utf-8?B?M0QvMWtIdVlzei9ZOWNVenlIRHFYVU1hWE1OM21mNWp1OUJmU3o3OXRKcHpp?= =?utf-8?B?bXN1MVRiak5wVE0xbGU3QXc2aGMwcTRlNnkycEVEeEtCcTg1WGVOckZyUXhi?= =?utf-8?B?Y0RDdGtYcldkUnZaWUdEK3ZDZEp0ZUkvcjNjRFdGYUZWZlZLWCs2UEFRVHJX?= =?utf-8?B?ZTJLeGNFbDZsTzluUXFhaFAzNW1SaTdTNytXYnpQMWZrOU8xZ21LVUVhdEda?= =?utf-8?B?QlRlSXViQmdGRS9GZDBKRW9PeUV1blZZK0xuNzBlcHZ4RTMyRGJYSWRMcERJ?= =?utf-8?B?NkZneElvQWxhYWd5QTI1QzNKT2puTWlNMUxuWE9RYm1iSGUzbUI3dXFEK3NW?= =?utf-8?B?aDZZZXNua0Yzb3JmSGVGWVFZeUFlellZWEJFdDQ5Qm1WamV5VmU3RjN3RUZI?= =?utf-8?B?NC9wOWY5ZHRxbXVOL0hNcjlSNGdRU2d5enY5bUxaOFUzbGNSN1lBcUQ4d1FV?= =?utf-8?B?YWhla21EcEJ0MklDK281Z2lUYWN6QUJoT0FjcDIxZVNROFhETTY5NXd1d3pG?= =?utf-8?Q?S9oOuENYqyGexeTrvfnnWHO55Rpi8aw/6DeCjMtMTZIT?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 786c9856-0826-4f4f-2a92-08dc2933cd44 X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB6332.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2024 05:56:04.5122 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6p+ViolUaW/2oavxkBmXsWx3zlWNztpwzOHNNUwdSEj35+uW6MKcF86U5TaDAsXxKFHhBSZk3rqVzufGaVf3DA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7341 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --8323329-2131945933-1707458164=:115611 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8BIT Hello Ferruh, I work on the patch update with detailed user guide for the table resize API. Regards, Gregory >>> So, by design, driver will keep the old table when it is resized. >>> - Can this have a performance impact, like when rules >>> updated/removed/inserted driver will need to look more tables? >>> - Or can this cause additional capacity complexity, like total number of >>> rules will be sum of rules in all tables, but new rules only can be >>> added to latest table, so number of rules can be more than size of >>> latest table. >>> - Or user can add more flows after resize() and this may not leave >>> enough room to update old rules to new table, what is expected behavior >>> for this case? >>> - Or if user did not updated rules at all after resize(), after each >>> rule deletion driver won't need to check if old table emptied and needs >>> to be freed? >>> - Or can user call resize() API multiple times, causing driver to >>> maintain multiple tables? How much memory overhead this may bring? >> >> After "resize, update, complete" sequence table performance must be >> the same as before resize. >> If application skiped updates or resize completion, performance is >> undefined. >> > > According below update, it is expected some time will pass until user > updates flows, during this time there may be some performance impact, > does it make sense to mention from it in API documentation? > > >> Driver must verify that total flows number does not exceed capacity set >> in table resize. >> > > OK, this is more complexity to driver > > > Is multiple resize() call supported? > > In the worst case, think about a case, for each rule application first > increase the size by one and adds that rule. Is this supported and > should there be a limit how many resize() can be done? > > >>> 'rte_flow_async_update_resized()' API is called per flow, won't this >>> force application to trace which flows are created in new table and >>> which are in old table, so pushing additional work to application. >> >> Application must trace what flows require update after table resize. >> As the general rule, all flows that were created before table resize >> call has returned must be updated: >> >> "old" flows |<-----------------resize------------>| "new" flows >> update keep >> unknown flow location: update >> >> ----------------------------TIME--------------------------------------> >> >> In MLX5 PMD, if update was called with a "new" flow, the call returns >> will success, without changing the flow. >> > > At least this makes life easy for the application, can we document this > behavior as API requirement? > > But still in a case there is high amount of flow insertion and deletion > happening in parallel, how application call the update(), it may still > need to keep list of flows to update. > >>> >>> Or what will happen if update() fails in the middle of update, should >>> user retry, should PMD restore back the moved rules? >>> >> >> If flow update call failed, it treated as failure during flow create, >> update or destroy. >> >>> >>> I understood the logic behind the dividing responsibility to multiple >>> APIs, and it makes sense, but it brings above complexities, and more >>> work to application. >>> Can it be possible to have monolithic API but only resize() part of it >>> is blocking and update() part and later remove table part done >>> asynchronously? >>> >> >> Table resize and single flow update operations consume approximately the >> same time duration. >> > > Ah, thanks. I was missing this bit of information. > > >> An update of a table with 1_000_000 flows will consume driver for too >> much time. >> During that time application will not be able to create, destroy or >> update existing "old" flows. >> Such operation must be coordinated with application. >> > > Got it. Briefly I was trying to highlight the complexities that multiple > APIs bring, and responsibility pushed to the application, > but above performance impact explains the design decision. > > Eventually application needs to get this hit, how application expected > to use these APIs? > First call resize(), after this point how application should handle > updating flows? > > > Can something like async updating flow work? Like driver returns success > but it sets a thread that in background moves flows in a loop, if it > gets the lock and release it per flow, this lets other thread get the > lock for other flow related operations? > >> A driver could provide a batch flows update, but I don’t see how it helps. >> It's ether update one or update all and the latter does not scale. >> >>> >>> I will also put more comment on the patch based on latest understanding. >>> >>> > > --8323329-2131945933-1707458164=:115611--