From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AEB0C430BC; Mon, 21 Aug 2023 08:56:58 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5D4C642D35; Mon, 21 Aug 2023 08:56:58 +0200 (CEST) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2076.outbound.protection.outlook.com [40.107.96.76]) by mails.dpdk.org (Postfix) with ESMTP id 9A6D940A7D for ; Mon, 21 Aug 2023 08:56:43 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JTXmTK/TdcAfNhmyGWJAvDAeJl1wzu/HZGVmkHggUjuBhHupJx5yzSXa1UJokakHIeCIVnmjk8pDzHNd1PAmMSQL4kjfjzxbqiWN1LRPKzZ8RWQrm6yzq4Y1lSdJfd6wv2xBzhNXzjZ3+3BOJBHIbq9713kzsPMjubSFXp/MVSQwjRegwD4b71agB9YhMgg8l/aPH69ey8uE7wDQCcm47yeg/d+SeozlI5Togv6/Rz40FwIfaTsjpuysbmO0lBNHBLvdje+f4CnyTAWRISDIj/dD7K0mmG8PiiqQ07lOQAVarvgCGkWQjCYy59LwR6ZeEvmWkkFEfsf//zH8YSAERw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fSqNr5xgyfyDGXO34CmJ7SphfzcHlCHkE8p852KWII0=; b=asw8HWHx5frh7EnZUWqHqbt9eT5MB45kOUyd8zs+3cnhuIAVIo91oAVQTH2O2RecoCbryUMu1vDNV5pOic3cmoO7iweU4HfE/1VkY8sgmxURC1CW5/UFLNuyHoXNYdwdjFAqjLXA/eO+zBqlkbWnwmBuYTd/cnF020LPWDgfT3WKaeSQYmBYZZcyOBvlGASItfqUtSarYPXSwk2+IC4astF6N4EZzw6asBJPeDcMH6z5FnkumZXKVMTrGqlD8qjg1hiit7p9qRuyrJNaYgFKAvbWnftHSd0i2pYMMo98jdXJU1WjDtmAS/vtfRydRGGmxbNyaI1QkcM+qaG7m1U66A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fSqNr5xgyfyDGXO34CmJ7SphfzcHlCHkE8p852KWII0=; b=FfyGM/EBDHJTJjgfSBSmrBeXfU0sEJRuBIA0lMD9A8X0PQvD9uZ2n0Taoqcy4sSqfBYV6TLSZM/bNB75vTj+BlQB35uKQL+tbmT/hhaO20JDfaY/HvEdk4ZdBvHhV24vIifrvJdgNVtk2z8vy19LNLvGO6V7FSk41Im2KtrT+u9QHFXI+aVe8/cBUNPV7gAedL2Cr49pd5VbKHAw5N1+WQkr2Ev4coornymRe2gQ7210NYoCKJUAb8tToe1G5whsAnVyAkGjcih73EaF83BqwjADlkEP2tZnwoNrmhmW0AO2YuRl/PkV+AUUXvAAl06YJ1iSX72GgYHnxyDrUY06vA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CO6PR12MB5459.namprd12.prod.outlook.com (2603:10b6:303:13b::16) by SA3PR12MB7973.namprd12.prod.outlook.com (2603:10b6:806:305::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.20; Mon, 21 Aug 2023 06:56:40 +0000 Received: from CO6PR12MB5459.namprd12.prod.outlook.com ([fe80::46b7:f479:cba8:9adf]) by CO6PR12MB5459.namprd12.prod.outlook.com ([fe80::46b7:f479:cba8:9adf%7]) with mapi id 15.20.6699.022; Mon, 21 Aug 2023 06:56:40 +0000 Message-ID: <1ea7fb69-0a6e-4e61-843e-127f3ccd35b3@nvidia.com> Date: Mon, 21 Aug 2023 14:56:32 +0800 User-Agent: Mozilla Thunderbird Subject: Re: MLX5 PMD access ring library private data To: Honnappa Nagarahalli , Stephen Hemminger Cc: "dev@dpdk.org" , Matan Azrad , "viacheslavo@nvidia.com" , Tyler Retzlaff , Wathsala Wathawana Vithanage , nd , Konstantin Ananyev References: <20230817070658.45576e6d@hermes.local> <1fbe4f48-e652-4887-8949-ad9cd15e2c26@nvidia.com> <7fd66332-a7cf-4947-ad5b-e1af0c5148e0@nvidia.com> Content-Language: en-US From: Jack Min In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: TYCP286CA0229.JPNP286.PROD.OUTLOOK.COM (2603:1096:400:3c7::8) To CO6PR12MB5459.namprd12.prod.outlook.com (2603:10b6:303:13b::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR12MB5459:EE_|SA3PR12MB7973:EE_ X-MS-Office365-Filtering-Correlation-Id: 299a0b8f-2046-400e-e938-08dba213c589 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: PCklvd3B/u4mqt+d3YkH8Qe/SmzD84BxrcFyo6Xeh6y70e/YdSVPOC9WeCc+vS5hbBPnJqHxL8h13rlaAJlbBbVEFR+OpuAV/GElpyeCggyN0KKoleV7Yp9dBJRjppMkWravccFVK7oWVA4U7TFHbvw1pVXLNSLm6c++Jmw2QCj/jIcdgyKX7b6nQmZQLZS+YkI6DA3UvYrraGcMqyjaqR0ueSpXHLRly8+WGgfmlDyEbPapb8LEiP3vRhRVO2M3IwIc4+f+5TspqooQAFIlAg5jifz/wjnqUGYzO+VWiQnu/XgftKBoIRLqfGEb732oLSpb3oR5hsJj3iNUnKQUCCG1O5yyKdILNwkxe5u8+5vWjxqFYYXgUtRVvrlO6fdjmaFBr6uWOCD9OAirI7DWR1aa1jD7FPEa1tUz3hEf2eF5eMPoX+9UY/RZle5zn19Xz9g3n6VaI/nvCK6Dos/oZPQX6ReCI4ekZZVqFhvNRASykxZ6sZniH1c0H7i5rbiYuSKbmbEVZBNnOlREg+qZadW7bNc5N0eEMZZ1jejPeU7fGmuQzq6wDF2obCuKtWHs/p2XJwYTrGT8vyFkEDDEdJsYP6mW00IkdBHnatEwWuZuryH/PuxHy144kjb/Fb6q1dZmEtA/chFLmdRbo0Z9reQcM3z97+jaQAoBY0QT3SV2CM8b6L94l9Rl4ceUpmP4 X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CO6PR12MB5459.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(136003)(396003)(39860400002)(376002)(366004)(346002)(186009)(1800799009)(451199024)(66556008)(54906003)(66946007)(66476007)(316002)(6512007)(110136005)(8676002)(2616005)(8936002)(4326008)(36756003)(41300700001)(478600001)(966005)(6666004)(38100700002)(53546011)(6486002)(6506007)(2906002)(83380400001)(86362001)(31696002)(31686004)(5660300002)(26005)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UmxnM3VWcDU0NC90aVdqL2tWWkk3eE9XcGZFVXI4bzN5cWRQUWwrc1ZzaXVx?= =?utf-8?B?Z0pjTmdsY0xEMkFaQlAxSVc1c1J6S2ZmMU5QWUM0a2lMWlU1M2pHQnF1MXM1?= =?utf-8?B?OXR0VVFJME9FSnFTTStVLy8ydDdQSkhyNDdMK0dvUjJERlRSbXJXSWJncHlE?= =?utf-8?B?a2VyeWdHc0NuYVlSMXA1Rmxsd2lJL0cvQmpHZ0dRSEdNbHdQWTVVbTVhWUIz?= =?utf-8?B?bVdzczFadnhKOHgxR1RSR0p1UVBwbkVlMDdxaU5oVUZpeE5PRFlGdDRqaFZp?= =?utf-8?B?YUhWZzZGNi9VQXErUDZIR0FMOTZob2Evam94VzNQZkEwKzgxU2dVd3lJRXpZ?= =?utf-8?B?K050dXBVRnJHdFZyMHBiWGFZOTRuTUlOczhEdDBId3o5RVM0WHU4N0RKNlNE?= =?utf-8?B?WldlRjBiMjlQQng2SEpMbWVvNUNuQmxIZ1A1WWJZbjZyTDgvaG5mdkwzZWN5?= =?utf-8?B?bzdXRSt1S2U3T3J2cGNyT1padjNzRi9rWlN6a01xck5uTlZLUStjdElwUDZW?= =?utf-8?B?ZTE1TEsxOEllQ0M1Yk95azZVUjhOeEdta2Y4NkYrQWpVSEJ6RUtrOVFhNW82?= =?utf-8?B?a1BuQVdGSmVJVE5vVTlCNDJvWFhqMktEenZlTmxCZmk2OWNOZGJmbHQzbWNs?= =?utf-8?B?VFVZbklrNHVLUGV2REJBWWhlVlI2aHB5enc3TWlxSkZjaklJKzhwVmFDZU5z?= =?utf-8?B?N0tmS2ppV2JFUG1SeFhPcjE1SDhrR05JTU96UklJeWZxUlU4VE93YWh3dENW?= =?utf-8?B?Lyt6TmZwa1pUNCtrTGtVbVF5N3ZTRHE2SDErbkVtSnlyVzlTWWt0WDBCaTBU?= =?utf-8?B?MmxDcFFkZnpKNlhCZ3czaFE5SFhMaVluaUYwc3llcXRvQWp1RU4vZWEyNlQ0?= =?utf-8?B?ZEdPNGpTa2ZjMGZmUlJrV0NLNUR2U0wvdElPaW9uYlc3eEJRb0o3TjF1Mnda?= =?utf-8?B?b2pBUTdEVk1SZW1vN2FLQTdFL01UY1JybitxbDdWc2Z1UHI0V0VDaVpvUGRV?= =?utf-8?B?Mjk4c0lmUkkySVZGVTJmZ3lGYjg1d1BROHJPLzI0Q2RpYVVuUVdEYnQ1WjNx?= =?utf-8?B?VTNDU0xzZFdMYnlwSjVkV0dYTFVTTVFtQjI5RGhzZk5yS2Q0dXJzZWxkZ2Zj?= =?utf-8?B?d3FGRFY0VW1KMnFTQkdVRnFoVWxkcWd5c1pnNXh0NlZ2bE95UzYxK1Uxa0Vx?= =?utf-8?B?MlRTT3lDTzYvdURhdTU5WHBLSHVkUEsvdlpXU0NFVkUyN2dWcjFPcWVNNy9x?= =?utf-8?B?N1ZLRDBDUjk1dnV5NzBrSlQreGpabzJtcytQekpNOHdYVitkbTV2QUVQRGpj?= =?utf-8?B?eHUxSmJicHhWTTl5MWNTbTJMeGF5TEhjVG5LQmh4RWJNZjNuc3cyWHFmTVdG?= =?utf-8?B?VWtSY3ZoSVFpWEVOU0dJTlY3RHlraFptN0JEa2pOczQrRFdSTGZmVDMzcHBk?= =?utf-8?B?V29OVkdJZ0FwZnNZWmV4d2pmZldzV0FoRUdBbDgxU1NOdmNUTDNnTE9RSkZQ?= =?utf-8?B?c2dDN0I2OGY3ZXJZbUdLRWM3Q2JGNmFHQ0RkSlVEYkFDMmZDeDdTU2h4SWho?= =?utf-8?B?MjBSVGsxbDF0VmhUWkRzdTlUenROYkMxbGhObEM2QnV0T1U5TzFJRUp2QktZ?= =?utf-8?B?a2tSWFdlTEZCRGs5VGhzdzQrSTlMNkNBZkg0VGFjL1lIaU50WVhEQ05hL1FR?= =?utf-8?B?M2hjWW9lN05QZHNGR2lWWlk3VGVMNEpYVGxjaHJNV044TzV4Q3hraGlNSzRt?= =?utf-8?B?VkQ4RWR5VHNSZXhsRVpGemhRaDRINzErUTE3OVhvL1kybDZLMXd0aVU1bTVO?= =?utf-8?B?SUxKelp2U3R5MlZ2Y0d1Ykdtdkx2RUs1bXBUSzlRVEtXdUU2Qk5GYTcyRWpy?= =?utf-8?B?emNsZ1J2dFh4Ull2b21DVWxmSlhoa3ZjWjU1UGZCM2FYM0RtSkhDRTQvZ0Rr?= =?utf-8?B?cHMyOWZPYmNZTGIwejcyZ285TUtsa0NIbnJmRUkxcTNOTEJLUDEvQkFhMEc5?= =?utf-8?B?blVrUVFhd092R1YwL0NrL2pDbVZVem40MDNSaFR1MUx6b2hlOGhpWWhyaE0v?= =?utf-8?B?WUpyRDJiWmM4ZEdETEpPLzJ1QlNZeEkwa3UwSGhnN0dHajVLYzRmb0FCYWor?= =?utf-8?Q?SbcwC66sDb3Pz8hzna4BbaDAq?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 299a0b8f-2046-400e-e938-08dba213c589 X-MS-Exchange-CrossTenant-AuthSource: CO6PR12MB5459.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Aug 2023 06:56:40.6935 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MGJVK8Dbxwdcf7gnUaZZIa35j9ly7kswwqoCKQYGfvazdoKOZmvvF4Cl6IOQJ0EDgQZge5HMm2YpEvnoQtVPrA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB7973 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 2023/8/21 14:06, Honnappa Nagarahalli wrote: > > It would be good if you could fix the email on your side to text > format. Comments inline. > Seems a wrong setting on my email client. Sorry. > > *From:* Jack Min > *Sent:* Friday, August 18, 2023 8:35 PM > *To:* Honnappa Nagarahalli ; Stephen > Hemminger > *Cc:* dev@dpdk.org; Matan Azrad ; > viacheslavo@nvidia.com; Tyler Retzlaff ; > Wathsala Wathawana Vithanage ; nd > *Subject:* Re: MLX5 PMD access ring library private data > > On 2023/8/18 21:59, Honnappa Nagarahalli wrote: > > *From:* Jack Min > *Sent:* Friday, August 18, 2023 12:57 AM > *To:* Honnappa Nagarahalli > ; Stephen Hemminger > > *Cc:* dev@dpdk.org; Matan Azrad > ; viacheslavo@nvidia.com; Tyler Retzlaff > > ; Wathsala Wathawana > Vithanage > ; nd > > *Subject:* Re: MLX5 PMD access ring library private data > > On 2023/8/18 12:30, Honnappa Nagarahalli wrote: > > > > > > -----Original Message----- > > From: Jack Min > > Sent: Thursday, August 17, 2023 9:32 PM > > To: Stephen Hemminger ; Honnappa > > Nagarahalli > > Cc:dev@dpdk.org; Matan Azrad ; > > viacheslavo@nvidia.com; Tyler Retzlaff ; > > Wathsala Wathawana Vithanage ; nd > > > > Subject: Re: MLX5 PMD access ring library private data > > > > On 2023/8/17 22:06, Stephen Hemminger wrote: > > On Thu, 17 Aug 2023 05:06:20 +0000 > > Honnappa Nagarahalli wrote: > > > > Hi Matan, Viacheslav, > >       Tyler pointed out that the function > > __mlx5_hws_cnt_pool_enqueue_revert is accessing the ring private structure > > members (prod.head and prod.tail) directly. Even though ' struct rte_ring' is a > > public structure (mainly because the library provides inline functions), the > > structure members are considered private to the ring library. So, this needs to > > be corrected. > > > > It looks like the function __mlx5_hws_cnt_pool_enqueue_revert is trying > > to revert things that were enqueued. It is not clear to me why this > > functionality is required. Can you provide the use case for this? We can > > discuss possible solutions. > > How can reverting be thread safe? Consumer could have already looked at > > them? > > > > Hey, > > > > In our case, this ring is SC/SP, only accessed by one thread > > (enqueue/dequeue/revert). > > You could implement a more simpler and more efficient (For ex: such an implementation would not need any atomic operations, would require less number of cache lines) ring for this. > > Is this function being used in the dataplane? > > Yes,  we can have our own version of ring (no atomic operations) > but basic operation are still as same as rte_ring. > > Since rte ring has been well-designed and tested sufficiently, so > there is no strong reason to re-write a new simple version of it > until today :) > > > > > > > > The scenario we have "revert" is: > > > >  We use ring to manager our HW objects (counter in this case) and for each > > core (thread) has "cache" (a SC/SP ring) for sake of performance. > > > > 1. Get objects from "cache" firstly, if cache is empty, we fetch a bulk of free > > objects from global ring into cache. > > > > 2. Put (free) objects also into "cache" firstly, if cache is full, we flush a bulk of > > objects into global ring in order to make some rooms in cache. > > > > However, this HW object cannot be immediately reused after free. It needs > > time to be reset and then can be used again. > > > > So when we flush cache, we want to keep the first enqueued objects still stay > > there because they have more chance already be reset than the latest > > enqueued objects. > > > > Only flush recently enqueued objects back into global ring, act as "LIFO" > > behavior. > > > > This is why we require "revert" enqueued objects. > > You could use 'rte_ring_free_count' API before you enqueue to check for available space. > > Only when cache is full (rte_ring_free_count() is zero), we revert > X objects. > > If there is still  one free slot we will not trigger revert (flush). > > */[Honnappa]/* May be I was not clear in my recommendation. What I > am saying is, you could call ‘rte_ring_free_count’ to check if you > have enough space on the cache ring. If there is not enough space > you can enqueue the new objects on the global ring. Pseudo code below: > > If (rte_ring_free_count(cache_ring) > n) { > >              > > } else { > >              > > } > > Hey, > > Then next n objects will still enqueue into global ring, not into > cache , right? ( we enqueue nnnn objects continually) > > Our requirement is like this: > > if (rte_ring_free_count(cache_ring) > 0) { > >          > > } else { /* cache is full */ > >       > >      > > */[Honnappa] /*Understood. IMO, this is a unique requirement. Ring > library does not support this and is not designed for this. As per the > guidelines and past agreements, accessing structure members in ring > structures is not allowed. > Alright. Now I'm aware of this. Do we have a document about this? I probably overlooked it... > I think a simple implementation like [1] would suffice your needs. > > [1] > https://patches.dpdk.org/project/dpdk/patch/20230821060420.3509667-1-honnappa.nagarahalli@arm.com/ > > } > Yes, mostly. It will be better if we have a "zero-copy" version, like: rte_st_ring_dequeue_zc_at_head_burst_elem_start() -Jack