From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 201A3430A8; Sun, 20 Aug 2023 03:41:20 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 96853410D7; Sun, 20 Aug 2023 03:41:19 +0200 (CEST) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2043.outbound.protection.outlook.com [40.107.220.43]) by mails.dpdk.org (Postfix) with ESMTP id EEF1440A80 for ; Sun, 20 Aug 2023 03:41:17 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gDyWiGMESnRDYjqfTDujaLOs1lf5LY0G/0yqUqnb18V8KdIACBXmIAKZ/GzEsxQtObfu5iSvk8/9cPWyO9CzvXgif9Qi1EH7CfYE7hNLR9oZF/i2DV6QYP0TVmd4ZuWGjVG6DwmRnBZleScCTqOplGqJSA+z3dpDS0e3lt0OXcpZvic/XEvJlawZHOEBGkwIdbicL98bcJkCGQ8futkw26hD4XjwY7eyLywK2wRVR3VOjTEpjXtpIdUWjhlESMEF58IcBcOZWDFOXW17BboDqbZ7FGOUwGTSAW6slbDrunm0/nLCr/xEjua5LOBgc6sHvzPDNNH7eocdxXMok0lAxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2nz3fFJ5dyGHzX3AIEKJZSRm07SOHY3OeK0VWm1Mse0=; b=bGEQAWj7zLBt2AHRDLRoNoXntUUUuCZVAhuntJe7BvgDOtyZCmXPMZF14kof8DKfZOV+Xz3scd+DMkLnxGIhBs+xJ6VehWnQJBC5Rc7UiWA8kcvFqDBnwXdN45V7IVDjG7nX3aDvfXadl/CHgq6etyikBubJWwoMVJFdQrF0ygF6htYzfOvuoB1icly2OrCoHxdt3HmbaNILMxW4h3Q2hwIAIy2ib+lvBRpDWdUWr5a5C2v8+1zU8Z2sqjtK2in7HovMqlan7XzzzkkK1jmSV773Vjpq0421lVHdn7VUiSKmcgwAkVYl7TSc/CmoggeiIBbv7sR1wrOrmXZ0/lsfDQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2nz3fFJ5dyGHzX3AIEKJZSRm07SOHY3OeK0VWm1Mse0=; b=cB2uUmW5s9xDxSBL7+hH4pJk13oEyuftuyk7V4Qf4YhsHRQflV71mNnIfrdxgiA8wgMeGsS2Mxc9bQ8X1NEru0X/M+D/fePO6MJnDlSlCxF3mCTTVodPYLZEqWeW20Xk1yF0lo/shZgpg0UWsfmZRMJCxkIorxC+XcBK7sp55OqpqAh/28bi4v8Jq9CBfOpzaXfG1vDsgLtVfPbQwKXL2IzMvpgmG5urNHnC2CWUdDRjQqvoNwQvQw2B9JthSqxvV6IQXInVa/QefuT8unHb9D1X8VoW6mfmJIohM8ejOt9Wb4uYGKt/Zfs1pMb5d4L+CMKKQ2b+JsvszMt4/E9LCg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CO6PR12MB5459.namprd12.prod.outlook.com (2603:10b6:303:13b::16) by MW6PR12MB8833.namprd12.prod.outlook.com (2603:10b6:303:23f::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.20; Sun, 20 Aug 2023 01:41:14 +0000 Received: from CO6PR12MB5459.namprd12.prod.outlook.com ([fe80::46b7:f479:cba8:9adf]) by CO6PR12MB5459.namprd12.prod.outlook.com ([fe80::46b7:f479:cba8:9adf%7]) with mapi id 15.20.6699.020; Sun, 20 Aug 2023 01:41:14 +0000 Content-Type: multipart/alternative; boundary="------------lk0v5RNy0z2i1XDg6pA9OYA8" Message-ID: <13740d86-ee3a-4a9e-8d00-8172ed680970@nvidia.com> Date: Sun, 20 Aug 2023 09:41:06 +0800 User-Agent: Mozilla Thunderbird Subject: Re: MLX5 PMD access ring library private data Content-Language: en-US To: Konstantin Ananyev , Konstantin Ananyev , Stephen Hemminger , Honnappa Nagarahalli Cc: "dev@dpdk.org" , Matan Azrad , "viacheslavo@nvidia.com" , Tyler Retzlaff , Wathsala Wathawana Vithanage , nd References: <20230817070658.45576e6d@hermes.local> <63206979-911f-439b-816d-ee5c1c67f195@nvidia.com> <9850f8f4-bc5b-3af7-c538-99a26aab6d9c@yandex.ru> From: Jack Min In-Reply-To: <9850f8f4-bc5b-3af7-c538-99a26aab6d9c@yandex.ru> X-ClientProxiedBy: TYAPR01CA0175.jpnprd01.prod.outlook.com (2603:1096:404:ba::19) To CO6PR12MB5459.namprd12.prod.outlook.com (2603:10b6:303:13b::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR12MB5459:EE_|MW6PR12MB8833:EE_ X-MS-Office365-Filtering-Correlation-Id: ab2b2ca9-d298-4dc7-0ace-08dba11e8a1e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: P4Uvr2AJRPJnMKFAs8rUhRAsjJRkmk8g3SbAaHAWGJftMNJM6kjd5uoo+R0bDrIru5+Vv8U2jG0ugmegb8eOt4b1UWqWvn1EPvxW+wr7/OAXbAYhietMSAFXQ6hpXiWQvVuJLTdDtaDE13z6hWNbe+GISyP/jVy0jbI+afqBOB3x1xlmNC8XrStmETozij09aM5aUochuCYoRvHVCVk9ws92aCqBm6JRM9XnoDZW3AxxdRd1blbqjo06lLehsY5otkh8QD68ZDtfAk/gNghyOcFDXjRBQZLz4h/ihAmEttK65ipq7OHZEY4mhkGEG/JgmmPMvBbu4xMUcL+RqJ2bNYUP2+t9bE8j+jF0lLlAGknjlBrTMFOAGB90clITtxh6jT8s8B/uHd9AM9W0oQDR0ezVZFMRJTRCOS+P6eivMmKWCbBv0VOMwqKtMr2V02RgYXDsKM7kwmHZvHBIZ8dNoEHtdAgqAoGLHDBzk+KAggI7UIBzZC+RVQMczBD773Wu8JMaKqlcYEr9RksI8Lk7yb75dI0vodhiAQTf5c80QjaZRaupR6iGr/9YWqd6dyTc6j2UJosCJECy/gjriMDUPsuN5g7Tr7O6rl+zpk4taIXJWRPJpUEVacNAxh9nAyJyyTmuhJJRH2QcYuv8Wsq4JA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CO6PR12MB5459.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(39860400002)(376002)(396003)(366004)(136003)(346002)(451199024)(1800799009)(186009)(86362001)(110136005)(5660300002)(2616005)(41300700001)(316002)(2906002)(66946007)(66556008)(54906003)(66476007)(31686004)(8936002)(4326008)(8676002)(478600001)(31696002)(6666004)(6486002)(53546011)(33964004)(38100700002)(6506007)(6512007)(26005)(36756003)(83380400001)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Q2xSYjJSOWtSZlFBaDFqdXc0cGx6bllkRFVCb2JuallSTDUwWTRpQUk2anF3?= =?utf-8?B?c2pYT3krVkl6UkpCS1NUaGZRODR2S1lVeWpxUUlvOStZT3drcHo3MENoVUF5?= =?utf-8?B?UHJER3VRMUxaK1FPd3lmcXUvRjF6U2ZMNGNXSndwWHBzRDBjYktVVjBmMmxa?= =?utf-8?B?L1VFY2RGNE44U0o3RnMvZkl1WnVxWGc0STZwbElhZWlObzdyYXEzZVBrSEhx?= =?utf-8?B?cHFDMVMyOVdLUGczMGxTbWNDelBYaG0wWkdFd3FGZkpvS3JlWkdQVFJwdmtC?= =?utf-8?B?clRXYVhzVE5CNEtRQXFNVm9MMEdkV3lvK1JtTEU1N0ZNN1BTcW1OUzhwVGo2?= =?utf-8?B?bzkxZ0ErNkJKSTlCVUFnTEQ4b1pPVkx4K0pxcG9TY0hmRU9ZL3VEYlA3bzg3?= =?utf-8?B?QnlpWDFYdWZVQzJQZE9MT1Z1QWJ4bktYMEx3Sm5Ed1RHTGxnK1Q2WVVvVDAx?= =?utf-8?B?N0ZvQnZhK2xhRWJLeENEeXZJVEZUd1dpL0JocFlubkY0djdqMEtRdE91TnRx?= =?utf-8?B?OVgwbzFhVU9rK0I5aGh3Z3pyRWlLMkRLTisxb1RzY2Qzd2w4eFdjWEJiMzg4?= =?utf-8?B?Zm52RlU3MVVrcnAxL0VkcFNQdUNLRWJOZmsyNDc4SHQ2SksxZFJMb0x6aDF5?= =?utf-8?B?YlpEWldrek9QWExQc3g2WEFLWDFPVnBhdkFVRGI4aWg0NitLSFN0YU8yRXE2?= =?utf-8?B?Vm5lck1UUnVLdDJDTlprdERVdU04L0k1V1FZOC8zdWxwaldFY1pLem04RGVQ?= =?utf-8?B?V28xRFBrR2xLeFpJY09id1dJclNGdkVGQXQwcGdZYzVZbzR3ZTNJNTNrTi9Q?= =?utf-8?B?VFFlOERCUzYxWUhKclpSd1VuYWVFVVhuT0F6enZtZFdtcGZ2bkxVbitWMnJG?= =?utf-8?B?ZTB5SEwxd0d5ZWtCTktvL21OZmM5TWNyRXZ3NnkxVUl4WU1MMUsrd3lScXV5?= =?utf-8?B?Q1BoWThSWkxCdlI5SkZzdVV4SXBuRC84MkFPeURIdHV6bWQxSzdtYU93aDVu?= =?utf-8?B?cEtWeEJCRUN1SkdTL3czdXRwK2t0SFlEM21BSDlRNXJBSnhrOG5qRmVySGdn?= =?utf-8?B?eGV3dGpQcUZ3TU9keWppVFA2QVpkT1Y5aXNlYlRNbmx5bVFDVkdnNzFOK0lF?= =?utf-8?B?UFhObm54V0JzZ2U4cGdsdGZSWUFYY2xUQWdwMjFkUkhxZnpDRHRLTzV0dFlu?= =?utf-8?B?bCtHME1jQW5FanhZUE0yY3ZmS3pVcXpFSno3aEowdjNhNXNRcnVPMjBybVcz?= =?utf-8?B?NEJrZ2pwbXh6UjA2OTlNRUVBMTFrZy9zZjUvT3hmTTA0Y3JxU0tYS0tiazZE?= =?utf-8?B?SHpxKzVsSEsyUmhYQTNrcWJVVzBGWThmdkF0RGgvSXh6MDdzWFBxalhTVmwz?= =?utf-8?B?bWMwQlJyK2FqL3NiWS85SlF0QkxzK05jZ0V2WnZzWkwrbmQ2UHUxWHVWYk9Z?= =?utf-8?B?Smw3bGdseG5EOW5iV0hhTkFEcE1mcmZnTDlFVzNIaXhkazdZYkljRmowblRL?= =?utf-8?B?dnh0dWc1VFZacVB2UVY3a2ZlWU5hS0toN0FmcW1pNnVxS1Fna29wQS9sYW5X?= =?utf-8?B?ZXlkV0xRM3JCanpDUlFFYm9kZkRuK28rdEdjZDNKZ3UyTk5Za29vZDRqMnlJ?= =?utf-8?B?SDdyTDc5WWRGeTZiOVdPRkpDQkl3Zi9zR1lIaVFFSDNMWEJ1V2tmR2RTQm9a?= =?utf-8?B?Z3dkSm9GOStXblNjaktqNXVjNjRyZVQzWXJDUDZ2ZEV0Z1ZqWnJzREsyakdB?= =?utf-8?B?R1VnYVlGdUQyaEJQWEQzYklRSU9KakdoU3lMYVlralVqaXArNTJpYTYyYXhk?= =?utf-8?B?OFVZRytheTFQUkUvRXYwdUNNQTVYL1pjYzRKOEFCYjZGeC9XRkI4L3hQUG1l?= =?utf-8?B?NHJEcEw0TU5PZWJsYWQrRHkvRURpREh3YU9mdHhDWjJpVUZqMmJMUGpMNVZ0?= =?utf-8?B?NXMySTNoRDdEdDN0SEFnaHNnTnBJRndlVGZYeHFJR2JySVhTMmJqZ3lsNk5T?= =?utf-8?B?bTdWTWdSdWFjK0F5SG1lREJYSll1KzFVK3B0ekRwRFIxa0lCRGRWNEI3YVc1?= =?utf-8?B?NlZkcVkzbXdEWEExYUtQK3FwYzJBVkNqVDZDNWxsaVN0bUwwY3BlUVRpMlZh?= =?utf-8?Q?pYOpN7iukhl5yeMmxT4f7Q+W4?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: ab2b2ca9-d298-4dc7-0ace-08dba11e8a1e X-MS-Exchange-CrossTenant-AuthSource: CO6PR12MB5459.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Aug 2023 01:41:14.3809 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vJ/lL1d2OHBZhw3ftk9Zn6iDPwZlRpG9D+LHmlgewiUCkxdqEMHJq1ggBtzZcyEAG5Mn8OQrFSM1rd0gIwsobg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW6PR12MB8833 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --------------lk0v5RNy0z2i1XDg6pA9OYA8 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2023/8/19 19:57, Konstantin Ananyev wrote: > 18/08/2023 10:38, Jack Min пишет: >> On 2023/8/18 17:05, Konstantin Ananyev wrote: >>>> On 2023/8/17 22:06, Stephen Hemminger wrote: >>>>> On Thu, 17 Aug 2023 05:06:20 +0000 >>>>> Honnappa Nagarahalli wrote: >>>>> >>>>>> Hi Matan, Viacheslav, >>>>>>     Tyler pointed out that the function >>>>>> __mlx5_hws_cnt_pool_enqueue_revert is accessing the ring private >>>>>> structure members >>>> (prod.head and prod.tail) directly. Even though ' struct rte_ring' >>>> is a public structure (mainly because the library provides inline >>>> functions), the structure members are considered private to the >>>> ring library. So, this needs to be corrected. >>>>>> It looks like the function __mlx5_hws_cnt_pool_enqueue_revert is >>>>>> trying to revert things that were enqueued. It is not clear to >>>> me why this functionality is required. Can you provide the use case >>>> for this? We can discuss possible solutions. >>>>> How can reverting be thread safe? Consumer could have already >>>>> looked at them? >>>> Hey, >>>> >>>> In our case, this ring is SC/SP, only accessed by one thread >>>> (enqueue/dequeue/revert). >>>> >>>> The scenario we have "revert" is: >>>> >>>>    We use ring to manager our HW objects (counter in this case) and >>>> for >>>> each core (thread) has "cache" (a SC/SP ring) for sake of performance. >>>> >>>> 1. Get objects from "cache" firstly, if cache is empty, we fetch a >>>> bulk >>>> of free objects from global ring into cache. >>>> >>>> 2. Put (free) objects also into "cache" firstly, if cache is full, we >>>> flush a bulk of objects into global ring in order to make some >>>> rooms in >>>> cache. >>>> >>>> However, this HW object cannot be immediately reused after free. It >>>> needs time to be reset and then can be used again. >>>> >>>> So when we flush cache, we want to keep the first enqueued objects >>>> still >>>> stay there because they have more chance already be reset than the >>>> latest enqueued objects. >>>> >>>> Only flush recently enqueued objects back into global ring, act as >>>> "LIFO" behavior. >>>> >>>> This is why we require "revert" enqueued objects. >>>> >>> Wouldn't then simple stack fit you better? >>> Something like lib/stack/rte_stack_std.h, but even without spinlock >>> around? >> >> No, stack is always a "LIFO" struct, right? > > Yep. > >> >> Here first we need this cache works as "FIFO" in most cases (get/put) >> because the first enqueued objects have more chance that are already >> reset so can reuse them. >> >> We only require "LIFO" behavior when "flush" cache in order to make >> some room, so next free will be quick because it happens in our local >> cache, needn't access global ring. >> >> In short, we require a struct supports "FIFO" and "LIFO". > > Ok, thanks fro explanation. > So you need a ring, but with an ability to revert prod N elements > back, right? Yes, exactly! --------------lk0v5RNy0z2i1XDg6pA9OYA8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 2023/8/19 19:57, Konstantin Ananyev wrote:
18/08/2023 10:38, Jack Min пишет:
On 2023/8/18 17:05, Konstantin Ananyev wrote:
On 2023/8/17 22:06, Stephen Hemminger wrote:
On Thu, 17 Aug 2023 05:06:20 +0000
Honnappa Nagarahalli<Honnappa.Nagarahalli@arm.com>  wrote:

Hi Matan, Viacheslav,
    Tyler pointed out that the function __mlx5_hws_cnt_pool_enqueue_revert is accessing the ring private structure members
(prod.head and prod.tail) directly. Even though ' struct rte_ring' is a public structure (mainly because the library provides inline
functions), the structure members are considered private to the ring library. So, this needs to be corrected.
It looks like the function __mlx5_hws_cnt_pool_enqueue_revert is trying to revert things that were enqueued. It is not clear to
me why this functionality is required. Can you provide the use case for this? We can discuss possible solutions.
How can reverting be thread safe? Consumer could have already looked at them?
Hey,

In our case, this ring is SC/SP, only accessed by one thread
(enqueue/dequeue/revert).

The scenario we have "revert" is:

   We use ring to manager our HW objects (counter in this case) and for
each core (thread) has "cache" (a SC/SP ring) for sake of performance.

1. Get objects from "cache" firstly, if cache is empty, we fetch a bulk
of free objects from global ring into cache.

2. Put (free) objects also into "cache" firstly, if cache is full, we
flush a bulk of objects into global ring in order to make some rooms in
cache.

However, this HW object cannot be immediately reused after free. It
needs time to be reset and then can be used again.

So when we flush cache, we want to keep the first enqueued objects still
stay there because they have more chance already be reset than the
latest enqueued objects.

Only flush recently enqueued objects back into global ring, act as
"LIFO" behavior.

This is why we require "revert" enqueued objects.

Wouldn't then simple stack fit you better?
Something like lib/stack/rte_stack_std.h, but even without spinlock around?

No, stack is always a "LIFO" struct, right?

Yep.


Here first we need this cache works as "FIFO" in most cases (get/put) because the first enqueued objects have more chance that are already reset so can reuse them.

We only require "LIFO" behavior when "flush" cache in order to make some room, so next free will be quick because it happens in our local cache, needn't access global ring.

In short, we require a struct supports "FIFO" and "LIFO".

Ok, thanks fro explanation.
So you need a ring, but with an ability to revert prod N elements back, right?

Yes, exactly!
--------------lk0v5RNy0z2i1XDg6pA9OYA8--