From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9328943095; Fri, 18 Aug 2023 11:38:27 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7E4EF40ED9; Fri, 18 Aug 2023 11:38:27 +0200 (CEST) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2061.outbound.protection.outlook.com [40.107.94.61]) by mails.dpdk.org (Postfix) with ESMTP id 8E8AC40395 for ; Fri, 18 Aug 2023 11:38:25 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eVgWR6oExbz+2W+0ksi/KMqPtEvnOHdZ8hEkCLB/4cJHZ1reGxQvi+qfdAs1QAwxNnRd/s5WbBkgYmaXtirt4V4tUfxlOy/D57vbXiu/3b2cJuyAEznMYRNPl8OkfsdngmzplOGjYkv4G/syec3DTqKbNV8wQn8H+UH1WOHVtdy1WWEmqJbZcPsb2siZVySWC9xMTbGrh68gBa3WUAkPlP3/UKWiIFJ0JFoqXWXk2SwFOah1aEx0eoJxQPLlmyMQDmRqJx6D0p9b01tKzVbt7h1e4O3NCCSErgVDdBpQlH3j8l9YPRJdUnVpCq+eV7iMnkkdPS6ilnN3c+rn3IgDiw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=F7PhJcfSbrVawxI0XUPTmT3GD56IXM56Rz7hRe6Sx44=; b=G0OtLOtgVlpT+dFdCtwxyA5+g2LwzRgxbD+orIcJlKEBZsOAcqgL1uYPHg1bOpidsC+YCZogAIM6skFPtgOo0jQl+7u6codAbhUd6D39YonQomf0v378weIpyjFb2+8lw9emNCvZg6slbR/whqyf70LIrvBhfG3zdCEHggFM2Ep0w3alfWzOdlvvfGqyM7FcDxyfYUJLNstM2rx/gtcK1Nlb65VOsoCUfSwMVFz0ojOx7+HAYrO7QLkeNCPh4zqwioQ3QvEpmsxoMe3Gs7R6WVJTrZDS0euZ0g8+ppOPGiTeLvwQRYWmS+66HoxBIjZTCUYiVe1HFCIyt/lek69sqQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=F7PhJcfSbrVawxI0XUPTmT3GD56IXM56Rz7hRe6Sx44=; b=bTg4V5qQZ25+byFAGrNQJYtEgDFzIK4UALpTh9EYOHwnT4YHe+KHYvVwkEBCOfgUKK6Tw+Q6w3ZUJ94p8Cmw+hqajlVboJm4lMRglhOnfhAd4zWlnLwriTCMrQd7x/h02D8bbKkHKE8A4SSC3MbxRhf11VnZsj/kTzobNJJQMyvZF7Pwel21kHcoevk19F6LIGPpaeKJNjXPvfUZ0AA66aD/28RP4DLrgoPIYQjnQPlMGBjiNTJM1Yb7wDyfXmmaLg76blq8DgsgwSJGKaJzBfSokvo4epqOTPmhdc6pvk+6FulCcb4954F3UlR7TsYY4XG1MdYi9/knx8NtEJ2eeQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CO6PR12MB5459.namprd12.prod.outlook.com (2603:10b6:303:13b::16) by SN7PR12MB7252.namprd12.prod.outlook.com (2603:10b6:806:2ac::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6678.31; Fri, 18 Aug 2023 09:38:23 +0000 Received: from CO6PR12MB5459.namprd12.prod.outlook.com ([fe80::46b7:f479:cba8:9adf]) by CO6PR12MB5459.namprd12.prod.outlook.com ([fe80::46b7:f479:cba8:9adf%7]) with mapi id 15.20.6699.020; Fri, 18 Aug 2023 09:38:23 +0000 Content-Type: multipart/alternative; boundary="------------FZCLfCfKuwX2Bned480SJRPv" Message-ID: <63206979-911f-439b-816d-ee5c1c67f195@nvidia.com> Date: Fri, 18 Aug 2023 17:38:14 +0800 User-Agent: Mozilla Thunderbird Subject: Re: MLX5 PMD access ring library private data Content-Language: en-US To: Konstantin Ananyev , Stephen Hemminger , Honnappa Nagarahalli Cc: "dev@dpdk.org" , Matan Azrad , "viacheslavo@nvidia.com" , Tyler Retzlaff , Wathsala Wathawana Vithanage , nd References: <20230817070658.45576e6d@hermes.local> From: Jack Min In-Reply-To: X-ClientProxiedBy: TY2PR02CA0050.apcprd02.prod.outlook.com (2603:1096:404:e2::14) To CO6PR12MB5459.namprd12.prod.outlook.com (2603:10b6:303:13b::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR12MB5459:EE_|SN7PR12MB7252:EE_ X-MS-Office365-Filtering-Correlation-Id: 58ad2355-841e-4e5a-8cce-08db9fcedd40 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qvIZXLg30MH7T/tZdSiVU02QCm2l9rcEPAooQGXtsPUufWjFtEHvkqxHui/hWJWXmvO7inooyPea5UdcC+J84OpcwyTJO8RQ6vXpqyoP7ZHQRm9h1Xv145bhXaKKifu+tsIOi6czWbVhl+7UifT2GU6vvvBTS5f+sNF/4gyiPbSuR/6OGBtYxAwDWT/N1mqkXHAMhuatxPmjUtrrEFzBtcAxy5FpE/HW5MT6VIpW5sFIudVhiQho7Op1IwLIoG60li13i/ckesJUeNtwAnlhKXykaqTioREmdN3EO7ynvyYpPeomE63sRfZxwMgOzd1XrNBFpVVo8nZhgqqocEB1zzOatDuvLlbtGpzDwBxG8MSp/GzzGpgtmgWjRQ6G5ipZxZ++j44Amc95MNRJVpgK3/lPOe3fjzcpDKfJWG+WJ690z1kU+FqFm796mQOEOGcd1BQ772EmKiLSjgpK5OQdy02o6tyP2TnSOnbyGpk1Qc6DWHiYIgFs8rhmsWaDGvgcbPZUbG2G8a9SQeNjPZWFHOfRkHgq7k3Ps2WuBeGrrc5YYXBeeVtqXjcvy9khyVPkceBxcxJUaWUmIWUEGL/mtFjAugFnwnp2X5dMliPfTJN8Q0Vd7J0YKXqct/pr1AWLbCL1GJuN2LISBMp3flV8MA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CO6PR12MB5459.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(376002)(346002)(366004)(136003)(396003)(39860400002)(451199024)(1800799009)(186009)(31686004)(36756003)(86362001)(31696002)(38100700002)(5660300002)(2616005)(66556008)(66476007)(33964004)(478600001)(6506007)(6666004)(110136005)(6486002)(54906003)(316002)(66946007)(53546011)(41300700001)(4326008)(8936002)(26005)(6512007)(8676002)(2906002)(83380400001)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?VUdieG13VkhLRStzRGdDdVpCemhLUWJCd0lFeUIySHE5ZHhEeU9IVjZNQzlL?= =?utf-8?B?bXhxZnlld0ZrTkhDRXBpR2RReFQ3WVRBTDRERGYzdklleGRoaUdrd0xSczk4?= =?utf-8?B?aEVOcHlaQTBBRlQzUEVIcEVxRWNMbE9wcWFnUEFIdkQ1akhXTmlXc1FzZ1hU?= =?utf-8?B?OThGSVNEd0twZlU1cm9oOU02NmsxL1llVXB3RWdTWEJjWU1yZHFoR2VhcHRm?= =?utf-8?B?SWVmOFAvWTNRT3FmWjZlMlFNTGhEU1RqUXUzZ21GM3dqKzEvbDEyaW5LQS85?= =?utf-8?B?MHpNR1hRbnNnUURzdllWNU92aGRXMU15MVhIc2V1YTVnZ0RrUXZTY0hMa3NW?= =?utf-8?B?MkhEemJScnVTTnJ2QmlOMDA5eWNvMDFXajcyd2E4U2txbmp0S0c3YTNib1Bs?= =?utf-8?B?RGNGWDNsVXVEVUw5SThCNXkyVnpLOFBMOTYyRFkrZkptOVowcUlqampFVC84?= =?utf-8?B?bXFQQmw2aE9oWklMWDdxMUU3Z1I1eHZYN0xLMHJRdHFmUnlUVGFMVWRjUFQ1?= =?utf-8?B?TlNGSTgrSmZhM0FaektkVGcxWWhuWW5MOTZWUkVQT1NYbUFxM0Z1aGxmRC9Q?= =?utf-8?B?UDBYL3J3b2VKUDRIMTBUR1U3Y1FOWUd0cVYyekM3NGE5djluTWhxV25pMGhW?= =?utf-8?B?NzR5Y3NHS2RjN2NpK3BneVk1WjBsaTF6cXhEUGNDWGJFK0lmaVZxaE91TDZ2?= =?utf-8?B?MmhaWDEzQnFCNDlPa0FBR0dZTEphK2hOVExZMEt5d3NkSkU5WkFqanZKQU96?= =?utf-8?B?dnJRRmZvQTNFUExzMEhBZkZNZzI0aGZnN09hNzE2RGNBYnNvNzZwaHFNQi9n?= =?utf-8?B?a1hEdW1UYnorUS9jL0tHS1I3ZVd4dXNCMFM0aDg5MDl0SVFpYTBrMjhGTHpZ?= =?utf-8?B?NDlYckFZWmlDdG00U1hpdnIvMytxbkJRc0cwMDlnRVhyQlhHTnFpWVp1Rmw0?= =?utf-8?B?NHE3aWhrWmtPKzVmeG9HSE0yUDl1R0lnd3JNajJST2tKWFVQRUd3bi9tbml3?= =?utf-8?B?c3pYZWRlM2F3QkxxNjFRMDZxeDlGdDVrQUFpeDRFSkdwK21vSWpZZ1JjcGhm?= =?utf-8?B?VlowK2tNandlTmZWalI3Qkdvb05TVjR3b2J0WkJHdjJsNzFBYnNrS0twMExW?= =?utf-8?B?QjV4WU5pWTM5RFI5VVBoOUw4ekJaQjZnZE1MSmVFZitzQVdkUnV3bE04bmN4?= =?utf-8?B?ZU54RTNScTJXNXRqU0Z6NzcxaUxka1BXckppUmd6TnY3T3YzTWt5b1hpdGJE?= =?utf-8?B?cWJtaDV0WjFibC8raXhhak15VWZRSVYrOTBXNUV1NURYNGFBUnQrME0xUHVN?= =?utf-8?B?bC9zNDBjSkdGUWEwMVNJUjR3MnZMaHVCcHNuR2UzWjh2L1JJTjV0U2FLTWpE?= =?utf-8?B?VnRpc21IVG43Nkwzb243Yml0TTdOUUJocHpkQmE0d0FQZEQwSHdiRUZ1L2pw?= =?utf-8?B?Vi96VlU0clNQZjhzaTg3Sy9BQnE1dFBjZlpQQ0JXbU40bGNyd1ovYm9TcGpO?= =?utf-8?B?TjRCb2N3c3k5OFp2UFlEQWhkaCtuamtEdWRuUEJxWWxQU0FteVkwMEJ5aGFV?= =?utf-8?B?ejUxQnFaSmtENDZUem5KRSt6U01OM0hZN3c3UW0xYU5xa3VHWlNERmR0cCtt?= =?utf-8?B?MHpSR2NEUWhtUGxQWmNqZnlXQjJkSVdzWjkzYnFNcDB6SlZWcFNwR1pkYmh6?= =?utf-8?B?ZlFzUWZneUdLd1NGZkFhNjN1cmpFQ1NPdzl1Y0hJTGw4NnpGbkdqLzFzQXQx?= =?utf-8?B?QVJrd3J1VVBrbFJPK09pVWNiQmt3SUptWXUydS8rZEhiZ0lrVVk5RWIyOWc5?= =?utf-8?B?S1JDWXM0L1k4QnNmdmlEOXBaRE0yT3IweGFIZGx3RkhDQUpyczFJdmlEVExP?= =?utf-8?B?R0NVZUFncldDam5RWU9HQm16N09zUGJHNlpYK2FDM2FhTnAyVkkzZWxxbXBL?= =?utf-8?B?RkU5VFRHOWE5aGNBQU83TnRXa3pXTWtEU1hpYmZFNnNtcXIyQk5EQWZPRi8r?= =?utf-8?B?MVpPa3d3ZWVXNHVGYlZ4UVU0a3dkRk9CSU1oZ1NCTTI1OGEvTCtXaXh4MUVE?= =?utf-8?B?MW5jem1ZVlg4OExDWThIL0txRWpyZDhsUGxmWmtFLzVIQ2IwUTgxTXNEeUx1?= =?utf-8?Q?yMJxRZstR0obUpwWnuveDqws0?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 58ad2355-841e-4e5a-8cce-08db9fcedd40 X-MS-Exchange-CrossTenant-AuthSource: CO6PR12MB5459.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Aug 2023 09:38:22.9089 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: JSoQZOO0xjc3AnipFisRc713dFkb2qobybK+O+EMTZO60K0VOT/iAh1CbN8zHfngxbcmFbXCBHEAtGECwG1YDw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7252 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --------------FZCLfCfKuwX2Bned480SJRPv Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2023/8/18 17:05, Konstantin Ananyev wrote: > >> On 2023/8/17 22:06, Stephen Hemminger wrote: >>> On Thu, 17 Aug 2023 05:06:20 +0000 >>> Honnappa Nagarahalli wrote: >>> >>>> Hi Matan, Viacheslav, >>>> Tyler pointed out that the function __mlx5_hws_cnt_pool_enqueue_revert is accessing the ring private structure members >> (prod.head and prod.tail) directly. Even though ' struct rte_ring' is a public structure (mainly because the library provides inline >> functions), the structure members are considered private to the ring library. So, this needs to be corrected. >>>> It looks like the function __mlx5_hws_cnt_pool_enqueue_revert is trying to revert things that were enqueued. It is not clear to >> me why this functionality is required. Can you provide the use case for this? We can discuss possible solutions. >>> How can reverting be thread safe? Consumer could have already looked at them? >> Hey, >> >> In our case, this ring is SC/SP, only accessed by one thread >> (enqueue/dequeue/revert). >> >> The scenario we have "revert" is: >> >>  We use ring to manager our HW objects (counter in this case) and for >> each core (thread) has "cache" (a SC/SP ring) for sake of performance. >> >> 1. Get objects from "cache" firstly, if cache is empty, we fetch a bulk >> of free objects from global ring into cache. >> >> 2. Put (free) objects also into "cache" firstly, if cache is full, we >> flush a bulk of objects into global ring in order to make some rooms in >> cache. >> >> However, this HW object cannot be immediately reused after free. It >> needs time to be reset and then can be used again. >> >> So when we flush cache, we want to keep the first enqueued objects still >> stay there because they have more chance already be reset than the >> latest enqueued objects. >> >> Only flush recently enqueued objects back into global ring, act as >> "LIFO" behavior. >> >> This is why we require "revert" enqueued objects. >> > Wouldn't then simple stack fit you better? > Something like lib/stack/rte_stack_std.h, but even without spinlock around? No, stack is always a "LIFO" struct, right? Here first we need this cache works as "FIFO" in most cases (get/put) because the first enqueued objects have more chance that are already reset so can reuse them. We only require "LIFO" behavior when "flush" cache in order to make some room, so next free will be quick because it happens in our local cache, needn't access global ring. In short, we require a struct supports "FIFO" and "LIFO". -Jack --------------FZCLfCfKuwX2Bned480SJRPv Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 2023/8/18 17:05, Konstantin Ananyev wrote:


On 2023/8/17 22:06, Stephen Hemminger wrote:

On Thu, 17 Aug 2023 05:06:20 +0000
Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com> wrote:


Hi Matan, Viacheslav,
	Tyler pointed out that the function __mlx5_hws_cnt_pool_enqueue_revert is accessing the ring private structure members

(prod.head and prod.tail) directly. Even though ' struct rte_ring' is a public structure (mainly because the library provides inline
functions), the structure members are considered private to the ring library. So, this needs to be corrected.

It looks like the function __mlx5_hws_cnt_pool_enqueue_revert is trying to revert things that were enqueued. It is not clear to

me why this functionality is required. Can you provide the use case for this? We can discuss possible solutions.

How can reverting be thread safe? Consumer could have already looked at them?

Hey,

In our case, this ring is SC/SP, only accessed by one thread
(enqueue/dequeue/revert).

The scenario we have "revert" is:

  We use ring to manager our HW objects (counter in this case) and for
each core (thread) has "cache" (a SC/SP ring) for sake of performance.

1. Get objects from "cache" firstly, if cache is empty, we fetch a bulk
of free objects from global ring into cache.

2. Put (free) objects also into "cache" firstly, if cache is full, we
flush a bulk of objects into global ring in order to make some rooms in
cache.

However, this HW object cannot be immediately reused after free. It
needs time to be reset and then can be used again.

So when we flush cache, we want to keep the first enqueued objects still
stay there because they have more chance already be reset than the
latest enqueued objects.

Only flush recently enqueued objects back into global ring, act as
"LIFO" behavior.

This is why we require "revert" enqueued objects.


Wouldn't then simple stack fit you better?
Something like lib/stack/rte_stack_std.h, but even without spinlock around?

No, stack is always a "LIFO" struct, right? 

Here first we need this cache works as "FIFO" in most cases (get/put) because the first enqueued objects have more chance that are already reset so can reuse them.

We only require "LIFO" behavior when "flush" cache in order to make some room, so next free will be quick because it happens in our local cache, needn't access global ring.

In short, we require a struct supports "FIFO" and "LIFO".

-Jack


    

  
  


--------------FZCLfCfKuwX2Bned480SJRPv--