From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <Honnappa.Nagarahalli@arm.com>
Received: from EUR03-AM5-obe.outbound.protection.outlook.com
 (mail-eopbgr30079.outbound.protection.outlook.com [40.107.3.79])
 by dpdk.org (Postfix) with ESMTP id C3539288C
 for <dev@dpdk.org>; Fri, 18 Jan 2019 07:48:48 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; 
 s=selector1-arm-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=SnITcdbVnMBstLh7xg70pSe8i/MLGYmB8Y8Bl+7lhL4=;
 b=poILzuPECVzfszSgIyLE0FMiKrzOHv6pTBMBspiRtdCGFoeR7ZbmnK9716b7Hf59DZ9LfcuRTzsDOhiiFZesQgOJxAbJ9tY057rVNNV27tSh49hUSXUQbj3VGT/JrMsNXnzCEoYniWZdkvRYmFTyN6y67KhH3xA1Ab9UHoTBQaI=
Received: from AM6PR08MB3672.eurprd08.prod.outlook.com (20.177.115.76) by
 AM6PR08MB4487.eurprd08.prod.outlook.com (20.179.18.20) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.1516.13; Fri, 18 Jan 2019 06:48:47 +0000
Received: from AM6PR08MB3672.eurprd08.prod.outlook.com
 ([fe80::25ec:2db7:d268:2b7b]) by AM6PR08MB3672.eurprd08.prod.outlook.com
 ([fe80::25ec:2db7:d268:2b7b%2]) with mapi id 15.20.1537.018; Fri, 18 Jan 2019
 06:48:47 +0000
From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
To: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>, "dev@dpdk.org"
 <dev@dpdk.org>, "stephen@networkplumber.org" <stephen@networkplumber.org>,
 "paulmck@linux.ibm.com" <paulmck@linux.ibm.com>
CC: "Gavin Hu (Arm Technology China)" <Gavin.Hu@arm.com>, Dharmik Thakkar
 <Dharmik.Thakkar@arm.com>, nd <nd@arm.com>, nd <nd@arm.com>
Thread-Topic: [RFC v2 1/2] rcu: add RCU library supporting QSBR mechanism
Thread-Index: AQHUmZwkA3lN9JDuXEiclvOGJBGOtKWwUixggACXpdCAAUkpAIACgqOw
Date: Fri, 18 Jan 2019 06:48:46 +0000
Message-ID: <AM6PR08MB367272B4EEA528BD914B1648989C0@AM6PR08MB3672.eurprd08.prod.outlook.com>
References: <20181122033055.3431-1-honnappa.nagarahalli@arm.com>
 <20181222021420.5114-1-honnappa.nagarahalli@arm.com>
 <20181222021420.5114-2-honnappa.nagarahalli@arm.com>
 <2601191342CEEE43887BDE71AB977258010D904212@irsmsx105.ger.corp.intel.com>
 <AM6PR08MB36728CA160ADA84FC3D901E998810@AM6PR08MB3672.eurprd08.prod.outlook.com>
 <2601191342CEEE43887BDE71AB977258010D904AC7@irsmsx105.ger.corp.intel.com>
In-Reply-To: <2601191342CEEE43887BDE71AB977258010D904AC7@irsmsx105.ger.corp.intel.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
authentication-results: spf=none (sender IP is )
 smtp.mailfrom=Honnappa.Nagarahalli@arm.com; 
x-originating-ip: [217.140.103.75]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; AM6PR08MB4487;
 6:wh/dhCZNYfYDigdYGbrNTgbZftjirzeDp12gkoX/1TgMWA0R+8Snit7H4CixDkJ9P9U8n4Obhqn3u8bnwjFa6b4dT3igHRvrSLx3c9Y5Cjm/NaYowvC6Uihstk9qltf/8CixxIcSNwsPRUTRlloXyO+QF1QsQ+xJXa+VniqGwchXIHUkyzR1tEXL7sHS74P8rCvoPor7Tlqb6zu5fal+OXHwFLjMwX7NzeEgTYd+Rd0Mw3PShxxa6nXvaRuLOaSivKTKA8k+40BZmC2+OxMU/UpG5GoM2WwiUI4IFJ+JZErQS+VnnzYVRKvCRrzpryiBBGPm33dhdYE9+c+LABMWJyPWQWOxhQDULRhRBqZ85VHPR/Z98gYcin0K3zogUE2uEP7nuwsBAGyQz7xQZxC8/w1FIdvtUVyZeSR4AQdcypuZfWy9HAqGPMtcGXQhu4E3z3q4EW4JmR4m4jIY4jzcHg==;
 5:QzlyhdSM4+15ECJCm8FhlCGi/gsNwGVo09oEdJc7chYkJN0xbV6JziwtKt51/VIDioZ0jPve4P/FQ+Ej3C+W+V0noeDyelSBNqlmRPEp/dfHTxQpWt3eaPtJMDjnrh1JHLHPhv8PogNGdU68WAy2WdHkq06oCm5y2OlXXGANstFNBp+ZSzzQtWKpYtSBlqf5fV3rV40JHO3eUlLot+OCyA==;
 7:an23oPDl22D6vTkIVut+2w3z8lHDvC3SElqBDax83S7tYEq7nIu2Dv8C5edoMagD99zLJRNLwtm62WTouVxFjH9T/yeK5cn6HvFNgar5bVfv2NwgpRxFlfroQ6zV8e4SDGI2EkRG6nBPf8VIcwKozQ==
x-ms-exchange-antispam-srfa-diagnostics: SOS;SOR;
x-ms-office365-filtering-correlation-id: 4fc53a29-4186-47c1-d658-08d67d10fefc
x-ms-office365-filtering-ht: Tenant
x-microsoft-antispam: BCL:0; PCL:0;
 RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600109)(711020)(4618075)(2017052603328)(7153060)(7193020);
 SRVR:AM6PR08MB4487; 
x-ms-traffictypediagnostic: AM6PR08MB4487:
nodisclaimer: True
x-microsoft-antispam-prvs: <AM6PR08MB4487D28FD759D3B39D7A5D9B989C0@AM6PR08MB4487.eurprd08.prod.outlook.com>
x-forefront-prvs: 0921D55E4F
x-forefront-antispam-report: SFV:NSPM;
 SFS:(10009020)(346002)(39860400002)(396003)(376002)(366004)(136003)(199004)(189003)(76176011)(5660300001)(54906003)(8676002)(71190400001)(81166006)(6116002)(3846002)(316002)(345774005)(110136005)(2201001)(305945005)(478600001)(86362001)(71200400001)(106356001)(105586002)(8936002)(7736002)(81156014)(74316002)(68736007)(256004)(6246003)(55016002)(229853002)(97736004)(186003)(6506007)(93886005)(4326008)(53936002)(33656002)(25786009)(102836004)(446003)(11346002)(486006)(2501003)(26005)(7696005)(72206003)(66066001)(2906002)(99286004)(6436002)(9686003)(476003)(14454004);
 DIR:OUT; SFP:1101; SCL:1; SRVR:AM6PR08MB4487;
 H:AM6PR08MB3672.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en;
 PTR:InfoNoRecords; A:1; MX:1; 
received-spf: None (protection.outlook.com: arm.com does not designate
 permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: OonDE+PV8KRHYWFURsDG+on+CGgrOsC8VZzQgA8L7NotVTRPJG9NrSbAQhpV2YYg1gU8naDezy6Rs7qtA1vovu5UmCoWKgZBOz1Z3mR7fRAa/0FI4VXRPpGj9yH4QQ7WhkF12Z/xVJOXVVot4S7cyl9qZXQzZcsvFWKuHnXffk0WUT2fKLC8K6tYvOBvisdhEmIHer+mLRYRjmEMsQ+jL/PyfIeHawaOnaaOIVEgRIxIF0GASPYXapvrNXhCmyBrP/xQjp6Qx6qlber+dW+qTy1qfj81UI5A0vXryVPT0IdxEljlMcFdC6xNs27TphMSnoCnGpA41OAGMzOtYb+PpAJL7Yik8//ph2C0QYD3+wlth7yqd1qXyAPdI+vxNT0YdJ3QjGM2JD5BNW+wzU6p8k8tBesUt2b99i9SnTrj5EI=
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: arm.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4fc53a29-4186-47c1-d658-08d67d10fefc
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Jan 2019 06:48:47.0035 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: f34e5979-57d9-4aaa-ad4d-b122a662184d
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB4487
Subject: Re: [dpdk-dev] [RFC v2 1/2] rcu: add RCU library supporting QSBR
	mechanism
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 18 Jan 2019 06:48:48 -0000

>=20
> > > ...
> > >
> > > > diff --git a/lib/librte_rcu/rte_rcu_qsbr.h
> > > > b/lib/librte_rcu/rte_rcu_qsbr.h new file mode 100644 index
> > > > 000000000..c818e77fd
> > > > --- /dev/null
> > > > +++ b/lib/librte_rcu/rte_rcu_qsbr.h
> > > > @@ -0,0 +1,321 @@
> > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > + * Copyright (c) 2018 Arm Limited  */
> > > > +
> > > > +#ifndef _RTE_RCU_QSBR_H_
> > > > +#define _RTE_RCU_QSBR_H_
> > > > +
> > > > +/**
> > > > + * @file
> > > > + * RTE Quiescent State Based Reclamation (QSBR)
> > > > + *
> > > > + * Quiescent State (QS) is any point in the thread execution
> > > > + * where the thread does not hold a reference to shared memory.
> > > > + * A critical section for a data structure can be a quiescent
> > > > +state for
> > > > + * another data structure.
> > > > + *
> > > > + * This library provides the ability to identify quiescent state.
> > > > + */
> > > > +
> > > > +#ifdef __cplusplus
> > > > +extern "C" {
> > > > +#endif
> > > > +
> > > > +#include <stdio.h>
> > > > +#include <stdint.h>
> > > > +#include <errno.h>
> > > > +#include <rte_common.h>
> > > > +#include <rte_memory.h>
> > > > +#include <rte_lcore.h>
> > > > +#include <rte_debug.h>
> > > > +
> > > > +/**< Maximum number of reader threads supported. */ #define
> > > > +RTE_RCU_MAX_THREADS 128
> > > > +
> > > > +#if !RTE_IS_POWER_OF_2(RTE_RCU_MAX_THREADS)
> > > > +#error RTE_RCU_MAX_THREADS must be a power of 2 #endif
> > > > +
> > > > +/**< Number of array elements required for the bit-map */ #define
> > > > +RTE_QSBR_BIT_MAP_ELEMS
> (RTE_RCU_MAX_THREADS/(sizeof(uint64_t)
> > > * 8))
> > > > +
> > > > +/* Thread IDs are stored as a bitmap of 64b element array. Given
> > > > +thread id
> > > > + * needs to be converted to index into the array and the id
> > > > +within
> > > > + * the array element.
> > > > + */
> > > > +#define RTE_QSBR_THR_INDEX_SHIFT 6 #define
> RTE_QSBR_THR_ID_MASK
> > > > +0x3f
> > > > +
> > > > +/* Worker thread counter */
> > > > +struct rte_rcu_qsbr_cnt {
> > > > +	uint64_t cnt; /**< Quiescent state counter. */ }
> > > > +__rte_cache_aligned;
> > > > +
> > > > +/**
> > > > + * RTE thread Quiescent State structure.
> > > > + */
> > > > +struct rte_rcu_qsbr {
> > > > +	uint64_t reg_thread_id[RTE_QSBR_BIT_MAP_ELEMS]
> > > __rte_cache_aligned;
> > > > +	/**< Registered reader thread IDs - reader threads reporting
> > > > +	 * on this QS variable represented in a bit map.
> > > > +	 */
> > > > +
> > > > +	uint64_t token __rte_cache_aligned;
> > > > +	/**< Counter to allow for multiple simultaneous QS queries */
> > > > +
> > > > +	struct rte_rcu_qsbr_cnt w[RTE_RCU_MAX_THREADS]
> > > __rte_cache_aligned;
> > > > +	/**< QS counter for each reader thread, counts upto
> > > > +	 * current value of token.
> > >
> > > As I understand you decided to stick with neutral thread_id and let
> > > user define what exactly thread_id is (lcore, syste, thread id, somet=
hing
> else)?
> > Yes, that is correct. I will reply to the other thread to continue the =
discussion.
> >
> > > If so, can you probably get rid of RTE_RCU_MAX_THREADS limitation?
> > I am not seeing this as a limitation. The user can change this if requi=
red. May
> be I should change it as follows:
> > #ifndef RTE_RCU_MAX_THREADS
> > #define RTE_RCU_MAX_THREADS 128
> > #endif
>=20
> Yep, that's better, though it would still require user to rebuild the cod=
e if he
> would like to increase total number of threads supported.
Agree

> Though it seems relatively simply to extend current code to support dynam=
ic
> max thread num here (2 variable arrays plus shift value plus mask).
Agree, supporting dynamic 'max thread num' is simple. But this means memory=
 needs to be allocated to the arrays. The API 'rte_rcu_qsbr_init' has to ta=
ke max thread num as the parameter. We also have to introduce another API t=
o free this memory. This will become very similar to alloc/free APIs I had =
in the v1.
I hope I am following you well, please correct me if not.

>=20
> >
> > > I.E. struct rte_rcu_qsbr_cnt w[] and allow user at init time to
> > > define max number of threads allowed.
> > > Or something like:
> > > #define RTE_RCU_QSBR_DEF(name, max_thread) struct name { \
> > >   uint64_t reg_thread_id[ALIGN_CEIL(max_thread, 64)  >> 6]; \
> > >    ...
> > >    struct rte_rcu_qsbr_cnt w[max_thread]; \ }
> > I am trying to understand this. I am not following why 'name' is
> > required? Would the user call 'RTE_RCU_QSBR_DEF' in the application
> header file?
>=20
> My thought here was to allow user to define his own structures, depending=
 on
> the number of max threads he needs/wants:
> RTE_RCU_QSBR_DEF(rte_rcu_qsbr_128, 128);
> RTE_RCU_QSBR_DEF(rte_rcu_qsbr_64, 64); ...
Thank you for the clarification, I follow you now. However, it will not sol=
ve the problem of dynamic max thread num. Changes to the max number of thre=
ads will require recompilation.

> Konstantin