From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 802C0A2EFC for ; Sun, 13 Oct 2019 06:36:53 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 4754C1D16B; Sun, 13 Oct 2019 06:36:53 +0200 (CEST) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-eopbgr60078.outbound.protection.outlook.com [40.107.6.78]) by dpdk.org (Postfix) with ESMTP id 92F851D156 for ; Sun, 13 Oct 2019 06:36:51 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bsCsKHh/f7IN+3f+ou2G2jQZrxVpk1qxNkJqTo0X6iw=; b=OO36V7Tjl1zBsxal2O1Amd7QSWHJd4i3UW0fmvFxLYy153fHmrTTeHY0Ghqehmyh/dQVlDJm4dFMr0LyKhq53YdhJ+/aBO8ugIafcwW4QmDFUuECy1jtBdort307Va3SGCWF5xm9+6y5YlOlVYShofHhbpMA2Bs8X5hd2929oF8= Received: from DB7PR08CA0035.eurprd08.prod.outlook.com (2603:10a6:5:16::48) by VI1PR08MB4335.eurprd08.prod.outlook.com (2603:10a6:803:fc::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2347.22; Sun, 13 Oct 2019 04:36:48 +0000 Received: from DB5EUR03FT008.eop-EUR03.prod.protection.outlook.com (2a01:111:f400:7e0a::205) by DB7PR08CA0035.outlook.office365.com (2603:10a6:5:16::48) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.2347.21 via Frontend Transport; Sun, 13 Oct 2019 04:36:47 +0000 Authentication-Results: spf=temperror (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=none action=none header.from=arm.com; Received-SPF: TempError (protection.outlook.com: error in processing during lookup of arm.com: DNS Timeout) Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DB5EUR03FT008.mail.protection.outlook.com (10.152.20.98) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.2305.15 via Frontend Transport; Sun, 13 Oct 2019 04:36:47 +0000 Received: ("Tessian outbound 3fba803f6da3:v33"); Sun, 13 Oct 2019 04:36:44 +0000 X-CR-MTA-TID: 64aa7808 Received: from 91a1c2ccf4ac.2 (ip-172-16-0-2.eu-west-1.compute.internal [104.47.10.53]) by 64aa7808-outbound-1.mta.getcheckrecipient.com id F6918005-1D5A-4219-8FBB-6CB3DDA1DBD6.1; Sun, 13 Oct 2019 04:36:38 +0000 Received: from EUR03-DB5-obe.outbound.protection.outlook.com (mail-db5eur03lp2053.outbound.protection.outlook.com [104.47.10.53]) by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 91a1c2ccf4ac.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384); Sun, 13 Oct 2019 04:36:38 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=flg1BCGsJELiDI3c55AD00FGl9T15b9002NUx+lHOXu6DRcLIe5mHUSvuS0cGdx195sx6xBjU5q4+lfKhVVZGmXst8KuE+Qyy4J3wpdlCmMC7IqgH3XE1i+vNmtmXnZH/QtF2pg3FwM1dkDrus5SWY42ByS+0/xGbN77PIxZdoJlqxtdTXDUHvH27g0AD++yjnH8I0M8z2Ambxp0drF5RbV218c/vVleYcpiEwMN/W5PwUA9nb0X3QPoWwuODsJZsgYG1+LX7V8pu8VkbDwCwc7byCdh7pqKv1fMUY6KGP895MlL1ZVeSPdynatRl9Zz7cx84wD2M7dsvysV+/0k+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bsCsKHh/f7IN+3f+ou2G2jQZrxVpk1qxNkJqTo0X6iw=; b=Qj7QT0E5w36/AZPdiHZZoOUE3gdy3eQMNjr3OKg2vh/H/SGCk+e5Yb+gcmDKZNhbZog3Z1x6kSmoheodxCBR+r+tzdytKkqMAERBkbFFh/MBO08YtW6xWixMYZa59mU6u3dDFdSB6GVodcrBrTf3JJOtrDDuGpo9JNft66Who4DtWf5H46A1KR5RFbXHBcyEAdpBAksR0wXuCC/to5F5+XlXW6ohgJjyzr60fno1DvlHQ9xYaRTghUkFVQmFdQ2AeW5rPhs8ZuF3feWKN/8JZE+/xBW6ibyg2z54fWdrKnTj6AVXhEzumTRdOeyRRVzsbKlpokuGvvIWk+EYQV7CMA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bsCsKHh/f7IN+3f+ou2G2jQZrxVpk1qxNkJqTo0X6iw=; b=OO36V7Tjl1zBsxal2O1Amd7QSWHJd4i3UW0fmvFxLYy153fHmrTTeHY0Ghqehmyh/dQVlDJm4dFMr0LyKhq53YdhJ+/aBO8ugIafcwW4QmDFUuECy1jtBdort307Va3SGCWF5xm9+6y5YlOlVYShofHhbpMA2Bs8X5hd2929oF8= Received: from VE1PR08MB5149.eurprd08.prod.outlook.com (20.179.30.27) by VE1PR08MB4910.eurprd08.prod.outlook.com (10.255.114.207) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2347.16; Sun, 13 Oct 2019 04:36:36 +0000 Received: from VE1PR08MB5149.eurprd08.prod.outlook.com ([fe80::8c82:8d9c:c78d:22a6]) by VE1PR08MB5149.eurprd08.prod.outlook.com ([fe80::8c82:8d9c:c78d:22a6%7]) with mapi id 15.20.2347.021; Sun, 13 Oct 2019 04:36:36 +0000 From: Honnappa Nagarahalli To: "Ananyev, Konstantin" , "Richardson, Bruce" , "Medvedkin, Vladimir" , "olivier.matz@6wind.com" CC: "dev@dpdk.org" , "stephen@networkplumber.org" , "paulmck@linux.ibm.com" , "Gavin Hu (Arm Technology China)" , Dharmik Thakkar , "Ruifeng Wang (Arm Technology China)" , nd , "Ruifeng Wang (Arm Technology China)" , Honnappa Nagarahalli , nd Thread-Topic: [PATCH v3 1/3] lib/lpm: integrate RCU QSBR Thread-Index: AQHVeIYn7Ch9Psq+bkuvfni/VY08BadO8EQAgAkGgoA= Date: Sun, 13 Oct 2019 04:36:36 +0000 Message-ID: References: <20190906094534.36060-1-ruifeng.wang@arm.com> <20191001182857.43867-1-honnappa.nagarahalli@arm.com> <20191001182857.43867-2-honnappa.nagarahalli@arm.com> <2601191342CEEE43887BDE71AB9772580191971F35@irsmsx105.ger.corp.intel.com> In-Reply-To: <2601191342CEEE43887BDE71AB9772580191971F35@irsmsx105.ger.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: f85638a2-68ad-486c-bdcf-a6151374db50.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; x-originating-ip: [217.140.111.135] x-ms-publictraffictype: Email X-MS-Office365-Filtering-Correlation-Id: e91ab2f9-5305-48df-d522-08d74f96f51a X-MS-Office365-Filtering-HT: Tenant X-MS-TrafficTypeDiagnostic: VE1PR08MB4910:|VE1PR08MB4910:|VI1PR08MB4335: x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true x-ms-oob-tlc-oobclassifiers: OLM:3044;OLM:3044; x-forefront-prvs: 01894AD3B8 X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(136003)(396003)(376002)(346002)(366004)(199004)(189003)(51444003)(486006)(4326008)(26005)(74316002)(99286004)(305945005)(316002)(52536014)(7696005)(76176011)(7736002)(30864003)(110136005)(5660300002)(25786009)(54906003)(33656002)(2906002)(476003)(11346002)(446003)(6506007)(76116006)(186003)(6116002)(3846002)(86362001)(102836004)(478600001)(81156014)(66946007)(8676002)(81166006)(71200400001)(6436002)(14454004)(9686003)(71190400001)(14444005)(256004)(2501003)(66556008)(66476007)(64756008)(66446008)(55016002)(229853002)(8936002)(66066001)(6246003); DIR:OUT; SFP:1101; SCL:1; SRVR:VE1PR08MB4910; H:VE1PR08MB5149.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: X2oox4gBobAYrZXMaoH6k1DnUcmduIfPdSvfywdlfmcCXlzzND1PcF4/oOtn+0KegnXvPa0Oy+JvB0dMFLX1qOVez2w4rzWwEjwJAR/r6TO+rpjuagOh4lJlQAyMVDvGI1s44AvoNbTdzTITV5U+DcV5ifXUpAw+EbidjTHPDhe/yjp8iUBtwiRGhIE5ta/UZsP8sw+GQ1KiDTg4GWkfa23gUF5AkXMqjVPjqRNlLBaJUOaE7SXiVsCtbU9dEEnQu7K00WrThq8wDLWnIw5IUuOeh/sUAzIHLbZF7cPYO7+lwt9QRq5BbjIP1UlHCBcNZP+EV0+u8qaX18I/T9UmeWnkouzY1dGUsyfM58LF5Jl7+CdqiiB372hxAlzQRX0WmV9hBq4E9g9J0LsYoCdZJUbsjjUIEqMtXanB1HDY4+A= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1PR08MB4910 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB5EUR03FT008.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; IPV:CAL; SCL:-1; CTRY:IE; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(346002)(136003)(396003)(376002)(51444003)(189003)(199004)(2906002)(52536014)(2501003)(46406003)(74316002)(8676002)(8746002)(81166006)(81156014)(8936002)(7736002)(305945005)(356004)(76130400001)(66066001)(4326008)(30864003)(50466002)(5660300002)(316002)(486006)(33656002)(22756006)(54906003)(110136005)(70586007)(6246003)(70206006)(25786009)(63350400001)(229853002)(102836004)(26005)(186003)(6506007)(86362001)(26826003)(478600001)(97756001)(126002)(476003)(14454004)(11346002)(446003)(9686003)(7696005)(23726003)(3846002)(6116002)(47776003)(99286004)(336012)(76176011)(14444005)(55016002); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR08MB4335; H:64aa7808-outbound-1.mta.getcheckrecipient.com; FPR:; SPF:TempError; LANG:en; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; MX:1; A:1; X-MS-Office365-Filtering-Correlation-Id-Prvs: 11b64d5e-0df2-4067-a898-08d74f96eee4 NoDisclaimer: True X-Forefront-PRVS: 01894AD3B8 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7DANT8TrzrjI6J72CN5xN7vucfgAp97iF3LKCZ8YhnVMSiEe6T+QsgJjFNc+BBDiSdFbCXmfDZJyL9DyEyvU1brgUo46AfyiNpjXHDZVtc6OM1t/ft4jaVUliqORjZ6udacELGl0L0BIaILoWcTA5lBOCAPss1zYwPhiXHHwrStcXuJSEi15iLwUXkSyk7c3ThMyWNwShGMe6vxVvMMXLEQEy9inhw1Lk72k2F7QRMvPZKUTd7yl9xp+bbiI7mWDWq6Ak/0IiTWJgv5FWJk8F3rYS6SLVPvUxKlyraa+Ut8ouni9RKDtfGaePoGdhHeKiikGShdcmbHhheh3hGeiadsXzhPeUQTwz76kDYbpnCBDDfRIU1a4SfR2CA7ol5yDKh3NBcVqVPetbs1Vu98/VuUJO2UuBqvcjA9zRLksYaA= X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Oct 2019 04:36:47.1449 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e91ab2f9-5305-48df-d522-08d74f96f51a X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR08MB4335 Subject: Re: [dpdk-dev] [PATCH v3 1/3] lib/lpm: integrate RCU QSBR X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > Hi guys, I have tried to consolidate design related questions here. If I have missed= anything, please add. >=20 > > > > From: Ruifeng Wang > > > > Currently, the tbl8 group is freed even though the readers might be > > using the tbl8 group entries. The freed tbl8 group can be reallocated > > quickly. This results in incorrect lookup results. > > > > RCU QSBR process is integrated for safe tbl8 group reclaim. > > Refer to RCU documentation to understand various aspects of > > integrating RCU library into other libraries. > > > > Signed-off-by: Ruifeng Wang > > Reviewed-by: Honnappa Nagarahalli > > --- > > lib/librte_lpm/Makefile | 3 +- > > lib/librte_lpm/meson.build | 2 + > > lib/librte_lpm/rte_lpm.c | 102 +++++++++++++++++++++++++---- > > lib/librte_lpm/rte_lpm.h | 21 ++++++ > > lib/librte_lpm/rte_lpm_version.map | 6 ++ > > 5 files changed, 122 insertions(+), 12 deletions(-) > > > > diff --git a/lib/librte_lpm/Makefile b/lib/librte_lpm/Makefile index > > a7946a1c5..ca9e16312 100644 > > --- a/lib/librte_lpm/Makefile > > +++ b/lib/librte_lpm/Makefile > > @@ -6,9 +6,10 @@ include $(RTE_SDK)/mk/rte.vars.mk # library name > > LIB =3D librte_lpm.a > > > > +CFLAGS +=3D -DALLOW_EXPERIMENTAL_API > > CFLAGS +=3D -O3 > > CFLAGS +=3D $(WERROR_FLAGS) -I$(SRCDIR) -LDLIBS +=3D -lrte_eal -lrte_h= ash > > +LDLIBS +=3D -lrte_eal -lrte_hash -lrte_rcu > > > > EXPORT_MAP :=3D rte_lpm_version.map > > > > diff --git a/lib/librte_lpm/meson.build b/lib/librte_lpm/meson.build > > index a5176d8ae..19a35107f 100644 > > --- a/lib/librte_lpm/meson.build > > +++ b/lib/librte_lpm/meson.build > > @@ -2,9 +2,11 @@ > > # Copyright(c) 2017 Intel Corporation > > > > version =3D 2 > > +allow_experimental_apis =3D true > > sources =3D files('rte_lpm.c', 'rte_lpm6.c') headers =3D > > files('rte_lpm.h', 'rte_lpm6.h') # since header files have different > > names, we can install all vector headers # without worrying about > > which architecture we actually need headers +=3D > > files('rte_lpm_altivec.h', 'rte_lpm_neon.h', 'rte_lpm_sse.h') deps += =3D > > ['hash'] > > +deps +=3D ['rcu'] > > diff --git a/lib/librte_lpm/rte_lpm.c b/lib/librte_lpm/rte_lpm.c index > > 3a929a1b1..ca58d4b35 100644 > > --- a/lib/librte_lpm/rte_lpm.c > > +++ b/lib/librte_lpm/rte_lpm.c > > @@ -1,5 +1,6 @@ > > /* SPDX-License-Identifier: BSD-3-Clause > > * Copyright(c) 2010-2014 Intel Corporation > > + * Copyright(c) 2019 Arm Limited > > */ > > > > #include > > @@ -381,6 +382,8 @@ rte_lpm_free_v1604(struct rte_lpm *lpm) > > > > rte_mcfg_tailq_write_unlock(); > > > > + if (lpm->dq) > > + rte_rcu_qsbr_dq_delete(lpm->dq); > > rte_free(lpm->tbl8); > > rte_free(lpm->rules_tbl); > > rte_free(lpm); > > @@ -390,6 +393,59 @@ BIND_DEFAULT_SYMBOL(rte_lpm_free, _v1604, > 16.04); > > MAP_STATIC_SYMBOL(void rte_lpm_free(struct rte_lpm *lpm), > > rte_lpm_free_v1604); > > > > +struct __rte_lpm_rcu_dq_entry { > > + uint32_t tbl8_group_index; > > + uint32_t pad; > > +}; > > + > > +static void > > +__lpm_rcu_qsbr_free_resource(void *p, void *data) { > > + struct rte_lpm_tbl_entry zero_tbl8_entry =3D {0}; > > + struct __rte_lpm_rcu_dq_entry *e =3D > > + (struct __rte_lpm_rcu_dq_entry *)data; > > + struct rte_lpm_tbl_entry *tbl8 =3D (struct rte_lpm_tbl_entry *)p; > > + > > + /* Set tbl8 group invalid */ > > + __atomic_store(&tbl8[e->tbl8_group_index], &zero_tbl8_entry, > > + __ATOMIC_RELAXED); > > +} > > + > > +/* Associate QSBR variable with an LPM object. > > + */ > > +int > > +rte_lpm_rcu_qsbr_add(struct rte_lpm *lpm, struct rte_rcu_qsbr *v) { > > + char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE]; > > + struct rte_rcu_qsbr_dq_parameters params; > > + > > + if ((lpm =3D=3D NULL) || (v =3D=3D NULL)) { > > + rte_errno =3D EINVAL; > > + return 1; > > + } > > + > > + if (lpm->dq) { > > + rte_errno =3D EEXIST; > > + return 1; > > + } > > + > > + /* Init QSBR defer queue. */ > > + snprintf(rcu_dq_name, sizeof(rcu_dq_name), "LPM_RCU_%s", lpm- > >name); > > + params.name =3D rcu_dq_name; > > + params.size =3D lpm->number_tbl8s; > > + params.esize =3D sizeof(struct __rte_lpm_rcu_dq_entry); > > + params.f =3D __lpm_rcu_qsbr_free_resource; > > + params.p =3D lpm->tbl8; > > + params.v =3D v; > > + lpm->dq =3D rte_rcu_qsbr_dq_create(¶ms); > > + if (lpm->dq =3D=3D NULL) { > > + RTE_LOG(ERR, LPM, "LPM QS defer queue creation failed\n"); > > + return 1; > > + } >=20 > Few thoughts about that function: Few things to keep in mind, the goal of the design is to make it easy for t= he applications to adopt lock-free algorithms. The reclamation process in t= he writer is a major portion of code one has to write for using lock-free a= lgorithms. The current design is such that the writer does not have to chan= ge any code or write additional code other than calling 'rte_lpm_rcu_qsbr_a= dd'. > It names rcu_qsbr_add() but in fact it allocates defer queue for give rcu= var. > So first thought - is it always necessary? This is part of the design. If the application does not want to use this in= tegrated logic then, it does not have to call this API. It can use the RCU = defer APIs to implement its own logic. But, if I ask the question, does thi= s integrated logic address most of the use cases of the LPM library, I thin= k the answer is yes. > For some use-cases I suppose user might be ok to wait for quiescent state > change > inside tbl8_free()? Yes, that is a possibility (for ex: no frequent route changes). But, I thin= k that is very trivial for the application to implement. Though, the LPM li= brary has to separate the 'delete' and 'free' operations. Similar operation= s are provided in rte_hash library. IMO, we should follow consistent approa= ch. > Another thing you do allocate defer queue, but it is internal, so user ca= n't call > reclaim() manually, which looks strange. > Why not to return defer_queue pointer to the user, so he can call reclaim= () > himself at appropriate time? The intention of the design is to take the complexity away from the user of= LPM library. IMO, the current design will address most uses cases of LPM l= ibrary. If we expose the 2 parameters (when to trigger reclamation and how = much to reclaim) in the 'rte_lpm_rcu_qsbr_add' API, it should provide enoug= h flexibility to the application. > Third thing - you always allocate defer queue with size equal to number o= f > tbl8. > Though I understand it could be up to 16M tbl8 groups inside the LPM. > Do we really need defer queue that long? No, we do not need it to be this long. It is this long today to avoid retur= ning no-space on the defer queue error. > Especially considering that current rcu_defer_queue will start reclamati= on > when 1/8 of defer_quueue becomes full and wouldn't reclaim more then > 1/16 of it. > Probably better to let user to decide himself how long defer_queue he nee= ds > for that LPM? It makes sense to expose it to the user if the writer-writer concurrency is= lock-free (no memory allocation allowed to expand the defer queue size whe= n the queue is full). However, LPM is not lock-free on the writer side. If = we think the writer could be lock-free in the future, it has to be exposed = to the user.=20 >=20 > Konstantin Pulling questions/comments from other threads: Can we leave reclamation to some other house-keeping thread to do (sort of = garbage collector). Or such mode is not supported/planned? [Honnappa] If the reclamation cost is small, the current method provides ad= vantages over having a separate thread to do reclamation. I did not plan to= provide such an option. But may be it makes sense to keep the options open= (especially from ABI perspective). May be we should add a flags field whic= h will allow us to implement different methods in the future? >=20 >=20 > > + > > + return 0; > > +} > > + > > /* > > * Adds a rule to the rule table. > > * > > @@ -679,14 +735,15 @@ tbl8_alloc_v20(struct rte_lpm_tbl_entry_v20 > > *tbl8) } > > > > static int32_t > > -tbl8_alloc_v1604(struct rte_lpm_tbl_entry *tbl8, uint32_t > > number_tbl8s) > > +__tbl8_alloc_v1604(struct rte_lpm *lpm) > > { > > uint32_t group_idx; /* tbl8 group index. */ > > struct rte_lpm_tbl_entry *tbl8_entry; > > > > /* Scan through tbl8 to find a free (i.e. INVALID) tbl8 group. */ > > - for (group_idx =3D 0; group_idx < number_tbl8s; group_idx++) { > > - tbl8_entry =3D &tbl8[group_idx * > RTE_LPM_TBL8_GROUP_NUM_ENTRIES]; > > + for (group_idx =3D 0; group_idx < lpm->number_tbl8s; group_idx++) { > > + tbl8_entry =3D &lpm->tbl8[group_idx * > > + > RTE_LPM_TBL8_GROUP_NUM_ENTRIES]; > > /* If a free tbl8 group is found clean it and set as VALID. */ > > if (!tbl8_entry->valid_group) { > > struct rte_lpm_tbl_entry new_tbl8_entry =3D { @@ - > 712,6 +769,21 @@ > > tbl8_alloc_v1604(struct rte_lpm_tbl_entry *tbl8, uint32_t number_tbl8s) > > return -ENOSPC; > > } > > > > +static int32_t > > +tbl8_alloc_v1604(struct rte_lpm *lpm) { > > + int32_t group_idx; /* tbl8 group index. */ > > + > > + group_idx =3D __tbl8_alloc_v1604(lpm); > > + if ((group_idx < 0) && (lpm->dq !=3D NULL)) { > > + /* If there are no tbl8 groups try to reclaim some. */ > > + if (rte_rcu_qsbr_dq_reclaim(lpm->dq) =3D=3D 0) > > + group_idx =3D __tbl8_alloc_v1604(lpm); > > + } > > + > > + return group_idx; > > +} > > + > > static void > > tbl8_free_v20(struct rte_lpm_tbl_entry_v20 *tbl8, uint32_t > > tbl8_group_start) { @@ -728,13 +800,21 @@ tbl8_free_v20(struct > > rte_lpm_tbl_entry_v20 *tbl8, uint32_t tbl8_group_start) } > > > > static void > > -tbl8_free_v1604(struct rte_lpm_tbl_entry *tbl8, uint32_t > > tbl8_group_start) > > +tbl8_free_v1604(struct rte_lpm *lpm, uint32_t tbl8_group_start) > > { > > - /* Set tbl8 group invalid*/ > > struct rte_lpm_tbl_entry zero_tbl8_entry =3D {0}; > > + struct __rte_lpm_rcu_dq_entry e; > > > > - __atomic_store(&tbl8[tbl8_group_start], &zero_tbl8_entry, > > - __ATOMIC_RELAXED); > > + if (lpm->dq !=3D NULL) { > > + e.tbl8_group_index =3D tbl8_group_start; > > + e.pad =3D 0; > > + /* Push into QSBR defer queue. */ > > + rte_rcu_qsbr_dq_enqueue(lpm->dq, (void *)&e); > > + } else { > > + /* Set tbl8 group invalid*/ > > + __atomic_store(&lpm->tbl8[tbl8_group_start], > &zero_tbl8_entry, > > + __ATOMIC_RELAXED); > > + } > > } > > > > static __rte_noinline int32_t > > @@ -1037,7 +1117,7 @@ add_depth_big_v1604(struct rte_lpm *lpm, > > uint32_t ip_masked, uint8_t depth, > > > > if (!lpm->tbl24[tbl24_index].valid) { > > /* Search for a free tbl8 group. */ > > - tbl8_group_index =3D tbl8_alloc_v1604(lpm->tbl8, lpm- > >number_tbl8s); > > + tbl8_group_index =3D tbl8_alloc_v1604(lpm); > > > > /* Check tbl8 allocation was successful. */ > > if (tbl8_group_index < 0) { > > @@ -1083,7 +1163,7 @@ add_depth_big_v1604(struct rte_lpm *lpm, > uint32_t ip_masked, uint8_t depth, > > } /* If valid entry but not extended calculate the index into Table8.= */ > > else if (lpm->tbl24[tbl24_index].valid_group =3D=3D 0) { > > /* Search for free tbl8 group. */ > > - tbl8_group_index =3D tbl8_alloc_v1604(lpm->tbl8, lpm- > >number_tbl8s); > > + tbl8_group_index =3D tbl8_alloc_v1604(lpm); > > > > if (tbl8_group_index < 0) { > > return tbl8_group_index; > > @@ -1818,7 +1898,7 @@ delete_depth_big_v1604(struct rte_lpm *lpm, > uint32_t ip_masked, > > */ > > lpm->tbl24[tbl24_index].valid =3D 0; > > __atomic_thread_fence(__ATOMIC_RELEASE); > > - tbl8_free_v1604(lpm->tbl8, tbl8_group_start); > > + tbl8_free_v1604(lpm, tbl8_group_start); > > } else if (tbl8_recycle_index > -1) { > > /* Update tbl24 entry. */ > > struct rte_lpm_tbl_entry new_tbl24_entry =3D { @@ -1834,7 > +1914,7 @@ > > delete_depth_big_v1604(struct rte_lpm *lpm, uint32_t ip_masked, > > __atomic_store(&lpm->tbl24[tbl24_index], > &new_tbl24_entry, > > __ATOMIC_RELAXED); > > __atomic_thread_fence(__ATOMIC_RELEASE); > > - tbl8_free_v1604(lpm->tbl8, tbl8_group_start); > > + tbl8_free_v1604(lpm, tbl8_group_start); > > } > > #undef group_idx > > return 0; > > diff --git a/lib/librte_lpm/rte_lpm.h b/lib/librte_lpm/rte_lpm.h index > > 906ec4483..49c12a68d 100644 > > --- a/lib/librte_lpm/rte_lpm.h > > +++ b/lib/librte_lpm/rte_lpm.h > > @@ -1,5 +1,6 @@ > > /* SPDX-License-Identifier: BSD-3-Clause > > * Copyright(c) 2010-2014 Intel Corporation > > + * Copyright(c) 2019 Arm Limited > > */ > > > > #ifndef _RTE_LPM_H_ > > @@ -21,6 +22,7 @@ > > #include > > #include > > #include > > +#include > > > > #ifdef __cplusplus > > extern "C" { > > @@ -186,6 +188,7 @@ struct rte_lpm { > > __rte_cache_aligned; /**< LPM tbl24 table. */ > > struct rte_lpm_tbl_entry *tbl8; /**< LPM tbl8 table. */ > > struct rte_lpm_rule *rules_tbl; /**< LPM rules. */ > > + struct rte_rcu_qsbr_dq *dq; /**< RCU QSBR defer queue.*/ > > }; > > > > /** > > @@ -248,6 +251,24 @@ rte_lpm_free_v20(struct rte_lpm_v20 *lpm); > void > > rte_lpm_free_v1604(struct rte_lpm *lpm); > > > > +/** > > + * Associate RCU QSBR variable with an LPM object. > > + * > > + * @param lpm > > + * the lpm object to add RCU QSBR > > + * @param v > > + * RCU QSBR variable > > + * @return > > + * On success - 0 > > + * On error - 1 with error code set in rte_errno. > > + * Possible rte_errno codes are: > > + * - EINVAL - invalid pointer > > + * - EEXIST - already added QSBR > > + * - ENOMEM - memory allocation failure > > + */ > > +__rte_experimental > > +int rte_lpm_rcu_qsbr_add(struct rte_lpm *lpm, struct rte_rcu_qsbr > > +*v); > > + > > /** > > * Add a rule to the LPM table. > > * > > diff --git a/lib/librte_lpm/rte_lpm_version.map > > b/lib/librte_lpm/rte_lpm_version.map > > index 90beac853..b353aabd2 100644 > > --- a/lib/librte_lpm/rte_lpm_version.map > > +++ b/lib/librte_lpm/rte_lpm_version.map > > @@ -44,3 +44,9 @@ DPDK_17.05 { > > rte_lpm6_lookup_bulk_func; > > > > } DPDK_16.04; > > + > > +EXPERIMENTAL { > > + global: > > + > > + rte_lpm_rcu_qsbr_add; > > +}; > > -- > > 2.17.1