From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 100F2A0561; Mon, 20 Apr 2020 10:19:47 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id ED41B1D41D; Mon, 20 Apr 2020 10:19:46 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by dpdk.org (Postfix) with ESMTP id 709161D416 for ; Mon, 20 Apr 2020 10:19:45 +0200 (CEST) IronPort-SDR: aM+FHELif9xVfdzk4ePUCGlx+sE3h+huPlNeaI1jm6I2pra/kGnnqIW701476uCXQJqLYCTR31 /0yHeXkPfrCw== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Apr 2020 01:19:44 -0700 IronPort-SDR: 53px35UQsVXP+jEG5RvUJNXObbioKsqcI0jrcr0LfwiTcZMkJjD7lErHukEOvn9e+ZfMO/EpF+ lT2KvfA4qEbw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,406,1580803200"; d="scan'208";a="246836636" Received: from orsmsx104.amr.corp.intel.com ([10.22.225.131]) by fmsmga008.fm.intel.com with ESMTP; 20 Apr 2020 01:19:44 -0700 Received: from ORSEDG002.ED.cps.intel.com (10.7.248.5) by ORSMSX104.amr.corp.intel.com (10.22.225.131) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 20 Apr 2020 01:19:43 -0700 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (104.47.59.173) by edgegateway.intel.com (134.134.137.101) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 20 Apr 2020 01:19:43 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=K5CDBSzcsTAVZgMtI2f0o6brwdY+MuEwHj3tuO1g4D6mYSUC/ND7PJulWMbu14yCHf+DVTJcDCtmR8jjhw7xnuYKPx4aEQD7YXexrBbZ7t9aGiTG5j3f80rVr4DtyKdsTv++nxePg2Dx8oAXxMs2bc2K/gCRCH8T/8Hc9KKLRt2ucer8aNO/izXJaKrn28nOsZjBtNKccCs+8MoMve5SqrNvXD1v8d6IcEevKKHVrVcOoFKxNG72CUZKFUB0DPdyjIkP+g+6CiIWX8/DligtqtbSfpooni9Rq5BfBy75DeG4q7pu0N44OIflAlg6NGGigij2cnLwBzShe3m7ZbdP1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rIBg87goYNOYkq4qBEz36cQIuLJtqh22xvn04gbHWz8=; b=R8ZZWaIwjlaLgMifXJ+picPPfvdPRECko2TNGNTfOfwMwKt0WRIyQcRVVhXkkWWfR03EGIrEycONHK0cUS5Sy8joIS1f9yRiiH3STiQ12RW6h4aobYp7FfHIIOVAQEQt3+ufBU4XHNs22DjsPM4RKWDdsluygAGgEd0MMoJPc9S0QLAFEmG+7dj0gRcNXuRiA6dDRuhi4xRSzUFOwtnHO6OBdL2shfqpV80QZfarP3wVxlXBOo8bLA+BbdOwhzoiYb3JfskXOql47/vvKvZY0xbfQLMsT9z6vY/njOkX9HpWFrdYFKygNmcvm8U9DaLqPKa834pmfGwiMOKU5xgKSw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rIBg87goYNOYkq4qBEz36cQIuLJtqh22xvn04gbHWz8=; b=zZlWaFX8xnIK8peZ5ao1liWkJJFw4lkG7lu9xrekRhGou3EpAactsM1uCMkZvRfkt4dag+xerRF3G/sMIVqZWbLaKlCh4+aCOz+9uv4SRRfomQ3qMKbnHOhudVlOyB1bFbTrT35hK4/aTfTAT37Ck/kruyJe4LWLUaypXcO9S04= Received: from BYAPR11MB3301.namprd11.prod.outlook.com (2603:10b6:a03:7f::26) by BYAPR11MB2837.namprd11.prod.outlook.com (2603:10b6:a02:c6::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2921.29; Mon, 20 Apr 2020 08:19:42 +0000 Received: from BYAPR11MB3301.namprd11.prod.outlook.com ([fe80::f8cb:58cd:e958:fff4]) by BYAPR11MB3301.namprd11.prod.outlook.com ([fe80::f8cb:58cd:e958:fff4%6]) with mapi id 15.20.2921.027; Mon, 20 Apr 2020 08:19:42 +0000 From: "Ananyev, Konstantin" To: Honnappa Nagarahalli , "stephen@networkplumber.org" , "Medvedkin, Vladimir" CC: "dev@dpdk.org" , Ruifeng Wang , Dharmik Thakkar , nd , nd Thread-Topic: [PATCH v4 1/4] lib/rcu: add resource reclamation APIs Thread-Index: AQHWCeeyx2wQaIFUDkKfKuH/bhXgO6ht5FcAgBNKl4CAAJCtEA== Date: Mon, 20 Apr 2020 08:19:41 +0000 Message-ID: References: <20191001062917.35578-1-honnappa.nagarahalli@arm.com> <20200403184142.7729-1-honnappa.nagarahalli@arm.com> <20200403184142.7729-2-honnappa.nagarahalli@arm.com> In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-reaction: no-action dlp-version: 11.2.0.6 authentication-results: spf=none (sender IP is ) smtp.mailfrom=konstantin.ananyev@intel.com; x-originating-ip: [192.198.151.170] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: a375b863-87c7-49cf-0c2d-08d7e5039398 x-ms-traffictypediagnostic: BYAPR11MB2837: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-forefront-prvs: 03793408BA x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR11MB3301.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(10019020)(366004)(136003)(346002)(39860400002)(396003)(376002)(52536014)(478600001)(54906003)(110136005)(33656002)(6636002)(7696005)(71200400001)(6506007)(4326008)(2906002)(66446008)(64756008)(66556008)(66476007)(81156014)(8676002)(66946007)(76116006)(5660300002)(8936002)(316002)(9686003)(186003)(55016002)(86362001)(26005)(60764002); DIR:OUT; SFP:1102; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Oyl/UcHb3q+YY9aNXNPFlid8JBeztwz8x8Hddf9hpjBjhDSNtTset09Ju8efDWjqk/UMZvI0JD4g/v+sXEPhse0dNyYFaxHI4g6OglP7PjLUyo/s19Mqc8D7oBz1ABMlzAOLwuSYKhZDtU5hn0IRyUGS46OyiW7k9gVLKYVsr+uld8dmdcHitz543gUIPla3uLfvVp8hoc2cp9ZwTmvrhFHA4RxpvlTt2FFI4+oScVuEN8XFHiWugAjM3IuS1esMO6g4QXRpmcmfwBO+STsjh35G29mP7faDRwRvFVi5FezG0IZZMvMv7YPTElYgIRcFZrX9WIP6eYpzIAFcxNCZN50XvOfRl9dpWyOKVeyU7HoUQHDpjty0BgDQuWWtLPJ04dxLh+aQ15kCuwGvKAndGem+OKrgzfEBDgCvssUnQeAwqnSaXut6iVinzAGSrHeRLj7vmRq9mWeVgllPJLPT7jOUez4XiRKuKBZYyqmWiPu+wPknIYObp8tyfW7kXcir x-ms-exchange-antispam-messagedata: 2NywBkK/PQvh/+/NPPKzWp1iyyGjWH8DFrX8GnHUnXaDChMyvAQxmQZHdTvPJxbCyjVlhQ7CAI5bj7UZcB8Ykz0yGLDuJzByDZCwBtg3XuzoaVto5Y775BQK7f/w5x1ELqaJgLxwqJuTzAz86nmZIw== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: a375b863-87c7-49cf-0c2d-08d7e5039398 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Apr 2020 08:19:41.6064 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: vj/nEgepzHAQGLuOB+9BpAKDGyxM9D2srnfNnL27scpZfdM+UJWbhFs7icUhfH4bKKeejSFdpYWWXv6dLO797mm8lppJ76PneW9TFTOcdyg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB2837 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH v4 1/4] lib/rcu: add resource reclamation APIs X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > > > + > > > + /* Enqueue the token and resource. Generating the token > > > + * and enqueuing (token + resource) on the queue is not an > > > + * atomic operation. This might result in tokens enqueued > > > + * out of order on the queue. So, some tokens might wait > > > + * longer than they are required to be reclaimed. > > > + */ > > > + char data[dq->esize]; > > > + memcpy(data, &token, __RTE_QSBR_TOKEN_SIZE); > > > + memcpy(data + __RTE_QSBR_TOKEN_SIZE, e, > > > + dq->esize - __RTE_QSBR_TOKEN_SIZE); > > > + /* Check the status as enqueue might fail since the other thread > > > + * might have used up the freed space. > > > + * Enqueue uses the configured flags when the DQ was created. > > > + */ > > > + if (rte_ring_enqueue_elem(dq->r, data, dq->esize) !=3D 0) { > > > + rte_log(RTE_LOG_ERR, rte_rcu_log_type, > > > + "%s(): Enqueue failed\n", __func__); > > > + /* Note that the token generated above is not used. > > > + * Other than wasting tokens, it should not cause any > > > + * other issues. > > > + */ > > > + rte_log(RTE_LOG_INFO, rte_rcu_log_type, > > > + "%s(): Skipped enqueuing token =3D %"PRIu64"\n", > > > + __func__, token); > > > + > > > + rte_errno =3D ENOSPC; > > > + return 1; > > > + } > > > > > > Just as a thought: in theory if we'll use MP_HTS(/SP) ring we can avoid > > wasting RCU tokens: > > > > if (rte_ring_enqueue_elem_bulk_start(dq->r, 1, NULL) !=3D 0) { > > token =3D rte_rcu_qsbr_start(dq->v); > > memcpy(data, &token, __RTE_QSBR_TOKEN_SIZE); > > rte_ring_enqueue_elem_finish(dq->r, data, dq->esize, 1); } > > > > Though it might slowdown things if we'll have a lot of parallel dq_enqu= eue. > > So not sure is it worth it or not. > Adding peek APIs for RTS would be better. That should take care of the pa= rallel dw_enqueue. Not sure if I gave you the comment. My ring > patch supported these APIs. AFAIK, peek API is not possible for RTS mode. Probably you are talking about Scatter-Gather API introduced in your RFC (_reserve_; update ring entries manually; _commit_)? Anyway, if there is no much value in my idea above, then feel free to drop = it. >=20 > > > > > + > > > + rte_log(RTE_LOG_INFO, rte_rcu_log_type, > > > + "%s(): Enqueued token =3D %"PRIu64"\n", __func__, token); > > > + > > > + return 0; > > > +} > > > + > > > +/* Reclaim resources from the defer queue. */ int > > > +rte_rcu_qsbr_dq_reclaim(struct rte_rcu_qsbr_dq *dq, unsigned int n, > > > + unsigned int *freed, unsigned int *pending) { > > > + uint32_t cnt; > > > + uint64_t token; > > > + > > > + if (dq =3D=3D NULL || n =3D=3D 0) { > > > + rte_log(RTE_LOG_ERR, rte_rcu_log_type, > > > + "%s(): Invalid input parameter\n", __func__); > > > + rte_errno =3D EINVAL; > > > + > > > + return 1; > > > + } > > > + > > > + cnt =3D 0; > > > + > > > + char e[dq->esize]; > > > + /* Check reader threads quiescent state and reclaim resources */ > > > + while ((cnt < n) && > > > + (rte_ring_dequeue_bulk_elem_start(dq->r, e, > > > + dq->esize, 1, NULL) !=3D 0)) { > > > > Another thought - any point to use burst_elem_start() here to retrieve = more > > then 1 elem in one go? Something like: > I think it makes sense. >=20 > > char e[32][dq->size]; > > while ((cnt < n) { > > k =3D RTE_MAX(32, cnt - n); > > k =3D rte_ring_dequeue_burst_elem_start(dq->r, e, dq->esize, k, NULL); > > if (k =3D 0) > > break; > > for (i =3D 0; i !=3D k; i++) { > > memcpy(&token, e[i], sizeof(uint64_t)); > > if (rte_rcu_qsbr_check(dq->v, token, false) !=3D 1) > > break; > > } > > k =3D i; > > rte_ring_dequeue_elem_finish(dq->r, k); > > for (i =3D 0; i !=3D k; i++) > > dq->free_fn(dq->p, e[i] + __RTE_QSBR_TOKEN_SIZE); > I think it also makes sense to change the free_fn to take 'n' number of t= okens. >=20 > > n +=3D k; > > if (k =3D=3D 0) > > break; > > > > ? > > Also if at enqueue we guarantee strict ordrer (via > > enqueue_start/enqueue_finish), then here we probably can do _check_ fro= m > > the last retrieved token here? > > In theory that might help to minimize number of checks. > > I.E. do: > > for (i =3D k; i-- !=3D0; ) { > > memcpy(&token, e[i], sizeof(uint64_t)); > > if (rte_rcu_qsbr_check(dq->v, token, false) !=3D 1) > There is a higher chance that later tokens are not acked. This introduces= more polling of the counters. > The rte_rcu_qsbr_check has an optimization. While acking the current toke= n, it will also caches the greatest token acked. It uses the cached > token for the subsequent calls. I think this provides a better optimizati= on. Ok.=20