From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 08053A0A0A; Fri, 22 Jan 2021 14:14:55 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CF1D2140FBE; Fri, 22 Jan 2021 14:14:55 +0100 (CET) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id 75BE2140FB7 for ; Fri, 22 Jan 2021 14:14:53 +0100 (CET) IronPort-SDR: VYT4HgSXXqhS1ZZfW8MN3LIFrL0hNJW7YLpQx1hsxdnmTxyVBAZHA4JaqMA2wKRPsmh2YXsYK/ sILnj2l/hh3g== X-IronPort-AV: E=McAfee;i="6000,8403,9871"; a="176869496" X-IronPort-AV: E=Sophos;i="5.79,366,1602572400"; d="scan'208";a="176869496" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Jan 2021 05:08:49 -0800 IronPort-SDR: KbSQBnWECEm1YWHZvESkOjPaH2UlL5MqppV+DCHp+G7D0jNSIzvUlVMX0aIYvYxwbftJhEh39r jWxX2bJdkvyw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.79,366,1602572400"; d="scan'208";a="574643034" Received: from orsmsx605.amr.corp.intel.com ([10.22.229.18]) by fmsmga005.fm.intel.com with ESMTP; 22 Jan 2021 05:08:49 -0800 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX605.amr.corp.intel.com (10.22.229.18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 22 Jan 2021 05:08:49 -0800 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Fri, 22 Jan 2021 05:08:49 -0800 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.170) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Fri, 22 Jan 2021 05:08:48 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KjYjqYlCt7j0Oau2+EkgDHSHk99NCbLUxIxFUt1byz2ham7G/JdfPn0d4ALhozn2EpOyoNHRzqQ5OYIG1KIjx5nuc0z+SeCdRgIkDuHGyN0cVM18LRIhLRSp+22gRd78pI5OUncLEjc44C3+rMDxo0fTNeD7F+EuC0nz4EVyrDgAuC/d+L+AXQ3xG6C6SxrHAwgcR3U+ToFd7+1kBbBL5z2nBBYDo7cpYD5VixRXaLFnLiv01VFqoFGFh8MBoMI9gB+ji/9Vxcg2dH2plzn/txjIOw1y9c2Y3iNPeowNO3G5bg2ozbAWj/Jgjicyt/jQJMSucGveTkGIE4d/mDy3lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=y7/xPC5kle5Z2tdHvYJTJSQPUMMxHGT+uq0wcIn6wI8=; b=WxIfpJ4+xXo6szabeaMi4Wu4wPwH5W7li1/q/hzB0Zh7t7I8fxkOG4rP7SFpVClYKURxaEmoN1ydeUSlxiCLrWnb2AZjdJu8Vk1W+UZ53pV6mclIKkhehP8bdU4FiaxaYO9BY98y7i810SzjJXaH+G6sxE2Y3d2AUlhmPDSlhnM5plIpsxlIDhYO/EcEVBcXybsjYlZW9zilve5MRkw17jC2ywN5y4l19ZILiBi0x9KGAXtBweHjD1v99QgNnPcbBLUVfx9zJMA5k5mzkrceXyxsdWYjY7gT/9F00Agza54G7uGk2WILD2V59voSbOy0+FKeY9izoIO7OMsuXBpqDQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=y7/xPC5kle5Z2tdHvYJTJSQPUMMxHGT+uq0wcIn6wI8=; b=k/Z3JjrPWRdYd885oJ0P3lyAjgIPRAPnT3HGNbftwfc7lFjj5XgzCl/VH0I0oENRGFVhnbxQ43IbXMOsrCIiAEMDIbWjPNmZluGzZO2DZ8a1ghAgGy0lL/ntL3DbiDDhFNH31uoSfp3wg4sgCeDvo2WIgUqP6BAXe9n/HFmcBKA= Received: from BYAPR11MB3301.namprd11.prod.outlook.com (2603:10b6:a03:7f::26) by BYAPR11MB3253.namprd11.prod.outlook.com (2603:10b6:a03:77::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.12; Fri, 22 Jan 2021 13:08:47 +0000 Received: from BYAPR11MB3301.namprd11.prod.outlook.com ([fe80::1152:1426:8a4f:c755]) by BYAPR11MB3301.namprd11.prod.outlook.com ([fe80::1152:1426:8a4f:c755%4]) with mapi id 15.20.3784.013; Fri, 22 Jan 2021 13:08:47 +0000 From: "Ananyev, Konstantin" To: Feifei Wang , Honnappa Nagarahalli CC: "dev@dpdk.org" , "nd@arm.com" Thread-Topic: [PATCH v1 3/3] ring: rename and refactor ring library Thread-Index: AQHW14qBH2SQ6bbJJ0qiJgdUKwT3Yqozzsbw Date: Fri, 22 Jan 2021 13:08:46 +0000 Message-ID: References: <20201221111359.22013-1-feifei.wang2@arm.com> <20201221111359.22013-4-feifei.wang2@arm.com> In-Reply-To: <20201221111359.22013-4-feifei.wang2@arm.com> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-reaction: no-action dlp-version: 11.5.1.3 authentication-results: arm.com; dkim=none (message not signed) header.d=none;arm.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [46.7.39.127] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: bfb585ec-944f-4c06-0a6c-08d8bed6da80 x-ms-traffictypediagnostic: BYAPR11MB3253: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Tu9PHFH24BVACHutDLObRoAmahZjsfCbi7lWyoBVRop4I/6qcjaTDtfpUD6YG+XYQq9sMeIx0kbbG65vjdv6JRIr57usRSpzrvNHdkn+kRwXXV8H3HhhWYq6F+pNg8/9LJIn/+Kqrc+08pYYMezrrW5z7OSBnwkaRCbpQuLJHhKwk2M5rZB+6su9ehtvevYiYJhwjmzeY27lrjYVJImlXIiOjs833VsIEBQy+/46e4NJd9MEsIwQlzeMLWED8RjMyNBowPWDwUKj48STxd+oC3udI13daJiDRfbW0ZNDDXNKMSIBnuH5WKHK5Y0J48/Erk4f8ooL5Xo1T8KaWBVFbH99wnjmNfXxJGsRZ8m6JFa0YQgR3cshHV0ZZ4noau6zSgHTU0zrIyZErya6dzT7hQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR11MB3301.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(346002)(136003)(366004)(376002)(396003)(39860400002)(54906003)(86362001)(478600001)(30864003)(9686003)(4326008)(110136005)(316002)(55016002)(83380400001)(33656002)(26005)(186003)(6506007)(71200400001)(7696005)(52536014)(2906002)(5660300002)(8936002)(66476007)(66556008)(64756008)(8676002)(76116006)(66446008)(66946007)(559001)(579004); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?PIuMkktftD0ZTXKZnhUK21m6ZT8l1MCoNeUaz3Cf/cjlTkvLJ3nKZgifLDzy?= =?us-ascii?Q?967o+SeIYaJsAorAH614+gUK8HDh08dOqYjEghJ+jpvnCAv3Wzi9cF3Lwjhi?= =?us-ascii?Q?mLfYkRfZ+qTcDJ1pmqvqTPeGbjqjp5a0N+MNJrmXYtKuqFfPSZbhXnLqPGMu?= =?us-ascii?Q?3eDOsDK6eQDTA/vE6Vkc4t/Nw/ecH5ug2LwFj8OKnzzJpzcNSwJBfpSq2qz0?= =?us-ascii?Q?8fQjjMmq+eKQ7Xhpidz/79572CRTWuysNEID0mUw/4ODxh2Kw0uycV4go3Ij?= =?us-ascii?Q?8uAxzFnoynyw+yEWBIaOt1pPclgDdeWvFPz/WLu8Zi9RkbIGFQc1914/7meZ?= =?us-ascii?Q?LHeiy5QhNzlL2Wpi1UEkdxGDAJeE6zL+GU+N+uOabhuOSIxDkHmB9x5zwgRy?= =?us-ascii?Q?NlHZE45sCXK5tGFZxaJBrFJyvmSSTf2BEpCx0vMDrmPmDEQ2ULr1fTsg2AMv?= =?us-ascii?Q?1RvTv2aAYO2uRFWlAP8ZfTxed4GQL9kGaU33bmx/YV8lRKZbBwJSllLZKZAs?= =?us-ascii?Q?nGwxa0FlrzHnawf5DV04+3FMKwP3rAsnH/jZZVAnSGr5XGFZLoQKGqMev03L?= =?us-ascii?Q?wyW6Dp0SghlGqawDRK6sZZKgnvPVOhr31wGjnSXTGtNPDG6aOBlkiz/8z84v?= =?us-ascii?Q?SVcPipYWA+CAH95TyRDlxLVid/3c/lz/TXFeHlYkGg7idAJ6nhY7k44e/T6G?= =?us-ascii?Q?0qmo+dH+/bWKPwvEpK2LY4NDoiajHc5b2dy0tVqP1KuxMRtudPcVSARVcbe8?= =?us-ascii?Q?DuukaU+0QBqtaD/3Jr9ajNzS91TAycLIgqBF350t7uIIi6H28Thatamh3Dh+?= =?us-ascii?Q?cJdNgD5sykP33WVcpA3P+3fqy/I0qKWT+bdoHADzbI1Rq3jRAR3+OZzu3KKJ?= =?us-ascii?Q?uqew5x9ZFXgu1c4UGBRnMta8uvapob0RDTRn4czKWfG8m+GhxfyQvsOLN1vv?= =?us-ascii?Q?mu2SfqelsZTZGPd12xyshHZILchw2qLw3H9OY2J+pGrC0XZT4a3boOkKtj2I?= =?us-ascii?Q?GMeOxfiLcTM9CuGbJR/asiWm81O+cIb8GAC4upzbEGPXLaM=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3301.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: bfb585ec-944f-4c06-0a6c-08d8bed6da80 X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Jan 2021 13:08:46.9869 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: C9TvmVsj2AMkuvXvP2wjQS9mEaKMyQTl+e8usnu+vWc6AfJVS8NtLaO/5YGNmGVWoN3+eLfghA9495R5nVPVkfeje1ZHtPLJTf299jb4Nek= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB3253 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH v1 3/3] ring: rename and refactor ring library X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > For legacy modes, rename ring_generic/c11 to ring_generic/c11_pvt. > Furthermore, add new file ring_elem_pvt.h which includes ring_do_eq/deq > and ring element copy/delete APIs. >=20 > For other modes, rename xx_c11_mem to xx_elem_pvt. Move all private APIs > into these new header files. >=20 > Suggested-by: Honnappa Nagarahalli > Signed-off-by: Feifei Wang > Reviewed-by: Honnappa Nagarahalli > Reviewed-by: Ruifeng Wang > --- > lib/librte_ring/meson.build | 15 +- > .../{rte_ring_c11_mem.h =3D> ring_c11_pvt.h} | 9 +- > lib/librte_ring/ring_elem_pvt.h | 385 ++++++++++++++++++ > ...{rte_ring_generic.h =3D> ring_generic_pvt.h} | 6 +- > ...ring_hts_c11_mem.h =3D> ring_hts_elem_pvt.h} | 88 +++- > ...ng_peek_c11_mem.h =3D> ring_peek_elem_pvt.h} | 75 +++- > ...ring_rts_c11_mem.h =3D> ring_rts_elem_pvt.h} | 88 +++- > lib/librte_ring/rte_ring_elem.h | 374 +---------------- > lib/librte_ring/rte_ring_hts.h | 84 +--- > lib/librte_ring/rte_ring_peek.h | 71 +--- > lib/librte_ring/rte_ring_peek_zc.h | 2 +- > lib/librte_ring/rte_ring_rts.h | 84 +--- > 12 files changed, 646 insertions(+), 635 deletions(-) > rename lib/librte_ring/{rte_ring_c11_mem.h =3D> ring_c11_pvt.h} (96%) > create mode 100644 lib/librte_ring/ring_elem_pvt.h > rename lib/librte_ring/{rte_ring_generic.h =3D> ring_generic_pvt.h} (98%= ) > rename lib/librte_ring/{rte_ring_hts_c11_mem.h =3D> ring_hts_elem_pvt.h}= (60%) > rename lib/librte_ring/{rte_ring_peek_c11_mem.h =3D> ring_peek_elem_pvt.= h} (62%) > rename lib/librte_ring/{rte_ring_rts_c11_mem.h =3D> ring_rts_elem_pvt.h}= (62%) >=20 Sorry, but I don't understand the purpose of that patch.=20 As I remember by DPDK naming convention all installable headers should have 'rte_' prefix. Same for public defines (RTE_). Why to abandon it here? > diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build > index 36fdcb6a5..98eac5810 100644 > --- a/lib/librte_ring/meson.build > +++ b/lib/librte_ring/meson.build > @@ -2,15 +2,16 @@ > # Copyright(c) 2017 Intel Corporation >=20 > sources =3D files('rte_ring.c') > -headers =3D files('rte_ring.h', > +headers =3D files('ring_c11_pvt.h', > + 'ring_elem_pvt.h', > + 'ring_generic_pvt.h', > + 'ring_hts_elem_pvt.h', > + 'ring_peek_elem_pvt.h', > + 'ring_rts_elem_pvt.h', > + 'rte_ring.h', > 'rte_ring_core.h', > 'rte_ring_elem.h', > - 'rte_ring_c11_mem.h', > - 'rte_ring_generic.h', > 'rte_ring_hts.h', > - 'rte_ring_hts_c11_mem.h', > 'rte_ring_peek.h', > - 'rte_ring_peek_c11_mem.h', > 'rte_ring_peek_zc.h', > - 'rte_ring_rts.h', > - 'rte_ring_rts_c11_mem.h') > + 'rte_ring_rts.h') > diff --git a/lib/librte_ring/rte_ring_c11_mem.h b/lib/librte_ring/ring_c1= 1_pvt.h > similarity index 96% > rename from lib/librte_ring/rte_ring_c11_mem.h > rename to lib/librte_ring/ring_c11_pvt.h > index 7f5eba262..9f2f5318f 100644 > --- a/lib/librte_ring/rte_ring_c11_mem.h > +++ b/lib/librte_ring/ring_c11_pvt.h > @@ -7,8 +7,8 @@ > * Used as BSD-3 Licensed with permission from Kip Macy. > */ >=20 > -#ifndef _RTE_RING_C11_MEM_H_ > -#define _RTE_RING_C11_MEM_H_ > +#ifndef _RING_C11_PVT_H_ > +#define _RING_C11_PVT_H_ >=20 > static __rte_always_inline void > __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val, > @@ -69,9 +69,6 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned = int is_sp, > /* Ensure the head is read before tail */ > __atomic_thread_fence(__ATOMIC_ACQUIRE); >=20 > - /* load-acquire synchronize with store-release of ht->tail > - * in update_tail. > - */ > cons_tail =3D __atomic_load_n(&r->cons.tail, > __ATOMIC_ACQUIRE); >=20 > @@ -178,4 +175,4 @@ __rte_ring_move_cons_head(struct rte_ring *r, int is_= sc, > return n; > } >=20 > -#endif /* _RTE_RING_C11_MEM_H_ */ > +#endif /* _RING_C11_PVT_H_ */ > diff --git a/lib/librte_ring/ring_elem_pvt.h b/lib/librte_ring/ring_elem_= pvt.h > new file mode 100644 > index 000000000..8003e5edc > --- /dev/null > +++ b/lib/librte_ring/ring_elem_pvt.h > @@ -0,0 +1,385 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * > + * Copyright (c) 2017,2018 HXT-semitech Corporation. > + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org > + * All rights reserved. > + * Derived from FreeBSD's bufring.h > + * Used as BSD-3 Licensed with permission from Kip Macy. > + */ > + > +#ifndef _RING_ELEM_PVT_H_ > +#define _RING_ELEM_PVT_H_ > + > +static __rte_always_inline void > +__rte_ring_enqueue_elems_32(struct rte_ring *r, const uint32_t size, > + uint32_t idx, const void *obj_table, uint32_t n) > +{ > + unsigned int i; > + uint32_t *ring =3D (uint32_t *)&r[1]; > + const uint32_t *obj =3D (const uint32_t *)obj_table; > + if (likely(idx + n < size)) { > + for (i =3D 0; i < (n & ~0x7); i +=3D 8, idx +=3D 8) { > + ring[idx] =3D obj[i]; > + ring[idx + 1] =3D obj[i + 1]; > + ring[idx + 2] =3D obj[i + 2]; > + ring[idx + 3] =3D obj[i + 3]; > + ring[idx + 4] =3D obj[i + 4]; > + ring[idx + 5] =3D obj[i + 5]; > + ring[idx + 6] =3D obj[i + 6]; > + ring[idx + 7] =3D obj[i + 7]; > + } > + switch (n & 0x7) { > + case 7: > + ring[idx++] =3D obj[i++]; /* fallthrough */ > + case 6: > + ring[idx++] =3D obj[i++]; /* fallthrough */ > + case 5: > + ring[idx++] =3D obj[i++]; /* fallthrough */ > + case 4: > + ring[idx++] =3D obj[i++]; /* fallthrough */ > + case 3: > + ring[idx++] =3D obj[i++]; /* fallthrough */ > + case 2: > + ring[idx++] =3D obj[i++]; /* fallthrough */ > + case 1: > + ring[idx++] =3D obj[i++]; /* fallthrough */ > + } > + } else { > + for (i =3D 0; idx < size; i++, idx++) > + ring[idx] =3D obj[i]; > + /* Start at the beginning */ > + for (idx =3D 0; i < n; i++, idx++) > + ring[idx] =3D obj[i]; > + } > +} > + > +static __rte_always_inline void > +__rte_ring_enqueue_elems_64(struct rte_ring *r, uint32_t prod_head, > + const void *obj_table, uint32_t n) > +{ > + unsigned int i; > + const uint32_t size =3D r->size; > + uint32_t idx =3D prod_head & r->mask; > + uint64_t *ring =3D (uint64_t *)&r[1]; > + const unaligned_uint64_t *obj =3D (const unaligned_uint64_t *)obj_table= ; > + if (likely(idx + n < size)) { > + for (i =3D 0; i < (n & ~0x3); i +=3D 4, idx +=3D 4) { > + ring[idx] =3D obj[i]; > + ring[idx + 1] =3D obj[i + 1]; > + ring[idx + 2] =3D obj[i + 2]; > + ring[idx + 3] =3D obj[i + 3]; > + } > + switch (n & 0x3) { > + case 3: > + ring[idx++] =3D obj[i++]; /* fallthrough */ > + case 2: > + ring[idx++] =3D obj[i++]; /* fallthrough */ > + case 1: > + ring[idx++] =3D obj[i++]; > + } > + } else { > + for (i =3D 0; idx < size; i++, idx++) > + ring[idx] =3D obj[i]; > + /* Start at the beginning */ > + for (idx =3D 0; i < n; i++, idx++) > + ring[idx] =3D obj[i]; > + } > +} > + > +static __rte_always_inline void > +__rte_ring_enqueue_elems_128(struct rte_ring *r, uint32_t prod_head, > + const void *obj_table, uint32_t n) > +{ > + unsigned int i; > + const uint32_t size =3D r->size; > + uint32_t idx =3D prod_head & r->mask; > + rte_int128_t *ring =3D (rte_int128_t *)&r[1]; > + const rte_int128_t *obj =3D (const rte_int128_t *)obj_table; > + if (likely(idx + n < size)) { > + for (i =3D 0; i < (n & ~0x1); i +=3D 2, idx +=3D 2) > + memcpy((void *)(ring + idx), > + (const void *)(obj + i), 32); > + switch (n & 0x1) { > + case 1: > + memcpy((void *)(ring + idx), > + (const void *)(obj + i), 16); > + } > + } else { > + for (i =3D 0; idx < size; i++, idx++) > + memcpy((void *)(ring + idx), > + (const void *)(obj + i), 16); > + /* Start at the beginning */ > + for (idx =3D 0; i < n; i++, idx++) > + memcpy((void *)(ring + idx), > + (const void *)(obj + i), 16); > + } > +} > + > +/* the actual enqueue of elements on the ring. > + * Placed here since identical code needed in both > + * single and multi producer enqueue functions. > + */ > +static __rte_always_inline void > +__rte_ring_enqueue_elems(struct rte_ring *r, uint32_t prod_head, > + const void *obj_table, uint32_t esize, uint32_t num) > +{ > + /* 8B and 16B copies implemented individually to retain > + * the current performance. > + */ > + if (esize =3D=3D 8) > + __rte_ring_enqueue_elems_64(r, prod_head, obj_table, num); > + else if (esize =3D=3D 16) > + __rte_ring_enqueue_elems_128(r, prod_head, obj_table, num); > + else { > + uint32_t idx, scale, nr_idx, nr_num, nr_size; > + > + /* Normalize to uint32_t */ > + scale =3D esize / sizeof(uint32_t); > + nr_num =3D num * scale; > + idx =3D prod_head & r->mask; > + nr_idx =3D idx * scale; > + nr_size =3D r->size * scale; > + __rte_ring_enqueue_elems_32(r, nr_size, nr_idx, > + obj_table, nr_num); > + } > +} > + > +static __rte_always_inline void > +__rte_ring_dequeue_elems_32(struct rte_ring *r, const uint32_t size, > + uint32_t idx, void *obj_table, uint32_t n) > +{ > + unsigned int i; > + uint32_t *ring =3D (uint32_t *)&r[1]; > + uint32_t *obj =3D (uint32_t *)obj_table; > + if (likely(idx + n < size)) { > + for (i =3D 0; i < (n & ~0x7); i +=3D 8, idx +=3D 8) { > + obj[i] =3D ring[idx]; > + obj[i + 1] =3D ring[idx + 1]; > + obj[i + 2] =3D ring[idx + 2]; > + obj[i + 3] =3D ring[idx + 3]; > + obj[i + 4] =3D ring[idx + 4]; > + obj[i + 5] =3D ring[idx + 5]; > + obj[i + 6] =3D ring[idx + 6]; > + obj[i + 7] =3D ring[idx + 7]; > + } > + switch (n & 0x7) { > + case 7: > + obj[i++] =3D ring[idx++]; /* fallthrough */ > + case 6: > + obj[i++] =3D ring[idx++]; /* fallthrough */ > + case 5: > + obj[i++] =3D ring[idx++]; /* fallthrough */ > + case 4: > + obj[i++] =3D ring[idx++]; /* fallthrough */ > + case 3: > + obj[i++] =3D ring[idx++]; /* fallthrough */ > + case 2: > + obj[i++] =3D ring[idx++]; /* fallthrough */ > + case 1: > + obj[i++] =3D ring[idx++]; /* fallthrough */ > + } > + } else { > + for (i =3D 0; idx < size; i++, idx++) > + obj[i] =3D ring[idx]; > + /* Start at the beginning */ > + for (idx =3D 0; i < n; i++, idx++) > + obj[i] =3D ring[idx]; > + } > +} > + > +static __rte_always_inline void > +__rte_ring_dequeue_elems_64(struct rte_ring *r, uint32_t prod_head, > + void *obj_table, uint32_t n) > +{ > + unsigned int i; > + const uint32_t size =3D r->size; > + uint32_t idx =3D prod_head & r->mask; > + uint64_t *ring =3D (uint64_t *)&r[1]; > + unaligned_uint64_t *obj =3D (unaligned_uint64_t *)obj_table; > + if (likely(idx + n < size)) { > + for (i =3D 0; i < (n & ~0x3); i +=3D 4, idx +=3D 4) { > + obj[i] =3D ring[idx]; > + obj[i + 1] =3D ring[idx + 1]; > + obj[i + 2] =3D ring[idx + 2]; > + obj[i + 3] =3D ring[idx + 3]; > + } > + switch (n & 0x3) { > + case 3: > + obj[i++] =3D ring[idx++]; /* fallthrough */ > + case 2: > + obj[i++] =3D ring[idx++]; /* fallthrough */ > + case 1: > + obj[i++] =3D ring[idx++]; /* fallthrough */ > + } > + } else { > + for (i =3D 0; idx < size; i++, idx++) > + obj[i] =3D ring[idx]; > + /* Start at the beginning */ > + for (idx =3D 0; i < n; i++, idx++) > + obj[i] =3D ring[idx]; > + } > +} > + > +static __rte_always_inline void > +__rte_ring_dequeue_elems_128(struct rte_ring *r, uint32_t prod_head, > + void *obj_table, uint32_t n) > +{ > + unsigned int i; > + const uint32_t size =3D r->size; > + uint32_t idx =3D prod_head & r->mask; > + rte_int128_t *ring =3D (rte_int128_t *)&r[1]; > + rte_int128_t *obj =3D (rte_int128_t *)obj_table; > + if (likely(idx + n < size)) { > + for (i =3D 0; i < (n & ~0x1); i +=3D 2, idx +=3D 2) > + memcpy((void *)(obj + i), (void *)(ring + idx), 32); > + switch (n & 0x1) { > + case 1: > + memcpy((void *)(obj + i), (void *)(ring + idx), 16); > + } > + } else { > + for (i =3D 0; idx < size; i++, idx++) > + memcpy((void *)(obj + i), (void *)(ring + idx), 16); > + /* Start at the beginning */ > + for (idx =3D 0; i < n; i++, idx++) > + memcpy((void *)(obj + i), (void *)(ring + idx), 16); > + } > +} > + > +/* the actual dequeue of elements from the ring. > + * Placed here since identical code needed in both > + * single and multi producer enqueue functions. > + */ > +static __rte_always_inline void > +__rte_ring_dequeue_elems(struct rte_ring *r, uint32_t cons_head, > + void *obj_table, uint32_t esize, uint32_t num) > +{ > + /* 8B and 16B copies implemented individually to retain > + * the current performance. > + */ > + if (esize =3D=3D 8) > + __rte_ring_dequeue_elems_64(r, cons_head, obj_table, num); > + else if (esize =3D=3D 16) > + __rte_ring_dequeue_elems_128(r, cons_head, obj_table, num); > + else { > + uint32_t idx, scale, nr_idx, nr_num, nr_size; > + > + /* Normalize to uint32_t */ > + scale =3D esize / sizeof(uint32_t); > + nr_num =3D num * scale; > + idx =3D cons_head & r->mask; > + nr_idx =3D idx * scale; > + nr_size =3D r->size * scale; > + __rte_ring_dequeue_elems_32(r, nr_size, nr_idx, > + obj_table, nr_num); > + } > +} > + > +/* Between load and load. there might be cpu reorder in weak model > + * (powerpc/arm). > + * There are 2 choices for the users > + * 1.use rmb() memory barrier > + * 2.use one-direction load_acquire/store_release barrier > + * It depends on performance test results. > + */ > +#ifdef RTE_USE_C11_MEM_MODEL > +#include "ring_c11_pvt.h" > +#else > +#include "ring_generic_pvt.h" > +#endif > + > +/** > + * @internal Enqueue several objects on the ring > + * > + * @param r > + * A pointer to the ring structure. > + * @param obj_table > + * A pointer to a table of objects. > + * @param esize > + * The size of ring element, in bytes. It must be a multiple of 4. > + * This must be the same value used while creating the ring. Otherwise > + * the results are undefined. > + * @param n > + * The number of objects to add in the ring from the obj_table. > + * @param behavior > + * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a rin= g > + * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from rin= g > + * @param is_sp > + * Indicates whether to use single producer or multi-producer head upd= ate > + * @param free_space > + * returns the amount of space after the enqueue operation has finishe= d > + * @return > + * Actual number of objects enqueued. > + * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n only. > + */ > +static __rte_always_inline unsigned int > +__rte_ring_do_enqueue_elem(struct rte_ring *r, const void *obj_table, > + unsigned int esize, unsigned int n, > + enum rte_ring_queue_behavior behavior, unsigned int is_sp, > + unsigned int *free_space) > +{ > + uint32_t prod_head, prod_next; > + uint32_t free_entries; > + > + n =3D __rte_ring_move_prod_head(r, is_sp, n, behavior, > + &prod_head, &prod_next, &free_entries); > + if (n =3D=3D 0) > + goto end; > + > + __rte_ring_enqueue_elems(r, prod_head, obj_table, esize, n); > + > + __rte_ring_update_tail(&r->prod, prod_head, prod_next, is_sp, 1); > +end: > + if (free_space !=3D NULL) > + *free_space =3D free_entries - n; > + return n; > +} > + > +/** > + * @internal Dequeue several objects from the ring > + * > + * @param r > + * A pointer to the ring structure. > + * @param obj_table > + * A pointer to a table of objects. > + * @param esize > + * The size of ring element, in bytes. It must be a multiple of 4. > + * This must be the same value used while creating the ring. Otherwise > + * the results are undefined. > + * @param n > + * The number of objects to pull from the ring. > + * @param behavior > + * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a rin= g > + * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from rin= g > + * @param is_sc > + * Indicates whether to use single consumer or multi-consumer head upd= ate > + * @param available > + * returns the number of remaining ring entries after the dequeue has = finished > + * @return > + * - Actual number of objects dequeued. > + * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n only= . > + */ > +static __rte_always_inline unsigned int > +__rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table, > + unsigned int esize, unsigned int n, > + enum rte_ring_queue_behavior behavior, unsigned int is_sc, > + unsigned int *available) > +{ > + uint32_t cons_head, cons_next; > + uint32_t entries; > + > + n =3D __rte_ring_move_cons_head(r, (int)is_sc, n, behavior, > + &cons_head, &cons_next, &entries); > + if (n =3D=3D 0) > + goto end; > + > + __rte_ring_dequeue_elems(r, cons_head, obj_table, esize, n); > + > + __rte_ring_update_tail(&r->cons, cons_head, cons_next, is_sc, 0); > + > +end: > + if (available !=3D NULL) > + *available =3D entries - n; > + return n; > +} > + > +#endif /* _RING_ELEM_PVT_H_ */ > diff --git a/lib/librte_ring/rte_ring_generic.h b/lib/librte_ring/ring_ge= neric_pvt.h > similarity index 98% > rename from lib/librte_ring/rte_ring_generic.h > rename to lib/librte_ring/ring_generic_pvt.h > index 37c62b8d6..fc46a27b2 100644 > --- a/lib/librte_ring/rte_ring_generic.h > +++ b/lib/librte_ring/ring_generic_pvt.h > @@ -7,8 +7,8 @@ > * Used as BSD-3 Licensed with permission from Kip Macy. > */ >=20 > -#ifndef _RTE_RING_GENERIC_H_ > -#define _RTE_RING_GENERIC_H_ > +#ifndef _RING_GENERIC_PVT_H_ > +#define _RING_GENERIC_PVT_H_ >=20 > static __rte_always_inline void > __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val, > @@ -170,4 +170,4 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigne= d int is_sc, > return n; > } >=20 > -#endif /* _RTE_RING_GENERIC_H_ */ > +#endif /* _RING_GENERIC_PVT_H_ */ > diff --git a/lib/librte_ring/rte_ring_hts_c11_mem.h b/lib/librte_ring/rin= g_hts_elem_pvt.h > similarity index 60% > rename from lib/librte_ring/rte_ring_hts_c11_mem.h > rename to lib/librte_ring/ring_hts_elem_pvt.h > index 16e54b6ff..9268750b0 100644 > --- a/lib/librte_ring/rte_ring_hts_c11_mem.h > +++ b/lib/librte_ring/ring_hts_elem_pvt.h > @@ -7,8 +7,8 @@ > * Used as BSD-3 Licensed with permission from Kip Macy. > */ >=20 > -#ifndef _RTE_RING_HTS_C11_MEM_H_ > -#define _RTE_RING_HTS_C11_MEM_H_ > +#ifndef _RING_HTS_ELEM_PVT_H_ > +#define _RING_HTS_ELEM_PVT_H_ >=20 > /** > * @file rte_ring_hts_c11_mem.h > @@ -161,4 +161,86 @@ __rte_ring_hts_move_cons_head(struct rte_ring *r, un= signed int num, > return n; > } >=20 > -#endif /* _RTE_RING_HTS_C11_MEM_H_ */ > +/** > + * @internal Enqueue several objects on the HTS ring. > + * > + * @param r > + * A pointer to the ring structure. > + * @param obj_table > + * A pointer to a table of objects. > + * @param esize > + * The size of ring element, in bytes. It must be a multiple of 4. > + * This must be the same value used while creating the ring. Otherwise > + * the results are undefined. > + * @param n > + * The number of objects to add in the ring from the obj_table. > + * @param behavior > + * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a rin= g > + * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from rin= g > + * @param free_space > + * returns the amount of space after the enqueue operation has finishe= d > + * @return > + * Actual number of objects enqueued. > + * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n only. > + */ > +static __rte_always_inline unsigned int > +__rte_ring_do_hts_enqueue_elem(struct rte_ring *r, const void *obj_table= , > + uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior, > + uint32_t *free_space) > +{ > + uint32_t free, head; > + > + n =3D __rte_ring_hts_move_prod_head(r, n, behavior, &head, &free); > + > + if (n !=3D 0) { > + __rte_ring_enqueue_elems(r, head, obj_table, esize, n); > + __rte_ring_hts_update_tail(&r->hts_prod, head, n, 1); > + } > + > + if (free_space !=3D NULL) > + *free_space =3D free - n; > + return n; > +} > + > +/** > + * @internal Dequeue several objects from the HTS ring. > + * > + * @param r > + * A pointer to the ring structure. > + * @param obj_table > + * A pointer to a table of objects. > + * @param esize > + * The size of ring element, in bytes. It must be a multiple of 4. > + * This must be the same value used while creating the ring. Otherwise > + * the results are undefined. > + * @param n > + * The number of objects to pull from the ring. > + * @param behavior > + * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a rin= g > + * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from rin= g > + * @param available > + * returns the number of remaining ring entries after the dequeue has = finished > + * @return > + * - Actual number of objects dequeued. > + * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n only= . > + */ > +static __rte_always_inline unsigned int > +__rte_ring_do_hts_dequeue_elem(struct rte_ring *r, void *obj_table, > + uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior, > + uint32_t *available) > +{ > + uint32_t entries, head; > + > + n =3D __rte_ring_hts_move_cons_head(r, n, behavior, &head, &entries); > + > + if (n !=3D 0) { > + __rte_ring_dequeue_elems(r, head, obj_table, esize, n); > + __rte_ring_hts_update_tail(&r->hts_cons, head, n, 0); > + } > + > + if (available !=3D NULL) > + *available =3D entries - n; > + return n; > +} > + > +#endif /* _RING_HTS_ELEM_PVT_H_ */ > diff --git a/lib/librte_ring/rte_ring_peek_c11_mem.h b/lib/librte_ring/ri= ng_peek_elem_pvt.h > similarity index 62% > rename from lib/librte_ring/rte_ring_peek_c11_mem.h > rename to lib/librte_ring/ring_peek_elem_pvt.h > index 283c7e70b..1c57bcdd6 100644 > --- a/lib/librte_ring/rte_ring_peek_c11_mem.h > +++ b/lib/librte_ring/ring_peek_elem_pvt.h > @@ -7,8 +7,8 @@ > * Used as BSD-3 Licensed with permission from Kip Macy. > */ >=20 > -#ifndef _RTE_RING_PEEK_C11_MEM_H_ > -#define _RTE_RING_PEEK_C11_MEM_H_ > +#ifndef _RING_PEEK_ELEM_PVT_H_ > +#define _RING_PEEK_ELEM_PVT_H_ >=20 > /** > * @file rte_ring_peek_c11_mem.h > @@ -107,4 +107,73 @@ __rte_ring_hts_set_head_tail(struct rte_ring_hts_hea= dtail *ht, uint32_t tail, > __atomic_store_n(&ht->ht.raw, p.raw, __ATOMIC_RELEASE); > } >=20 > -#endif /* _RTE_RING_PEEK_C11_MEM_H_ */ > +/** > + * @internal This function moves prod head value. > + */ > +static __rte_always_inline unsigned int > +__rte_ring_do_enqueue_start(struct rte_ring *r, uint32_t n, > + enum rte_ring_queue_behavior behavior, uint32_t *free_space) > +{ > + uint32_t free, head, next; > + > + switch (r->prod.sync_type) { > + case RTE_RING_SYNC_ST: > + n =3D __rte_ring_move_prod_head(r, RTE_RING_SYNC_ST, n, > + behavior, &head, &next, &free); > + break; > + case RTE_RING_SYNC_MT_HTS: > + n =3D __rte_ring_hts_move_prod_head(r, n, behavior, > + &head, &free); > + break; > + case RTE_RING_SYNC_MT: > + case RTE_RING_SYNC_MT_RTS: > + default: > + /* unsupported mode, shouldn't be here */ > + RTE_ASSERT(0); > + n =3D 0; > + free =3D 0; > + } > + > + if (free_space !=3D NULL) > + *free_space =3D free - n; > + return n; > +} > + > +/** > + * @internal This function moves cons head value and copies up to *n* > + * objects from the ring to the user provided obj_table. > + */ > +static __rte_always_inline unsigned int > +__rte_ring_do_dequeue_start(struct rte_ring *r, void *obj_table, > + uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior, > + uint32_t *available) > +{ > + uint32_t avail, head, next; > + > + switch (r->cons.sync_type) { > + case RTE_RING_SYNC_ST: > + n =3D __rte_ring_move_cons_head(r, RTE_RING_SYNC_ST, n, > + behavior, &head, &next, &avail); > + break; > + case RTE_RING_SYNC_MT_HTS: > + n =3D __rte_ring_hts_move_cons_head(r, n, behavior, > + &head, &avail); > + break; > + case RTE_RING_SYNC_MT: > + case RTE_RING_SYNC_MT_RTS: > + default: > + /* unsupported mode, shouldn't be here */ > + RTE_ASSERT(0); > + n =3D 0; > + avail =3D 0; > + } > + > + if (n !=3D 0) > + __rte_ring_dequeue_elems(r, head, obj_table, esize, n); > + > + if (available !=3D NULL) > + *available =3D avail - n; > + return n; > +} > + > +#endif /* _RING_PEEK_ELEM_PVT_H_ */ > diff --git a/lib/librte_ring/rte_ring_rts_c11_mem.h b/lib/librte_ring/rin= g_rts_elem_pvt.h > similarity index 62% > rename from lib/librte_ring/rte_ring_rts_c11_mem.h > rename to lib/librte_ring/ring_rts_elem_pvt.h > index 327f22796..cbcec73eb 100644 > --- a/lib/librte_ring/rte_ring_rts_c11_mem.h > +++ b/lib/librte_ring/ring_rts_elem_pvt.h > @@ -7,8 +7,8 @@ > * Used as BSD-3 Licensed with permission from Kip Macy. > */ >=20 > -#ifndef _RTE_RING_RTS_C11_MEM_H_ > -#define _RTE_RING_RTS_C11_MEM_H_ > +#ifndef _RING_RTS_ELEM_PVT_H_ > +#define _RING_RTS_ELEM_PVT_H_ >=20 > /** > * @file rte_ring_rts_c11_mem.h > @@ -176,4 +176,86 @@ __rte_ring_rts_move_cons_head(struct rte_ring *r, ui= nt32_t num, > return n; > } >=20 > -#endif /* _RTE_RING_RTS_C11_MEM_H_ */ > +/** > + * @internal Enqueue several objects on the RTS ring. > + * > + * @param r > + * A pointer to the ring structure. > + * @param obj_table > + * A pointer to a table of objects. > + * @param esize > + * The size of ring element, in bytes. It must be a multiple of 4. > + * This must be the same value used while creating the ring. Otherwise > + * the results are undefined. > + * @param n > + * The number of objects to add in the ring from the obj_table. > + * @param behavior > + * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a rin= g > + * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from rin= g > + * @param free_space > + * returns the amount of space after the enqueue operation has finishe= d > + * @return > + * Actual number of objects enqueued. > + * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n only. > + */ > +static __rte_always_inline unsigned int > +__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, const void *obj_table= , > + uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior, > + uint32_t *free_space) > +{ > + uint32_t free, head; > + > + n =3D __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free); > + > + if (n !=3D 0) { > + __rte_ring_enqueue_elems(r, head, obj_table, esize, n); > + __rte_ring_rts_update_tail(&r->rts_prod); > + } > + > + if (free_space !=3D NULL) > + *free_space =3D free - n; > + return n; > +} > + > +/** > + * @internal Dequeue several objects from the RTS ring. > + * > + * @param r > + * A pointer to the ring structure. > + * @param obj_table > + * A pointer to a table of objects. > + * @param esize > + * The size of ring element, in bytes. It must be a multiple of 4. > + * This must be the same value used while creating the ring. Otherwise > + * the results are undefined. > + * @param n > + * The number of objects to pull from the ring. > + * @param behavior > + * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a rin= g > + * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from rin= g > + * @param available > + * returns the number of remaining ring entries after the dequeue has = finished > + * @return > + * - Actual number of objects dequeued. > + * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n only= . > + */ > +static __rte_always_inline unsigned int > +__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void *obj_table, > + uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior, > + uint32_t *available) > +{ > + uint32_t entries, head; > + > + n =3D __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries); > + > + if (n !=3D 0) { > + __rte_ring_dequeue_elems(r, head, obj_table, esize, n); > + __rte_ring_rts_update_tail(&r->rts_cons); > + } > + > + if (available !=3D NULL) > + *available =3D entries - n; > + return n; > +} > + > +#endif /* _RING_RTS_ELEM_PVT_H_ */ > diff --git a/lib/librte_ring/rte_ring_elem.h b/lib/librte_ring/rte_ring_e= lem.h > index 57344d47d..119b5c0b6 100644 > --- a/lib/librte_ring/rte_ring_elem.h > +++ b/lib/librte_ring/rte_ring_elem.h > @@ -21,6 +21,7 @@ extern "C" { > #endif >=20 > #include > +#include >=20 > /** > * Calculate the memory size needed for a ring with given element size > @@ -105,379 +106,6 @@ ssize_t rte_ring_get_memsize_elem(unsigned int esiz= e, unsigned int count); > struct rte_ring *rte_ring_create_elem(const char *name, unsigned int esi= ze, > unsigned int count, int socket_id, unsigned int flags); >=20 > -static __rte_always_inline void > -__rte_ring_enqueue_elems_32(struct rte_ring *r, const uint32_t size, > - uint32_t idx, const void *obj_table, uint32_t n) > -{ > - unsigned int i; > - uint32_t *ring =3D (uint32_t *)&r[1]; > - const uint32_t *obj =3D (const uint32_t *)obj_table; > - if (likely(idx + n < size)) { > - for (i =3D 0; i < (n & ~0x7); i +=3D 8, idx +=3D 8) { > - ring[idx] =3D obj[i]; > - ring[idx + 1] =3D obj[i + 1]; > - ring[idx + 2] =3D obj[i + 2]; > - ring[idx + 3] =3D obj[i + 3]; > - ring[idx + 4] =3D obj[i + 4]; > - ring[idx + 5] =3D obj[i + 5]; > - ring[idx + 6] =3D obj[i + 6]; > - ring[idx + 7] =3D obj[i + 7]; > - } > - switch (n & 0x7) { > - case 7: > - ring[idx++] =3D obj[i++]; /* fallthrough */ > - case 6: > - ring[idx++] =3D obj[i++]; /* fallthrough */ > - case 5: > - ring[idx++] =3D obj[i++]; /* fallthrough */ > - case 4: > - ring[idx++] =3D obj[i++]; /* fallthrough */ > - case 3: > - ring[idx++] =3D obj[i++]; /* fallthrough */ > - case 2: > - ring[idx++] =3D obj[i++]; /* fallthrough */ > - case 1: > - ring[idx++] =3D obj[i++]; /* fallthrough */ > - } > - } else { > - for (i =3D 0; idx < size; i++, idx++) > - ring[idx] =3D obj[i]; > - /* Start at the beginning */ > - for (idx =3D 0; i < n; i++, idx++) > - ring[idx] =3D obj[i]; > - } > -} > - > -static __rte_always_inline void > -__rte_ring_enqueue_elems_64(struct rte_ring *r, uint32_t prod_head, > - const void *obj_table, uint32_t n) > -{ > - unsigned int i; > - const uint32_t size =3D r->size; > - uint32_t idx =3D prod_head & r->mask; > - uint64_t *ring =3D (uint64_t *)&r[1]; > - const unaligned_uint64_t *obj =3D (const unaligned_uint64_t *)obj_table= ; > - if (likely(idx + n < size)) { > - for (i =3D 0; i < (n & ~0x3); i +=3D 4, idx +=3D 4) { > - ring[idx] =3D obj[i]; > - ring[idx + 1] =3D obj[i + 1]; > - ring[idx + 2] =3D obj[i + 2]; > - ring[idx + 3] =3D obj[i + 3]; > - } > - switch (n & 0x3) { > - case 3: > - ring[idx++] =3D obj[i++]; /* fallthrough */ > - case 2: > - ring[idx++] =3D obj[i++]; /* fallthrough */ > - case 1: > - ring[idx++] =3D obj[i++]; > - } > - } else { > - for (i =3D 0; idx < size; i++, idx++) > - ring[idx] =3D obj[i]; > - /* Start at the beginning */ > - for (idx =3D 0; i < n; i++, idx++) > - ring[idx] =3D obj[i]; > - } > -} > - > -static __rte_always_inline void > -__rte_ring_enqueue_elems_128(struct rte_ring *r, uint32_t prod_head, > - const void *obj_table, uint32_t n) > -{ > - unsigned int i; > - const uint32_t size =3D r->size; > - uint32_t idx =3D prod_head & r->mask; > - rte_int128_t *ring =3D (rte_int128_t *)&r[1]; > - const rte_int128_t *obj =3D (const rte_int128_t *)obj_table; > - if (likely(idx + n < size)) { > - for (i =3D 0; i < (n & ~0x1); i +=3D 2, idx +=3D 2) > - memcpy((void *)(ring + idx), > - (const void *)(obj + i), 32); > - switch (n & 0x1) { > - case 1: > - memcpy((void *)(ring + idx), > - (const void *)(obj + i), 16); > - } > - } else { > - for (i =3D 0; idx < size; i++, idx++) > - memcpy((void *)(ring + idx), > - (const void *)(obj + i), 16); > - /* Start at the beginning */ > - for (idx =3D 0; i < n; i++, idx++) > - memcpy((void *)(ring + idx), > - (const void *)(obj + i), 16); > - } > -} > - > -/* the actual enqueue of elements on the ring. > - * Placed here since identical code needed in both > - * single and multi producer enqueue functions. > - */ > -static __rte_always_inline void > -__rte_ring_enqueue_elems(struct rte_ring *r, uint32_t prod_head, > - const void *obj_table, uint32_t esize, uint32_t num) > -{ > - /* 8B and 16B copies implemented individually to retain > - * the current performance. > - */ > - if (esize =3D=3D 8) > - __rte_ring_enqueue_elems_64(r, prod_head, obj_table, num); > - else if (esize =3D=3D 16) > - __rte_ring_enqueue_elems_128(r, prod_head, obj_table, num); > - else { > - uint32_t idx, scale, nr_idx, nr_num, nr_size; > - > - /* Normalize to uint32_t */ > - scale =3D esize / sizeof(uint32_t); > - nr_num =3D num * scale; > - idx =3D prod_head & r->mask; > - nr_idx =3D idx * scale; > - nr_size =3D r->size * scale; > - __rte_ring_enqueue_elems_32(r, nr_size, nr_idx, > - obj_table, nr_num); > - } > -} > - > -static __rte_always_inline void > -__rte_ring_dequeue_elems_32(struct rte_ring *r, const uint32_t size, > - uint32_t idx, void *obj_table, uint32_t n) > -{ > - unsigned int i; > - uint32_t *ring =3D (uint32_t *)&r[1]; > - uint32_t *obj =3D (uint32_t *)obj_table; > - if (likely(idx + n < size)) { > - for (i =3D 0; i < (n & ~0x7); i +=3D 8, idx +=3D 8) { > - obj[i] =3D ring[idx]; > - obj[i + 1] =3D ring[idx + 1]; > - obj[i + 2] =3D ring[idx + 2]; > - obj[i + 3] =3D ring[idx + 3]; > - obj[i + 4] =3D ring[idx + 4]; > - obj[i + 5] =3D ring[idx + 5]; > - obj[i + 6] =3D ring[idx + 6]; > - obj[i + 7] =3D ring[idx + 7]; > - } > - switch (n & 0x7) { > - case 7: > - obj[i++] =3D ring[idx++]; /* fallthrough */ > - case 6: > - obj[i++] =3D ring[idx++]; /* fallthrough */ > - case 5: > - obj[i++] =3D ring[idx++]; /* fallthrough */ > - case 4: > - obj[i++] =3D ring[idx++]; /* fallthrough */ > - case 3: > - obj[i++] =3D ring[idx++]; /* fallthrough */ > - case 2: > - obj[i++] =3D ring[idx++]; /* fallthrough */ > - case 1: > - obj[i++] =3D ring[idx++]; /* fallthrough */ > - } > - } else { > - for (i =3D 0; idx < size; i++, idx++) > - obj[i] =3D ring[idx]; > - /* Start at the beginning */ > - for (idx =3D 0; i < n; i++, idx++) > - obj[i] =3D ring[idx]; > - } > -} > - > -static __rte_always_inline void > -__rte_ring_dequeue_elems_64(struct rte_ring *r, uint32_t prod_head, > - void *obj_table, uint32_t n) > -{ > - unsigned int i; > - const uint32_t size =3D r->size; > - uint32_t idx =3D prod_head & r->mask; > - uint64_t *ring =3D (uint64_t *)&r[1]; > - unaligned_uint64_t *obj =3D (unaligned_uint64_t *)obj_table; > - if (likely(idx + n < size)) { > - for (i =3D 0; i < (n & ~0x3); i +=3D 4, idx +=3D 4) { > - obj[i] =3D ring[idx]; > - obj[i + 1] =3D ring[idx + 1]; > - obj[i + 2] =3D ring[idx + 2]; > - obj[i + 3] =3D ring[idx + 3]; > - } > - switch (n & 0x3) { > - case 3: > - obj[i++] =3D ring[idx++]; /* fallthrough */ > - case 2: > - obj[i++] =3D ring[idx++]; /* fallthrough */ > - case 1: > - obj[i++] =3D ring[idx++]; /* fallthrough */ > - } > - } else { > - for (i =3D 0; idx < size; i++, idx++) > - obj[i] =3D ring[idx]; > - /* Start at the beginning */ > - for (idx =3D 0; i < n; i++, idx++) > - obj[i] =3D ring[idx]; > - } > -} > - > -static __rte_always_inline void > -__rte_ring_dequeue_elems_128(struct rte_ring *r, uint32_t prod_head, > - void *obj_table, uint32_t n) > -{ > - unsigned int i; > - const uint32_t size =3D r->size; > - uint32_t idx =3D prod_head & r->mask; > - rte_int128_t *ring =3D (rte_int128_t *)&r[1]; > - rte_int128_t *obj =3D (rte_int128_t *)obj_table; > - if (likely(idx + n < size)) { > - for (i =3D 0; i < (n & ~0x1); i +=3D 2, idx +=3D 2) > - memcpy((void *)(obj + i), (void *)(ring + idx), 32); > - switch (n & 0x1) { > - case 1: > - memcpy((void *)(obj + i), (void *)(ring + idx), 16); > - } > - } else { > - for (i =3D 0; idx < size; i++, idx++) > - memcpy((void *)(obj + i), (void *)(ring + idx), 16); > - /* Start at the beginning */ > - for (idx =3D 0; i < n; i++, idx++) > - memcpy((void *)(obj + i), (void *)(ring + idx), 16); > - } > -} > - > -/* the actual dequeue of elements from the ring. > - * Placed here since identical code needed in both > - * single and multi producer enqueue functions. > - */ > -static __rte_always_inline void > -__rte_ring_dequeue_elems(struct rte_ring *r, uint32_t cons_head, > - void *obj_table, uint32_t esize, uint32_t num) > -{ > - /* 8B and 16B copies implemented individually to retain > - * the current performance. > - */ > - if (esize =3D=3D 8) > - __rte_ring_dequeue_elems_64(r, cons_head, obj_table, num); > - else if (esize =3D=3D 16) > - __rte_ring_dequeue_elems_128(r, cons_head, obj_table, num); > - else { > - uint32_t idx, scale, nr_idx, nr_num, nr_size; > - > - /* Normalize to uint32_t */ > - scale =3D esize / sizeof(uint32_t); > - nr_num =3D num * scale; > - idx =3D cons_head & r->mask; > - nr_idx =3D idx * scale; > - nr_size =3D r->size * scale; > - __rte_ring_dequeue_elems_32(r, nr_size, nr_idx, > - obj_table, nr_num); > - } > -} > - > -/* Between load and load. there might be cpu reorder in weak model > - * (powerpc/arm). > - * There are 2 choices for the users > - * 1.use rmb() memory barrier > - * 2.use one-direction load_acquire/store_release barrier > - * It depends on performance test results. > - * By default, move common functions to rte_ring_generic.h > - */ > -#ifdef RTE_USE_C11_MEM_MODEL > -#include "rte_ring_c11_mem.h" > -#else > -#include "rte_ring_generic.h" > -#endif > - > -/** > - * @internal Enqueue several objects on the ring > - * > - * @param r > - * A pointer to the ring structure. > - * @param obj_table > - * A pointer to a table of objects. > - * @param esize > - * The size of ring element, in bytes. It must be a multiple of 4. > - * This must be the same value used while creating the ring. Otherwise > - * the results are undefined. > - * @param n > - * The number of objects to add in the ring from the obj_table. > - * @param behavior > - * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a rin= g > - * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from rin= g > - * @param is_sp > - * Indicates whether to use single producer or multi-producer head upd= ate > - * @param free_space > - * returns the amount of space after the enqueue operation has finishe= d > - * @return > - * Actual number of objects enqueued. > - * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n only. > - */ > -static __rte_always_inline unsigned int > -__rte_ring_do_enqueue_elem(struct rte_ring *r, const void *obj_table, > - unsigned int esize, unsigned int n, > - enum rte_ring_queue_behavior behavior, unsigned int is_sp, > - unsigned int *free_space) > -{ > - uint32_t prod_head, prod_next; > - uint32_t free_entries; > - > - n =3D __rte_ring_move_prod_head(r, is_sp, n, behavior, > - &prod_head, &prod_next, &free_entries); > - if (n =3D=3D 0) > - goto end; > - > - __rte_ring_enqueue_elems(r, prod_head, obj_table, esize, n); > - > - __rte_ring_update_tail(&r->prod, prod_head, prod_next, is_sp, 1); > -end: > - if (free_space !=3D NULL) > - *free_space =3D free_entries - n; > - return n; > -} > - > -/** > - * @internal Dequeue several objects from the ring > - * > - * @param r > - * A pointer to the ring structure. > - * @param obj_table > - * A pointer to a table of objects. > - * @param esize > - * The size of ring element, in bytes. It must be a multiple of 4. > - * This must be the same value used while creating the ring. Otherwise > - * the results are undefined. > - * @param n > - * The number of objects to pull from the ring. > - * @param behavior > - * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a rin= g > - * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from rin= g > - * @param is_sc > - * Indicates whether to use single consumer or multi-consumer head upd= ate > - * @param available > - * returns the number of remaining ring entries after the dequeue has = finished > - * @return > - * - Actual number of objects dequeued. > - * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n only= . > - */ > -static __rte_always_inline unsigned int > -__rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table, > - unsigned int esize, unsigned int n, > - enum rte_ring_queue_behavior behavior, unsigned int is_sc, > - unsigned int *available) > -{ > - uint32_t cons_head, cons_next; > - uint32_t entries; > - > - n =3D __rte_ring_move_cons_head(r, (int)is_sc, n, behavior, > - &cons_head, &cons_next, &entries); > - if (n =3D=3D 0) > - goto end; > - > - __rte_ring_dequeue_elems(r, cons_head, obj_table, esize, n); > - > - __rte_ring_update_tail(&r->cons, cons_head, cons_next, is_sc, 0); > - > -end: > - if (available !=3D NULL) > - *available =3D entries - n; > - return n; > -} > - > /** > * Enqueue several objects on the ring (multi-producers safe). > * > diff --git a/lib/librte_ring/rte_ring_hts.h b/lib/librte_ring/rte_ring_ht= s.h > index 359b15771..bdbdafc9f 100644 > --- a/lib/librte_ring/rte_ring_hts.h > +++ b/lib/librte_ring/rte_ring_hts.h > @@ -29,89 +29,7 @@ > extern "C" { > #endif >=20 > -#include > - > -/** > - * @internal Enqueue several objects on the HTS ring. > - * > - * @param r > - * A pointer to the ring structure. > - * @param obj_table > - * A pointer to a table of objects. > - * @param esize > - * The size of ring element, in bytes. It must be a multiple of 4. > - * This must be the same value used while creating the ring. Otherwise > - * the results are undefined. > - * @param n > - * The number of objects to add in the ring from the obj_table. > - * @param behavior > - * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a rin= g > - * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from rin= g > - * @param free_space > - * returns the amount of space after the enqueue operation has finishe= d > - * @return > - * Actual number of objects enqueued. > - * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n only. > - */ > -static __rte_always_inline unsigned int > -__rte_ring_do_hts_enqueue_elem(struct rte_ring *r, const void *obj_table= , > - uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior, > - uint32_t *free_space) > -{ > - uint32_t free, head; > - > - n =3D __rte_ring_hts_move_prod_head(r, n, behavior, &head, &free); > - > - if (n !=3D 0) { > - __rte_ring_enqueue_elems(r, head, obj_table, esize, n); > - __rte_ring_hts_update_tail(&r->hts_prod, head, n, 1); > - } > - > - if (free_space !=3D NULL) > - *free_space =3D free - n; > - return n; > -} > - > -/** > - * @internal Dequeue several objects from the HTS ring. > - * > - * @param r > - * A pointer to the ring structure. > - * @param obj_table > - * A pointer to a table of objects. > - * @param esize > - * The size of ring element, in bytes. It must be a multiple of 4. > - * This must be the same value used while creating the ring. Otherwise > - * the results are undefined. > - * @param n > - * The number of objects to pull from the ring. > - * @param behavior > - * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a rin= g > - * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from rin= g > - * @param available > - * returns the number of remaining ring entries after the dequeue has = finished > - * @return > - * - Actual number of objects dequeued. > - * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n only= . > - */ > -static __rte_always_inline unsigned int > -__rte_ring_do_hts_dequeue_elem(struct rte_ring *r, void *obj_table, > - uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior, > - uint32_t *available) > -{ > - uint32_t entries, head; > - > - n =3D __rte_ring_hts_move_cons_head(r, n, behavior, &head, &entries); > - > - if (n !=3D 0) { > - __rte_ring_dequeue_elems(r, head, obj_table, esize, n); > - __rte_ring_hts_update_tail(&r->hts_cons, head, n, 0); > - } > - > - if (available !=3D NULL) > - *available =3D entries - n; > - return n; > -} > +#include >=20 > /** > * Enqueue several objects on the HTS ring (multi-producers safe). > diff --git a/lib/librte_ring/rte_ring_peek.h b/lib/librte_ring/rte_ring_p= eek.h > index 45f707dc7..0dd402be4 100644 > --- a/lib/librte_ring/rte_ring_peek.h > +++ b/lib/librte_ring/rte_ring_peek.h > @@ -48,39 +48,7 @@ > extern "C" { > #endif >=20 > -#include > - > -/** > - * @internal This function moves prod head value. > - */ > -static __rte_always_inline unsigned int > -__rte_ring_do_enqueue_start(struct rte_ring *r, uint32_t n, > - enum rte_ring_queue_behavior behavior, uint32_t *free_space) > -{ > - uint32_t free, head, next; > - > - switch (r->prod.sync_type) { > - case RTE_RING_SYNC_ST: > - n =3D __rte_ring_move_prod_head(r, RTE_RING_SYNC_ST, n, > - behavior, &head, &next, &free); > - break; > - case RTE_RING_SYNC_MT_HTS: > - n =3D __rte_ring_hts_move_prod_head(r, n, behavior, > - &head, &free); > - break; > - case RTE_RING_SYNC_MT: > - case RTE_RING_SYNC_MT_RTS: > - default: > - /* unsupported mode, shouldn't be here */ > - RTE_ASSERT(0); > - n =3D 0; > - free =3D 0; > - } > - > - if (free_space !=3D NULL) > - *free_space =3D free - n; > - return n; > -} > +#include >=20 > /** > * Start to enqueue several objects on the ring. > @@ -248,43 +216,6 @@ rte_ring_enqueue_finish(struct rte_ring *r, void * c= onst *obj_table, > rte_ring_enqueue_elem_finish(r, obj_table, sizeof(uintptr_t), n); > } >=20 > -/** > - * @internal This function moves cons head value and copies up to *n* > - * objects from the ring to the user provided obj_table. > - */ > -static __rte_always_inline unsigned int > -__rte_ring_do_dequeue_start(struct rte_ring *r, void *obj_table, > - uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior, > - uint32_t *available) > -{ > - uint32_t avail, head, next; > - > - switch (r->cons.sync_type) { > - case RTE_RING_SYNC_ST: > - n =3D __rte_ring_move_cons_head(r, RTE_RING_SYNC_ST, n, > - behavior, &head, &next, &avail); > - break; > - case RTE_RING_SYNC_MT_HTS: > - n =3D __rte_ring_hts_move_cons_head(r, n, behavior, > - &head, &avail); > - break; > - case RTE_RING_SYNC_MT: > - case RTE_RING_SYNC_MT_RTS: > - default: > - /* unsupported mode, shouldn't be here */ > - RTE_ASSERT(0); > - n =3D 0; > - avail =3D 0; > - } > - > - if (n !=3D 0) > - __rte_ring_dequeue_elems(r, head, obj_table, esize, n); > - > - if (available !=3D NULL) > - *available =3D avail - n; > - return n; > -} > - > /** > * Start to dequeue several objects from the ring. > * Note that user has to call appropriate dequeue_finish() > diff --git a/lib/librte_ring/rte_ring_peek_zc.h b/lib/librte_ring/rte_rin= g_peek_zc.h > index cb3bbd067..bc8252a18 100644 > --- a/lib/librte_ring/rte_ring_peek_zc.h > +++ b/lib/librte_ring/rte_ring_peek_zc.h > @@ -72,7 +72,7 @@ > extern "C" { > #endif >=20 > -#include > +#include >=20 > /** > * Ring zero-copy information structure. > diff --git a/lib/librte_ring/rte_ring_rts.h b/lib/librte_ring/rte_ring_rt= s.h > index afc12abe2..83d9903e2 100644 > --- a/lib/librte_ring/rte_ring_rts.h > +++ b/lib/librte_ring/rte_ring_rts.h > @@ -56,89 +56,7 @@ > extern "C" { > #endif >=20 > -#include > - > -/** > - * @internal Enqueue several objects on the RTS ring. > - * > - * @param r > - * A pointer to the ring structure. > - * @param obj_table > - * A pointer to a table of objects. > - * @param esize > - * The size of ring element, in bytes. It must be a multiple of 4. > - * This must be the same value used while creating the ring. Otherwise > - * the results are undefined. > - * @param n > - * The number of objects to add in the ring from the obj_table. > - * @param behavior > - * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a rin= g > - * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from rin= g > - * @param free_space > - * returns the amount of space after the enqueue operation has finishe= d > - * @return > - * Actual number of objects enqueued. > - * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n only. > - */ > -static __rte_always_inline unsigned int > -__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, const void *obj_table= , > - uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior, > - uint32_t *free_space) > -{ > - uint32_t free, head; > - > - n =3D __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free); > - > - if (n !=3D 0) { > - __rte_ring_enqueue_elems(r, head, obj_table, esize, n); > - __rte_ring_rts_update_tail(&r->rts_prod); > - } > - > - if (free_space !=3D NULL) > - *free_space =3D free - n; > - return n; > -} > - > -/** > - * @internal Dequeue several objects from the RTS ring. > - * > - * @param r > - * A pointer to the ring structure. > - * @param obj_table > - * A pointer to a table of objects. > - * @param esize > - * The size of ring element, in bytes. It must be a multiple of 4. > - * This must be the same value used while creating the ring. Otherwise > - * the results are undefined. > - * @param n > - * The number of objects to pull from the ring. > - * @param behavior > - * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a rin= g > - * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from rin= g > - * @param available > - * returns the number of remaining ring entries after the dequeue has = finished > - * @return > - * - Actual number of objects dequeued. > - * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n only= . > - */ > -static __rte_always_inline unsigned int > -__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void *obj_table, > - uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior, > - uint32_t *available) > -{ > - uint32_t entries, head; > - > - n =3D __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries); > - > - if (n !=3D 0) { > - __rte_ring_dequeue_elems(r, head, obj_table, esize, n); > - __rte_ring_rts_update_tail(&r->rts_cons); > - } > - > - if (available !=3D NULL) > - *available =3D entries - n; > - return n; > -} > +#include >=20 > /** > * Enqueue several objects on the RTS ring (multi-producers safe). > -- > 2.17.1