From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5024EA0577; Tue, 14 Apr 2020 06:29:03 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7AB111C06C; Tue, 14 Apr 2020 06:29:02 +0200 (CEST) Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2046.outbound.protection.outlook.com [40.107.21.46]) by dpdk.org (Postfix) with ESMTP id E50C61BF02 for ; Tue, 14 Apr 2020 06:29:00 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3/PgS4fZ2HvmeIiP26xMLDBtkc6wwPL4aBZbnWezpjc=; b=+J2Chw5xm8quGNGLgihmAz31etT0l8PetIp3IVZEmkOh5qB3vtuxBpWJ/Lro8hmhmS72cY3FxtXWymky3qatPz+x+R1iUqHs9iFNGeySLA3D2uAh/g/VoTydG7zxj7z8+6xM9C/Q74r3ZCEQfAcYIpQEx1oUTne/c28kDjxfPL4= Received: from DB6PR07CA0065.eurprd07.prod.outlook.com (2603:10a6:6:2a::27) by DB6PR0802MB2232.eurprd08.prod.outlook.com (2603:10a6:4:83::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2900.26; Tue, 14 Apr 2020 04:28:57 +0000 Received: from DB5EUR03FT004.eop-EUR03.prod.protection.outlook.com (2603:10a6:6:2a:cafe::f7) by DB6PR07CA0065.outlook.office365.com (2603:10a6:6:2a::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2921.12 via Frontend Transport; Tue, 14 Apr 2020 04:28:57 +0000 Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=bestguesspass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DB5EUR03FT004.mail.protection.outlook.com (10.152.20.128) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2900.15 via Frontend Transport; Tue, 14 Apr 2020 04:28:57 +0000 Received: ("Tessian outbound 55454527ea3b:v50"); Tue, 14 Apr 2020 04:28:57 +0000 X-CR-MTA-TID: 64aa7808 Received: from b10dab1be2a2.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 9D87E8D8-BAB1-4DB5-BA80-59CC10E527D3.1; Tue, 14 Apr 2020 04:28:52 +0000 Received: from EUR03-DB5-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id b10dab1be2a2.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Tue, 14 Apr 2020 04:28:52 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=e8TgLsLIji5pMEymCRtQwFAXWKTTA1nvnyDTDPwN9Q61gJVvIYgDS4Vow5cDI0Hsa/a63wmDlhwXbLtuQPpHxYSLQBs3y4NvdgHjKMQGGKQlmtd15BugJ+GdUqrGjYa9P9nV3PIO/K2Zf2IDH2eVq78P5GB2R05+9fLiN4SGCGx9C/QFpqawXFqUnlaREkRNg7UIHWmXbyv6fRK5y+diCPFXm9eMl6n7M/CQNzmvi3SsOFM2My28p93I3tKDxmkucQu9HBjWDqN9r0XyNQ8peZ79STvbQQvuKMtORGYki97gjA5N8/o5bAYnLMZaGq8LvckREreSKvZ8sMM3AWqWXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3/PgS4fZ2HvmeIiP26xMLDBtkc6wwPL4aBZbnWezpjc=; b=KQTZy+sUwKxXGdtZed3CQ3VxCIwtijJ69Rlm9TEzPf9uDAVj3X3aZW8t43//HSeubTjpTsNRAbQQKqZfHLssWGQQYh1T0AY/SCOP5NHL2fDANAMXGGte2jmqkcwqN5thPYowSSYqz+XKU3sKGzAWo1TdXc25hv6O0QNtY7lfiNcZgX3ySjis3037IPRa2m/6V0X33f2CdzqheeY7ruzuKI3obY6wTwJRnRJ8uBMQLaWFmLTojy+nig2yVHaM23C0vTTX/j7HIwJ1rTHbaQ1JyRQ6Fy578RGGer4o1DS7SBDhv9wmbiYRey/c+ubQeu8RA2mHrYNiFDZcd+5dWbcezw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3/PgS4fZ2HvmeIiP26xMLDBtkc6wwPL4aBZbnWezpjc=; b=+J2Chw5xm8quGNGLgihmAz31etT0l8PetIp3IVZEmkOh5qB3vtuxBpWJ/Lro8hmhmS72cY3FxtXWymky3qatPz+x+R1iUqHs9iFNGeySLA3D2uAh/g/VoTydG7zxj7z8+6xM9C/Q74r3ZCEQfAcYIpQEx1oUTne/c28kDjxfPL4= Received: from DBBPR08MB4646.eurprd08.prod.outlook.com (2603:10a6:10:f5::16) by DBBPR08MB4428.eurprd08.prod.outlook.com (2603:10a6:10:ce::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2900.24; Tue, 14 Apr 2020 04:28:51 +0000 Received: from DBBPR08MB4646.eurprd08.prod.outlook.com ([fe80::1870:afc4:b90f:609d]) by DBBPR08MB4646.eurprd08.prod.outlook.com ([fe80::1870:afc4:b90f:609d%5]) with mapi id 15.20.2900.028; Tue, 14 Apr 2020 04:28:50 +0000 From: Honnappa Nagarahalli To: Konstantin Ananyev , "dev@dpdk.org" CC: "david.marchand@redhat.com" , "jielong.zjl@antfin.com" , nd , Honnappa Nagarahalli , nd Thread-Topic: [PATCH v3 9/9] ring: add C11 memory model for new sync modes Thread-Index: AQHWCd9fcecsz8DEM0eTUdPrKGt//6h4DDzw Date: Tue, 14 Apr 2020 04:28:50 +0000 Message-ID: References: <20200402220959.29885-1-konstantin.ananyev@intel.com> <20200403174235.23308-1-konstantin.ananyev@intel.com> <20200403174235.23308-10-konstantin.ananyev@intel.com> In-Reply-To: <20200403174235.23308-10-konstantin.ananyev@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 39a4afb2-6485-4039-bc1c-42b3f01913cb.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; x-originating-ip: [70.113.25.165] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: e6cd77ca-6aa6-4980-6de5-08d7e02c590f x-ms-traffictypediagnostic: DBBPR08MB4428:|DBBPR08MB4428:|DB6PR0802MB2232: x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:9508;OLM:9508; x-forefront-prvs: 0373D94D15 X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBBPR08MB4646.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(10009020)(4636009)(396003)(366004)(376002)(346002)(136003)(39860400002)(2906002)(86362001)(52536014)(4326008)(7696005)(33656002)(8936002)(81156014)(76116006)(8676002)(6506007)(26005)(71200400001)(30864003)(55016002)(66946007)(64756008)(5660300002)(110136005)(186003)(66446008)(54906003)(9686003)(316002)(66556008)(478600001)(66476007); DIR:OUT; SFP:1101; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: EQ4K/MCh5k9mrL+uBG61VY+hY9vvMLrLYFYfPEBnZ9YDjhBpxgjKHeIyrWsvh2yKx0Peylmga+T+C1ymPzEEthV14zh9vb4b5ytfNmcQ65AbUef+5isxO3t9HRZA+zySmLs7ZrvoiwsKyDeIfK/betdAqOELvxaxISESejnHHxBmJMT05n2EFG2lEC3kyKIboPHlHE3GwzAysIOaCY3+AmTMFphwp57gpcQn0YvgMBBVxdxI7jV+NQ8CPiELeZ3JEfBTNz/jZwNlHKH1Uz6Hd42HodXShHj0oTSLOqBiytKo/Ks1EtJ+mrf9/ua3jUQapVSW9aMrxHjyNmd72pzTV4oIx4H7cCNdqaorWEpLC2oGzN8DqtcdsiJxO3NM23aZLzA0BKw3kHrlSIhzP6Kgvoa7j6DsIHky/SeM5gtSUhJrCi0rbUHULGyBcb1mZP6G x-ms-exchange-antispam-messagedata: vsdQyEyYfOiaIt0Fko873ESFNcchkfrxw487cDFhYdJLrzuLAuajgYAFfQqoQ90/qrGXTXlrh0wbf7geLhY7ham3H63INOE7Z1a6lLBJTMmVf7o6SbuLgxHBexAQ+/1HeR0hOsM0N9e6NmJtiryOSA== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR08MB4428 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB5EUR03FT004.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFTY:; SFS:(10009020)(4636009)(136003)(39860400002)(346002)(376002)(396003)(46966005)(82740400003)(81166007)(336012)(86362001)(356005)(30864003)(186003)(8676002)(81156014)(4326008)(5660300002)(8936002)(70586007)(52536014)(110136005)(33656002)(47076004)(70206006)(26005)(55016002)(2906002)(26826003)(316002)(478600001)(6506007)(9686003)(7696005)(54906003); DIR:OUT; SFP:1101; X-MS-Office365-Filtering-Correlation-Id-Prvs: 5e5197d8-93e0-4e88-bc32-08d7e02c5544 X-Forefront-PRVS: 0373D94D15 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: WiJLH6HENMFStuBEI8K5kJsifdCrrwlZBcQOcOtPLI4twtrnMZ7QiAoQfNB/6rDm6ZHn4UQ5NnoYqo9JXaC9JEJowvH8PoeDFt0JBgswseTQ5EO7YRjAfgyGM1fsKaJZe00yPOhrMS8Dakr8oYSy1sZfdg6kGiisF2KTZ0b3mDOmFuMngWZVaBZlqdwTw4KEQ9fI7elwK0hKpSLHh8LHdGp3glzfHvLALX6Dh6EPHg0t+77LDmYvxUglkoUzbAhMLvZlSo9+BkrF1MuYWtlwF+l1+UjN+nOEPQOhJWpAoETuI2OWGryGUpj7NLO5/DYGkLySjvGLOzivOQscxR+6HUxlE0mTVW/s4QsTOisMNTG22pikrwI3AWlogzS8MLbySVhO4HWOVgwPqdpOrIQJFEYbhpnSu+xmszvg3/iQau4F6NIoJ09XQeYjmZFzEWqBgtcEWEUY1qO4YPMU/Pqs0n+R4P/07VYm0f3oYkxeKHYTS1tYyJ9VQva9M0piitZoZHOblo1e26M5nF3r6avQsg== X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Apr 2020 04:28:57.3520 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e6cd77ca-6aa6-4980-6de5-08d7e02c590f X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR0802MB2232 Subject: Re: [dpdk-dev] [PATCH v3 9/9] ring: add C11 memory model for new sync modes X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Konstantin, It would be good to blend this commit with the other commits. Few comments= inline. > Subject: [PATCH v3 9/9] ring: add C11 memory model for new sync modes >=20 > Add C11 atomics based implementation for RTS and HTS head/tail update > primitivies. >=20 > Signed-off-by: Konstantin Ananyev > --- > lib/librte_ring/Makefile | 4 +- > lib/librte_ring/meson.build | 2 + > lib/librte_ring/rte_ring_hts.h | 4 + > lib/librte_ring/rte_ring_hts_c11_mem.h | 222 +++++++++++++++++++++++++ > lib/librte_ring/rte_ring_hts_elem.h | 4 + > lib/librte_ring/rte_ring_rts.h | 4 + > lib/librte_ring/rte_ring_rts_c11_mem.h | 198 ++++++++++++++++++++++ > lib/librte_ring/rte_ring_rts_elem.h | 4 + > 8 files changed, 441 insertions(+), 1 deletion(-) create mode 100644 > lib/librte_ring/rte_ring_hts_c11_mem.h > create mode 100644 lib/librte_ring/rte_ring_rts_c11_mem.h >=20 > diff --git a/lib/librte_ring/Makefile b/lib/librte_ring/Makefile index > 5f8662737..927d105bf 100644 > --- a/lib/librte_ring/Makefile > +++ b/lib/librte_ring/Makefile > @@ -22,9 +22,11 @@ SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include :=3D > rte_ring.h \ > rte_ring_hts.h \ > rte_ring_hts_elem.h \ > rte_ring_hts_generic.h \ > + rte_ring_hts_c11_mem.h \ > rte_ring_peek.h \ > rte_ring_rts.h \ > rte_ring_rts_elem.h \ > - rte_ring_rts_generic.h > + rte_ring_rts_generic.h \ > + rte_ring_rts_c11_mem.h >=20 > include $(RTE_SDK)/mk/rte.lib.mk > diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build in= dex > f5f84dc6e..f2e37a8e4 100644 > --- a/lib/librte_ring/meson.build > +++ b/lib/librte_ring/meson.build > @@ -7,10 +7,12 @@ headers =3D files('rte_ring.h', > 'rte_ring_c11_mem.h', > 'rte_ring_generic.h', > 'rte_ring_hts.h', > + 'rte_ring_hts_c11_mem.h', > 'rte_ring_hts_elem.h', > 'rte_ring_hts_generic.h', > 'rte_ring_peek.h', > 'rte_ring_rts.h', > + 'rte_ring_rts_c11_mem.h', > 'rte_ring_rts_elem.h', > 'rte_ring_rts_generic.h') >=20 > diff --git a/lib/librte_ring/rte_ring_hts.h b/lib/librte_ring/rte_ring_ht= s.h index > 062d7be6c..ddaa47ff1 100644 > --- a/lib/librte_ring/rte_ring_hts.h > +++ b/lib/librte_ring/rte_ring_hts.h > @@ -29,7 +29,11 @@ > extern "C" { > #endif >=20 > +#ifdef RTE_USE_C11_MEM_MODEL > +#include > +#else > #include > +#endif >=20 > /** > * @internal Enqueue several objects on the HTS ring. > diff --git a/lib/librte_ring/rte_ring_hts_c11_mem.h > b/lib/librte_ring/rte_ring_hts_c11_mem.h > new file mode 100644 > index 000000000..0218d0e7d > --- /dev/null > +++ b/lib/librte_ring/rte_ring_hts_c11_mem.h > @@ -0,0 +1,222 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * > + * Copyright (c) 2010-2020 Intel Corporation > + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org > + * All rights reserved. > + * Derived from FreeBSD's bufring.h > + * Used as BSD-3 Licensed with permission from Kip Macy. > + */ > + > +#ifndef _RTE_RING_HTS_C11_MEM_H_ > +#define _RTE_RING_HTS_C11_MEM_H_ > + > +/** > + * @file rte_ring_hts_c11_mem.h > + * It is not recommended to include this file directly, > + * include instead. > + * Contains internal helper functions for head/tail sync (HTS) ring mode= . > + * For more information please refer to . > + */ > + > +/** > + * @internal get current tail value. > + * Check that user didn't request to move tail above the head. > + * In that situation: > + * - return zero, that will cause abort any pending changes and > + * return head to its previous position. > + * - throw an assert in debug mode. > + */ > +static __rte_always_inline uint32_t > +__rte_ring_hts_get_tail(struct rte_ring_hts_headtail *ht, uint32_t *tail= , > + uint32_t num) > +{ > + uint32_t n; > + union rte_ring_ht_pos p; > + > + p.raw =3D __atomic_load_n(&ht->ht.raw, __ATOMIC_RELAXED); > + n =3D p.pos.head - p.pos.tail; > + > + RTE_ASSERT(n >=3D num); > + num =3D (n >=3D num) ? num : 0; > + > + *tail =3D p.pos.tail; > + return num; > +} > + > +/** > + * @internal set new values for head and tail as one atomic 64 bit opera= tion. > + * Should be used only in conjunction with __rte_ring_hts_get_tail. > + */ > +static __rte_always_inline void > +__rte_ring_hts_set_head_tail(struct rte_ring_hts_headtail *ht, uint32_t = tail, > + uint32_t num, uint32_t enqueue) > +{ > + union rte_ring_ht_pos p; > + > + RTE_SET_USED(enqueue); > + > + p.pos.head =3D tail + num; > + p.pos.tail =3D p.pos.head; > + > + __atomic_store_n(&ht->ht.raw, p.raw, __ATOMIC_RELEASE); } > + > +static __rte_always_inline void > +__rte_ring_hts_update_tail(struct rte_ring_hts_headtail *ht, uint32_t nu= m, > + uint32_t enqueue) > +{ > + uint32_t tail; > + > + num =3D __rte_ring_hts_get_tail(ht, &tail, num); > + __rte_ring_hts_set_head_tail(ht, tail, num, enqueue); } > + > +/** > + * @internal waits till tail will become equal to head. > + * Means no writer/reader is active for that ring. > + * Suppose to work as serialization point. > + */ > +static __rte_always_inline void > +__rte_ring_hts_head_wait(const struct rte_ring_hts_headtail *ht, > + union rte_ring_ht_pos *p) > +{ > + p->raw =3D __atomic_load_n(&ht->ht.raw, __ATOMIC_ACQUIRE); > + > + while (p->pos.head !=3D p->pos.tail) { > + rte_pause(); > + p->raw =3D __atomic_load_n(&ht->ht.raw, > __ATOMIC_ACQUIRE); > + } > +} > + > +/** > + * @internal This function updates the producer head for enqueue > + * > + * @param r > + * A pointer to the ring structure > + * @param is_sp > + * Indicates whether multi-producer path is needed or not > + * @param n > + * The number of elements we will want to enqueue, i.e. how far should= the > + * head be moved > + * @param behavior > + * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a rin= g > + * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from > ring > + * @param old_head > + * Returns head value as it was before the move, i.e. where enqueue st= arts > + * @param new_head > + * Returns the current/new head value i.e. where enqueue finishes > + * @param free_entries > + * Returns the amount of free space in the ring BEFORE head was moved > + * @return > + * Actual number of objects enqueued. > + * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n only. > + */ > +static __rte_always_inline unsigned int > +__rte_ring_hts_move_prod_head(struct rte_ring *r, unsigned int num, > + enum rte_ring_queue_behavior behavior, uint32_t *old_head, > + uint32_t *free_entries) > +{ > + uint32_t n; > + union rte_ring_ht_pos np, op; > + > + const uint32_t capacity =3D r->capacity; > + > + do { > + /* Reset n to the initial burst count */ > + n =3D num; > + > + /* wait for tail to be equal to head, , acquire point */ > + __rte_ring_hts_head_wait(&r->hts_prod, &op); > + > + /* > + * The subtraction is done between two unsigned 32bits value > + * (the result is always modulo 32 bits even if we have > + * *old_head > cons_tail). So 'free_entries' is always between > 0 > + * and capacity (which is < size). > + */ > + *free_entries =3D capacity + r->cons.tail - op.pos.head; > + > + /* check that we have enough room in ring */ > + if (unlikely(n > *free_entries)) > + n =3D (behavior =3D=3D RTE_RING_QUEUE_FIXED) ? > + 0 : *free_entries; > + > + if (n =3D=3D 0) > + break; > + > + np.pos.tail =3D op.pos.tail; > + np.pos.head =3D op.pos.head + n; > + > + } while (__atomic_compare_exchange_n(&r->hts_prod.ht.raw, > + &op.raw, np.raw, > + 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED) =3D=3D 0); __ATOMIC_RELEASE can be __ATOMIC_RELAXED. The RELEASE while updating after = the elements are written is enough. > + > + *old_head =3D op.pos.head; > + return n; > +} > + > +/** > + * @internal This function updates the consumer head for dequeue > + * > + * @param r > + * A pointer to the ring structure > + * @param is_sc > + * Indicates whether multi-consumer path is needed or not > + * @param n > + * The number of elements we will want to enqueue, i.e. how far should= the > + * head be moved > + * @param behavior > + * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a rin= g > + * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from > ring > + * @param old_head > + * Returns head value as it was before the move, i.e. where dequeue st= arts > + * @param new_head > + * Returns the current/new head value i.e. where dequeue finishes > + * @param entries > + * Returns the number of entries in the ring BEFORE head was moved > + * @return > + * - Actual number of objects dequeued. > + * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n only= . > + */ > +static __rte_always_inline unsigned int > +__rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num, > + enum rte_ring_queue_behavior behavior, uint32_t *old_head, > + uint32_t *entries) > +{ > + uint32_t n; > + union rte_ring_ht_pos np, op; > + > + /* move cons.head atomically */ > + do { > + /* Restore n as it may change every loop */ > + n =3D num; > + > + /* wait for tail to be equal to head */ > + __rte_ring_hts_head_wait(&r->hts_cons, &op); > + > + /* The subtraction is done between two unsigned 32bits value > + * (the result is always modulo 32 bits even if we have > + * cons_head > prod_tail). So 'entries' is always between 0 > + * and size(ring)-1. > + */ > + *entries =3D r->prod.tail - op.pos.head; > + > + /* Set the actual entries for dequeue */ > + if (n > *entries) > + n =3D (behavior =3D=3D RTE_RING_QUEUE_FIXED) ? 0 : > *entries; > + > + if (unlikely(n =3D=3D 0)) > + break; > + > + np.pos.tail =3D op.pos.tail; > + np.pos.head =3D op.pos.head + n; > + > + } while (__atomic_compare_exchange_n(&r->hts_cons.ht.raw, > + &op.raw, np.raw, > + 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED) =3D=3D 0); Same here, RELEASE can be RELAXED. > + > + *old_head =3D op.pos.head; > + return n; > +} > + > +#endif /* _RTE_RING_HTS_C11_MEM_H_ */ > /** > * @internal Enqueue several objects on the RTS ring. > diff --git a/lib/librte_ring/rte_ring_rts_c11_mem.h > b/lib/librte_ring/rte_ring_rts_c11_mem.h > new file mode 100644 > index 000000000..b72901497 > --- /dev/null > +++ b/lib/librte_ring/rte_ring_rts_c11_mem.h > @@ -0,0 +1,198 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * > + * Copyright (c) 2010-2017 Intel Corporation > + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org > + * All rights reserved. > + * Derived from FreeBSD's bufring.h > + * Used as BSD-3 Licensed with permission from Kip Macy. > + */ > + > +#ifndef _RTE_RING_RTS_C11_MEM_H_ > +#define _RTE_RING_RTS_C11_MEM_H_ > + > +/** > + * @file rte_ring_rts_c11_mem.h > + * It is not recommended to include this file directly, > + * include instead. > + * Contains internal helper functions for Relaxed Tail Sync (RTS) ring m= ode. > + * For more information please refer to . > + */ > + > +/** > + * @internal This function updates tail values. > + */ > +static __rte_always_inline void > +__rte_ring_rts_update_tail(struct rte_ring_rts_headtail *ht) { > + union rte_ring_ht_poscnt h, ot, nt; > + > + /* > + * If there are other enqueues/dequeues in progress that > + * might preceded us, then don't update tail with new value. > + */ > + > + ot.raw =3D __atomic_load_n(&ht->tail.raw, __ATOMIC_ACQUIRE); This can be RELAXED. This thread is reading the value that it updated earli= er, so it should be able to see the value it updated. > + > + do { > + /* on 32-bit systems we have to do atomic read here */ > + h.raw =3D __atomic_load_n(&ht->head.raw, > __ATOMIC_RELAXED); > + > + nt.raw =3D ot.raw; > + if (++nt.val.cnt =3D=3D h.val.cnt) > + nt.val.pos =3D h.val.pos; > + > + } while (__atomic_compare_exchange_n(&ht->tail.raw, &ot.raw, > nt.raw, > + 0, __ATOMIC_RELEASE, __ATOMIC_ACQUIRE) =3D=3D 0); } > + > +/** > + * @internal This function waits till head/tail distance wouldn't > + * exceed pre-defined max value. > + */ > +static __rte_always_inline void > +__rte_ring_rts_head_wait(const struct rte_ring_rts_headtail *ht, > + union rte_ring_ht_poscnt *h) > +{ > + uint32_t max; > + > + max =3D ht->htd_max; > + h->raw =3D __atomic_load_n(&ht->head.raw, __ATOMIC_ACQUIRE); > + > + while (h->val.pos - ht->tail.val.pos > max) { > + rte_pause(); > + h->raw =3D __atomic_load_n(&ht->head.raw, > __ATOMIC_ACQUIRE); > + } > +} > + > +/** > + * @internal This function updates the producer head for enqueue. > + * > + * @param r > + * A pointer to the ring structure > + * @param is_sp > + * Indicates whether multi-producer path is needed or not > + * @param n > + * The number of elements we will want to enqueue, i.e. how far should= the > + * head be moved > + * @param behavior > + * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items from a rin= g > + * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from > ring > + * @param old_head > + * Returns head value as it was before the move, i.e. where enqueue st= arts > + * @param new_head > + * Returns the current/new head value i.e. where enqueue finishes > + * @param free_entries > + * Returns the amount of free space in the ring BEFORE head was moved > + * @return > + * Actual number of objects enqueued. > + * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n only. > + */ > +static __rte_always_inline uint32_t > +__rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num, > + enum rte_ring_queue_behavior behavior, uint32_t *old_head, > + uint32_t *free_entries) > +{ > + uint32_t n; > + union rte_ring_ht_poscnt nh, oh; > + > + const uint32_t capacity =3D r->capacity; > + > + do { > + /* Reset n to the initial burst count */ > + n =3D num; > + > + /* read prod head (may spin on prod tail, acquire point) */ > + __rte_ring_rts_head_wait(&r->rts_prod, &oh); > + > + /* > + * The subtraction is done between two unsigned 32bits value > + * (the result is always modulo 32 bits even if we have > + * *old_head > cons_tail). So 'free_entries' is always between > 0 > + * and capacity (which is < size). > + */ > + *free_entries =3D capacity + r->cons.tail - oh.val.pos; > + > + /* check that we have enough room in ring */ > + if (unlikely(n > *free_entries)) > + n =3D (behavior =3D=3D RTE_RING_QUEUE_FIXED) ? > + 0 : *free_entries; > + > + if (n =3D=3D 0) > + break; > + > + nh.val.pos =3D oh.val.pos + n; > + nh.val.cnt =3D oh.val.cnt + 1; > + > + } while (__atomic_compare_exchange_n(&r->rts_prod.head.raw, > + &oh.raw, nh.raw, > + 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED) =3D=3D 0); > + > + *old_head =3D oh.val.pos; > + return n; > +} > + > +/** > + * @internal This function updates the consumer head for dequeue > + * > + * @param r > + * A pointer to the ring structure > + * @param is_sc > + * Indicates whether multi-consumer path is needed or not > + * @param n > + * The number of elements we will want to enqueue, i.e. how far should= the > + * head be moved > + * @param behavior > + * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from a rin= g > + * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from > ring > + * @param old_head > + * Returns head value as it was before the move, i.e. where dequeue st= arts > + * @param new_head > + * Returns the current/new head value i.e. where dequeue finishes > + * @param entries > + * Returns the number of entries in the ring BEFORE head was moved > + * @return > + * - Actual number of objects dequeued. > + * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n only= . > + */ > +static __rte_always_inline unsigned int > +__rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num, > + enum rte_ring_queue_behavior behavior, uint32_t *old_head, > + uint32_t *entries) > +{ > + uint32_t n; > + union rte_ring_ht_poscnt nh, oh; > + > + /* move cons.head atomically */ > + do { > + /* Restore n as it may change every loop */ > + n =3D num; > + > + /* read cons head (may spin on cons tail, acquire point) */ > + __rte_ring_rts_head_wait(&r->rts_cons, &oh); > + > + /* The subtraction is done between two unsigned 32bits value > + * (the result is always modulo 32 bits even if we have > + * cons_head > prod_tail). So 'entries' is always between 0 > + * and size(ring)-1. > + */ > + *entries =3D r->prod.tail - oh.val.pos; > + > + /* Set the actual entries for dequeue */ > + if (n > *entries) > + n =3D (behavior =3D=3D RTE_RING_QUEUE_FIXED) ? 0 : > *entries; > + > + if (unlikely(n =3D=3D 0)) > + break; > + > + nh.val.pos =3D oh.val.pos + n; > + nh.val.cnt =3D oh.val.cnt + 1; > + > + } while (__atomic_compare_exchange_n(&r->rts_cons.head.raw, > + &oh.raw, nh.raw, > + 1, __ATOMIC_RELEASE, __ATOMIC_RELAXED) =3D=3D 0); > + > + *old_head =3D oh.val.pos; > + return n; > +} > + > +#endif /* _RTE_RING_RTS_C11_MEM_H_ */ > diff --git a/lib/librte_ring/rte_ring_rts_elem.h > b/lib/librte_ring/rte_ring_rts_elem.h > index 71a331b23..23d8aeec7 100644 > --- a/lib/librte_ring/rte_ring_rts_elem.h > +++ b/lib/librte_ring/rte_ring_rts_elem.h > @@ -24,7 +24,11 @@ > extern "C" { > #endif >=20 > +#ifdef RTE_USE_C11_MEM_MODEL > +#include > +#else > #include > +#endif >=20 > /** > * @internal Enqueue several objects on the RTS ring. > -- > 2.17.1