From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BDEB542FD1; Sat, 5 Aug 2023 00:51:26 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3F404410EE; Sat, 5 Aug 2023 00:51:26 +0200 (CEST) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (mail-am6eur05on2062.outbound.protection.outlook.com [40.107.22.62]) by mails.dpdk.org (Postfix) with ESMTP id 8FC7040A81 for ; Sat, 5 Aug 2023 00:51:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uYLp1LD7T9vJHu7qDFPx6CcfSJy1ZZ77epMd2kL2Fak=; b=2miEtuVo7N4iPQtXfCdHDjpzUmE+TPPRK69pXDc1nDuDfrl/S1MvyBuDSvBsTEkOMg9a2G7rKH9Mfiocrsyw+d69ShcMJa28GFSTk7fhn6mb2lywzKZM3Y974xGHc50P+3fImx1xnAYkbX2moR6/Dm+NqwbZ95IAJMfkjKyjBX0= Received: from AM6P195CA0107.EURP195.PROD.OUTLOOK.COM (2603:10a6:209:86::48) by GV1PR08MB10370.eurprd08.prod.outlook.com (2603:10a6:150:a7::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6652.20; Fri, 4 Aug 2023 22:51:16 +0000 Received: from AM7EUR03FT057.eop-EUR03.prod.protection.outlook.com (2603:10a6:209:86:cafe::ef) by AM6P195CA0107.outlook.office365.com (2603:10a6:209:86::48) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6652.22 via Frontend Transport; Fri, 4 Aug 2023 22:51:16 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM7EUR03FT057.mail.protection.outlook.com (100.127.140.117) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6678.10 via Frontend Transport; Fri, 4 Aug 2023 22:51:16 +0000 Received: ("Tessian outbound e1fdbe8a48d3:v145"); Fri, 04 Aug 2023 22:51:15 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: f5858632d2357b75 X-CR-MTA-TID: 64aa7808 Received: from 64d1ed82ca64.2 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 88904D1B-A2CB-4023-8B47-BC27563A711B.1; Fri, 04 Aug 2023 22:51:04 +0000 Received: from EUR01-HE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 64d1ed82ca64.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Fri, 04 Aug 2023 22:51:04 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=X1hQsXnF0BZc0eIocck2S5sqD5vA86JVSbId+KqRe1pjymPdZ88pE5I2IMvdLBDcv4I7gIwNcJBGZE3zNA7pta2xrpCr796YIM5asZ4TNhlyVQJq5dy4ubhf1Sv17SHijBNnt0Jkjk4jhJNNtyyP/VYi2APng5eO784k021H6eZTPHkUOBTQcdQqsejulfI9FpeI9HYWTlvBkrLSoNDahp/ClMr/9ANr74xRGzVw88DoRsAOMpkhSoX1x5XdpieF/0mYIBzHlFTJ3DvN9tJbb9Iw4QPlQjQZ1aNUugt185TObr+zGBoRdBNULOgYfnRTGuTx9/UR2jRfxbjlUA18Ag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=uYLp1LD7T9vJHu7qDFPx6CcfSJy1ZZ77epMd2kL2Fak=; b=ECyTCa02O50jyq1tL5bBrH0I+rC424s12C6hHTLAfM5xxtQ9Ljo6onKo+Zc0sGE9q33VHsbpdR/LkTo3Oi3DzU+b+319EttZHXQTXflsVv+aag1nUU2B2+AecwV/AUw4feL78cEL26pgJreQjmaglLfhOoZAAj2kV3fvjOybocu/sEaLYzLwBPLBqUuNm9svZl1BoyWkRAVDy5KyYoDSc/++WlqTFt8SAxQWcGG85lkcnnoUGuU4YSJLDbjhbkOc8CNwBrpTkpPjmX//xmHzyoMZHjz/vXmAUqV1ZzxposUisFbd7+SLnienws5er6r0QkAJurRCKVGI+ymNX1Oamg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uYLp1LD7T9vJHu7qDFPx6CcfSJy1ZZ77epMd2kL2Fak=; b=2miEtuVo7N4iPQtXfCdHDjpzUmE+TPPRK69pXDc1nDuDfrl/S1MvyBuDSvBsTEkOMg9a2G7rKH9Mfiocrsyw+d69ShcMJa28GFSTk7fhn6mb2lywzKZM3Y974xGHc50P+3fImx1xnAYkbX2moR6/Dm+NqwbZ95IAJMfkjKyjBX0= Received: from DB5PR08MB10022.eurprd08.prod.outlook.com (2603:10a6:10:4a7::14) by DB9PR08MB7650.eurprd08.prod.outlook.com (2603:10a6:10:30d::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6631.47; Fri, 4 Aug 2023 22:50:57 +0000 Received: from DB5PR08MB10022.eurprd08.prod.outlook.com ([fe80::5d06:d2f8:c197:b5f7]) by DB5PR08MB10022.eurprd08.prod.outlook.com ([fe80::5d06:d2f8:c197:b5f7%3]) with mapi id 15.20.6652.022; Fri, 4 Aug 2023 22:50:57 +0000 From: Wathsala Wathawana Vithanage To: Konstantin Ananyev , Honnappa Nagarahalli , "konstantin.v.ananyev@yandex.ru" , "thomas@monjalon.net" , Ruifeng Wang CC: "dev@dpdk.org" , nd , Justin He , nd Subject: RE: [RFC] ring: further performance improvements with C11 Thread-Topic: [RFC] ring: further performance improvements with C11 Thread-Index: AQHZn8XqJVzwOdI7LE+bgMCe4o+n9K+MTC6AgEq/lwCAA/7/sA== Date: Fri, 4 Aug 2023 22:50:57 +0000 Message-ID: References: <20230615201335.919563-1-wathsala.vithanage@arm.com> <20230615201335.919563-2-wathsala.vithanage@arm.com> <67a8987cb0d5456b9c99887402ea30af@huawei.com> In-Reply-To: <67a8987cb0d5456b9c99887402ea30af@huawei.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 759C20E1AC2FEE4F84EC269B372E46DE.0 Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; x-ms-traffictypediagnostic: DB5PR08MB10022:EE_|DB9PR08MB7650:EE_|AM7EUR03FT057:EE_|GV1PR08MB10370:EE_ X-MS-Office365-Filtering-Correlation-Id: 6aa07f11-f77d-4108-322d-08db953d4f81 x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-checkrecipientrouted: true nodisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: jdFXdOXHk4RuN1jWxyTggi/wef3epOxrfPm2lvYjjEVDSHggl2zhkU0VBcj777ez5AQRhRi1tiBVseGECT/1A7MuCYNGSpjTRlMWJIQiezFbQoAOa/eaGji6vzW7Esh1furrtqfkjEozKsPMORJzfmGgslbfZg5SZyQyrGcDjeaTE9GQCzE/cq9ajcAFOG/74cPEvJHadJhYR/VqmLp5D57Kaz3o/J58+iDGPmuveojDd4IyQIgRe8R0cV9EX+lzqxxqUqlqvrfaHC/F72P0YzfmU6UVTXzWqIucueYcY3JUzXzxatCdW0um4l7k43eVBgHeRsr2levEME4hrZVxj/PcGm5k7SBGXzjFfIxNpJ1lkk5v5zzEuU9/GP9602RBJGPoyIgMmmLkbqpUyMwlGSqr8qi9tCvvGOAvZnY+BYkjbS7fLd7W6vFBZnzKjxItrEWO9TH0PJbwSo4tqurn9yIh/C65BmzLu30WhjLIxSsrUivvHi6+FZr/NhlrVqA+bSMwm56DgNHnUKAD2fmQ4sI33lgYO4LcHJKBwteYdKkKTEEDL0+218udoWDCftfm8mwNAaMC4cBcHRgNLLDaxgAwtMTwiJYrM2Jn6RGBrD97o+h/T/TuqVS2rkB0XcTG X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DB5PR08MB10022.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(376002)(366004)(39860400002)(346002)(136003)(396003)(186006)(1800799003)(451199021)(38070700005)(33656002)(86362001)(55016003)(122000001)(478600001)(54906003)(110136005)(38100700002)(8676002)(83380400001)(6506007)(26005)(53546011)(8936002)(52536014)(9686003)(966005)(71200400001)(7696005)(66556008)(6636002)(2906002)(316002)(4326008)(66476007)(66446008)(41300700001)(76116006)(64756008)(5660300002)(66946007)(23180200003); DIR:OUT; SFP:1101; Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR08MB7650 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM7EUR03FT057.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 6ab5300e-155d-451d-30e5-08db953d4418 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: xlhbLZXIWWCXhVAm6MYisYE+TBwerciEhnpkR2K3D3kO2WiDxt423+elAjQ1TLnp1XXWTzzWK0cWmAltQjWaW011mDtmQmufpXbQcihf/DC2NpJh/CiqgcqL7Z9IrycJDo7IN2RbkDEQyX74hFHFMe/FXFZ69PDH2b23KaGy40/I0DvAOzAYROnk4S56RVLnN4hWLeGg/hTaiSOZVbuW+gg+PI+5zrwlTbRh+KnAyoj55jLaQysEMEBc8VMFJ/ZvtohNf50ezIjRVu0PLpJXc//JERGrGJlVAaayy5upbBPFcur5ALdzZCG3Z7YUL4mTp0lDtLkD/vecCLO95s4t7665bTI5cAF6cbkkp4Bc6DdPUyI24TiQBgSWkZnWZ/pVOuLgrKYN2M9PW6PlGX29r7IBgh40+jjHrwmdcWGp+/1BnxAZEhxUhKF8pCkVsQrYNIXOgAQFIt4NFDfimQGCM4zCTlyTbWg7qzfo4MonYXNx5o9n03eXXOdWUl33K4lFIzmWMEs0hyKjXtLrxRJPJxLy9P41uBiGmvPyJv5xKOK5ABqYDjsehKx3VJrgrsjXEXBe49/7wnYKhVC1X1GX6GS/20lf/H8pMxMm7fEm9jhxKjBG/BGSjgmM3NkhZ/ZZazztnV3NjW+UwG8ADRPqUuHWy5Z1EPnJbi/+k0PkcZd4kffa5UlcP7pQJJjGme/VPWnw/Y4G+UvtXQzNNGeYw0ucrHpYN+xseFLiYSz6ARsvVSZ7SVKo5CYFe1Bdgi/8ZykVYxMsnCsxkggyH5KwEV2CTEgnfZD6pM2JIEOhX5k= X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230028)(4636009)(376002)(39860400002)(346002)(136003)(396003)(186006)(1800799003)(451199021)(82310400008)(40470700004)(46966006)(36840700001)(7696005)(54906003)(110136005)(41300700001)(478600001)(53546011)(5660300002)(8936002)(8676002)(336012)(52536014)(4326008)(6636002)(316002)(9686003)(966005)(6506007)(70586007)(70206006)(26005)(40460700003)(2906002)(83380400001)(55016003)(47076005)(40480700001)(36860700001)(356005)(82740400003)(81166007)(33656002)(86362001)(525324003)(23180200003); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Aug 2023 22:51:16.1842 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6aa07f11-f77d-4108-322d-08db953d4f81 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM7EUR03FT057.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV1PR08MB10370 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Konstantin Ananyev > Sent: Wednesday, August 2, 2023 4:43 AM > To: Wathsala Wathawana Vithanage ; > Honnappa Nagarahalli ; > konstantin.v.ananyev@yandex.ru; thomas@monjalon.net; Ruifeng Wang > > Cc: dev@dpdk.org; nd ; Justin He > Subject: RE: [RFC] ring: further performance improvements with C11 >=20 >=20 >=20 > > For improved performance over the current C11 based ring > > implementation following changes were made. > > (1) Replace tail store with RELEASE semantics in > > __rte_ring_update_tail with a RELEASE fence. Replace load of the tail > > with ACQUIRE semantics in __rte_ring_move_prod_head and > > __rte_ring_move_cons_head with ACQUIRE fences. > > (2) Remove ACQUIRE fences between load of the old_head and load of the > > cons_tail in __rte_ring_move_prod_head and __rte_ring_move_cons_head. > > These two fences are not required for the safety of the ring library. >=20 > Hmm... with these changes, aren't we re-introducing the old bug fixed by = this > commit: Cover letter explains why this barrier does not solve what it intends to so= lve and Why it should not matter. https://mails.dpdk.org/archives/dev/2023-June/270874.html >=20 > commit 9bc2cbb007c0a3335c5582357ae9f6d37ea0b654 > Author: Jia He > Date: Fri Nov 10 03:30:42 2017 +0000 >=20 > ring: guarantee load/load order in enqueue and dequeue >=20 > We watched a rte panic of mbuf_autotest in our qualcomm arm64 server > (Amberwing). >=20 > Root cause: > In __rte_ring_move_cons_head() > ... > do { > /* Restore n as it may change every loop */ > n =3D max; >=20 > *old_head =3D r->cons.head; //1st load > const uint32_t prod_tail =3D r->prod.tail; //2nd load >=20 > In weak memory order architectures (powerpc,arm), the 2nd load might = be > reodered before the 1st load, that makes *entries is bigger than we w= anted. > This nasty reording messed enque/deque up. > .... > ? >=20 > > > > Signed-off-by: Wathsala Vithanage > > Reviewed-by: Honnappa Nagarahalli > > Reviewed-by: Ruifeng Wang > > --- > > .mailmap | 1 + > > lib/ring/rte_ring_c11_pvt.h | 35 ++++++++++++++++++++--------------- > > 2 files changed, 21 insertions(+), 15 deletions(-) > > > > diff --git a/.mailmap b/.mailmap > > index 4018f0fc47..367115d134 100644 > > --- a/.mailmap > > +++ b/.mailmap > > @@ -1430,6 +1430,7 @@ Walter Heymans > > > Wang Sheng-Hui Wangyu (Eric) > > Waterman Cao > > +Wathsala Vithanage > > Weichun Chen Wei Dai > > Weifeng Li diff --git > > a/lib/ring/rte_ring_c11_pvt.h b/lib/ring/rte_ring_c11_pvt.h index > > f895950df4..63fe58ce9e 100644 > > --- a/lib/ring/rte_ring_c11_pvt.h > > +++ b/lib/ring/rte_ring_c11_pvt.h > > @@ -16,6 +16,13 @@ __rte_ring_update_tail(struct rte_ring_headtail *ht, > uint32_t old_val, > > uint32_t new_val, uint32_t single, uint32_t enqueue) { > > RTE_SET_USED(enqueue); > > + /* > > + * Updating of ht->tail cannot happen before elements are added to or > > + * removed from the ring, as it could result in data races between > > + * producer and consumer threads. Therefore we need a release > > + * barrier here. > > + */ > > + rte_atomic_thread_fence(__ATOMIC_RELEASE); > > > > /* > > * If there are other enqueues/dequeues in progress that preceded > > us, @@ -24,7 +31,7 @@ __rte_ring_update_tail(struct rte_ring_headtail > *ht, uint32_t old_val, > > if (!single) > > rte_wait_until_equal_32(&ht->tail, old_val, > __ATOMIC_RELAXED); > > > > - __atomic_store_n(&ht->tail, new_val, __ATOMIC_RELEASE); > > + __atomic_store_n(&ht->tail, new_val, __ATOMIC_RELAXED); > > } > > > > /** > > @@ -66,14 +73,8 @@ __rte_ring_move_prod_head(struct rte_ring *r, > unsigned int is_sp, > > /* Reset n to the initial burst count */ > > n =3D max; > > > > - /* Ensure the head is read before tail */ > > - __atomic_thread_fence(__ATOMIC_ACQUIRE); > > - > > - /* load-acquire synchronize with store-release of ht->tail > > - * in update_tail. > > - */ > > cons_tail =3D __atomic_load_n(&r->cons.tail, > > - __ATOMIC_ACQUIRE); > > + __ATOMIC_RELAXED); > > > > /* The subtraction is done between two unsigned 32bits value > > * (the result is always modulo 32 bits even if we have @@ - > 100,6 > > +101,11 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int > is_sp, > > 0, __ATOMIC_RELAXED, > > __ATOMIC_RELAXED); > > } while (unlikely(success =3D=3D 0)); > > + /* > > + * Ensure that updates to the ring doesn't rise above > > + * load of the new_head in SP and MP cases. > > + */ > > + rte_atomic_thread_fence(__ATOMIC_ACQUIRE); > > return n; > > } > > > > @@ -142,14 +148,8 @@ __rte_ring_move_cons_head(struct rte_ring *r, int > is_sc, > > /* Restore n as it may change every loop */ > > n =3D max; > > > > - /* Ensure the head is read before tail */ > > - __atomic_thread_fence(__ATOMIC_ACQUIRE); > > - > > - /* this load-acquire synchronize with store-release of ht->tail > > - * in update_tail. > > - */ > > prod_tail =3D __atomic_load_n(&r->prod.tail, > > - __ATOMIC_ACQUIRE); > > + __ATOMIC_RELAXED); > > > > /* The subtraction is done between two unsigned 32bits value > > * (the result is always modulo 32 bits even if we have @@ - > 175,6 > > +175,11 @@ __rte_ring_move_cons_head(struct rte_ring *r, int is_sc, > > 0, > __ATOMIC_RELAXED, > > __ATOMIC_RELAXED); > > } while (unlikely(success =3D=3D 0)); > > + /* > > + * Ensure that updates to the ring doesn't rise above > > + * load of the new_head in SP and MP cases. > > + */ > > + rte_atomic_thread_fence(__ATOMIC_ACQUIRE); > > return n; > > } > > > > -- > > 2.25.1 > >