From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7B75DA2EFC for ; Mon, 14 Oct 2019 12:00:35 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 2BC4B1C13A; Mon, 14 Oct 2019 12:00:35 +0200 (CEST) Received: from EUR03-VE1-obe.outbound.protection.outlook.com (mail-eopbgr50043.outbound.protection.outlook.com [40.107.5.43]) by dpdk.org (Postfix) with ESMTP id 9DC951C0B4; Mon, 14 Oct 2019 12:00:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JwdtJwXVBWIBPTT+gSipBPytUh+UWIbg1Wh/p0ZtKnw=; b=EeUE6l8w82ylihJnD5fi1zG8CWVelRK+tWoT44siYnHvu7MtBduydDd2SVizWKzd/wgcDGg4hYnnUv6LaawRqdOjjGgw47TTO0hfE+sREGOfFEllW5N2w7BnySHR1+S85tO4QHL7l32Cum8tvZQMzMKK9MbhkdGG/XN/4xbGGV0= Received: from VI1PR08CA0085.eurprd08.prod.outlook.com (2603:10a6:800:d3::11) by DBBPR08MB4555.eurprd08.prod.outlook.com (2603:10a6:10:cb::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2347.21; Mon, 14 Oct 2019 10:00:27 +0000 Received: from VE1EUR03FT030.eop-EUR03.prod.protection.outlook.com (2a01:111:f400:7e09::208) by VI1PR08CA0085.outlook.office365.com (2603:10a6:800:d3::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.2347.16 via Frontend Transport; Mon, 14 Oct 2019 10:00:27 +0000 Authentication-Results: spf=temperror (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=none action=none header.from=arm.com; Received-SPF: TempError (protection.outlook.com: error in processing during lookup of arm.com: DNS Timeout) Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by VE1EUR03FT030.mail.protection.outlook.com (10.152.18.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.2305.15 via Frontend Transport; Mon, 14 Oct 2019 10:00:25 +0000 Received: ("Tessian outbound 6481c7fa5a3c:v33"); Mon, 14 Oct 2019 10:00:23 +0000 X-CR-MTA-TID: 64aa7808 Received: from d474485dc645.2 (ip-172-16-0-2.eu-west-1.compute.internal [104.47.12.57]) by 64aa7808-outbound-1.mta.getcheckrecipient.com id 8D40F139-13B3-497F-81DD-0F642804216F.1; Mon, 14 Oct 2019 10:00:18 +0000 Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-db3eur04lp2057.outbound.protection.outlook.com [104.47.12.57]) by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id d474485dc645.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384); Mon, 14 Oct 2019 10:00:18 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IS1E6dr7lQuv/L0eeGTrahcPl6PyN3Ih/HQPsgdijc1Hf/rXQkigm9BcBW/+cWFI54ImMla/iGgGUJxXeb3CLg97EjFQqngdo25klRo69j8+N31WjeHODXyEghakS1VDmYBTLhOHDasUO7UvBLjQylI4jF359q91urZy1zGXKzc4C1hh6ypP7Zj/Pz+oSlzKU1M4qu4lusO521C1vqwgCZARCy5NWg5feSR9ThqfqC2tCyoATQ7OxZLIf9EqxOV4xSTXK4x16WTcHmLdjbJZ+/m4rA/IpioXmJD/TBr1Zsnw9wCsiQSQkmUS8ZjDZ32KQ+bM5VRLTIWHRypUpzPiDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JwdtJwXVBWIBPTT+gSipBPytUh+UWIbg1Wh/p0ZtKnw=; b=b2bc/LNnAo/RQsy4Cuzg4l9ePV2drgvSgQJ//otV8e2h3flwt4/dCscPHMCOlcoVLXBOu2AGLXAigYH4nE4zQldFgJYSv7N7nEV7jivXJIjgoaeMROb38USikKt96m1iMAkT7mlvQ4PClpWkxLd8glTL1/FQ1RHIzgltFpwZf+3nGwNJfrb7wz6OH4eyq/lq9pidOlDq8EySVLVACQ6VcTxGlAp4+V7gJj2nSGS4sU9DAMfB31LEEQvvhGrkidYjSFUegoI+tbD+f2pcbah1QE1Wrga//DxoVNNWewAxxBlkFN3PKaDWqPOx2w9dC7PLya1OGIUBIAqW7VxfKmVoLA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JwdtJwXVBWIBPTT+gSipBPytUh+UWIbg1Wh/p0ZtKnw=; b=EeUE6l8w82ylihJnD5fi1zG8CWVelRK+tWoT44siYnHvu7MtBduydDd2SVizWKzd/wgcDGg4hYnnUv6LaawRqdOjjGgw47TTO0hfE+sREGOfFEllW5N2w7BnySHR1+S85tO4QHL7l32Cum8tvZQMzMKK9MbhkdGG/XN/4xbGGV0= Received: from AM0PR08MB3986.eurprd08.prod.outlook.com (20.178.118.90) by AM0PR08MB3508.eurprd08.prod.outlook.com (20.177.110.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2347.16; Mon, 14 Oct 2019 10:00:16 +0000 Received: from AM0PR08MB3986.eurprd08.prod.outlook.com ([fe80::8106:8373:8559:3c07]) by AM0PR08MB3986.eurprd08.prod.outlook.com ([fe80::8106:8373:8559:3c07%7]) with mapi id 15.20.2347.021; Mon, 14 Oct 2019 10:00:16 +0000 From: "Ruifeng Wang (Arm Technology China)" To: Honnappa Nagarahalli , "david.hunt@intel.com" CC: "dev@dpdk.org" , "hkalra@marvell.com" , "Gavin Hu (Arm Technology China)" , nd , "stable@dpdk.org" , nd , nd Thread-Topic: [PATCH v2 1/2] lib/distributor: fix deadlock issue for aarch64 Thread-Index: AQHVgKcYxHcKiTVgzk6083x+SlBRBadX22QAgAIPFuA= Date: Mon, 14 Oct 2019 10:00:16 +0000 Message-ID: References: <20191008095524.1585-1-ruifeng.wang@arm.com> <20191012024352.23545-1-ruifeng.wang@arm.com> <20191012024352.23545-2-ruifeng.wang@arm.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 5a1466e3-ecd9-457d-ad88-a4b605623104.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Ruifeng.Wang@arm.com; x-originating-ip: [113.29.88.7] x-ms-publictraffictype: Email X-MS-Office365-Filtering-Correlation-Id: fdcd8e19-1666-419f-ebfc-08d7508d55f5 X-MS-Office365-Filtering-HT: Tenant X-MS-TrafficTypeDiagnostic: AM0PR08MB3508:|AM0PR08MB3508:|DBBPR08MB4555: x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true x-ms-oob-tlc-oobclassifiers: OLM:8273;OLM:8273; x-forefront-prvs: 01901B3451 X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(396003)(366004)(346002)(376002)(136003)(189003)(199004)(13464003)(66476007)(66446008)(76176011)(66556008)(4326008)(64756008)(81156014)(26005)(33656002)(186003)(66946007)(81166006)(8676002)(2906002)(6436002)(256004)(14444005)(7696005)(316002)(53546011)(102836004)(2501003)(54906003)(55236004)(76116006)(99286004)(110136005)(6506007)(476003)(74316002)(25786009)(86362001)(486006)(305945005)(7736002)(66066001)(6246003)(446003)(11346002)(9686003)(229853002)(8936002)(55016002)(71190400001)(71200400001)(52536014)(478600001)(5660300002)(30864003)(6116002)(14454004)(3846002); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR08MB3508; H:AM0PR08MB3986.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: KC9chCXVAU1tnaD3SKIHLQKYBgjVS1mJsmQVZGE2QwnGPHXCjO624+zzbjFhAi/HEXNJvtMBRwlZSinzXWwBSfRgHLLgjwm7xI2PKTYPpkEgPrY7lP9OA3scXRclmuTtGGzujMuKeb/41njp21DZAAaThg8MAvSkk8JU9BoDB/5fL/GXZVW5D/Jp4JbqSQx8G0j6eACJhh3DA077vLjHzVgAcb5SQJP8TVZU9qrvVAUdpHGxwQdXLm81hfQIFyD/0wE8xU5nmBalliLXfemUHquq+hchqJvT/8X556izJSiltv3dWhFZqIybQpQufc7nBG3mHfRinfyrEBNYQfaenOwm5E9gQYccc+bPvKRs1XdYFV5gEpeO84Jx8niM/XyA8zaHA4nNB8ad/k9Mmf/4bCLhKsWpyacl5uIimtjk1lQ= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR08MB3508 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Ruifeng.Wang@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: VE1EUR03FT030.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; IPV:CAL; SCL:-1; CTRY:IE; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(346002)(396003)(136003)(376002)(13464003)(189003)(199004)(46406003)(478600001)(30864003)(26826003)(81166006)(229853002)(8676002)(81156014)(2906002)(336012)(14444005)(66066001)(99286004)(4326008)(450100002)(86362001)(63350400001)(8936002)(126002)(305945005)(9686003)(52536014)(23726003)(25786009)(3846002)(6116002)(55016002)(8746002)(486006)(446003)(476003)(11346002)(50466002)(5660300002)(47776003)(54906003)(36906005)(316002)(110136005)(76130400001)(2501003)(70206006)(70586007)(186003)(7696005)(26005)(7736002)(14454004)(33656002)(102836004)(97756001)(76176011)(53546011)(6506007)(356004)(6246003)(74316002)(22756006); DIR:OUT; SFP:1101; SCL:1; SRVR:DBBPR08MB4555; H:64aa7808-outbound-1.mta.getcheckrecipient.com; FPR:; SPF:TempError; LANG:en; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; MX:1; A:1; X-MS-Office365-Filtering-Correlation-Id-Prvs: 4b3e8ef2-741d-4487-0836-08d7508d508a NoDisclaimer: True X-Forefront-PRVS: 01901B3451 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: QT5WSzjm0golpIYwalUZNyugd8s+Ih/I8EwAYxX0NlagX3NVAT+lLWNjTDzaU44POiXXsu2nh7O4XpD9kAnIe9N/4G30i6tgE+OqZhxdRuDpgcj+SU+jiDsqZldCGWbJXtH9qOGlp0OwanvUi4/fFi60DuI1NzFRV7dxueN5dD64PwaKqLwx+Jmd8uq6Z5D3WsTtQw1007KvXJ/KC3uYWqZLPktMLom/2u+Gkyg+4Z8tWUQa7bs2AM1Vdkq8fekkofMpHIqMkqRmoZ8hkXLL2ZT+hqWDaKbbt1+9kc/Y+gVF85Bw3ETfgvW4tyW1/JKOQt0NYsGkNyd20UiloWMAxDZ6bbT/lduARtIg25r5MO0ZDrksm0uJE4m/qS7M+6tYnwwVRUwLZSBgXW/ptIhOsnCs/xmK7o72Py6x0VezvAQ= X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Oct 2019 10:00:25.7223 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: fdcd8e19-1666-419f-ebfc-08d7508d55f5 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR08MB4555 Subject: Re: [dpdk-stable] [PATCH v2 1/2] lib/distributor: fix deadlock issue for aarch64 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Sender: "stable" > -----Original Message----- > From: Honnappa Nagarahalli > Sent: Sunday, October 13, 2019 10:32 > To: Ruifeng Wang (Arm Technology China) ; > david.hunt@intel.com > Cc: dev@dpdk.org; hkalra@marvell.com; Gavin Hu (Arm Technology China) > ; nd ; Ruifeng Wang (Arm Technology > China) ; stable@dpdk.org; Honnappa Nagarahalli > ; nd > Subject: RE: [PATCH v2 1/2] lib/distributor: fix deadlock issue for aarch= 64 >=20 > Hi Ruifeng, > Typically, we have followed the convention of adding comments > whenever C11 atomic APIs are used. Can you please add comments > indicating why acquire or release semantics are used? >=20 OK. Comments will be added to explain acquire/release semantics used. > > -----Original Message----- > > From: Ruifeng Wang > > Sent: Friday, October 11, 2019 9:44 PM > > To: david.hunt@intel.com > > Cc: dev@dpdk.org; hkalra@marvell.com; Gavin Hu (Arm Technology China) > > ; Honnappa Nagarahalli > > ; nd ; Ruifeng Wang > (Arm > > Technology China) ; stable@dpdk.org > > Subject: [PATCH v2 1/2] lib/distributor: fix deadlock issue for > > aarch64 > > > > Distributor and worker threads rely on data structs in cache line for > > synchronization. The shared data structs were not protected. > > This caused deadlock issue on weaker memory ordering platforms as > aarch64. > > Fix this issue by adding memory barriers to ensure synchronization > > among cores. > > > > Bugzilla ID: 342 > > Fixes: 775003ad2f96 ("distributor: add new burst-capable library") > > Cc: stable@dpdk.org > > > > Signed-off-by: Ruifeng Wang > > Reviewed-by: Gavin Hu > > --- > > lib/librte_distributor/meson.build | 5 ++ > > lib/librte_distributor/rte_distributor.c | 39 ++++++++++------ > > lib/librte_distributor/rte_distributor_v20.c | 49 > > +++++++++++++------- > > 3 files changed, 63 insertions(+), 30 deletions(-) > > > > diff --git a/lib/librte_distributor/meson.build > > b/lib/librte_distributor/meson.build > > index dba7e3b2a..26577dbc1 100644 > > --- a/lib/librte_distributor/meson.build > > +++ b/lib/librte_distributor/meson.build > > @@ -9,3 +9,8 @@ else > > endif > > headers =3D files('rte_distributor.h') > > deps +=3D ['mbuf'] > > + > > +# for clang 32-bit compiles we need libatomic for 64-bit atomic ops > > +if > > +cc.get_id() =3D=3D 'clang' and dpdk_conf.get('RTE_ARCH_64') =3D=3D fal= se > > + ext_deps +=3D cc.find_library('atomic') endif > > diff --git a/lib/librte_distributor/rte_distributor.c > > b/lib/librte_distributor/rte_distributor.c > > index 21eb1fb0a..b653146d0 100644 > > --- a/lib/librte_distributor/rte_distributor.c > > +++ b/lib/librte_distributor/rte_distributor.c > > @@ -50,7 +50,8 @@ rte_distributor_request_pkt_v1705(struct > > rte_distributor *d, > > > > retptr64 =3D &(buf->retptr64[0]); > > /* Spin while handshake bits are set (scheduler clears it) */ > > - while (unlikely(*retptr64 & RTE_DISTRIB_GET_BUF)) { > > + while (unlikely(__atomic_load_n(retptr64, __ATOMIC_ACQUIRE) > > + & RTE_DISTRIB_GET_BUF)) { > > rte_pause(); > > uint64_t t =3D rte_rdtsc()+100; > > > > @@ -76,7 +77,8 @@ rte_distributor_request_pkt_v1705(struct > > rte_distributor *d, > > * Finally, set the GET_BUF to signal to distributor that cache > > * line is ready for processing > > */ > > - *retptr64 |=3D RTE_DISTRIB_GET_BUF; > > + __atomic_store_n(retptr64, *retptr64 | RTE_DISTRIB_GET_BUF, > > + __ATOMIC_RELEASE); > > } > > BIND_DEFAULT_SYMBOL(rte_distributor_request_pkt, _v1705, 17.05); > > MAP_STATIC_SYMBOL(void rte_distributor_request_pkt(struct > > rte_distributor *d, @@ -99,7 +101,8 @@ > > rte_distributor_poll_pkt_v1705(struct > > rte_distributor *d, > > } > > > > /* If bit is set, return */ > > - if (buf->bufptr64[0] & RTE_DISTRIB_GET_BUF) > > + if (__atomic_load_n(&(buf->bufptr64[0]), __ATOMIC_ACQUIRE) > > + & RTE_DISTRIB_GET_BUF) > > return -1; > > > > /* since bufptr64 is signed, this should be an arithmetic shift */ > > @@ - > > 115,7 +118,8 @@ rte_distributor_poll_pkt_v1705(struct rte_distributor *= d, > > * mbuf pointers, so toggle the bit so scheduler can start working > > * on the next cacheline while we're working. > > */ > > - buf->bufptr64[0] |=3D RTE_DISTRIB_GET_BUF; > > + __atomic_store_n(&(buf->bufptr64[0]), > > + buf->bufptr64[0] | RTE_DISTRIB_GET_BUF, > > __ATOMIC_RELEASE); > > > > return count; > > } > > @@ -174,6 +178,7 @@ rte_distributor_return_pkt_v1705(struct > > rte_distributor *d, > > return -EINVAL; > > } > > > > + __atomic_thread_fence(__ATOMIC_ACQUIRE); > > for (i =3D 0; i < RTE_DIST_BURST_SIZE; i++) > > /* Switch off the return bit first */ > > buf->retptr64[i] &=3D ~RTE_DISTRIB_RETURN_BUF; @@ -183,7 > > +188,8 @@ rte_distributor_return_pkt_v1705(struct rte_distributor *d, > > RTE_DISTRIB_FLAG_BITS) | > > RTE_DISTRIB_RETURN_BUF; > > > > /* set the GET_BUF but even if we got no returns */ > > - buf->retptr64[0] |=3D RTE_DISTRIB_GET_BUF; > > + __atomic_store_n(&(buf->retptr64[0]), > > + buf->retptr64[0] | RTE_DISTRIB_GET_BUF, > > __ATOMIC_RELEASE); > > > > return 0; > > } > > @@ -273,7 +279,8 @@ handle_returns(struct rte_distributor *d, unsigned > > int wkr) > > unsigned int count =3D 0; > > unsigned int i; > > > > - if (buf->retptr64[0] & RTE_DISTRIB_GET_BUF) { > > + if (__atomic_load_n(&(buf->retptr64[0]), __ATOMIC_ACQUIRE) > > + & RTE_DISTRIB_GET_BUF) { > > for (i =3D 0; i < RTE_DIST_BURST_SIZE; i++) { > > if (buf->retptr64[i] & RTE_DISTRIB_RETURN_BUF) { > > oldbuf =3D ((uintptr_t)(buf->retptr64[i] >> @@ > > -287,7 +294,7 @@ handle_returns(struct rte_distributor *d, unsigned int > wkr) > > d->returns.start =3D ret_start; > > d->returns.count =3D ret_count; > > /* Clear for the worker to populate with more returns */ > > - buf->retptr64[0] =3D 0; > > + __atomic_store_n(&(buf->retptr64[0]), 0, > > __ATOMIC_RELEASE); > > } > > return count; > > } > > @@ -307,7 +314,8 @@ release(struct rte_distributor *d, unsigned int wkr= ) > > struct rte_distributor_buffer *buf =3D &(d->bufs[wkr]); > > unsigned int i; > > > > - while (!(d->bufs[wkr].bufptr64[0] & RTE_DISTRIB_GET_BUF)) > > + while (!(__atomic_load_n(&(d->bufs[wkr].bufptr64[0]), > > __ATOMIC_ACQUIRE) > > + & RTE_DISTRIB_GET_BUF)) > > rte_pause(); > > > > handle_returns(d, wkr); > > @@ -328,7 +336,8 @@ release(struct rte_distributor *d, unsigned int wkr= ) > > d->backlog[wkr].count =3D 0; > > > > /* Clear the GET bit */ > > - buf->bufptr64[0] &=3D ~RTE_DISTRIB_GET_BUF; > > + __atomic_store_n(&(buf->bufptr64[0]), > > + buf->bufptr64[0] & ~RTE_DISTRIB_GET_BUF, > > __ATOMIC_RELEASE); > > return buf->count; > > > > } > > @@ -355,7 +364,8 @@ rte_distributor_process_v1705(struct > > rte_distributor *d, > > if (unlikely(num_mbufs =3D=3D 0)) { > > /* Flush out all non-full cache-lines to workers. */ > > for (wid =3D 0 ; wid < d->num_workers; wid++) { > > - if (d->bufs[wid].bufptr64[0] & > RTE_DISTRIB_GET_BUF) > > { > > + if (__atomic_load_n(&(d->bufs[wid].bufptr64[0]), > > + __ATOMIC_ACQUIRE) & > > RTE_DISTRIB_GET_BUF) { > > release(d, wid); > > handle_returns(d, wid); > > } > > @@ -367,7 +377,8 @@ rte_distributor_process_v1705(struct > > rte_distributor *d, > > uint16_t matches[RTE_DIST_BURST_SIZE]; > > unsigned int pkts; > > > > - if (d->bufs[wkr].bufptr64[0] & RTE_DISTRIB_GET_BUF) > > + if (__atomic_load_n(&(d->bufs[wkr].bufptr64[0]), > > + __ATOMIC_ACQUIRE) & RTE_DISTRIB_GET_BUF) > > d->bufs[wkr].count =3D 0; > > > > if ((num_mbufs - next_idx) < RTE_DIST_BURST_SIZE) @@ - > > 465,7 +476,8 @@ rte_distributor_process_v1705(struct rte_distributor > > *d, > > > > /* Flush out all non-full cache-lines to workers. */ > > for (wid =3D 0 ; wid < d->num_workers; wid++) > > - if ((d->bufs[wid].bufptr64[0] & RTE_DISTRIB_GET_BUF)) > > + if ((__atomic_load_n(&(d->bufs[wid].bufptr64[0]), > > + __ATOMIC_ACQUIRE) & RTE_DISTRIB_GET_BUF)) > > release(d, wid); > > > > return num_mbufs; > > @@ -574,7 +586,8 @@ rte_distributor_clear_returns_v1705(struct > > rte_distributor *d) > > > > /* throw away returns, so workers can exit */ > > for (wkr =3D 0; wkr < d->num_workers; wkr++) > > - d->bufs[wkr].retptr64[0] =3D 0; > > + __atomic_store_n(&(d->bufs[wkr].retptr64[0]), 0, > > + __ATOMIC_RELEASE); > > } > > BIND_DEFAULT_SYMBOL(rte_distributor_clear_returns, _v1705, 17.05); > > MAP_STATIC_SYMBOL(void rte_distributor_clear_returns(struct > > rte_distributor *d), diff --git > > a/lib/librte_distributor/rte_distributor_v20.c > > b/lib/librte_distributor/rte_distributor_v20.c > > index cdc0969a8..41411e3c1 100644 > > --- a/lib/librte_distributor/rte_distributor_v20.c > > +++ b/lib/librte_distributor/rte_distributor_v20.c > > @@ -34,9 +34,10 @@ rte_distributor_request_pkt_v20(struct > > rte_distributor_v20 *d, > > union rte_distributor_buffer_v20 *buf =3D &d->bufs[worker_id]; > > int64_t req =3D (((int64_t)(uintptr_t)oldpkt) << > RTE_DISTRIB_FLAG_BITS) > > | RTE_DISTRIB_GET_BUF; > > - while (unlikely(buf->bufptr64 & RTE_DISTRIB_FLAGS_MASK)) > > + while (unlikely(__atomic_load_n(&(buf->bufptr64), > > __ATOMIC_ACQUIRE) > > + & RTE_DISTRIB_FLAGS_MASK)) > > rte_pause(); > > - buf->bufptr64 =3D req; > > + __atomic_store_n(&(buf->bufptr64), req, __ATOMIC_RELEASE); > > } > > VERSION_SYMBOL(rte_distributor_request_pkt, _v20, 2.0); > > > > @@ -45,7 +46,8 @@ rte_distributor_poll_pkt_v20(struct > > rte_distributor_v20 *d, > > unsigned worker_id) > > { > > union rte_distributor_buffer_v20 *buf =3D &d->bufs[worker_id]; > > - if (buf->bufptr64 & RTE_DISTRIB_GET_BUF) > > + if (__atomic_load_n(&(buf->bufptr64), __ATOMIC_ACQUIRE) > > + & RTE_DISTRIB_GET_BUF) > > return NULL; > > > > /* since bufptr64 is signed, this should be an arithmetic shift */ > > @@ - > > 73,7 +75,7 @@ rte_distributor_return_pkt_v20(struct rte_distributor_v20 > *d, > > union rte_distributor_buffer_v20 *buf =3D &d->bufs[worker_id]; > > uint64_t req =3D (((int64_t)(uintptr_t)oldpkt) << > > RTE_DISTRIB_FLAG_BITS) > > | RTE_DISTRIB_RETURN_BUF; > > - buf->bufptr64 =3D req; > > + __atomic_store_n(&(buf->bufptr64), req, __ATOMIC_RELEASE); > > return 0; > > } > > VERSION_SYMBOL(rte_distributor_return_pkt, _v20, 2.0); @@ -117,7 > > +119,7 @@ handle_worker_shutdown(struct rte_distributor_v20 *d, > > unsigned int > > wkr) { > > d->in_flight_tags[wkr] =3D 0; > > d->in_flight_bitmask &=3D ~(1UL << wkr); > > - d->bufs[wkr].bufptr64 =3D 0; > > + __atomic_store_n(&(d->bufs[wkr].bufptr64), 0, > __ATOMIC_RELEASE); > > if (unlikely(d->backlog[wkr].count !=3D 0)) { > > /* On return of a packet, we need to move the > > * queued packets for this core elsewhere. > > @@ -165,18 +167,23 @@ process_returns(struct rte_distributor_v20 *d) > > const int64_t data =3D d->bufs[wkr].bufptr64; > > uintptr_t oldbuf =3D 0; > > > > - if (data & RTE_DISTRIB_GET_BUF) { > > + if (__atomic_load_n(&data, __ATOMIC_ACQUIRE) > > + & RTE_DISTRIB_GET_BUF) { > > flushed++; > > if (d->backlog[wkr].count) > > - d->bufs[wkr].bufptr64 =3D > > - backlog_pop(&d- > > >backlog[wkr]); > > + __atomic_store_n(&(d->bufs[wkr].bufptr64), > > + backlog_pop(&d->backlog[wkr]), > > + __ATOMIC_RELEASE); > > else { > > - d->bufs[wkr].bufptr64 =3D > > RTE_DISTRIB_GET_BUF; > > + __atomic_store_n(&(d->bufs[wkr].bufptr64), > > + RTE_DISTRIB_GET_BUF, > > + __ATOMIC_RELEASE); > > d->in_flight_tags[wkr] =3D 0; > > d->in_flight_bitmask &=3D ~(1UL << wkr); > > } > > oldbuf =3D data >> RTE_DISTRIB_FLAG_BITS; > > - } else if (data & RTE_DISTRIB_RETURN_BUF) { > > + } else if (__atomic_load_n(&data, __ATOMIC_ACQUIRE) > > + & RTE_DISTRIB_RETURN_BUF) { > > handle_worker_shutdown(d, wkr); > > oldbuf =3D data >> RTE_DISTRIB_FLAG_BITS; > > } > > @@ -251,21 +258,26 @@ rte_distributor_process_v20(struct > > rte_distributor_v20 *d, > > } > > } > > > > - if ((data & RTE_DISTRIB_GET_BUF) && > > + if ((__atomic_load_n(&data, __ATOMIC_ACQUIRE) > > + & RTE_DISTRIB_GET_BUF) && > > (d->backlog[wkr].count || next_mb)) { > > > > if (d->backlog[wkr].count) > > - d->bufs[wkr].bufptr64 =3D > > - backlog_pop(&d- > > >backlog[wkr]); > > + __atomic_store_n(&(d->bufs[wkr].bufptr64), > > + backlog_pop(&d- > > >backlog[wkr]), > > + __ATOMIC_RELEASE); > > > > else { > > - d->bufs[wkr].bufptr64 =3D next_value; > > + __atomic_store_n(&(d->bufs[wkr].bufptr64), > > + next_value, > > + __ATOMIC_RELEASE); > > d->in_flight_tags[wkr] =3D new_tag; > > d->in_flight_bitmask |=3D (1UL << wkr); > > next_mb =3D NULL; > > } > > oldbuf =3D data >> RTE_DISTRIB_FLAG_BITS; > > - } else if (data & RTE_DISTRIB_RETURN_BUF) { > > + } else if (__atomic_load_n(&data, __ATOMIC_ACQUIRE) > > + & RTE_DISTRIB_RETURN_BUF) { > > handle_worker_shutdown(d, wkr); > > oldbuf =3D data >> RTE_DISTRIB_FLAG_BITS; > > } > > @@ -280,13 +292,16 @@ rte_distributor_process_v20(struct > > rte_distributor_v20 *d, > > * if they are ready */ > > for (wkr =3D 0; wkr < d->num_workers; wkr++) > > if (d->backlog[wkr].count && > > - (d->bufs[wkr].bufptr64 & > > RTE_DISTRIB_GET_BUF)) { > > + (__atomic_load_n(&(d->bufs[wkr].bufptr64), > > + __ATOMIC_ACQUIRE) & > > RTE_DISTRIB_GET_BUF)) { > > > > int64_t oldbuf =3D d->bufs[wkr].bufptr64 >> > > RTE_DISTRIB_FLAG_BITS; > > store_return(oldbuf, d, &ret_start, &ret_count); > > > > - d->bufs[wkr].bufptr64 =3D backlog_pop(&d- > > >backlog[wkr]); > > + __atomic_store_n(&(d->bufs[wkr].bufptr64), > > + backlog_pop(&d->backlog[wkr]), > > + __ATOMIC_RELEASE); > > } > > > > d->returns.start =3D ret_start; > > -- > > 2.17.1