From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 02C7CA0528; Sat, 11 Jul 2020 01:48:07 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id C712A1DA22; Sat, 11 Jul 2020 01:48:06 +0200 (CEST) Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2076.outbound.protection.outlook.com [40.107.21.76]) by dpdk.org (Postfix) with ESMTP id 275BB1D8EF for ; Sat, 11 Jul 2020 01:48:04 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8ZqpwAj0jsTnOZGsAD0SFMX6tTzj8BYyN2jj68RtehI=; b=K/dXo8QbT+YlV4HyFh5O1XYQHtGJbUOWVhNrqHHARUp23j5OKoo5y0VMEFW00uAYgVQ3DH1UIz937JTEEkhO7zwxGyrd5G9DkbCPMMFOOZemxBoxp0rUVjbyvQPJ5tMNZhP4xOWJ1o25yW8i2LcPhldFdUZI5AXgNzB4YiD2tLw= Received: from AM5PR0601CA0069.eurprd06.prod.outlook.com (2603:10a6:206::34) by DBBPR08MB4853.eurprd08.prod.outlook.com (2603:10a6:10:d5::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3174.21; Fri, 10 Jul 2020 23:48:02 +0000 Received: from AM5EUR03FT027.eop-EUR03.prod.protection.outlook.com (2603:10a6:206:0:cafe::f) by AM5PR0601CA0069.outlook.office365.com (2603:10a6:206::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3174.21 via Frontend Transport; Fri, 10 Jul 2020 23:48:02 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=bestguesspass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM5EUR03FT027.mail.protection.outlook.com (10.152.16.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3174.21 via Frontend Transport; Fri, 10 Jul 2020 23:48:02 +0000 Received: ("Tessian outbound 1dc58800d5dd:v62"); Fri, 10 Jul 2020 23:48:01 +0000 X-CR-MTA-TID: 64aa7808 Received: from ba567a206739.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 2FDCB403-BF92-4618-8099-E52A779E0D25.1; Fri, 10 Jul 2020 23:47:56 +0000 Received: from EUR04-DB3-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id ba567a206739.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Fri, 10 Jul 2020 23:47:56 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nNV9F5LlbsTA9GSq9gwY9HTc3jegG1csEY3rH5wpoRoylLqGChtSeK7QWwI6/2QGmFUUXwdmXXsdGRW6fKNhbBaKdEjPVBH12KJn9EHPB0ehjAnhsZXA6hKj7lWkskt6fqfK35AR3sQl6hVgZDGDcC4Nyic/vQVRHNmycHqFdPGJjMNkDRW9Pv1T9uAYBpuI9Cs4cLinxGmesiT4MzJnv/NJ8Oxfvd5GunK2ZFeMgXIpjdgLsvxC2btHeoQROs61SFH/lqoyOGsWmQ30BUog0exhuz8uY2DuMYMr28gy9qFmfdD+DCzIDnn3COT7n0OLGF8GcKKKlX1FBZ2RhjDDng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8ZqpwAj0jsTnOZGsAD0SFMX6tTzj8BYyN2jj68RtehI=; b=oEE2/FhBwvLNJAwFwiXMNOrMmRh+Pxd3rBEavNea8DC8Ujk8lzPKVHsELvZlvq4cfMLVojORrE9DcyrRPM7fm2EYqtdqPbUNqo4aYsVPI6a+vnQaHJENxvCVi42YpY4w3gPVSkQ/c7LMIwGaB0b8SbHE7I8EvRq8LV0gQpx1j3lfeudS73EZH8xSc6tghhH4GArW4qWShuoP0rrzc813v7wEAqjSeLUA3Ie03gqqfIqK4bp+R86YbIKz4D/Mc1lhzC00CyWJCAmm2HRhDjWDtvuusvROE6YJPQXce1kxJZ6v8jLfOQBMZSsr8JSrAFH+pNQe+rf6MTFdz2XjQpEvtQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8ZqpwAj0jsTnOZGsAD0SFMX6tTzj8BYyN2jj68RtehI=; b=K/dXo8QbT+YlV4HyFh5O1XYQHtGJbUOWVhNrqHHARUp23j5OKoo5y0VMEFW00uAYgVQ3DH1UIz937JTEEkhO7zwxGyrd5G9DkbCPMMFOOZemxBoxp0rUVjbyvQPJ5tMNZhP4xOWJ1o25yW8i2LcPhldFdUZI5AXgNzB4YiD2tLw= Received: from DB6PR0802MB2216.eurprd08.prod.outlook.com (2603:10a6:4:85::9) by DB7PR08MB3131.eurprd08.prod.outlook.com (2603:10a6:5:1e::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3174.21; Fri, 10 Jul 2020 23:47:55 +0000 Received: from DB6PR0802MB2216.eurprd08.prod.outlook.com ([fe80::9d1d:207b:e89d:199d]) by DB6PR0802MB2216.eurprd08.prod.outlook.com ([fe80::9d1d:207b:e89d:199d%10]) with mapi id 15.20.3174.024; Fri, 10 Jul 2020 23:47:55 +0000 From: Honnappa Nagarahalli To: "thomas@monjalon.net" , Phil Yang CC: "dev@dpdk.org" , "david.marchand@redhat.com" , "drc@linux.vnet.ibm.com" , "jerinj@marvell.com" , "konstantin.ananyev@intel.com" , Ola Liljedahl , Ruifeng Wang , nd , "john.mcnamara@intel.com" , "bruce.richardson@intel.com" , Honnappa Nagarahalli , nd Thread-Topic: [dpdk-dev] [PATCH v6 1/4] doc: add generic atomic deprecation section Thread-Index: AQHWVxSIhq5cz44f0Uahpu9qDTM52Q== Date: Fri, 10 Jul 2020 23:47:55 +0000 Message-ID: References: <1590483667-10318-1-git-send-email-phil.yang@arm.com> <1594115449-13750-1-git-send-email-phil.yang@arm.com> <1594115449-13750-2-git-send-email-phil.yang@arm.com> <1746729.2ERU3Mzd2g@thomas> In-Reply-To: <1746729.2ERU3Mzd2g@thomas> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: b0fe359f-ddb5-4428-b3ea-b11606780f1e.0 x-checkrecipientchecked: true Authentication-Results-Original: monjalon.net; dkim=none (message not signed) header.d=none; monjalon.net; dmarc=none action=none header.from=arm.com; x-originating-ip: [70.112.90.121] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 6b81f016-437e-4500-6d90-08d8252baee0 x-ms-traffictypediagnostic: DB7PR08MB3131:|DBBPR08MB4853: x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:10000;OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: FAY8RYP6jt6WFGVMAmBZIc4upOYJRURLhtA2/SeAox6t74wUjNB3XQEC8SOx1Wid8VyvcduBbET9AELdknNTPH7/7rhzTI5Q3xXXwuR9tzYNfxVmWrRbIp2Eue7xCZ+PEhgsBDwV08KStDHNRo4y6zgDdmNB4nfGRUNYZD7JJVruaybuhJ9Lca2aUwvHKEn8gijzZiKRaWUqRUZ2qLmmo1sw7J3iNfebIuR+PNrWALLkFGEfQrbmb4f93bNFR9He40/JjA/41wAg6k/DsKQZVzbp4tVJGlxi8LWlGbYeztvtUDr6f7r+uwhdgWy1nvTdSKMovdByzjp8fr3TIUNbXW72XnxWDR/Kbr4Gg2ne2bvS0eDxKBTXMOHPoBD556KskAaEK1DoC3Ye3u+HjUrBxH9zawXqbUxw3sRbILmtVYU/5+rNp2uVvWyLNUf1XB4H X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DB6PR0802MB2216.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(39860400002)(376002)(136003)(366004)(346002)(396003)(110136005)(54906003)(86362001)(83380400001)(9686003)(6636002)(2906002)(316002)(5660300002)(8676002)(8936002)(478600001)(33656002)(55016002)(52536014)(7696005)(6506007)(4326008)(26005)(71200400001)(186003)(66556008)(66446008)(66946007)(66476007)(76116006)(64756008)(41533002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: I2gwQIkz7awlrQJvMgi8QGSZ2uHTVSpdmLwtgODC52Ju0nbo/YjsdKiZFIaNpgplSaM7AEhyogMMtwbYgLmJHxfE0YJgDRw8owAyz/pyfw1PR0DXp0mkd4+m0Fc95YXpILzHiLXPiEfNFehX79sL0r8tzn+KxETBq9wbfpkytGgBv5c7lWuF8vLL4/0m3OEcdv1Ys9q84tW4NqQFgsXHB4Sw6c6UVt4hVrJ8X/0IdPEUtyv3cBxVmbgQoStvTsBNssHqExNPX9utKV14kEO4cqslX2xPKUpFCAX2UMPvhpIbvTu81RQ5kxqjvyeKZLW3nUv+QaZBSFtWZGbfubhMa1imjcDFCCBKhl4Cd5zqza6nQni7AbXb09PMOxC+cn6Uw+S1gPb+m/ICNoqlISGokRMfJ/UfGfCf7SbYgjHPvIl7FnLyk/7S8sOfY0GSmsSapnQ7qNIxaWdeaKE1905XD14ewB/iJlPxOTRllo8bMIS1hO9+H97F7KJvOgcBTiHH Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB7PR08MB3131 Original-Authentication-Results: monjalon.net; dkim=none (message not signed) header.d=none; monjalon.net; dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM5EUR03FT027.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFTY:; SFS:(4636009)(39860400002)(346002)(376002)(136003)(396003)(46966005)(336012)(70586007)(5660300002)(52536014)(70206006)(6636002)(47076004)(81166007)(26005)(186003)(478600001)(55016002)(9686003)(356005)(110136005)(83380400001)(6506007)(4326008)(82740400003)(7696005)(316002)(36906005)(86362001)(33656002)(82310400002)(54906003)(2906002)(8676002)(8936002)(41533002); DIR:OUT; SFP:1101; X-MS-Office365-Filtering-Correlation-Id-Prvs: 3b669162-a190-48a8-f593-08d8252bab36 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Tgy+bbGsSGxlFIO0aMDILSX4nC0AQI7icjMzsfWg28P+ZZFi/2FfhegJB+7+ecijogDVI/Se5WV7HkMTOlHZzpvreNi2QTJc9wTr+PyaSqaSV1XITu+mwWZ2hgCrYSUhSnkfXHtrULqvec3A2ek4n97B1K4ety7UAA4BrnYKYXElFEDoFNIJCvtHs4oCR8JTWsdOI2ipHgRginWi6zcHuQt75DvzOX8pSyfe0DHuPyN6Xs/0+XZRve09GujP/q8u7pGnaMNfwGdc1WwgIb+ASL4yUcgpJN1ktaMObbzKJJEKDpiCZoojtrqMsCXYw7CtNoOM/BoqqcNxGY7Rf5KsyjvSOMcABaoJtD/eSQB/I1nge/VvMqZaEElfFDM4ehaCkg91sm27ftRZrtCSIgrWaI5ZStDhYySR5Vhax6Npl1NqAK2nywllPmjVDG6cGI3gHSJYvmtf3GjuR53PWm1kN9p+ifccCpNHnmNc2hlCjbzb5UKtZShiBA5j5DzPsXFJ X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jul 2020 23:48:02.0150 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6b81f016-437e-4500-6d90-08d8252baee0 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM5EUR03FT027.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR08MB4853 Subject: Re: [dpdk-dev] [PATCH v6 1/4] doc: add generic atomic deprecation section X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > Subject: Re: [dpdk-dev] [PATCH v6 1/4] doc: add generic atomic deprecatio= n > section >=20 > Interestingly, John, our doc maintainer is not Cc'ed. > I add him. > Please use --cc-cmd devtools/get-maintainer.sh I am expecting a review fr= om > an x86 maintainer as well. > If no maintainer replies, ping them. >=20 > 07/07/2020 11:50, Phil Yang: > > Add deprecating the generic rte_atomic_xx APIs to c11 atomic built-ins > > guide and examples. > [...] > > +Atomic Operations: Use C11 Atomic Built-ins > > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > + > > +DPDK `generic rte_atomic > > + > +de/generic/rte_atomic.h>`_ operations are >=20 > Why this github link on 20.02? >=20 > Please try to keep lines small. >=20 > > +implemented by `__sync built-ins > `_. >=20 > Long links should be on their own line to avoid long lines. >=20 > > +These __sync built-ins result in full barriers on aarch64, which are > > +unnecessary in many use cases. They can be replaced by `__atomic > > +built-ins Builtins.html>`_ that conform to the C11 memory model and provide finer > memory order control. > > + > > +So replacing the rte_atomic operations with __atomic built-ins might > > +improve performance for aarch64 machines. `More details > content/uploads/sites/35/2019/10/StateofC11Code.pdf>`_. >=20 > "More details." > Please make a sentence. The full stop is after the link. But, I think we will remove this link as w= ell. >=20 > > + > > +Some typical optimization cases are listed below: > > + > > +Atomicity > > +^^^^^^^^^ > > + > > +Some use cases require atomicity alone, the ordering of the memory > > +operations does not matter. For example the packets statistics in the > `vhost > ` > _ example application. >=20 > Again github. > If you really want a web link, use code.dpdk.org or doc.dpdk.org/api >=20 > But why giving code example at all? >=20 > > + > > +It just updates the number of transmitted packets, no subsequent > > +logic depends on these counters. So the RELAXED memory ordering is > sufficient: > > + > > +.. code-block:: c > > + > > + static __rte_always_inline void > > + virtio_xmit(struct vhost_dev *dst_vdev, struct vhost_dev *src_vdev= , > > + struct rte_mbuf *m) > > + { > > + ... > > + ... > > + if (enable_stats) { > > + __atomic_add_fetch(&dst_vdev->stats.rx_total_atomic, 1, > __ATOMIC_RELAXED); > > + __atomic_add_fetch(&dst_vdev->stats.rx_atomic, ret, > __ATOMIC_RELAXED); > > + ... > > + } > > + } >=20 > I don't see how adding real code helps here. > Why not just mentioning __atomic_add_fetch and __ATOMIC_RELAXED? >=20 > > + > > +One-way Barrier > > +^^^^^^^^^^^^^^^ > > + > > +Some use cases allow for memory reordering in one way while requiring > > +memory ordering in the other direction. > > + > > +For example, the memory operations before the `lock > > + > +de/generic/rte_spinlock.h#L66>`_ can move to the critical section, > > +but the memory operations in the critical section cannot move above > > +the lock. In this case, the full memory barrier in the CAS operation > > +can be replaced to ACQUIRE. On the other hand, the memory operations > > +after the `unlock > generic/rte_spinlock.h#L88>`_ can move to the critical section, but the > memory operations in the critical section cannot move below the unlock. S= o > the full barrier in the STORE operation can be replaced with RELEASE. >=20 > Again github links instead of our doxygen. >=20 > > + > > +Reader-Writer Concurrency > > +^^^^^^^^^^^^^^^^^^^^^^^^^ >=20 > No blank line here? Will fix >=20 > > +Lock-free reader-writer concurrency is one of the common use cases in > DPDK. > > + > > +The payload or the data that the writer wants to communicate to the > > +reader, can be written with RELAXED memory order. However, the guard > > +variable should be written with RELEASE memory order. This ensures > > +that the store to guard variable is observable only after the store to > payload is observable. > > +Refer to `rte_hash insert > h.c#L737>`_ for an example. >=20 > Hum... >=20 > > + > > +.. code-block:: c > > + > > + static inline int32_t > > + rte_hash_cuckoo_insert_mw(const struct rte_hash *h, > > + ... > > + int32_t *ret_val) > > + { > > + ... > > + ... > > + > > + /* Insert new entry if there is room in the primary > > + * bucket. > > + */ > > + for (i =3D 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { > > + /* Check if slot is available */ > > + if (likely(prim_bkt->key_idx[i] =3D=3D EMPTY_SLOT)) { > > + prim_bkt->sig_current[i] =3D sig; > > + /* Store to signature and key should not > > + * leak after the store to key_idx. i.e. > > + * key_idx is the guard variable for signature > > + * and key. > > + */ > > + __atomic_store_n(&prim_bkt->key_idx[i], > > + new_idx, > > + __ATOMIC_RELEASE); > > + break; > > + } > > + } > > + > > + ... > > + } > > + > > +Correspondingly, on the reader side, the guard variable should be > > +read with ACQUIRE memory order. The payload or the data the writer > > +communicated, can be read with RELAXED memory order. This ensures > > +that, if the store to guard variable is observable, the store to paylo= ad is > also observable. Refer to `rte_hash lookup > h.c#L1215>`_ for an example. > > + > > +.. code-block:: c > > + > > + static inline int32_t > > + search_one_bucket_lf(const struct rte_hash *h, const void *key, > uint16_t sig, > > + void **data, const struct rte_hash_bucket *bkt) > > + { > > + ... > > + > > + for (i =3D 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { > > + .... > > + if (bkt->sig_current[i] =3D=3D sig) { > > + key_idx =3D __atomic_load_n(&bkt->key_idx[i], > > + __ATOMIC_ACQUIRE); > > + if (key_idx !=3D EMPTY_SLOT) { > > + k =3D (struct rte_hash_key *) ((char *)keys + > > + key_idx * h->key_entry_size); > > + > > + if (rte_hash_cmp_eq(key, k->key, h) =3D=3D 0) { > > + if (data !=3D NULL) { > > + *data =3D __atomic_load_n(&k->pdata, > > + __ATOMIC_ACQUIRE); > > + } > > + > > + /* > > + * Return index where key is stored, > > + * subtracting the first dummy index > > + */ > > + return key_idx - 1; > > + } > > + ... > > + } > > + >=20 > NACK for the big chunks of real code. > Please use words and avoid code. >=20 > If you insist on keeping code in doc, I will make you responsible of upda= ting > all the code we have already in the doc :) Ok, understood, will re-spin. >=20