From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bon0088.outbound.protection.outlook.com [157.56.111.88]) by dpdk.org (Postfix) with ESMTP id AFB864A63 for ; Tue, 3 Nov 2015 17:53:45 +0100 (CET) Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.Jacob@caviumnetworks.com; Received: from localhost.localdomain (111.93.218.67) by BY2PR0701MB1976.namprd07.prod.outlook.com (10.163.155.22) with Microsoft SMTP Server (TLS) id 15.1.312.18; Tue, 3 Nov 2015 16:53:43 +0000 Date: Tue, 3 Nov 2015 22:23:21 +0530 From: Jerin Jacob To: "Ananyev, Konstantin" Message-ID: <20151103165318.GA19474@localhost.localdomain> References: <1446565921-18088-1-git-send-email-jerin.jacob@caviumnetworks.com> <2601191342CEEE43887BDE71AB97725836AB8F01@irsmsx105.ger.corp.intel.com> <20151103161834.GA18450@localhost.localdomain> <2601191342CEEE43887BDE71AB97725836AB8F55@irsmsx105.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <2601191342CEEE43887BDE71AB97725836AB8F55@irsmsx105.ger.corp.intel.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-Originating-IP: [111.93.218.67] X-ClientProxiedBy: BM1PR01CA0010.INDPRD01.PROD.OUTLOOK.COM (25.163.198.145) To BY2PR0701MB1976.namprd07.prod.outlook.com (25.163.155.22) X-Microsoft-Exchange-Diagnostics: 1; BY2PR0701MB1976; 2:43GB8zXdFyys/GB4ndEpgT94FN1YaDG9CnNu7ud6IPaNj8eqDF6Eoobf/A4Smxp2QY6SNZ4WQ/UWrimnt97c+WQ3TF1+CVt6/paxhEa61wIPpcHvRG4yYD6opZ+tO76Z4l5ygfFgur2kf3/iBBnyZmIMdftBALzXIt1vTeoK/XQ=; 3:ftnLJbX/BDxan49FB21yo4FcxGF4Pqz6RhLLsB9/6y3iPSOuvkjjDL45BjzYoZf3N4Wsd9dpq4CBmBaAFZtj2IJFvAq0UGnSMVIa6raL1rIW/aGjY09G1vglcAma1VLC7IhcFRF8qXgKoDk3Glf0cg==; 25:pUHAd9Sxlehak5hLFbIrPds2qNqei9sZn8h5XO06iLf7+JK7051v42CGBbdu4d27hBmSrdUysxZmuaPIPxRa2GFABeedqAFNC8M4I2UhGEZoD8JN7tLCMY2BjpxCsXVHzTidpcZ9oj4k4lnCGoHfKF9kMfe3lnezsgA2XnzaD8RNUS7m8+IKX0hd/PTI0oXlPh1oPFo8A2bHlkjDGDR12kVwPEMu29Y6upDEdpwp7PSBloS/dzstel7uanIytJSDWBLKgfvDhsGH9s/75eL36g== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BY2PR0701MB1976; X-Microsoft-Exchange-Diagnostics: 1; BY2PR0701MB1976; 20:4ORsuZbKsF85bwKtZkxqcDiQR6uu1fY9xD1RTd0l6/gQhMNpvFt0VVwQ5agtAvx4CBrl9oA+E9oDV1C+k/rZDCgtk7mLB/V1NeI3C7uugm0XzfvhFA4InokRmVWt7s6G3pQGne3HCiL46SgkfRLvYPsgt7elrht+4TPEwoInVT5H9DTXAjkeTkHewt8+fVGUCfRQX/VgaTQXB0opM4eUiMq5D2fiiNWesaG1o3nKSZo92rXo7kZ1BiWztYlIX94HeUnYr/TJXN4bY9UQeF7DFQ57dH5DuMOHmQFw49XrIF0MUX54tOCbcJfwkB92ughgyO01wc+PS71c7OGdBtLrQMW/79ax/DOxCzemGstZ/S+4pfWwqrdM9kcAQKrCu12la81qURAWUKVl9Mzl19xUOqyt4J/l1WI3ODuSC1sRWlb+1GQig2sfx10/QdoWURZg/H6IrSdUKJiWiXDn4n4+wIFWh8v9pY/26kgZwtx11+zmsZvAzoWk0h7IbKMY7U1jqtZho9uS6nFzuAUCX9rntf2gTuqjcaT5hmtDiISLByXea52JChyVDz8rnBhI+2FLJ1QcCz+ov1kXMaUa9e1Hy1EXzqkcd76brbHwg9teX0Y= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(236414709691187); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(5005006)(520078)(8121501046)(10201501046)(3002001); SRVR:BY2PR0701MB1976; BCL:0; PCL:0; RULEID:; SRVR:BY2PR0701MB1976; X-Microsoft-Exchange-Diagnostics: 1; BY2PR0701MB1976; 4:zc20k7MA+dWtGP1+rToHIVIFj08zuugC1t8mtY5iZ1v6ImZcRptnXXaqcp7KvaDTxdTUiK8AVVwfrCp38NCYm6O69A2VxEm+xcumNce2wuaM+nGlw+EMy0xseyotY2thF5z6ErSjgSk0mrTdjtU1bHnnTZn6zQKjGUJ/KKE6PkPnq0pVIcCsiDkwDkpho+HcotQkKj6GgH1qV2w9Wq811pjyENJ3RGecIhBfO0XCLcqyvc5jtSiTL2K1AWlbqjitslr7UNXeB1EH48UFLFfTfW5Ka7yccQtIVsrB0HI56byu3PX2LWkzTWMYQIWY4w2jQ0wuA+jGb4q8t1xd6MlbhTjcCxsP/8XPV0UkVeg0/z6nxYWU7HgO5GqaJ4B4Tl4f X-Forefront-PRVS: 0749DC2CE6 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6069001)(6009001)(24454002)(199003)(377454003)(13464003)(57704003)(189002)(40100003)(5008740100001)(50466002)(97756001)(47776003)(87976001)(46406003)(5004730100002)(15395725005)(93886004)(42186005)(92566002)(105586002)(106356001)(81156007)(33656002)(97736004)(4001350100001)(189998001)(61506002)(50986999)(110136002)(23726002)(122386002)(77096005)(83506001)(5009440100003)(5001960100002)(15975445007)(66066001)(86362001)(101416001)(19580395003)(54356999)(5007970100001)(76176999)(2950100001)(19580405001)(7099028); DIR:OUT; SFP:1101; SCL:1; SRVR:BY2PR0701MB1976; H:localhost.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: caviumnetworks.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BY2PR0701MB1976; 23:nuPjqcEAzdZvleCDoQxalsYzumKhfLSFcM1UdWu?= =?us-ascii?Q?9QKgtJ5brTKcIrtvJfVPEaou/cWkX7rfnwo5Vr2ICccVuZa7lD75jCWvjA3e?= =?us-ascii?Q?cE9roLwHXW8A3fPXKxZoYyUFJSiWMCNucdCw6pOkqSlE75E0OIyA2b0edAGL?= =?us-ascii?Q?odWz6YBuiFjjlGmFXPGYKPlGTuqa385vfqfz4NAMAwGp2+RmY/gih1F2dHTQ?= =?us-ascii?Q?5YkbADNlU6LgP49qAwg82s6K0sl1Qq9uwakukryUKzftB7k1t8h9+txi426T?= =?us-ascii?Q?2AKDvH0NIbItOYL8ezwN5Gh7ccKqspgynI+NgIzJXUDoTkxBqc0Lj+zCgBkP?= =?us-ascii?Q?dRsCb0Mp5vAR8yiebhdjsvtwJF+P7NwIa+Sp0t+jt3fMnxyHS8HHyQORgApF?= =?us-ascii?Q?1L2oQDmvkC1nvGbPUhwDrFBG3r13peIqaZbVlivi0G4cy64m+i2l/VjrC3Ro?= =?us-ascii?Q?UYYWnTMRwucz9d7amL8LSMoa+xnqDpGEQ4FhW1Q/FAaZWtyfh6ime8BckGW5?= =?us-ascii?Q?UAhWvxuW7vYSGy2R+o7bVUKcf/iC9wfj0FNi1tkqMIm1pVGlbQSNUKPPwhnz?= =?us-ascii?Q?ZT5wN/GHDxlRWVd2Jmn1CpfB4WtxhHxWjGQPhscc81PqHUvVoP0/m3wZuW95?= =?us-ascii?Q?Fyuq+Ls2jrBItI0qJurC0VmVaBzR0JAzAqSMIV4BHxp+LLq3rIzIh/JXQrg8?= =?us-ascii?Q?a1yxm2j/VWpLURWofqcnKu0cDG4vkn0HoLOhDKhcipLNSOwjwdvv5s2j7zCN?= =?us-ascii?Q?YO+2tcaT/EJH3OAWHmhuyS7ccjz3lhzerd3HUxIJVDXu0e79678RYWihHiCz?= =?us-ascii?Q?3rFcCMixp9BskohodfPXgMvv+EShYXIT6QG7FZnAAA5WUKMEE94ILqnOY3IQ?= =?us-ascii?Q?KDnPNIhiyNyNAxDnSr9jXHQBBuD/0X1ibdj2yiKX8KnHg1LuJjkaK/2EF9jM?= =?us-ascii?Q?EBU5aKHYbSMeygj1qJEWljiZZNT3tw08zGk2OByGZ0ESCxH47ILxlN7PGdv4?= =?us-ascii?Q?dw7bSUh6An/53Vq73atuI8q+1JlCADSjghXjJ/cxWj86fEfEREhKzMl2/XUD?= =?us-ascii?Q?j0l0aFNvU27n9hHyePbYp/BFX2FufA0PNNWTJmdNio14KuYe1OL9o1EvBNWi?= =?us-ascii?Q?XIJEv8jMZ1jXpHkJfluSDQ6yKRvqkJO915rKlH2hmUnpQ9ugJyWHVaGq/vO2?= =?us-ascii?Q?mxqHkTYoQ/WK3QKJklfLe4xGVJWqXX9p/RvP57+CiDkyoaPL6f5l9lUg/nTR?= =?us-ascii?Q?zNnYZyj5KlcjWtMX1Quc=3D?= X-Microsoft-Exchange-Diagnostics: 1; BY2PR0701MB1976; 5:H3kDCbH9XvSPTNB0db5N1ZC0Ie6CHGeFHOjBvGsgb3UUIoVzMFm2flSGwOA7BTkiXhH5aenSqeHsSddjAHa3uCE+CQTpsvpHgzrGm1jDXTPteozWnZih+fxj0eouEpgVbzGA/JwRk1l8HDn6f1vkSg==; 24:6AXQFSiruKWryuTVwYITrmMWDl5jCexh4vxJGVb25D5fNfKnv48A0ZwKM0VS+5FCvx7rKrdf9ItxWcUBEtouUvIldYixewX410ioKvjaHAg=; 20:X2KjO2SBwgg3n4nRta7c1548Cmx/f9b4onhBo2ETxYDDu8ezcZ6UeR+SwyXvRV2P57BWApKBdKE+RE8Ch1cxsw== SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2015 16:53:43.0858 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR0701MB1976 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] [RFC ][PATCH] Introduce RTE_ARCH_STRONGLY_ORDERED_MEM_OPS configuration parameter X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 16:53:46 -0000 On Tue, Nov 03, 2015 at 04:28:00PM +0000, Ananyev, Konstantin wrote: > > > > -----Original Message----- > > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com] > > Sent: Tuesday, November 03, 2015 4:19 PM > > To: Ananyev, Konstantin > > Cc: dev@dpdk.org > > Subject: Re: [dpdk-dev] [RFC ][PATCH] Introduce RTE_ARCH_STRONGLY_ORDERED_MEM_OPS configuration parameter > > > > On Tue, Nov 03, 2015 at 03:57:24PM +0000, Ananyev, Konstantin wrote: > > > > > > > > > > -----Original Message----- > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jerin Jacob > > > > Sent: Tuesday, November 03, 2015 3:52 PM > > > > To: dev@dpdk.org > > > > Subject: [dpdk-dev] [RFC ][PATCH] Introduce RTE_ARCH_STRONGLY_ORDERED_MEM_OPS configuration parameter > > > > > > > > rte_ring implementation needs explicit memory barrier > > > > in weakly ordered architecture like ARM unlike > > > > strongly ordered architecture like X86 > > > > > > > > Introducing RTE_ARCH_STRONGLY_ORDERED_MEM_OPS > > > > configuration to abstract such dependency so that other > > > > weakly ordered architectures can reuse this infrastructure. > > > > > > Looks a bit clumsy. > > > Please try to follow this suggestion instead: > > > http://dpdk.org/ml/archives/dev/2015-October/025505.html > > > > Make sense. Do we agree on a macro that is defined based upon > > RTE_ARCH_STRONGLY_ORDERED_MEM_OP to remove clumsy #ifdef every where ? > > Why do we need that macro at all? > Why just not have architecture specific macro as was discussed in that thread? > > So for intel somewhere inside > lib/librte_eal/common/include/arch/x86/rte_atomic.h > > it would be: > > #define rte_smp_wmb() rte_compiler_barrier() > > For arm inside lib/librte_eal/common/include/arch/x86/rte_atomic.h > > #define rte_smp_wmb() rte_wmb() I am not sure about the other architecture but in armv8 device memory (typically mapped through NIC PCIe BAR space) are strongly ordered. So there is one more dimension to the equation(normal memory or device memory). IMO rte_smp_wmb() -> rte_wmb() mapping to deal with device memory may not be correct on arm64 ? Thoughts ? > > And so on. > > I think it was already an attempt (not finished) to do similar stuff for ppc: > http://dpdk.org/dev/patchwork/patch/5884/ > > Konstantin > > > > > Jerin > > > > > > > > Konstantin > > > > > > > > > > > Signed-off-by: Jerin Jacob > > > > --- > > > > config/common_bsdapp | 5 +++++ > > > > config/common_linuxapp | 5 +++++ > > > > config/defconfig_arm64-armv8a-linuxapp-gcc | 1 + > > > > config/defconfig_arm64-thunderx-linuxapp-gcc | 1 + > > > > lib/librte_ring/rte_ring.h | 20 ++++++++++++++++++++ > > > > 5 files changed, 32 insertions(+) > > > > > > > > diff --git a/config/common_bsdapp b/config/common_bsdapp > > > > index b37dcf4..c8d1f63 100644 > > > > --- a/config/common_bsdapp > > > > +++ b/config/common_bsdapp > > > > @@ -79,6 +79,11 @@ CONFIG_RTE_FORCE_INTRINSICS=n > > > > CONFIG_RTE_ARCH_STRICT_ALIGN=n > > > > > > > > # > > > > +# Machine has strongly-ordered memory operations on normal memory like x86 > > > > +# > > > > +CONFIG_RTE_ARCH_STRONGLY_ORDERED_MEM_OPS=y > > > > + > > > > +# > > > > # Compile to share library > > > > # > > > > CONFIG_RTE_BUILD_SHARED_LIB=n > > > > diff --git a/config/common_linuxapp b/config/common_linuxapp > > > > index 0de43d5..d040a74 100644 > > > > --- a/config/common_linuxapp > > > > +++ b/config/common_linuxapp > > > > @@ -79,6 +79,11 @@ CONFIG_RTE_FORCE_INTRINSICS=n > > > > CONFIG_RTE_ARCH_STRICT_ALIGN=n > > > > > > > > # > > > > +# Machine has strongly-ordered memory operations on normal memory like x86 > > > > +# > > > > +CONFIG_RTE_ARCH_STRONGLY_ORDERED_MEM_OPS=y > > > > + > > > > +# > > > > # Compile to share library > > > > # > > > > CONFIG_RTE_BUILD_SHARED_LIB=n > > > > diff --git a/config/defconfig_arm64-armv8a-linuxapp-gcc b/config/defconfig_arm64-armv8a-linuxapp-gcc > > > > index 6ea38a5..5289152 100644 > > > > --- a/config/defconfig_arm64-armv8a-linuxapp-gcc > > > > +++ b/config/defconfig_arm64-armv8a-linuxapp-gcc > > > > @@ -37,6 +37,7 @@ CONFIG_RTE_ARCH="arm64" > > > > CONFIG_RTE_ARCH_ARM64=y > > > > CONFIG_RTE_ARCH_64=y > > > > CONFIG_RTE_ARCH_ARM_NEON=y > > > > +CONFIG_RTE_ARCH_STRONGLY_ORDERED_MEM_OPS=n > > > > > > > > CONFIG_RTE_FORCE_INTRINSICS=y > > > > > > > > diff --git a/config/defconfig_arm64-thunderx-linuxapp-gcc b/config/defconfig_arm64-thunderx-linuxapp-gcc > > > > index e8fccc7..79fa9e6 100644 > > > > --- a/config/defconfig_arm64-thunderx-linuxapp-gcc > > > > +++ b/config/defconfig_arm64-thunderx-linuxapp-gcc > > > > @@ -37,6 +37,7 @@ CONFIG_RTE_ARCH="arm64" > > > > CONFIG_RTE_ARCH_ARM64=y > > > > CONFIG_RTE_ARCH_64=y > > > > CONFIG_RTE_ARCH_ARM_NEON=y > > > > +CONFIG_RTE_ARCH_STRONGLY_ORDERED_MEM_OPS=n > > > > > > > > CONFIG_RTE_FORCE_INTRINSICS=y > > > > > > > > diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h > > > > index af68888..1ccd186 100644 > > > > --- a/lib/librte_ring/rte_ring.h > > > > +++ b/lib/librte_ring/rte_ring.h > > > > @@ -457,7 +457,12 @@ __rte_ring_mp_do_enqueue(struct rte_ring *r, void * const *obj_table, > > > > > > > > /* write entries in ring */ > > > > ENQUEUE_PTRS(); > > > > + > > > > +#ifdef RTE_ARCH_STRONGLY_ORDERED_MEM_OPS > > > > rte_compiler_barrier(); > > > > +#else > > > > + rte_wmb(); > > > > +#endif > > > > > > > > /* if we exceed the watermark */ > > > > if (unlikely(((mask + 1) - free_entries + n) > r->prod.watermark)) { > > > > @@ -552,7 +557,12 @@ __rte_ring_sp_do_enqueue(struct rte_ring *r, void * const *obj_table, > > > > > > > > /* write entries in ring */ > > > > ENQUEUE_PTRS(); > > > > + > > > > +#ifdef RTE_ARCH_STRONGLY_ORDERED_MEM_OPS > > > > rte_compiler_barrier(); > > > > +#else > > > > + rte_wmb(); > > > > +#endif > > > > > > > > /* if we exceed the watermark */ > > > > if (unlikely(((mask + 1) - free_entries + n) > r->prod.watermark)) { > > > > @@ -643,7 +653,12 @@ __rte_ring_mc_do_dequeue(struct rte_ring *r, void **obj_table, > > > > > > > > /* copy in table */ > > > > DEQUEUE_PTRS(); > > > > + > > > > +#ifdef RTE_ARCH_STRONGLY_ORDERED_MEM_OPS > > > > rte_compiler_barrier(); > > > > +#else > > > > + rte_rmb(); > > > > +#endif > > > > > > > > /* > > > > * If there are other dequeues in progress that preceded us, > > > > @@ -727,7 +742,12 @@ __rte_ring_sc_do_dequeue(struct rte_ring *r, void **obj_table, > > > > > > > > /* copy in table */ > > > > DEQUEUE_PTRS(); > > > > + > > > > +#ifdef RTE_ARCH_STRONGLY_ORDERED_MEM_OPS > > > > rte_compiler_barrier(); > > > > +#else > > > > + rte_rmb(); > > > > +#endif > > > > > > > > __RING_STAT_ADD(r, deq_success, n); > > > > r->cons.tail = cons_next; > > > > -- > > > > 2.1.0 > > >