From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM03-CO1-obe.outbound.protection.outlook.com (mail-co1nam03on0047.outbound.protection.outlook.com [104.47.40.47]) by dpdk.org (Postfix) with ESMTP id C19101DB1 for ; Mon, 18 Jul 2016 04:47:29 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=/W/O1ZmE6wB9glrvXplS5mBj9F2aaci7wi3AYd6559Q=; b=Fvj3jKiop/HT91iCrSaSpsK0qjx44kdH+WkEYACkFEV0txp1IKMLhmHVFs1uPBMCU9q10kHm2/d5QbI9O/o22M0V7Mcq3QJi7CtybFB5MWsZwck6/uvyNzTKptCzkmw/0oElR1/3tAUnG/ORdY2/PvCOdv/fLHbspS0kXJIl7Lo= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.Jacob@cavium.com; Received: from localhost.localdomain (122.166.170.128) by BN3PR0701MB1718.namprd07.prod.outlook.com (10.163.39.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.544.10; Mon, 18 Jul 2016 02:47:23 +0000 Date: Mon, 18 Jul 2016 08:17:00 +0530 From: Jerin Jacob To: "Ananyev, Konstantin" CC: "Kuusisaari, Juhamatti" , "'dev@dpdk.org'" , "Jan Viktorin (viktorin@rehivetech.com)" , "Chao Zhu (bjzhuc@cn.ibm.com)" , Message-ID: <20160718024659.GA4537@localhost.localdomain> References: <2601191342CEEE43887BDE71AB97725836B7C916@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB97725836B7D20B@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB97725836B7D628@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB97725836B7D850@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB97725836B7DD85@irsmsx105.ger.corp.intel.com> <20160715062905.GA6473@localhost.localdomain> <2601191342CEEE43887BDE71AB97725836B7E280@irsmsx105.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <2601191342CEEE43887BDE71AB97725836B7E280@irsmsx105.ger.corp.intel.com> User-Agent: Mutt/1.6.1 (2016-04-27) X-Originating-IP: [122.166.170.128] X-ClientProxiedBy: BM1PR01CA0016.INDPRD01.PROD.OUTLOOK.COM (10.163.198.151) To BN3PR0701MB1718.namprd07.prod.outlook.com (10.163.39.17) X-MS-Office365-Filtering-Correlation-Id: 6f534884-4974-432b-5e0c-08d3aeb5da49 X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1718; 2:haWoF/8WN++ezn3YzIofmRvwk91eXOguvrmo7cnIJ/YnbpiSYmQ3D0ux0MG6bl7KDP8JqDv9e04gEJ2UrDbnGxx0EpXsE9UicDGU1SyPMh2ccOQm62FX643KhlGL+BV4GIzRynwVF9aJ08RSTNF8cySXaRj0rrD4I0LdhxUEH1Qzn87kPtFRnHEgte+66sTU; 3:hzc/OZ1rzuIxuJOlYjQdY3O/nUgtWjbDoveR1SmDtZD3FaHiDA4Der1zztkKmZGF8SutWb8cvq5cId+yl7ICVDID97TnlYsrS6m6kWV/6TdNbDgTlxujNYsHmoGS+O4n X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN3PR0701MB1718; X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1718; 25:VyFbSDK5Fem+D2Ke6i43KdLaw5KJDtolvARb1HhFJATnJAVSMRT+1gZxxTXmlzVNN0BWfW8bPr57LV+6zlN/SK1VAmtbsJOGX11Z8YIwvk6gqKn+066iVKY92XKw48VOf04IQTsAQTDo0j5B7eCArUBM0ZiSqwyRDrtmzuj8G32CBJt54oArK6OXzhvBzZdKcanYkZaDa2mONOFM7jTpq/8ANN6AT9lJsu1bWes+0Ibh7mplkCXfQkPM0ohZkLtHJLOQVi1FfzhtZy55Bo2murskLiLoJUA0cVdNl9uK/7V1buQCMrlLcpyz4dgnaBttzeiSfG3M7/2UDknstvB1ez3OLZwV35TGHihm9aeKJc3bCa/VRhDwwTAtQYDu5s5SpEJmRrgI0bd7HqHFgRpOvuIgC/9bd7nfn0j1hm128d+VOG9xFEavTbukA9KSFSGQ0luK7a9VLA5lVFYh1l8MU4zR7m/cLbfJf8m7AZgQvkMmh3Eeqy943z3vqhB4WsbNdh57KjdoqkwPF7v/XV1cefUIcdAZAYW/DSH7ex/1dbDLFyhSbn/dFvx7CXNf7ZQWIV3yYUhrspmKLKFDe1dwsqC3v6T2Z8WCpR0f4Uw3XKxks3BCHR9pdu6ff9FlJsgiUR0gz7dKgAEyLhGjZc3UGpiHGk9FFBCYAVULRWo3dF947nom21e0yya0gy1VCM/QEP65ws5QY7YSsZ90CoQlLvZqaWM0ZWDVFr9KRFFFZuCxTrc4gcCEEZn/r8R+M8A5cjLKSCnzIIe2qbbTwbu7WScFvt5GsLVHMTviOnpBN69bj6LARjIYwlOHUpAO0a13fX//FWZQLHx8g2xJgXvpFO93CLvoVVQy0OGglArMvnwpNAWUnW5LETXAHiMbaHXQ X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1718; 31:jzS82qndsWQ5uaCuuFyhYt09HMqW73F3C25iBGHfve5ufnC3Y7V7ZjdB142uZXuSElkYyGVx5iD67mYUp/7woABbjWRc63OwNLBop6p1/RLkPEUFg+isK89C4Ogus7+dfdfn6NVqyOcfEjl5D5wkO4QfTEdpKR7vSZVHpQmbsjxXixuhk8TRepigR/L2cEcwRqRwj9bWf3ZVUJUjHZ7myQ==; 20:vt/SE2L21GuDe8YU8xbaoescIOdvAfVOXymGDFRxUkxWInaU+/lnUdQbFo/PpWzkVHy8AqV/ZmyMQtmpEy5v7satUgorO8QWsJzZM9uOX0eYed7zGLKXUBxYP3gYkqtOKk6vlmhB06wYRL2pMlWbk3PnKQGMa1621WkheTylcldsE4EaVyumzEeanYySjjUkpBXypI9SdgcBQgP4t4CAclV5gVQ0tNRA4UbyF+sIyUTZ6CsnJcfQI+gDJKewMc2ts8xN/xvc+SyAIAAdLiZPLZYYwOTviTtjJhxAxiBTlvinL7GedeTLjT5Uy9zA+j10X4UNC8svPw2UFtuRUtnspLoLwqQNaejvdkGr13qv6BNR+NNhLrh8wGi4uePg6Bf7xvpqiekZcFLvypN6u9hhESUk3D++oouiXmmTipQ58xMWLI/Pb++mN67xXvywNNqJMViYuqAjDtr/19H93zm8fIGwLXrjToNE0CPtGUo0xxDtpmEUThREwmrFzRu5N2EvJ3blmt4/sCeygsA+VtjKRGvrJOT3ROk4BCnRIVy6asIAvleF33J+nCldqkg1/p4iT+nduBuZROikkQjQSGtyjohjH2MRt0amxJ8OObQIkP8= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(180628864354917)(22074186197030)(183786458502308); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046); SRVR:BN3PR0701MB1718; BCL:0; PCL:0; RULEID:; SRVR:BN3PR0701MB1718; X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1718; 4:f/0WDlQmQyw+agGjLpEoJMsh9wH55WFlwMH91x88pQNfp3g47nGNfsCJAMTDnE3iCU+QK11hlIr+ueE8IgserNBWVHWftWZGV1+QovXnndVsbFXQcRUxA28Q25ah5W0MKOfdvZMUMKa01ttpWNaWHVrIxjMc5B/YShMVTU4k/t0+xespdvk4OVkNRVkNpf6Z9xomVxdfaiwY2XyiaTWzMWb9zbz3lPRUTUlSsBNwfTrRxPWymwNWI5ECjalKUtgjUAcz5+OYXDDt2xXupJV/Rge07Y4jw4vuaihfQcByQUYOW8SooBEmppYDvFJt0SiSQpAE+LA0cor/0JcL0Mfot+MbDdiJT0ftMRGSwyMihFFJaDwH7Kri4nWMp+E7+3eTCC7HL0mW40LPTllC4rbHCZcPvq3h5E25LiSplmwNCYdsgsuJYW8p00iwhdNhBruAUmY4u1e0X7cw84Q4QziWZ24Pn4pRO4JvfBgtYCx3HV4= X-Forefront-PRVS: 00073DB75F X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(6069001)(7916002)(24454002)(76104003)(189002)(51444003)(199003)(7846002)(15975445007)(110136002)(8676002)(77096005)(61506002)(1720100001)(189998001)(83506001)(50466002)(586003)(68736007)(42186005)(19580395003)(93886004)(3846002)(97756001)(105586002)(6116002)(92566002)(66066001)(15395725005)(9686002)(81156014)(81166006)(4326007)(23726003)(2950100001)(2906002)(1076002)(47776003)(4001350100001)(46406003)(7736002)(106356001)(8666005)(33656002)(97736004)(50986999)(54356999)(76176999)(305945005)(561944003)(101416001)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:BN3PR0701MB1718; H:localhost.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN3PR0701MB1718; 23:XkZtR84lQuiNsVx+3HadJDkKpk9Rcga7nzhZxW7?= =?us-ascii?Q?Qox5hA2GzRmwgTL2+dgRNrotUYZpPWzeP8GNQGhCOq9J/HKynZbuKBw2LECu?= =?us-ascii?Q?MTZvEvfcS5z/80uhPaKKR8PZewLzCgRSW0jEGFJpBtqu3Ke4cjNO7WEr6gev?= =?us-ascii?Q?xPyHL9B/eU/7Jur6pindQT+FMX4FKydLkoRtWuYDV8zwVfxIJGJa88+FY5Vp?= =?us-ascii?Q?g4iW6QPU++QH+rddIdwYMv/B3A4tTYNubhpSp0O3pNzDa10ON5M4ZNa8H/rv?= =?us-ascii?Q?K2/YaTNpVZy109e4H6KhL4xx123lTPWGxEBgAX0Nm3OaoXVV1BvcSYrs5aoZ?= =?us-ascii?Q?cTn5k09qkYVGoEJBuAisTShNU/CoubJwyyFj/0V2bWINlPjv5wI4gsyYQvAP?= =?us-ascii?Q?bW3KVw4PMO6wg0jZsyfddou0LbGAcGNk1Opv6NmyITyBhQjSH0VSsfJQ1Vg+?= =?us-ascii?Q?SuxAb1e4wdTmUhodrln3fa/jOn0Is43oNWzslqczpZ1HCURnZc8ENceM12nA?= =?us-ascii?Q?fVHUmkC1N1bvR9iFR3Lv599NCoWdfv54E4iZFH8b9Ws65SJYLwo3z1w3yKIp?= =?us-ascii?Q?Kh1mPO6UzyOPXVC9jIYCr7hBOyFpsT+ZR8LsytFBYY/OAPEzZKNHHh3kSEBn?= =?us-ascii?Q?HWCzbY8EQJQkYp83GcsCrfH8qDNe/AoQTAS+4BOo/sGMoOJzCwaPDPg2J9+o?= =?us-ascii?Q?HqIKVVqrzvoycD5iZ3MuYtLQDbfEHJ2JNNWsCoQvVBOQ3Odswn6FZweHEUpv?= =?us-ascii?Q?GWk7byKpi3wjFhjPi9/4wY2pFZUxth/bqVuuqcNbwCefaYVzel4aikj9M43d?= =?us-ascii?Q?MvEYwJo9+6f03UhF4jvm2oKhahkF8oe1o/4BMDIHnTxJMFNdfvACYnqnGNNf?= =?us-ascii?Q?eXLxts5PU2K115mm7Y/sJdQaUjpplolY0GsezgQ7GW+OR7n93HIbrgemEo6T?= =?us-ascii?Q?ODJ0kCuTaBJygGvonn/V/ENMGhHFPqXdQoFGXDRFCTJutgqQkJ7ZnTvpvFeE?= =?us-ascii?Q?IX1VtynOntTsXrB3NHsPVI90eIed86VA8nhuA8yPEog3iVg10eYXSb/CHySc?= =?us-ascii?Q?3AzCqWQyCziIlJ7MFFAEwx47oUZjmrEXu3iiBMco7CdEtX/pv7r9rOc0hhRS?= =?us-ascii?Q?+ojqBQCz/k6uCF0tXS6AEDX3/oQZIdormtW7Tk7vdorFibW4bAlhhwyzMJ2E?= =?us-ascii?Q?z+sB+Iekqlvgrvm0tVL3znmfCVtJjGUJpZZFX6tHVsbWDRdux2thjsFF4P8q?= =?us-ascii?Q?0xl2zT99/t8rFyul4bt1AIKUX4RBPRmKMAcj+I9/Hs/HNMpQA6mQ4CSiFpqB?= =?us-ascii?Q?ibzcw+HvD0gENX5I9zzFU6TnE6t5KQHdBibcykBDHt4PU?= X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1718; 6:bLIWJ1hILEWenQpe30+QZUWcQUSXQk3QExDtYYXi1bKL6U2MKIA7lVOlrpA6fncG3kILeGhj+L+eVEhULvEG2M1WXRVP2oLMmPo2QgpNQSv9zRyP5n61zSx8NjV5zHMO9gk/IeeqJFKSe6sHwbPPLkwHef+b7WtvyQpz4tm7jnms2qwyQf1651Y5xMfux0svaA5mo08lb71NR7hqJyX8c87xabq2qZc3Rk8LdpzrbIdRNTq7jUmKpRnR8u0vmouL22wc9vFIfYUE5pMcFrpjeOdb/K0W/ulMzo3gsbpm/Q0=; 5:DVt7I5vOLvLghUN5DXWD1JrkqPyyKBi5cwNCcUP6J2fHQbpVq3Ud7+hwwaIs9g/REQVDn98/ttn9Vn0/nWspZdCCslwdkxG+HGIQjJFoBzCc0wl0NWb/z/Q9kIL9RbP+FaVdWsiG4EjayK0AVoOrTw==; 24:VeECyj+K4gZrJkv85DLIwgyjnrMt5nha18N5bg67fafwLgWY/w5LJcjb4fg7Wgvv0iUK6kFOBE7zLwcM45bmlJFveOyaLrQ/pVKHG09JDec=; 7:MWXHqf57RwoZROng3uOOQqCLSqE+cmqhtl9jKK66fd9Dep9VqrN9aCfG8CLa5HEuemel5UidY6TRIjD0Rj7Wae/JABw70vNLi8G6lGsYo15wkk+9wMPSQBnhtdN3+jzR06RS0hL3S2FPjFjcovZGq60mz0wurI8tJENAurMPUh+kwfOoYozj2HYDahwFs/W6xD9P9gNXOgaEoMs0wRSGejBiYLpkgA6E3s6HMVXw6EfUyRp72pWRKK1OR7/TqO9/ SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2016 02:47:23.5733 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR0701MB1718 Subject: Re: [dpdk-dev] [PATCH] lib: move rte_ring read barrier to correct location X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jul 2016 02:47:30 -0000 On Fri, Jul 15, 2016 at 10:34:40AM +0000, Ananyev, Konstantin wrote: > Hi Jerin, > > > > > > > > > > > The CPU also > > > > knows already the value that will be written to cons.tail and that > > > > value does not depend on the previous read either. The CPU does not know we are planning to do a spinlock there, so it might do things > > out-of-order without proper dependencies. > > > > > > > > > For __rte_ring_sc_do_dequeue(), I think you right, we might need > > > > > something stronger. > > > > > I don't want to put rte_smp_mb() here as it would cause full HW > > > > > barrier even on machines with strong memory order (IA). > > > > > I think that rte_smp_wmb() might be enough here: > > > > > it would force cpu to wait till writes in DEQUEUE_PTRS() are > > > > > become visible, which means reads have to be completed too. > > > > > > > > In practice I think that rte_smp_wmb() would work fine, even though > > > > it is not strictly according to the book. Below solution would be my > > > > proposal as a fix to the issue of sc dequeueing (and also to mc dequeueing, if we have the problem of CPU completely ignoring the > > spinlock in reality there): > > > > > > > > DEQUEUE_PTRS(); > > > > .. > > > > rte_smp_wmb(); > > > > r->cons.tail = cons_next; > > > > > > As I said in previous email - it looks good for me for > > > _rte_ring_sc_do_dequeue(), but I am interested to hear what ARM and PPC maintainers think about it. > > > Jan, Jerin do you have any comments on it? > > > > Actually it is NOT performance effective and difficult to capture the ORDER dependency with plane store and load barriers on WEAK > > ordered machines. > > Beyond plane store and load barriers, We need to express #LoadLoad, #LoadStore,#StoreStore barrier dependency with Acquire and > > Release Semantics in Arch neutral code(Looks like this is compiler barrier on IA) http://preshing.com/20120913/acquire-and-release- > > semantics/ > > > > For instance, Full barrier CAS(__sync_bool_compare_and_swap) will not be required for weak ordered machine in MP case. > > I can send out a RFC version of ring implementation changes required with acquire-and-release-semantics. > > If it has performance degradation on IA then we can separate it out through conditional compilation flag. > > > > GCC Built-in Functions for Memory Model Aware Atomic Operations https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html > > I am not sure what exactly changes you are planning, > but I suppose I'd just wait for your RFC here. > Though my question was: what do you think about current _rte_ring_sc_do_dequeue()? > Do you agree that rmb() is not sufficient here and does Juhamatti patch: > http://dpdk.org/dev/patchwork/patch/14846/ > looks good to you? > It looks good to me ,and I am going to ACK it, but thought you'd better > have a look too. I think rte_smp_rmb() is enough in this case i.e DEQUEUE_PTRS(); (i.e. READ/LOAD) rte_smp_rmb(); // DMB ISHLD .. r->cons.tail = cons_next; (i.e WRITE/STORE) rte_smp_rmb()/DMB ISHLD will create the barrier that r->cons.tail write will wait for earlier LOADS to complete. I think we need only LOAD-STORE barrier here. STORE-STORE barrier would also work here as DEQUEUE_PTRS has loads from ring memory and store to local memory. IMO, as far as barrier is concerned we may need to wait for only LOADS to complete before we update the r->cons.tail Waiting for STORES(to local memory) to complete will be overkill here. http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0024a/CHDGACJD.html ISHLD provides LOAD-STORE barrier maps rte_smp_rmb() for arm64 ISHST provides STORE-STORE barrier maps rte_smp_wmb() for arm64 Jerin > Thanks > Konstantin > > > > > > Thoughts ? > > > > Jerin > > > > > Chao, sorry but I still not sure why PPC is considered as architecture with strong memory ordering? > > > Might be I am missing something obvious here. > > > Thank > > > Konstantin > > >