From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM04-BN3-obe.outbound.protection.outlook.com (mail-eopbgr680049.outbound.protection.outlook.com [40.107.68.49]) by dpdk.org (Postfix) with ESMTP id 305371B1FB; Thu, 12 Jul 2018 19:08:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WzEgyJeJw8B5MVX5xrfayNoxVpsDZNYj73hB4kN/8rQ=; b=EM+xCfRYGA1is6mbQg0RWeTNPShGoRnuAwrsVlUVaY6cVkd22KduyGoQtk/S7ULSCKG45vWEQJvYD60Cv+aiYRaHC7Gj1gBOR5OAPWvKjpDvJrGWKZTlJh4HCHndZs9q/kGdOI3gl2GDE6y8tgmuKezi1OjXNk07OE0iJqVmRv0= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.JacobKollanukkaran@cavium.com; Received: from jerin (50.233.148.155) by SN2PR07MB2527.namprd07.prod.outlook.com (2603:10b6:804:6::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.930.21; Thu, 12 Jul 2018 17:08:48 +0000 Date: Thu, 12 Jul 2018 22:38:41 +0530 From: Jerin Jacob To: Takeshi Yoshimura Cc: dev@dpdk.org, stable@dpdk.org, Takeshi Yoshimura Message-ID: <20180712170839.GA11626@jerin> References: <20180712024414.4756-1-t.yoshimura8869@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180712024414.4756-1-t.yoshimura8869@gmail.com> User-Agent: Mutt/1.10.0 (2018-05-17) X-Originating-IP: [50.233.148.155] X-ClientProxiedBy: BYAPR07CA0036.namprd07.prod.outlook.com (2603:10b6:a02:bc::49) To SN2PR07MB2527.namprd07.prod.outlook.com (2603:10b6:804:6::27) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 26fa1328-34f4-4c37-aaa0-08d5e81a2360 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989117)(5600053)(711020)(4534165)(4627221)(201703031133081)(201702281549075)(8990107)(2017052603328)(7153060)(7193020); SRVR:SN2PR07MB2527; X-Microsoft-Exchange-Diagnostics: 1; SN2PR07MB2527; 3:dnS+VLSOiwoE1iJ83Z9cA0UIfcnWroqpKc6KWvt6prfVJOrRYMyIV8fXdSFrAZQlNvJu3fXUVY3btNwJ3tLhElWhOCYLJhji3U6idXbmxw3/C7NYv8SYpNzWt4JnjecMl1JTLh3h36JfdGLIiOt/30b57fI8ZEiQIoLkPO0mBwtoJMwe2u8SGEASlvwM8qIFOMCl7ANTzWYNXSGSVpn8aWbsYtBXNWbBx4/g9UvnR0OchFNilQXw938iTS4no1N5; 25:U+gKw+2D1mg8uiI8CSs05avsFqI/c1KVw++Lt5FOQHi56VvTvnniho/KvjHrOn0jaPQG5EnTgRiueXPNvdxflHnWsWSz+ytCDtQKXsa4ltljySkyvBRMBpqEhbSh35qUuZpG4HP+vmTMk/DysM4SSICZZy/t3KuAqnIJY5x6LdItt883n+HHDHDEOJVH/y7MJZ2+TuBr9D+4rcEcxLUauX+Y4lvQYnMseXwXzWTNEwoJfcsA+fGhWEa96w7+8xz22TRP18DhkKrMOPUL+3+EQgwu58aYYTscza8WsG4DzoKpIA9635Yyw0Iss1uTewfjE9MciquMuPOGifDHJDmXJw==; 31:O8DWWZBmlQRYOw7Y4RfUkDCkQvBmUkVZVgs4e9KfJK+JP7JmVswC7+qQvmZ84WgLrpUqUmahmvoCpxFhnnGlLFaUb5ns4+Q28gJ051Zu0OAqkCIaFIICn2pNY8c4FskOJSqZOyTEMUZ6J7hYu67kKHFC17UgU5wuXp7jN9p3/xDPLtvKxUnJsUS8iN2aq6IAwLCZxThnY9A4U48Q8ysoewIV1O9t7M9IZG1wLw17EKs= X-MS-TrafficTypeDiagnostic: SN2PR07MB2527: X-Microsoft-Exchange-Diagnostics: 1; SN2PR07MB2527; 20:irUKNPcSC1tLp8PrBJlxioA+oFc/es30EXwkp/NAPRjtekey+WIbGUaiB/I3XBch3T2xag0FbQce5ew82ELjLBZUeYGpdQBSNoTOlYeyqv3RzuJuYuf1uzyoUv8OCzKgUjkOR+q4PtptBQKGRCe1yajYHVUcY7fkFWH4w7RPcV3lHvf0NtuBKZiwUGWgMB3+nUx09ix8A8NznVJ1ww3YMpKeLHulykX2za4NWVRTXy4NFK2toV+e8KNGIG3ZzdembJXJlPQsLBBWd2jKqgMQoQVhEuaEMcQCF1nwEG+ugs+7Mm8L+HdqI57TPV/Il9fJUMHgEKcplxG3rCcTTurAw2HJf0ntxSoNE0TmdSQ204fRdRw1q0O5ug1Ik/lsqMI/fEC1g4feqhAW+mXR/LxudUO3qzQ3vqCh2sPQYbMJ4h0/kpXLF4BRBsjsBgC81uBdM9r0Xh1pcUw4JcmBGtIKB0QWvOfRZWb07NcNYAe+sRI0f3h3jUrHMEc3ECGI4oKrU/H0UoqXXPs1A0oSDjzEMAXzO+/Jn1EBJgK2rfJ3anafrWglGcPEsjZmGUeBt4fIG1hs9CrnzDM5BOSPGk2q+Q8rHgL92QuFv+3Z0/cv4l4= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(85827821059158)(104084551191319); X-MS-Exchange-SenderADCheck: 1 X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(10201501046)(93006095)(3002001)(3231311)(944501410)(52105095)(149027)(150027)(6041310)(20161123558120)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123560045)(6072148)(201708071742011)(7699016); SRVR:SN2PR07MB2527; BCL:0; PCL:0; RULEID:; SRVR:SN2PR07MB2527; X-Microsoft-Exchange-Diagnostics: 1; SN2PR07MB2527; 4:QM85iytaw1MQaIMfZdSThHYjuFawzOX7MQCHkYwfmLJ4LJWk+x4LbDgLwBzVBvu1CkT16Kcd5BGtB02QGrnmPa5Oo0+mRTqXeLZFxdiKVLnh5+MVMgXCsCl2g40Olkgvrz67DHZd9zjQq2493IApjDDGoivvgCOtiKWqnftoE7/gWr97Lls9RgXDDKoqU48Y9yuKkHVu/DHTcIc6pu19hJ108k0kS9UbOrS4Aw5l5yATbLx838tKxpChSMF+YfdeW90+J8FVkHTbQg2PqPl3STngiZqb+t0INVil7nc61JMrOcNb6X+r2/yfKtuEPuNiyHOTKJ5nKP/MlZ/LLLC4HQyrvt6vhzK4jpFDP7Lf8jE= X-Forefront-PRVS: 0731AA2DE6 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(39860400002)(366004)(396003)(136003)(346002)(376002)(199004)(13464003)(189003)(16586007)(58126008)(6496006)(7736002)(33716001)(6116002)(316002)(50466002)(72206003)(478600001)(81156014)(1076002)(81166006)(8936002)(229853002)(33656002)(3846002)(11346002)(23726003)(305945005)(8676002)(4326008)(42882007)(26005)(25786009)(55016002)(476003)(16526019)(6246003)(186003)(68736007)(2906002)(6666003)(14444005)(97736004)(44832011)(486006)(9686003)(956004)(53936002)(39060400002)(47776003)(105586002)(66066001)(446003)(6916009)(386003)(76176011)(5660300001)(33896004)(52116002)(106356001)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:SN2PR07MB2527; H:jerin; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; SN2PR07MB2527; 23:qr6bPXwmMu1Sb/dD/XkPSCCE3zp0iy+Ck42OWiEkK?= =?us-ascii?Q?ucqvGXzGPi9buFU953gGNEkFI1TP/vWmFvEn2a1dhT2ekpO3TViYHQLBqMKc?= =?us-ascii?Q?brdS+quDbjyWoWTl02AYWI4NcB5YdCnd1tgdiaq0uyJDwUJbxTdxDIz3GZLd?= =?us-ascii?Q?bJQbKb1bYn4+ShTh0bySmI6PAT8Ii0CFAvJaaRhEzcvJQGJe17RSB2IP41pd?= =?us-ascii?Q?UMhyKCIB/kGccRfkfAZuO3AmUoZMZ/b8WNywkoqIOoMnKv0LXcJhhkllypLc?= =?us-ascii?Q?qkkQjNvxI/eZ4WsVHZ84gq/lA3Lig5E1kOR1i6jgaKoSjk+G3LH/oUJW1Hhw?= =?us-ascii?Q?DtFw1DNscEtTjrwGlNCQHgyfb7WkEnbuiURmmC9q1vqzHt/seOLbrANlf/yU?= =?us-ascii?Q?b2VVMlFbft/5axyT6L+pgy8Ud5ANTOzRLR8rxGsZChuwyzJ/MCQ2cj2dQLAo?= =?us-ascii?Q?kt6GiBy1+pzMVIpkMVM/JqdPMq3mVYQQDO1Br/VCt6aC4GDP6zPeC4P9fsTz?= =?us-ascii?Q?1U/Ce0tWifRMXyE4ddEZ6mEFPwpD1VUjj6nYpq0GAwLhDeNPt36XR2ZxOybV?= =?us-ascii?Q?EgKP2bblz82rNFYsej35f9A6jxZQHiXbaXjYMXvysVrDXCqbMAn/YRPMulWo?= =?us-ascii?Q?VJcovBTFv0VkhjtVYMmdJ4+1m80sLPUQLFY6tiXf/NKbe58Mr9TcCDly5GLO?= =?us-ascii?Q?CsZfBwRudjSwd1M9jrUNqkUnwzSVikZ6BmItmUNl6joreeno7QGgZIPpSRMz?= =?us-ascii?Q?lbU6/4E/lzPUhZ6N/sEjf4V96fkL9WMvjvy7FZJCE31iaO4+PTFV8bq066lH?= =?us-ascii?Q?JRDqLKSgmmbKV4yZeCzBzwV6KP/8iR4Q7gaU55i6viwip2fWlmTXkdHnO9S2?= =?us-ascii?Q?PTf0yqJYAS9IpsCyEuyErjppJdGvvhxv3IH1Ms3cgU+V+rae+WCVCl7LxMIh?= =?us-ascii?Q?ulR8PWa5xZGUCfpB+dYjlb5dlJxuA6sWgE4Mk6KHyHYCaIHpZz4IufVQWY3Z?= =?us-ascii?Q?Pv6EGr/WBv9RYEJ99dMnp117KdNUqOVRbPQbJ710beIOP0OXgxR4ANeOA/Gb?= =?us-ascii?Q?ifbizoKBEpQfgQ4XEWRtidJGhjRB4PEApjrrEcdyYxVXMpi25E3m8906GONK?= =?us-ascii?Q?bMsXgX1GG4Hk+vlfHTHfkq/qTXB2LQREeIlSC70Ke3RRApaJz3rncqKSgxTS?= =?us-ascii?Q?b1AnkRqPtApP77tRSxvDEt/UvY/M8XunNL/RqtU1/H9Yn685p72+1+Qe9Lz7?= =?us-ascii?Q?LGUK/mECd+L1WQ22iUAjks48G1fB3OGu0bz6x0EYS1k1au/nioYOAVtCjvqs?= =?us-ascii?Q?OrMFpjYuCqIN9WsNT8Krqrc2am0GqAWaKCszxwF7sjdokxNqhT/h2xwGSVxH?= =?us-ascii?Q?4ZPQA=3D=3D?= X-Microsoft-Antispam-Message-Info: ruUGEs6AqfzqEOAb3wf3KuKX3qBm75uV8Mc1hOk96CJf6Imnxn6e3WsRYtfQZJeIJ2OY/BKmQaCfF75d1BvQScPiEiM30xA8XukbSSfQgi2buzEU0uZZJNJ+MLYX2jPs4aq0kUplV5WGO9q9MgEgtMif9+vNscYnp3IKsiYDA3nZnhMm0dj/E4b1hpWBr9Hv8EAwnL1g6LYH8rB0lZVfEzLQGQBHiYxKbLXunjBascSHDWTc0rfRC1JpVDw7IYCMl1xlC4Lih7ScooZXPfnLJJ4O3q7NQheatO1sbqV1UohYlCaDeCYlleM223g4o0BGjh7mY8PgjSB3MHkitBs6I0PcaDfA/HW/nqngRsww7ak= X-Microsoft-Exchange-Diagnostics: 1; SN2PR07MB2527; 6:t91Yvv2aZ7ygq+HdNGVtPbwBm7ppGRCTLuDNlg5ev+dwjy8b8njWwsqnufBiVNUVKJte9B1YrN/n7/cHsFdKG0D2etdEhZer9RIA9AQnghZPCtba/PZkB+iEuXoaX2DscwPUCUjXgugiFIBscd6pvI8Z0iDVIh5ndDfdJNRMeTDFpNjUvnNX7FSPn6TirIFhYjeF6WXCnr8g5rZjC9BYBJr5hdN0/PTMwL5eWuEFntr2XJhkZTESpb/EINfHK+/iKxpNbGyoIEKfeC1PKzPEWO59SyR1qOvQkaJMyhon+Ymhpz2l1L1oSdoccwZn4eVi/CDta5U+s0WDNfuQYi65ep0YcTy9UYR8LEryh6QxkA+kngH27chlLHTo6ZCXRmu/BJG499U5Alb2yUrdtkejEdNtDdRZ+J5NOkGxD/6xGNXpy9jMpMaNGdFLpe9NVBG3ANI+j+UELij8tJaKhf83+A==; 5:0VFPNCozhPz0mg8278tRFSKqsx3fBArVfgFS2/g1oNHDo84cnT94/Mf3PHBNSresaFGeXf48q6YeAvYtP2FPQyYGgrNMJ+cQQF04NeBg1GEaI6YFyI+DEepMT7vIDXTVFh9vj1psOVWdKGkDs/ROkG0DchVqCqvh621wpj4/LMY=; 24:OJo0yYvFCqef+8cfz/O6F1vTn2nPa+G/n+AqUlwrD1Rv6csu544hixUbFwdvaRM79P5vgUW1tnrGk2RSPhbj3kWVS32Q0tidOq/LivKWlg0= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; SN2PR07MB2527; 7:6D/iTCZ4gj74sDZjH1+QeKLjspiZHqqzFrTB/KeyeqPdvUk8obCOt4sSylDxufD3Gs09hjpxCIvpptwnEPT2tRfx9C/pAKB6UHJGtyn2dm7SBpFQOx65fMK5jAXsY592yfV4A0dj+3suV1Tm4PvLsnUq01KQ80u8/gcUKjpyh8641rEx/+2cAv1IE77YjAhE0p3t1aFHRkGfqK1s5MwnA4hlrOFIB8SonfgpwgwWoe31Fn6xkJW/REuX1FoazMvI X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jul 2018 17:08:48.0766 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 26fa1328-34f4-4c37-aaa0-08d5e81a2360 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN2PR07MB2527 Subject: Re: [dpdk-dev] [PATCH] rte_ring: fix racy dequeue/enqueue in ppc64 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jul 2018 17:08:52 -0000 -----Original Message----- > Date: Thu, 12 Jul 2018 11:44:14 +0900 > From: Takeshi Yoshimura > To: dev@dpdk.org > Cc: Takeshi Yoshimura , stable@dpdk.org, Takeshi > Yoshimura > Subject: [dpdk-dev] [PATCH] rte_ring: fix racy dequeue/enqueue in ppc64 > X-Mailer: git-send-email 2.15.1 > > External Email > > SPDK blobfs encountered a crash around rte_ring dequeues in ppc64. > It uses a single consumer and multiple producers for a rte_ring. > The problem was a load-load reorder in rte_ring_sc_dequeue_bulk(). Adding rte_smp_rmb() cause performance regression on non x86 platforms. Having said that, load-load barrier can be expressed very well with C11 memory model. I guess ppc64 supports C11 memory model. If so, Could you try CONFIG_RTE_RING_USE_C11_MEM_MODEL=y for ppc64 and check original issue? > > The reordered loads happened on r->prod.tail in > __rte_ring_move_cons_head() (rte_ring_generic.h) and ring[idx] in > DEQUEUE_PTRS() (rte_ring.h). They have a load-load control > dependency, but the code does not satisfy it. Note that they are > not reordered if __rte_ring_move_cons_head() with is_sc != 1 because > cmpset invokes a read barrier. > > The paired stores on these loads are in ENQUEUE_PTRS() and > update_tail(). Simplified code around the reorder is the following. > > Consumer Producer > load idx[ring] > store idx[ring] > store r->prod.tail > load r->prod.tail > > In this case, the consumer loads old idx[ring] and confirms the load > is valid with the new r->prod.tail. > > I added a read barrier in the case where __IS_SC is passed to > __rte_ring_move_cons_head(). I also fixed __rte_ring_move_prod_head() > to avoid similar problems with a single producer. > > Cc: stable@dpdk.org > > Signed-off-by: Takeshi Yoshimura > --- > lib/librte_ring/rte_ring_generic.h | 10 ++++++---- > 1 file changed, 6 insertions(+), 4 deletions(-) > > diff --git a/lib/librte_ring/rte_ring_generic.h b/lib/librte_ring/rte_ring_generic.h > index ea7dbe5b9..477326180 100644 > --- a/lib/librte_ring/rte_ring_generic.h > +++ b/lib/librte_ring/rte_ring_generic.h > @@ -90,9 +90,10 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp, > return 0; > > *new_head = *old_head + n; > - if (is_sp) > + if (is_sp) { > + rte_smp_rmb(); > r->prod.head = *new_head, success = 1; > - else > + } else > success = rte_atomic32_cmpset(&r->prod.head, > *old_head, *new_head); > } while (unlikely(success == 0)); > @@ -158,9 +159,10 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc, > return 0; > > *new_head = *old_head + n; > - if (is_sc) > + if (is_sc) { > + rte_smp_rmb(); > r->cons.head = *new_head, success = 1; > - else > + } else > success = rte_atomic32_cmpset(&r->cons.head, *old_head, > *new_head); > } while (unlikely(success == 0)); > -- > 2.17.1 >