From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM03-DM3-obe.outbound.protection.outlook.com (mail-dm3nam03on0061.outbound.protection.outlook.com [104.47.41.61]) by dpdk.org (Postfix) with ESMTP id 055A41B31C for ; Thu, 12 Oct 2017 19:23:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=86xCKDpJzkxB0ajdH3EoXd1uwdchY3J7LSFg92Hb7UQ=; b=hOW0rQ7i+uetTmDCAuVyqFMxz8YGZZnDR2sM2vKV6pVg6O5qV16QJRlUc4gsp2Z6oalEVXd7+XBL6LwFNTSQdhhcVjB7xAAlrRuDWHPoRiypC6he1WkAxKpEqOoGwEXYJoHeLh3wpi6muaxLTEOs8N6EA/dPBmdsmHOPmLpiQTg= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.JacobKollanukkaran@cavium.com; Received: from jerin (106.201.58.193) by SN2PR07MB2526.namprd07.prod.outlook.com (2603:10b6:804:6::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.77.7; Thu, 12 Oct 2017 17:23:33 +0000 Date: Thu, 12 Oct 2017 22:53:12 +0530 From: Jerin Jacob To: "Ananyev, Konstantin" Cc: Olivier MATZ , Jia He , "dev@dpdk.org" , "jia.he@hxt-semitech.com" , "jie2.liu@hxt-semitech.com" , "bing.zhao@hxt-semitech.com" Message-ID: <20171012172311.GA8524@jerin> References: <20171010095636.4507-1-hejianet@gmail.com> <20171012155350.j34ddtivxzd27pag@platinum> <2601191342CEEE43887BDE71AB9772585FAA859F@IRSMSX103.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2601191342CEEE43887BDE71AB9772585FAA859F@IRSMSX103.ger.corp.intel.com> User-Agent: Mutt/1.9.1 (2017-09-22) X-Originating-IP: [106.201.58.193] X-ClientProxiedBy: PN1PR0101CA0024.INDPRD01.PROD.OUTLOOK.COM (2603:1096:c00:e::34) To SN2PR07MB2526.namprd07.prod.outlook.com (2603:10b6:804:6::26) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 20800f53-5313-4795-a61f-08d51195f902 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254152)(2017052603199)(201703131423075)(201703031133081)(201702281549075); SRVR:SN2PR07MB2526; X-Microsoft-Exchange-Diagnostics: 1; SN2PR07MB2526; 3:NlvO0BF1TmpG2mLVxjUjz+lXlqtoAHyVTlwvxPxCB2wSxf7WKvKGg4SEgLsVorad+E3a6JPUF1eFa+7OXhrzo46LEmy1laW9Kai8Pn/DOIwU9N1KSPWJ5fn/1lKKKdlvb5MbJ/baSLCGI7KroQfyr/IKwNEruccKS0ICPj4+Tqd90J7jbArRE9LkHDbsNgsisGB6+3KCHzZwmv1jeTGGevdnSlDFIYkTUWwN3ubzdDVYdBPgX8A3pRVdhQa6jPy6; 25:DobhnVXt8BOQuQcTAHSsS/pQ7Wbtr2VRdRGGt/fbAae7AJGThPxy1Y0mkuZRcbKX4OIZQcu2cT/0rFr7BEJBotEfic9T9NBLbo5/JHxRRHdTOvQ2xft1YeqTZtNZ6pVwGxzbH/y/O/jFCFDSA9VU08m7DQzcNeFgqWIZKdhc/nGRg+JWvJxu/uGWfXzjLHeZYXiwhg/BVLJjDSiZF7v+hkdL1MhtnWy3qzcj8KsjDfaShYdvYWi3B09GPf3svvCuBd72Xtt0JQIW1IVBC3JmVDLIYs9i2oK/9P7QZatI20PDoY00tMtaa4AtAkCIvdY7DyVWNwsdZSGoGQZCFrJS/A==; 31:lRgJuEuQG8gwID5szyQvAyL76atSv5WIHU0EEC4pcEDrJXMJgIgHiSN5B6Yoe7nl1Fr125/HUm2PP1tye/bNMBQ5WoKJFKT0yjZWsx/ltIh1wG6q1TBRsZDCoi/+jqO50oCtpSWJ+d68zt0KlUYKgFFT8/qY4+tWKnasMCnSyKPL3TvZ4CYTShAg+BT6sLn7w4hvkbAPg6kiSjx2UacwXqixzTda6hZLUGRnbn7xFhM= X-MS-TrafficTypeDiagnostic: SN2PR07MB2526: X-Microsoft-Exchange-Diagnostics: 1; SN2PR07MB2526; 20:NqaHlGS+ZvZqw6h5Tr06X2mcbRGalrzYbxuV0ijRGpD0zISsnma7b5BIQvlWjiOdljzwSE96dUjwbExSiYMdhN9tAPbQ+xWXefUZTZD/eFUmYilpp/GOdWrOXc75NOmCY4GoStiqW/iXQg+rMozW1Em4SJ6WYJsswaEbr7c3Z+hgC3MtRbYtyZedb2TvvGvaMBTEI3ajN0rNlS9aXfjZvb2g5WVDhW5gT3kYeEMrbZf54lpx1uYES1wIKwCmn8Gfpmm6drVELzMu6l/kK5YG0TSqaYCgZpt5DmqYIfhtsMCgzAcGTlddvf4TbDxWynQgJBr0aT1Mka9u5/z+qCoq/wFMloNkgJEXrUlQzl91x8HeQGoplDqlE/GbUVh3PvuXS49Wu1Y2pOa+ttWdBYJvcCR07Lw2k1xRQnHs2CZdKAyp3HL/LgVAj6yNCl6ojYexcu09JkavPYIcasbWi7MGEstCa/TjAAcIC7Z2Mm/SBWqYP0NQuzRSduoQ4sETJlys/lw1dUnjzZHhpIZwAffQmygX7IoHpGiVxeVAFZW7FsyWNdjKZ4aEmt6OZg9AlTNrqU/DC8/82FVkHgwRmhNrrg5DFNV7HOO9+IoWVEgjP80=; 4:CPHHqsD2U7b4E58Pgn7b4OyGY/j7F2Ml3yCCGN21jheJztRW3crEZlIjue4Z1xsi4I4f3UtFQxBt9Q9MZ55Pl1HdyjZ0KM9uYPentEnokHT5eXCe3di+46JFGyiy3bDd9q8jQ+BRmuHUVBefzxYHiaks2VNlJUTbi4fjSmN6zcN7ltR6mFFc89v7ESGRtbLXEBfcEsw/wnrxMwN2nesJWvqw/gBSrW6CPOCWBVABUmA0THp2gQEznPRWJJqRFFkf9SbZ4fsmveouOfyi1O5OA2h2ZeESbVy5ozo8tt8OHPQ= X-Exchange-Antispam-Report-Test: UriScan:(228905959029699); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(93006095)(10201501046)(100000703101)(100105400095)(3002001)(6041248)(20161123558100)(20161123564025)(20161123562025)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:SN2PR07MB2526; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:SN2PR07MB2526; X-Forefront-PRVS: 04583CED1A X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(346002)(376002)(199003)(51234002)(189002)(24454002)(13464003)(377454003)(81166006)(81156014)(53546010)(76176999)(97736004)(66066001)(50986999)(54906003)(5009440100003)(54356999)(6246003)(50466002)(4326008)(23726003)(53376002)(72206003)(1076002)(189998001)(6496005)(6116002)(3846002)(39060400002)(25786009)(16526018)(229853002)(8676002)(83506001)(8936002)(966005)(478600001)(55016002)(305945005)(7736002)(5660300001)(101416001)(33716001)(53936002)(6306002)(9686003)(68736007)(47776003)(106356001)(33656002)(6666003)(2950100002)(316002)(42882006)(16586007)(105586002)(58126008)(6916009)(2906002)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:SN2PR07MB2526; H:jerin; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; SN2PR07MB2526; 23:BqUcXIdPA/CYMl4g0RGNBmLpZpoEoIvSZ4trUeUfv?= =?us-ascii?Q?C9f8jMDI6VuFgT8MTkDtedGcCWnxq0BM8DpS5rJW/5SfZQ14TlW0JklZl6ss?= =?us-ascii?Q?NoeNliOXd2RGq6HuzNwlYUgF6gLmohRq8W8m2Q3T3kVIl2lj4x13bBuKEuBI?= =?us-ascii?Q?36uU85jgw3vbkb/yc2uhHbALg/cKcmeMMR/jTHeh4QPKGU8RdoTS/NCSO+Sg?= =?us-ascii?Q?P4yXb8RUBYza5WAD2jbqTPr7ZIGCLEL6b+Kz7ob/R5tX26IMy35sYDMPmE7s?= =?us-ascii?Q?XBbOMIWWE7J65bXimwW2N8pygMiG2lg32niodO0A7VKwAhe2ihfSBPLeaDzE?= =?us-ascii?Q?Pqalt1I8YEdi4jvcyoAeIjxS9fAp1sjW4gRmajZGzkLg2R3vqI1gDmuXceNZ?= =?us-ascii?Q?59nf8UUTvFOGXeMo9OTVCVL+E5JBUGmstOmxhWwlu4y8c48qhOTJ3GV+i/MN?= =?us-ascii?Q?arpXXGLSa3IM59JCCwg3MrFp5BMHKH+62CIE5M1FLoidrPI7dtFhL7qVRyQE?= =?us-ascii?Q?4m7Arc9SAMVGWV+jK6kzXTWPubRq1I7QD/arzjvJ7o8nKtiXAHWjoM8wBJEq?= =?us-ascii?Q?GGBGRsvpko+jP4YnVLERfARK1EgX3VypMomncjQWZ0ezrcy5+9Ua6lSdg+ME?= =?us-ascii?Q?MjkENW4InDg306NkDZ2e5m08Sr9DBUxYwBBot6zSqGV10A6svtRrSBCpp3S3?= =?us-ascii?Q?9Org22vuaR3C6rQIz5hGfd1gFFfyW1/GLl5kIwBthJP7kq5cb85eSGqu5KQs?= =?us-ascii?Q?cv2RkfrxywrOr5MbiD1dW2hJkhM5zrwfquyQmUuY+YbjjTEz1uFKSEdYzNUC?= =?us-ascii?Q?KM9zcWPD59Ym8fXXmUTatnRe1LdzDnO2sFXS3H4XLTvW0NxJbT4wYIXNa54K?= =?us-ascii?Q?UYYKtjtHyXMOKeYPevPc0WNPB7855nJWhPonatHKIjWVqdShKPC7fkkYPx2A?= =?us-ascii?Q?uLJ0IRXO/xGwrW20L8LHX771srLbsSCkblQ175VD7/+vbLypwHkgGnfaUCSg?= =?us-ascii?Q?34s28kz/1f+kSGV4ZCXCc1hRhym2JhzfVyTO1E6H9AxdkAs1K2MNbYLz5xMC?= =?us-ascii?Q?cTN7VeCU1JARzeAimr1uk7eofju2eKv0ggPjp4McAp8I3q+fVnBXQxdBKkeW?= =?us-ascii?Q?tXqiady2667bctEMXI4fW7VGwY9gjpLVNl+z/CIxBc1vcpHbzgNqVqHabe3+?= =?us-ascii?Q?TwUiecMiDzJeW9GWHLTFcu38xS2V7E+boXZFvGC/vB1PAMQZiEVLohOItvZe?= =?us-ascii?Q?i8KujUZNQALRGXFpmUzUkNWuQw5Qbh4fljcHcojy2kTs+Mr60KPOWK5jRyFG?= =?us-ascii?Q?9nS/G5+hLxyAyp8qcU8Mao7JkEnVL3bRNcBqf/jZTBM7zGK/aW/qcabuLawj?= =?us-ascii?Q?ehjTsRuPvoC4sJndG3/Tzbe3SAq+KIgehzaellwQ1ymNxEglUPVFkCLptg85?= =?us-ascii?Q?rlfEfRfMQ=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; SN2PR07MB2526; 6:KmJBdNQRA5F53jmBEFX5AzUwVH+8zkQaE9cPIQ1PhsLYMs5iuwncx4kCvB3OqFSg8Rwf6hDd+D7FU2C0SQ8mQrlOWhqeJyeeIAubFkY2q7wSgIvHECThRxeQ4/wkd3dERKbU0yQ/BcPjlP0m8Y2O9EwGk6BY95aSdHPNS+icmwwtac+tByRNuAU3rx3znkXoHSKaHIzIr7FOjgR1THtMZ8DMOfK7xc23aNwPRDjCVkHw57HFc/77k2O0FSW4py8ptlQnriZssO7oGsKG8ATAaGaZXipz9Q4aqkjP1K6ubIkt3y6aevd3M4jXCl4XL++2t6CCd7ieqME//otg4on6vA==; 5:5GOzls/dlLhSzH9Ti9Ci4acxI2gBp7wIQLkbEzzfKOp7zHsLkugR5IFbwSYPFhHVeJCe/WlEyabdasUPuLxaDh4CXKBM4SSARJGSFqDnxS7U94VNGY3pzFy8PeYpKDD38ObC/+RYl61CMbjze/IQNQ==; 24:NAT2InkD2sZMiWBnV6XTuyHCHS4BwXYKhUbpxIz5FjxdjFjVZ38GpEyyVB4B9oMgm0b8hfei3/KaX7HjTjMJyIk/Wq/bf/VXXddYYmvsg3Q=; 7:TxgVpKy5Ug3XugKH+KoqmoNX42uyRg2+lFJ/kf7dZw7wutwQwJ+orgbiUGXwg2102HKQFdsSPp1LGg/TBt29Z3PhE+tt12wwSm8nxLDumAQQUpxoRw8W9r7JNYh3S2eFYd0P5qXtfedRfEuubHLPkqvGvaLwnzGlDtw7gI6iHZq4ToruvctGrVLENJTAS5hcRjwPonyOY258JwizPiILgHO0MjnorQeDFfm9qoV+1yE= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Oct 2017 17:23:33.3448 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN2PR07MB2526 Subject: Re: [dpdk-dev] [PATCH] ring: guarantee ordering of cons/prod loading when doing enqueue/dequeue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Oct 2017 17:23:39 -0000 -----Original Message----- > Date: Thu, 12 Oct 2017 17:05:50 +0000 > From: "Ananyev, Konstantin" > To: Olivier MATZ , Jia He > CC: "dev@dpdk.org" , "jia.he@hxt-semitech.com" > , "jie2.liu@hxt-semitech.com" > , "bing.zhao@hxt-semitech.com" > , "jerin.jacob@caviumnetworks.com" > > Subject: RE: [PATCH] ring: guarantee ordering of cons/prod loading when > doing enqueue/dequeue > > Hi guys, > > > -----Original Message----- > > From: Olivier MATZ [mailto:olivier.matz@6wind.com] > > Sent: Thursday, October 12, 2017 4:54 PM > > To: Jia He > > Cc: dev@dpdk.org; jia.he@hxt-semitech.com; jie2.liu@hxt-semitech.com; bing.zhao@hxt-semitech.com; Ananyev, Konstantin > > ; jerin.jacob@caviumnetworks.com > > Subject: Re: [PATCH] ring: guarantee ordering of cons/prod loading when doing enqueue/dequeue > > > > Hi, > > > > On Tue, Oct 10, 2017 at 05:56:36PM +0800, Jia He wrote: > > > Before this patch: > > > In __rte_ring_move_cons_head() > > > ... > > > do { > > > /* Restore n as it may change every loop */ > > > n = max; > > > > > > *old_head = r->cons.head; //1st load > > > const uint32_t prod_tail = r->prod.tail; //2nd load > > > > > > In weak memory order architectures(powerpc,arm), the 2nd load might be > > > reodered before the 1st load, that makes *entries is bigger than we wanted. > > > This nasty reording messed enque/deque up. > > > > > > cpu1(producer) cpu2(consumer) cpu3(consumer) > > > load r->prod.tail > > > in enqueue: > > > load r->cons.tail > > > load r->prod.head > > > > > > store r->prod.tail > > > > > > load r->cons.head > > > load r->prod.tail > > > ... > > > store r->cons.{head,tail} > > > load r->cons.head > > > > > > THEN,r->cons.head will be bigger than prod_tail, then make *entries very big > > > > > > After this patch, the old cons.head will be recaculated after failure of > > > rte_atomic32_cmpset > > > > > > There is no such issue in X86 cpu, because X86 is strong memory order model > > > > > > Signed-off-by: Jia He > > > Signed-off-by: jia.he@hxt-semitech.com > > > Signed-off-by: jie2.liu@hxt-semitech.com > > > Signed-off-by: bing.zhao@hxt-semitech.com > > > > > > --- > > > lib/librte_ring/rte_ring.h | 8 ++++++++ > > > 1 file changed, 8 insertions(+) > > > > > > diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h > > > index 5e9b3b7..15c72e2 100644 > > > --- a/lib/librte_ring/rte_ring.h > > > +++ b/lib/librte_ring/rte_ring.h > > > @@ -409,6 +409,10 @@ __rte_ring_move_prod_head(struct rte_ring *r, int is_sp, > > > n = max; > > > > > > *old_head = r->prod.head; > > > + > > > + /* load of prod.tail can't be reordered before cons.head */ > > > + rte_smp_rmb(); > > > + > > > const uint32_t cons_tail = r->cons.tail; > > > /* > > > * The subtraction is done between two unsigned 32bits value > > > @@ -517,6 +521,10 @@ __rte_ring_move_cons_head(struct rte_ring *r, int is_sc, > > > n = max; > > > > > > *old_head = r->cons.head; > > > + > > > + /* load of prod.tail can't be reordered before cons.head */ > > > + rte_smp_rmb(); > > > + > > > const uint32_t prod_tail = r->prod.tail; > > > /* The subtraction is done between two unsigned 32bits value > > > * (the result is always modulo 32 bits even if we have > > > -- > > > 2.7.4 > > > > > > > The explanation convinces me. > > > > However, since it's in a critical path, it would be good to have other > > opinions. This patch reminds me this discussion, that was also related to > > memory barrier, but at another place: > > http://dpdk.org/ml/archives/dev/2016-July/043765.html > > Lead to that patch: http://dpdk.org/browse/dpdk/commit/?id=ecc7d10e448e > > But finally reverted: http://dpdk.org/browse/dpdk/commit/?id=c3acd92746c3 > > > > Konstatin, Jerin, do you have any comment? > > For IA, as rte_smp_rmb() is just a compiler_barrier, that patch shouldn't make any difference, > but I can't see how read reordering would screw things up here... > Probably just me and arm or ppc guys could explain what will be the problem > if let say cons.tail will be read before prod.head in __rte_ring_move_prod_head(). > I wonder Is there a simple test-case to reproduce that problem (on arm or ppc)? > Probably new test-case for rte_ring autotest is needed, or is it possible to reproduce > it with existing one? On the same lines, Jia He, jie2.liu, bing.zhao, Is this patch based on code review or do you saw this issue on any of the arm/ppc target? arm64 will have performance impact with this change. > Konstantin