From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM01-BY2-obe.outbound.protection.outlook.com (mail-by2nam01on0061.outbound.protection.outlook.com [104.47.34.61]) by dpdk.org (Postfix) with ESMTP id 3D9BB1AEF4 for ; Fri, 20 Oct 2017 07:43:45 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=0+saVl3oyARaCfhZ53ktZCWMhA2g6bXFMoRBwyOURMM=; b=E0JymoOzxZOMq6g5bJuVClAzh5pLU6F/XmpNMdYxHkYC9R9LhYMUZZ0BDs5dtD2CcWUjsJWRbRuLY2tIfMSl81jFz6vgZVkgbxMA4xTi1yG4IR5lV5W8cT66gXY87ISLrBOZlyLBzDJa4G/uksk0YgemqERrvCbVrKX7unJc+Rk= Received: from jerin (171.76.118.225) by CO2PR07MB2518.namprd07.prod.outlook.com (10.166.200.152) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.77.7; Fri, 20 Oct 2017 05:43:39 +0000 Date: Fri, 20 Oct 2017 11:13:21 +0530 From: Jerin Jacob To: Jia He Cc: "Ananyev, Konstantin" , "Zhao, Bing" , Olivier MATZ , "dev@dpdk.org" , "jia.he@hxt-semitech.com" , "jie2.liu@hxt-semitech.com" , "bing.zhao@hxt-semitech.com" , "Richardson, Bruce" Message-ID: <20171020054319.GA4249@jerin> References: <20171012155350.j34ddtivxzd27pag@platinum> <2601191342CEEE43887BDE71AB9772585FAA859F@IRSMSX103.ger.corp.intel.com> <20171012172311.GA8524@jerin> <2601191342CEEE43887BDE71AB9772585FAAB171@IRSMSX103.ger.corp.intel.com> <8806e2bd-c57b-03ff-a315-0a311690f1d9@163.com> <2601191342CEEE43887BDE71AB9772585FAAB404@IRSMSX103.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772585FAAB570@IRSMSX103.ger.corp.intel.com> <3e580cd7-2854-d855-be9c-7c4ce06e3ed5@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <3e580cd7-2854-d855-be9c-7c4ce06e3ed5@gmail.com> User-Agent: Mutt/1.9.1 (2017-09-22) X-Originating-IP: [171.76.118.225] X-ClientProxiedBy: PN1PR01CA0099.INDPRD01.PROD.OUTLOOK.COM (10.174.144.15) To CO2PR07MB2518.namprd07.prod.outlook.com (10.166.200.152) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0968fc1c-60d6-4da9-5007-08d5177d861b X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(4534020)(4602075)(4627075)(201703031133081)(201702281549075)(2017052603229); SRVR:CO2PR07MB2518; X-Microsoft-Exchange-Diagnostics: 1; CO2PR07MB2518; 3:X9F4XKL2G4tWD5l/mfANsRk5e9lkMjv7I+RP6bmmCDkqV2By6UPkhNlDrex4jvSj21gF7L4qKr3z2NdDMuSdc8O3lYWdtkxAieHKQm24SIVHagqndSzZ4sWG5yeAxnYXftpCQ75VmrZrYWQywqkDYI8WsM0xJCkL67/uMMrZ3GBxSwfN7iU2wRFFWMNj/5wNhG96GXYou9VpWeQLXgs8yCeO0cvMLzfdC2MpyK/c1Qb2k53x4WgKAdNjWxroR+8p; 25:kE3BIOC0J9cEPQHdTTnkqsfHjcSXiLu6hIrWrUlMP2/JpPLASxvIpLEAT4Xlw8rbzBnPX5vtZOltT78QmQkKB5y9RxMVYv0/D4n3hdtQ9NcAv2ZhL5P89KD4pgIhFSK94Pb1nIYJ9M+Lo0wIB9+tROEc4fCNZSIJ1mHuWvUzmzIwPAnPsA3oqBTf/S/iNQ0ONCmAi1nwk466a865cOgBOyyCG1ascveyif8ZrGKtSGm9av5VIN7w0Eieha2v8nhFhe2Dpx90OyOBRLxDRSIZqOMJKYalDQsgOqhnNFcRVtD7GkJEHVRreOS8+wXdwz9a4b/Q5upKI256WxuCLgVtXQ==; 31:bJApOqr8/bnPcOvlZM0SUIX0nhNSooQuQ9hOyaN2w1GBtgGXb0Y8TYCor0VfKv59XBpVzz2X59ttZWfinIL8iXh34D4PjynLK65rycPiz5uiG/Ad1JPyiqOd7MPJVU42kw3mOZM48/g2NsjtAK6/lZ/2SdigN8V3ZgRia0VLWQTl8kHu8V1MCofajc/0/rB6DAO8z6N7fcsyLaE6Z0kSdZEMjdkb2K2Mwo2v5t9O6j8= X-MS-TrafficTypeDiagnostic: CO2PR07MB2518: Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.JacobKollanukkaran@cavium.com; X-Microsoft-Exchange-Diagnostics: 1; CO2PR07MB2518; 20:vgSSJBznF5FUrrGNtgKtUrJJXuUhD9O8FI7tbo/ynh3hLype7P2bclrYtZCud6ukTqvQ4YSAoxL1/AyRL5Wh2mwegQzdKgN+cwruad+Vq0VxzYJmrKtbKwuOC5TD/cohMz/vXYFMZi25B68AQQXMZRxfYcsDAKPXmYM6GS28zLDIcwuRr2cWWnT/nJ9lDEaslQmnrNmPFEgINqcrbVbEOwY2upofFQ5wB3BxscTuJtDgBxhZDDB6vEvFVBe39a7wa6tsryXTIoBbecY8T81naoaXfClUK+0mDdDG2t2A6/Oeks8xLlt2cjtqMRoPB9KcC9P7FYvEyjKifZzKaMvcTW//u10OD1BAkjAjoDA1WYerJPn+R5QZj6+3H9rEk5b7uuO6YBir5mEKGifkzicUGkMIXx71yEknxGWasRhHf3eJxZyLqm4bHf7busWmSOSJxH8DWUZX7nNUkl83xIdm2DJ971FhoqNXahzyWowYAJhd67vz+3jHsP7my6yTWZMcmYpxsppyoUd4mu1I1H+5IdGdwtyCLS73JxaFq/hyhgCYbk+6SwWj5lqU42TYlZBZEffqZZ+94C4v0TJLiSMXoCJjaGvUeu1//quisq5/qho= X-Exchange-Antispam-Report-Test: UriScan:(158342451672863)(166708455590820)(130843839470238)(228905959029699); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(10201501046)(3231020)(93006095)(3002001)(100000703101)(100105400095)(6041248)(20161123560025)(20161123558100)(20161123555025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:CO2PR07MB2518; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:CO2PR07MB2518; X-Microsoft-Exchange-Diagnostics: 1; CO2PR07MB2518; 4:EhoPfhbaGEm1MT2KllUKlDnncKiILwGYdTLJR81wsHlG/bWmdFaW12sMtwr15GqgoSqOrWAvn1aOjZrbYw0kJxkaHufqlOBbg19oXZCL2yIHG0DA8jyPREyKzMEuucVX75Jq//50MP4QmRarQ6Lwl/m515e+1GJDR/SJKzhcyTlZcyvb7hbCD2xSRAI5pyQSm/T+hwIooejAjM9NWOIif+zkEs5R6E5pByNJm8hif6MIg26VOEF0P0aw+2eIhSRu/hSYw+rXRR6ai/TiViu794P5dJNRctLkN/SaoyQPyxBA5cJQIuQkGsxb7+3FA2cYHrUjDKFnxCPL1FmhYq6PUtYfOdsKZ6dlWOhFxnN4ZVXftEJ8LabRdnD6O9A+A9C5o9dk6NKa2nOXMcNVGjBwzg== X-Forefront-PRVS: 0466CA5A45 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(346002)(376002)(24454002)(13464003)(199003)(189002)(66066001)(1411001)(316002)(966005)(33656002)(6496005)(47776003)(2950100002)(42882006)(6306002)(53546010)(97736004)(93886005)(106356001)(105586002)(1076002)(6666003)(6116002)(6916009)(53936002)(39060400002)(2906002)(16526018)(8936002)(23756003)(58126008)(229853002)(2870700001)(54906003)(81156014)(9686003)(81166006)(50466002)(8676002)(189998001)(53376002)(25786009)(68736007)(478600001)(50986999)(101416001)(54356999)(3846002)(305945005)(4326008)(76176999)(55016002)(33716001)(7736002)(6246003)(5660300001)(83506002)(72206003)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:CO2PR07MB2518; H:jerin; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?iso-8859-1?Q?1; CO2PR07MB2518; 23:gmjyAZc4wDiz0P6ynKbLdrKi9tvVriqzR+BYv13?= =?iso-8859-1?Q?PMbjxkiuKRhHYkHx5AFUK9PVkLACa8PKH/ulLpJHiutx32Lh/uFwR+CqcO?= =?iso-8859-1?Q?CiWCnyySQ+go5FRc8Dw+kds+t0j4AOyYt0BVOhtp5HGfqro38tuJ1QfFQA?= =?iso-8859-1?Q?DvTS9AfD8JpDklk08W85krAQs3OziMDD3rK400gYDdMAOoJN538P4sY0f2?= =?iso-8859-1?Q?pTA2KwramDCSVVvxuDfj+C4NuakyBQ41gQiJUHRRmlQca8bXAA4q5t9AnY?= =?iso-8859-1?Q?GIJTrGTDQwmUZrb3HmowpvxLIO5GsQVsvIOFu3W9ZMW8yasp3w9i/kFOAJ?= =?iso-8859-1?Q?YHIpInjI9P7SBnrxgJpJC9kFwIgh+xED3PcaFPuD+j3M/due6H244hMu76?= =?iso-8859-1?Q?YUSxNzJuG+6cnMdc6pWvTiOk9j3m/ynujczfXnQLdQozG1Iu3xZOA0lM/k?= =?iso-8859-1?Q?YB5ap/f36qdecSPcRO8f4WDGrk3ilO1BY6nvueGIif8quXU/G6iuTlVlqT?= =?iso-8859-1?Q?Ov77Kg7wvlN5o1qfMZoCbr5NPsOAEB8glMOx6XOBZ7SLvkcLEZtKLgVk8b?= =?iso-8859-1?Q?SAVvhyzGjytsCSV5YYs+NXbnZVFGlQNg9Yo0ehFo5quPOOcj2JJ6elxo2p?= =?iso-8859-1?Q?4f9vmVaZ5g1mvovubK7hNtwof9LTSnmtMkXMekIuoFcPo+8ehzzBJktRm9?= =?iso-8859-1?Q?6xgIw5VjqvnfSARNmjunVXOg2lD/HwK7K1sH6B4+7Dg8h5fpjlu8UXYUXs?= =?iso-8859-1?Q?vDTlU8Ib9cw3ggYe/HnE9/zCuv8WfZDANzHWs3BATF9tKMlM8esHcdzHqB?= =?iso-8859-1?Q?TqGSVt7ihxaKYMpbSKXtwiWG6Uyr+HId29R/Ovwxzh7havY1aLYP8vac9b?= =?iso-8859-1?Q?5oIoXbrERM4IKS0T3NWZftvXNBg1eHOs+U0BV+4Zc5QcUFEtyq2ER7Qehg?= =?iso-8859-1?Q?DXIPpxpvk3pW8Zkc6immOG+s9LyXyydeHnpt7l3hJflwU41Z6oTS4eLeXE?= =?iso-8859-1?Q?voTohlF9AxgcvGBNWcpUxJsyXirPmWccgo45lvcGDCbCdabSLJzRKjrpHu?= =?iso-8859-1?Q?hBqQAkh0yIkH2Kpu+2pKztUdYJ96XzgGGEHASS4N8VZNRIFo4Lk7r85mcs?= =?iso-8859-1?Q?uQui0IowX/EbF+RvUu8vTD+idoEJeXVx39CQjjgS0pvj14WPr1NLjploEo?= =?iso-8859-1?Q?vVNneLfghCipr6cX3xXByumTjz1tfn0gpL6/STvThn9Uzrnr1BZhL2a2bn?= =?iso-8859-1?Q?wTm1jfsMDRcAL0voiRP6OjM6qwfcQBURo4o7tGGcqc9J1I1wgaXcNQNi4N?= =?iso-8859-1?Q?OZx0dtZ9HJ9wQrlf28lzNpdZ65RX6uetGgiPL4kKjbg8y2F7JVD7wzoDgE?= =?iso-8859-1?Q?iga8pir/Pxwi9feKHz+ioh3jOvzztgQPk94lwuYPwJoFD71jawYiFT88og?= =?iso-8859-1?Q?YKMNu2UNBr5JUS7F46VlHEJgqEnJaLiTj1d?= X-Microsoft-Exchange-Diagnostics: 1; CO2PR07MB2518; 6:ayLIvcMzCcyRvt3Ph499nZW4ZinM5fodwOCpULTcZIkKv+f9E0u77XUoUQs/+hvEqHuDkZWzEIqp7x+9VWyzyNUkAraYL1vH+GEXbHS+OCbShQZT0GsUr5DaV0OHzOHqBgPEyEb9R31nljpWUlM18Y7+RSOiwOgijmOOqyW4fODQsSi61+mFxjneLY3WRQ5g1LYCI+MyK9fi7wRx+7EocF7Lv8vv9kcjuKyZpRqm+/m+5guvTOUYwtgERa1rWHqp9ip1YLmroPif1O1Npv8mVZsTA2+gGYxRghsR8bhG/zGeBhz3EmYa9vnD9IyV1TVElV/D3ZGLHmdzDyMXQdyiWw==; 5:zo1Z8K4Vg2xN0bvfiVptarAxw4NRAkcjVlRkisGWVPzycCM1ici+yxTuVVboy3OoBJ9SV/CtT5skIWPuCt1lDAET5UlElUR4XkmLz/SojkKdNl27XTxSX/Us1cq6htu4e2yKdf1ECgkaFAfPozfqDg==; 24:f11YpvBjH8lEHa4UXDDVuq/pniUnVlpYIYT/tu2SEKhlsXnl3KR8q5bRKrwhxywpc4yIO4LZkZRHWihyjXc0PF1H3HuVJhvS/6Jw3HCldD0=; 7:dLWgA0l78kn2fOtbeJuIw9Ji2YPbHJ1IQR/oy8n2HQdj3sK/F+maA/2qvC6yiT4sOuAqGL0GJAVL7ciomGnQErUBx0mz6YHbjh8KVexoMPGp5TTDsLAO0P3F3AnBld6ACdKivT8oF45UpG0Jnd+XdKTv2Mx4dXzMmYh1aIYZisPf7HH9ghkOXgk0iWYEMhUUjvobQqfEzuY4+vTOn7E+nxlPlEb4wJNbZA8N+5r1Los= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Oct 2017 05:43:39.6978 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO2PR07MB2518 Subject: Re: [dpdk-dev] [PATCH] ring: guarantee ordering of cons/prod loading when doing enqueue/dequeue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Oct 2017 05:43:47 -0000 -----Original Message----- > Date: Fri, 20 Oct 2017 09:57:58 +0800 > From: Jia He > To: "Ananyev, Konstantin" , "Zhao, Bing" > , Jerin Jacob > Cc: Olivier MATZ , "dev@dpdk.org" , > "jia.he@hxt-semitech.com" , > "jie2.liu@hxt-semitech.com" , > "bing.zhao@hxt-semitech.com" , "Richardson, > Bruce" > Subject: Re: [dpdk-dev] [PATCH] ring: guarantee ordering of cons/prod > loading when doing enqueue/dequeue > User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 > Thunderbird/52.4.0 > > > > On 10/20/2017 4:02 AM, Ananyev, Konstantin Wrote: > > > > > -----Original Message----- > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ananyev, Konstantin > > > Sent: Thursday, October 19, 2017 3:15 PM > > > To: Zhao, Bing ; Jia He ; Jerin Jacob > > > Cc: Olivier MATZ ; dev@dpdk.org; jia.he@hxt-semitech.com; jie2.liu@hxt-semitech.com; bing.zhao@hxt- > > > semitech.com > > > Subject: Re: [dpdk-dev] [PATCH] ring: guarantee ordering of cons/prod loading when doing enqueue/dequeue > > > > > > > Hi, > > > > > > > > On 2017/10/19 18:02, Ananyev, Konstantin wrote: > > > > > Hi Jia, > > > > > > > > > > > Hi > > > > > > > > > > > > > > > > > > On 10/13/2017 9:02 AM, Jia He Wrote: > > > > > > > Hi Jerin > > > > > > > > > > > > > > > > > > > > > On 10/13/2017 1:23 AM, Jerin Jacob Wrote: > > > > > > > > -----Original Message----- > > > > > > > > > Date: Thu, 12 Oct 2017 17:05:50 +0000 > > > > > > > > > > > > > > > [...] > > > > > > > > On the same lines, > > > > > > > > > > > > > > > > Jia He, jie2.liu, bing.zhao, > > > > > > > > > > > > > > > > Is this patch based on code review or do you saw this issue on any of > > > > > > > > the > > > > > > > > arm/ppc target? arm64 will have performance impact with this change. > > > > > > sorry, miss one important information > > > > > > Our platform is an aarch64 server with 46 cpus. > > > > > > If we reduced the involved cpu numbers, the bug occurred less frequently. > > > > > > > > > > > > Yes, mb barrier impact the performance, but correctness is more > > > > > > important, isn't it ;-) > > > > > > Maybe we can  find any other lightweight barrier here? > > > > > > > > > > > > Cheers, > > > > > > Jia > > > > > > > Based on mbuf_autotest, the rte_panic will be invoked in seconds. > > > > > > > > > > > > > > PANIC in test_refcnt_iter(): > > > > > > > (lcore=0, iter=0): after 10s only 61 of 64 mbufs left free > > > > > > > 1: [./test(rte_dump_stack+0x38) [0x58d868]] > > > > > > > Aborted (core dumped) > > > > > > > > > > > > So is it only reproducible with mbuf refcnt test? > > > > > Could it be reproduced with some 'pure' ring test > > > > > (no mempools/mbufs refcnt, etc.)? > > > > > The reason I am asking - in that test we also have mbuf refcnt updates > > > > > (that's what for that test was created) and we are doing some optimizations here too > > > > > to avoid excessive atomic updates. > > > > > BTW, if the problem is not reproducible without mbuf refcnt, > > > > > can I suggest to extend the test with: > > > > > - add a check that enqueue() operation was successful > > > > > - walk through the pool and check/printf refcnt of each mbuf. > > > > > Hopefully that would give us some extra information what is going wrong here. > > > > > Konstantin > > > > > > > > > > > > > > Currently, the issue is only found in this case here on the ARM > > > > platform, not sure how it is going with the X86_64 platform > > > I understand that it is only reproducible on arm so far. > > > What I am asking - with dpdk is there any other way to reproduce it (on arm) > > > except then running mbuf_autotest? > > > Something really simple that not using mbuf/mempool etc? > > > Just do dequeue/enqueue from multiple threads and check data integrity at the end? > > > If not - what makes you think that the problem is precisely in rte_ring code? > > > Why not in rte_mbuf let say? > > Actually I reread your original mail and finally get your point. > > If I understand you correctly the problem with read reordering here is that > > after we read prot.tail but before we read cons.head > > both cons.head and prod.tail might be updated, > > but for us prod.tail change might be unnoticed. > > As an example: > > time 0 (cons.head == 0, prod.tail == 0): > > prod_tail = r->prod.tail; /* due read reordering */ > > /* prod_tail == 0 */ > > > > time 1 (cons.head ==5, prod.tail == 5): > > *old_head = r->cons.head; > > /* cons.head == 5 */ > > *entries = (prod_tail - *old_head); > > /* *entries == (0 - 5) == 0xFFFFFFFB */ > > > > And we will move our cons.head forward, even though there are no filled entries in the ring. > > Is that right? > Yes > > As I side notice, I wonder why we have here: > > *entries = (prod_tail - *old_head); > > instead of: > > *entries = r->size + prod_tail - *old_head; > > ? > Yes, I agree with you at this code line. > But reordering will still mess up things even after this change(I have > tested, still the same as before) > I thought the *entries is a door to prevent consumer from moving forward too > fast than the producer. > But in some cases, it is correct that prod_tail is smaller than *old_head > due to  the cirular queue. > In other cases, it is incorrect because of read/read reordering. > > AFAICT, the root cause here is the producer tail and cosumer head are > dependant on each other. > Thus a memory barrier is neccessary. Yes. The barrier is necessary. In fact, upstream freebsd fixed this issue for arm64. DPDK ring implementation is derived from freebsd's buf_ring.h. https://github.com/freebsd/freebsd/blob/master/sys/sys/buf_ring.h#L166 I think, the only outstanding issue is, how to reduce the performance impact for arm64. I believe using accurate/release semantics instead of rte_smp_rmb() will reduce the performance overhead like similar ring implementations below, freebsd: https://github.com/freebsd/freebsd/blob/master/sys/sys/buf_ring.h#L166 odp: https://github.com/Linaro/odp/blob/master/platform/linux-generic/pktio/ring.c Jia, 1) Can you verify the use of accurate/release semantics fixes the problem in your platform? like use of atomic_load_acq* in the reference code. 2) If so, What is the overhead between accurate/release and plane smp_smb() barriers. Based on that we need decide what path to take. Note: This issue wont come in all the arm64 implementation. it comes on arm64 implementation with OOO(out of order) implementations. > > Cheers, > Jia > > > > > Konstantin > > > > > > . In another > > > > mail of this thread, we've made a simple test based on this and captured > > > > some information and I pasted there.(I pasted the patch there :-)) > > > Are you talking about that one: > > > http://dpdk.org/dev/patchwork/patch/30405/ > > > ? > > > It still uses test/test/test_mbuf.c..., > > > but anyway I don't really understand how mbuf_autotest supposed > > > to work with these changes: > > > @@ -730,7 +739,7 @@ test_refcnt_iter(unsigned int lcore, unsigned int iter, > > > rte_ring_enqueue(refcnt_mbuf_ring, m); > > > } > > > } > > > - rte_pktmbuf_free(m); > > > + // rte_pktmbuf_free(m); > > > } > > > @@ -741,6 +750,12 @@ test_refcnt_iter(unsigned int lcore, unsigned int iter, > > > while (!rte_ring_empty(refcnt_mbuf_ring)) > > > ; > > > > > > + if (NULL != m) { > > > + if (1 != rte_mbuf_refcnt_read(m)) > > > + printf("m ref is %u\n", rte_mbuf_refcnt_read(m)); > > > + rte_pktmbuf_free(m); > > > + } > > > + > > > /* check that all mbufs are back into mempool by now */ > > > for (wn = 0; wn != REFCNT_MAX_TIMEOUT; wn++) { > > > if ((i = rte_mempool_avail_count(refcnt_pool)) == n) { > > > > > > That means all your mbufs (except the last one) will still be allocated. > > > So the test would fail - as it should, I think. > > > > > > > And > > > > it seems that Juhamatti & Jacod found some reverting action several > > > > months ago. > > > Didn't get that one either. > > > Konstantin >