From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-ve1eur01on0053.outbound.protection.outlook.com [104.47.1.53]) by dpdk.org (Postfix) with ESMTP id A2B1E8DF0 for ; Wed, 25 Apr 2018 18:44:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=8I8wipS6UjI+ionBzWG39mFBAmAJKt9agk2T497IKsg=; b=ebRmACxKuIj6Yn+5kQG4E78YLvSrQlIo3q0RxVas6mK0tlRMt4nIb5EYCs42yXFKUXYLLHIFBkvfQ6HSGYWIwXaanaLznVvHdTL0Ej12FE6bMzXmWMogs2XmHfYJy/CQtfS3OmPAAHCSYh1MBFmAv3gHG2eOBaRAk7xlZSTd0cg= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; Received: from yongseok-MBP.local (209.116.155.178) by DB6PR0501MB2038.eurprd05.prod.outlook.com (2603:10a6:4:6::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.696.13; Wed, 25 Apr 2018 16:44:31 +0000 Date: Wed, 25 Apr 2018 09:44:17 -0700 From: Yongseok Koh To: "Ananyev, Konstantin" Cc: "Lu, Wenzhuo" , "Wu, Jingjing" , "olivier.matz@6wind.com" , "dev@dpdk.org" , "adrien.mazarguil@6wind.com" , "nelio.laranjeiro@6wind.com" Message-ID: <20180425164416.GA3268@yongseok-MBP.local> References: <20180310012532.15809-1-yskoh@mellanox.com> <20180419011105.9694-1-yskoh@mellanox.com> <2601191342CEEE43887BDE71AB977258AE91994F@IRSMSX102.ger.corp.intel.com> <20180424020427.GA83470@yongseok-MBP.local> <2601191342CEEE43887BDE71AB977258AEBCF960@IRSMSX102.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2601191342CEEE43887BDE71AB977258AEBCF960@IRSMSX102.ger.corp.intel.com> User-Agent: Mutt/1.9.3 (2018-01-21) X-Originating-IP: [209.116.155.178] X-ClientProxiedBy: CY4PR04CA0065.namprd04.prod.outlook.com (2603:10b6:910:4f::30) To DB6PR0501MB2038.eurprd05.prod.outlook.com (2603:10a6:4:6::20) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(48565401081)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:DB6PR0501MB2038; X-Microsoft-Exchange-Diagnostics: 1; DB6PR0501MB2038; 3:in30dgnBMTalmWDjQiIP5tGHNCInWJW6sn3fHOYMpJjKl80CMq7uUm8odrRNbCfNMlaFj/g4zLSD2NKLygUgBmI2V9Jlja/u9X9s6wM8Sls/WGtHAyaYgYZPhmrybo2FpaUOA0lYnumhTA58h6/yBByFiKWpQqTly1TM7oRsFMbRWXF0IjXP823ECUJBtduH43Bwxd2J6C5tpN0gTA/aV2wb9E/enE1I+nFx2eHLNtfLO+xksS8+RvB+7aELsVrM; 25:MGYlXy10gryuqxJutkOU1idqcSE1nh3oieUxXG5lEG/wS2qmGW7HW5Cjd5mtLnSNjSW79I1yAsoVasUP3jGxxgqu7LNz+X/8S6GQlZlaDsw9pcJw9brqHgvsALf4ddu2hvwSqUp02gngK//46769Kbgna/emJCxEGXebVEE1kTDoOg0FuCMW8+jQfkAK6EdSaFmMhNbaGCldicQqK8Jmu/xhg1821OKO6MeW6YfrQTagkRX9wnLkbnrUlstyZ0s53ZlQLXUqSsjT+sKMQFGQS0ixeypZkZLwYxYuFz8FZIqXodJ+h5KoF2qeRfKTHIUbJXk/2eyzEFQqr4jXBc7IfQ==; 31:h/RmCX2pX0fQA+zyyT+2AfVQ5q72/P9yNOEbR38iSkwUSkX8pXG4KXCkktaZng9cLbyLvpurMX55tLZiQ6M6vyzwF6p4M5F/79fKBpCKUdPMDugZzV974lHT1Kg8xiD1uQQUj8P+aoW39Vk2fE1GcrY5e7KqArbPkE33+n7bunfTvZc6FpQjPdtacbLJk0eAGUjjg1iUM2gR039Wd00tfdK7Rgkw/WwoYDAb5Jl7/sg= X-MS-TrafficTypeDiagnostic: DB6PR0501MB2038: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; DB6PR0501MB2038; 20:aKGrQIUmJcH8ma0zxb3brB9GTNIo2UoywSzOw4E14coKr6JvhEBAmEPlgvq02UbYtDUUM1cNCNCEqosP3SfRzFNOVAMZtSXj7jbd9R8vzIb4RqnVCWxij3yJ7SsK9HiJO1GfxR7bryWxZWgKKZ5uUDWA0B6vg+9PYp6FfmCsyqWp95AT1onTdxNn5OpbG14ndS7Rg5rjkER44GyQ3EV+mMhvbxJ+eXA9Vw1cGlZP+DKMO09weBurBvzoB4jftjnKYKEhG+2z+Ikp/ZxdMMbQNJt78v2he3P6XwyLA6Vd58kloTDUeJ+i3Qju4YJL+f60KkuTim7R/Wc6LN6ICjfw4DQ9xyfpZeu2rWnCoew36ox8mIR4TGL7M/uuWUxeDxJJEdOHX9bHEyPPolYAfvixUDGXbvqO/QW7DC5NNEaeC7FPSXOegVUJvsws4tXKCohkDOvBqnxFBacPhFwUmz04utKUBoL9G287JUVWdX3VUkBqg/Slun5+glpdYRIDghIg; 4:eQ7ZcetPk8JqCKxR2N0aEAzjdHqOzzzXr0dX8YGuPtT2cJxbG1sJg3Mi12tSv61IBewsSmIPq/R514m8X/iOlJBpCa3y2F44QP4CFYNFe6SDLLe06bI9+xQZOo0NDI/44lXqI096rgxezBIfPZuqUz60aZuaofg0UNgBe8wlJokyuQd1thg/c5j3nbibK9pspGVfZOK8g8SH/GfdqP8VDLOVpbq5vYhl+XHivHo9R77258GwZLDKlMsuTSdIC7gT6lsmApzJWRWBpGL8rTrBBZ0V1PasoxSbuxY8dPeJE4hu243nOE5cwGrdJbkNhpAeBQWqG53k8DBuExxu9ON5F7SbKxL5kM8n9kac0hLqmPs= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(131327999870524)(17755550239193); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(10201501046)(93006095)(93001095)(3231232)(944501410)(52105095)(3002001)(6055026)(6041310)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123562045)(20161123558120)(20161123564045)(6072148)(201708071742011); SRVR:DB6PR0501MB2038; BCL:0; PCL:0; RULEID:; SRVR:DB6PR0501MB2038; X-Forefront-PRVS: 06530126A4 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(346002)(396003)(39860400002)(366004)(376002)(39380400002)(189003)(199004)(1076002)(66066001)(7736002)(58126008)(305945005)(6116002)(105586002)(3846002)(345774005)(4326008)(16586007)(9686003)(26005)(316002)(476003)(55016002)(956004)(11346002)(446003)(52116002)(23726003)(5890100001)(7696005)(5660300001)(97736004)(98436002)(53936002)(16526019)(6506007)(33896004)(186003)(33656002)(59450400001)(386003)(8676002)(54906003)(106356001)(68736007)(229853002)(47776003)(2906002)(486006)(25786009)(86362001)(6666003)(6246003)(93886005)(81166006)(81156014)(50466002)(76176011)(478600001)(6916009)(8936002)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:DB6PR0501MB2038; H:yongseok-MBP.local; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DB6PR0501MB2038; 23:Sclg6oBCGqvhnrffMqZ7dj3YEx/IQGiB8UoBu8c?= =?us-ascii?Q?jVdz2dxfOIfSr6INGmu/sbGD7A331tWAjEhMGcrP8QSsdRAti0RHfVWRpTU7?= =?us-ascii?Q?d03lMKa8DlgJeG5jZ4Os6dCftpxsWOBKv3275zaQusKv94d0Xy4Lvumod6EK?= =?us-ascii?Q?fGx2lzhNMG0mGgRGXdU3DNlvs2/zgPReFj/lntvAl/2/RsrobGFnGD9EbaJq?= =?us-ascii?Q?XQF7X2I3RQdUQXsGLJAbSsJde5qDTA3GTYGRlxsEvcS1rybCqy7ubdL02B+J?= =?us-ascii?Q?KJgQluuNj2RcPNs8xbu3g5oAYrO3WiBP5kd4OhCf5W911bjbMgEaKjeHMFDg?= =?us-ascii?Q?+iDhJAZpnbXu8a9l94e/GC0TgU+CMT9leKiNrO6Dy5qX50SzOPM2rqPzQKiw?= =?us-ascii?Q?z16aBHOgp+Rk9U7pn4oP9YVj3XZC/JKr3kuLxmTeL75Y0XnFDkyY7KM8ZOcb?= =?us-ascii?Q?WHVvZGKGRWuYqyQTQE2QkKWjKgbd3SDjPF3IhnU5d0p/On2z0YaWKK3AReTO?= =?us-ascii?Q?o7S7EOBvsj1R72Axj4o2aSf6FK4v51+k74DJRsk9/mVeqDuHeVMMwvtlchuH?= =?us-ascii?Q?kb5o3Dgdb6v2wN8Y7/iSpqPP52jvF3WYVr4VTRu5MI+xHaYDHGTah95NYQBl?= =?us-ascii?Q?cM6PZxZHxYSVukCODEadpKlTb4omoJQ6+xXIcK1XkMMQhHWNaHYyk1X3r5qN?= =?us-ascii?Q?tEQJNugGltf7O/fhvApFve9b7hC4pDK+RbcXYlIzUYzCearpeyTZARWEN/LK?= =?us-ascii?Q?xwUc4FrO1ZRFjb+C0Jl2QApeigCtFvNtTlhtdAkP4b6r6BiucWIbawXWLjV+?= =?us-ascii?Q?E6xpi6DIwIB/dmjp3bTUAjAnP/DgsySNfv2Hmv52xX8M8XgQ6xSBg7YFo5EO?= =?us-ascii?Q?Fx8hE9NYr7YLb1Z1W0aHFYJc2jdDqbOPytjAeGxXHGBuKmqSpbM8sXcKTEIJ?= =?us-ascii?Q?rC0KMDPvBDA/4WuhZqSdjIJU/RS5yuVVFhwfXaoxmeYephJgdjq6p2LDmT6/?= =?us-ascii?Q?kV+iHXI+j/8fY/RG+K3+GpC9sg5+ExrsJ1I3k5b0BNrXOssZXq4P2QnoEoxb?= =?us-ascii?Q?K3352dIQEYVzax8z+Ep2ILFdFEI03z9sOveVU14fkZXobIQKR90MCkekjeXm?= =?us-ascii?Q?5CvL9Gz/7kCBf9JB9MyRJmIxwk3cIgzpDwFqPzoMaFanXBbbPZ1Bd7vkm07U?= =?us-ascii?Q?Pi1TNdia74myvILJBnuzXMOaxRsJG9VE6wEJiDK4RRP2Y+pDX1o8X9qMusBM?= =?us-ascii?Q?BlCjo03PtSBKi4wXsOoRiCGUObI4kTKaJA5ublHN3ZoLTKA8VOBsf46POrvZ?= =?us-ascii?Q?jI5EbmBsRSIBnxJ44nXMF038BFe2L+FlK8m0X6Fhac9szqu76M2yAa7OFCUF?= =?us-ascii?Q?Ya093JFuoOakMpcaEow6QVhJgAJRIMTYWwmrOEmZcX7njMxHT?= X-Microsoft-Antispam-Message-Info: 13SkTJ7tTjQNZ3J+/rpizsFYdebV/NO1GxFL90gSAcWTdje9wfTWhL+7yeb0V/b45h+jbKB7lEdo70FhOMdZZyJI5MT0VA7EWaGJL9LW9niVWUoTmnz3OY1Jim4cKxC4NtWrJu/ASMAZHfcF+u/N0LYUADGkVccnIts1HTIyzJ39V0Tf9APx3UMw7tqAmG/G X-Microsoft-Exchange-Diagnostics: 1; DB6PR0501MB2038; 6:qQjgwn92xUdRMS/8Ej3PshxWWTP213OR9z8ZJQO8LA7bzvn4aWc3esIz2IC0RDFns4Ft/PMPb5J35K1Lnyy0n2oODc+wJiiYt/fl7kBJDJ9yv1HbGRjTI5IeKwdSB210CF6GXLQDP9K7AZ+Am2sMLu+Cm/AHrbXwcQ4g1xg84EWB9KX2TOJ3csiC34C2x3y7S4tRtJ0NXEmPqq73A5nfGg7wU2DPcksl2BEdzIlUXFo4Y5BShFtEMFsvqlLHSxbdAYDKGWoX1ABYKqGnD2YXeUHk5R5xL2ugRjHzw2X7dXmsSzNCvaRZ5l9YoTBWfoiM1dKZxW1cGsv1ViTsZnHAIEFQKjx4Qy/cgCJ15murWE6o5H2FUtBxclXFKQgnzsgD1QdXrVoEvVM1EZXAe3RNlv/GizWmQUjSG3lNzMuPiXB5n1om66pZKJjPQ0kgADwNSYP7wXklsDt/K0K1i2zjKQ==; 5:jkA0iEAKuOdS+HcNYlArwXZbio84abiFUGMRpTKWh5p1s2W0NgsAEQKFMBZ5fKR7be9Dtz7uJWbKrJ6oHrgdQmYL3QwNL2Oojk7ZgOvylSBsmXyKJf/DglOCB6GH/v474/6BoL9V/H40MmiPIi9JW8qEJD+lQdZmikqpxYtiowQ=; 24:OgNjdLVYH+7yULg4qr20oiwgCW7ai9sdrQswGd9+KgByCTfuE9RbcK/CFaUKJjDwW4HYnxHl0dHupZG+h2nPbraiLFLLsNYNGP58eQGg1b8= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; DB6PR0501MB2038; 7:7ruJjdVhmFCjrdYe2GveZ4UkP+JbKwvintmZlEqtjf9wQIOdtAVFQkNiQWVBpw3EQzz2abZZmRfbtG3XuCCna8k1yWeVtAAvTL8D5HYScXs8bwGey0ELQx7Yh3BLsmcCEwN9gEmAsU9nbdatwIFhKbPNLtzJTWgKvI2LxzZGGVEN2UMUFTmxhnaQVH4tEOBMXkTrIbRoAnyc/WEivRwuIFW5+8Jil9nzn69tLM3j/QmHvxaPPc/aZkFnZmC5g+eh X-MS-Office365-Filtering-Correlation-Id: 85bdfc13-dae0-484d-ae37-08d5aacbd26f X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Apr 2018 16:44:31.4968 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 85bdfc13-dae0-484d-ae37-08d5aacbd26f X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR0501MB2038 Subject: Re: [dpdk-dev] [PATCH v3 1/2] mbuf: support attaching external buffer to mbuf X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2018 16:44:37 -0000 On Wed, Apr 25, 2018 at 01:16:38PM +0000, Ananyev, Konstantin wrote: > > > > > > On Mon, Apr 23, 2018 at 11:53:04AM +0000, Ananyev, Konstantin wrote: > > [...] > > > > @@ -693,9 +711,14 @@ rte_mbuf_to_baddr(struct rte_mbuf *md) > > > > #define RTE_MBUF_INDIRECT(mb) ((mb)->ol_flags & IND_ATTACHED_MBUF) > > > > > > > > /** > > > > + * Returns TRUE if given mbuf has external buffer, or FALSE otherwise. > > > > + */ > > > > +#define RTE_MBUF_HAS_EXTBUF(mb) ((mb)->ol_flags & EXT_ATTACHED_MBUF) > > > > + > > > > +/** > > > > * Returns TRUE if given mbuf is direct, or FALSE otherwise. > > > > */ > > > > -#define RTE_MBUF_DIRECT(mb) (!RTE_MBUF_INDIRECT(mb)) > > > > +#define RTE_MBUF_DIRECT(mb) (!RTE_MBUF_INDIRECT(mb) && !RTE_MBUF_HAS_EXTBUF(mb)) > > > > > > As a nit: > > > RTE_MBUF_DIRECT(mb) (((mb)->ol_flags & (IND_ATTACHED_MBUF | EXT_ATTACHED_MBUF)) == 0) > > > > It was for better readability and I expected compiler did the same. > > But, if you still want this way, I can change it. > > I know compilers are quite smart these days, but you never know for sure, > so yes, I think better to do that explicitly. Okay. > > [...] > > > > /** > > > > - * Detach an indirect packet mbuf. > > > > + * @internal used by rte_pktmbuf_detach(). > > > > + * > > > > + * Decrement the reference counter of the external buffer. When the > > > > + * reference counter becomes 0, the buffer is freed by pre-registered > > > > + * callback. > > > > + */ > > > > +static inline void > > > > +__rte_pktmbuf_free_extbuf(struct rte_mbuf *m) > > > > +{ > > > > + struct rte_mbuf_ext_shared_info *shinfo; > > > > + > > > > + RTE_ASSERT(RTE_MBUF_HAS_EXTBUF(m)); > > > > + > > > > + shinfo = rte_mbuf_ext_shinfo(m); > > > > + > > > > + if (rte_extbuf_refcnt_update(shinfo, -1) == 0) > > > > + shinfo->free_cb(m->buf_addr, shinfo->fcb_opaque); > > > > > > > > > I understand the reason but extra function call for each external mbuf - seems quite expensive. > > > Wonder is it possible to group them somehow and amortize the cost? > > > > Good point. I thought about it today. > > > > Comparing to the regular mbuf, maybe three differences. a) free function isn't > > inlined but a real branch. b) no help from core local cache like mempool's c) no > > free_bulk func like rte_mempool_put_bulk(). But these look quite costly and > > complicated for the external buffer attachment. > > > > For example, to free it in bulk, external buffers should be grouped as the > > buffers would have different callback functions. To do that, I have to make an > > API to pre-register an external buffer group to prepare resources for the bulk > > free. Then, buffers can't be anonymous anymore but have to be registered in > > advance. If so, it would be better to use existing APIs, especially when a user > > wants high throughput... > > > > Let me know if you have better idea to implement it. Then, I'll gladly take > > that. Or, we can push any improvement patch in the next releases. > > I don't have any extra-smart thoughts here. > One option I thought about - was to introduce group of external buffers with > common free routine (I think o mentioned it already). > Second - hide all that external buffer management inside mempool, > i.e. if user wants to use external buffers he create a mempool > (with rte_mbuf_ext_shared_info as elements?), then attach external buffer to shinfo > and call mbuf_attach_external(mbuf, shinfo). > Though for free we can just call mempool_put(shinfo) and let particular implementation > decide when/how call free_cb(), etc. I don't want to restrict external buffer to mempool object. Especially for storage users, they want to use **any** buffer, even coming outside of DPDK. However, will open a follow-up discussion for this in the next release window probably with more measurement data. Thank you for suggestions. Yongseok