From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-db5eur01on0075.outbound.protection.outlook.com [104.47.2.75]) by dpdk.org (Postfix) with ESMTP id 17A09F3E for ; Tue, 24 Apr 2018 04:04:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=85mnfjl2gC9hAvdf93ZUXx2NLRasrZ+HgqN7XOHDRf4=; b=obohniR6NJ7Yq3qrAJxVJ7cnuKO45wH2kFzL37LMVlMG8fN7/5h1Smwzr9NZ21souQ7ykkpIsf4WaLw2Xk+0N2/CERQRFK+8xCYAQ/Qt+ahC5JVcIuH4dtm2qACZzega9w6ONgb/c6GEXa/n/POBAJ3p7w8Ltm0dTZ70Ki2X2Ek= Received: from yongseok-MBP.local (209.116.155.178) by DB6PR0501MB2040.eurprd05.prod.outlook.com (2603:10a6:4:6::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.696.13; Tue, 24 Apr 2018 02:04:44 +0000 Date: Mon, 23 Apr 2018 19:04:27 -0700 From: Yongseok Koh To: "Ananyev, Konstantin" Cc: "Lu, Wenzhuo" , "Wu, Jingjing" , "olivier.matz@6wind.com" , "dev@dpdk.org" , "adrien.mazarguil@6wind.com" , "nelio.laranjeiro@6wind.com" Message-ID: <20180424020427.GA83470@yongseok-MBP.local> References: <20180310012532.15809-1-yskoh@mellanox.com> <20180419011105.9694-1-yskoh@mellanox.com> <2601191342CEEE43887BDE71AB977258AE91994F@IRSMSX102.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2601191342CEEE43887BDE71AB977258AE91994F@IRSMSX102.ger.corp.intel.com> User-Agent: Mutt/1.9.3 (2018-01-21) X-Originating-IP: [209.116.155.178] X-ClientProxiedBy: MWHPR1201CA0019.namprd12.prod.outlook.com (2603:10b6:301:4a::29) To DB6PR0501MB2040.eurprd05.prod.outlook.com (2603:10a6:4:6::22) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(48565401081)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:DB6PR0501MB2040; X-Microsoft-Exchange-Diagnostics: 1; DB6PR0501MB2040; 3:rrt2ADrf9h6y3jpHq63FDMBfBUqGkB8FUdl18JySGnsrjw9bW0aAu9StB3cJq+mKOV+DkPkdiBqO8+gIOs5JSFA7cqV6NnHQdmx2SE5clQGH0hje/YyHp55TdYgxLMkHq9DkIWhfEULzyPYIaxRSMO/T5z8O7c/eTc6FoPVghvZ3G3W1tXPJHeqmILgl+CyXlY3adYuWHgSDYrr5WwtPO/6qj4e2hLiMm+DSWHTP5poEggwQ/5v8Tio45H+JLupv; 25:rIhUkPGQbCbtkowNXmWPaNf/EA8rBLO/zcgeNZR/bZuM+mi2ua8ImXCB4sTlyQJ24I6R3aZMMcyRu4l6tryFa7cyTlhz3RTU0i/NtTyPcWe2CIO3tJeXrXEe0TkRlfzEvMweCOn5LcmvR5bD0uOcUC9ZBX8VabHfWhfW/tiiUpld1zC8Vi9kcUXTd1LLtIxCW8L4zAmioqPr0hoSLr0VgFPXrzWc4TDM63dRWSfYb2rsgjEuKsl1Tk/Y0oLwJGu8Cn5D+n0vR67Z7TitgJ4o4LQQMJ4+qMOc3/+6430ORai+4vtdBupfWiKy27mbRMJaTC+e1imYt/dsxyWddLDlAQ==; 31:C0zvFfXaaERjbL4UCctxxXKs4QAa/nWHOnhEBqqmmq7/+T8GnpAEMaapFnjt+kfM47Q99faW2Qb1Rm0Wbj/krG2AcUXJZjkB0IJM03sW8NUGZAVZ1Qhj8a0z7MQAQ7AfyiY6vcVjABr3HBInPHyuu2jSoLlaw64IkpQD0pfz185cKgFpL6uVimsewT1G6aQBXj/ThLS+amWEXazKNUP5jhIome/HGhlvuuTpVx42ULo= X-MS-TrafficTypeDiagnostic: DB6PR0501MB2040: Authentication-Results: outbound.protection.outlook.com; spf=skipped (originating message); dkim=none (message not signed) header.d=none; dmarc=none action=none header.from=mellanox.com; X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; DB6PR0501MB2040; 20:8V57uXkub/T70YYJf9w6ffhaEInEo6NFkkBJiGrpBre1XncvlcafppqWfCnSyT0b/SGFqCF7kO7dqz6biJxxEEBvmaki7oDTUI4tTAOSXr848bH3CU7GzCjpVgjZKOy3edh5h/EosvJ5bVVOmEuy+dkNzmdXqYzx/v6Q65DR5iqqNM5HPL6I08154+EPUOLhnFpFGS145Wj51l7piSgRbYWERvAwAMcIYx1HpKnbLfRI3xIBmxDU1LhktNszbiHxyFwhICanMzX8+YAPusiO36g/LqcnguWJvAmUkYVWtGaDYjtrZFm+s/Z5/1U7QBcB9Xk9WRN5FVtXOBQW8RrYzI/coegQrCqF6WMVVz6FG8gPuKkXagcKtzfzI0NH1X/fLJEIqA/uhZYrK8epnU51HDON0x49Cb225IzxOJoVVi5m19wBnHJdy4qSTccD64EATdYzBFQji920tu/DCTXRS/fTTNDJG2SNIU5tgv2msbqVfRqOJwn+X7K6levDYSNV; 4:IfeMDJqMI/7RnwE5/WyKDERvHdn4HhcCDHn0nZRVqkygmIuPHnZ3b6q3NToKXeg7fkWyV3VfqfuCY0AT8xfJLPhVbuO2+/aB0nCuvB3JVXHYREBaouPcXx8VNmvjTv3T0krnWe06NMU+Gb2+DAkQm3TCs14Om5TDPQk/T985eg/5aIhb+FHD8CvZaGVqcsYCFPvjx3EzNahOvWkx4bfMmxCDkZgXpb750FdCZPeH3lCfqwK1bxwivCM++8OvWitmOjopVOSuWcr0QjSwOJhM75Xm0QOMFvY88yiNZh29XJVQU/gIodtmy9M8zdjvHuU+ X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(131327999870524); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(3002001)(3231232)(944501410)(52105095)(93006095)(93001095)(10201501046)(6055026)(6041310)(20161123564045)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(20161123562045)(6072148)(201708071742011); SRVR:DB6PR0501MB2040; BCL:0; PCL:0; RULEID:; SRVR:DB6PR0501MB2040; X-Forefront-PRVS: 0652EA5565 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(346002)(376002)(39860400002)(366004)(396003)(39380400002)(476003)(81166006)(8676002)(186003)(76176011)(8936002)(305945005)(16526019)(956004)(446003)(11346002)(7736002)(5890100001)(50466002)(33896004)(316002)(386003)(7696005)(52116002)(26005)(59450400001)(54906003)(6506007)(16586007)(23726003)(1076002)(86362001)(3846002)(6116002)(4326008)(229853002)(55016002)(98436002)(33656002)(25786009)(6666003)(9686003)(2906002)(6246003)(53936002)(6916009)(5660300001)(47776003)(478600001)(66066001)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:DB6PR0501MB2040; H:yongseok-MBP.local; FPR:; SPF:None; LANG:en; MLV:sfv; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DB6PR0501MB2040; 23:yvNJ0LtQGocoagQ97NQc4Xi6Gcp3huVqetvaA1F?= =?us-ascii?Q?OE38w84OGy+cVsy5jVJxFzyzIHhgnSo4gb8XfpDPNKi25HmsLzf32WXEKlim?= =?us-ascii?Q?p0v5TdrMLdrKTYgYGsVNTQtI0ZIo4Mcan7QN0e99/4pAokm6XD/VQNIdfxcJ?= =?us-ascii?Q?PDmsBXZmebFcbaLQW6chlUd5AlYLSJ2rw1weHO7LJp/zEuXZkqWsMA5ceMrF?= =?us-ascii?Q?zbriayK14yZQuF0wCLx0JRSpep/u3cn7qqKqMPDOySi/KIVVxZ16elGaHW1M?= =?us-ascii?Q?u3e22fbVH2m3Plbch1o3Sq2GlcKiQSG6DshzYytruZFyY1brycD04B9h78cJ?= =?us-ascii?Q?GV8vCLUeycNyal8BWL5P3FwHUIRoPIFVDH4C+U3kADpelUf6nO1ewkrTkb6I?= =?us-ascii?Q?X5BIfDCNjQieKo35UNvoR7Pcvr5rn869yhwO4qDGTeqEfxCLKnOeSLU0aCci?= =?us-ascii?Q?eigOyTKM6naT4yx1IilvpTbUGzw+eygOK//qIuo4ucynvn9LiTWYrkGwu5sR?= =?us-ascii?Q?J/1thRXK2wjc8vT/nliPgoUwHM+7xdz7IZHc7av/J7GGPQ+UeTsVliiksSit?= =?us-ascii?Q?k9LLoxZzdOXEeY6Zj9lSap+4niQKFYjrP6mo4e6qEbOIc803ydmQ/RO7B2Bs?= =?us-ascii?Q?E4eL92IkdGVpam8yOBqhLXzDc8Y4Lb06BswxwQmAN8lR7oFQFQkUfrLe3jbt?= =?us-ascii?Q?m6duotaa8Ba7RQK9qtLN+pf3lz3CeMjv4BucxI+So9Cr5XZZNVkSb9UQdVK3?= =?us-ascii?Q?QINyy/hMN/RXvCg/bEL9lkA122rhU66x8QyHciNyo0vEY19lIC5rKZ0nun8A?= =?us-ascii?Q?J396IqWDPwJFCxkxKfNMqC6JxIFllqQFeDxq3NyMBHmSwLdbBrGfUsIFZ/mE?= =?us-ascii?Q?pF+7MLaFbccPx90vcsSXtqIIlrZ6o7FUL0o5uj+l2WStxA7qEwiPOr3KjlGb?= =?us-ascii?Q?gmTVqeuTJHsvDHGwFWDQXGwDXU82odrVGVYFCXa1i79VxEzKqtOn13vwEvCt?= =?us-ascii?Q?cFToTpf/xBuCTtT7gMWL07PVSvFeVeRxqTShm5ufAxrVJx9SRYtnz06zgREF?= =?us-ascii?Q?m3mCooCEPNOdWEFYArakwt5Bai/PfNPQpl7eeEGU8j+wvZp7QKKTItKj5VZG?= =?us-ascii?Q?+b2gCZ+Sf0SDS9SMFGNqyMcTToKsDUbSXHDF2W7kZF2D+yMly/Q4N4nlog6a?= =?us-ascii?Q?dVTrDO/o8/dbJnOQ=3D?= X-Microsoft-Antispam-Message-Info: 7MXxXD8UHhFfm61OXljQQC0dL7JiqqJZbvdz2iolT2nyXYmO/6wKsFXD05413Dlo18WPlbYVGxhQTj9ysGxnW7Mgy6OmcbHVkqKsrEm4fYVNDdckZnVUas/YbntPJ85GtDmUkXajKdu3qLK1PQ7Ew2UyBb0i3dTUemb2lfYhnfHe1ovWt2EVssRD1X9r6nJA X-Microsoft-Exchange-Diagnostics: 1; DB6PR0501MB2040; 6:fYcZn3GiXOfdkMFPZ2N8BfzdXOqqEStbwenjmjMFbRcVamAI9Ht9NHy/8mFk0t3Sct/NvwGdIqH1hdM6ZvcNgMhLyF4rIt2jKuVw6j61+fPe+2267jPEXCzJbD403FHaG9WKu+RvKCVNrjvRZ0GVs6Lb2IrYjrU56Ra3N0Ddc2AO4pya3Kkm7UJaFjfvIO0XP6wlQCpDUQ+WAwVLh25sIv4aP9daB1He95d1C2vKHDk7/OXbe6hNY3D0/Yt95YsLU7jJ5+VrLH3Hp0sp96iGB4c6+p+Hr+QyT9fYfTEr7n2/21HtIz7DcjTDkIi5nS0RzU3x18OSVC/B4PdnSRQjfpZfIpVE99uAsPHNuDP2ZWmfSb3JYOJZrHgPFLAbv1XSAbp3g2VBU/lO5KquKKn89jh8DT3sZggnhXVPVcBHdtyItckd26R9w0TYrQTV3HW/JVNZ/5fIJLfnXAMSBr4NdQ==; 5:y8ECHzxwJ0GAU27SJ0wt9kI+KbNdCnWV90ta3Yz1RkTfQVAgFq5qUxpkR0hMsGkCfb01Jgal4K0PS7p27ibmVAz1NyrQwMp7SSi0fUt19xQ+h+3McGftv9qGoGhaltDnKt2AXoXJXQmNNhaL7sis3XxG4m/Fa6iU/cyI5d7z06k=; 24:x47IULqbdXPwjNtj3vz28VOGG+UjxFS+yZEgCApzROrbxfPl/MQxzsFE5AWQRTLoaFV22HUnIOZNJRSbrNbpBC+21t8m/nk679+6h347jG0= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; DB6PR0501MB2040; 7:C0dL8vjl4RFc6DyyApKwwGrZwDHmH3tM5bIv6W4I5p3vHIHRiykW8zSngtzrw4/thIrysZCeTJhBDf2FVRkUqN7DX8w4s432GFAWqIKW1qBzQXdFXsYUV2ynmnSMAibszjxI+vngvcWxFdW7NkTgr8P7SqdLLZVlIccORMQ3rfLUAW5lGTAnmOBOx5ubnlFduBCI6qLCaVgULbuQQLmXfbRSewEZXDUXU59TjtNISenMXnv4d1bE+5J34Z6TXNHQ X-MS-Office365-Filtering-Correlation-Id: d43ffe79-cdc9-43a9-8735-08d5a987c0ba X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2018 02:04:44.7390 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d43ffe79-cdc9-43a9-8735-08d5a987c0ba X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR0501MB2040 Subject: Re: [dpdk-dev] [PATCH v3 1/2] mbuf: support attaching external buffer to mbuf X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2018 02:04:48 -0000 On Mon, Apr 23, 2018 at 11:53:04AM +0000, Ananyev, Konstantin wrote: [...] > > @@ -693,9 +711,14 @@ rte_mbuf_to_baddr(struct rte_mbuf *md) > > #define RTE_MBUF_INDIRECT(mb) ((mb)->ol_flags & IND_ATTACHED_MBUF) > > > > /** > > + * Returns TRUE if given mbuf has external buffer, or FALSE otherwise. > > + */ > > +#define RTE_MBUF_HAS_EXTBUF(mb) ((mb)->ol_flags & EXT_ATTACHED_MBUF) > > + > > +/** > > * Returns TRUE if given mbuf is direct, or FALSE otherwise. > > */ > > -#define RTE_MBUF_DIRECT(mb) (!RTE_MBUF_INDIRECT(mb)) > > +#define RTE_MBUF_DIRECT(mb) (!RTE_MBUF_INDIRECT(mb) && !RTE_MBUF_HAS_EXTBUF(mb)) > > As a nit: > RTE_MBUF_DIRECT(mb) (((mb)->ol_flags & (IND_ATTACHED_MBUF | EXT_ATTACHED_MBUF)) == 0) It was for better readability and I expected compiler did the same. But, if you still want this way, I can change it. [...] > > +static inline char * __rte_experimental > > +rte_pktmbuf_attach_extbuf(struct rte_mbuf *m, void *buf_addr, > > + uint16_t buf_len, rte_mbuf_extbuf_free_callback_t free_cb, > > + void *fcb_opaque) > > +{ > > + void *buf_end = RTE_PTR_ADD(buf_addr, buf_len); > > + struct rte_mbuf_ext_shared_info *shinfo; > > + > > + shinfo = RTE_PTR_ALIGN_FLOOR(RTE_PTR_SUB(buf_end, sizeof(*shinfo)), > > + sizeof(uintptr_t)); > > + > > + if ((void *)shinfo <= buf_addr) > > + return NULL; > > + > > + m->buf_addr = buf_addr; > > + m->buf_iova = rte_mempool_virt2iova(buf_addr); > > > That wouldn't work for arbitrary extern buffer. > Only for the one that is an element in some other mempool. > For arbitrary external buffer - callee has to provide PA for it plus guarantee that > it's VA would be locked down. > From other side - if your intention is just to use only elements of other mempools - > No need to have free_cb(). mempool_put should do. Of course, I didn't mean that. That was a mistake. Please refer to my reply to Olivier. [...] > > /** > > - * Detach an indirect packet mbuf. > > + * @internal used by rte_pktmbuf_detach(). > > + * > > + * Decrement the reference counter of the external buffer. When the > > + * reference counter becomes 0, the buffer is freed by pre-registered > > + * callback. > > + */ > > +static inline void > > +__rte_pktmbuf_free_extbuf(struct rte_mbuf *m) > > +{ > > + struct rte_mbuf_ext_shared_info *shinfo; > > + > > + RTE_ASSERT(RTE_MBUF_HAS_EXTBUF(m)); > > + > > + shinfo = rte_mbuf_ext_shinfo(m); > > + > > + if (rte_extbuf_refcnt_update(shinfo, -1) == 0) > > + shinfo->free_cb(m->buf_addr, shinfo->fcb_opaque); > > > I understand the reason but extra function call for each external mbuf - seems quite expensive. > Wonder is it possible to group them somehow and amortize the cost? Good point. I thought about it today. Comparing to the regular mbuf, maybe three differences. a) free function isn't inlined but a real branch. b) no help from core local cache like mempool's c) no free_bulk func like rte_mempool_put_bulk(). But these look quite costly and complicated for the external buffer attachment. For example, to free it in bulk, external buffers should be grouped as the buffers would have different callback functions. To do that, I have to make an API to pre-register an external buffer group to prepare resources for the bulk free. Then, buffers can't be anonymous anymore but have to be registered in advance. If so, it would be better to use existing APIs, especially when a user wants high throughput... Let me know if you have better idea to implement it. Then, I'll gladly take that. Or, we can push any improvement patch in the next releases. Thanks, Yongseok