From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR03-DB5-obe.outbound.protection.outlook.com (mail-eopbgr40055.outbound.protection.outlook.com [40.107.4.55]) by dpdk.org (Postfix) with ESMTP id 8248C10BD for ; Tue, 24 Apr 2018 13:47:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=l5b138Pqco9AguNmQdlGodLEgLRWc3katCo6wD3pD+8=; b=BhNIhxG6H0VY4eGK32Z+WeI5YyXNEXeAqprSDNV/bV7qQGFckYVXqNGu4mb8kSSkakgIAxYpzfmF88ER3e/DKoz0sHQ6kewVBhLRbppxoCjOS1m9rpZ7b+CzD4WH/kkW6DHHeoc3hGNB61suc5E44MSEaQVK9eiE8QNxUCHVC2w= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; Received: from yongseok-MBP.local (73.222.116.174) by AM5PR0501MB2035.eurprd05.prod.outlook.com (2603:10a6:203:1a::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.696.13; Tue, 24 Apr 2018 11:47:26 +0000 Date: Tue, 24 Apr 2018 04:47:08 -0700 From: Yongseok Koh To: Stephen Hemminger Cc: wenzhuo.lu@intel.com, jingjing.wu@intel.com, olivier.matz@6wind.com, dev@dpdk.org, konstantin.ananyev@intel.com, adrien.mazarguil@6wind.com, nelio.laranjeiro@6wind.com Message-ID: <20180424114704.GC84555@yongseok-MBP.local> References: <20180310012532.15809-1-yskoh@mellanox.com> <20180424013854.33749-1-yskoh@mellanox.com> <20180423220107.4d221e6e@xeon-e3> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180423220107.4d221e6e@xeon-e3> User-Agent: Mutt/1.9.3 (2018-01-21) X-Originating-IP: [73.222.116.174] X-ClientProxiedBy: SN4PR0501CA0036.namprd05.prod.outlook.com (2603:10b6:803:40::49) To AM5PR0501MB2035.eurprd05.prod.outlook.com (2603:10a6:203:1a::21) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(48565401081)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:AM5PR0501MB2035; X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2035; 3:ssVdmeBxCB2ikb422Sv4VxA0nlaeN34eMXayxIS5uINaRp+P+Xo8ODrOe6PQhj0hRrt2vTOsVK2kTyffqh62GP3+i6S4JuxDe4y7jOdBweg5sHU/BUtTXGaF8BHOs7Idbg1IXTt8GQkC49fR9dIVYyJZoWUItFdJXcnKFSMVVQ4KGOt4LTB+SYVMq7csUwuG4tqjHG5LHUfBUWc4jao5zNQQJHF/xy2plSFLtirh7D7334/8GNmziZAYt7cJdS+I; 25:515z5qUbGS4sSIcZSW9MzxM1PpZca+k0+O7xxm79+7I2SZhRD9ZlhLZA9fyhmQ0Mex+pUKXLcCsVehhEewyf6ZksNKE8LEjIJywI8cP9Ur4/A2Dlw0jrwncARQk27QXF8CmDjrXtIiTLr2D+SrlTXZ2nRHfY6mOXfXr5Anqrz8jpgZQhBDxpFAAaWLVYI/EXDHvOJOgBDVir7kt3ovLuM040+a1QqybZesqj9XTGMW0QAWZPPZz8FwIM+3MiPiJE+8lc3etjS7PhivU/PW4sF6VvJ+3hHef2XfbNSC2/F2Goh2EFHznghX71hoz+JY19CUuRKbhMAPrNSHKWA+6KAQ==; 31:BjyFukaHrJvj9iQ2J6WQJdzaVw8i1swZRo0mEQSGVaoBNi3b8C5RDd7k/pEJgWlOZ0pjK3FjI05oRcp1MEQaDwKxSOS5cRfG00nw50Eektf2X1ph/9qz2MQnNRey4VTbcu+8aU7Qjrd3tCPqVsrpFM++nCJZq7Eb/oDDyudrO/LRGjeZ3yajprdO9pJS70P/vPgXwn+zz66qMoD7/jyML7XinHEfFtVl2dCIIStxhNA= X-MS-TrafficTypeDiagnostic: AM5PR0501MB2035: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2035; 20:0HRfZalbvcEn2Mkf8mLvT68YV3gOWy6d/HV266G9Ar8o1xvl+v2MlDsF2rBzQkGUOWKmlLNyEHQM9ecWim66TC9n5Aa4r2om6CmdHX5Vc5dK44QNSxPLjC7Ia1WYMgFzfnD8i865gw+hXn/waaknVECumDUQvKzz5HLi9FWEgUEtOoTtKETwatGKYGSxHL55kpRwL1m8QYtHhyk/Z62UnjfKvYyVa7IxUofZXTmcWQyF1G5LAK8hDEPoOm5WMC2Y3x7YZILe5+xcOVHWvwnn3PJVzJBk/DUl4lwvvs8yjvoLWgD238XKGrhav32Nr6Bs5r86L+fj6kd6Y1VmR3NO01PT6GHZR79eWuu91TeQD/VZWcZl2WnwK3LI9FuHur1O6zsuFrAef7uo2eKjV+vSmuYxyYqBf8TVXlcRPcsU0vJWAkDF9B9w0z/aijCsiLuEiVw2CeTz+jy0mRzsaCfFf9hHGvt+WX9TeoMZMVYrylBph4vodO8MpYkQvejlo0HR; 4:S0b/xVzwDS2LNzHUO/2U+JW7s2k4nKWDvXJAlGhtJkRnqFkEWWqEd1ytpqUfWF9vQQW+M/9mjJBPoDZ9/C8EjlnsxDe+rr02CiUk5T7ZAMEoz9UPD5o/eUpYI9iyLP6H2X3DpkyGXjJjWWC+KDk5n95o+DTJIf4m9gXto350codBoYnkjb+gIn/I78ppVynIL7sJkv4k7dl35cNF0/vEnIiDQHsxeSsZjc0vPzXLmPp3tbQS/vhbmDn9TVddPlvrb4oXrfiNLK5+BHrgh8lPvxc9PBKjKMtFjuztMOzFTq0lvJmc5boRUBGS25bln6HV X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(278428928389397); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(3231232)(944501410)(52105095)(93006095)(93001095)(3002001)(10201501046)(6055026)(6041310)(20161123562045)(20161123558120)(20161123560045)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:AM5PR0501MB2035; BCL:0; PCL:0; RULEID:; SRVR:AM5PR0501MB2035; X-Forefront-PRVS: 0652EA5565 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(346002)(396003)(39860400002)(376002)(39380400002)(366004)(199004)(189003)(4326008)(6666003)(11346002)(6916009)(2906002)(446003)(50466002)(6246003)(16526019)(5890100001)(186003)(98436002)(9686003)(478600001)(229853002)(25786009)(476003)(486006)(956004)(55016002)(5660300001)(53936002)(47776003)(66066001)(97736004)(386003)(316002)(8676002)(6506007)(33896004)(81166006)(8936002)(81156014)(68736007)(76176011)(58126008)(16586007)(59450400001)(26005)(33656002)(305945005)(106356001)(105586002)(23726003)(52116002)(7696005)(1076002)(86362001)(7736002)(3846002)(6116002)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:AM5PR0501MB2035; H:yongseok-MBP.local; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; AM5PR0501MB2035; 23:L158mDVCgCowUo2fbkaJTCbuX9vn6hdgZ1iUuNS?= =?us-ascii?Q?027j0qWrvlH8KWeZ2xcEiHmTuVb28SJem/tFelYNthQn0VCvRhRB4589iHB8?= =?us-ascii?Q?cGIhawtU/jpvMbzKSYVrvXAEMOwTrI2AlVGgWf4e4dALsekRUfQkee8LMomk?= =?us-ascii?Q?VaqUh0WFMZY83609Ny/U4JGzdumhc6WW37M5uYha5lRhQ0SCry1LLbEN0HNe?= =?us-ascii?Q?Tj7Jq1a+LbzPP4TU8JQLU8TxgC3tpZQwDx16jrmeTZgw3avsTjdNaDoZR0di?= =?us-ascii?Q?iZnNOcM2TTdr/8wa8akzgXxSXRvY9fiEUKArRN8QxXzYXpKq0hAC7eRWP4ju?= =?us-ascii?Q?oECAiVD3ZdSuaR8Akh88WeAC+igEQOETzREKT4ixNUjplxwGyLe4oS/kaFD8?= =?us-ascii?Q?3sk1U+96t7eIj7qh59OjsyfVvSQ45mClPgfkXFHPp1DIbLSGzoJbeRwPpFET?= =?us-ascii?Q?VbfO33PdAffwxhutgiw2v0Dd7R08kz5wNabBq0HT25bpSH66eWN0Scg3Gfxe?= =?us-ascii?Q?gc/961NwI9lhagor8bMdNMc5t+GgZvN1a4hI/kUNn2EUeq9apGNnQvZEgrjo?= =?us-ascii?Q?SeieXu3uD7wwBNItcRw3Ag8/n2EDui+kXeLFLX5hFAsR/nE16hSSGSNjjs8x?= =?us-ascii?Q?yHepBtAqFX5HlEea0A9lKmp4f/Z6m+KxJM/rpzbj4n1Xr2QWPEahy4xzBgR0?= =?us-ascii?Q?/ehfy+LyTuVGZUPnSc1wbw/lz/RMIv10w/3wIL7uXRu+PJ5wWY0IbZM/GAN2?= =?us-ascii?Q?M++bEbb+luOiUT5lIpRaRSit7GX2Wg63SDNDWzJFpAS8z6THWtfIByX56cDh?= =?us-ascii?Q?D+SA3p2jGFreR1a0uFxBRpRsjqu8VEEx/MLqwSqpCF+ywIkQkBt2ZT2Hsz5L?= =?us-ascii?Q?5ub9K/i7PEwciWgJqFMASqIBESlB0v2bgN2Cqc2mG+4scPPTtxX2OkDdHZca?= =?us-ascii?Q?QMZMoID8mTIqrh68AhruPfJAP5FxgbG5NmRmK9jM4sVHhmIe/Cz9TI8wfXM/?= =?us-ascii?Q?FcafIS2EFOkKCW52rCr0Qc4M0/4Y2jQUQ+jYyjPPFL9Qqk/2CCrnPqbvmO3/?= =?us-ascii?Q?6/yWqW6teW3Xi8ZJo4wlen+SWd34QYZ2/G/NFWFhCTzcVrdFIBMKUaz9Q0pr?= =?us-ascii?Q?wgdZyQRa7n+8mpYgTlo/raGBbH1MUYbaHcB97GWodGL2KyNHkaaQBqIiDqwW?= =?us-ascii?Q?nDYvtjV9ISlHrZgBrX4ubUBcTMfaEEAjUZL4NOkkl8p78fl8VQU9LQpjw8Mc?= =?us-ascii?Q?DkuJ+z+Z6fMQq8RZ78lQDCDmYynEpGhfRUcFcuE2QxKFST14ay/HUEP/L2do?= =?us-ascii?Q?w+nq+wDLTgYVUehpbEcqpNBbH68e1KS5q+G/1p9+lSm42?= X-Microsoft-Antispam-Message-Info: S43hMXe5hiFhoIgtplXujAxKJ2MHvwk0aoMfJjZUE3gYKTNU/ZWxYbGHErpLvKIRM3iwkYtBZBd5foQq1kVjkmXlI58q1L3QIjDauLe7fGU5pMMa+EoHIvDo6Xuh1ofAnsTaxXuQ0p2HhBNydc7HhrSxmgJ3Y/HIMOXX6ov7HcFt7PcaKH/he+xZvek/zyRb X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2035; 6:wnUlTFtnV1cVDikSx26epIWaGNylioYXro3DxZEDMz2WCJdeEKV/jeIO8DE4R5kYTYJhez4OG7JdyFzXE9fAzQTzFpZRL4e7z/ps2KwaIpvqn/KNt38GXMQwKrC67ua9zAVVaj1e8mKkQE8jaOwdEZXeQC/8C/3LtPkOCuLgCDdJtA/VHlWh9D1F9rVCsWAFVEqENJvP38pgjS58IG5WBXFpqsTn0NtuacXgJBCjUS3uVsQnshXDbN+VMok1jHwHSpKcr/ufgZxlAx8N8j6MAQ7pu4HOXBxnwg2EEMZWUbI8wRVWMJ4shZbKyvjuIXL2a/NT9/T+C+RAYG+/iRwUbh5mSNarGR7sdAAibo9Hj3VxU5w7TLXfU21HTDmiU2Vv07yyQ3RoOKaEtQ86lq4lltMz3+zqSBj3BFQTp+/k7W/jPaoPyoEr5oqURlPKV0PJVwjaqxcs+5Y3yDGqdYvCOQ==; 5:Ieh3GFlV1hTbErDseKnvxAbupGAl1683+u4kin6UFSjI9FBkTnCIOU2a50mmT0aD1vo9U/hancKzRnXdJGoiPD5dy82zNfmm2atcCrpbDz7eKxwcrY8DV96g6W/mB7Nu/5YXBd7l0QBs7iPVp18y3FwUSna+521aj1zYVBDsetE=; 24:kof8tQrwwI0eJwPWDJ1QUc1+GY5hCWbDwiR0guN1Nuq5lReZZzAvvgFRCKPQfuRickI3NH3i7e7YuLLbFC6tBcdpAMAlenFITmjwvNoAkQc= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2035; 7:snG6v4UYVhJLpzY2Cknt2dT7SNWx8UcjkVx9iUsCf/d76xs+up/NVuTHDCgluqXR4kZFIQtxyRYUe/ZoWdp1BsPJpeJ5yONy/NSfek38NfnaG0S2tSg1GzAZdQzgC1bxciV4SGB+1xLiSqTmdkFSyHqNiNmi/PRErD7Mpc7uqvQhV+knm3PvJw2H/kzA2ztF/Yzfg3zEG0tFR7pRgKba3caua1E/rc71e1AICy8erAMQT5TQtVxmD3DbFoDO4Fer X-MS-Office365-Filtering-Correlation-Id: 0c32fa13-29df-446d-5eac-08d5a9d927ea X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2018 11:47:26.8117 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0c32fa13-29df-446d-5eac-08d5a9d927ea X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5PR0501MB2035 Subject: Re: [dpdk-dev] [PATCH v4 1/2] mbuf: support attaching external buffer to mbuf X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2018 11:47:30 -0000 On Mon, Apr 23, 2018 at 10:01:07PM -0700, Stephen Hemminger wrote: > On Mon, 23 Apr 2018 18:38:53 -0700 > Yongseok Koh wrote: > > > This patch introduces a new way of attaching an external buffer to a mbuf. > > > > Attaching an external buffer is quite similar to mbuf indirection in > > replacing buffer addresses and length of a mbuf, but a few differences: > > - When an indirect mbuf is attached, refcnt of the direct mbuf would be > > 2 as long as the direct mbuf itself isn't freed after the attachment. > > In such cases, the buffer area of a direct mbuf must be read-only. But > > external buffer has its own refcnt and it starts from 1. Unless > > multiple mbufs are attached to a mbuf having an external buffer, the > > external buffer is writable. > > - There's no need to allocate buffer from a mempool. Any buffer can be > > attached with appropriate free callback. > > - Smaller metadata is required to maintain shared data such as refcnt. > > > > Signed-off-by: Yongseok Koh > > I think this is a good idea. It looks more useful than indirect mbuf's for > the use case where received data needs to come from a non mempool area. Actually Olivier's idea and I just implemented it for my need. :-) > Does it have any performance impact? I would hope it doesn't impact > applications not using external buffers. It would have little. The only change which can impact regular cases is in rte_pktmbuf_prefree_seg(). This critical path inlines rte_pktmbuf_detach() and it becomes a little longer - a few more instructions to update refcnt and branch to user-provided callback. In io fwd of testpmd with single core, I'm not seeing any noticeable drop. > Is it possible to start with a refcnt > 1 for the mbuf? I am thinking > of the case in netvsc where data is received into an area returned > from the host. The area is an RNDIS buffer and may contain several > packets. A useful optimization would be for the driver return > mbufs which point to that buffer where starting refcnt value > is the number of packets in the buffer. When refcnt goes to > 0 the buffer would be returned to the host. That's actually my use-case for mlx5 PMD. mlx5 device supports "Multi-Packet Rx Queue". The device can pack multiple packets into a single Rx buffer to reduce PCIe overhead of control transactions. And it is also quite common for FPGA based NICs. What I've done is allocating a big buffer (from a PMD-private mempool) and reserve a space in the head to store metadata to manage another refcnt, which gets decremented by registered callback func. And the callback func will free the whole chunk if the refcnt gets to zero. +--+----+--------------+---+----+--------------+---+---+- - - | |head|mbuf1 data |sh |head|mbuf2 data |sh | | | |room| |inf|room| |inf| | +--+----+--------------+---+----+--------------+---+---+- - - ^ | Metadata for the whole chunk, having another refcnt managed by PMD. fcb_opaque will have this pointer so that the callback func knows it. > One other problem with this is that it adds an additional buffer > management constraint on the application. If for example the > mbuf's are going into a TCP stack and TCP can have very slow > readers; then the receive buffer might have a long lifetime. > Since the receive buffers are limited, eventually the receive > area runs out and no more packets are received. Much fingerpointing > and angry users ensue.. In such a case (buffer depletion), I memcpy the Rx packet to mbuf instead of attaching it to the mbuf until buffers get available. Seems unavoidable penalty but better than dropping packets. Thanks, Yongseok