From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-eopbgr10061.outbound.protection.outlook.com [40.107.1.61]) by dpdk.org (Postfix) with ESMTP id A16D42C16 for ; Wed, 12 Dec 2018 13:55:52 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dfpRFYKlAAasi44MlSM+6dFXFNEYwJyzmvk4kKcmoZo=; b=gr3yAwsz4u8rb/RLAfZwNIBYpsdFy1q6+FfoUA3T2K4UVvpUkHF6wWqsgVoKaQbUt1tZJ6J94PgtxMuMHybtPQ+8EeFShRowxGtnrb7ldZKoPZcNMD3yIFJ19JSNaIV8XixkdGSGbj5ZNVGfQtTlVU7+v9Imm6tJEWQKbZ91hN8= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB4012.eurprd05.prod.outlook.com (52.134.68.29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1425.18; Wed, 12 Dec 2018 12:55:44 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::dcbc:4578:3018:50f3]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::dcbc:4578:3018:50f3%5]) with mapi id 15.20.1404.026; Wed, 12 Dec 2018 12:55:44 +0000 From: Yongseok Koh To: "Burakov, Anatoly" CC: Shahaf Shuler , "dev@dpdk.org" , Thomas Monjalon , "shreyansh.jain@nxp.com" Thread-Topic: [PATCH 0/4] Allow using external memory without malloc Thread-Index: AQHUh+o+UL0xxxq8xkaRCgNf8gc/VqVq9YKAgAEoOoCAALbIgIAOT5KA Date: Wed, 12 Dec 2018 12:55:43 +0000 Message-ID: <20181212125528.GA5538@minint-98vp2qg> References: <7F52E2C1-4552-48E7-9AFA-BEAF4D8E45D7@mellanox.com> <57d7ad23-bebd-f788-5ac9-540804ecf5da@intel.com> In-Reply-To: <57d7ad23-bebd-f788-5ac9-540804ecf5da@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: BYAPR07CA0032.namprd07.prod.outlook.com (2603:10b6:a02:bc::45) To DB3PR0502MB3980.eurprd05.prod.outlook.com (2603:10a6:8:10::27) authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [69.181.245.183] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DB3PR0502MB4012; 6:r7VMYJc598AeoH4zLxkFVJ+vjesEiKr8FY6GShn2OGrCz5yjF6TssTnDhRZAVjqCqv97S03EKAlO35iHIGEAAlYrXYctkW5NoIbku3TgOVcVoksGbnd7JmBeXWAASDaUn3gju5WaTLfbwVbKx5A14z2N9q3SKcg6xCNm+2ee+PS6X//VdruGFfMSlBHyvws3QG0eFj6cnslv3DF85KAzfKYWop6yXYdmfY/zBN3fIfO7QRNfWbErEMS4E6/5w1ov4kJ4lFeKwrCJ7zbpmqVTcbyPUl9Yk6h2TEA/3g4iYtpjZy94QUUAzxH6yL5fgPP6htpexmptTwBlNoRDPTq3r739BYkjyueViRPg+ZWcLbzA+dean56Vk1v0izA3fdahayoBkSCiHfZ3AudFKnYg86PqxAnwtNeLQG8MA7XfzBBfs10VTCQ+CUR5UJrxgbTt7QnqTw5wApFjAiMpjzw1aw==; 5:DFAVchDPmPlgWP5GCs1AyoKXT6HJvHw3b2x/7wNNXRRLGDSoRPcvvudjKTMfacuR77WWMe1/aaelT3qCnrlmSIob5UY/91N6Dwx6Pe761j32xhDwIVDrEI8p1nwz0Hq1vXrX+lyaaovg9AmyaO0mvxTwLrV4rGDeCrvArzF9h9I=; 7:w/4ztnVIHD8ThOzP4X7tfH+ukADUalpT/9DsDcoXx63SqiaW4Wdf2x5RSCxJLqg8NWAlp31OxVF09Nr9a86tYZgWKfQNDIJaL/yAq2oqCyQQx5x/cOm3BIVBUCl9zjW6r+cP5bNgJtAPclMna6BDKA== x-ms-office365-filtering-correlation-id: 1e3517dc-1b7b-44ca-ada2-08d660312083 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390098)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:DB3PR0502MB4012; x-ms-traffictypediagnostic: DB3PR0502MB4012: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(10201501046)(3231455)(999002)(944501520)(52105112)(93006095)(93001095)(3002001)(6055026)(148016)(149066)(150057)(6041310)(20161123558120)(20161123560045)(20161123562045)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699051)(76991095); SRVR:DB3PR0502MB4012; BCL:0; PCL:0; RULEID:; SRVR:DB3PR0502MB4012; x-forefront-prvs: 0884AAA693 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(7916004)(136003)(346002)(366004)(39860400002)(376002)(396003)(69234005)(199004)(189003)(52116002)(966005)(71200400001)(71190400001)(68736007)(86362001)(106356001)(6436002)(5660300001)(105586002)(6916009)(7736002)(33716001)(1076002)(81156014)(81166006)(8936002)(478600001)(14454004)(305945005)(5024004)(2906002)(229853002)(6512007)(6116002)(4326008)(97736004)(476003)(446003)(256004)(53546011)(102836004)(6246003)(186003)(8676002)(9686003)(66066001)(386003)(26005)(6306002)(486006)(25786009)(33656002)(6506007)(33896004)(6486002)(76176011)(99286004)(53936002)(93886005)(11346002)(316002)(3846002)(54906003)(45080400002); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB4012; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: xlWHd2FbCrE9XF1i5Tn3zzTwmKoHjPfSRw2fuw/WRvYWXRqFGC+ykk+VbW3RNG8zTFRx1mg/abK9r1cxJEqHG39gsJMZLniah3U7bnjEUKv0DRE1r9RiaQIeGfQgXy99Ij4np6BkqfrPq2NVQA5bJkbBmaz3RX4dk3VjMhzFLLrafJnOefV+NvrLA1yNiE+PtrmGX5L33PSYpOvw6j3opQTsMAx3R1E/o2WqD6eYZqz0hHl87lXqQWMiXBuByReu8x25GurUzLHi6svfllu6rD8eyOo/rhErFNdDmct4nETfR1tIpnVMaSoBYX5YJyED spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-ID: <44B154FFEC94584193FAF3EC508DFCF2@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1e3517dc-1b7b-44ca-ada2-08d660312083 X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Dec 2018 12:55:43.9571 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB4012 Subject: Re: [dpdk-dev] [PATCH 0/4] Allow using external memory without malloc X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Dec 2018 12:55:52 -0000 On Mon, Dec 03, 2018 at 10:23:03AM +0000, Burakov, Anatoly wrote: > On 02-Dec-18 11:28 PM, Yongseok Koh wrote: > >=20 > > > On Dec 1, 2018, at 9:48 PM, Shahaf Shuler wrot= e: > > >=20 > > > Hi Anatoly, > > >=20 > > > Thursday, November 29, 2018 3:49 PM, Anatoly Burakov: > > > > Subject: [PATCH 0/4] Allow using external memory without malloc > > > >=20 > > > > Currently, the only way to use externally allocated memory is throu= gh > > > > rte_malloc API's. While this is fine for a lot of use cases, it may= not be suitable > > > > for certain other use cases like manual memory management, etc. > > > >=20 > > > > This patchset adds another API to register memory segments with DPD= K (so > > > > that API's like ``rte_mem_virt2memseg`` could be relied on by PMD's= and > > > > such), but not create a malloc heap out of them. > > > >=20 > > > > Aside from the obvious (not adding memory to a heap), the other maj= or > > > > difference between this API and the ``rte_malloc_heap_*`` external = memory > > > > functions is the fact that no DMA mapping is performed automaticall= y. > > > >=20 > > > > This really draws a line in the sand, and there are now two ways of= doing > > > > things - do everything automatically (using the ``rte_malloc_heap_*= `` API's), > > > > or do everything manually (``rte_extmem_*`` and future DMA mapping = API > > > > [1] that would replace ``rte_vfio_dma_map``). This way, the consist= ency of > > > > API is kept, and flexibility is also allowed. > > > >=20 > > >=20 > > > As you know I like the idea. > > > One question though, do you see a use case for application to have ex= ternally allocated memory which needs to be registered to the DPDK subsyste= m however not being used for DMA? > > > My only guess would be so helper libraries which requires the memory = allocation from user (however it doesn't seems like a good API). > > >=20 > > > If no use case, maybe it is better to merge between the two (rte_extm= em_* and rte_dma_map) to have a single call for app to register and DMA map= the memory. The rte_mem_virt2memseg is not something application needs to = understand, it is used internally by PMDs or other libs. > >=20 > > Just FYI. > > My implementation for mlx4/5 doesn't need to have a separate registrati= on for > > DMA by rte_dma_map() as long as it is included in the memseg list. Regi= stration > > is done only if Lkey lookup misses and only mem free event is taken to = release > > it. From my end, the reason why we wanted to have a generic DMA registr= ation was > > because some people doesn't want to use the new API to make the ext mem= included > > in the memseg list but want to simply call the API for DMA registration= . > >=20 > > In a nutshell, mlx4/5 needs users use either rte_extmem_register() or > > rte_dma_map(). However, it is no problem to call both. >=20 > It would be good to create a segment when using rte_dma_map(). > Unfortunately, that's not realistic :) >=20 > Registering memory within DPDK does not necessarily have to be performed = by > the primary process - whichever process that wants to create the table, c= an > do so, and later processes have to attach to the memory area. There's als= o > no way to know if memory segment can be attached to - this is a question > only application can answer. >=20 > In other words, there's no way to combine rte_dma_map() and > rte_extmem_register() into one call and keep multiprocess support. Sorry for late reply. I was away for a while. I understood your point that rte_dma_map() can't create a segment but isn't= the opposite possible? I still have a question about rte_extmem_register/unregister/attach/detach(). Why don't these APIs genera= te memory events? Do you define the memory events are limited to memories for malloc? What if some app wants to know the events even if it is extmem? Wha= t makes difference between two types of extmem (one for malloc heap and the o= ther for just memseg) in generating the events? I've reviewed your patches and all look good :-) But it is still unclear to= me. Thanks, Yongseok > > > > [1] > > > > https://emea01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%= 2Fma > > > > ils.dpdk.org%2Farchives%2Fdev%2F2018- > > > > November%2F118175.html&data=3D02%7C01%7Cshahafs%40mellanox.co > > > > m%7C007a9234feaf42c82f6508d656015eb1%7Ca652971c7d2e4d9ba6a4d1492 > > > > 56f461b%7C0%7C0%7C636790961244424277&sdata=3DYqwcPEEhJM3I7Toe > > > > Ne%2BGcbeo%2FmPbYEnNFckoA7ES2Hg%3D&reserved=3D0 > > > >=20 > > > > Note: at the time of this writing, there's no release notes > > > > template, so no release notes updates in this patchset. > > > > They will be added in later revisions. > > > >=20 > > > > Anatoly Burakov (4): > > > > malloc: separate creating memseg list and malloc heap > > > > malloc: separate destroying memseg list and heap data > > > > mem: allow registering external memory areas > > > > mem: allow usage of non-heap external memory in multiprocess > > > >=20 > > > > .../prog_guide/env_abstraction_layer.rst | 63 +++++++-- > > > > lib/librte_eal/common/eal_common_memory.c | 116 > > > > +++++++++++++++++ > > > > lib/librte_eal/common/include/rte_memory.h | 122 > > > > ++++++++++++++++++ > > > > lib/librte_eal/common/malloc_heap.c | 104 +++++++++++---- > > > > lib/librte_eal/common/malloc_heap.h | 15 ++- > > > > lib/librte_eal/common/rte_malloc.c | 115 +++++++--------= -- > > > > lib/librte_eal/rte_eal_version.map | 4 + > > > > 7 files changed, 434 insertions(+), 105 deletions(-) > > > >=20 > > > > -- > > > > 2.17.1 > >=20 > >=20 >=20 >=20 > --=20 > Thanks, > Anatoly