From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <yskoh@mellanox.com>
Received: from EUR02-HE1-obe.outbound.protection.outlook.com
 (mail-eopbgr10061.outbound.protection.outlook.com [40.107.1.61])
 by dpdk.org (Postfix) with ESMTP id A16D42C16
 for <dev@dpdk.org>; Wed, 12 Dec 2018 13:55:52 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com;
 s=selector1;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=dfpRFYKlAAasi44MlSM+6dFXFNEYwJyzmvk4kKcmoZo=;
 b=gr3yAwsz4u8rb/RLAfZwNIBYpsdFy1q6+FfoUA3T2K4UVvpUkHF6wWqsgVoKaQbUt1tZJ6J94PgtxMuMHybtPQ+8EeFShRowxGtnrb7ldZKoPZcNMD3yIFJ19JSNaIV8XixkdGSGbj5ZNVGfQtTlVU7+v9Imm6tJEWQKbZ91hN8=
Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by
 DB3PR0502MB4012.eurprd05.prod.outlook.com (52.134.68.29) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.1425.18; Wed, 12 Dec 2018 12:55:44 +0000
Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com
 ([fe80::dcbc:4578:3018:50f3]) by DB3PR0502MB3980.eurprd05.prod.outlook.com
 ([fe80::dcbc:4578:3018:50f3%5]) with mapi id 15.20.1404.026; Wed, 12 Dec 2018
 12:55:44 +0000
From: Yongseok Koh <yskoh@mellanox.com>
To: "Burakov, Anatoly" <anatoly.burakov@intel.com>
CC: Shahaf Shuler <shahafs@mellanox.com>, "dev@dpdk.org" <dev@dpdk.org>,
 Thomas Monjalon <thomas@monjalon.net>, "shreyansh.jain@nxp.com"
 <shreyansh.jain@nxp.com>
Thread-Topic: [PATCH 0/4] Allow using external memory without malloc
Thread-Index: AQHUh+o+UL0xxxq8xkaRCgNf8gc/VqVq9YKAgAEoOoCAALbIgIAOT5KA
Date: Wed, 12 Dec 2018 12:55:43 +0000
Message-ID: <20181212125528.GA5538@minint-98vp2qg>
References: <cover.1543495935.git.anatoly.burakov@intel.com>
 <DB7PR05MB44263F105C5207974228EF4EC3AD0@DB7PR05MB4426.eurprd05.prod.outlook.com>
 <7F52E2C1-4552-48E7-9AFA-BEAF4D8E45D7@mellanox.com>
 <57d7ad23-bebd-f788-5ac9-540804ecf5da@intel.com>
In-Reply-To: <57d7ad23-bebd-f788-5ac9-540804ecf5da@intel.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-clientproxiedby: BYAPR07CA0032.namprd07.prod.outlook.com
 (2603:10b6:a02:bc::45) To DB3PR0502MB3980.eurprd05.prod.outlook.com
 (2603:10a6:8:10::27)
authentication-results: spf=none (sender IP is )
 smtp.mailfrom=yskoh@mellanox.com; 
x-ms-exchange-messagesentrepresentingtype: 1
x-originating-ip: [69.181.245.183]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; DB3PR0502MB4012;
 6:r7VMYJc598AeoH4zLxkFVJ+vjesEiKr8FY6GShn2OGrCz5yjF6TssTnDhRZAVjqCqv97S03EKAlO35iHIGEAAlYrXYctkW5NoIbku3TgOVcVoksGbnd7JmBeXWAASDaUn3gju5WaTLfbwVbKx5A14z2N9q3SKcg6xCNm+2ee+PS6X//VdruGFfMSlBHyvws3QG0eFj6cnslv3DF85KAzfKYWop6yXYdmfY/zBN3fIfO7QRNfWbErEMS4E6/5w1ov4kJ4lFeKwrCJ7zbpmqVTcbyPUl9Yk6h2TEA/3g4iYtpjZy94QUUAzxH6yL5fgPP6htpexmptTwBlNoRDPTq3r739BYkjyueViRPg+ZWcLbzA+dean56Vk1v0izA3fdahayoBkSCiHfZ3AudFKnYg86PqxAnwtNeLQG8MA7XfzBBfs10VTCQ+CUR5UJrxgbTt7QnqTw5wApFjAiMpjzw1aw==;
 5:DFAVchDPmPlgWP5GCs1AyoKXT6HJvHw3b2x/7wNNXRRLGDSoRPcvvudjKTMfacuR77WWMe1/aaelT3qCnrlmSIob5UY/91N6Dwx6Pe761j32xhDwIVDrEI8p1nwz0Hq1vXrX+lyaaovg9AmyaO0mvxTwLrV4rGDeCrvArzF9h9I=;
 7:w/4ztnVIHD8ThOzP4X7tfH+ukADUalpT/9DsDcoXx63SqiaW4Wdf2x5RSCxJLqg8NWAlp31OxVF09Nr9a86tYZgWKfQNDIJaL/yAq2oqCyQQx5x/cOm3BIVBUCl9zjW6r+cP5bNgJtAPclMna6BDKA==
x-ms-office365-filtering-correlation-id: 1e3517dc-1b7b-44ca-ada2-08d660312083
x-ms-office365-filtering-ht: Tenant
x-microsoft-antispam: BCL:0; PCL:0;
 RULEID:(2390098)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020);
 SRVR:DB3PR0502MB4012; 
x-ms-traffictypediagnostic: DB3PR0502MB4012:
x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr
x-microsoft-antispam-prvs: <DB3PR0502MB4012CAB196D5BD2B3DA87E33C3A70@DB3PR0502MB4012.eurprd05.prod.outlook.com>
x-ms-exchange-senderadcheck: 1
x-exchange-antispam-report-cfa-test: BCL:0; PCL:0;
 RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(10201501046)(3231455)(999002)(944501520)(52105112)(93006095)(93001095)(3002001)(6055026)(148016)(149066)(150057)(6041310)(20161123558120)(20161123560045)(20161123562045)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699051)(76991095);
 SRVR:DB3PR0502MB4012; BCL:0; PCL:0; RULEID:; SRVR:DB3PR0502MB4012; 
x-forefront-prvs: 0884AAA693
x-forefront-antispam-report: SFV:NSPM;
 SFS:(10009020)(7916004)(136003)(346002)(366004)(39860400002)(376002)(396003)(69234005)(199004)(189003)(52116002)(966005)(71200400001)(71190400001)(68736007)(86362001)(106356001)(6436002)(5660300001)(105586002)(6916009)(7736002)(33716001)(1076002)(81156014)(81166006)(8936002)(478600001)(14454004)(305945005)(5024004)(2906002)(229853002)(6512007)(6116002)(4326008)(97736004)(476003)(446003)(256004)(53546011)(102836004)(6246003)(186003)(8676002)(9686003)(66066001)(386003)(26005)(6306002)(486006)(25786009)(33656002)(6506007)(33896004)(6486002)(76176011)(99286004)(53936002)(93886005)(11346002)(316002)(3846002)(54906003)(45080400002);
 DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB4012;
 H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en;
 PTR:InfoNoRecords; A:1; MX:1; 
received-spf: None (protection.outlook.com: mellanox.com does not designate
 permitted sender hosts)
x-microsoft-antispam-message-info: xlWHd2FbCrE9XF1i5Tn3zzTwmKoHjPfSRw2fuw/WRvYWXRqFGC+ykk+VbW3RNG8zTFRx1mg/abK9r1cxJEqHG39gsJMZLniah3U7bnjEUKv0DRE1r9RiaQIeGfQgXy99Ij4np6BkqfrPq2NVQA5bJkbBmaz3RX4dk3VjMhzFLLrafJnOefV+NvrLA1yNiE+PtrmGX5L33PSYpOvw6j3opQTsMAx3R1E/o2WqD6eYZqz0hHl87lXqQWMiXBuByReu8x25GurUzLHi6svfllu6rD8eyOo/rhErFNdDmct4nETfR1tIpnVMaSoBYX5YJyED
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="us-ascii"
Content-ID: <44B154FFEC94584193FAF3EC508DFCF2@eurprd05.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: Mellanox.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1e3517dc-1b7b-44ca-ada2-08d660312083
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Dec 2018 12:55:43.9571 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB4012
Subject: Re: [dpdk-dev] [PATCH 0/4] Allow using external memory without
	malloc
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Wed, 12 Dec 2018 12:55:52 -0000

On Mon, Dec 03, 2018 at 10:23:03AM +0000, Burakov, Anatoly wrote:
> On 02-Dec-18 11:28 PM, Yongseok Koh wrote:
> >=20
> > > On Dec 1, 2018, at 9:48 PM, Shahaf Shuler <shahafs@mellanox.com> wrot=
e:
> > >=20
> > > Hi Anatoly,
> > >=20
> > > Thursday, November 29, 2018 3:49 PM, Anatoly Burakov:
> > > > Subject: [PATCH 0/4] Allow using external memory without malloc
> > > >=20
> > > > Currently, the only way to use externally allocated memory is throu=
gh
> > > > rte_malloc API's. While this is fine for a lot of use cases, it may=
 not be suitable
> > > > for certain other use cases like manual memory management, etc.
> > > >=20
> > > > This patchset adds another API to register memory segments with DPD=
K (so
> > > > that API's like ``rte_mem_virt2memseg`` could be relied on by PMD's=
 and
> > > > such), but not create a malloc heap out of them.
> > > >=20
> > > > Aside from the obvious (not adding memory to a heap), the other maj=
or
> > > > difference between this API and the ``rte_malloc_heap_*`` external =
memory
> > > > functions is the fact that no DMA mapping is performed automaticall=
y.
> > > >=20
> > > > This really draws a line in the sand, and there are now two ways of=
 doing
> > > > things - do everything automatically (using the ``rte_malloc_heap_*=
`` API's),
> > > > or do everything manually (``rte_extmem_*`` and future DMA mapping =
API
> > > > [1] that would replace ``rte_vfio_dma_map``). This way, the consist=
ency of
> > > > API is kept, and flexibility is also allowed.
> > > >=20
> > >=20
> > > As you know I like the idea.
> > > One question though, do you see a use case for application to have ex=
ternally allocated memory which needs to be registered to the DPDK subsyste=
m however not being used for DMA?
> > > My only guess would be so helper libraries which requires the memory =
allocation from user (however it doesn't seems like a good API).
> > >=20
> > > If no use case, maybe it is better to merge between the two (rte_extm=
em_* and rte_dma_map) to have a single call for app to register and DMA map=
 the memory. The rte_mem_virt2memseg is not something application needs to =
understand, it is used internally by PMDs or other libs.
> >=20
> > Just FYI.
> > My implementation for mlx4/5 doesn't need to have a separate registrati=
on for
> > DMA by rte_dma_map() as long as it is included in the memseg list. Regi=
stration
> > is done only if Lkey lookup misses and only mem free event is taken to =
release
> > it. From my end, the reason why we wanted to have a generic DMA registr=
ation was
> > because some people doesn't want to use the new API to make the ext mem=
 included
> > in the memseg list but want to simply call the API for DMA registration=
.
> >=20
> > In a nutshell, mlx4/5 needs users use either rte_extmem_register() or
> > rte_dma_map(). However, it is no problem to call both.
>=20
> It would be good to create a segment when using rte_dma_map().
> Unfortunately, that's not realistic :)
>=20
> Registering memory within DPDK does not necessarily have to be performed =
by
> the primary process - whichever process that wants to create the table, c=
an
> do so, and later processes have to attach to the memory area. There's als=
o
> no way to know if memory segment can be attached to - this is a question
> only application can answer.
>=20
> In other words, there's no way to combine rte_dma_map() and
> rte_extmem_register() into one call and keep multiprocess support.

Sorry for late reply. I was away for a while.

I understood your point that rte_dma_map() can't create a segment but isn't=
 the
opposite possible? I still have a question about
rte_extmem_register/unregister/attach/detach(). Why don't these APIs genera=
te
memory events? Do you define the memory events are limited to memories for
malloc? What if some app wants to know the events even if it is extmem? Wha=
t
makes difference between two types of extmem (one for malloc heap and the o=
ther
for just memseg) in generating the events?

I've reviewed your patches and all look good :-) But it is still unclear to=
 me.


Thanks,
Yongseok

> > > > [1]
> > > > https://emea01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%=
2Fma
> > > > ils.dpdk.org%2Farchives%2Fdev%2F2018-
> > > > November%2F118175.html&amp;data=3D02%7C01%7Cshahafs%40mellanox.co
> > > > m%7C007a9234feaf42c82f6508d656015eb1%7Ca652971c7d2e4d9ba6a4d1492
> > > > 56f461b%7C0%7C0%7C636790961244424277&amp;sdata=3DYqwcPEEhJM3I7Toe
> > > > Ne%2BGcbeo%2FmPbYEnNFckoA7ES2Hg%3D&amp;reserved=3D0
> > > >=20
> > > > Note: at the time of this writing, there's no release notes
> > > >       template, so no release notes updates in this patchset.
> > > >       They will be added in later revisions.
> > > >=20
> > > > Anatoly Burakov (4):
> > > >   malloc: separate creating memseg list and malloc heap
> > > >   malloc: separate destroying memseg list and heap data
> > > >   mem: allow registering external memory areas
> > > >   mem: allow usage of non-heap external memory in multiprocess
> > > >=20
> > > > .../prog_guide/env_abstraction_layer.rst      |  63 +++++++--
> > > > lib/librte_eal/common/eal_common_memory.c     | 116
> > > > +++++++++++++++++
> > > > lib/librte_eal/common/include/rte_memory.h    | 122
> > > > ++++++++++++++++++
> > > > lib/librte_eal/common/malloc_heap.c           | 104 +++++++++++----
> > > > lib/librte_eal/common/malloc_heap.h           |  15 ++-
> > > > lib/librte_eal/common/rte_malloc.c            | 115 +++++++--------=
--
> > > > lib/librte_eal/rte_eal_version.map            |   4 +
> > > > 7 files changed, 434 insertions(+), 105 deletions(-)
> > > >=20
> > > > --
> > > > 2.17.1
> >=20
> >=20
>=20
>=20
> --=20
> Thanks,
> Anatoly