From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-eopbgr70055.outbound.protection.outlook.com [40.107.7.55]) by dpdk.org (Postfix) with ESMTP id 404F81B9BB for ; Fri, 14 Dec 2018 10:56:31 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dvEevM0LoyGZo6DgUIF0xgmBDyZQpGf3q/lWFgCJgSw=; b=gz3/Q285pNZFObNmp7ink/IagyJOow/YFXiESBpDxwGq7oYOT4M0LF8aB7YMLhbGGleGTTHcAtVVVqmeXsAozDw5LvuJFKCqTkCr6fG4DIf1P/Gh6cEXq348GT56337mSrDJg9VYLT/ZEF00LrlXySnq45ppEZaJcQxjozKgmFw= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB3977.eurprd05.prod.outlook.com (52.134.70.150) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1404.22; Fri, 14 Dec 2018 09:56:29 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::dcbc:4578:3018:50f3]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::dcbc:4578:3018:50f3%5]) with mapi id 15.20.1404.026; Fri, 14 Dec 2018 09:56:29 +0000 From: Yongseok Koh To: Anatoly Burakov CC: "dev@dpdk.org" , John McNamara , Marko Kovacevic , Shahaf Shuler , Thomas Monjalon , "shreyansh.jain@nxp.com" Thread-Topic: [PATCH 4/4] mem: allow usage of non-heap external memory in multiprocess Thread-Index: AQHUh+o+tnzVI/057k+EiqEso+Y5IqV+FrIA Date: Fri, 14 Dec 2018 09:56:29 +0000 Message-ID: <20181214095620.GD12221@mtidpdk.mti.labs.mlnx> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: BYAPR04CA0026.namprd04.prod.outlook.com (2603:10b6:a03:40::39) To DB3PR0502MB3980.eurprd05.prod.outlook.com (2603:10a6:8:10::27) authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [209.116.155.178] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DB3PR0502MB3977; 6:jnxkI+ogwfQW3S0pfZsSayA5uw4qaTITnBlMaz3NosumdZgaT2DWln/NqmGpxN6O7LFwG7KkWIiJzqIjOPYehluEV1k5OXUUSKNj0k424TiCq1fq/0IhsrnbR6RI8zSSajk1Umqr+dle6rqdiPhEc0ZRRUKMs7ns54n/oYCE5rwSmSzUJiKcBZBGHWBOJFfzh0IjOA5mgwJgD+bfj55tPtwz9AiG1IBIW4vBvwFPJrgcN9sgGGeksrbz2R2pyCs6nRcd6FrE8uji+Td8NqpYZ6Q1wsUq5eFe1tPxnAJftFa6W1LaFD3QOtD9bPdy9aA9G9PoAfhIH57Vl21PBoLSt1Lx9TKfJzkWd/oApvFxD90lqsJc/FrNPSdgnq1dgb6FLBvIJ8jyQmss2l0hWyyrbrNlUOFJwLNjPWLBNfDMIvWjkiAcWUDRvg0Rs+eQLrpGrr/FND7nyKJxx3WDjQgK+A==; 5:q/E9eP8uLgoLh72ue3Benc7kggqD8aWTzGFvVcJcoepRlR+YQHMv+Y84Y9ATJ8w0msGSlg1NtrsLyHoPDRi9C8ZxmKFGXzTKat/jMoNS8HwkIXQoASfMpVxMtBLmNtmTdfzoufNe+LmGt8rTuLQdxu/N+5aWcqHAyxw5yAG7DBg=; 7:ZErKQ8bPRS/W6pTyp71ZLIYvRCflC1RPs9q15xtBj4iQj5T/IxR2wLfm+1/VZRwSyr65mnWZOE3GOvmq5TTkejSEgc5hkvyQ7DhXlbUPgI7ccFvQUL65lZ1NU3693IqqOcgZtUbCCD2LtrTukyByPg== x-ms-office365-filtering-correlation-id: 59217504-fc23-4811-eb73-08d661aa6b0d x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:DB3PR0502MB3977; x-ms-traffictypediagnostic: DB3PR0502MB3977: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(3230021)(999002)(6040522)(2401047)(8121501046)(5005006)(10201501046)(3002001)(3231475)(944501520)(52105112)(93006095)(93001095)(6055026)(148016)(149066)(150057)(6041310)(20161123564045)(20161123560045)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(201708071742011)(7699051)(76991095); SRVR:DB3PR0502MB3977; BCL:0; PCL:0; RULEID:; SRVR:DB3PR0502MB3977; x-forefront-prvs: 08864C38AC x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(376002)(346002)(39860400002)(136003)(396003)(189003)(199004)(71190400001)(102836004)(386003)(11346002)(486006)(7736002)(6506007)(2906002)(446003)(33656002)(476003)(71200400001)(54906003)(256004)(6916009)(4326008)(186003)(14444005)(305945005)(97736004)(5024004)(105586002)(81166006)(8936002)(1076002)(6246003)(14454004)(81156014)(53936002)(6512007)(6116002)(316002)(68736007)(26005)(99286004)(52116002)(6486002)(3846002)(9686003)(478600001)(229853002)(106356001)(6436002)(86362001)(5660300001)(33896004)(76176011)(66066001)(8676002)(25786009); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB3977; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: YJ+NQxaB87xwEqzcEBJMreTr2ZnWHGcvIlxwXPivdECfo2l+g8Ni408B9pWdEqtTIdp6oBy0k9b8ZKUqVrTqsmaXDp0x2BJWteAtZ/uVLHzOb8iytLAyzzyQ9uTKJcpLOFFMyflhnrsybCTfmrfp9zF2aNj/aJwfgqsjzu18fkcgvY6Vsz0ARfmNBhmz9JsfteEOK7G9ZHTRDQ9w7aiXBw/j7mVApjhO3KnXwvJ8Ghrj+VWqgLEZ3YmHRDTfb0/gtPQAwfOfk8QfRlPpNALOBbXOTpQ53TZ5QMyamDeahUcruTRtMg0L9vgMJIvyUH8r spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-ID: <1B38D0E9FF213F45B34D9164091A385F@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 59217504-fc23-4811-eb73-08d661aa6b0d X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Dec 2018 09:56:29.1265 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB3977 Subject: Re: [dpdk-dev] [PATCH 4/4] mem: allow usage of non-heap external memory in multiprocess X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Dec 2018 09:56:31 -0000 On Thu, Nov 29, 2018 at 01:48:35PM +0000, Anatoly Burakov wrote: > Add multiprocess support for externally allocated memory areas that > are not added to DPDK heap (and add relevant doc sections). >=20 > Signed-off-by: Anatoly Burakov > --- Acked-by: Yongseok Koh Thanks > .../prog_guide/env_abstraction_layer.rst | 3 + > lib/librte_eal/common/eal_common_memory.c | 42 +++++++++++++ > lib/librte_eal/common/include/rte_memory.h | 59 +++++++++++++++++++ > lib/librte_eal/rte_eal_version.map | 2 + > 4 files changed, 106 insertions(+) >=20 > diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides= /prog_guide/env_abstraction_layer.rst > index d7799b626..b0491bf2d 100644 > --- a/doc/guides/prog_guide/env_abstraction_layer.rst > +++ b/doc/guides/prog_guide/env_abstraction_layer.rst > @@ -276,11 +276,14 @@ The expected workflow is as follows: > * Register memory within DPDK > - If IOVA table is not specified, IOVA addresses will be assumed to = be > unavailable > + - Other processes must attach to the memory area before they can use= it > * Perform DMA mapping with ``rte_vfio_dma_map`` if needed > * Use the memory area in your application > * If memory area is no longer needed, it can be unregistered > - If the area was mapped for DMA, unmapping must be performed before > unregistering memory > + - Other processes must detach from the memory area before it can be > + unregistered > =20 > Since these externally allocated memory areas will not be managed by DPD= K, it is > therefore up to the user application to decide how to use them and what = to do > diff --git a/lib/librte_eal/common/eal_common_memory.c b/lib/librte_eal/c= ommon/eal_common_memory.c > index a2e085ae8..67b445c31 100644 > --- a/lib/librte_eal/common/eal_common_memory.c > +++ b/lib/librte_eal/common/eal_common_memory.c > @@ -849,6 +849,48 @@ rte_extmem_unregister(void *va_addr, size_t len) > return ret; > } > =20 > +static int > +sync_memory(void *va_addr, size_t len, bool attach) > +{ > + struct rte_mem_config *mcfg =3D rte_eal_get_configuration()->mem_config= ; > + struct rte_memseg_list *msl; > + int ret =3D 0; > + > + if (va_addr =3D=3D NULL || len =3D=3D 0) { > + rte_errno =3D EINVAL; > + return -1; > + } > + rte_rwlock_write_lock(&mcfg->memory_hotplug_lock); > + > + /* find our segment */ > + msl =3D malloc_heap_find_external_seg(va_addr, len); > + if (msl =3D=3D NULL) { > + rte_errno =3D ENOENT; > + ret =3D -1; > + goto unlock; > + } > + if (attach) > + ret =3D rte_fbarray_attach(&msl->memseg_arr); > + else > + ret =3D rte_fbarray_detach(&msl->memseg_arr); > + > +unlock: > + rte_rwlock_write_unlock(&mcfg->memory_hotplug_lock); > + return ret; > +} > + > +int __rte_experimental > +rte_extmem_attach(void *va_addr, size_t len) > +{ > + return sync_memory(va_addr, len, true); > +} > + > +int __rte_experimental > +rte_extmem_detach(void *va_addr, size_t len) > +{ > + return sync_memory(va_addr, len, false); > +} > + > /* init memory subsystem */ > int > rte_eal_memory_init(void) > diff --git a/lib/librte_eal/common/include/rte_memory.h b/lib/librte_eal/= common/include/rte_memory.h > index 4a43c1a9e..050bb6d8e 100644 > --- a/lib/librte_eal/common/include/rte_memory.h > +++ b/lib/librte_eal/common/include/rte_memory.h > @@ -435,6 +435,10 @@ rte_memseg_get_fd_offset_thread_unsafe(const struct = rte_memseg *ms, > * @note This API will not perform any DMA mapping. It is expected that = user > * will do that themselves. > * > + * @note Before accessing this memory in other processes, it needs to be > + * attached in each of those processes by calling ``rte_extmem_attach`= ` in > + * each other process. > + * > * @param va_addr > * Start of virtual area to register > * @param len > @@ -472,6 +476,9 @@ rte_extmem_register(void *va_addr, size_t len, rte_io= va_t iova_addrs[], > * @note This API will not perform any DMA unmapping. It is expected tha= t user > * will do that themselves. > * > + * @note Before calling this function, all other processes must call > + * ``rte_extmem_detach`` to detach from the memory area. > + * > * @param va_addr > * Start of virtual area to unregister > * @param len > @@ -486,6 +493,58 @@ rte_extmem_register(void *va_addr, size_t len, rte_i= ova_t iova_addrs[], > int __rte_experimental > rte_extmem_unregister(void *va_addr, size_t len); > =20 > +/** > + * @warning > + * @b EXPERIMENTAL: this API may change without prior notice > + * > + * Attach to external memory chunk registered in another process. > + * > + * @note Using this API is mutually exclusive with ``rte_malloc`` family= of > + * API's. > + * > + * @note This API will not perform any DMA mapping. It is expected that = user > + * will do that themselves. > + * > + * @param va_addr > + * Start of virtual area to register > + * @param len > + * Length of virtual area to register > + * > + * @return > + * - 0 on success > + * - -1 in case of error, with rte_errno set to one of the following: > + * EINVAL - one of the parameters was invalid > + * ENOENT - memory chunk was not found > + */ > +int __rte_experimental > +rte_extmem_attach(void *va_addr, size_t len); > + > +/** > + * @warning > + * @b EXPERIMENTAL: this API may change without prior notice > + * > + * Detach from external memory chunk registered in another process. > + * > + * @note Using this API is mutually exclusive with ``rte_malloc`` family= of > + * API's. > + * > + * @note This API will not perform any DMA unmapping. It is expected tha= t user > + * will do that themselves. > + * > + * @param va_addr > + * Start of virtual area to unregister > + * @param len > + * Length of virtual area to unregister > + * > + * @return > + * - 0 on success > + * - -1 in case of error, with rte_errno set to one of the following: > + * EINVAL - one of the parameters was invalid > + * ENOENT - memory chunk was not found > + */ > +int __rte_experimental > +rte_extmem_detach(void *va_addr, size_t len); > + > /** > * Dump the physical memory layout to a file. > * > diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_= version.map > index 593691a14..eb5f7b9cb 100644 > --- a/lib/librte_eal/rte_eal_version.map > +++ b/lib/librte_eal/rte_eal_version.map > @@ -296,6 +296,8 @@ EXPERIMENTAL { > rte_devargs_remove; > rte_devargs_type_count; > rte_eal_cleanup; > + rte_extmem_attach; > + rte_extmem_detach; > rte_extmem_register; > rte_extmem_unregister; > rte_fbarray_attach; > --=20 > 2.17.1