From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-eopbgr30046.outbound.protection.outlook.com [40.107.3.46]) by dpdk.org (Postfix) with ESMTP id 815434CB5 for ; Thu, 13 Sep 2018 09:44:17 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Djsf/CqAGDMeEkzO9OE9HmCt9zLw2hFS8SF3PJ9FdYA=; b=XwyAxvE5AEiDe29z1iNE2lW/BrA43poSFeZKRxcsR+Mv5rBJt8mpNoqZFwowlsshilUsuwkf9VGSBwY3T0QvAbOTrt8od6IZxXU/MXsaUpet+CgTzHs8TANs0xPasMn2ATBmC2MOAySer5rvvZYz8/lr+gShJEqG8WfRDIPmsvM= Received: from DB7PR05MB4426.eurprd05.prod.outlook.com (52.134.109.15) by DB7PR05MB4939.eurprd05.prod.outlook.com (20.176.235.151) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1122.18; Thu, 13 Sep 2018 07:44:15 +0000 Received: from DB7PR05MB4426.eurprd05.prod.outlook.com ([fe80::c8e7:d9c1:5054:693b]) by DB7PR05MB4426.eurprd05.prod.outlook.com ([fe80::c8e7:d9c1:5054:693b%6]) with mapi id 15.20.1122.020; Thu, 13 Sep 2018 07:44:15 +0000 From: Shahaf Shuler To: Anatoly Burakov , "dev@dpdk.org" CC: "laszlo.madarassy@ericsson.com" , "laszlo.vadkerti@ericsson.com" , "andras.kovacs@ericsson.com" , "winnie.tian@ericsson.com" , "daniel.andrasi@ericsson.com" , "janos.kobor@ericsson.com" , "srinath.mannam@broadcom.com" , "scott.branden@broadcom.com" , "ajit.khaparde@broadcom.com" , "keith.wiles@intel.com" , "bruce.richardson@intel.com" , Thomas Monjalon Thread-Topic: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in DPDK Thread-Index: AQHURFDf2xhCgwt9L0+pUaIzcsQ80aTt4bgQ Date: Thu, 13 Sep 2018 07:44:15 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; x-originating-ip: [193.47.165.251] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DB7PR05MB4939; 6:+ViBy4Q+oEMaLuYdnCKYbHKsGEUksBE/Nal1LR1AO8rg2uWLudHSzfl19oC5DFTtCDcQsvgm1b+8phgbxvOXQb9cQAuN/8nf+jtWa6XR6q/Wv1NfEnJNqzNaNQdjs/p3DSSpqIPw1IbXshhIf1bMSpZT0cdMSd26YRw4fXcE5Vjo3gRoONW6V9H+GCpYneu88MTrhxqGBPDpwNkvnJ88rvDxpZGGfopx2jUx9KPwbB40OGkxEjkmB5S4QBSqYCGECYsG7ZHoKQNNr8XK6u1ONaXCONb0VFCooKPRwzFed2dfy7HsG5x4dqr/BSyMS4NV6pKGGFTKD47HNRGw5jmP8qTYcbDqxuznTgflsDwoC6ou9nUC6fyh0TI4p1tA/XkBnmoHh+j+yHznzkG6T+W/CqL/XIaFtIXUd7qBluQiVG16x63uHcH37d8ps5znA+Z1XmT9By1ztXacusxAfxGePQ==; 5:i5vgojNRkabRah5Wjos8laOmCP8cQbldHRgH8gQHyrGyDJ7veAKldjd4DJKm/eQwA3EdZMFCe9zoZdeZjyuOY43674eNFflWEXuTD3oNkql4L+R6laHtMEgYU2ttY09AVunSZxI5/B5cTw97OKfA7LPmcOqT5FMs9lrYsl7Y6R0=; 7:IGufxm61os+zD8o/VDESASeYVN/XO28I/iJPTbGvMR9h4yEr/beO+v0Chu/5+msH9SX9zUaqoAe89XBbjQkNA7iRdKuC/iw/4bhPLg1nAXsUzNa06ZevePlCAMvkPi7OwvLLPvOkLWtoVjeTD8H2op+MDf5/uaVM8TaOGmEj3eJIpmp6kXygt/lXpxz42/Dy3WAArKpC+rltLOi0UGHJ/sjOscCiECaDQmpsXRvgkyGyqIY53NnIz2bcS7joM1+E x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 449df17e-f2ea-4f1b-2e82-08d6194cb44a x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989137)(4534165)(4627221)(201703031133081)(201702281549075)(8990107)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:DB7PR05MB4939; x-ms-traffictypediagnostic: DB7PR05MB4939: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(3231311)(944501410)(52105095)(3002001)(10201501046)(93006095)(93001095)(6055026)(149027)(150027)(6041310)(20161123564045)(20161123562045)(20161123558120)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699050); SRVR:DB7PR05MB4939; BCL:0; PCL:0; RULEID:; SRVR:DB7PR05MB4939; x-forefront-prvs: 07943272E1 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(366004)(39860400002)(376002)(346002)(136003)(51914003)(199004)(189003)(229853002)(97736004)(8676002)(2900100001)(68736007)(14454004)(561944003)(8936002)(33656002)(105586002)(6116002)(81166006)(81156014)(26005)(2906002)(106356001)(3846002)(6506007)(102836004)(9686003)(7696005)(76176011)(99286004)(7416002)(316002)(110136005)(55016002)(54906003)(66066001)(5250100002)(53936002)(5024004)(14444005)(256004)(4326008)(25786009)(186003)(5660300001)(6246003)(446003)(11346002)(476003)(486006)(86362001)(2501003)(7736002)(478600001)(74316002)(6436002)(305945005); DIR:OUT; SFP:1101; SCL:1; SRVR:DB7PR05MB4939; H:DB7PR05MB4426.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: S0GryIeeOtFbHGUmIJzotP+b9dqoznainv51bXurv5S5dDT+Hfo9di3JgtCuqumSyb8UL0+SBARCR+WKWMrEjJT4ipc8KypOXKEFQfMtLW+P9YXg2pTSRwPSZK2PqgQ0is/9Rm/7lCz7Ozbg41SBkru4iJu+7TWFvOobLQbzK3Wv0ObbI0Kejire8SRO/BWaYrcL7KhnZNHYDW+mXekIoN0VPwif/cs3J/ZTKKi3NHoKZiY1dz+SgAGa6TfQMzW2Uk0/t6552d5oqHQcAU+FuNg1WdjgkbE15uBGZyhLi6yKX721luPf7Mguo7qhl/oZUXXeV4U7ClJ9tO9i3BxuodhUed7iUAVx6CIe7VcQr04= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 449df17e-f2ea-4f1b-2e82-08d6194cb44a X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Sep 2018 07:44:15.1412 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB7PR05MB4939 Subject: Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in DPDK X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Sep 2018 07:44:17 -0000 Hi Anatoly, First thanks for the patchset, it is a great enhancement.=20 See question below.=20 Tuesday, September 4, 2018 4:12 PM, Anatoly Burakov: > Subject: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in > DPDK >=20 > This is a proposal to enable using externally allocated memory in DPDK. >=20 > In a nutshell, here is what is being done here: >=20 > - Index internal malloc heaps by NUMA node index, rather than NUMA > node itself (external heaps will have ID's in order of creation) > - Add identifier string to malloc heap, to uniquely identify it > - Each new heap will receive a unique socket ID that will be used by > allocator to decide from which heap (internal or external) to > allocate requested amount of memory > - Allow creating named heaps and add/remove memory to/from those > heaps > - Allocate memseg lists at runtime, to keep track of IOVA addresses > of externally allocated memory > - If IOVA addresses aren't provided, use RTE_BAD_IOVA > - Allow malloc and memzones to allocate from external heaps > - Allow other data structures to allocate from externall heaps >=20 > The responsibility to ensure memory is accessible before using it is on t= he > shoulders of the user - there is no checking done with regards to validit= y of > the memory (nor could there be...). That makes sense. However who should be in-charge of mapping this memory fo= r dma access? The user or internally be the PMD when encounter the first packet or while = traversing the existing mempools?=20 >=20 > The general approach is to create heap and add memory into it. For any ot= her > process wishing to use the same memory, said memory must first be > attached (otherwise some things will not work). >=20 > A design decision was made to make multiprocess synchronization a manual > process. Due to underlying issues with attaching to fbarrays in secondary > processes, this design was deemed to be better because we don't want to > fail to create external heap in the primary because something in the > secondary has failed when in fact we may not eve have wanted this memory > to be accessible in the secondary in the first place. >=20 > Using external memory in multiprocess is *hard*, because not only memory > space needs to be preallocated, but it also needs to be attached in each > process to allow other processes to access the page table. The attach API= call > may or may not succeed, depending on memory layout, for reasons similar t= o > other multiprocess failures. This is treated as a "known issue" for this = release. >=20 > RFC -> v1 changes: > - Removed the "named heaps" API, allocate using fake socket ID instead > - Added multiprocess support > - Everything is now thread-safe > - Numerous bugfixes and API improvements >=20 > Anatoly Burakov (16): > mem: add length to memseg list > mem: allow memseg lists to be marked as external > malloc: index heaps using heap ID rather than NUMA node > mem: do not check for invalid socket ID > flow_classify: do not check for invalid socket ID > pipeline: do not check for invalid socket ID > sched: do not check for invalid socket ID > malloc: add name to malloc heaps > malloc: add function to query socket ID of named heap > malloc: allow creating malloc heaps > malloc: allow destroying heaps > malloc: allow adding memory to named heaps > malloc: allow removing memory from named heaps > malloc: allow attaching to external memory chunks > malloc: allow detaching from external memory > test: add unit tests for external memory support >=20 > config/common_base | 1 + > config/rte_config.h | 1 + > drivers/bus/fslmc/fslmc_vfio.c | 7 +- > drivers/bus/pci/linux/pci.c | 2 +- > drivers/net/mlx4/mlx4_mr.c | 3 + > drivers/net/mlx5/mlx5.c | 5 +- > drivers/net/mlx5/mlx5_mr.c | 3 + > drivers/net/virtio/virtio_user/vhost_kernel.c | 5 +- > lib/librte_eal/bsdapp/eal/eal.c | 3 + > lib/librte_eal/bsdapp/eal/eal_memory.c | 9 +- > lib/librte_eal/common/eal_common_memory.c | 9 +- > lib/librte_eal/common/eal_common_memzone.c | 8 +- > .../common/include/rte_eal_memconfig.h | 6 +- > lib/librte_eal/common/include/rte_malloc.h | 181 +++++++++ > .../common/include/rte_malloc_heap.h | 3 + > lib/librte_eal/common/include/rte_memory.h | 9 + > lib/librte_eal/common/malloc_heap.c | 287 +++++++++++-- > lib/librte_eal/common/malloc_heap.h | 17 + > lib/librte_eal/common/rte_malloc.c | 383 ++++++++++++++++- > lib/librte_eal/linuxapp/eal/eal.c | 3 + > lib/librte_eal/linuxapp/eal/eal_memalloc.c | 12 +- > lib/librte_eal/linuxapp/eal/eal_memory.c | 4 +- > lib/librte_eal/linuxapp/eal/eal_vfio.c | 17 +- > lib/librte_eal/rte_eal_version.map | 7 + > lib/librte_flow_classify/rte_flow_classify.c | 3 +- > lib/librte_mempool/rte_mempool.c | 31 +- > lib/librte_pipeline/rte_pipeline.c | 3 +- > lib/librte_sched/rte_sched.c | 2 +- > test/test/Makefile | 1 + > test/test/autotest_data.py | 14 +- > test/test/meson.build | 1 + > test/test/test_external_mem.c | 384 ++++++++++++++++++ > test/test/test_malloc.c | 3 + > test/test/test_memzone.c | 3 + > 34 files changed, 1346 insertions(+), 84 deletions(-) create mode 10064= 4 > test/test/test_external_mem.c >=20 > -- > 2.17.1