From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <yskoh@mellanox.com>
Received: from EUR04-HE1-obe.outbound.protection.outlook.com
(mail-eopbgr70057.outbound.protection.outlook.com [40.107.7.57])
by dpdk.org (Postfix) with ESMTP id EEBE51B96B
for <dev@dpdk.org>; Fri, 14 Dec 2018 10:34:16 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com;
s=selector1;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=EvoW3BWRxtsrDRTH3Kubb8kGP6ZiHL8QTL5sT/EC9cQ=;
b=Oci1DHj/W8+0iBH7zP1V65ykE87MqtE2+wnx+pp1Nu+I4SYtEzXG/r5csAPcG0Muh3l3Zv5fVXKCq3woK6d9Hu+N4UwiR5Q+938tigrF8oOLyAoWX9GJR5F9fS70vce/2pQ8gHsjQCER4vyjfa/bzaUwB9c/rKKR8SBNilh9OtQ=
Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by
DB3PR0502MB4090.eurprd05.prod.outlook.com (52.134.72.159) with Microsoft SMTP
Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
15.20.1425.22; Fri, 14 Dec 2018 09:34:15 +0000
Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com
([fe80::dcbc:4578:3018:50f3]) by DB3PR0502MB3980.eurprd05.prod.outlook.com
([fe80::dcbc:4578:3018:50f3%5]) with mapi id 15.20.1404.026; Fri, 14 Dec 2018
09:34:15 +0000
From: Yongseok Koh <yskoh@mellanox.com>
To: Anatoly Burakov <anatoly.burakov@intel.com>
CC: "dev@dpdk.org" <dev@dpdk.org>, Shahaf Shuler <shahafs@mellanox.com>,
Thomas Monjalon <thomas@monjalon.net>, "shreyansh.jain@nxp.com"
<shreyansh.jain@nxp.com>
Thread-Topic: [PATCH 2/4] malloc: separate destroying memseg list and heap data
Thread-Index: AQHUh+o9Nu/mwoBtska1pmH26GwjdqV+EH2A
Date: Fri, 14 Dec 2018 09:34:15 +0000
Message-ID: <20181214093406.GB12221@mtidpdk.mti.labs.mlnx>
References: <cover.1543495935.git.anatoly.burakov@intel.com>
<54cb382e91119cfbef7c34db4406f8ea39428511.1543495935.git.anatoly.burakov@intel.com>
In-Reply-To: <54cb382e91119cfbef7c34db4406f8ea39428511.1543495935.git.anatoly.burakov@intel.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-clientproxiedby: BYAPR05CA0088.namprd05.prod.outlook.com
(2603:10b6:a03:e0::29) To DB3PR0502MB3980.eurprd05.prod.outlook.com
(2603:10a6:8:10::27)
authentication-results: spf=none (sender IP is )
smtp.mailfrom=yskoh@mellanox.com;
x-ms-exchange-messagesentrepresentingtype: 1
x-originating-ip: [209.116.155.178]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; DB3PR0502MB4090;
6:FzOYLwCUmHgmKOaFj4VRiEMCEZjVF7DYZ8pwcZJyDqlT5AVViZhA2o+AE9+S4KkmZpi5KKmQk0Cf7oEUmj9Mzl/+2eQUAISCagzOoN/O73Q7D0bzL/YZCWBnafy2+i28jctIW+hoHiTXNy0BLPGyGDY7KuzQYghDDw9b480PwI5sd6i579EQ6D93vdskSnEKoN02slyQS9/fd3kmsPUStmx1iRmNCXNEbkxUpT+WUsguifEg/la+6ze9F1kU7/bXHj4GeF2Hf89Z+vuTR8pgFHMhFke7vmwMnbISf06lm3uGrc9R5olCgJQB3PWU0XbvFeKXdVTw/53DwRpYH0ZbeLKs0MiK6pWGCfkstC0lo3VEl75GM+hEHl1i11R5XbdXdWaL97cmfuirHPbtb7azzBJQey+5kX0UW+JoEnJx8XypQLfTQUbpcUs9vF9TzFpuFKFo5Qr/P7N1KyCkK1PENA==;
5:wJ0ffmZESJClhZg/ybJc/VvukCUR1mcaa7KzXtewG0Y6T5k/e3XW3fGgCz6CLog6MFMLMe/3SgC0qZzLqibL58Je69AEShxr1jLGVn0Flm0mGI00JSCvU7IrBDlyxIkEaL8D0dtNIPhOHT00PPSpvYp4ztEYuY+/APw7ogDlsCU=;
7:EifBsqPYT+eF3MKmy9fEBZ5JgTodw2Yp5Tu/pTYvq2L4gtLNJJq/omBnv6H6zi+GC2riSjVx0U0oUE0a1W8RcYd3gHv0bUDFj0ORvLj/7uuGYhWLwti9B3+sN3bDFK6L6hGQmuDI+8wNb+wO7wgygA==
x-ms-office365-filtering-correlation-id: 63ee0709-77c1-44a0-a571-08d661a75010
x-ms-office365-filtering-ht: Tenant
x-microsoft-antispam: BCL:0; PCL:0;
RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020);
SRVR:DB3PR0502MB4090;
x-ms-traffictypediagnostic: DB3PR0502MB4090:
x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr
x-microsoft-antispam-prvs: <DB3PR0502MB4090E79C828E97A9AB99930CC3A10@DB3PR0502MB4090.eurprd05.prod.outlook.com>
x-ms-exchange-senderadcheck: 1
x-exchange-antispam-report-cfa-test: BCL:0; PCL:0;
RULEID:(8211001083)(3230021)(999002)(6040522)(2401047)(5005006)(8121501046)(3231475)(944501520)(52105112)(93006095)(93001095)(10201501046)(3002001)(6055026)(148016)(149066)(150057)(6041310)(20161123558120)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123562045)(201708071742011)(7699051)(76991095);
SRVR:DB3PR0502MB4090; BCL:0; PCL:0; RULEID:; SRVR:DB3PR0502MB4090;
x-forefront-prvs: 08864C38AC
x-forefront-antispam-report: SFV:NSPM;
SFS:(10009020)(346002)(376002)(396003)(136003)(39860400002)(366004)(189003)(199004)(6436002)(102836004)(105586002)(6506007)(33896004)(386003)(8936002)(81166006)(52116002)(2906002)(7736002)(81156014)(8676002)(76176011)(256004)(575784001)(316002)(99286004)(71200400001)(71190400001)(66066001)(86362001)(54906003)(97736004)(14444005)(5024004)(446003)(5660300001)(476003)(25786009)(68736007)(305945005)(486006)(14454004)(11346002)(53936002)(26005)(229853002)(6916009)(3846002)(6512007)(9686003)(6246003)(106356001)(6116002)(4326008)(33656002)(1076002)(478600001)(186003)(6486002);
DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB4090;
H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en;
PTR:InfoNoRecords; MX:1; A:1;
received-spf: None (protection.outlook.com: mellanox.com does not designate
permitted sender hosts)
x-microsoft-antispam-message-info: gZc/8aLjJC9fCfor7XuWHm8W2rSOYYmAmjuWUXJ+0mrtxPwPNo4FdSFapab3OcUS0Q62HgGiKJe9+A/PTmAyaFO0UdFNhxN8IvMQLEjTZIrrbx7qjEfLKK33/05ctbjSIa0EDx1edY1OG6tzkxYJFInGz/Z24fOOIArOV6Hyc4ziJvoBrk9AU6mUBiBrehyW8oBE3O1CzCKphsorU0Ofb/AoYMYdFWFfCeciW2kAh2vFxQhRpVE2D+g5VctV5t7R6N5HCkYS+Tl5ZogG7Z7cWoEGYde48UaO3iqenGhtx53a/DxFM0m6fuWT9EusyCK8
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="us-ascii"
Content-ID: <D95A89A37BA5F54BA424DAACB1450DF0@eurprd05.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: Mellanox.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 63ee0709-77c1-44a0-a571-08d661a75010
X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Dec 2018 09:34:15.2396 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB4090
Subject: Re: [dpdk-dev] [PATCH 2/4] malloc: separate destroying memseg list
and heap data
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 14 Dec 2018 09:34:17 -0000
On Thu, Nov 29, 2018 at 01:48:33PM +0000, Anatoly Burakov wrote:
> Currently, destroying external heap chunk and its memseg list is
> part of one process. When we will gain the ability to unregister
> external memory from DPDK that doesn't have any heap structures
> associated with it, we need to be able to find and destroy
> memseg lists as well as heap data separately.
>=20
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
Acked-by: Yongseok Koh <yskoh@mellanox.com>
Thanks
> lib/librte_eal/common/malloc_heap.c | 70 +++++++++++++++----
> lib/librte_eal/common/malloc_heap.h | 6 ++
> lib/librte_eal/common/rte_malloc.c | 104 ++++++++++------------------
> 3 files changed, 102 insertions(+), 78 deletions(-)
>=20
> diff --git a/lib/librte_eal/common/malloc_heap.c b/lib/librte_eal/common/=
malloc_heap.c
> index 25693481f..fa0cb0799 100644
> --- a/lib/librte_eal/common/malloc_heap.c
> +++ b/lib/librte_eal/common/malloc_heap.c
> @@ -1067,12 +1067,9 @@ malloc_heap_dump(struct malloc_heap *heap, FILE *f=
)
> }
> =20
> static int
> -destroy_seg(struct malloc_elem *elem, size_t len)
> +destroy_elem(struct malloc_elem *elem, size_t len)
> {
> struct malloc_heap *heap =3D elem->heap;
> - struct rte_memseg_list *msl;
> -
> - msl =3D elem->msl;
> =20
> /* notify all subscribers that a memory area is going to be removed */
> eal_memalloc_mem_event_notify(RTE_MEM_EVENT_FREE, elem, len);
> @@ -1085,13 +1082,6 @@ destroy_seg(struct malloc_elem *elem, size_t len)
> =20
> memset(elem, 0, sizeof(*elem));
> =20
> - /* destroy the fbarray backing this memory */
> - if (rte_fbarray_destroy(&msl->memseg_arr) < 0)
> - return -1;
> -
> - /* reset the memseg list */
> - memset(msl, 0, sizeof(*msl));
> -
> return 0;
> }
> =20
> @@ -1158,6 +1148,62 @@ malloc_heap_create_external_seg(void *va_addr, rte=
_iova_t iova_addrs[],
> return msl;
> }
> =20
> +struct extseg_walk_arg {
> + void *va_addr;
> + size_t len;
> + struct rte_memseg_list *msl;
> +};
> +
> +static int
> +extseg_walk(const struct rte_memseg_list *msl, void *arg)
> +{
> + struct rte_mem_config *mcfg =3D rte_eal_get_configuration()->mem_config=
;
> + struct extseg_walk_arg *wa =3D arg;
> +
> + if (msl->base_va =3D=3D wa->va_addr && msl->len =3D=3D wa->len) {
> + unsigned int found_idx;
> +
> + /* msl is const */
> + found_idx =3D msl - mcfg->memsegs;
> + wa->msl =3D &mcfg->memsegs[found_idx];
> + return 1;
> + }
> + return 0;
> +}
> +
> +struct rte_memseg_list *
> +malloc_heap_find_external_seg(void *va_addr, size_t len)
> +{
> + struct extseg_walk_arg wa;
> + int res;
> +
> + wa.va_addr =3D va_addr;
> + wa.len =3D len;
> +
> + res =3D rte_memseg_list_walk_thread_unsafe(extseg_walk, &wa);
> +
> + if (res !=3D 1) {
> + /* 0 means nothing was found, -1 shouldn't happen */
> + if (res =3D=3D 0)
> + rte_errno =3D ENOENT;
> + return NULL;
> + }
> + return wa.msl;
> +}
> +
> +int
> +malloc_heap_destroy_external_seg(struct rte_memseg_list *msl)
> +{
> + /* destroy the fbarray backing this memory */
> + if (rte_fbarray_destroy(&msl->memseg_arr) < 0)
> + return -1;
> +
> + /* reset the memseg list */
> + memset(msl, 0, sizeof(*msl));
> +
> + return 0;
> +}
> +
> int
> malloc_heap_add_external_memory(struct malloc_heap *heap,
> struct rte_memseg_list *msl)
> @@ -1206,7 +1252,7 @@ malloc_heap_remove_external_memory(struct malloc_he=
ap *heap, void *va_addr,
> rte_errno =3D EBUSY;
> return -1;
> }
> - return destroy_seg(elem, len);
> + return destroy_elem(elem, len);
> }
> =20
> int
> diff --git a/lib/librte_eal/common/malloc_heap.h b/lib/librte_eal/common/=
malloc_heap.h
> index 255a315b8..ca9ff666f 100644
> --- a/lib/librte_eal/common/malloc_heap.h
> +++ b/lib/librte_eal/common/malloc_heap.h
> @@ -44,6 +44,12 @@ malloc_heap_create_external_seg(void *va_addr, rte_iov=
a_t iova_addrs[],
> unsigned int n_pages, size_t page_sz, const char *seg_name,
> unsigned int socket_id);
> =20
> +struct rte_memseg_list *
> +malloc_heap_find_external_seg(void *va_addr, size_t len);
> +
> +int
> +malloc_heap_destroy_external_seg(struct rte_memseg_list *msl);
> +
> int
> malloc_heap_add_external_memory(struct malloc_heap *heap,
> struct rte_memseg_list *msl);
> diff --git a/lib/librte_eal/common/rte_malloc.c b/lib/librte_eal/common/r=
te_malloc.c
> index 66bfe63c3..9a82e3386 100644
> --- a/lib/librte_eal/common/rte_malloc.c
> +++ b/lib/librte_eal/common/rte_malloc.c
> @@ -396,6 +396,7 @@ rte_malloc_heap_memory_remove(const char *heap_name, =
void *va_addr, size_t len)
> {
> struct rte_mem_config *mcfg =3D rte_eal_get_configuration()->mem_config=
;
> struct malloc_heap *heap =3D NULL;
> + struct rte_memseg_list *msl;
> int ret;
> =20
> if (heap_name =3D=3D NULL || va_addr =3D=3D NULL || len =3D=3D 0 ||
> @@ -420,9 +421,19 @@ rte_malloc_heap_memory_remove(const char *heap_name,=
void *va_addr, size_t len)
> goto unlock;
> }
> =20
> + msl =3D malloc_heap_find_external_seg(va_addr, len);
> + if (msl =3D=3D NULL) {
> + ret =3D -1;
> + goto unlock;
> + }
> +
> rte_spinlock_lock(&heap->lock);
> ret =3D malloc_heap_remove_external_memory(heap, va_addr, len);
> rte_spinlock_unlock(&heap->lock);
> + if (ret !=3D 0)
> + goto unlock;
> +
> + ret =3D malloc_heap_destroy_external_seg(msl);
> =20
> unlock:
> rte_rwlock_write_unlock(&mcfg->memory_hotplug_lock);
> @@ -430,63 +441,12 @@ rte_malloc_heap_memory_remove(const char *heap_name=
, void *va_addr, size_t len)
> return ret;
> }
> =20
> -struct sync_mem_walk_arg {
> - void *va_addr;
> - size_t len;
> - int result;
> - bool attach;
> -};
> -
> -static int
> -sync_mem_walk(const struct rte_memseg_list *msl, void *arg)
> -{
> - struct rte_mem_config *mcfg =3D rte_eal_get_configuration()->mem_config=
;
> - struct sync_mem_walk_arg *wa =3D arg;
> - size_t len =3D msl->page_sz * msl->memseg_arr.len;
> -
> - if (msl->base_va =3D=3D wa->va_addr &&
> - len =3D=3D wa->len) {
> - struct rte_memseg_list *found_msl;
> - int msl_idx, ret;
> -
> - /* msl is const */
> - msl_idx =3D msl - mcfg->memsegs;
> - found_msl =3D &mcfg->memsegs[msl_idx];
> -
> - if (wa->attach) {
> - ret =3D rte_fbarray_attach(&found_msl->memseg_arr);
> - } else {
> - /* notify all subscribers that a memory area is about to
> - * be removed
> - */
> - eal_memalloc_mem_event_notify(RTE_MEM_EVENT_FREE,
> - msl->base_va, msl->len);
> - ret =3D rte_fbarray_detach(&found_msl->memseg_arr);
> - }
> -
> - if (ret < 0) {
> - wa->result =3D -rte_errno;
> - } else {
> - /* notify all subscribers that a new memory area was
> - * added
> - */
> - if (wa->attach)
> - eal_memalloc_mem_event_notify(
> - RTE_MEM_EVENT_ALLOC,
> - msl->base_va, msl->len);
> - wa->result =3D 0;
> - }
> - return 1;
> - }
> - return 0;
> -}
> -
> static int
> sync_memory(const char *heap_name, void *va_addr, size_t len, bool attac=
h)
> {
> struct rte_mem_config *mcfg =3D rte_eal_get_configuration()->mem_config=
;
> struct malloc_heap *heap =3D NULL;
> - struct sync_mem_walk_arg wa;
> + struct rte_memseg_list *msl;
> int ret;
> =20
> if (heap_name =3D=3D NULL || va_addr =3D=3D NULL || len =3D=3D 0 ||
> @@ -513,23 +473,35 @@ sync_memory(const char *heap_name, void *va_addr, s=
ize_t len, bool attach)
> }
> =20
> /* find corresponding memseg list to sync to */
> - wa.va_addr =3D va_addr;
> - wa.len =3D len;
> - wa.result =3D -ENOENT; /* fail unless explicitly told to succeed */
> - wa.attach =3D attach;
> -
> - /* we're already holding a read lock */
> - rte_memseg_list_walk_thread_unsafe(sync_mem_walk, &wa);
> -
> - if (wa.result < 0) {
> - rte_errno =3D -wa.result;
> + msl =3D malloc_heap_find_external_seg(va_addr, len);
> + if (msl =3D=3D NULL) {
> ret =3D -1;
> - } else {
> - /* notify all subscribers that a new memory area was added */
> - if (attach)
> + goto unlock;
> + }
> +
> + if (attach) {
> + ret =3D rte_fbarray_attach(&msl->memseg_arr);
> + if (ret =3D=3D 0) {
> + /* notify all subscribers that a new memory area was
> + * added.
> + */
> eal_memalloc_mem_event_notify(RTE_MEM_EVENT_ALLOC,
> va_addr, len);
> - ret =3D 0;
> + } else {
> + ret =3D -1;
> + goto unlock;
> + }
> + } else {
> + /* notify all subscribers that a memory area is about to
> + * be removed.
> + */
> + eal_memalloc_mem_event_notify(RTE_MEM_EVENT_FREE,
> + msl->base_va, msl->len);
> + ret =3D rte_fbarray_detach(&msl->memseg_arr);
> + if (ret < 0) {
> + ret =3D -1;
> + goto unlock;
> + }
> }
> unlock:
> rte_rwlock_read_unlock(&mcfg->memory_hotplug_lock);
> --=20
> 2.17.1