From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-eopbgr70057.outbound.protection.outlook.com [40.107.7.57]) by dpdk.org (Postfix) with ESMTP id EEBE51B96B for ; Fri, 14 Dec 2018 10:34:16 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EvoW3BWRxtsrDRTH3Kubb8kGP6ZiHL8QTL5sT/EC9cQ=; b=Oci1DHj/W8+0iBH7zP1V65ykE87MqtE2+wnx+pp1Nu+I4SYtEzXG/r5csAPcG0Muh3l3Zv5fVXKCq3woK6d9Hu+N4UwiR5Q+938tigrF8oOLyAoWX9GJR5F9fS70vce/2pQ8gHsjQCER4vyjfa/bzaUwB9c/rKKR8SBNilh9OtQ= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB4090.eurprd05.prod.outlook.com (52.134.72.159) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1425.22; Fri, 14 Dec 2018 09:34:15 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::dcbc:4578:3018:50f3]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::dcbc:4578:3018:50f3%5]) with mapi id 15.20.1404.026; Fri, 14 Dec 2018 09:34:15 +0000 From: Yongseok Koh To: Anatoly Burakov CC: "dev@dpdk.org" , Shahaf Shuler , Thomas Monjalon , "shreyansh.jain@nxp.com" Thread-Topic: [PATCH 2/4] malloc: separate destroying memseg list and heap data Thread-Index: AQHUh+o9Nu/mwoBtska1pmH26GwjdqV+EH2A Date: Fri, 14 Dec 2018 09:34:15 +0000 Message-ID: <20181214093406.GB12221@mtidpdk.mti.labs.mlnx> References: <54cb382e91119cfbef7c34db4406f8ea39428511.1543495935.git.anatoly.burakov@intel.com> In-Reply-To: <54cb382e91119cfbef7c34db4406f8ea39428511.1543495935.git.anatoly.burakov@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: BYAPR05CA0088.namprd05.prod.outlook.com (2603:10b6:a03:e0::29) To DB3PR0502MB3980.eurprd05.prod.outlook.com (2603:10a6:8:10::27) authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [209.116.155.178] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DB3PR0502MB4090; 6:FzOYLwCUmHgmKOaFj4VRiEMCEZjVF7DYZ8pwcZJyDqlT5AVViZhA2o+AE9+S4KkmZpi5KKmQk0Cf7oEUmj9Mzl/+2eQUAISCagzOoN/O73Q7D0bzL/YZCWBnafy2+i28jctIW+hoHiTXNy0BLPGyGDY7KuzQYghDDw9b480PwI5sd6i579EQ6D93vdskSnEKoN02slyQS9/fd3kmsPUStmx1iRmNCXNEbkxUpT+WUsguifEg/la+6ze9F1kU7/bXHj4GeF2Hf89Z+vuTR8pgFHMhFke7vmwMnbISf06lm3uGrc9R5olCgJQB3PWU0XbvFeKXdVTw/53DwRpYH0ZbeLKs0MiK6pWGCfkstC0lo3VEl75GM+hEHl1i11R5XbdXdWaL97cmfuirHPbtb7azzBJQey+5kX0UW+JoEnJx8XypQLfTQUbpcUs9vF9TzFpuFKFo5Qr/P7N1KyCkK1PENA==; 5:wJ0ffmZESJClhZg/ybJc/VvukCUR1mcaa7KzXtewG0Y6T5k/e3XW3fGgCz6CLog6MFMLMe/3SgC0qZzLqibL58Je69AEShxr1jLGVn0Flm0mGI00JSCvU7IrBDlyxIkEaL8D0dtNIPhOHT00PPSpvYp4ztEYuY+/APw7ogDlsCU=; 7:EifBsqPYT+eF3MKmy9fEBZ5JgTodw2Yp5Tu/pTYvq2L4gtLNJJq/omBnv6H6zi+GC2riSjVx0U0oUE0a1W8RcYd3gHv0bUDFj0ORvLj/7uuGYhWLwti9B3+sN3bDFK6L6hGQmuDI+8wNb+wO7wgygA== x-ms-office365-filtering-correlation-id: 63ee0709-77c1-44a0-a571-08d661a75010 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:DB3PR0502MB4090; x-ms-traffictypediagnostic: DB3PR0502MB4090: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(3230021)(999002)(6040522)(2401047)(5005006)(8121501046)(3231475)(944501520)(52105112)(93006095)(93001095)(10201501046)(3002001)(6055026)(148016)(149066)(150057)(6041310)(20161123558120)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123562045)(201708071742011)(7699051)(76991095); SRVR:DB3PR0502MB4090; BCL:0; PCL:0; RULEID:; SRVR:DB3PR0502MB4090; x-forefront-prvs: 08864C38AC x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(346002)(376002)(396003)(136003)(39860400002)(366004)(189003)(199004)(6436002)(102836004)(105586002)(6506007)(33896004)(386003)(8936002)(81166006)(52116002)(2906002)(7736002)(81156014)(8676002)(76176011)(256004)(575784001)(316002)(99286004)(71200400001)(71190400001)(66066001)(86362001)(54906003)(97736004)(14444005)(5024004)(446003)(5660300001)(476003)(25786009)(68736007)(305945005)(486006)(14454004)(11346002)(53936002)(26005)(229853002)(6916009)(3846002)(6512007)(9686003)(6246003)(106356001)(6116002)(4326008)(33656002)(1076002)(478600001)(186003)(6486002); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB4090; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: gZc/8aLjJC9fCfor7XuWHm8W2rSOYYmAmjuWUXJ+0mrtxPwPNo4FdSFapab3OcUS0Q62HgGiKJe9+A/PTmAyaFO0UdFNhxN8IvMQLEjTZIrrbx7qjEfLKK33/05ctbjSIa0EDx1edY1OG6tzkxYJFInGz/Z24fOOIArOV6Hyc4ziJvoBrk9AU6mUBiBrehyW8oBE3O1CzCKphsorU0Ofb/AoYMYdFWFfCeciW2kAh2vFxQhRpVE2D+g5VctV5t7R6N5HCkYS+Tl5ZogG7Z7cWoEGYde48UaO3iqenGhtx53a/DxFM0m6fuWT9EusyCK8 spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 63ee0709-77c1-44a0-a571-08d661a75010 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Dec 2018 09:34:15.2396 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB4090 Subject: Re: [dpdk-dev] [PATCH 2/4] malloc: separate destroying memseg list and heap data X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Dec 2018 09:34:17 -0000 On Thu, Nov 29, 2018 at 01:48:33PM +0000, Anatoly Burakov wrote: > Currently, destroying external heap chunk and its memseg list is > part of one process. When we will gain the ability to unregister > external memory from DPDK that doesn't have any heap structures > associated with it, we need to be able to find and destroy > memseg lists as well as heap data separately. >=20 > Signed-off-by: Anatoly Burakov > --- Acked-by: Yongseok Koh Thanks > lib/librte_eal/common/malloc_heap.c | 70 +++++++++++++++---- > lib/librte_eal/common/malloc_heap.h | 6 ++ > lib/librte_eal/common/rte_malloc.c | 104 ++++++++++------------------ > 3 files changed, 102 insertions(+), 78 deletions(-) >=20 > diff --git a/lib/librte_eal/common/malloc_heap.c b/lib/librte_eal/common/= malloc_heap.c > index 25693481f..fa0cb0799 100644 > --- a/lib/librte_eal/common/malloc_heap.c > +++ b/lib/librte_eal/common/malloc_heap.c > @@ -1067,12 +1067,9 @@ malloc_heap_dump(struct malloc_heap *heap, FILE *f= ) > } > =20 > static int > -destroy_seg(struct malloc_elem *elem, size_t len) > +destroy_elem(struct malloc_elem *elem, size_t len) > { > struct malloc_heap *heap =3D elem->heap; > - struct rte_memseg_list *msl; > - > - msl =3D elem->msl; > =20 > /* notify all subscribers that a memory area is going to be removed */ > eal_memalloc_mem_event_notify(RTE_MEM_EVENT_FREE, elem, len); > @@ -1085,13 +1082,6 @@ destroy_seg(struct malloc_elem *elem, size_t len) > =20 > memset(elem, 0, sizeof(*elem)); > =20 > - /* destroy the fbarray backing this memory */ > - if (rte_fbarray_destroy(&msl->memseg_arr) < 0) > - return -1; > - > - /* reset the memseg list */ > - memset(msl, 0, sizeof(*msl)); > - > return 0; > } > =20 > @@ -1158,6 +1148,62 @@ malloc_heap_create_external_seg(void *va_addr, rte= _iova_t iova_addrs[], > return msl; > } > =20 > +struct extseg_walk_arg { > + void *va_addr; > + size_t len; > + struct rte_memseg_list *msl; > +}; > + > +static int > +extseg_walk(const struct rte_memseg_list *msl, void *arg) > +{ > + struct rte_mem_config *mcfg =3D rte_eal_get_configuration()->mem_config= ; > + struct extseg_walk_arg *wa =3D arg; > + > + if (msl->base_va =3D=3D wa->va_addr && msl->len =3D=3D wa->len) { > + unsigned int found_idx; > + > + /* msl is const */ > + found_idx =3D msl - mcfg->memsegs; > + wa->msl =3D &mcfg->memsegs[found_idx]; > + return 1; > + } > + return 0; > +} > + > +struct rte_memseg_list * > +malloc_heap_find_external_seg(void *va_addr, size_t len) > +{ > + struct extseg_walk_arg wa; > + int res; > + > + wa.va_addr =3D va_addr; > + wa.len =3D len; > + > + res =3D rte_memseg_list_walk_thread_unsafe(extseg_walk, &wa); > + > + if (res !=3D 1) { > + /* 0 means nothing was found, -1 shouldn't happen */ > + if (res =3D=3D 0) > + rte_errno =3D ENOENT; > + return NULL; > + } > + return wa.msl; > +} > + > +int > +malloc_heap_destroy_external_seg(struct rte_memseg_list *msl) > +{ > + /* destroy the fbarray backing this memory */ > + if (rte_fbarray_destroy(&msl->memseg_arr) < 0) > + return -1; > + > + /* reset the memseg list */ > + memset(msl, 0, sizeof(*msl)); > + > + return 0; > +} > + > int > malloc_heap_add_external_memory(struct malloc_heap *heap, > struct rte_memseg_list *msl) > @@ -1206,7 +1252,7 @@ malloc_heap_remove_external_memory(struct malloc_he= ap *heap, void *va_addr, > rte_errno =3D EBUSY; > return -1; > } > - return destroy_seg(elem, len); > + return destroy_elem(elem, len); > } > =20 > int > diff --git a/lib/librte_eal/common/malloc_heap.h b/lib/librte_eal/common/= malloc_heap.h > index 255a315b8..ca9ff666f 100644 > --- a/lib/librte_eal/common/malloc_heap.h > +++ b/lib/librte_eal/common/malloc_heap.h > @@ -44,6 +44,12 @@ malloc_heap_create_external_seg(void *va_addr, rte_iov= a_t iova_addrs[], > unsigned int n_pages, size_t page_sz, const char *seg_name, > unsigned int socket_id); > =20 > +struct rte_memseg_list * > +malloc_heap_find_external_seg(void *va_addr, size_t len); > + > +int > +malloc_heap_destroy_external_seg(struct rte_memseg_list *msl); > + > int > malloc_heap_add_external_memory(struct malloc_heap *heap, > struct rte_memseg_list *msl); > diff --git a/lib/librte_eal/common/rte_malloc.c b/lib/librte_eal/common/r= te_malloc.c > index 66bfe63c3..9a82e3386 100644 > --- a/lib/librte_eal/common/rte_malloc.c > +++ b/lib/librte_eal/common/rte_malloc.c > @@ -396,6 +396,7 @@ rte_malloc_heap_memory_remove(const char *heap_name, = void *va_addr, size_t len) > { > struct rte_mem_config *mcfg =3D rte_eal_get_configuration()->mem_config= ; > struct malloc_heap *heap =3D NULL; > + struct rte_memseg_list *msl; > int ret; > =20 > if (heap_name =3D=3D NULL || va_addr =3D=3D NULL || len =3D=3D 0 || > @@ -420,9 +421,19 @@ rte_malloc_heap_memory_remove(const char *heap_name,= void *va_addr, size_t len) > goto unlock; > } > =20 > + msl =3D malloc_heap_find_external_seg(va_addr, len); > + if (msl =3D=3D NULL) { > + ret =3D -1; > + goto unlock; > + } > + > rte_spinlock_lock(&heap->lock); > ret =3D malloc_heap_remove_external_memory(heap, va_addr, len); > rte_spinlock_unlock(&heap->lock); > + if (ret !=3D 0) > + goto unlock; > + > + ret =3D malloc_heap_destroy_external_seg(msl); > =20 > unlock: > rte_rwlock_write_unlock(&mcfg->memory_hotplug_lock); > @@ -430,63 +441,12 @@ rte_malloc_heap_memory_remove(const char *heap_name= , void *va_addr, size_t len) > return ret; > } > =20 > -struct sync_mem_walk_arg { > - void *va_addr; > - size_t len; > - int result; > - bool attach; > -}; > - > -static int > -sync_mem_walk(const struct rte_memseg_list *msl, void *arg) > -{ > - struct rte_mem_config *mcfg =3D rte_eal_get_configuration()->mem_config= ; > - struct sync_mem_walk_arg *wa =3D arg; > - size_t len =3D msl->page_sz * msl->memseg_arr.len; > - > - if (msl->base_va =3D=3D wa->va_addr && > - len =3D=3D wa->len) { > - struct rte_memseg_list *found_msl; > - int msl_idx, ret; > - > - /* msl is const */ > - msl_idx =3D msl - mcfg->memsegs; > - found_msl =3D &mcfg->memsegs[msl_idx]; > - > - if (wa->attach) { > - ret =3D rte_fbarray_attach(&found_msl->memseg_arr); > - } else { > - /* notify all subscribers that a memory area is about to > - * be removed > - */ > - eal_memalloc_mem_event_notify(RTE_MEM_EVENT_FREE, > - msl->base_va, msl->len); > - ret =3D rte_fbarray_detach(&found_msl->memseg_arr); > - } > - > - if (ret < 0) { > - wa->result =3D -rte_errno; > - } else { > - /* notify all subscribers that a new memory area was > - * added > - */ > - if (wa->attach) > - eal_memalloc_mem_event_notify( > - RTE_MEM_EVENT_ALLOC, > - msl->base_va, msl->len); > - wa->result =3D 0; > - } > - return 1; > - } > - return 0; > -} > - > static int > sync_memory(const char *heap_name, void *va_addr, size_t len, bool attac= h) > { > struct rte_mem_config *mcfg =3D rte_eal_get_configuration()->mem_config= ; > struct malloc_heap *heap =3D NULL; > - struct sync_mem_walk_arg wa; > + struct rte_memseg_list *msl; > int ret; > =20 > if (heap_name =3D=3D NULL || va_addr =3D=3D NULL || len =3D=3D 0 || > @@ -513,23 +473,35 @@ sync_memory(const char *heap_name, void *va_addr, s= ize_t len, bool attach) > } > =20 > /* find corresponding memseg list to sync to */ > - wa.va_addr =3D va_addr; > - wa.len =3D len; > - wa.result =3D -ENOENT; /* fail unless explicitly told to succeed */ > - wa.attach =3D attach; > - > - /* we're already holding a read lock */ > - rte_memseg_list_walk_thread_unsafe(sync_mem_walk, &wa); > - > - if (wa.result < 0) { > - rte_errno =3D -wa.result; > + msl =3D malloc_heap_find_external_seg(va_addr, len); > + if (msl =3D=3D NULL) { > ret =3D -1; > - } else { > - /* notify all subscribers that a new memory area was added */ > - if (attach) > + goto unlock; > + } > + > + if (attach) { > + ret =3D rte_fbarray_attach(&msl->memseg_arr); > + if (ret =3D=3D 0) { > + /* notify all subscribers that a new memory area was > + * added. > + */ > eal_memalloc_mem_event_notify(RTE_MEM_EVENT_ALLOC, > va_addr, len); > - ret =3D 0; > + } else { > + ret =3D -1; > + goto unlock; > + } > + } else { > + /* notify all subscribers that a memory area is about to > + * be removed. > + */ > + eal_memalloc_mem_event_notify(RTE_MEM_EVENT_FREE, > + msl->base_va, msl->len); > + ret =3D rte_fbarray_detach(&msl->memseg_arr); > + if (ret < 0) { > + ret =3D -1; > + goto unlock; > + } > } > unlock: > rte_rwlock_read_unlock(&mcfg->memory_hotplug_lock); > --=20 > 2.17.1