From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <yskoh@mellanox.com>
Received: from EUR04-VI1-obe.outbound.protection.outlook.com
 (mail-eopbgr80050.outbound.protection.outlook.com [40.107.8.50])
 by dpdk.org (Postfix) with ESMTP id A64E84CE4
 for <dev@dpdk.org>; Sat, 29 Sep 2018 02:09:51 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com;
 s=selector1;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=yuVmuW+LcwddF3caZw9CMPNvc7XvZnHWnQ6qQgmsIKg=;
 b=VBqc2zngC5UvA8V086fd7XXkDo7PAKdxJZYl7cbBg5xDh16KBnUAHIc3RpGoOZHsPSfWzH4meR5YXMUuAFChlyCn9tFRRyEg1bf8QoXflGf10hDntdgU0dlERRHt529P76ovdPIw11Pe63FWWqhdOjSRKVXenEC/7WSgUdT3UgA=
Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by
 DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.1164.22; Sat, 29 Sep 2018 00:09:47 +0000
Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com
 ([fe80::1cb0:661b:ecab:6045]) by DB3PR0502MB3980.eurprd05.prod.outlook.com
 ([fe80::1cb0:661b:ecab:6045%2]) with mapi id 15.20.1164.024; Sat, 29 Sep 2018
 00:09:47 +0000
From: Yongseok Koh <yskoh@mellanox.com>
To: Anatoly Burakov <anatoly.burakov@intel.com>
CC: "dev@dpdk.org" <dev@dpdk.org>, Neil Horman <nhorman@tuxdriver.com>, John
 McNamara <john.mcnamara@intel.com>, Marko Kovacevic
 <marko.kovacevic@intel.com>, Hemant Agrawal <hemant.agrawal@nxp.com>,
 Shreyansh Jain <shreyansh.jain@nxp.com>, Matan Azrad <matan@mellanox.com>,
 Shahaf Shuler <shahafs@mellanox.com>, Maxime Coquelin
 <maxime.coquelin@redhat.com>, Tiwei Bie <tiwei.bie@intel.com>, Zhihong Wang
 <zhihong.wang@intel.com>, Bruce Richardson <bruce.richardson@intel.com>,
 Olivier Matz <olivier.matz@6wind.com>, Andrew Rybchenko
 <arybchenko@solarflare.com>, "laszlo.madarassy@ericsson.com"
 <laszlo.madarassy@ericsson.com>, "laszlo.vadkerti@ericsson.com"
 <laszlo.vadkerti@ericsson.com>, "andras.kovacs@ericsson.com"
 <andras.kovacs@ericsson.com>, "winnie.tian@ericsson.com"
 <winnie.tian@ericsson.com>, "daniel.andrasi@ericsson.com"
 <daniel.andrasi@ericsson.com>, "janos.kobor@ericsson.com"
 <janos.kobor@ericsson.com>, "geza.koblo@ericsson.com"
 <geza.koblo@ericsson.com>, "srinath.mannam@broadcom.com"
 <srinath.mannam@broadcom.com>, "scott.branden@broadcom.com"
 <scott.branden@broadcom.com>, "ajit.khaparde@broadcom.com"
 <ajit.khaparde@broadcom.com>, "keith.wiles@intel.com"
 <keith.wiles@intel.com>, Thomas Monjalon <thomas@monjalon.net>,
 "alejandro.lucero@netronome.com" <alejandro.lucero@netronome.com>
Thread-Topic: [PATCH v6 02/21] mem: allow memseg lists to be marked as external
Thread-Index: AQHUVk+61TOtLurNJkeA066DYqkcxKUGZMIA
Date: Sat, 29 Sep 2018 00:09:46 +0000
Message-ID: <20180929000929.GA91924@minint-98vp2qg>
References: <cover.1537960448.git.anatoly.burakov@intel.com>
 <cover.1538044725.git.anatoly.burakov@intel.com>
 <ed151fe0f8e3644030208942c289c76ed95e4494.1538044725.git.anatoly.burakov@intel.com>
In-Reply-To: <ed151fe0f8e3644030208942c289c76ed95e4494.1538044725.git.anatoly.burakov@intel.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-clientproxiedby: BN6PR11CA0012.namprd11.prod.outlook.com
 (2603:10b6:405:2::22) To DB3PR0502MB3980.eurprd05.prod.outlook.com
 (2603:10a6:8:10::27)
authentication-results: spf=none (sender IP is )
 smtp.mailfrom=yskoh@mellanox.com; 
x-ms-exchange-messagesentrepresentingtype: 1
x-originating-ip: [69.181.245.183]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; DB3PR0502MB3980;
 6:WaePEuxmSEysFj+0NtfLO4m+erhAlYaNhuwsf5wT3cW97mItElbowe02tPUV5eB7pnQYrdne6AuPPBz9ju/tMEt90P39Ne7Es20s2SLSw3sltbz32rKnwpk2Yaj8DLtOd40kQ9U+ERjrQnOODZHitXEJheWcRPWWfaB9ySFYyjHyzByDhZCUOVfgqHagXwAXv2wAlsrLD6wTkhLvguQQghq7m0W3mP66qmdaAhg+ZF6Qm1eCdlGWx8YPa8VIpIUoMuXxT5TCTykxWBwkEcbT8kjyITKJulwHxm1M/zcmPJ9TGpJsL2QWhmu4Q5MAXRuY7EOqJ05gpYzbR3ae7rFFougVOQrEhBcj+dEViLvOnSj6FjvDRxgsph+N0crQUAf2iq2bCkDlZ9D/0iCYCwCobr+4hqfDhU4TQqhSfBnC+YLOMLQb7K6Nu3EOTrrjT02xeUxREv1BK+u1DnhX2FE4cQ==;
 5:2xqAE4quYk5CieaGPK1mKDINlgGKclx6YU3EUBEl9lKAL1vq0eX/dP+fLqXWWVxAHGtV9giNAj11eHqTrWK+JPNd/vaTZd3vcVNAJkdTTNihKWUqhpGY/7XQmPQTVfs41tbFmv8LTGZaF03nEdAqoDNc9tddXzGp9z4BbQMgGs8=;
 7:zWJnJs2Zbt0f49KFmol/1MScddIRyuk6Q9SiW1dtchMDcjTURB2iIfMildnp+4/m3RVz8DzI8DaSyta3R9kvERDTAf4uQ1HL0ZjJGK2jbntWfCprFalg10HwLvfjfC6Z9PZMo2LzwETRO5rwRCTnbZK90nfjvLyLbUbbBESf42KBuYqWaZZDzE3M9f4feeY3uYH6UN/5t7JkCYa6KG0Bu8WP2IzdoPHJ4UvMXl2OwFYmivgxPVvY+LyWiW+RGhSE
x-ms-office365-filtering-correlation-id: a3a0f06a-237b-4b9a-7fb9-08d6259fdd52
x-ms-office365-filtering-ht: Tenant
x-microsoft-antispam: BCL:0; PCL:0;
 RULEID:(7020095)(4652040)(8989299)(4534165)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020);
 SRVR:DB3PR0502MB3980; 
x-ms-traffictypediagnostic: DB3PR0502MB3980:
x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr
x-microsoft-antispam-prvs: <DB3PR0502MB3980362A117996B7A03C36E5C3ED0@DB3PR0502MB3980.eurprd05.prod.outlook.com>
x-exchange-antispam-report-test: UriScan:(228905959029699);
x-ms-exchange-senderadcheck: 1
x-exchange-antispam-report-cfa-test: BCL:0; PCL:0;
 RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(10201501046)(93006095)(93001095)(3231355)(944501410)(52105095)(3002001)(6055026)(149066)(150057)(6041310)(20161123562045)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123560045)(201708071742011)(7699051);
 SRVR:DB3PR0502MB3980; BCL:0; PCL:0; RULEID:; SRVR:DB3PR0502MB3980; 
x-forefront-prvs: 0810818DA0
x-forefront-antispam-report: SFV:NSPM;
 SFS:(10009020)(7916004)(136003)(366004)(376002)(39860400002)(396003)(346002)(189003)(199004)(105586002)(33656002)(2900100001)(478600001)(486006)(76176011)(386003)(34290500001)(106356001)(6506007)(33896004)(3846002)(6116002)(86362001)(68736007)(5250100002)(6916009)(53936002)(1076002)(14454004)(14444005)(26005)(186003)(256004)(7416002)(25786009)(71200400001)(8676002)(81166006)(81156014)(305945005)(66066001)(6246003)(39060400002)(102836004)(8936002)(9686003)(54906003)(6512007)(316002)(97736004)(11346002)(446003)(6436002)(476003)(99286004)(7736002)(2906002)(52116002)(229853002)(71190400001)(5660300001)(33716001)(4326008)(6486002)(4006002);
 DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB3980;
 H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en;
 PTR:InfoNoRecords; MX:1; A:1; 
received-spf: None (protection.outlook.com: mellanox.com does not designate
 permitted sender hosts)
x-microsoft-antispam-message-info: fw0ozhQto0P4nTmwGeXvyBqzQ9tc5KwkaIBFS/HGXFDzjrIiokBGkaZrnf1NatD6YraHVjUcHrmbQWRd0BXn+N+br3oRravc2qQ025M/opW/q0F5m9iUeVDvIQj8k1dtTztDaDRubaEHtpja+c+omAKOEgPLdikXHPpL00QiGlMsNiL+ciQccn018yXtKZOjBAerWJF0zHXiO3Y/UNck73ErXrbsxf7em89ZWsiKrbkJOtM8N0SKYXmwI23EnpLc5HuFA5ev6vq1Fpfl7pB4Bwf/U81SZE30DOXYU0tPko+eAC+QUW0PYzom0XCdE4J8TT+gAG2knXyJjbvaTatXcCoqN1+owcRAL6nkUns6Qos=
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="us-ascii"
Content-ID: <8CF24C7E500B474C822C96A8F09FBFCF@eurprd05.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: Mellanox.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a3a0f06a-237b-4b9a-7fb9-08d6259fdd52
X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Sep 2018 00:09:46.9109 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB3980
Subject: Re: [dpdk-dev] [PATCH v6 02/21] mem: allow memseg lists to be
	marked as external
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Sat, 29 Sep 2018 00:09:52 -0000

On Thu, Sep 27, 2018 at 11:40:59AM +0100, Anatoly Burakov wrote:
> When we allocate and use DPDK memory, we need to be able to
> differentiate between DPDK hugepage segments and segments that
> were made part of DPDK but are externally allocated. Add such
> a property to memseg lists.
>=20
> This breaks the ABI, so bump the EAL library ABI version and
> document the change in release notes. This also breaks a few
> internal assumptions about memory contiguousness, so adjust
> malloc code in a few places.
>=20
> All current calls for memseg walk functions were adjusted to
> ignore external segments where it made sense.
>=20
> Mempools is a special case, because we may be asked to allocate
> a mempool on a specific socket, and we need to ignore all page
> sizes on other heaps or other sockets. Previously, this
> assumption of knowing all page sizes was not a problem, but it
> will be now, so we have to match socket ID with page size when
> calculating minimum page size for a mempool.
>=20
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> Acked-by: Andrew Rybchenko <arybchenko@solarflare.com>
> ---
>=20
> Notes:
>     v3:
>     - Add comment to explain the process of picking up minimum
>       page sizes for mempool
>    =20
>     v2:
>     - Add documentation changes and ABI break
>    =20
>     v1:
>     - Adjust all calls to memseg walk functions to ignore external
>       segments where it made sense to do so
>=20
>  doc/guides/rel_notes/deprecation.rst          | 15 --------
>  doc/guides/rel_notes/release_18_11.rst        | 13 ++++++-
>  drivers/bus/fslmc/fslmc_vfio.c                |  7 ++--
>  drivers/net/mlx4/mlx4_mr.c                    |  3 ++
>  drivers/net/mlx5/mlx5.c                       |  5 ++-
>  drivers/net/mlx5/mlx5_mr.c                    |  3 ++
>  drivers/net/virtio/virtio_user/vhost_kernel.c |  5 ++-
>  lib/librte_eal/bsdapp/eal/Makefile            |  2 +-
>  lib/librte_eal/bsdapp/eal/eal.c               |  3 ++
>  lib/librte_eal/bsdapp/eal/eal_memory.c        |  7 ++--
>  lib/librte_eal/common/eal_common_memory.c     |  3 ++
>  .../common/include/rte_eal_memconfig.h        |  1 +
>  lib/librte_eal/common/include/rte_memory.h    |  9 +++++
>  lib/librte_eal/common/malloc_elem.c           | 10 ++++--
>  lib/librte_eal/common/malloc_heap.c           |  9 +++--
>  lib/librte_eal/common/rte_malloc.c            |  2 +-
>  lib/librte_eal/linuxapp/eal/Makefile          |  2 +-
>  lib/librte_eal/linuxapp/eal/eal.c             | 10 +++++-
>  lib/librte_eal/linuxapp/eal/eal_memalloc.c    |  9 +++++
>  lib/librte_eal/linuxapp/eal/eal_vfio.c        | 17 ++++++---
>  lib/librte_eal/meson.build                    |  2 +-
>  lib/librte_mempool/rte_mempool.c              | 35 ++++++++++++++-----
>  test/test/test_malloc.c                       |  3 ++
>  test/test/test_memzone.c                      |  3 ++
>  24 files changed, 134 insertions(+), 44 deletions(-)
>=20
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/=
deprecation.rst
> index 138335dfb..d2aec64d1 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -11,21 +11,6 @@ API and ABI deprecation notices are to be posted here.
>  Deprecation Notices
>  -------------------
> =20
> -* eal: certain structures will change in EAL on account of upcoming exte=
rnal
> -  memory support. Aside from internal changes leading to an ABI break, t=
he
> -  following externally visible changes will also be implemented:
> -
> -  - ``rte_memseg_list`` will change to include a boolean flag indicating
> -    whether a particular memseg list is externally allocated. This will =
have
> -    implications for any users of memseg-walk-related functions, as they=
 will
> -    now have to skip externally allocated segments in most cases if the =
intent
> -    is to only iterate over internal DPDK memory.
> -  - ``socket_id`` parameter across the entire DPDK will gain additional =
meaning,
> -    as some socket ID's will now be representing externally allocated me=
mory. No
> -    changes will be required for existing code as backwards compatibilit=
y will
> -    be kept, and those who do not use this feature will not see these ex=
tra
> -    socket ID's.
> -
>  * eal: both declaring and identifying devices will be streamlined in v18=
.11.
>    New functions will appear to query a specific port from buses, classes=
 of
>    device and device drivers. Device declaration will be made coherent wi=
th the
> diff --git a/doc/guides/rel_notes/release_18_11.rst b/doc/guides/rel_note=
s/release_18_11.rst
> index bc9b74ec4..5fc71e208 100644
> --- a/doc/guides/rel_notes/release_18_11.rst
> +++ b/doc/guides/rel_notes/release_18_11.rst
> @@ -91,6 +91,13 @@ API Changes
>    flag the MAC can be properly configured in any case. This is particula=
rly
>    important for bonding.
> =20
> +* eal: The following API changes were made in 18.11:
> +
> +  - ``rte_memseg_list`` structure now has an additional flag indicating =
whether
> +    the memseg list is externally allocated. This will have implications=
 for any
> +    users of memseg-walk-related functions, as they will now have to ski=
p
> +    externally allocated segments in most cases if the intent is to only=
 iterate
> +    over internal DPDK memory.
> =20
>  ABI Changes
>  -----------
> @@ -107,6 +114,10 @@ ABI Changes
>     =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D
> =20
> =20
> +* eal: EAL library ABI version was changed due to previously announced w=
ork on
> +       supporting external memory in DPDK. Structure ``rte_memseg_list``=
 now has
> +       a new flag indicating whether the memseg list refers to external =
memory.
> +
>  Removed Items
>  -------------
> =20
> @@ -152,7 +163,7 @@ The libraries prepended with a plus sign were increme=
nted in this version.
>       librte_compressdev.so.1
>       librte_cryptodev.so.5
>       librte_distributor.so.1
> -     librte_eal.so.8
> +   + librte_eal.so.9
>       librte_ethdev.so.10
>       librte_eventdev.so.4
>       librte_flow_classify.so.1
> diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfi=
o.c
> index 4c2cd2a87..2e9244fb7 100644
> --- a/drivers/bus/fslmc/fslmc_vfio.c
> +++ b/drivers/bus/fslmc/fslmc_vfio.c
> @@ -317,12 +317,15 @@ fslmc_unmap_dma(uint64_t vaddr, uint64_t iovaddr __=
rte_unused, size_t len)
>  }
> =20
>  static int
> -fslmc_dmamap_seg(const struct rte_memseg_list *msl __rte_unused,
> -		 const struct rte_memseg *ms, void *arg)
> +fslmc_dmamap_seg(const struct rte_memseg_list *msl, const struct rte_mem=
seg *ms,
> +		void *arg)
>  {
>  	int *n_segs =3D arg;
>  	int ret;
> =20
> +	if (msl->external)
> +		return 0;
> +
>  	ret =3D fslmc_map_dma(ms->addr_64, ms->iova, ms->len);
>  	if (ret)
>  		DPAA2_BUS_ERR("Unable to VFIO map (addr=3D%p, len=3D%zu)",
> diff --git a/drivers/net/mlx4/mlx4_mr.c b/drivers/net/mlx4/mlx4_mr.c
> index d23d3c613..9f5d790b6 100644
> --- a/drivers/net/mlx4/mlx4_mr.c
> +++ b/drivers/net/mlx4/mlx4_mr.c
> @@ -496,6 +496,9 @@ mr_find_contig_memsegs_cb(const struct rte_memseg_lis=
t *msl,
>  {
>  	struct mr_find_contig_memsegs_data *data =3D arg;
> =20
> +	if (msl->external)
> +		return 0;
> +

Because memory free event for external memory is available, current design =
of
mlx4/mlx5 memory mgmt can accommodate the new external memory support. So,
please remove it so that PMD can traverse external memory as well.

>  	if (data->addr < ms->addr_64 || data->addr >=3D ms->addr_64 + len)
>  		return 0;
>  	/* Found, save it and stop walking. */
> diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
> index 30d4e70a7..c90e1d8ce 100644
> --- a/drivers/net/mlx5/mlx5.c
> +++ b/drivers/net/mlx5/mlx5.c
> @@ -568,11 +568,14 @@ static struct rte_pci_driver mlx5_driver;
>  static void *uar_base;
> =20
>  static int
> -find_lower_va_bound(const struct rte_memseg_list *msl __rte_unused,
> +find_lower_va_bound(const struct rte_memseg_list *msl,
>  		const struct rte_memseg *ms, void *arg)
>  {
>  	void **addr =3D arg;
> =20
> +	if (msl->external)
> +		return 0;
> +

This one is fine.
But can you please remove the blank line?
That's a rule by former maintainers. :-)

>  	if (*addr =3D=3D NULL)
>  		*addr =3D ms->addr;
>  	else
> diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
> index 1d1bcb5fe..fd4345f9c 100644
> --- a/drivers/net/mlx5/mlx5_mr.c
> +++ b/drivers/net/mlx5/mlx5_mr.c
> @@ -486,6 +486,9 @@ mr_find_contig_memsegs_cb(const struct rte_memseg_lis=
t *msl,
>  {
>  	struct mr_find_contig_memsegs_data *data =3D arg;
> =20
> +	if (msl->external)
> +		return 0;
> +

Like I mentioned, please remove it.

If those two changes in mlx4/5_mr.c are removed, for the whole patch,

Acked-by: Yongseok Koh <yskoh@mellanox.com>

Thanks
Yongseok