From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-AM5-obe.outbound.protection.outlook.com (mail-eopbgr00080.outbound.protection.outlook.com [40.107.0.80]) by dpdk.org (Postfix) with ESMTP id AB7D658C6; Sun, 21 Jan 2018 07:58:12 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=7IvEyqbiMdbQYmnvczIpf1Zoo13UbUAqL0XgMOiYq+U=; b=CN884nVdkSidS9pgNs131nXKOy9EOhuXpzmpXxChuCaTL56f6V3g5tuZXft3WUtvr2FAbk6NVAa8jPQDIFozMaht1Jrkv/D+AcIWBemhIohGhg7ekZWJBxEorkQbbzWnn1FKz/85/4OVJIQq3UJABRZGERsbTKdMWfFmBWcnsbU= Received: from VI1PR05MB3149.eurprd05.prod.outlook.com (10.170.237.142) by VI1PR05MB3296.eurprd05.prod.outlook.com (10.170.238.29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.407.7; Sun, 21 Jan 2018 06:58:09 +0000 Received: from VI1PR05MB3149.eurprd05.prod.outlook.com ([fe80::789c:3f06:bb88:e29c]) by VI1PR05MB3149.eurprd05.prod.outlook.com ([fe80::789c:3f06:bb88:e29c%13]) with mapi id 15.20.0407.012; Sun, 21 Jan 2018 06:58:09 +0000 From: Shahaf Shuler To: Olivier Matz , "dev@dpdk.org" , Adrien Mazarguil , =?iso-8859-1?Q?N=E9lio_Laranjeiro?= , "Yongseok Koh" CC: "stable@dpdk.org" Thread-Topic: [dpdk-stable] [PATCH v2 2/2] net/mlx5: fix allocation when no memory on device NUMA node Thread-Index: AQHTkUItxN2r59Le7E6l0sJql2VKvqN95o/A Date: Sun, 21 Jan 2018 06:58:09 +0000 Message-ID: References: <20180118130043.31773-1-olivier.matz@6wind.com> <20180119162517.4451-1-olivier.matz@6wind.com> <20180119162517.4451-2-olivier.matz@6wind.com> In-Reply-To: <20180119162517.4451-2-olivier.matz@6wind.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; x-originating-ip: [31.154.10.107] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; VI1PR05MB3296; 7:RGA0dvUJKpQH84G/VqpyObsASSJG9E9f3xLVNPjx1G0Q+aiGOCQl7yQV0oEYlCEaB72s9eTUTj78t8ETuj4XF/TIruAMJCH5plKH7GzopNepxlhMZzeUKQMjaYL8oQof9jcajsbanoSyHwa0PZK7/dzID80CYOqE5yrTQ+gcxcCp/cd4+I4eHVtc3Q8im4qq2CJo97VovhfdCQZC8KEcbTKWzRz5pDyLlGqOCWKZmhjbNBKE2dJU+CqREPqOmqUT x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 613c75d5-626e-4f73-7154-08d5609c54ed x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(3008032)(48565401081)(2017052603307)(7153060)(7193020); SRVR:VI1PR05MB3296; x-ms-traffictypediagnostic: VI1PR05MB3296: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040501)(2401047)(5005006)(8121501046)(93006095)(93001095)(10201501046)(3002001)(3231023)(2400081)(944501161)(6055026)(6041288)(20161123564045)(20161123560045)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(6072148)(201708071742011); SRVR:VI1PR05MB3296; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:VI1PR05MB3296; x-forefront-prvs: 0559FB9674 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(396003)(39380400002)(366004)(346002)(376002)(199004)(189003)(2900100001)(6246003)(7736002)(66066001)(3660700001)(14454004)(33656002)(3846002)(6116002)(6506007)(102836004)(81156014)(4326008)(316002)(74316002)(81166006)(76176011)(2906002)(59450400001)(106356001)(68736007)(478600001)(8936002)(99286004)(105586002)(7696005)(110136005)(305945005)(6436002)(25786009)(97736004)(86362001)(3280700002)(5660300001)(5250100002)(229853002)(2501003)(6636002)(9686003)(53936002)(2950100002)(26005)(55016002); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR05MB3296; H:VI1PR05MB3149.eurprd05.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: JTlUfmdMVCvPePt7k87YNyjnjkjtoU0Urbu7ZjrKwg4bVGDO1TDwSAt2ywBpF2HVhPtkz7510ELRqimOdi3J8Q== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 613c75d5-626e-4f73-7154-08d5609c54ed X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Jan 2018 06:58:09.7086 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR05MB3296 Subject: Re: [dpdk-stable] [PATCH v2 2/2] net/mlx5: fix allocation when no memory on device NUMA node X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jan 2018 06:58:12 -0000 Friday, January 19, 2018 6:25 PM, Olivier Matz: on the same numa node than the device, it is > preferable to fallback on another socket instead of failing. >=20 > Fixes: 1e3a39f72d5d ("net/mlx5: allocate verbs object into shared memory"= ) > Cc: stable@dpdk.org >=20 > Signed-off-by: Olivier Matz > Signed-off-by: Nelio Laranjeiro > --- >=20 > This new version of the patch was provided by Nelio (thanks), I validated= it > on my platform. I just did minimal changes to fix the checkpatch issues i= n the > comments of mlx5.h (/** instead of /*). Per my understanding the below patch is to select the socket on which to cr= eate the Verbs object based on the ethdev configuration rather than the PCI= numa node. While it introduce the infrastructure to do fallback to other socket id, it= is not yet used.=20 I think the commit log should be modified to better explain this patch. >=20 > drivers/net/mlx5/mlx5.c | 14 ++++++++++++-- > drivers/net/mlx5/mlx5.h | 20 ++++++++++++++++++++ > drivers/net/mlx5/mlx5_rxq.c | 4 ++++ > drivers/net/mlx5/mlx5_txq.c | 4 ++++ > 4 files changed, 40 insertions(+), 2 deletions(-) >=20 > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index > 1c95f3520..7a04ccf98 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -139,10 +139,20 @@ mlx5_alloc_verbs_buf(size_t size, void *data) > struct priv *priv =3D data; > void *ret; > size_t alignment =3D sysconf(_SC_PAGESIZE); > + unsigned int socket =3D SOCKET_ID_ANY; >=20 > + if (priv->verbs_alloc_ctx.type =3D=3D > MLX5_VERSB_ALLOC_TYPE_TX_QUEUE) { > + const struct mlx5_txq_ctrl *ctrl =3D priv->verbs_alloc_ctx.obj; > + > + socket =3D ctrl->socket; > + } else if (priv->verbs_alloc_ctx.type =3D=3D > + MLX5_VERSB_ALLOC_TYPE_RX_QUEUE) { > + const struct mlx5_rxq_ctrl *ctrl =3D priv->verbs_alloc_ctx.obj; > + > + socket =3D ctrl->socket; > + } > assert(data !=3D NULL); > - ret =3D rte_malloc_socket(__func__, size, alignment, > - priv->dev->device->numa_node); > + ret =3D rte_malloc_socket(__func__, size, alignment, socket); > DEBUG("Extern alloc size: %lu, align: %lu: %p", size, alignment, ret); > return ret; > } > diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index > e740a4e77..abcae95b8 100644 > --- a/drivers/net/mlx5/mlx5.h > +++ b/drivers/net/mlx5/mlx5.h > @@ -123,6 +123,24 @@ struct mlx5_dev_config { > int inline_max_packet_sz; /* Max packet size for inlining. */ }; >=20 > +/** > + * Type of objet being allocated. > + */ > +enum mlx5_verbs_alloc_type { > + MLX5_VERSB_ALLOC_TYPE_NONE, > + MLX5_VERSB_ALLOC_TYPE_TX_QUEUE, > + MLX5_VERSB_ALLOC_TYPE_RX_QUEUE, > +}; > + > +/** > + * Verbs allocator needs a context to know in the callback which kind > +of > + * resources it is allocating. > + */ > +struct mlx5_verbs_alloc_ctx { > + enum mlx5_verbs_alloc_type type; /* Kind of object being allocated. > */ > + const void *obj; /* Pointer to the DPDK object. */ }; > + > struct priv { > struct rte_eth_dev *dev; /* Ethernet device of master process. */ > struct ibv_context *ctx; /* Verbs context. */ @@ -164,6 +182,8 @@ > struct priv { > int primary_socket; /* Unix socket for primary process. */ > struct rte_intr_handle intr_handle_socket; /* Interrupt handler. */ > struct mlx5_dev_config config; /* Device configuration. */ > + struct mlx5_verbs_alloc_ctx verbs_alloc_ctx; > + /* Context for Verbs allocator. */ > }; >=20 > /** > diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c > index 950472754..a43a67526 100644 > --- a/drivers/net/mlx5/mlx5_rxq.c > +++ b/drivers/net/mlx5/mlx5_rxq.c > @@ -655,6 +655,8 @@ mlx5_priv_rxq_ibv_new(struct priv *priv, uint16_t > idx) >=20 > assert(rxq_data); > assert(!rxq_ctrl->ibv); > + priv->verbs_alloc_ctx.type =3D MLX5_VERSB_ALLOC_TYPE_RX_QUEUE; > + priv->verbs_alloc_ctx.obj =3D rxq_ctrl; > tmpl =3D rte_calloc_socket(__func__, 1, sizeof(*tmpl), 0, > rxq_ctrl->socket); > if (!tmpl) { > @@ -818,6 +820,7 @@ mlx5_priv_rxq_ibv_new(struct priv *priv, uint16_t > idx) > DEBUG("%p: Verbs Rx queue %p: refcnt %d", (void *)priv, > (void *)tmpl, rte_atomic32_read(&tmpl->refcnt)); > LIST_INSERT_HEAD(&priv->rxqsibv, tmpl, next); > + priv->verbs_alloc_ctx.type =3D MLX5_VERSB_ALLOC_TYPE_NONE; > return tmpl; > error: > if (tmpl->wq) > @@ -828,6 +831,7 @@ mlx5_priv_rxq_ibv_new(struct priv *priv, uint16_t > idx) > claim_zero(ibv_destroy_comp_channel(tmpl->channel)); > if (tmpl->mr) > priv_mr_release(priv, tmpl->mr); > + priv->verbs_alloc_ctx.type =3D MLX5_VERSB_ALLOC_TYPE_NONE; > return NULL; > } >=20 > diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c > index 26db15a4f..b43cc9ed0 100644 > --- a/drivers/net/mlx5/mlx5_txq.c > +++ b/drivers/net/mlx5/mlx5_txq.c > @@ -396,6 +396,8 @@ mlx5_priv_txq_ibv_new(struct priv *priv, uint16_t > idx) > int ret =3D 0; >=20 > assert(txq_data); > + priv->verbs_alloc_ctx.type =3D MLX5_VERSB_ALLOC_TYPE_TX_QUEUE; > + priv->verbs_alloc_ctx.obj =3D txq_ctrl; > if (mlx5_getenv_int("MLX5_ENABLE_CQE_COMPRESSION")) { > ERROR("MLX5_ENABLE_CQE_COMPRESSION must never be > set"); > goto error; > @@ -526,12 +528,14 @@ mlx5_priv_txq_ibv_new(struct priv *priv, uint16_t > idx) > DEBUG("%p: Verbs Tx queue %p: refcnt %d", (void *)priv, > (void *)txq_ibv, rte_atomic32_read(&txq_ibv->refcnt)); > LIST_INSERT_HEAD(&priv->txqsibv, txq_ibv, next); > + priv->verbs_alloc_ctx.type =3D MLX5_VERSB_ALLOC_TYPE_NONE; > return txq_ibv; > error: > if (tmpl.cq) > claim_zero(ibv_destroy_cq(tmpl.cq)); > if (tmpl.qp) > claim_zero(ibv_destroy_qp(tmpl.qp)); > + priv->verbs_alloc_ctx.type =3D MLX5_VERSB_ALLOC_TYPE_NONE; > return NULL; > } >=20 > -- > 2.11.0