From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id 9BCD6A00E6 for ; Thu, 21 Mar 2019 13:14:53 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 2687E1B4D2; Thu, 21 Mar 2019 13:14:48 +0100 (CET) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-eopbgr140055.outbound.protection.outlook.com [40.107.14.55]) by dpdk.org (Postfix) with ESMTP id 8AD611B4D0 for ; Thu, 21 Mar 2019 13:14:46 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WspzuQ5119BiYnTL4vSG03J4OcIgkd51daLbbk3xC9U=; b=kSfZkNikfcmmCCri+g0+7epAIz8q9OBrg9wT1S7dqA88rfqIhgy9BC/OJCuqpPwleDO4fEoMocWhTF6bdN9dvQFgRDsJlnn7mv1Ry3GjpVg9QNV0TJZZXhP5Y6+kEyqgxSYUVQC8j1gulciPyGoJ0/PBhppkB+zFaSx6D6HGziA= Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com (52.133.45.150) by AM0PR0502MB3827.eurprd05.prod.outlook.com (52.133.47.141) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1730.15; Thu, 21 Mar 2019 12:14:45 +0000 Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003]) by AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003%2]) with mapi id 15.20.1730.013; Thu, 21 Mar 2019 12:14:45 +0000 From: Shahaf Shuler To: Slava Ovsiienko , "dev@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH 06/14] net/mlx5: add IB shared context alloc/free functions Thread-Index: AQHU373AIdoBP+bg40abQLmKhZWuR6YV520g Date: Thu, 21 Mar 2019 12:14:45 +0000 Message-ID: References: <1551376985-11096-1-git-send-email-viacheslavo@mellanox.com> <1553155888-27498-1-git-send-email-viacheslavo@mellanox.com> <1553155888-27498-7-git-send-email-viacheslavo@mellanox.com> In-Reply-To: <1553155888-27498-7-git-send-email-viacheslavo@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; x-originating-ip: [31.154.10.105] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: fb8996ae-1aa4-49fa-fe33-08d6adf6ce34 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020); SRVR:AM0PR0502MB3827; x-ms-traffictypediagnostic: AM0PR0502MB3827: x-microsoft-antispam-prvs: x-forefront-prvs: 0983EAD6B2 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(396003)(346002)(39860400002)(136003)(376002)(189003)(199004)(81166006)(97736004)(33656002)(68736007)(74316002)(25786009)(7736002)(305945005)(110136005)(316002)(14454004)(2906002)(446003)(2501003)(14444005)(52536014)(11346002)(476003)(71190400001)(256004)(5024004)(86362001)(486006)(99286004)(186003)(5660300002)(66066001)(26005)(7696005)(53936002)(76176011)(102836004)(55016002)(9686003)(6436002)(71200400001)(229853002)(3846002)(8936002)(81156014)(478600001)(6246003)(30864003)(106356001)(6506007)(6116002)(105586002)(309714004); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR0502MB3827; H:AM0PR0502MB3795.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: Oy8v2GhJRd8qGeybIm/HytgFtb/xtqlfqp1v7lRspQCeyd2CwYUgQjQbFYTHp5ZG9LxIFtdFn0/BihOENqSSw83ORbPLeyhMz2KdR7g5mMgF/NQhImgQb7X4Z6ZbulLN9z5iG56PzCQvQpWa0j5/D6b8soemsr7Z+GfZQv6b6WX+aiI5IIws0eR8WAp5+IfNJGlUjKYavZvi4Vn/jYbFP4+71Nam87kmS6590BXq/pCvfuUUtX/tYqJYGx47EqUVUvWdBKeVIF7gdPh1JWXumCTsxARFWHOdlmAbSTQ6ULEaUfSRLuQ1mmXBX/TDlWXrOn+O6KI03cBBdjVBJJKNl53S2KBZrVWdiY9xNP12b9aba88+0ZOO3X0tytlNaivl4GW+qymiaNN2sEwYQsUTwrMBU5LmxBoQZZsuzwhY6s0= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: fb8996ae-1aa4-49fa-fe33-08d6adf6ce34 X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Mar 2019 12:14:45.1772 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR0502MB3827 Subject: Re: [dpdk-dev] [PATCH 06/14] net/mlx5: add IB shared context alloc/free functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190321121445.3h8rJ7v-4RatBXMLgO21Obp8sD3yeRMq59WcDO_nnFU@z> Thursday, March 21, 2019 10:11 AM, Viacheslav Ovsiienko: > Subject: [dpdk-dev] [PATCH 06/14] net/mlx5: add IB shared context > alloc/free functions >=20 > The functions to allocate and free shared IB context for multiport is add= ed. > The IB device context, Protection Domain, device attributes, Infiniband > names are going to be relocated to the shared structure from the device > private one. mlx5_dev_spawn() is updated to create shared context. >=20 > Signed-off-by: Viacheslav Ovsiienko Patch #2 in the series should be part of this commit, since it doesn't make= sense to be a standalone.=20 > --- > drivers/net/mlx5/mlx5.c | 234 > ++++++++++++++++++++++++++++++++++++------------ > 1 file changed, 176 insertions(+), 58 deletions(-) >=20 > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index > 100e9f4..b3060de 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -140,6 +140,150 @@ struct mlx5_dev_spawn_data { > struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */ }; >=20 > +static LIST_HEAD(, mlx5_ibv_shared) mlx5_ibv_list =3D > +LIST_HEAD_INITIALIZER(); > + > +/** > + * Allocate shared IB device context. If there is multiport device the > + * master and representors will share this context, if there is single > + * port dedicated IB device, the context will be used by only given > + * port due to unification. > + * > + * Routine first searches the context for the spesified IB device name, > + * if found the shared context assumed and reference counter is > incremented. > + * If no context found the new one is created and initialized with > +specified > + * IB device context and parameters. > + * > + * @param[in] spawn > + * Pointer to the IB device attributes (name, port, etc). > + * > + * @return > + * Pointer to mlx5_ibv_shared object on success, > + * otherwise NULL and rte_errno is set. > + */ > +static struct mlx5_ibv_shared * > +mlx5_alloc_shared_ibctx(const struct mlx5_dev_spawn_data *spawn) { > + struct mlx5_ibv_shared *sh; > + int err =3D 0; Since you having a global list you need to make the access thread safe.=20 > + > + assert(spawn); > + /* Search for IB context by device name. */ > + LIST_FOREACH(sh, &mlx5_ibv_list, next) { > + if (!strcmp(sh->ibdev_name, spawn->ibv_dev->name)) { > + assert(!sh->secondary); How do you enforce secondary not to call this function?=20 > + sh->refcnt++; > + return sh; > + } > + } > + /* No device found, we have to create new sharted context. */ > + assert(spawn->max_port); > + sh =3D rte_zmalloc("ethdev shared ib context", > + sizeof(struct mlx5_ibv_shared) + > + spawn->max_port * > + sizeof(struct mlx5_ibv_shared_port), > + RTE_CACHE_LINE_SIZE); > + if (!sh) { > + DRV_LOG(ERR, "shared context allocation failure"); > + rte_errno =3D ENOMEM; > + return NULL; > + } > + /* Try to open IB device with DV first, then usual Verbs. */ > + errno =3D 0; > + sh->ctx =3D mlx5_glue->dv_open_device(spawn->ibv_dev); This one is only for primary processes right?=20 > + if (sh->ctx) { > + sh->devx =3D 1; > + DRV_LOG(DEBUG, "DevX is supported"); > + } else { > + sh->ctx =3D mlx5_glue->open_device(spawn->ibv_dev); > + if (!sh->ctx) { > + err =3D errno ? errno : ENODEV; > + goto error; > + } > + DRV_LOG(DEBUG, "DevX is NOT supported"); > + } > + err =3D mlx5_glue->query_device_ex(sh->ctx, NULL, &sh- > >device_attr); > + if (err) { > + DRV_LOG(DEBUG, "ibv_query_device_ex() failed"); > + goto error; > + } > + sh->refcnt =3D 1; > + sh->max_port =3D spawn->max_port; > + strncpy(sh->ibdev_name, sh->ctx->device->name, > + sizeof(sh->ibdev_name)); > + strncpy(sh->ibdev_path, sh->ctx->device->ibdev_path, > + sizeof(sh->ibdev_path)); > + if (rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY) { > + /* > + * For secondary process we just open the IB device > + * and get attributes, there will no be real usage > + * of this structure, the secondary process will > + * use one from prpimary. > + */ > + sh->secondary =3D 1; Secondary process should not open a new device. it should use the primary d= evice + private structure for everything. In fact, secondary process should not call this function nor reference any = shared object.=20 > + return sh; > + } > + sh->pd =3D mlx5_glue->alloc_pd(sh->ctx); > + if (sh->pd =3D=3D NULL) { > + DRV_LOG(ERR, "PD allocation failure"); > + err =3D ENOMEM; > + goto error; > + } > + LIST_INSERT_HEAD(&mlx5_ibv_list, sh, next); > + return sh; > +error: > + assert(sh); > + if (sh->pd) > + claim_zero(mlx5_glue->dealloc_pd(sh->pd)); > + if (sh->ctx) > + claim_zero(mlx5_glue->close_device(sh->ctx)); > + rte_free(sh); > + assert(err > 0); > + rte_errno =3D err; > + return NULL; > +} > + > +/** > + * Free shared IB device context. Decrement counter and if zero free > + * all allocated resources and close handles. > + * > + * @param[in] sh > + * Pointer to mlx5_ibv_shared object to free > + */ > +static void > +mlx5_free_shared_ibctx(struct mlx5_ibv_shared *sh) { #ifndef NDEBUG > + /* Check the object presence in the list. */ > + struct mlx5_ibv_shared *lctx; > + > + LIST_FOREACH(lctx, &mlx5_ibv_list, next) > + if (lctx =3D=3D sh) > + break; > + assert(lctx); > + if (lctx !=3D sh) { > + DRV_LOG(ERR, "Freeing non-existing shared IB context"); > + return; > + } > +#endif > + assert(sh); > + assert(sh->refcnt); > + if (--sh->refcnt) > + return; > + /* Zero reference counter, we should release resources. */ > + if (rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY) { > + assert(sh->secondary); > + assert(sh->ctx); > + assert(!sh->pd); > + } > + LIST_REMOVE(sh, next); Secondary process is not allowed to do all of the below.=20 > + if (sh->pd) > + claim_zero(mlx5_glue->dealloc_pd(sh->pd)); > + if (sh->ctx) > + claim_zero(mlx5_glue->close_device(sh->ctx)); > + rte_free(sh); > +} > + > + > /** > * Prepare shared data between primary and secondary process. > */ > @@ -289,12 +433,10 @@ struct mlx5_dev_spawn_data { > } > mlx5_mprq_free_mp(dev); > mlx5_mr_release(dev); > - if (priv->pd !=3D NULL) { > - assert(priv->ctx !=3D NULL); > - claim_zero(mlx5_glue->dealloc_pd(priv->pd)); > - claim_zero(mlx5_glue->close_device(priv->ctx)); > - } else > - assert(priv->ctx =3D=3D NULL); > + assert(priv->sh); > + if (priv->sh) > + mlx5_free_shared_ibctx(priv->sh); > + priv->sh =3D NULL; > if (priv->rss_conf.rss_key !=3D NULL) > rte_free(priv->rss_conf.rss_key); > if (priv->reta_idx !=3D NULL) > @@ -744,11 +886,8 @@ struct mlx5_dev_spawn_data { > struct mlx5_dev_config config) > { > const struct mlx5_switch_info *switch_info =3D &spawn->info; > - struct ibv_device *ibv_dev =3D spawn->ibv_dev; > - struct ibv_context *ctx =3D NULL; > - struct ibv_device_attr_ex attr; > + struct mlx5_ibv_shared *sh; > struct ibv_port_attr port_attr; > - struct ibv_pd *pd =3D NULL; > struct mlx5dv_context dv_attr =3D { .comp_mask =3D 0 }; > struct rte_eth_dev *eth_dev =3D NULL; > struct mlx5_priv *priv =3D NULL; > @@ -807,18 +946,10 @@ struct mlx5_dev_spawn_data { > } > /* Prepare shared data between primary and secondary process. */ > mlx5_prepare_shared_data(); > - errno =3D 0; > - ctx =3D mlx5_glue->dv_open_device(ibv_dev); > - if (ctx) { > - config.devx =3D 1; > - DRV_LOG(DEBUG, "DEVX is supported"); > - } else { > - ctx =3D mlx5_glue->open_device(ibv_dev); > - if (!ctx) { > - rte_errno =3D errno ? errno : ENODEV; > - return NULL; > - } > - } > + sh =3D mlx5_alloc_shared_ibctx(spawn); > + if (!sh) > + return NULL; > + config.devx =3D sh->devx; > #ifdef HAVE_IBV_MLX5_MOD_SWP > dv_attr.comp_mask |=3D MLX5DV_CONTEXT_MASK_SWP; #endif @@ > -832,7 +963,7 @@ struct mlx5_dev_spawn_data { #ifdef > HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT > dv_attr.comp_mask |=3D MLX5DV_CONTEXT_MASK_STRIDING_RQ; > #endif > - mlx5_glue->dv_query_device(ctx, &dv_attr); > + mlx5_glue->dv_query_device(sh->ctx, &dv_attr); > if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) { > if (dv_attr.flags & > MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) { > DRV_LOG(DEBUG, "enhanced MPW is supported"); > @@ -917,11 +1048,6 @@ struct mlx5_dev_spawn_data { > " old OFED/rdma-core version or firmware configuration"); > #endif > config.mpls_en =3D mpls_en; > - err =3D mlx5_glue->query_device_ex(ctx, NULL, &attr); > - if (err) { > - DEBUG("ibv_query_device_ex() failed"); > - goto error; > - } > DRV_LOG(DEBUG, "naming Ethernet device \"%s\"", name); > if (rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY) { > eth_dev =3D rte_eth_dev_attach_secondary(name); > @@ -957,11 +1083,11 @@ struct mlx5_dev_spawn_data { > */ > eth_dev->rx_pkt_burst =3D > mlx5_select_rx_function(eth_dev); > eth_dev->tx_pkt_burst =3D > mlx5_select_tx_function(eth_dev); > - claim_zero(mlx5_glue->close_device(ctx)); > + mlx5_free_shared_ibctx(sh); > return eth_dev; > } > /* Check port status. */ > - err =3D mlx5_glue->query_port(ctx, spawn->ibv_port, &port_attr); > + err =3D mlx5_glue->query_port(sh->ctx, spawn->ibv_port, &port_attr); > if (err) { > DRV_LOG(ERR, "port query failed: %s", strerror(err)); > goto error; > @@ -975,13 +1101,7 @@ struct mlx5_dev_spawn_data { > DRV_LOG(DEBUG, "port is not active: \"%s\" (%d)", > mlx5_glue->port_state_str(port_attr.state), > port_attr.state); > - /* Allocate protection domain. */ > - pd =3D mlx5_glue->alloc_pd(ctx); > - if (pd =3D=3D NULL) { > - DRV_LOG(ERR, "PD allocation failure"); > - err =3D ENOMEM; > - goto error; > - } > + /* Allocate private eth device data. */ > priv =3D rte_zmalloc("ethdev private structure", > sizeof(*priv), > RTE_CACHE_LINE_SIZE); > @@ -990,13 +1110,11 @@ struct mlx5_dev_spawn_data { > err =3D ENOMEM; > goto error; > } > - priv->ctx =3D ctx; > - strncpy(priv->ibdev_name, priv->ctx->device->name, > - sizeof(priv->ibdev_name)); > - strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, > - sizeof(priv->ibdev_path)); > - priv->device_attr =3D attr; > - priv->pd =3D pd; > + priv->sh =3D sh; > + priv->ctx =3D sh->ctx; > + priv->ibv_port =3D spawn->ibv_port; > + priv->device_attr =3D sh->device_attr; > + priv->pd =3D sh->pd; > priv->mtu =3D ETHER_MTU; > #ifndef RTE_ARCH_64 > /* Initialize UAR access locks for 32bit implementations. */ @@ - > 1051,7 +1169,8 @@ struct mlx5_dev_spawn_data { > strerror(rte_errno)); > goto error; > } > - config.hw_csum =3D !!(attr.device_cap_flags_ex & > IBV_DEVICE_RAW_IP_CSUM); > + config.hw_csum =3D !!(sh->device_attr.device_cap_flags_ex & > + IBV_DEVICE_RAW_IP_CSUM); > DRV_LOG(DEBUG, "checksum offloading is %ssupported", > (config.hw_csum ? "" : "not ")); > #if !defined(HAVE_IBV_DEVICE_COUNTERS_SET_V42) && \ @@ -1065,7 > +1184,7 @@ struct mlx5_dev_spawn_data { > } > #endif > config.ind_table_max_size =3D > - attr.rss_caps.max_rwq_indirection_table_size; > + sh->device_attr.rss_caps.max_rwq_indirection_table_size; > /* > * Remove this check once DPDK supports larger/variable > * indirection tables. > @@ -1074,18 +1193,18 @@ struct mlx5_dev_spawn_data { > config.ind_table_max_size =3D ETH_RSS_RETA_SIZE_512; > DRV_LOG(DEBUG, "maximum Rx indirection table size is %u", > config.ind_table_max_size); > - config.hw_vlan_strip =3D !!(attr.raw_packet_caps & > + config.hw_vlan_strip =3D !!(sh->device_attr.raw_packet_caps & >=20 > IBV_RAW_PACKET_CAP_CVLAN_STRIPPING); > DRV_LOG(DEBUG, "VLAN stripping is %ssupported", > (config.hw_vlan_strip ? "" : "not ")); > - config.hw_fcs_strip =3D !!(attr.raw_packet_caps & > + config.hw_fcs_strip =3D !!(sh->device_attr.raw_packet_caps & > IBV_RAW_PACKET_CAP_SCATTER_FCS); > DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", > (config.hw_fcs_strip ? "" : "not ")); #if > defined(HAVE_IBV_WQ_FLAG_RX_END_PADDING) > - hw_padding =3D !!attr.rx_pad_end_addr_align; > + hw_padding =3D !!sh->device_attr.rx_pad_end_addr_align; > #elif defined(HAVE_IBV_WQ_FLAGS_PCI_WRITE_END_PADDING) > - hw_padding =3D !!(attr.device_cap_flags_ex & > + hw_padding =3D !!(sh->device_attr.device_cap_flags_ex & > IBV_DEVICE_PCI_WRITE_END_PADDING); > #endif > if (config.hw_padding && !hw_padding) { @@ -1094,11 +1213,11 @@ > struct mlx5_dev_spawn_data { > } else if (config.hw_padding) { > DRV_LOG(DEBUG, "Rx end alignment padding is enabled"); > } > - config.tso =3D (attr.tso_caps.max_tso > 0 && > - (attr.tso_caps.supported_qpts & > + config.tso =3D (sh->device_attr.tso_caps.max_tso > 0 && > + (sh->device_attr.tso_caps.supported_qpts & > (1 << IBV_QPT_RAW_PACKET))); > if (config.tso) > - config.tso_max_payload_sz =3D attr.tso_caps.max_tso; > + config.tso_max_payload_sz =3D sh- > >device_attr.tso_caps.max_tso; > /* > * MPW is disabled by default, while the Enhanced MPW is enabled > * by default. > @@ -1257,7 +1376,8 @@ struct mlx5_dev_spawn_data { > .free =3D &mlx5_free_verbs_buf, > .data =3D priv, > }; > - mlx5_glue->dv_set_context_attr(ctx, > MLX5DV_CTX_ATTR_BUF_ALLOCATORS, > + mlx5_glue->dv_set_context_attr(sh->ctx, > + MLX5DV_CTX_ATTR_BUF_ALLOCATORS, > (void *)((uintptr_t)&alctr)); > /* Bring Ethernet device up. */ > DRV_LOG(DEBUG, "port %u forcing Ethernet interface up", @@ - > 1311,15 +1431,13 @@ struct mlx5_dev_spawn_data { > if (eth_dev !=3D NULL) > eth_dev->data->dev_private =3D NULL; > } > - if (pd) > - claim_zero(mlx5_glue->dealloc_pd(pd)); > if (eth_dev !=3D NULL) { > /* mac_addrs must not be freed alone because part of > dev_private */ > eth_dev->data->mac_addrs =3D NULL; > rte_eth_dev_release_port(eth_dev); > } > - if (ctx) > - claim_zero(mlx5_glue->close_device(ctx)); > + if (sh) > + mlx5_free_shared_ibctx(sh); > assert(err > 0); > rte_errno =3D err; > return NULL; > -- > 1.8.3.1