From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-VE1-obe.outbound.protection.outlook.com (mail-eopbgr20087.outbound.protection.outlook.com [40.107.2.87]) by dpdk.org (Postfix) with ESMTP id B9F6A2C28 for ; Tue, 26 Mar 2019 13:10:42 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mg1BtU26cyiEW4NAUqax4IKUprBpcy9RzoaxJ3BzHUw=; b=ZR9BXl8aXT8jv9aXe2R2r8zB6dNtSz0gX0p9ls2xfZaiwX5z8bPHNv8I+spwFMClAcKjlLBTF33znH2wuWOesQ3g2wYf/zu6CAegxyYEKtWyyOhgKAf4I6K3ievNY56wFscSpT1GIfYBm13XmyoKyKgXGd1uz6tRsmHOLgSpzYA= Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com (52.133.45.150) by AM0PR0502MB3939.eurprd05.prod.outlook.com (52.133.41.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1730.18; Tue, 26 Mar 2019 12:10:41 +0000 Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003]) by AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003%2]) with mapi id 15.20.1750.014; Tue, 26 Mar 2019 12:10:41 +0000 From: Shahaf Shuler To: Slava Ovsiienko , "dev@dpdk.org" Thread-Topic: [PATCH v2 05/13] net/mlx5: add IB shared context alloc/free functions Thread-Index: AQHU4yzPEaFRhPQspkSdXaG+m2Kb0KYd0thQ Date: Tue, 26 Mar 2019 12:10:41 +0000 Message-ID: References: <1553155888-27498-1-git-send-email-viacheslavo@mellanox.com> <1553533414-9911-1-git-send-email-viacheslavo@mellanox.com> <1553533414-9911-6-git-send-email-viacheslavo@mellanox.com> In-Reply-To: <1553533414-9911-6-git-send-email-viacheslavo@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; x-originating-ip: [31.154.10.105] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: fbd21a23-9fb2-4575-b657-08d6b1e410c9 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020); SRVR:AM0PR0502MB3939; x-ms-traffictypediagnostic: AM0PR0502MB3939: x-microsoft-antispam-prvs: x-forefront-prvs: 09888BC01D x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(136003)(39860400002)(396003)(346002)(376002)(366004)(199004)(189003)(14454004)(71200400001)(71190400001)(68736007)(9686003)(5660300002)(66066001)(74316002)(33656002)(30864003)(76176011)(81166006)(102836004)(8676002)(186003)(2906002)(7696005)(26005)(81156014)(86362001)(6506007)(478600001)(6116002)(8936002)(25786009)(3846002)(7736002)(316002)(6246003)(105586002)(14444005)(11346002)(446003)(256004)(5024004)(99286004)(6436002)(110136005)(476003)(305945005)(486006)(229853002)(97736004)(53936002)(53946003)(52536014)(55016002)(2501003)(106356001)(309714004); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR0502MB3939; H:AM0PR0502MB3795.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: TyRvASmSJPmi4eyzjuf8FVE4DPOgMi13kCmIpUwMqoMGUb5HMiWBEiG74LTdQNprr9Doh0YusW91Iwgmr6/xaZ9BAXVwxuohBmvksQFAC51zEt2ZIVhcc0ZxkwgKV3mhsFbqBldyxhTD5WMjU4w4rTjW7yM/g+jDk9Od85RR8kFvppBOMNGYEKYpN766KOZ03vQMgbE6DIc24tF/pIcyg+d9KI/SeVlZf/+EIs+8FJAkN/5Iz2LCodxMWxTeIjTIaRIvKv2fFszpEWyAHOOhtnYxKAHk7neyv25OIIVvf2QxMz4RNK7N6rukezCJONmBpTDMhCvC9PoKgFg7XvCSBiBLxilXZH99nbiWoDdzV8MltxTtfi50p6WGxHbC5+WaJn6czUhI96Yoy7dnMvEIIABJVZFci7aU7DCNrHMHSiY= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: fbd21a23-9fb2-4575-b657-08d6b1e410c9 X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Mar 2019 12:10:41.1028 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR0502MB3939 Subject: Re: [dpdk-dev] [PATCH v2 05/13] net/mlx5: add IB shared context alloc/free functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Mar 2019 12:10:43 -0000 Monday, March 25, 2019 7:03 PM, Viacheslav Ovsiienko: > Subject: [PATCH v2 05/13] net/mlx5: add IB shared context alloc/free > functions >=20 > The Mellanox NICs support SR-IOV and have E-Switch feature. > When SR-IOV is set up in switchdev mode and E-Switch is enabled we have > so called VF representors in the system. All representors belonging to th= e > same E-Switch are created on the basis of the single PCI function and wit= h > current implementation each representor has its own dedicated Infiniband > device and operates within its own Infiniband context. It is proposed to > provide representors as ports of the single Infiniband device and operate= on > the shared Infiniband context saving various resources. This patch introd= uces > appropriate structures. >=20 > Alsa the functions to allocate and free shared IB context for multiport a= re > added. The IB device context, Protection Domain, device attributes, > Infiniband names are going to be relocated to the shared structure from t= he > device private one. > mlx5_dev_spawn() is updated to support shared context. >=20 > Signed-off-by: Viacheslav Ovsiienko > --- > drivers/net/mlx5/mlx5.c | 298 +++++++++++++++++++++++++++++++++--- > ------------ > drivers/net/mlx5/mlx5.h | 32 ++++++ > 2 files changed, 235 insertions(+), 95 deletions(-) >=20 > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index > 4338254..4c9621a 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -140,6 +140,141 @@ struct mlx5_dev_spawn_data { > struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */ }; >=20 > +static LIST_HEAD(, mlx5_ibv_shared) mlx5_ibv_list =3D > +LIST_HEAD_INITIALIZER(); > + > +/** > + * Allocate shared IB device context. If there is multiport device the > + * master and representors will share this context, if there is single > + * port dedicated IB device, the context will be used by only given > + * port due to unification. > + * > + * Routine first searches the context for the spesified IB device name, > + * if found the shared context assumed and reference counter is > incremented. > + * If no context found the new one is created and initialized with > +specified > + * IB device context and parameters. > + * > + * @param[in] spawn > + * Pointer to the IB device attributes (name, port, etc). > + * > + * @return > + * Pointer to mlx5_ibv_shared object on success, > + * otherwise NULL and rte_errno is set. > + */ > +static struct mlx5_ibv_shared * > +mlx5_alloc_shared_ibctx(const struct mlx5_dev_spawn_data *spawn) { Sorry Slava, I mislead you. Even though it is not expected for probe to be called by multiple thread it= is safer to protect this list as it is global resources. It is not on the critical datapath, so no big concerns w/ mutex.=20 Please add proper synchronization when manipulating this shared contexts.=20 > + struct mlx5_ibv_shared *sh; > + int err =3D 0; > + > + assert(spawn); > + /* Secondary process should not create the shared context. */ > + assert(rte_eal_process_type() =3D=3D RTE_PROC_PRIMARY); > + /* Search for IB context by device name. */ > + LIST_FOREACH(sh, &mlx5_ibv_list, next) { > + if (!strcmp(sh->ibdev_name, spawn->ibv_dev->name)) { > + sh->refcnt++; > + return sh; > + } > + } > + /* No device found, we have to create new sharted context. */ > + assert(spawn->max_port); > + sh =3D rte_zmalloc("ethdev shared ib context", > + sizeof(struct mlx5_ibv_shared) + > + spawn->max_port * > + sizeof(struct mlx5_ibv_shared_port), > + RTE_CACHE_LINE_SIZE); > + if (!sh) { > + DRV_LOG(ERR, "shared context allocation failure"); > + rte_errno =3D ENOMEM; > + return NULL; > + } > + /* Try to open IB device with DV first, then usual Verbs. */ > + errno =3D 0; > + sh->ctx =3D mlx5_glue->dv_open_device(spawn->ibv_dev); > + if (sh->ctx) { > + sh->devx =3D 1; > + DRV_LOG(DEBUG, "DevX is supported"); > + } else { > + sh->ctx =3D mlx5_glue->open_device(spawn->ibv_dev); > + if (!sh->ctx) { > + err =3D errno ? errno : ENODEV; > + goto error; > + } > + DRV_LOG(DEBUG, "DevX is NOT supported"); > + } > + err =3D mlx5_glue->query_device_ex(sh->ctx, NULL, &sh- > >device_attr); > + if (err) { > + DRV_LOG(DEBUG, "ibv_query_device_ex() failed"); > + goto error; > + } > + sh->refcnt =3D 1; > + sh->max_port =3D spawn->max_port; > + strncpy(sh->ibdev_name, sh->ctx->device->name, > + sizeof(sh->ibdev_name)); > + strncpy(sh->ibdev_path, sh->ctx->device->ibdev_path, > + sizeof(sh->ibdev_path)); > + sh->pd =3D mlx5_glue->alloc_pd(sh->ctx); > + if (sh->pd =3D=3D NULL) { > + DRV_LOG(ERR, "PD allocation failure"); > + err =3D ENOMEM; > + goto error; > + } > + LIST_INSERT_HEAD(&mlx5_ibv_list, sh, next); > + return sh; > +error: > + assert(sh); > + if (sh->pd) > + claim_zero(mlx5_glue->dealloc_pd(sh->pd)); > + if (sh->ctx) > + claim_zero(mlx5_glue->close_device(sh->ctx)); > + rte_free(sh); > + assert(err > 0); > + rte_errno =3D err; > + return NULL; > +} > + > +/** > + * Free shared IB device context. Decrement counter and if zero free > + * all allocated resources and close handles. > + * > + * @param[in] sh > + * Pointer to mlx5_ibv_shared object to free > + */ > +static void > +mlx5_free_shared_ibctx(struct mlx5_ibv_shared *sh) { #ifndef NDEBUG > + /* Check the object presence in the list. */ > + struct mlx5_ibv_shared *lctx; > + > + LIST_FOREACH(lctx, &mlx5_ibv_list, next) > + if (lctx =3D=3D sh) > + break; > + assert(lctx); > + if (lctx !=3D sh) { > + DRV_LOG(ERR, "Freeing non-existing shared IB context"); > + return; > + } > +#endif > + assert(sh); > + assert(sh->refcnt); > + if (--sh->refcnt) > + return; > + /* Zero reference counter, we should release resources. */ > + if (rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY) { > + assert(sh->secondary); > + assert(sh->ctx); > + assert(!sh->pd); > + } > + LIST_REMOVE(sh, next); > + if (sh->pd) > + claim_zero(mlx5_glue->dealloc_pd(sh->pd)); > + if (sh->ctx) > + claim_zero(mlx5_glue->close_device(sh->ctx)); > + rte_free(sh); > +} > + > + > /** > * Prepare shared data between primary and secondary process. > */ > @@ -289,12 +424,10 @@ struct mlx5_dev_spawn_data { > } > mlx5_mprq_free_mp(dev); > mlx5_mr_release(dev); > - if (priv->pd !=3D NULL) { > - assert(priv->ctx !=3D NULL); > - claim_zero(mlx5_glue->dealloc_pd(priv->pd)); > - claim_zero(mlx5_glue->close_device(priv->ctx)); > - } else > - assert(priv->ctx =3D=3D NULL); > + assert(priv->sh); > + if (priv->sh) > + mlx5_free_shared_ibctx(priv->sh); > + priv->sh =3D NULL; > if (priv->rss_conf.rss_key !=3D NULL) > rte_free(priv->rss_conf.rss_key); > if (priv->reta_idx !=3D NULL) > @@ -744,11 +877,8 @@ struct mlx5_dev_spawn_data { > struct mlx5_dev_config config) > { > const struct mlx5_switch_info *switch_info =3D &spawn->info; > - struct ibv_device *ibv_dev =3D spawn->ibv_dev; > - struct ibv_context *ctx =3D NULL; > - struct ibv_device_attr_ex attr; > + struct mlx5_ibv_shared *sh; > struct ibv_port_attr port_attr; > - struct ibv_pd *pd =3D NULL; > struct mlx5dv_context dv_attr =3D { .comp_mask =3D 0 }; > struct rte_eth_dev *eth_dev =3D NULL; > struct mlx5_priv *priv =3D NULL; > @@ -807,18 +937,47 @@ struct mlx5_dev_spawn_data { > } > /* Prepare shared data between primary and secondary process. */ > mlx5_prepare_shared_data(); > - errno =3D 0; > - ctx =3D mlx5_glue->dv_open_device(ibv_dev); > - if (ctx) { > - config.devx =3D 1; > - DRV_LOG(DEBUG, "DEVX is supported"); > - } else { > - ctx =3D mlx5_glue->open_device(ibv_dev); > - if (!ctx) { > - rte_errno =3D errno ? errno : ENODEV; > - return NULL; > + DRV_LOG(DEBUG, "naming Ethernet device \"%s\"", name); > + if (rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY) { > + eth_dev =3D rte_eth_dev_attach_secondary(name); > + if (eth_dev =3D=3D NULL) { > + DRV_LOG(ERR, "can not attach rte ethdev"); > + rte_errno =3D ENOMEM; > + err =3D rte_errno; > + goto error; > + } > + eth_dev->device =3D dpdk_dev; > + eth_dev->dev_ops =3D &mlx5_dev_sec_ops; > + err =3D mlx5_uar_init_secondary(eth_dev); > + if (err) { > + err =3D rte_errno; > + goto error; > } > + /* Receive command fd from primary process */ > + err =3D mlx5_socket_connect(eth_dev); > + if (err < 0) { > + err =3D rte_errno; > + goto error; > + } > + /* Remap UAR for Tx queues. */ > + err =3D mlx5_tx_uar_remap(eth_dev, err); > + if (err) { > + err =3D rte_errno; > + goto error; > + } > + /* > + * Ethdev pointer is still required as input since > + * the primary device is not accessible from the > + * secondary process. > + */ > + eth_dev->rx_pkt_burst =3D > mlx5_select_rx_function(eth_dev); > + eth_dev->tx_pkt_burst =3D > mlx5_select_tx_function(eth_dev); > + return eth_dev; > } > + sh =3D mlx5_alloc_shared_ibctx(spawn); > + if (!sh) > + return NULL; > + config.devx =3D sh->devx; > #ifdef HAVE_IBV_MLX5_MOD_SWP > dv_attr.comp_mask |=3D MLX5DV_CONTEXT_MASK_SWP; #endif @@ > -832,7 +991,7 @@ struct mlx5_dev_spawn_data { #ifdef > HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT > dv_attr.comp_mask |=3D MLX5DV_CONTEXT_MASK_STRIDING_RQ; > #endif > - mlx5_glue->dv_query_device(ctx, &dv_attr); > + mlx5_glue->dv_query_device(sh->ctx, &dv_attr); > if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) { > if (dv_attr.flags & > MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) { > DRV_LOG(DEBUG, "enhanced MPW is supported"); > @@ -917,51 +1076,8 @@ struct mlx5_dev_spawn_data { > " old OFED/rdma-core version or firmware configuration"); > #endif > config.mpls_en =3D mpls_en; > - err =3D mlx5_glue->query_device_ex(ctx, NULL, &attr); > - if (err) { > - DEBUG("ibv_query_device_ex() failed"); > - goto error; > - } > - DRV_LOG(DEBUG, "naming Ethernet device \"%s\"", name); > - if (rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY) { > - eth_dev =3D rte_eth_dev_attach_secondary(name); > - if (eth_dev =3D=3D NULL) { > - DRV_LOG(ERR, "can not attach rte ethdev"); > - rte_errno =3D ENOMEM; > - err =3D rte_errno; > - goto error; > - } > - eth_dev->device =3D dpdk_dev; > - eth_dev->dev_ops =3D &mlx5_dev_sec_ops; > - err =3D mlx5_uar_init_secondary(eth_dev); > - if (err) { > - err =3D rte_errno; > - goto error; > - } > - /* Receive command fd from primary process */ > - err =3D mlx5_socket_connect(eth_dev); > - if (err < 0) { > - err =3D rte_errno; > - goto error; > - } > - /* Remap UAR for Tx queues. */ > - err =3D mlx5_tx_uar_remap(eth_dev, err); > - if (err) { > - err =3D rte_errno; > - goto error; > - } > - /* > - * Ethdev pointer is still required as input since > - * the primary device is not accessible from the > - * secondary process. > - */ > - eth_dev->rx_pkt_burst =3D > mlx5_select_rx_function(eth_dev); > - eth_dev->tx_pkt_burst =3D > mlx5_select_tx_function(eth_dev); > - claim_zero(mlx5_glue->close_device(ctx)); > - return eth_dev; > - } > /* Check port status. */ > - err =3D mlx5_glue->query_port(ctx, spawn->ibv_port, &port_attr); > + err =3D mlx5_glue->query_port(sh->ctx, spawn->ibv_port, &port_attr); > if (err) { > DRV_LOG(ERR, "port query failed: %s", strerror(err)); > goto error; > @@ -975,13 +1091,7 @@ struct mlx5_dev_spawn_data { > DRV_LOG(DEBUG, "port is not active: \"%s\" (%d)", > mlx5_glue->port_state_str(port_attr.state), > port_attr.state); > - /* Allocate protection domain. */ > - pd =3D mlx5_glue->alloc_pd(ctx); > - if (pd =3D=3D NULL) { > - DRV_LOG(ERR, "PD allocation failure"); > - err =3D ENOMEM; > - goto error; > - } > + /* Allocate private eth device data. */ > priv =3D rte_zmalloc("ethdev private structure", > sizeof(*priv), > RTE_CACHE_LINE_SIZE); > @@ -990,13 +1100,11 @@ struct mlx5_dev_spawn_data { > err =3D ENOMEM; > goto error; > } > - priv->ctx =3D ctx; > - strncpy(priv->ibdev_name, priv->ctx->device->name, > - sizeof(priv->ibdev_name)); > - strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, > - sizeof(priv->ibdev_path)); > - priv->device_attr =3D attr; > - priv->pd =3D pd; > + priv->sh =3D sh; > + priv->ctx =3D sh->ctx; > + priv->ibv_port =3D spawn->ibv_port; > + priv->device_attr =3D sh->device_attr; > + priv->pd =3D sh->pd; > priv->mtu =3D ETHER_MTU; > #ifndef RTE_ARCH_64 > /* Initialize UAR access locks for 32bit implementations. */ @@ - > 1062,7 +1170,8 @@ struct mlx5_dev_spawn_data { > strerror(rte_errno)); > goto error; > } > - config.hw_csum =3D !!(attr.device_cap_flags_ex & > IBV_DEVICE_RAW_IP_CSUM); > + config.hw_csum =3D !!(sh->device_attr.device_cap_flags_ex & > + IBV_DEVICE_RAW_IP_CSUM); > DRV_LOG(DEBUG, "checksum offloading is %ssupported", > (config.hw_csum ? "" : "not ")); > #if !defined(HAVE_IBV_DEVICE_COUNTERS_SET_V42) && \ @@ -1076,7 > +1185,7 @@ struct mlx5_dev_spawn_data { > } > #endif > config.ind_table_max_size =3D > - attr.rss_caps.max_rwq_indirection_table_size; > + sh->device_attr.rss_caps.max_rwq_indirection_table_size; > /* > * Remove this check once DPDK supports larger/variable > * indirection tables. > @@ -1085,18 +1194,18 @@ struct mlx5_dev_spawn_data { > config.ind_table_max_size =3D ETH_RSS_RETA_SIZE_512; > DRV_LOG(DEBUG, "maximum Rx indirection table size is %u", > config.ind_table_max_size); > - config.hw_vlan_strip =3D !!(attr.raw_packet_caps & > + config.hw_vlan_strip =3D !!(sh->device_attr.raw_packet_caps & >=20 > IBV_RAW_PACKET_CAP_CVLAN_STRIPPING); > DRV_LOG(DEBUG, "VLAN stripping is %ssupported", > (config.hw_vlan_strip ? "" : "not ")); > - config.hw_fcs_strip =3D !!(attr.raw_packet_caps & > + config.hw_fcs_strip =3D !!(sh->device_attr.raw_packet_caps & > IBV_RAW_PACKET_CAP_SCATTER_FCS); > DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", > (config.hw_fcs_strip ? "" : "not ")); #if > defined(HAVE_IBV_WQ_FLAG_RX_END_PADDING) > - hw_padding =3D !!attr.rx_pad_end_addr_align; > + hw_padding =3D !!sh->device_attr.rx_pad_end_addr_align; > #elif defined(HAVE_IBV_WQ_FLAGS_PCI_WRITE_END_PADDING) > - hw_padding =3D !!(attr.device_cap_flags_ex & > + hw_padding =3D !!(sh->device_attr.device_cap_flags_ex & > IBV_DEVICE_PCI_WRITE_END_PADDING); > #endif > if (config.hw_padding && !hw_padding) { @@ -1105,11 +1214,11 @@ > struct mlx5_dev_spawn_data { > } else if (config.hw_padding) { > DRV_LOG(DEBUG, "Rx end alignment padding is enabled"); > } > - config.tso =3D (attr.tso_caps.max_tso > 0 && > - (attr.tso_caps.supported_qpts & > + config.tso =3D (sh->device_attr.tso_caps.max_tso > 0 && > + (sh->device_attr.tso_caps.supported_qpts & > (1 << IBV_QPT_RAW_PACKET))); > if (config.tso) > - config.tso_max_payload_sz =3D attr.tso_caps.max_tso; > + config.tso_max_payload_sz =3D sh- > >device_attr.tso_caps.max_tso; > /* > * MPW is disabled by default, while the Enhanced MPW is enabled > * by default. > @@ -1243,7 +1352,7 @@ struct mlx5_dev_spawn_data { > } > } > #ifdef HAVE_MLX5DV_DR > - ns =3D mlx5dv_dr_create_ns(ctx, > + ns =3D mlx5dv_dr_create_ns(sh->ctx, >=20 > MLX5DV_DR_NS_DOMAIN_INGRESS_BYPASS); > if (ns =3D=3D NULL) { > DRV_LOG(ERR, "mlx5dv_dr_create_ns failed"); @@ - > 1251,7 +1360,7 @@ struct mlx5_dev_spawn_data { > goto error; > } > priv->rx_ns =3D ns; > - ns =3D mlx5dv_dr_create_ns(ctx, > + ns =3D mlx5dv_dr_create_ns(sh->ctx, >=20 > MLX5DV_DR_NS_DOMAIN_EGRESS_BYPASS); > if (ns =3D=3D NULL) { > DRV_LOG(ERR, "mlx5dv_dr_create_ns failed"); @@ - > 1268,7 +1377,8 @@ struct mlx5_dev_spawn_data { > .free =3D &mlx5_free_verbs_buf, > .data =3D priv, > }; > - mlx5_glue->dv_set_context_attr(ctx, > MLX5DV_CTX_ATTR_BUF_ALLOCATORS, > + mlx5_glue->dv_set_context_attr(sh->ctx, > + MLX5DV_CTX_ATTR_BUF_ALLOCATORS, > (void *)((uintptr_t)&alctr)); > /* Bring Ethernet device up. */ > DRV_LOG(DEBUG, "port %u forcing Ethernet interface up", @@ - > 1322,15 +1432,13 @@ struct mlx5_dev_spawn_data { > if (eth_dev !=3D NULL) > eth_dev->data->dev_private =3D NULL; > } > - if (pd) > - claim_zero(mlx5_glue->dealloc_pd(pd)); > if (eth_dev !=3D NULL) { > /* mac_addrs must not be freed alone because part of > dev_private */ > eth_dev->data->mac_addrs =3D NULL; > rte_eth_dev_release_port(eth_dev); > } > - if (ctx) > - claim_zero(mlx5_glue->close_device(ctx)); > + if (sh) > + mlx5_free_shared_ibctx(sh); > assert(err > 0); > rte_errno =3D err; > return NULL; > diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index > d6967a3..4568de7 100644 > --- a/drivers/net/mlx5/mlx5.h > +++ b/drivers/net/mlx5/mlx5.h > @@ -196,10 +196,42 @@ struct mlx5_flow_tbl_resource { #define > MLX5_MAX_TABLES 1024 #define MLX5_GROUP_FACTOR 1 >=20 > +/* Per port data of shared IB device. */ struct mlx5_ibv_shared_port { > + uint32_t ih_port_id; > + /* > + * Interrupt handler port_id. Used by shared interrupt > + * handler to find the corresponding rte_eth device > + * by IB port index. If value is equal or greater > + * RTE_MAX_ETHPORTS it means there is no subhandler > + * installed for specified IB port index. > + */ > +}; > + > +/* > + * Shared Infiniband device context for Master/Representors > + * which belong to same IB device with multiple IB ports. > + **/ > +struct mlx5_ibv_shared { > + LIST_ENTRY(mlx5_ibv_shared) next; > + uint32_t refcnt; > + uint32_t devx:1; /* Opened with DV. */ > + uint32_t max_port; /* Maximal IB device port index. */ > + struct ibv_context *ctx; /* Verbs/DV context. */ > + struct ibv_pd *pd; /* Protection Domain. */ > + char ibdev_name[IBV_SYSFS_NAME_MAX]; /* IB device name. */ > + char ibdev_path[IBV_SYSFS_PATH_MAX]; /* IB device path for > secondary */ > + struct ibv_device_attr_ex device_attr; /* Device properties. */ > + struct rte_intr_handle intr_handle; /* Interrupt handler for device. > */ > + struct mlx5_ibv_shared_port port[]; /* per device port data array. */ > +}; > + > struct mlx5_priv { > LIST_ENTRY(mlx5_priv) mem_event_cb; > /**< Called by memory event callback. */ > struct rte_eth_dev_data *dev_data; /* Pointer to device data. */ > + struct mlx5_ibv_shared *sh; /* Shared IB device context. */ > + uint32_t ibv_port; /* IB device port number. */ > struct ibv_context *ctx; /* Verbs context. */ > struct ibv_device_attr_ex device_attr; /* Device properties. */ > struct ibv_pd *pd; /* Protection Domain. */ > -- > 1.8.3.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id B8A89A05D3 for ; Tue, 26 Mar 2019 13:10:45 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 944092C60; Tue, 26 Mar 2019 13:10:44 +0100 (CET) Received: from EUR02-VE1-obe.outbound.protection.outlook.com (mail-eopbgr20087.outbound.protection.outlook.com [40.107.2.87]) by dpdk.org (Postfix) with ESMTP id B9F6A2C28 for ; Tue, 26 Mar 2019 13:10:42 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mg1BtU26cyiEW4NAUqax4IKUprBpcy9RzoaxJ3BzHUw=; b=ZR9BXl8aXT8jv9aXe2R2r8zB6dNtSz0gX0p9ls2xfZaiwX5z8bPHNv8I+spwFMClAcKjlLBTF33znH2wuWOesQ3g2wYf/zu6CAegxyYEKtWyyOhgKAf4I6K3ievNY56wFscSpT1GIfYBm13XmyoKyKgXGd1uz6tRsmHOLgSpzYA= Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com (52.133.45.150) by AM0PR0502MB3939.eurprd05.prod.outlook.com (52.133.41.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1730.18; Tue, 26 Mar 2019 12:10:41 +0000 Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003]) by AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003%2]) with mapi id 15.20.1750.014; Tue, 26 Mar 2019 12:10:41 +0000 From: Shahaf Shuler To: Slava Ovsiienko , "dev@dpdk.org" Thread-Topic: [PATCH v2 05/13] net/mlx5: add IB shared context alloc/free functions Thread-Index: AQHU4yzPEaFRhPQspkSdXaG+m2Kb0KYd0thQ Date: Tue, 26 Mar 2019 12:10:41 +0000 Message-ID: References: <1553155888-27498-1-git-send-email-viacheslavo@mellanox.com> <1553533414-9911-1-git-send-email-viacheslavo@mellanox.com> <1553533414-9911-6-git-send-email-viacheslavo@mellanox.com> In-Reply-To: <1553533414-9911-6-git-send-email-viacheslavo@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; x-originating-ip: [31.154.10.105] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: fbd21a23-9fb2-4575-b657-08d6b1e410c9 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020); SRVR:AM0PR0502MB3939; x-ms-traffictypediagnostic: AM0PR0502MB3939: x-microsoft-antispam-prvs: x-forefront-prvs: 09888BC01D x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(136003)(39860400002)(396003)(346002)(376002)(366004)(199004)(189003)(14454004)(71200400001)(71190400001)(68736007)(9686003)(5660300002)(66066001)(74316002)(33656002)(30864003)(76176011)(81166006)(102836004)(8676002)(186003)(2906002)(7696005)(26005)(81156014)(86362001)(6506007)(478600001)(6116002)(8936002)(25786009)(3846002)(7736002)(316002)(6246003)(105586002)(14444005)(11346002)(446003)(256004)(5024004)(99286004)(6436002)(110136005)(476003)(305945005)(486006)(229853002)(97736004)(53936002)(53946003)(52536014)(55016002)(2501003)(106356001)(309714004); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR0502MB3939; H:AM0PR0502MB3795.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: TyRvASmSJPmi4eyzjuf8FVE4DPOgMi13kCmIpUwMqoMGUb5HMiWBEiG74LTdQNprr9Doh0YusW91Iwgmr6/xaZ9BAXVwxuohBmvksQFAC51zEt2ZIVhcc0ZxkwgKV3mhsFbqBldyxhTD5WMjU4w4rTjW7yM/g+jDk9Od85RR8kFvppBOMNGYEKYpN766KOZ03vQMgbE6DIc24tF/pIcyg+d9KI/SeVlZf/+EIs+8FJAkN/5Iz2LCodxMWxTeIjTIaRIvKv2fFszpEWyAHOOhtnYxKAHk7neyv25OIIVvf2QxMz4RNK7N6rukezCJONmBpTDMhCvC9PoKgFg7XvCSBiBLxilXZH99nbiWoDdzV8MltxTtfi50p6WGxHbC5+WaJn6czUhI96Yoy7dnMvEIIABJVZFci7aU7DCNrHMHSiY= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: fbd21a23-9fb2-4575-b657-08d6b1e410c9 X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Mar 2019 12:10:41.1028 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR0502MB3939 Subject: Re: [dpdk-dev] [PATCH v2 05/13] net/mlx5: add IB shared context alloc/free functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190326121041.A8IYVmsm8-GPuMw_9WNV2JEHPcRqoF0Ae1TBQG8zic8@z> Monday, March 25, 2019 7:03 PM, Viacheslav Ovsiienko: > Subject: [PATCH v2 05/13] net/mlx5: add IB shared context alloc/free > functions >=20 > The Mellanox NICs support SR-IOV and have E-Switch feature. > When SR-IOV is set up in switchdev mode and E-Switch is enabled we have > so called VF representors in the system. All representors belonging to th= e > same E-Switch are created on the basis of the single PCI function and wit= h > current implementation each representor has its own dedicated Infiniband > device and operates within its own Infiniband context. It is proposed to > provide representors as ports of the single Infiniband device and operate= on > the shared Infiniband context saving various resources. This patch introd= uces > appropriate structures. >=20 > Alsa the functions to allocate and free shared IB context for multiport a= re > added. The IB device context, Protection Domain, device attributes, > Infiniband names are going to be relocated to the shared structure from t= he > device private one. > mlx5_dev_spawn() is updated to support shared context. >=20 > Signed-off-by: Viacheslav Ovsiienko > --- > drivers/net/mlx5/mlx5.c | 298 +++++++++++++++++++++++++++++++++--- > ------------ > drivers/net/mlx5/mlx5.h | 32 ++++++ > 2 files changed, 235 insertions(+), 95 deletions(-) >=20 > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index > 4338254..4c9621a 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -140,6 +140,141 @@ struct mlx5_dev_spawn_data { > struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */ }; >=20 > +static LIST_HEAD(, mlx5_ibv_shared) mlx5_ibv_list =3D > +LIST_HEAD_INITIALIZER(); > + > +/** > + * Allocate shared IB device context. If there is multiport device the > + * master and representors will share this context, if there is single > + * port dedicated IB device, the context will be used by only given > + * port due to unification. > + * > + * Routine first searches the context for the spesified IB device name, > + * if found the shared context assumed and reference counter is > incremented. > + * If no context found the new one is created and initialized with > +specified > + * IB device context and parameters. > + * > + * @param[in] spawn > + * Pointer to the IB device attributes (name, port, etc). > + * > + * @return > + * Pointer to mlx5_ibv_shared object on success, > + * otherwise NULL and rte_errno is set. > + */ > +static struct mlx5_ibv_shared * > +mlx5_alloc_shared_ibctx(const struct mlx5_dev_spawn_data *spawn) { Sorry Slava, I mislead you. Even though it is not expected for probe to be called by multiple thread it= is safer to protect this list as it is global resources. It is not on the critical datapath, so no big concerns w/ mutex.=20 Please add proper synchronization when manipulating this shared contexts.=20 > + struct mlx5_ibv_shared *sh; > + int err =3D 0; > + > + assert(spawn); > + /* Secondary process should not create the shared context. */ > + assert(rte_eal_process_type() =3D=3D RTE_PROC_PRIMARY); > + /* Search for IB context by device name. */ > + LIST_FOREACH(sh, &mlx5_ibv_list, next) { > + if (!strcmp(sh->ibdev_name, spawn->ibv_dev->name)) { > + sh->refcnt++; > + return sh; > + } > + } > + /* No device found, we have to create new sharted context. */ > + assert(spawn->max_port); > + sh =3D rte_zmalloc("ethdev shared ib context", > + sizeof(struct mlx5_ibv_shared) + > + spawn->max_port * > + sizeof(struct mlx5_ibv_shared_port), > + RTE_CACHE_LINE_SIZE); > + if (!sh) { > + DRV_LOG(ERR, "shared context allocation failure"); > + rte_errno =3D ENOMEM; > + return NULL; > + } > + /* Try to open IB device with DV first, then usual Verbs. */ > + errno =3D 0; > + sh->ctx =3D mlx5_glue->dv_open_device(spawn->ibv_dev); > + if (sh->ctx) { > + sh->devx =3D 1; > + DRV_LOG(DEBUG, "DevX is supported"); > + } else { > + sh->ctx =3D mlx5_glue->open_device(spawn->ibv_dev); > + if (!sh->ctx) { > + err =3D errno ? errno : ENODEV; > + goto error; > + } > + DRV_LOG(DEBUG, "DevX is NOT supported"); > + } > + err =3D mlx5_glue->query_device_ex(sh->ctx, NULL, &sh- > >device_attr); > + if (err) { > + DRV_LOG(DEBUG, "ibv_query_device_ex() failed"); > + goto error; > + } > + sh->refcnt =3D 1; > + sh->max_port =3D spawn->max_port; > + strncpy(sh->ibdev_name, sh->ctx->device->name, > + sizeof(sh->ibdev_name)); > + strncpy(sh->ibdev_path, sh->ctx->device->ibdev_path, > + sizeof(sh->ibdev_path)); > + sh->pd =3D mlx5_glue->alloc_pd(sh->ctx); > + if (sh->pd =3D=3D NULL) { > + DRV_LOG(ERR, "PD allocation failure"); > + err =3D ENOMEM; > + goto error; > + } > + LIST_INSERT_HEAD(&mlx5_ibv_list, sh, next); > + return sh; > +error: > + assert(sh); > + if (sh->pd) > + claim_zero(mlx5_glue->dealloc_pd(sh->pd)); > + if (sh->ctx) > + claim_zero(mlx5_glue->close_device(sh->ctx)); > + rte_free(sh); > + assert(err > 0); > + rte_errno =3D err; > + return NULL; > +} > + > +/** > + * Free shared IB device context. Decrement counter and if zero free > + * all allocated resources and close handles. > + * > + * @param[in] sh > + * Pointer to mlx5_ibv_shared object to free > + */ > +static void > +mlx5_free_shared_ibctx(struct mlx5_ibv_shared *sh) { #ifndef NDEBUG > + /* Check the object presence in the list. */ > + struct mlx5_ibv_shared *lctx; > + > + LIST_FOREACH(lctx, &mlx5_ibv_list, next) > + if (lctx =3D=3D sh) > + break; > + assert(lctx); > + if (lctx !=3D sh) { > + DRV_LOG(ERR, "Freeing non-existing shared IB context"); > + return; > + } > +#endif > + assert(sh); > + assert(sh->refcnt); > + if (--sh->refcnt) > + return; > + /* Zero reference counter, we should release resources. */ > + if (rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY) { > + assert(sh->secondary); > + assert(sh->ctx); > + assert(!sh->pd); > + } > + LIST_REMOVE(sh, next); > + if (sh->pd) > + claim_zero(mlx5_glue->dealloc_pd(sh->pd)); > + if (sh->ctx) > + claim_zero(mlx5_glue->close_device(sh->ctx)); > + rte_free(sh); > +} > + > + > /** > * Prepare shared data between primary and secondary process. > */ > @@ -289,12 +424,10 @@ struct mlx5_dev_spawn_data { > } > mlx5_mprq_free_mp(dev); > mlx5_mr_release(dev); > - if (priv->pd !=3D NULL) { > - assert(priv->ctx !=3D NULL); > - claim_zero(mlx5_glue->dealloc_pd(priv->pd)); > - claim_zero(mlx5_glue->close_device(priv->ctx)); > - } else > - assert(priv->ctx =3D=3D NULL); > + assert(priv->sh); > + if (priv->sh) > + mlx5_free_shared_ibctx(priv->sh); > + priv->sh =3D NULL; > if (priv->rss_conf.rss_key !=3D NULL) > rte_free(priv->rss_conf.rss_key); > if (priv->reta_idx !=3D NULL) > @@ -744,11 +877,8 @@ struct mlx5_dev_spawn_data { > struct mlx5_dev_config config) > { > const struct mlx5_switch_info *switch_info =3D &spawn->info; > - struct ibv_device *ibv_dev =3D spawn->ibv_dev; > - struct ibv_context *ctx =3D NULL; > - struct ibv_device_attr_ex attr; > + struct mlx5_ibv_shared *sh; > struct ibv_port_attr port_attr; > - struct ibv_pd *pd =3D NULL; > struct mlx5dv_context dv_attr =3D { .comp_mask =3D 0 }; > struct rte_eth_dev *eth_dev =3D NULL; > struct mlx5_priv *priv =3D NULL; > @@ -807,18 +937,47 @@ struct mlx5_dev_spawn_data { > } > /* Prepare shared data between primary and secondary process. */ > mlx5_prepare_shared_data(); > - errno =3D 0; > - ctx =3D mlx5_glue->dv_open_device(ibv_dev); > - if (ctx) { > - config.devx =3D 1; > - DRV_LOG(DEBUG, "DEVX is supported"); > - } else { > - ctx =3D mlx5_glue->open_device(ibv_dev); > - if (!ctx) { > - rte_errno =3D errno ? errno : ENODEV; > - return NULL; > + DRV_LOG(DEBUG, "naming Ethernet device \"%s\"", name); > + if (rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY) { > + eth_dev =3D rte_eth_dev_attach_secondary(name); > + if (eth_dev =3D=3D NULL) { > + DRV_LOG(ERR, "can not attach rte ethdev"); > + rte_errno =3D ENOMEM; > + err =3D rte_errno; > + goto error; > + } > + eth_dev->device =3D dpdk_dev; > + eth_dev->dev_ops =3D &mlx5_dev_sec_ops; > + err =3D mlx5_uar_init_secondary(eth_dev); > + if (err) { > + err =3D rte_errno; > + goto error; > } > + /* Receive command fd from primary process */ > + err =3D mlx5_socket_connect(eth_dev); > + if (err < 0) { > + err =3D rte_errno; > + goto error; > + } > + /* Remap UAR for Tx queues. */ > + err =3D mlx5_tx_uar_remap(eth_dev, err); > + if (err) { > + err =3D rte_errno; > + goto error; > + } > + /* > + * Ethdev pointer is still required as input since > + * the primary device is not accessible from the > + * secondary process. > + */ > + eth_dev->rx_pkt_burst =3D > mlx5_select_rx_function(eth_dev); > + eth_dev->tx_pkt_burst =3D > mlx5_select_tx_function(eth_dev); > + return eth_dev; > } > + sh =3D mlx5_alloc_shared_ibctx(spawn); > + if (!sh) > + return NULL; > + config.devx =3D sh->devx; > #ifdef HAVE_IBV_MLX5_MOD_SWP > dv_attr.comp_mask |=3D MLX5DV_CONTEXT_MASK_SWP; #endif @@ > -832,7 +991,7 @@ struct mlx5_dev_spawn_data { #ifdef > HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT > dv_attr.comp_mask |=3D MLX5DV_CONTEXT_MASK_STRIDING_RQ; > #endif > - mlx5_glue->dv_query_device(ctx, &dv_attr); > + mlx5_glue->dv_query_device(sh->ctx, &dv_attr); > if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) { > if (dv_attr.flags & > MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) { > DRV_LOG(DEBUG, "enhanced MPW is supported"); > @@ -917,51 +1076,8 @@ struct mlx5_dev_spawn_data { > " old OFED/rdma-core version or firmware configuration"); > #endif > config.mpls_en =3D mpls_en; > - err =3D mlx5_glue->query_device_ex(ctx, NULL, &attr); > - if (err) { > - DEBUG("ibv_query_device_ex() failed"); > - goto error; > - } > - DRV_LOG(DEBUG, "naming Ethernet device \"%s\"", name); > - if (rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY) { > - eth_dev =3D rte_eth_dev_attach_secondary(name); > - if (eth_dev =3D=3D NULL) { > - DRV_LOG(ERR, "can not attach rte ethdev"); > - rte_errno =3D ENOMEM; > - err =3D rte_errno; > - goto error; > - } > - eth_dev->device =3D dpdk_dev; > - eth_dev->dev_ops =3D &mlx5_dev_sec_ops; > - err =3D mlx5_uar_init_secondary(eth_dev); > - if (err) { > - err =3D rte_errno; > - goto error; > - } > - /* Receive command fd from primary process */ > - err =3D mlx5_socket_connect(eth_dev); > - if (err < 0) { > - err =3D rte_errno; > - goto error; > - } > - /* Remap UAR for Tx queues. */ > - err =3D mlx5_tx_uar_remap(eth_dev, err); > - if (err) { > - err =3D rte_errno; > - goto error; > - } > - /* > - * Ethdev pointer is still required as input since > - * the primary device is not accessible from the > - * secondary process. > - */ > - eth_dev->rx_pkt_burst =3D > mlx5_select_rx_function(eth_dev); > - eth_dev->tx_pkt_burst =3D > mlx5_select_tx_function(eth_dev); > - claim_zero(mlx5_glue->close_device(ctx)); > - return eth_dev; > - } > /* Check port status. */ > - err =3D mlx5_glue->query_port(ctx, spawn->ibv_port, &port_attr); > + err =3D mlx5_glue->query_port(sh->ctx, spawn->ibv_port, &port_attr); > if (err) { > DRV_LOG(ERR, "port query failed: %s", strerror(err)); > goto error; > @@ -975,13 +1091,7 @@ struct mlx5_dev_spawn_data { > DRV_LOG(DEBUG, "port is not active: \"%s\" (%d)", > mlx5_glue->port_state_str(port_attr.state), > port_attr.state); > - /* Allocate protection domain. */ > - pd =3D mlx5_glue->alloc_pd(ctx); > - if (pd =3D=3D NULL) { > - DRV_LOG(ERR, "PD allocation failure"); > - err =3D ENOMEM; > - goto error; > - } > + /* Allocate private eth device data. */ > priv =3D rte_zmalloc("ethdev private structure", > sizeof(*priv), > RTE_CACHE_LINE_SIZE); > @@ -990,13 +1100,11 @@ struct mlx5_dev_spawn_data { > err =3D ENOMEM; > goto error; > } > - priv->ctx =3D ctx; > - strncpy(priv->ibdev_name, priv->ctx->device->name, > - sizeof(priv->ibdev_name)); > - strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, > - sizeof(priv->ibdev_path)); > - priv->device_attr =3D attr; > - priv->pd =3D pd; > + priv->sh =3D sh; > + priv->ctx =3D sh->ctx; > + priv->ibv_port =3D spawn->ibv_port; > + priv->device_attr =3D sh->device_attr; > + priv->pd =3D sh->pd; > priv->mtu =3D ETHER_MTU; > #ifndef RTE_ARCH_64 > /* Initialize UAR access locks for 32bit implementations. */ @@ - > 1062,7 +1170,8 @@ struct mlx5_dev_spawn_data { > strerror(rte_errno)); > goto error; > } > - config.hw_csum =3D !!(attr.device_cap_flags_ex & > IBV_DEVICE_RAW_IP_CSUM); > + config.hw_csum =3D !!(sh->device_attr.device_cap_flags_ex & > + IBV_DEVICE_RAW_IP_CSUM); > DRV_LOG(DEBUG, "checksum offloading is %ssupported", > (config.hw_csum ? "" : "not ")); > #if !defined(HAVE_IBV_DEVICE_COUNTERS_SET_V42) && \ @@ -1076,7 > +1185,7 @@ struct mlx5_dev_spawn_data { > } > #endif > config.ind_table_max_size =3D > - attr.rss_caps.max_rwq_indirection_table_size; > + sh->device_attr.rss_caps.max_rwq_indirection_table_size; > /* > * Remove this check once DPDK supports larger/variable > * indirection tables. > @@ -1085,18 +1194,18 @@ struct mlx5_dev_spawn_data { > config.ind_table_max_size =3D ETH_RSS_RETA_SIZE_512; > DRV_LOG(DEBUG, "maximum Rx indirection table size is %u", > config.ind_table_max_size); > - config.hw_vlan_strip =3D !!(attr.raw_packet_caps & > + config.hw_vlan_strip =3D !!(sh->device_attr.raw_packet_caps & >=20 > IBV_RAW_PACKET_CAP_CVLAN_STRIPPING); > DRV_LOG(DEBUG, "VLAN stripping is %ssupported", > (config.hw_vlan_strip ? "" : "not ")); > - config.hw_fcs_strip =3D !!(attr.raw_packet_caps & > + config.hw_fcs_strip =3D !!(sh->device_attr.raw_packet_caps & > IBV_RAW_PACKET_CAP_SCATTER_FCS); > DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", > (config.hw_fcs_strip ? "" : "not ")); #if > defined(HAVE_IBV_WQ_FLAG_RX_END_PADDING) > - hw_padding =3D !!attr.rx_pad_end_addr_align; > + hw_padding =3D !!sh->device_attr.rx_pad_end_addr_align; > #elif defined(HAVE_IBV_WQ_FLAGS_PCI_WRITE_END_PADDING) > - hw_padding =3D !!(attr.device_cap_flags_ex & > + hw_padding =3D !!(sh->device_attr.device_cap_flags_ex & > IBV_DEVICE_PCI_WRITE_END_PADDING); > #endif > if (config.hw_padding && !hw_padding) { @@ -1105,11 +1214,11 @@ > struct mlx5_dev_spawn_data { > } else if (config.hw_padding) { > DRV_LOG(DEBUG, "Rx end alignment padding is enabled"); > } > - config.tso =3D (attr.tso_caps.max_tso > 0 && > - (attr.tso_caps.supported_qpts & > + config.tso =3D (sh->device_attr.tso_caps.max_tso > 0 && > + (sh->device_attr.tso_caps.supported_qpts & > (1 << IBV_QPT_RAW_PACKET))); > if (config.tso) > - config.tso_max_payload_sz =3D attr.tso_caps.max_tso; > + config.tso_max_payload_sz =3D sh- > >device_attr.tso_caps.max_tso; > /* > * MPW is disabled by default, while the Enhanced MPW is enabled > * by default. > @@ -1243,7 +1352,7 @@ struct mlx5_dev_spawn_data { > } > } > #ifdef HAVE_MLX5DV_DR > - ns =3D mlx5dv_dr_create_ns(ctx, > + ns =3D mlx5dv_dr_create_ns(sh->ctx, >=20 > MLX5DV_DR_NS_DOMAIN_INGRESS_BYPASS); > if (ns =3D=3D NULL) { > DRV_LOG(ERR, "mlx5dv_dr_create_ns failed"); @@ - > 1251,7 +1360,7 @@ struct mlx5_dev_spawn_data { > goto error; > } > priv->rx_ns =3D ns; > - ns =3D mlx5dv_dr_create_ns(ctx, > + ns =3D mlx5dv_dr_create_ns(sh->ctx, >=20 > MLX5DV_DR_NS_DOMAIN_EGRESS_BYPASS); > if (ns =3D=3D NULL) { > DRV_LOG(ERR, "mlx5dv_dr_create_ns failed"); @@ - > 1268,7 +1377,8 @@ struct mlx5_dev_spawn_data { > .free =3D &mlx5_free_verbs_buf, > .data =3D priv, > }; > - mlx5_glue->dv_set_context_attr(ctx, > MLX5DV_CTX_ATTR_BUF_ALLOCATORS, > + mlx5_glue->dv_set_context_attr(sh->ctx, > + MLX5DV_CTX_ATTR_BUF_ALLOCATORS, > (void *)((uintptr_t)&alctr)); > /* Bring Ethernet device up. */ > DRV_LOG(DEBUG, "port %u forcing Ethernet interface up", @@ - > 1322,15 +1432,13 @@ struct mlx5_dev_spawn_data { > if (eth_dev !=3D NULL) > eth_dev->data->dev_private =3D NULL; > } > - if (pd) > - claim_zero(mlx5_glue->dealloc_pd(pd)); > if (eth_dev !=3D NULL) { > /* mac_addrs must not be freed alone because part of > dev_private */ > eth_dev->data->mac_addrs =3D NULL; > rte_eth_dev_release_port(eth_dev); > } > - if (ctx) > - claim_zero(mlx5_glue->close_device(ctx)); > + if (sh) > + mlx5_free_shared_ibctx(sh); > assert(err > 0); > rte_errno =3D err; > return NULL; > diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index > d6967a3..4568de7 100644 > --- a/drivers/net/mlx5/mlx5.h > +++ b/drivers/net/mlx5/mlx5.h > @@ -196,10 +196,42 @@ struct mlx5_flow_tbl_resource { #define > MLX5_MAX_TABLES 1024 #define MLX5_GROUP_FACTOR 1 >=20 > +/* Per port data of shared IB device. */ struct mlx5_ibv_shared_port { > + uint32_t ih_port_id; > + /* > + * Interrupt handler port_id. Used by shared interrupt > + * handler to find the corresponding rte_eth device > + * by IB port index. If value is equal or greater > + * RTE_MAX_ETHPORTS it means there is no subhandler > + * installed for specified IB port index. > + */ > +}; > + > +/* > + * Shared Infiniband device context for Master/Representors > + * which belong to same IB device with multiple IB ports. > + **/ > +struct mlx5_ibv_shared { > + LIST_ENTRY(mlx5_ibv_shared) next; > + uint32_t refcnt; > + uint32_t devx:1; /* Opened with DV. */ > + uint32_t max_port; /* Maximal IB device port index. */ > + struct ibv_context *ctx; /* Verbs/DV context. */ > + struct ibv_pd *pd; /* Protection Domain. */ > + char ibdev_name[IBV_SYSFS_NAME_MAX]; /* IB device name. */ > + char ibdev_path[IBV_SYSFS_PATH_MAX]; /* IB device path for > secondary */ > + struct ibv_device_attr_ex device_attr; /* Device properties. */ > + struct rte_intr_handle intr_handle; /* Interrupt handler for device. > */ > + struct mlx5_ibv_shared_port port[]; /* per device port data array. */ > +}; > + > struct mlx5_priv { > LIST_ENTRY(mlx5_priv) mem_event_cb; > /**< Called by memory event callback. */ > struct rte_eth_dev_data *dev_data; /* Pointer to device data. */ > + struct mlx5_ibv_shared *sh; /* Shared IB device context. */ > + uint32_t ibv_port; /* IB device port number. */ > struct ibv_context *ctx; /* Verbs context. */ > struct ibv_device_attr_ex device_attr; /* Device properties. */ > struct ibv_pd *pd; /* Protection Domain. */ > -- > 1.8.3.1