From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id AD570A00E6 for ; Thu, 21 Mar 2019 13:15:55 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 3B5CA1B4F6; Thu, 21 Mar 2019 13:15:13 +0100 (CET) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-eopbgr140082.outbound.protection.outlook.com [40.107.14.82]) by dpdk.org (Postfix) with ESMTP id 2F0821B4F6 for ; Thu, 21 Mar 2019 13:15:12 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T3Z1sSjPmLQpjOgryXZLYgX1DOoWpHzNr7zuyQcA89I=; b=wWuMp8Cl/+b1yksj8BFX/itt8VcVM+ZNmYHfvNfz5tyVXUKvUJkIchaR3nIeCNhbM2oFToxAczkpNGZ5W8rEpzbvJQG/P3p+e9RrZyc+Zw45zncznD76BCIwVA+LyLMcm6/4nRCvjI2km/zD8rVRJAjUfYztAaFymE1g0QwRphA= Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com (52.133.45.150) by AM0PR0502MB3827.eurprd05.prod.outlook.com (52.133.47.141) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1730.15; Thu, 21 Mar 2019 12:15:11 +0000 Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003]) by AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003%2]) with mapi id 15.20.1730.013; Thu, 21 Mar 2019 12:15:11 +0000 From: Shahaf Shuler To: Slava Ovsiienko , "dev@dpdk.org" Thread-Topic: [PATCH 12/14] net/mlx5: update install/uninstall int handler routines Thread-Index: AQHU373ZnaYVRFjDE06NvmRmNxlBvqYV+PRQ Date: Thu, 21 Mar 2019 12:15:10 +0000 Message-ID: References: <1551376985-11096-1-git-send-email-viacheslavo@mellanox.com> <1553155888-27498-1-git-send-email-viacheslavo@mellanox.com> <1553155888-27498-13-git-send-email-viacheslavo@mellanox.com> In-Reply-To: <1553155888-27498-13-git-send-email-viacheslavo@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; x-originating-ip: [31.154.10.105] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 567a7279-6e9a-46a7-563f-08d6adf6dd9e x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020); SRVR:AM0PR0502MB3827; x-ms-traffictypediagnostic: AM0PR0502MB3827: x-microsoft-antispam-prvs: x-forefront-prvs: 0983EAD6B2 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(396003)(346002)(39860400002)(136003)(376002)(189003)(199004)(81166006)(97736004)(33656002)(68736007)(74316002)(25786009)(7736002)(305945005)(110136005)(316002)(14454004)(2906002)(446003)(2501003)(14444005)(52536014)(11346002)(476003)(71190400001)(256004)(5024004)(86362001)(486006)(99286004)(186003)(5660300002)(66066001)(26005)(7696005)(53936002)(76176011)(102836004)(55016002)(9686003)(6436002)(71200400001)(229853002)(3846002)(8676002)(8936002)(81156014)(15650500001)(478600001)(6246003)(106356001)(6506007)(6116002)(105586002); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR0502MB3827; H:AM0PR0502MB3795.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: ZlE01Bui6t9l4NU37Vy3k52EorIpOfTbGIg5H7iN9vNNIHpR5tiDQsJdOg0jhvEwUs9NaBLklQLAgzSW8qr5SQovC9bLH1HAoxOTLVhbywNNSgovGMqjmXL+0PmCqodPRbXJCr9hC5iB4xnZE0ZsvTmMGwpXA+fak22lZe3CPvxE8Je3pWCDlK7lzlJhzjB2uQLiWT9yKzgcUtgv4rXEckHnoCVS9A/1HYHR5aAvneZWc942YQBHQi1JK7fTgSB0h+Z950qKQLB0DuDFl7m/uXMySpBArUnosYnMfnpNTc0NLEPiQyvzOmrQUw29VNl1Qff9k1I88FL7agEeLg5pKfK9RsP/tlBTQsWqE9yEXZj0fpMCf7NcJO+iEa04p3VLlUWc3vQI0neovdHKbwA3y5LKYLogoJg5gspuUCC8rFc= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 567a7279-6e9a-46a7-563f-08d6adf6dd9e X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Mar 2019 12:15:10.9824 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR0502MB3827 Subject: Re: [dpdk-dev] [PATCH 12/14] net/mlx5: update install/uninstall int handler routines X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190321121510.MCGuLyEuiQ9LlrPriNpCo9rKIPiQavWPC6REFMV3xeM@z> Thursday, March 21, 2019 10:11 AM, Viacheslav Ovsiienko: > Subject: [PATCH 12/14] net/mlx5: update install/uninstall int handler rou= tines >=20 > We are implementing the support for multport Infiniband device withj > representors attached to these multiple ports. Asynchronous device event > notifications (link status change, removal event, etc.) should be shared > between ports. We are going to implement shared event handler and this > patch introduces appropriate device structure changes and updated event > handler install and uninstall routines. >=20 > Signed-off-by: Viacheslav Ovsiienko > --- > drivers/net/mlx5/mlx5.c | 14 ++++- > drivers/net/mlx5/mlx5.h | 3 +- > drivers/net/mlx5/mlx5_ethdev.c | 118 > ++++++++++++++++++++++++++++++++--------- > 3 files changed, 107 insertions(+), 28 deletions(-) >=20 > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index > 312c42b..44b7a87 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -165,6 +165,7 @@ struct mlx5_dev_spawn_data { { > struct mlx5_ibv_shared *sh; > int err =3D 0; > + uint32_t i; >=20 > assert(spawn); > /* Search for IB context by device name. */ @@ -212,6 +213,9 @@ > struct mlx5_dev_spawn_data { > sizeof(sh->ibdev_name)); > strncpy(sh->ibdev_path, sh->ctx->device->ibdev_path, > sizeof(sh->ibdev_path)); > + pthread_mutex_init(&sh->intr_mutex, NULL); > + for (i =3D 0; i < sh->max_port; i++) > + sh->port[i].port_id =3D RTE_MAX_ETHPORTS; Why you need struct here? You port array is not just of uint32_t type? > if (rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY) { > /* > * For secondary process we just open the IB device @@ - > 276,6 +280,15 @@ struct mlx5_dev_spawn_data { > assert(!sh->pd); > } > LIST_REMOVE(sh, next); > + /* > + * Ensure there is no async event handler installed. > + * Only primary process handles async device events. > + **/ > + assert(!sh->intr_cnt); > + if (sh->intr_cnt) > + rte_intr_callback_unregister > + (&sh->intr_handle, mlx5_dev_interrupt_handler, > sh); > + pthread_mutex_destroy(&sh->intr_mutex); > if (sh->pd) > claim_zero(mlx5_glue->dealloc_pd(sh->pd)); > if (sh->ctx) > @@ -283,7 +296,6 @@ struct mlx5_dev_spawn_data { > rte_free(sh); > } >=20 > - > /** > * Prepare shared data between primary and secondary process. > */ > diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index > d816d24..f23298e 100644 > --- a/drivers/net/mlx5/mlx5.h > +++ b/drivers/net/mlx5/mlx5.h > @@ -216,6 +216,8 @@ struct mlx5_ibv_shared { > char ibdev_name[IBV_SYSFS_NAME_MAX]; /* IB device name. */ > char ibdev_path[IBV_SYSFS_PATH_MAX]; /* IB device path for > secondary */ > struct ibv_device_attr_ex device_attr; /* Device properties. */ > + pthread_mutex_t intr_mutex; /* Interrupt config mutex. */ > + uint32_t intr_cnt; /* Interrupt handler reference counter. */ > struct rte_intr_handle intr_handle; /* Interrupt handler for device. > */ > struct mlx5_ibv_shared_port port[]; /* per device port data array. */ > }; @@ -245,7 +247,6 @@ struct mlx5_priv { > struct mlx5_txq_data *(*txqs)[]; /* TX queues. */ > struct rte_mempool *mprq_mp; /* Mempool for Multi-Packet RQ. > */ > struct rte_eth_rss_conf rss_conf; /* RSS configuration. */ > - struct rte_intr_handle intr_handle; /* Interrupt handler. */ > unsigned int (*reta_idx)[]; /* RETA index table. */ > unsigned int reta_idx_n; /* RETA index size. */ > struct mlx5_drop drop_queue; /* Flow drop queues. */ diff --git > a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index > 1b2173b..8358cd2 100644 > --- a/drivers/net/mlx5/mlx5_ethdev.c > +++ b/drivers/net/mlx5/mlx5_ethdev.c > @@ -1109,6 +1109,96 @@ int mlx5_fw_version_get(struct rte_eth_dev > *dev, char *fw_ver, size_t fw_size) } >=20 > /** > + * Uninstall shared asynchronous device events handler. > + * This function is implemeted to support event sharing > + * between multiple ports of single IB device. > + * > + * @param dev > + * Pointer to Ethernet device. > + */ > +static void > +mlx5_dev_shared_handler_uninstall(struct rte_eth_dev *dev) { > + struct mlx5_priv *priv =3D dev->data->dev_private; > + struct mlx5_ibv_shared *sh =3D priv->sh; > + > + if (rte_eal_process_type() !=3D RTE_PROC_PRIMARY) > + return; > + pthread_mutex_lock(&sh->intr_mutex); > + assert(priv->ibv_port); > + assert(priv->ibv_port <=3D sh->max_port); > + assert(dev->data->port_id < RTE_MAX_ETHPORTS); > + if (sh->port[priv->ibv_port - 1].port_id >=3D RTE_MAX_ETHPORTS) > + goto exit; > + assert(sh->port[priv->ibv_port - 1].port_id =3D=3D > + (uint32_t)dev->data->port_id); > + assert(sh->intr_cnt); > + sh->port[priv->ibv_port - 1].port_id =3D RTE_MAX_ETHPORTS; > + if (!sh->intr_cnt || --sh->intr_cnt) > + goto exit; > + rte_intr_callback_unregister(&sh->intr_handle, > + mlx5_dev_interrupt_handler, sh); > + sh->intr_handle.fd =3D 0; > + sh->intr_handle.type =3D RTE_INTR_HANDLE_UNKNOWN; > +exit: > + pthread_mutex_unlock(&sh->intr_mutex); > +} > + > +/** > + * Install shared asyncronous device events handler. > + * This function is implemeted to support event sharing > + * between multiple ports of single IB device. > + * > + * @param dev > + * Pointer to Ethernet device. > + */ > +static void > +mlx5_dev_shared_handler_install(struct rte_eth_dev *dev) { > + struct mlx5_priv *priv =3D dev->data->dev_private; > + struct mlx5_ibv_shared *sh =3D priv->sh; > + int ret; > + int flags; > + > + if (rte_eal_process_type() !=3D RTE_PROC_PRIMARY) > + return; > + pthread_mutex_lock(&sh->intr_mutex); > + assert(priv->ibv_port); > + assert(priv->ibv_port <=3D sh->max_port); > + assert(dev->data->port_id < RTE_MAX_ETHPORTS); > + if (sh->port[priv->ibv_port - 1].port_id < RTE_MAX_ETHPORTS) { I don't understand why need an array to understand handler is already exist= s.=20 Why not the refcnt? > + /* The handler is already installed for this port. */ > + assert(sh->intr_cnt++); Asserts are compiled only in debug mode. You should not put any logic (++) = into them.=20 > + goto exit; > + } > + sh->port[priv->ibv_port - 1].port_id =3D (uint32_t)dev->data->port_id; > + if (sh->intr_cnt) { > + sh->intr_cnt++; > + goto exit; > + } > + /* No shared handler installed. */ > + assert(sh->ctx->async_fd > 0); > + flags =3D fcntl(sh->ctx->async_fd, F_GETFL); > + ret =3D fcntl(sh->ctx->async_fd, F_SETFL, flags | O_NONBLOCK); > + if (ret) { > + DRV_LOG(INFO, "failed to change file descriptor" > + " async event queue"); > + /* Indicate there will be no interrupts. */ > + dev->data->dev_conf.intr_conf.lsc =3D 0; > + dev->data->dev_conf.intr_conf.rmv =3D 0; > + sh->port[priv->ibv_port - 1].port_id =3D RTE_MAX_ETHPORTS; > + goto exit; > + } > + sh->intr_handle.fd =3D sh->ctx->async_fd; > + sh->intr_handle.type =3D RTE_INTR_HANDLE_EXT; > + rte_intr_callback_register(&sh->intr_handle, > + mlx5_dev_interrupt_handler, sh); > + sh->intr_cnt++; > +exit: > + pthread_mutex_unlock(&sh->intr_mutex); > +} > + > +/** > * Uninstall interrupt handler. > * > * @param dev > @@ -1119,15 +1209,10 @@ int mlx5_fw_version_get(struct rte_eth_dev > *dev, char *fw_ver, size_t fw_size) { > struct mlx5_priv *priv =3D dev->data->dev_private; >=20 > - if (dev->data->dev_conf.intr_conf.lsc || > - dev->data->dev_conf.intr_conf.rmv) > - rte_intr_callback_unregister(&priv->intr_handle, > - mlx5_dev_interrupt_handler, > dev); > + mlx5_dev_shared_handler_uninstall(dev); > if (priv->primary_socket) > rte_intr_callback_unregister(&priv->intr_handle_socket, > mlx5_dev_handler_socket, dev); > - priv->intr_handle.fd =3D 0; > - priv->intr_handle.type =3D RTE_INTR_HANDLE_UNKNOWN; > priv->intr_handle_socket.fd =3D 0; > priv->intr_handle_socket.type =3D RTE_INTR_HANDLE_UNKNOWN; } > @@ -1142,28 +1227,9 @@ int mlx5_fw_version_get(struct rte_eth_dev > *dev, char *fw_ver, size_t fw_size) > mlx5_dev_interrupt_handler_install(struct rte_eth_dev *dev) { > struct mlx5_priv *priv =3D dev->data->dev_private; > - struct ibv_context *ctx =3D priv->sh->ctx; > int ret; > - int flags; >=20 > - assert(ctx->async_fd > 0); > - flags =3D fcntl(ctx->async_fd, F_GETFL); > - ret =3D fcntl(ctx->async_fd, F_SETFL, flags | O_NONBLOCK); > - if (ret) { > - DRV_LOG(INFO, > - "port %u failed to change file descriptor async event" > - " queue", > - dev->data->port_id); > - dev->data->dev_conf.intr_conf.lsc =3D 0; > - dev->data->dev_conf.intr_conf.rmv =3D 0; > - } > - if (dev->data->dev_conf.intr_conf.lsc || > - dev->data->dev_conf.intr_conf.rmv) { > - priv->intr_handle.fd =3D ctx->async_fd; > - priv->intr_handle.type =3D RTE_INTR_HANDLE_EXT; > - rte_intr_callback_register(&priv->intr_handle, > - mlx5_dev_interrupt_handler, dev); > - } > + mlx5_dev_shared_handler_install(dev); > ret =3D mlx5_socket_init(dev); > if (ret) > DRV_LOG(ERR, "port %u cannot initialise socket: %s", > -- > 1.8.3.1