From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <shahafs@mellanox.com>
Received: from EUR01-VE1-obe.outbound.protection.outlook.com
(mail-eopbgr140082.outbound.protection.outlook.com [40.107.14.82])
by dpdk.org (Postfix) with ESMTP id 2F0821B4F6
for <dev@dpdk.org>; Thu, 21 Mar 2019 13:15:12 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com;
s=selector1;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=T3Z1sSjPmLQpjOgryXZLYgX1DOoWpHzNr7zuyQcA89I=;
b=wWuMp8Cl/+b1yksj8BFX/itt8VcVM+ZNmYHfvNfz5tyVXUKvUJkIchaR3nIeCNhbM2oFToxAczkpNGZ5W8rEpzbvJQG/P3p+e9RrZyc+Zw45zncznD76BCIwVA+LyLMcm6/4nRCvjI2km/zD8rVRJAjUfYztAaFymE1g0QwRphA=
Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com (52.133.45.150) by
AM0PR0502MB3827.eurprd05.prod.outlook.com (52.133.47.141) with Microsoft SMTP
Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
15.20.1730.15; Thu, 21 Mar 2019 12:15:11 +0000
Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com
([fe80::84f3:7e92:7a51:1003]) by AM0PR0502MB3795.eurprd05.prod.outlook.com
([fe80::84f3:7e92:7a51:1003%2]) with mapi id 15.20.1730.013; Thu, 21 Mar 2019
12:15:11 +0000
From: Shahaf Shuler <shahafs@mellanox.com>
To: Slava Ovsiienko <viacheslavo@mellanox.com>, "dev@dpdk.org" <dev@dpdk.org>
Thread-Topic: [PATCH 12/14] net/mlx5: update install/uninstall int handler
routines
Thread-Index: AQHU373ZnaYVRFjDE06NvmRmNxlBvqYV+PRQ
Date: Thu, 21 Mar 2019 12:15:10 +0000
Message-ID: <AM0PR0502MB37950318C44F54097E1BEF6FC3420@AM0PR0502MB3795.eurprd05.prod.outlook.com>
References: <1551376985-11096-1-git-send-email-viacheslavo@mellanox.com>
<1553155888-27498-1-git-send-email-viacheslavo@mellanox.com>
<1553155888-27498-13-git-send-email-viacheslavo@mellanox.com>
In-Reply-To: <1553155888-27498-13-git-send-email-viacheslavo@mellanox.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
authentication-results: spf=none (sender IP is )
smtp.mailfrom=shahafs@mellanox.com;
x-originating-ip: [31.154.10.105]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 567a7279-6e9a-46a7-563f-08d6adf6dd9e
x-ms-office365-filtering-ht: Tenant
x-microsoft-antispam: BCL:0; PCL:0;
RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020);
SRVR:AM0PR0502MB3827;
x-ms-traffictypediagnostic: AM0PR0502MB3827:
x-microsoft-antispam-prvs: <AM0PR0502MB3827BFC9B4C002E63A554B8BC3420@AM0PR0502MB3827.eurprd05.prod.outlook.com>
x-forefront-prvs: 0983EAD6B2
x-forefront-antispam-report: SFV:NSPM;
SFS:(10009020)(366004)(396003)(346002)(39860400002)(136003)(376002)(189003)(199004)(81166006)(97736004)(33656002)(68736007)(74316002)(25786009)(7736002)(305945005)(110136005)(316002)(14454004)(2906002)(446003)(2501003)(14444005)(52536014)(11346002)(476003)(71190400001)(256004)(5024004)(86362001)(486006)(99286004)(186003)(5660300002)(66066001)(26005)(7696005)(53936002)(76176011)(102836004)(55016002)(9686003)(6436002)(71200400001)(229853002)(3846002)(8676002)(8936002)(81156014)(15650500001)(478600001)(6246003)(106356001)(6506007)(6116002)(105586002);
DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR0502MB3827;
H:AM0PR0502MB3795.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en;
PTR:InfoNoRecords; MX:1; A:1;
received-spf: None (protection.outlook.com: mellanox.com does not designate
permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: ZlE01Bui6t9l4NU37Vy3k52EorIpOfTbGIg5H7iN9vNNIHpR5tiDQsJdOg0jhvEwUs9NaBLklQLAgzSW8qr5SQovC9bLH1HAoxOTLVhbywNNSgovGMqjmXL+0PmCqodPRbXJCr9hC5iB4xnZE0ZsvTmMGwpXA+fak22lZe3CPvxE8Je3pWCDlK7lzlJhzjB2uQLiWT9yKzgcUtgv4rXEckHnoCVS9A/1HYHR5aAvneZWc942YQBHQi1JK7fTgSB0h+Z950qKQLB0DuDFl7m/uXMySpBArUnosYnMfnpNTc0NLEPiQyvzOmrQUw29VNl1Qff9k1I88FL7agEeLg5pKfK9RsP/tlBTQsWqE9yEXZj0fpMCf7NcJO+iEa04p3VLlUWc3vQI0neovdHKbwA3y5LKYLogoJg5gspuUCC8rFc=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: Mellanox.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 567a7279-6e9a-46a7-563f-08d6adf6dd9e
X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Mar 2019 12:15:10.9824 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR0502MB3827
Subject: Re: [dpdk-dev] [PATCH 12/14] net/mlx5: update install/uninstall int
handler routines
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Thu, 21 Mar 2019 12:15:12 -0000
Thursday, March 21, 2019 10:11 AM, Viacheslav Ovsiienko:
> Subject: [PATCH 12/14] net/mlx5: update install/uninstall int handler rou=
tines
>=20
> We are implementing the support for multport Infiniband device withj
> representors attached to these multiple ports. Asynchronous device event
> notifications (link status change, removal event, etc.) should be shared
> between ports. We are going to implement shared event handler and this
> patch introduces appropriate device structure changes and updated event
> handler install and uninstall routines.
>=20
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
> drivers/net/mlx5/mlx5.c | 14 ++++-
> drivers/net/mlx5/mlx5.h | 3 +-
> drivers/net/mlx5/mlx5_ethdev.c | 118
> ++++++++++++++++++++++++++++++++---------
> 3 files changed, 107 insertions(+), 28 deletions(-)
>=20
> diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index
> 312c42b..44b7a87 100644
> --- a/drivers/net/mlx5/mlx5.c
> +++ b/drivers/net/mlx5/mlx5.c
> @@ -165,6 +165,7 @@ struct mlx5_dev_spawn_data { {
> struct mlx5_ibv_shared *sh;
> int err =3D 0;
> + uint32_t i;
>=20
> assert(spawn);
> /* Search for IB context by device name. */ @@ -212,6 +213,9 @@
> struct mlx5_dev_spawn_data {
> sizeof(sh->ibdev_name));
> strncpy(sh->ibdev_path, sh->ctx->device->ibdev_path,
> sizeof(sh->ibdev_path));
> + pthread_mutex_init(&sh->intr_mutex, NULL);
> + for (i =3D 0; i < sh->max_port; i++)
> + sh->port[i].port_id =3D RTE_MAX_ETHPORTS;
Why you need struct here? You port array is not just of uint32_t type?
> if (rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY) {
> /*
> * For secondary process we just open the IB device @@ -
> 276,6 +280,15 @@ struct mlx5_dev_spawn_data {
> assert(!sh->pd);
> }
> LIST_REMOVE(sh, next);
> + /*
> + * Ensure there is no async event handler installed.
> + * Only primary process handles async device events.
> + **/
> + assert(!sh->intr_cnt);
> + if (sh->intr_cnt)
> + rte_intr_callback_unregister
> + (&sh->intr_handle, mlx5_dev_interrupt_handler,
> sh);
> + pthread_mutex_destroy(&sh->intr_mutex);
> if (sh->pd)
> claim_zero(mlx5_glue->dealloc_pd(sh->pd));
> if (sh->ctx)
> @@ -283,7 +296,6 @@ struct mlx5_dev_spawn_data {
> rte_free(sh);
> }
>=20
> -
> /**
> * Prepare shared data between primary and secondary process.
> */
> diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index
> d816d24..f23298e 100644
> --- a/drivers/net/mlx5/mlx5.h
> +++ b/drivers/net/mlx5/mlx5.h
> @@ -216,6 +216,8 @@ struct mlx5_ibv_shared {
> char ibdev_name[IBV_SYSFS_NAME_MAX]; /* IB device name. */
> char ibdev_path[IBV_SYSFS_PATH_MAX]; /* IB device path for
> secondary */
> struct ibv_device_attr_ex device_attr; /* Device properties. */
> + pthread_mutex_t intr_mutex; /* Interrupt config mutex. */
> + uint32_t intr_cnt; /* Interrupt handler reference counter. */
> struct rte_intr_handle intr_handle; /* Interrupt handler for device.
> */
> struct mlx5_ibv_shared_port port[]; /* per device port data array. */
> }; @@ -245,7 +247,6 @@ struct mlx5_priv {
> struct mlx5_txq_data *(*txqs)[]; /* TX queues. */
> struct rte_mempool *mprq_mp; /* Mempool for Multi-Packet RQ.
> */
> struct rte_eth_rss_conf rss_conf; /* RSS configuration. */
> - struct rte_intr_handle intr_handle; /* Interrupt handler. */
> unsigned int (*reta_idx)[]; /* RETA index table. */
> unsigned int reta_idx_n; /* RETA index size. */
> struct mlx5_drop drop_queue; /* Flow drop queues. */ diff --git
> a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index
> 1b2173b..8358cd2 100644
> --- a/drivers/net/mlx5/mlx5_ethdev.c
> +++ b/drivers/net/mlx5/mlx5_ethdev.c
> @@ -1109,6 +1109,96 @@ int mlx5_fw_version_get(struct rte_eth_dev
> *dev, char *fw_ver, size_t fw_size) }
>=20
> /**
> + * Uninstall shared asynchronous device events handler.
> + * This function is implemeted to support event sharing
> + * between multiple ports of single IB device.
> + *
> + * @param dev
> + * Pointer to Ethernet device.
> + */
> +static void
> +mlx5_dev_shared_handler_uninstall(struct rte_eth_dev *dev) {
> + struct mlx5_priv *priv =3D dev->data->dev_private;
> + struct mlx5_ibv_shared *sh =3D priv->sh;
> +
> + if (rte_eal_process_type() !=3D RTE_PROC_PRIMARY)
> + return;
> + pthread_mutex_lock(&sh->intr_mutex);
> + assert(priv->ibv_port);
> + assert(priv->ibv_port <=3D sh->max_port);
> + assert(dev->data->port_id < RTE_MAX_ETHPORTS);
> + if (sh->port[priv->ibv_port - 1].port_id >=3D RTE_MAX_ETHPORTS)
> + goto exit;
> + assert(sh->port[priv->ibv_port - 1].port_id =3D=3D
> + (uint32_t)dev->data->port_id);
> + assert(sh->intr_cnt);
> + sh->port[priv->ibv_port - 1].port_id =3D RTE_MAX_ETHPORTS;
> + if (!sh->intr_cnt || --sh->intr_cnt)
> + goto exit;
> + rte_intr_callback_unregister(&sh->intr_handle,
> + mlx5_dev_interrupt_handler, sh);
> + sh->intr_handle.fd =3D 0;
> + sh->intr_handle.type =3D RTE_INTR_HANDLE_UNKNOWN;
> +exit:
> + pthread_mutex_unlock(&sh->intr_mutex);
> +}
> +
> +/**
> + * Install shared asyncronous device events handler.
> + * This function is implemeted to support event sharing
> + * between multiple ports of single IB device.
> + *
> + * @param dev
> + * Pointer to Ethernet device.
> + */
> +static void
> +mlx5_dev_shared_handler_install(struct rte_eth_dev *dev) {
> + struct mlx5_priv *priv =3D dev->data->dev_private;
> + struct mlx5_ibv_shared *sh =3D priv->sh;
> + int ret;
> + int flags;
> +
> + if (rte_eal_process_type() !=3D RTE_PROC_PRIMARY)
> + return;
> + pthread_mutex_lock(&sh->intr_mutex);
> + assert(priv->ibv_port);
> + assert(priv->ibv_port <=3D sh->max_port);
> + assert(dev->data->port_id < RTE_MAX_ETHPORTS);
> + if (sh->port[priv->ibv_port - 1].port_id < RTE_MAX_ETHPORTS) {
I don't understand why need an array to understand handler is already exist=
s.=20
Why not the refcnt?
> + /* The handler is already installed for this port. */
> + assert(sh->intr_cnt++);
Asserts are compiled only in debug mode. You should not put any logic (++) =
into them.=20
> + goto exit;
> + }
> + sh->port[priv->ibv_port - 1].port_id =3D (uint32_t)dev->data->port_id;
> + if (sh->intr_cnt) {
> + sh->intr_cnt++;
> + goto exit;
> + }
> + /* No shared handler installed. */
> + assert(sh->ctx->async_fd > 0);
> + flags =3D fcntl(sh->ctx->async_fd, F_GETFL);
> + ret =3D fcntl(sh->ctx->async_fd, F_SETFL, flags | O_NONBLOCK);
> + if (ret) {
> + DRV_LOG(INFO, "failed to change file descriptor"
> + " async event queue");
> + /* Indicate there will be no interrupts. */
> + dev->data->dev_conf.intr_conf.lsc =3D 0;
> + dev->data->dev_conf.intr_conf.rmv =3D 0;
> + sh->port[priv->ibv_port - 1].port_id =3D RTE_MAX_ETHPORTS;
> + goto exit;
> + }
> + sh->intr_handle.fd =3D sh->ctx->async_fd;
> + sh->intr_handle.type =3D RTE_INTR_HANDLE_EXT;
> + rte_intr_callback_register(&sh->intr_handle,
> + mlx5_dev_interrupt_handler, sh);
> + sh->intr_cnt++;
> +exit:
> + pthread_mutex_unlock(&sh->intr_mutex);
> +}
> +
> +/**
> * Uninstall interrupt handler.
> *
> * @param dev
> @@ -1119,15 +1209,10 @@ int mlx5_fw_version_get(struct rte_eth_dev
> *dev, char *fw_ver, size_t fw_size) {
> struct mlx5_priv *priv =3D dev->data->dev_private;
>=20
> - if (dev->data->dev_conf.intr_conf.lsc ||
> - dev->data->dev_conf.intr_conf.rmv)
> - rte_intr_callback_unregister(&priv->intr_handle,
> - mlx5_dev_interrupt_handler,
> dev);
> + mlx5_dev_shared_handler_uninstall(dev);
> if (priv->primary_socket)
> rte_intr_callback_unregister(&priv->intr_handle_socket,
> mlx5_dev_handler_socket, dev);
> - priv->intr_handle.fd =3D 0;
> - priv->intr_handle.type =3D RTE_INTR_HANDLE_UNKNOWN;
> priv->intr_handle_socket.fd =3D 0;
> priv->intr_handle_socket.type =3D RTE_INTR_HANDLE_UNKNOWN; }
> @@ -1142,28 +1227,9 @@ int mlx5_fw_version_get(struct rte_eth_dev
> *dev, char *fw_ver, size_t fw_size)
> mlx5_dev_interrupt_handler_install(struct rte_eth_dev *dev) {
> struct mlx5_priv *priv =3D dev->data->dev_private;
> - struct ibv_context *ctx =3D priv->sh->ctx;
> int ret;
> - int flags;
>=20
> - assert(ctx->async_fd > 0);
> - flags =3D fcntl(ctx->async_fd, F_GETFL);
> - ret =3D fcntl(ctx->async_fd, F_SETFL, flags | O_NONBLOCK);
> - if (ret) {
> - DRV_LOG(INFO,
> - "port %u failed to change file descriptor async event"
> - " queue",
> - dev->data->port_id);
> - dev->data->dev_conf.intr_conf.lsc =3D 0;
> - dev->data->dev_conf.intr_conf.rmv =3D 0;
> - }
> - if (dev->data->dev_conf.intr_conf.lsc ||
> - dev->data->dev_conf.intr_conf.rmv) {
> - priv->intr_handle.fd =3D ctx->async_fd;
> - priv->intr_handle.type =3D RTE_INTR_HANDLE_EXT;
> - rte_intr_callback_register(&priv->intr_handle,
> - mlx5_dev_interrupt_handler, dev);
> - }
> + mlx5_dev_shared_handler_install(dev);
> ret =3D mlx5_socket_init(dev);
> if (ret)
> DRV_LOG(ERR, "port %u cannot initialise socket: %s",
> --
> 1.8.3.1
From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from dpdk.org (dpdk.org [92.243.14.124])
by dpdk.space (Postfix) with ESMTP id AD570A00E6
for <public@inbox.dpdk.org>; Thu, 21 Mar 2019 13:15:55 +0100 (CET)
Received: from [92.243.14.124] (localhost [127.0.0.1])
by dpdk.org (Postfix) with ESMTP id 3B5CA1B4F6;
Thu, 21 Mar 2019 13:15:13 +0100 (CET)
Received: from EUR01-VE1-obe.outbound.protection.outlook.com
(mail-eopbgr140082.outbound.protection.outlook.com [40.107.14.82])
by dpdk.org (Postfix) with ESMTP id 2F0821B4F6
for <dev@dpdk.org>; Thu, 21 Mar 2019 13:15:12 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com;
s=selector1;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=T3Z1sSjPmLQpjOgryXZLYgX1DOoWpHzNr7zuyQcA89I=;
b=wWuMp8Cl/+b1yksj8BFX/itt8VcVM+ZNmYHfvNfz5tyVXUKvUJkIchaR3nIeCNhbM2oFToxAczkpNGZ5W8rEpzbvJQG/P3p+e9RrZyc+Zw45zncznD76BCIwVA+LyLMcm6/4nRCvjI2km/zD8rVRJAjUfYztAaFymE1g0QwRphA=
Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com (52.133.45.150) by
AM0PR0502MB3827.eurprd05.prod.outlook.com (52.133.47.141) with Microsoft SMTP
Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
15.20.1730.15; Thu, 21 Mar 2019 12:15:11 +0000
Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com
([fe80::84f3:7e92:7a51:1003]) by AM0PR0502MB3795.eurprd05.prod.outlook.com
([fe80::84f3:7e92:7a51:1003%2]) with mapi id 15.20.1730.013; Thu, 21 Mar 2019
12:15:11 +0000
From: Shahaf Shuler <shahafs@mellanox.com>
To: Slava Ovsiienko <viacheslavo@mellanox.com>, "dev@dpdk.org" <dev@dpdk.org>
Thread-Topic: [PATCH 12/14] net/mlx5: update install/uninstall int handler
routines
Thread-Index: AQHU373ZnaYVRFjDE06NvmRmNxlBvqYV+PRQ
Date: Thu, 21 Mar 2019 12:15:10 +0000
Message-ID:
<AM0PR0502MB37950318C44F54097E1BEF6FC3420@AM0PR0502MB3795.eurprd05.prod.outlook.com>
References: <1551376985-11096-1-git-send-email-viacheslavo@mellanox.com>
<1553155888-27498-1-git-send-email-viacheslavo@mellanox.com>
<1553155888-27498-13-git-send-email-viacheslavo@mellanox.com>
In-Reply-To: <1553155888-27498-13-git-send-email-viacheslavo@mellanox.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
authentication-results: spf=none (sender IP is )
smtp.mailfrom=shahafs@mellanox.com;
x-originating-ip: [31.154.10.105]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 567a7279-6e9a-46a7-563f-08d6adf6dd9e
x-ms-office365-filtering-ht: Tenant
x-microsoft-antispam: BCL:0; PCL:0;
RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020);
SRVR:AM0PR0502MB3827;
x-ms-traffictypediagnostic: AM0PR0502MB3827:
x-microsoft-antispam-prvs: <AM0PR0502MB3827BFC9B4C002E63A554B8BC3420@AM0PR0502MB3827.eurprd05.prod.outlook.com>
x-forefront-prvs: 0983EAD6B2
x-forefront-antispam-report: SFV:NSPM;
SFS:(10009020)(366004)(396003)(346002)(39860400002)(136003)(376002)(189003)(199004)(81166006)(97736004)(33656002)(68736007)(74316002)(25786009)(7736002)(305945005)(110136005)(316002)(14454004)(2906002)(446003)(2501003)(14444005)(52536014)(11346002)(476003)(71190400001)(256004)(5024004)(86362001)(486006)(99286004)(186003)(5660300002)(66066001)(26005)(7696005)(53936002)(76176011)(102836004)(55016002)(9686003)(6436002)(71200400001)(229853002)(3846002)(8676002)(8936002)(81156014)(15650500001)(478600001)(6246003)(106356001)(6506007)(6116002)(105586002);
DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR0502MB3827;
H:AM0PR0502MB3795.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en;
PTR:InfoNoRecords; MX:1; A:1;
received-spf: None (protection.outlook.com: mellanox.com does not designate
permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: ZlE01Bui6t9l4NU37Vy3k52EorIpOfTbGIg5H7iN9vNNIHpR5tiDQsJdOg0jhvEwUs9NaBLklQLAgzSW8qr5SQovC9bLH1HAoxOTLVhbywNNSgovGMqjmXL+0PmCqodPRbXJCr9hC5iB4xnZE0ZsvTmMGwpXA+fak22lZe3CPvxE8Je3pWCDlK7lzlJhzjB2uQLiWT9yKzgcUtgv4rXEckHnoCVS9A/1HYHR5aAvneZWc942YQBHQi1JK7fTgSB0h+Z950qKQLB0DuDFl7m/uXMySpBArUnosYnMfnpNTc0NLEPiQyvzOmrQUw29VNl1Qff9k1I88FL7agEeLg5pKfK9RsP/tlBTQsWqE9yEXZj0fpMCf7NcJO+iEa04p3VLlUWc3vQI0neovdHKbwA3y5LKYLogoJg5gspuUCC8rFc=
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: Mellanox.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 567a7279-6e9a-46a7-563f-08d6adf6dd9e
X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Mar 2019 12:15:10.9824 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR0502MB3827
Subject: Re: [dpdk-dev] [PATCH 12/14] net/mlx5: update install/uninstall int
handler routines
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>
Message-ID: <20190321121510.MCGuLyEuiQ9LlrPriNpCo9rKIPiQavWPC6REFMV3xeM@z>
Thursday, March 21, 2019 10:11 AM, Viacheslav Ovsiienko:
> Subject: [PATCH 12/14] net/mlx5: update install/uninstall int handler rou=
tines
>=20
> We are implementing the support for multport Infiniband device withj
> representors attached to these multiple ports. Asynchronous device event
> notifications (link status change, removal event, etc.) should be shared
> between ports. We are going to implement shared event handler and this
> patch introduces appropriate device structure changes and updated event
> handler install and uninstall routines.
>=20
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
> drivers/net/mlx5/mlx5.c | 14 ++++-
> drivers/net/mlx5/mlx5.h | 3 +-
> drivers/net/mlx5/mlx5_ethdev.c | 118
> ++++++++++++++++++++++++++++++++---------
> 3 files changed, 107 insertions(+), 28 deletions(-)
>=20
> diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index
> 312c42b..44b7a87 100644
> --- a/drivers/net/mlx5/mlx5.c
> +++ b/drivers/net/mlx5/mlx5.c
> @@ -165,6 +165,7 @@ struct mlx5_dev_spawn_data { {
> struct mlx5_ibv_shared *sh;
> int err =3D 0;
> + uint32_t i;
>=20
> assert(spawn);
> /* Search for IB context by device name. */ @@ -212,6 +213,9 @@
> struct mlx5_dev_spawn_data {
> sizeof(sh->ibdev_name));
> strncpy(sh->ibdev_path, sh->ctx->device->ibdev_path,
> sizeof(sh->ibdev_path));
> + pthread_mutex_init(&sh->intr_mutex, NULL);
> + for (i =3D 0; i < sh->max_port; i++)
> + sh->port[i].port_id =3D RTE_MAX_ETHPORTS;
Why you need struct here? You port array is not just of uint32_t type?
> if (rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY) {
> /*
> * For secondary process we just open the IB device @@ -
> 276,6 +280,15 @@ struct mlx5_dev_spawn_data {
> assert(!sh->pd);
> }
> LIST_REMOVE(sh, next);
> + /*
> + * Ensure there is no async event handler installed.
> + * Only primary process handles async device events.
> + **/
> + assert(!sh->intr_cnt);
> + if (sh->intr_cnt)
> + rte_intr_callback_unregister
> + (&sh->intr_handle, mlx5_dev_interrupt_handler,
> sh);
> + pthread_mutex_destroy(&sh->intr_mutex);
> if (sh->pd)
> claim_zero(mlx5_glue->dealloc_pd(sh->pd));
> if (sh->ctx)
> @@ -283,7 +296,6 @@ struct mlx5_dev_spawn_data {
> rte_free(sh);
> }
>=20
> -
> /**
> * Prepare shared data between primary and secondary process.
> */
> diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index
> d816d24..f23298e 100644
> --- a/drivers/net/mlx5/mlx5.h
> +++ b/drivers/net/mlx5/mlx5.h
> @@ -216,6 +216,8 @@ struct mlx5_ibv_shared {
> char ibdev_name[IBV_SYSFS_NAME_MAX]; /* IB device name. */
> char ibdev_path[IBV_SYSFS_PATH_MAX]; /* IB device path for
> secondary */
> struct ibv_device_attr_ex device_attr; /* Device properties. */
> + pthread_mutex_t intr_mutex; /* Interrupt config mutex. */
> + uint32_t intr_cnt; /* Interrupt handler reference counter. */
> struct rte_intr_handle intr_handle; /* Interrupt handler for device.
> */
> struct mlx5_ibv_shared_port port[]; /* per device port data array. */
> }; @@ -245,7 +247,6 @@ struct mlx5_priv {
> struct mlx5_txq_data *(*txqs)[]; /* TX queues. */
> struct rte_mempool *mprq_mp; /* Mempool for Multi-Packet RQ.
> */
> struct rte_eth_rss_conf rss_conf; /* RSS configuration. */
> - struct rte_intr_handle intr_handle; /* Interrupt handler. */
> unsigned int (*reta_idx)[]; /* RETA index table. */
> unsigned int reta_idx_n; /* RETA index size. */
> struct mlx5_drop drop_queue; /* Flow drop queues. */ diff --git
> a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index
> 1b2173b..8358cd2 100644
> --- a/drivers/net/mlx5/mlx5_ethdev.c
> +++ b/drivers/net/mlx5/mlx5_ethdev.c
> @@ -1109,6 +1109,96 @@ int mlx5_fw_version_get(struct rte_eth_dev
> *dev, char *fw_ver, size_t fw_size) }
>=20
> /**
> + * Uninstall shared asynchronous device events handler.
> + * This function is implemeted to support event sharing
> + * between multiple ports of single IB device.
> + *
> + * @param dev
> + * Pointer to Ethernet device.
> + */
> +static void
> +mlx5_dev_shared_handler_uninstall(struct rte_eth_dev *dev) {
> + struct mlx5_priv *priv =3D dev->data->dev_private;
> + struct mlx5_ibv_shared *sh =3D priv->sh;
> +
> + if (rte_eal_process_type() !=3D RTE_PROC_PRIMARY)
> + return;
> + pthread_mutex_lock(&sh->intr_mutex);
> + assert(priv->ibv_port);
> + assert(priv->ibv_port <=3D sh->max_port);
> + assert(dev->data->port_id < RTE_MAX_ETHPORTS);
> + if (sh->port[priv->ibv_port - 1].port_id >=3D RTE_MAX_ETHPORTS)
> + goto exit;
> + assert(sh->port[priv->ibv_port - 1].port_id =3D=3D
> + (uint32_t)dev->data->port_id);
> + assert(sh->intr_cnt);
> + sh->port[priv->ibv_port - 1].port_id =3D RTE_MAX_ETHPORTS;
> + if (!sh->intr_cnt || --sh->intr_cnt)
> + goto exit;
> + rte_intr_callback_unregister(&sh->intr_handle,
> + mlx5_dev_interrupt_handler, sh);
> + sh->intr_handle.fd =3D 0;
> + sh->intr_handle.type =3D RTE_INTR_HANDLE_UNKNOWN;
> +exit:
> + pthread_mutex_unlock(&sh->intr_mutex);
> +}
> +
> +/**
> + * Install shared asyncronous device events handler.
> + * This function is implemeted to support event sharing
> + * between multiple ports of single IB device.
> + *
> + * @param dev
> + * Pointer to Ethernet device.
> + */
> +static void
> +mlx5_dev_shared_handler_install(struct rte_eth_dev *dev) {
> + struct mlx5_priv *priv =3D dev->data->dev_private;
> + struct mlx5_ibv_shared *sh =3D priv->sh;
> + int ret;
> + int flags;
> +
> + if (rte_eal_process_type() !=3D RTE_PROC_PRIMARY)
> + return;
> + pthread_mutex_lock(&sh->intr_mutex);
> + assert(priv->ibv_port);
> + assert(priv->ibv_port <=3D sh->max_port);
> + assert(dev->data->port_id < RTE_MAX_ETHPORTS);
> + if (sh->port[priv->ibv_port - 1].port_id < RTE_MAX_ETHPORTS) {
I don't understand why need an array to understand handler is already exist=
s.=20
Why not the refcnt?
> + /* The handler is already installed for this port. */
> + assert(sh->intr_cnt++);
Asserts are compiled only in debug mode. You should not put any logic (++) =
into them.=20
> + goto exit;
> + }
> + sh->port[priv->ibv_port - 1].port_id =3D (uint32_t)dev->data->port_id;
> + if (sh->intr_cnt) {
> + sh->intr_cnt++;
> + goto exit;
> + }
> + /* No shared handler installed. */
> + assert(sh->ctx->async_fd > 0);
> + flags =3D fcntl(sh->ctx->async_fd, F_GETFL);
> + ret =3D fcntl(sh->ctx->async_fd, F_SETFL, flags | O_NONBLOCK);
> + if (ret) {
> + DRV_LOG(INFO, "failed to change file descriptor"
> + " async event queue");
> + /* Indicate there will be no interrupts. */
> + dev->data->dev_conf.intr_conf.lsc =3D 0;
> + dev->data->dev_conf.intr_conf.rmv =3D 0;
> + sh->port[priv->ibv_port - 1].port_id =3D RTE_MAX_ETHPORTS;
> + goto exit;
> + }
> + sh->intr_handle.fd =3D sh->ctx->async_fd;
> + sh->intr_handle.type =3D RTE_INTR_HANDLE_EXT;
> + rte_intr_callback_register(&sh->intr_handle,
> + mlx5_dev_interrupt_handler, sh);
> + sh->intr_cnt++;
> +exit:
> + pthread_mutex_unlock(&sh->intr_mutex);
> +}
> +
> +/**
> * Uninstall interrupt handler.
> *
> * @param dev
> @@ -1119,15 +1209,10 @@ int mlx5_fw_version_get(struct rte_eth_dev
> *dev, char *fw_ver, size_t fw_size) {
> struct mlx5_priv *priv =3D dev->data->dev_private;
>=20
> - if (dev->data->dev_conf.intr_conf.lsc ||
> - dev->data->dev_conf.intr_conf.rmv)
> - rte_intr_callback_unregister(&priv->intr_handle,
> - mlx5_dev_interrupt_handler,
> dev);
> + mlx5_dev_shared_handler_uninstall(dev);
> if (priv->primary_socket)
> rte_intr_callback_unregister(&priv->intr_handle_socket,
> mlx5_dev_handler_socket, dev);
> - priv->intr_handle.fd =3D 0;
> - priv->intr_handle.type =3D RTE_INTR_HANDLE_UNKNOWN;
> priv->intr_handle_socket.fd =3D 0;
> priv->intr_handle_socket.type =3D RTE_INTR_HANDLE_UNKNOWN; }
> @@ -1142,28 +1227,9 @@ int mlx5_fw_version_get(struct rte_eth_dev
> *dev, char *fw_ver, size_t fw_size)
> mlx5_dev_interrupt_handler_install(struct rte_eth_dev *dev) {
> struct mlx5_priv *priv =3D dev->data->dev_private;
> - struct ibv_context *ctx =3D priv->sh->ctx;
> int ret;
> - int flags;
>=20
> - assert(ctx->async_fd > 0);
> - flags =3D fcntl(ctx->async_fd, F_GETFL);
> - ret =3D fcntl(ctx->async_fd, F_SETFL, flags | O_NONBLOCK);
> - if (ret) {
> - DRV_LOG(INFO,
> - "port %u failed to change file descriptor async event"
> - " queue",
> - dev->data->port_id);
> - dev->data->dev_conf.intr_conf.lsc =3D 0;
> - dev->data->dev_conf.intr_conf.rmv =3D 0;
> - }
> - if (dev->data->dev_conf.intr_conf.lsc ||
> - dev->data->dev_conf.intr_conf.rmv) {
> - priv->intr_handle.fd =3D ctx->async_fd;
> - priv->intr_handle.type =3D RTE_INTR_HANDLE_EXT;
> - rte_intr_callback_register(&priv->intr_handle,
> - mlx5_dev_interrupt_handler, dev);
> - }
> + mlx5_dev_shared_handler_install(dev);
> ret =3D mlx5_socket_init(dev);
> if (ret)
> DRV_LOG(ERR, "port %u cannot initialise socket: %s",
> --
> 1.8.3.1