From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id B69CFA0096 for ; Thu, 14 Mar 2019 13:36:41 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 545E64C9F; Thu, 14 Mar 2019 13:36:36 +0100 (CET) Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-eopbgr130041.outbound.protection.outlook.com [40.107.13.41]) by dpdk.org (Postfix) with ESMTP id 80A03239 for ; Thu, 14 Mar 2019 13:36:34 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=M5zGqC7++GFe76La++0H3geQpLi9Vjzh5YSf4jglMb0=; b=nTIfFIPEhlXJZxjkMeES7i6NkEBgHOpftHGkY0vBWkVmNfgRjKfALuLc+WSm2QupqeqK1pkQwudaTu9pK6Z5mbaNZnbWHA9LO04XIOb7uvLRz4umZmH01nQob0OIZ5kbCA9mkEs+DQ3dJXL/UME+ydO9KOxGDyfN88O/7Z7w+PE= Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com (52.133.45.150) by AM0PR0502MB3891.eurprd05.prod.outlook.com (52.133.45.154) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1709.13; Thu, 14 Mar 2019 12:36:32 +0000 Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003]) by AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003%2]) with mapi id 15.20.1709.011; Thu, 14 Mar 2019 12:36:32 +0000 From: Shahaf Shuler To: Yongseok Koh CC: "dev@dpdk.org" Thread-Topic: [PATCH 2/4] net/mlx5: replace IPC socket with EAL API Thread-Index: AQHU1LgR1u5J7N04B06fUDgYvOqaZKYK6kyw Date: Thu, 14 Mar 2019 12:36:32 +0000 Message-ID: References: <20190307073314.18324-1-yskoh@mellanox.com> <20190307073314.18324-3-yskoh@mellanox.com> In-Reply-To: <20190307073314.18324-3-yskoh@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; x-originating-ip: [31.154.10.105] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 80a81635-ff34-4c7b-33e7-08d6a879b0c2 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020); SRVR:AM0PR0502MB3891; x-ms-traffictypediagnostic: AM0PR0502MB3891: x-microsoft-antispam-prvs: x-forefront-prvs: 09760A0505 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(366004)(376002)(346002)(39860400002)(136003)(189003)(199004)(7696005)(99286004)(186003)(6116002)(14454004)(3846002)(446003)(11346002)(26005)(5660300002)(86362001)(486006)(14444005)(33656002)(256004)(476003)(52536014)(53936002)(71190400001)(71200400001)(66066001)(74316002)(8936002)(305945005)(106356001)(6436002)(8676002)(2906002)(6636002)(229853002)(68736007)(81166006)(81156014)(76176011)(105586002)(4326008)(6506007)(9686003)(30864003)(6862004)(55016002)(6246003)(102836004)(7736002)(25786009)(478600001)(316002)(97736004); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR0502MB3891; H:AM0PR0502MB3795.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: 0htoTDZhJgIB8Myha/qEQxWrztD6JhOZ1HGFaq/BGLUSKlJnJacq75yqNula8ToYqX4Ccxp3bOSdz2SDpbf5BSbQ6MZTh4gHkaCSGlkGV5ZwCvieWnC/LAKft0KBvv8mqcyT43wMl8X89JkDA7PXWCZDL/j5Uf9gndZkfdUlByB+q6e7rg9g0uiH0j+zzmO8rvDIyOn3ZMKxk8wBOeBh7zhfOzt7SZAfOKvI/n+VlSk/S5wUS2lMrWJcX+7YGBmfDFvGG8FsZ39b4HUcBrJU29MedRbFfQvArhVl2ByJVaep1F31NjaREWPxabN6jq+DqTV8vWzJ/sNJzBjDE4dy0h19frejwq1T8Y0NJnxj6YtcYQOfIfowrOM4uu3+WU9yNkPCfI6Sjezb+oTuIzOWBi7YY2cVEjMMJqrbnpBXZs4= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 80a81635-ff34-4c7b-33e7-08d6a879b0c2 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Mar 2019 12:36:32.8649 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR0502MB3891 Subject: Re: [dpdk-dev] [PATCH 2/4] net/mlx5: replace IPC socket with EAL API X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190314123632.Ap3js0pdJx3Vzm3FOxFYTqADhvUil2bf9PKE2qjJS5o@z> Hi Koh,=20 Thursday, March 7, 2019 9:33 AM, Yongseok Koh: > Subject: [PATCH 2/4] net/mlx5: replace IPC socket with EAL API >=20 > Socket API is used for IPC in order for secondary process to acquire Verb > command file descriptor. The FD is used to remap UAR address. The new > multi-process APIs (rte_mp) in EAL are newly introduced. mlx5_socket.c is > replaced with mlx5_mp.c, which uses the new APIs. >=20 > As it is PMD global infrastructure, only one IPC channel is established. > All the IPC message types may have port_id in the message if there is nee= d > to reference a specific device. >=20 > Signed-off-by: Yongseok Koh [...] > /** > diff --git a/drivers/net/mlx5/mlx5_mp.c b/drivers/net/mlx5/mlx5_mp.c new > file mode 100644 index 0000000000..19a1f25f0e > --- /dev/null > +++ b/drivers/net/mlx5/mlx5_mp.c > @@ -0,0 +1,126 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright 2019 6WIND S.A. > + * Copyright 2019 Mellanox Technologies, Ltd */ > + > +#include > +#include > +#include > + > +#include > +#include > +#include > + > +#include "mlx5.h" > +#include "mlx5_utils.h" > + > +/** > + * Initialize IPC message. > + * > + * @param[in] dev > + * Pointer to Ethernet structure. > + * @param[out] msg > + * Pointer to message to fill in. > + * @param[in] type > + * Message type. > + */ > +static inline void > +mp_init_msg(struct rte_eth_dev *dev, struct rte_mp_msg *msg, > + enum mlx5_mp_req_type type) > +{ > + struct mlx5_mp_param *param =3D (struct mlx5_mp_param *)msg- > >param; > + > + memset(msg, 0, sizeof(*msg)); > + strlcpy(msg->name, MLX5_MP_NAME, sizeof(msg->name)); > + msg->len_param =3D sizeof(*param); > + param->type =3D type; > + param->port_id =3D dev->data->port_id; > +} > + > +/** > + * IPC message handler of primary process. > + * > + * @param[in] dev > + * Pointer to Ethernet structure. > + * @param[in] peer > + * Pointer to the peer socket path. > + * > + * @return > + * 0 on success, a negative errno value otherwise and rte_errno is set= . > + */ > +static int > +mp_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer) > { > + struct rte_mp_msg mp_res; > + struct mlx5_mp_param *res =3D (struct mlx5_mp_param > *)mp_res.param; > + const struct mlx5_mp_param *param =3D > + (const struct mlx5_mp_param *)mp_msg->param; > + struct rte_eth_dev *dev =3D &rte_eth_devices[param->port_id]; Need to check dev is a valid one before we continue. If such case happen ne= ed error log (like you have for invalid req).=20 > + struct mlx5_priv *priv =3D dev->data->dev_private; > + int ret =3D 0; > + > + assert(rte_eal_process_type() =3D=3D RTE_PROC_PRIMARY); > + switch (param->type) { > + case MLX5_MP_REQ_VERBS_CMD_FD: > + mp_init_msg(dev, &mp_res, param->type); > + mp_res.num_fds =3D 1; > + mp_res.fds[0] =3D priv->ctx->cmd_fd; > + res->result =3D 0; > + ret =3D rte_mp_reply(&mp_res, peer); > + break; > + default: > + rte_errno =3D EINVAL; > + DRV_LOG(ERR, "port %u invalid mp request type", > + dev->data->port_id); > + return -rte_errno; > + } > + return ret; > +} > + > +/** > + * Request Verbs command file descriptor for mmap to the primary process= . > + * > + * @param[in] dev > + * Pointer to Ethernet structure. > + * > + * @return > + * fd on success, a negative errno value otherwise and rte_errno is se= t. > + */ > +int > +mlx5_mp_req_verbs_cmd_fd(struct rte_eth_dev *dev) { > + struct rte_mp_msg mp_req; > + struct rte_mp_msg *mp_res; > + struct rte_mp_reply mp_rep; > + struct mlx5_mp_param *res __rte_unused; > + struct timespec ts =3D {.tv_sec =3D 5, .tv_nsec =3D 0}; > + int cmd_fd; > + int ret; > + > + assert(rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY); > + mp_init_msg(dev, &mp_req, MLX5_MP_REQ_VERBS_CMD_FD); > + ret =3D rte_mp_request_sync(&mp_req, &mp_rep, &ts); > + if (ret) { > + DRV_LOG(ERR, > + "port %u failed to get command FD from primary > process", > + dev->data->port_id); > + return -rte_errno; > + } > + assert(mp_rep.nb_received =3D=3D 1); > + mp_res =3D &mp_rep.msgs[0]; > + res =3D (struct mlx5_mp_param *)mp_res->param; > + assert(!res->result); Above should not be an assert rather and actual check.=20 > + assert(mp_res->num_fds =3D=3D 1); > + cmd_fd =3D mp_res->fds[0]; > + free(mp_rep.msgs); > + DRV_LOG(DEBUG, "port %u command FD from primary is %d", > + dev->data->port_id, cmd_fd); > + return cmd_fd; > +} > + > +void > +mlx5_mp_init(void) > +{ > + if (rte_eal_process_type() =3D=3D RTE_PROC_PRIMARY) > + rte_mp_action_register(MLX5_MP_NAME, > mp_primary_handle); } > diff --git a/drivers/net/mlx5/mlx5_socket.c > b/drivers/net/mlx5/mlx5_socket.c deleted file mode 100644 index > 41cac3c6aa..0000000000 > --- a/drivers/net/mlx5/mlx5_socket.c > +++ /dev/null > @@ -1,306 +0,0 @@ > -/* SPDX-License-Identifier: BSD-3-Clause > - * Copyright 2016 6WIND S.A. > - * Copyright 2016 Mellanox Technologies, Ltd > - */ > - > -#include > -#include > -#include > -#include > -#include > -#include > -#include > - > -#include "mlx5.h" > -#include "mlx5_utils.h" > - > -/** > - * Initialise the socket to communicate with the secondary process > - * > - * @param[in] dev > - * Pointer to Ethernet device. > - * > - * @return > - * 0 on success, a negative errno value otherwise and rte_errno is set= . > - */ > -int > -mlx5_socket_init(struct rte_eth_dev *dev) -{ > - struct mlx5_priv *priv =3D dev->data->dev_private; > - struct sockaddr_un sun =3D { > - .sun_family =3D AF_UNIX, > - }; > - int ret; > - int flags; > - > - /* > - * Close the last socket that was used to communicate > - * with the secondary process > - */ > - if (priv->primary_socket) > - mlx5_socket_uninit(dev); > - /* > - * Initialise the socket to communicate with the secondary > - * process. > - */ > - ret =3D socket(AF_UNIX, SOCK_STREAM, 0); > - if (ret < 0) { > - rte_errno =3D errno; > - DRV_LOG(WARNING, "port %u secondary process not > supported: %s", > - dev->data->port_id, strerror(errno)); > - goto error; > - } > - priv->primary_socket =3D ret; > - flags =3D fcntl(priv->primary_socket, F_GETFL, 0); > - if (flags =3D=3D -1) { > - rte_errno =3D errno; > - goto error; > - } > - ret =3D fcntl(priv->primary_socket, F_SETFL, flags | O_NONBLOCK); > - if (ret < 0) { > - rte_errno =3D errno; > - goto error; > - } > - snprintf(sun.sun_path, sizeof(sun.sun_path), "/var/tmp/%s_%d", > - MLX5_DRIVER_NAME, priv->primary_socket); > - remove(sun.sun_path); > - ret =3D bind(priv->primary_socket, (const struct sockaddr *)&sun, > - sizeof(sun)); > - if (ret < 0) { > - rte_errno =3D errno; > - DRV_LOG(WARNING, > - "port %u cannot bind socket, secondary process not" > - " supported: %s", > - dev->data->port_id, strerror(errno)); > - goto close; > - } > - ret =3D listen(priv->primary_socket, 0); > - if (ret < 0) { > - rte_errno =3D errno; > - DRV_LOG(WARNING, "port %u secondary process not > supported: %s", > - dev->data->port_id, strerror(errno)); > - goto close; > - } > - return 0; > -close: > - remove(sun.sun_path); > -error: > - claim_zero(close(priv->primary_socket)); > - priv->primary_socket =3D 0; > - return -rte_errno; > -} > - > -/** > - * Un-Initialise the socket to communicate with the secondary process > - * > - * @param[in] dev > - */ > -void > -mlx5_socket_uninit(struct rte_eth_dev *dev) -{ > - struct mlx5_priv *priv =3D dev->data->dev_private; > - > - MKSTR(path, "/var/tmp/%s_%d", MLX5_DRIVER_NAME, priv- > >primary_socket); > - claim_zero(close(priv->primary_socket)); > - priv->primary_socket =3D 0; > - claim_zero(remove(path)); > -} > - > -/** > - * Handle socket interrupts. > - * > - * @param dev > - * Pointer to Ethernet device. > - */ > -void > -mlx5_socket_handle(struct rte_eth_dev *dev) -{ > - struct mlx5_priv *priv =3D dev->data->dev_private; > - int conn_sock; > - int ret =3D 0; > - struct cmsghdr *cmsg =3D NULL; > - struct ucred *cred =3D NULL; > - char buf[CMSG_SPACE(sizeof(struct ucred))] =3D { 0 }; > - char vbuf[1024] =3D { 0 }; > - struct iovec io =3D { > - .iov_base =3D vbuf, > - .iov_len =3D sizeof(*vbuf), > - }; > - struct msghdr msg =3D { > - .msg_iov =3D &io, > - .msg_iovlen =3D 1, > - .msg_control =3D buf, > - .msg_controllen =3D sizeof(buf), > - }; > - int *fd; > - > - /* Accept the connection from the client. */ > - conn_sock =3D accept(priv->primary_socket, NULL, NULL); > - if (conn_sock < 0) { > - DRV_LOG(WARNING, "port %u connection failed: %s", > - dev->data->port_id, strerror(errno)); > - return; > - } > - ret =3D setsockopt(conn_sock, SOL_SOCKET, SO_PASSCRED, &(int){1}, > - sizeof(int)); > - if (ret < 0) { > - ret =3D errno; > - DRV_LOG(WARNING, "port %u cannot change socket > options: %s", > - dev->data->port_id, strerror(rte_errno)); > - goto error; > - } > - ret =3D recvmsg(conn_sock, &msg, MSG_WAITALL); > - if (ret < 0) { > - ret =3D errno; > - DRV_LOG(WARNING, "port %u received an empty message: > %s", > - dev->data->port_id, strerror(rte_errno)); > - goto error; > - } > - /* Expect to receive credentials only. */ > - cmsg =3D CMSG_FIRSTHDR(&msg); > - if (cmsg =3D=3D NULL) { > - DRV_LOG(WARNING, "port %u no message", dev->data- > >port_id); > - goto error; > - } > - if ((cmsg->cmsg_type =3D=3D SCM_CREDENTIALS) && > - (cmsg->cmsg_len >=3D sizeof(*cred))) { > - cred =3D (struct ucred *)CMSG_DATA(cmsg); > - assert(cred !=3D NULL); > - } > - cmsg =3D CMSG_NXTHDR(&msg, cmsg); > - if (cmsg !=3D NULL) { > - DRV_LOG(WARNING, "port %u message wrongly > formatted", > - dev->data->port_id); > - goto error; > - } > - /* Make sure all the ancillary data was received and valid. */ > - if ((cred =3D=3D NULL) || (cred->uid !=3D getuid()) || > - (cred->gid !=3D getgid())) { > - DRV_LOG(WARNING, "port %u wrong credentials", > - dev->data->port_id); > - goto error; > - } > - /* Set-up the ancillary data. */ > - cmsg =3D CMSG_FIRSTHDR(&msg); > - assert(cmsg !=3D NULL); > - cmsg->cmsg_level =3D SOL_SOCKET; > - cmsg->cmsg_type =3D SCM_RIGHTS; > - cmsg->cmsg_len =3D CMSG_LEN(sizeof(priv->ctx->cmd_fd)); > - fd =3D (int *)CMSG_DATA(cmsg); > - *fd =3D priv->ctx->cmd_fd; > - ret =3D sendmsg(conn_sock, &msg, 0); > - if (ret < 0) > - DRV_LOG(WARNING, "port %u cannot send response", > - dev->data->port_id); > -error: > - close(conn_sock); > -} > - > -/** > - * Connect to the primary process. > - * > - * @param[in] dev > - * Pointer to Ethernet structure. > - * > - * @return > - * fd on success, negative errno value otherwise and rte_errno is set. > - */ > -int > -mlx5_socket_connect(struct rte_eth_dev *dev) -{ > - struct mlx5_priv *priv =3D dev->data->dev_private; > - struct sockaddr_un sun =3D { > - .sun_family =3D AF_UNIX, > - }; > - int socket_fd =3D -1; > - int *fd =3D NULL; > - int ret; > - struct ucred *cred; > - char buf[CMSG_SPACE(sizeof(*cred))] =3D { 0 }; > - char vbuf[1024] =3D { 0 }; > - struct iovec io =3D { > - .iov_base =3D vbuf, > - .iov_len =3D sizeof(*vbuf), > - }; > - struct msghdr msg =3D { > - .msg_control =3D buf, > - .msg_controllen =3D sizeof(buf), > - .msg_iov =3D &io, > - .msg_iovlen =3D 1, > - }; > - struct cmsghdr *cmsg; > - > - ret =3D socket(AF_UNIX, SOCK_STREAM, 0); > - if (ret < 0) { > - rte_errno =3D errno; > - DRV_LOG(WARNING, "port %u cannot connect to primary", > - dev->data->port_id); > - goto error; > - } > - socket_fd =3D ret; > - snprintf(sun.sun_path, sizeof(sun.sun_path), "/var/tmp/%s_%d", > - MLX5_DRIVER_NAME, priv->primary_socket); > - ret =3D connect(socket_fd, (const struct sockaddr *)&sun, sizeof(sun)); > - if (ret < 0) { > - rte_errno =3D errno; > - DRV_LOG(WARNING, "port %u cannot connect to primary", > - dev->data->port_id); > - goto error; > - } > - cmsg =3D CMSG_FIRSTHDR(&msg); > - if (cmsg =3D=3D NULL) { > - rte_errno =3D EINVAL; > - DRV_LOG(DEBUG, "port %u cannot get first message", > - dev->data->port_id); > - goto error; > - } > - cmsg->cmsg_level =3D SOL_SOCKET; > - cmsg->cmsg_type =3D SCM_CREDENTIALS; > - cmsg->cmsg_len =3D CMSG_LEN(sizeof(*cred)); > - cred =3D (struct ucred *)CMSG_DATA(cmsg); > - if (cred =3D=3D NULL) { > - rte_errno =3D EINVAL; > - DRV_LOG(DEBUG, "port %u no credentials received", > - dev->data->port_id); > - goto error; > - } > - cred->pid =3D getpid(); > - cred->uid =3D getuid(); > - cred->gid =3D getgid(); > - ret =3D sendmsg(socket_fd, &msg, MSG_DONTWAIT); > - if (ret < 0) { > - rte_errno =3D errno; > - DRV_LOG(WARNING, > - "port %u cannot send credentials to primary: %s", > - dev->data->port_id, strerror(errno)); > - goto error; > - } > - ret =3D recvmsg(socket_fd, &msg, MSG_WAITALL); > - if (ret <=3D 0) { > - rte_errno =3D errno; > - DRV_LOG(WARNING, "port %u no message from primary: > %s", > - dev->data->port_id, strerror(errno)); > - goto error; > - } > - cmsg =3D CMSG_FIRSTHDR(&msg); > - if (cmsg =3D=3D NULL) { > - rte_errno =3D EINVAL; > - DRV_LOG(WARNING, "port %u no file descriptor received", > - dev->data->port_id); > - goto error; > - } > - fd =3D (int *)CMSG_DATA(cmsg); > - if (*fd < 0) { > - DRV_LOG(WARNING, "port %u no file descriptor received: > %s", > - dev->data->port_id, strerror(errno)); > - rte_errno =3D *fd; > - goto error; > - } > - ret =3D *fd; > - close(socket_fd); > - return ret; > -error: > - if (socket_fd !=3D -1) > - close(socket_fd); > - return -rte_errno; > -} > -- > 2.11.0