From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id C9E39A0597; Wed, 8 Apr 2020 11:05:48 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B05F71C11B; Wed, 8 Apr 2020 11:05:48 +0200 (CEST) Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2071.outbound.protection.outlook.com [40.107.21.71]) by dpdk.org (Postfix) with ESMTP id B5CF41C0DA for ; Wed, 8 Apr 2020 11:05:47 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TfR7UqJOdsPbs7OUiIe7GtZL9cgnA1ejITdUYILzOJ12kvGLyV3PgtOW0ykmZs08gW0R9exiseLaPSFWNgr33zEt80aGvYn2FscsHrjGnYHVA2nJIrH6Gzq70D80l66EY02ZDQEut6sjc2g/2GH6oSp62MlhhDcCJTbto8E1bGkEA2sNBG8RerA+1NwqR5n0G3fFgaRaCZwTIhl99a5oOb45A9EKDLrKuKV44E1cUQ6uwjK1F6q2jmld7brspFme50tn9qxueZDSLfke7t3RNnw9IxwTbVfnwXlOCtgCYLSdyVX5o1qCAmjKsq/CeyRq4MZOj/TSCWKnjBHmZp3fmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JMYtSn92Q/JTJ5an5O0DWjdZ8hQOeiHxGL2HkSNIVOM=; b=DYQ/bs+wuGPeoNLl1g4Y03IS9WmNpzk4fjnkuhEhuBdSYFLeHCo6N4CNn1fZ1t4XUmXjMgf0uHdCKpvNAxiZTBZ6INdwVOyllIrEArc2pfIL4GXm6CAYg7K1JXGqSbTBaoY26R1HP6J0mv7A4IVgGP3QtDXqGOyI+P22W0fLrUMC3aY/P9/HK1g+Z5/F0UWS4kC/jxY9kHHFA7YS7etDsPLC+7VuoX1VG+LyjTP1GH8h59zqsT3E60QXlOiZl6nW8S7F6R/e61YT9VogA4NspAqQtd7T8bn+xEjrTcXZsh6cQxZHV0wi7FEKc5KGWtvV8QA5R/zoWWQHsqNVVVjxFA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JMYtSn92Q/JTJ5an5O0DWjdZ8hQOeiHxGL2HkSNIVOM=; b=B/PDmtxNB8DM37neVnAu4slAw1+I1Td1lqyBNgYLY8pVm8j0BXWI1ztZEwk93kCBP/EuHrikR1Ob8PNmKhD0yokordGIaN+tvI1/69+H0q7i3BiO7QSNFF70rQtGkdWC+LnEden9uimX9gmf4JEYA/o2vIqNmhWzOmrNOA3TTl4= Received: from AM4PR05MB3265.eurprd05.prod.outlook.com (2603:10a6:205:8::26) by AM4PR05MB3267.eurprd05.prod.outlook.com (2603:10a6:205:7::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2878.20; Wed, 8 Apr 2020 09:05:46 +0000 Received: from AM4PR05MB3265.eurprd05.prod.outlook.com ([fe80::da5:7919:35c1:894]) by AM4PR05MB3265.eurprd05.prod.outlook.com ([fe80::da5:7919:35c1:894%6]) with mapi id 15.20.2878.018; Wed, 8 Apr 2020 09:05:46 +0000 From: Slava Ovsiienko To: Vu Pham , "dev@dpdk.org" CC: Ori Kam , Matan Azrad , Raslan Darawsheh , Vu Pham Thread-Topic: [PATCH v3 1/4] common/mlx5: refactor multi-process IPC handling codes to common driver Thread-Index: AQHWDP5DtATZfoujcE2FyXxy7nD9fKhu775g Date: Wed, 8 Apr 2020 09:05:46 +0000 Message-ID: References: <20200402192105.11606-1-vuhuoong@mellanox.com> <20200407170058.9274-1-vuhuong@mellanox.com> <20200407170058.9274-2-vuhuong@mellanox.com> In-Reply-To: <20200407170058.9274-2-vuhuong@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=viacheslavo@mellanox.com; x-originating-ip: [95.164.10.10] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: d9c57fbc-7592-45f2-e8df-08d7db9c0679 x-ms-traffictypediagnostic: AM4PR05MB3267:|AM4PR05MB3267: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtFwd x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:1060; x-forefront-prvs: 0367A50BB1 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM4PR05MB3265.eurprd05.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(10009020)(4636009)(396003)(366004)(136003)(346002)(376002)(39860400002)(66946007)(55016002)(66446008)(64756008)(66556008)(5660300002)(66476007)(86362001)(6506007)(107886003)(76116006)(52536014)(54906003)(7696005)(110136005)(2906002)(8936002)(316002)(9686003)(81166007)(81156014)(4326008)(8676002)(478600001)(53546011)(71200400001)(186003)(33656002)(26005); DIR:OUT; SFP:1101; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: u8mGgtCZVrRwb0B98YTHJfJOy1HYDi/V3ibg7XmV4foTaLSMxsgK2xdLREgcNI8UCvsJ6nlQ9xyFfMYVnjr42gMPJ2J5r7rtk2AoPery059RAgbWHz7JNed7nCZiKfP8dna4L6PgbH4buJ0IhtcVy1PvsLy7vNDSgN6N5IJn+LHpj4SOQJ57yTDJ4/xLewVtGx39z0iYwEH93ps91AqBwYLvoBntVisNb7k8KC+5x/kMhMAg6d+JHRR8ufkF6NMiFySAhlM1DmKsi5nzjPCKyLGBxPG70HF3gW7gzMtpTB7hAoIqqut+eNGFUsly/NDcbdvJikQuJNytdDBuvxOR5Z6ru+tJ/NjWf87mk4bZ7ZOo+64usXXQgGVzozf4NiJO3Zxz86SRftxtGIYmpQ4kcX8DqKuVEEbaDHsZYCH59aBIZXlCTfb8ncLcT17cLyNj x-ms-exchange-antispam-messagedata: s+wlepyYAaKHMO9PZkuIrRjtj3xboqbUx1J71QQt+nL0sTYj9Xdcmp0sZAqiuduOQ/klWEOVYTmvg2ZHHqiOOYfitZ9Z7ZIrWiBEcWrCZ6fdzHTBBYmGnUiX8+0nZohj9gTy8ApcJ/3E2+2JXLXWoA== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: d9c57fbc-7592-45f2-e8df-08d7db9c0679 X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Apr 2020 09:05:46.4979 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: oUm0QqPhgE2Ye6tQuhi7vFZ8audSuRx1QLBq9wCNwt5oGkq6zl+xf4TT1Ugiwz4GFXWcwyYSFxY+PbaDFaW8ohwp5agQsW63iMsV2q4mMnA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR05MB3267 Subject: Re: [dpdk-dev] [PATCH v3 1/4] common/mlx5: refactor multi-process IPC handling codes to common driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Vu Pham > Sent: Tuesday, April 7, 2020 20:01 > To: dev@dpdk.org > Cc: Slava Ovsiienko ; Ori Kam > ; Matan Azrad ; Raslan > Darawsheh ; Vu Pham > Subject: [PATCH v3 1/4] common/mlx5: refactor multi-process IPC handling > codes to common driver >=20 > Refactor common multi-process handling codes from net PMD to common > driver. Using tuple mp_id{name, port_id} as standard input parameter for = all > multi-process IPC APIs instead of using rte_eth_dev. >=20 > Signed-off-by: Vu Pham Acked-by: Viacheslav Ovsiienko > --- > drivers/common/mlx5/mlx5_common_mp.c | 188 > ++++++++++++++++++++++++ > drivers/common/mlx5/mlx5_common_mp.h | 98 ++++++++++++ > drivers/common/mlx5/rte_common_mlx5_version.map | 13 ++ > 3 files changed, 299 insertions(+) > create mode 100644 drivers/common/mlx5/mlx5_common_mp.c > create mode 100644 drivers/common/mlx5/mlx5_common_mp.h >=20 > diff --git a/drivers/common/mlx5/mlx5_common_mp.c > b/drivers/common/mlx5/mlx5_common_mp.c > new file mode 100644 > index 0000000000..da55143bc1 > --- /dev/null > +++ b/drivers/common/mlx5/mlx5_common_mp.c > @@ -0,0 +1,188 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright 2019 6WIND S.A. > + * Copyright 2019 Mellanox Technologies, Ltd */ > + > +#include > +#include > + > +#include > +#include > + > +#include "mlx5_common_mp.h" > +#include "mlx5_common_utils.h" > + > +/** > + * Request Memory Region creation to the primary process. > + * > + * @param[in] mp_id > + * ID of the MP process. > + * @param addr > + * Target virtual address to register. > + * > + * @return > + * 0 on success, a negative errno value otherwise and rte_errno is set= . > + */ > +int > +mlx5_mp_req_mr_create(struct mlx5_mp_id *mp_id, uintptr_t addr) { > + struct rte_mp_msg mp_req; > + struct rte_mp_msg *mp_res; > + struct rte_mp_reply mp_rep; > + struct mlx5_mp_param *req =3D (struct mlx5_mp_param > *)mp_req.param; > + struct mlx5_mp_param *res; > + struct timespec ts =3D {.tv_sec =3D MLX5_MP_REQ_TIMEOUT_SEC, .tv_nsec > =3D 0}; > + int ret; > + > + MLX5_ASSERT(rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY); > + mp_init_msg(mp_id, &mp_req, MLX5_MP_REQ_CREATE_MR); > + req->args.addr =3D addr; > + ret =3D rte_mp_request_sync(&mp_req, &mp_rep, &ts); > + if (ret) { > + DRV_LOG(ERR, "port %u request to primary process failed", > + mp_id->port_id); > + return -rte_errno; > + } > + MLX5_ASSERT(mp_rep.nb_received =3D=3D 1); > + mp_res =3D &mp_rep.msgs[0]; > + res =3D (struct mlx5_mp_param *)mp_res->param; > + ret =3D res->result; > + if (ret) > + rte_errno =3D -ret; > + free(mp_rep.msgs); > + return ret; > +} > + > +/** > + * Request Verbs queue state modification to the primary process. > + * > + * @param[in] mp_id > + * ID of the MP process. > + * @param sm > + * State modify parameters. > + * > + * @return > + * 0 on success, a negative errno value otherwise and rte_errno is set= . > + */ > +int > +mlx5_mp_req_queue_state_modify(struct mlx5_mp_id *mp_id, > + struct mlx5_mp_arg_queue_state_modify *sm) { > + struct rte_mp_msg mp_req; > + struct rte_mp_msg *mp_res; > + struct rte_mp_reply mp_rep; > + struct mlx5_mp_param *req =3D (struct mlx5_mp_param > *)mp_req.param; > + struct mlx5_mp_param *res; > + struct timespec ts =3D {.tv_sec =3D MLX5_MP_REQ_TIMEOUT_SEC, .tv_nsec > =3D 0}; > + int ret; > + > + MLX5_ASSERT(rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY); > + mp_init_msg(mp_id, &mp_req, > MLX5_MP_REQ_QUEUE_STATE_MODIFY); > + req->args.state_modify =3D *sm; > + ret =3D rte_mp_request_sync(&mp_req, &mp_rep, &ts); > + if (ret) { > + DRV_LOG(ERR, "port %u request to primary process failed", > + mp_id->port_id); > + return -rte_errno; > + } > + MLX5_ASSERT(mp_rep.nb_received =3D=3D 1); > + mp_res =3D &mp_rep.msgs[0]; > + res =3D (struct mlx5_mp_param *)mp_res->param; > + ret =3D res->result; > + free(mp_rep.msgs); > + return ret; > +} > + > +/** > + * Request Verbs command file descriptor for mmap to the primary process= . > + * > + * @param[in] mp_id > + * ID of the MP process. > + * > + * @return > + * fd on success, a negative errno value otherwise and rte_errno is se= t. > + */ > +int > +mlx5_mp_req_verbs_cmd_fd(struct mlx5_mp_id *mp_id) { > + struct rte_mp_msg mp_req; > + struct rte_mp_msg *mp_res; > + struct rte_mp_reply mp_rep; > + struct mlx5_mp_param *res; > + struct timespec ts =3D {.tv_sec =3D MLX5_MP_REQ_TIMEOUT_SEC, .tv_nsec > =3D 0}; > + int ret; > + > + MLX5_ASSERT(rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY); > + mp_init_msg(mp_id, &mp_req, MLX5_MP_REQ_VERBS_CMD_FD); > + ret =3D rte_mp_request_sync(&mp_req, &mp_rep, &ts); > + if (ret) { > + DRV_LOG(ERR, "port %u request to primary process failed", > + mp_id->port_id); > + return -rte_errno; > + } > + MLX5_ASSERT(mp_rep.nb_received =3D=3D 1); > + mp_res =3D &mp_rep.msgs[0]; > + res =3D (struct mlx5_mp_param *)mp_res->param; > + if (res->result) { > + rte_errno =3D -res->result; > + DRV_LOG(ERR, > + "port %u failed to get command FD from primary > process", > + mp_id->port_id); > + ret =3D -rte_errno; > + goto exit; > + } > + MLX5_ASSERT(mp_res->num_fds =3D=3D 1); > + ret =3D mp_res->fds[0]; > + DRV_LOG(DEBUG, "port %u command FD from primary is %d", > + mp_id->port_id, ret); > +exit: > + free(mp_rep.msgs); > + return ret; > +} > + > +/** > + * Initialize by primary process. > + */ > +int > +mlx5_mp_init_primary(const char *name, const rte_mp_t primary_action) { > + int ret; > + > + MLX5_ASSERT(rte_eal_process_type() =3D=3D RTE_PROC_PRIMARY); > + > + /* primary is allowed to not support IPC */ > + ret =3D rte_mp_action_register(name, primary_action); > + if (ret && rte_errno !=3D ENOTSUP) > + return -1; > + return 0; > +} > + > +/** > + * Un-initialize by primary process. > + */ > +void > +mlx5_mp_uninit_primary(const char *name) { > + MLX5_ASSERT(rte_eal_process_type() =3D=3D RTE_PROC_PRIMARY); > + rte_mp_action_unregister(name); > +} > + > +/** > + * Initialize by secondary process. > + */ > +int > +mlx5_mp_init_secondary(const char *name, const rte_mp_t > +secondary_action) { > + MLX5_ASSERT(rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY); > + return rte_mp_action_register(name, secondary_action); } > + > +/** > + * Un-initialize by secondary process. > + */ > +void > +mlx5_mp_uninit_secondary(const char *name) { > + MLX5_ASSERT(rte_eal_process_type() =3D=3D RTE_PROC_SECONDARY); > + rte_mp_action_unregister(name); > +} > diff --git a/drivers/common/mlx5/mlx5_common_mp.h > b/drivers/common/mlx5/mlx5_common_mp.h > new file mode 100644 > index 0000000000..7aab77acb2 > --- /dev/null > +++ b/drivers/common/mlx5/mlx5_common_mp.h > @@ -0,0 +1,98 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright 2018 6WIND S.A. > + * Copyright 2018 Mellanox Technologies, Ltd */ > + > +#ifndef RTE_PMD_MLX5_COMMON_MP_H_ > +#define RTE_PMD_MLX5_COMMON_MP_H_ > + > +/* Verbs header. */ > +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. > +*/ #ifdef PEDANTIC #pragma GCC diagnostic ignored "-Wpedantic" > +#endif > +#include > +#ifdef PEDANTIC > +#pragma GCC diagnostic error "-Wpedantic" > +#endif > + > +#include > +#include > + > +/* Request types for IPC. */ > +enum mlx5_mp_req_type { > + MLX5_MP_REQ_VERBS_CMD_FD =3D 1, > + MLX5_MP_REQ_CREATE_MR, > + MLX5_MP_REQ_START_RXTX, > + MLX5_MP_REQ_STOP_RXTX, > + MLX5_MP_REQ_QUEUE_STATE_MODIFY, > +}; > + > +struct mlx5_mp_arg_queue_state_modify { > + uint8_t is_wq; /* Set if WQ. */ > + uint16_t queue_id; /* DPDK queue ID. */ > + enum ibv_wq_state state; /* WQ requested state. */ }; > + > +/* Pameters for IPC. */ > +struct mlx5_mp_param { > + enum mlx5_mp_req_type type; > + int port_id; > + int result; > + RTE_STD_C11 > + union { > + uintptr_t addr; /* MLX5_MP_REQ_CREATE_MR */ > + struct mlx5_mp_arg_queue_state_modify state_modify; > + /* MLX5_MP_REQ_QUEUE_STATE_MODIFY */ > + } args; > +}; > + > +/* Identifier of a MP process */ > +struct mlx5_mp_id { > + char name[RTE_MP_MAX_NAME_LEN]; > + uint16_t port_id; > +}; > + > +/** Request timeout for IPC. */ > +#define MLX5_MP_REQ_TIMEOUT_SEC 5 > + > +/** > + * Initialize IPC message. > + * > + * @param[in] port_id > + * Port ID of the device. > + * @param[out] msg > + * Pointer to message to fill in. > + * @param[in] type > + * Message type. > + */ > +static inline void > +mp_init_msg(struct mlx5_mp_id *mp_id, struct rte_mp_msg *msg, > + enum mlx5_mp_req_type type) > +{ > + struct mlx5_mp_param *param =3D (struct mlx5_mp_param *)msg- > >param; > + > + memset(msg, 0, sizeof(*msg)); > + strlcpy(msg->name, mp_id->name, sizeof(msg->name)); > + msg->len_param =3D sizeof(*param); > + param->type =3D type; > + param->port_id =3D mp_id->port_id; > +} > + > +__rte_experimental > +int mlx5_mp_init_primary(const char *name, const rte_mp_t > +primary_action); __rte_experimental void mlx5_mp_uninit_primary(const > +char *name); __rte_experimental int mlx5_mp_init_secondary(const char > +*name, const rte_mp_t secondary_action); __rte_experimental void > +mlx5_mp_uninit_secondary(const char *name); __rte_experimental int > +mlx5_mp_req_mr_create(struct mlx5_mp_id *mp_id, uintptr_t addr); > +__rte_experimental int mlx5_mp_req_queue_state_modify(struct > mlx5_mp_id > +*mp_id, > + struct mlx5_mp_arg_queue_state_modify > *sm); __rte_experimental > +int mlx5_mp_req_verbs_cmd_fd(struct mlx5_mp_id *mp_id); > + > +#endif /* RTE_PMD_MLX5_COMMON_MP_H_ */ > diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map > b/drivers/common/mlx5/rte_common_mlx5_version.map > index aede2a0a51..265703d1c9 100644 > --- a/drivers/common/mlx5/rte_common_mlx5_version.map > +++ b/drivers/common/mlx5/rte_common_mlx5_version.map > @@ -48,4 +48,17 @@ DPDK_20.0.1 { > mlx5_nl_vlan_vmwa_delete; >=20 > mlx5_translate_port_name; > + > +}; > + > +EXPERIMENTAL { > + global: > + > + mlx5_mp_init_primary; > + mlx5_mp_uninit_primary; > + mlx5_mp_init_secondary; > + mlx5_mp_uninit_secondary; > + mlx5_mp_req_mr_create; > + mlx5_mp_req_queue_state_modify; > + mlx5_mp_req_verbs_cmd_fd; > }; > -- > 2.16.6