From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E3A55A0C43; Tue, 19 Oct 2021 12:35:37 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0DD014112E; Tue, 19 Oct 2021 12:35:37 +0200 (CEST) Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam08on2045.outbound.protection.outlook.com [40.107.102.45]) by mails.dpdk.org (Postfix) with ESMTP id B275A4113F for ; Tue, 19 Oct 2021 12:35:35 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=X+eE72ElxcrLbXPo/y/FOn4OBRAdDYj6nPcZVfgsES6CS/6VNS7n2FGoUrd7sapESQENsAzwgCkKSlGlWG85D2DAGIn9iGWOh9I3RD/yfJ7gdXdG2FTlsH7978RqmH1UuFv0Qgwsa+SxuqhNiF06YgTE7/W1FNubq/Ig4zzgBCryKvEr4L+WFY2kJLhYUsxafd502s4K0WZvalFjq2XZWcRIcDk1YS9wMFtFyVfGdl1PAKPTGRonQPgmgw9DYP2XRCVliCIGVX0jOOFy7b7JEjvx3mr1yIWu4zVYDDMuVWoBFjxYZcG/2IJK/diovAeVWUWXYvLp1dJnVwp/ozQIWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5L9vFvHiPV8u8qi/MS7ovPawk4kOCN18Zu/1gHQ4gng=; b=FAcV5E8LsUnOG5S/wHez1FrRyz0g/Ca00XVV0ufxXNOZENg1Gg0e4/q3r7IRx/y2v/k3T25GXU6NtiGZB08gnh4IesJfcYADJE44+dUe4o0hK69Of1JBby2HeJz+sGoDh+1H8C5SZjwevGuCya4hRhuAZRM2AYkBAEK5QV/iOH0MGYzeEjfbEzo6w3mmOPgNeqzQVsH1ap2pcKMbGYKq/iZwFDxQURDhOQOzzCDdqoXmg2lVhqHymiX5mAIh90b0ir0iuRsMq9SGUPzEfC0G2diTqFR2k/WD25GrMUyKCnSLlB/iIQnT5He8sTZHoFoHJzkYXKTfIeuDPQJ2+iNEpA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=ashroe.eu smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5L9vFvHiPV8u8qi/MS7ovPawk4kOCN18Zu/1gHQ4gng=; b=Br3au3eCTTFfwQSxL1n9Pyg/3NRFapXi7hso9d7Mvs/7EXTcpTR1/fmtO+JGsDTR0XEhsmL/DQ5oK8YtOB7yYlpN0mROfjHObQp4mraVC8F+GA+kMS28m03rPEBfoGvMTJkzzl6RHnFXXAw558RANP/YuykfM+HkixuwawJP84+y4dGs/4ChRRAmw653v6hmVN1PVTpL4asQw/NMFXeihOzWUvCW6qjV3GVWs3aCBvV/qWPl30tuPn4P33XaQNk9bs4c2HjqHJzefKmxajWAUnNJ5uLwHE4l71qAAxGBR3RZ8j1rGT7upnQMcO2XfYU8Gk7i3tcz6I0eo2o3+Kz3SA== Received: from DM5PR10CA0024.namprd10.prod.outlook.com (2603:10b6:4:2::34) by SA0PR12MB4352.namprd12.prod.outlook.com (2603:10b6:806:9c::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.18; Tue, 19 Oct 2021 10:35:34 +0000 Received: from DM6NAM11FT040.eop-nam11.prod.protection.outlook.com (2603:10b6:4:2:cafe::89) by DM5PR10CA0024.outlook.office365.com (2603:10b6:4:2::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4628.15 via Frontend Transport; Tue, 19 Oct 2021 10:35:34 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; ashroe.eu; dkim=none (message not signed) header.d=none;ashroe.eu; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT040.mail.protection.outlook.com (10.13.173.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:35:33 +0000 Received: from nvidia.com (172.20.187.5) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 10:35:28 +0000 From: Xueming Li To: CC: , Viacheslav Ovsiienko , Lior Margalit , Matan Azrad , "Ray Kinsella" Date: Tue, 19 Oct 2021 18:34:54 +0800 Message-ID: <20211019103501.2216840-2-xuemingl@nvidia.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211019103501.2216840-1-xuemingl@nvidia.com> References: <20210927083256.337450-1-xuemingl@nvidia.com> <20211019103501.2216840-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL105.nvidia.com (172.20.187.12) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 51d6191c-9518-4cfd-29c9-08d992ec2e53 X-MS-TrafficTypeDiagnostic: SA0PR12MB4352: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2512; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: frxdGf0AvvdLX1qYFJ1+J/0suYJvbjFXZeVMM2UEeZ5jCKd8K8mgvTEVYti9Qj4z+lVT3aMDsBP2DV07RTfBXWqs7Ie4luF8vjlUXgJwZkUayFpsgplmB2TWEpfzS1c0rPqyWLbFGjIToqpgIrwcmsxRfFyfgBxoqXl9UU1BIkep4r9d8nsj6+z1q4kbiGQKApRW5YNAjwLIADtXE/zIizXZ3OGfAcHEkgUIhJtuh2pNixb6rWoRrpNhoUTAebGXMY7gfqIj+0WuObkvYBpSYkvnHzQF2LE3aF0MWW5BzAflwizUcg70IEKkuSqEWZLWjJ3fO8OYjjpn0xfSeANV+FqKB8ZNHL00FgDIwu4s61MJCAOmuVuZagx5Tz1Ji358znZXlRVWrRSJyaJ6aR1Oxj65x3Cuw0Y0i3vxjhZaLUSzq30996juBYZX/Uu3IDgwxO8f4Erw35huQQWM3HEXeAy5HHyjCONVnkTWNB9kq1HNfFZzmrtSNv/JFHusuncV93Iko8EWqq5fqmQlWevf5kslkUUZvwZ/4/UiVmbBmO7CgvVx1KJIyInGp/LpssxXDOWPFpy27Ax3+KttPlig8CpLIAKm1pMxbZXc5xZV6M5UV3urd3nrKqoptKO/aereyPZl5VLiyQ+RlaglZbslWRI9lA0F/Gdpb9NzCqjLNfwKXq5uKtGblRgeVBHUFk5IQLBU69JqeebS1lK0pv6kDg== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(4326008)(70206006)(1076003)(6916009)(83380400001)(55016002)(426003)(36756003)(82310400003)(2616005)(7696005)(16526019)(6666004)(86362001)(2906002)(8936002)(70586007)(54906003)(8676002)(36860700001)(26005)(186003)(47076005)(36906005)(6286002)(336012)(7636003)(356005)(316002)(5660300002)(508600001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2021 10:35:33.5615 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 51d6191c-9518-4cfd-29c9-08d992ec2e53 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT040.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4352 Subject: [dpdk-dev] [PATCH v3 1/8] common/mlx5: add netlink API to get RDMA port state X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Introduce netlink API to get rdma port state. Port state is restrieved based on RDMA device name and port index. Signed-off-by: Xueming Li Acked-by: Viacheslav Ovsiienko --- drivers/common/mlx5/linux/meson.build | 2 + drivers/common/mlx5/linux/mlx5_nl.c | 136 +++++++++++++++++++------- drivers/common/mlx5/linux/mlx5_nl.h | 2 + drivers/common/mlx5/version.map | 1 + 4 files changed, 106 insertions(+), 35 deletions(-) diff --git a/drivers/common/mlx5/linux/meson.build b/drivers/common/mlx5/linux/meson.build index cbea58f557d..2dcd27b7786 100644 --- a/drivers/common/mlx5/linux/meson.build +++ b/drivers/common/mlx5/linux/meson.build @@ -175,6 +175,8 @@ has_sym_args = [ 'RDMA_NLDEV_ATTR_DEV_NAME' ], [ 'HAVE_RDMA_NLDEV_ATTR_PORT_INDEX', 'rdma/rdma_netlink.h', 'RDMA_NLDEV_ATTR_PORT_INDEX' ], + [ 'HAVE_RDMA_NLDEV_ATTR_PORT_STATE', 'rdma/rdma_netlink.h', + 'RDMA_NLDEV_ATTR_PORT_STATE' ], [ 'HAVE_RDMA_NLDEV_ATTR_NDEV_INDEX', 'rdma/rdma_netlink.h', 'RDMA_NLDEV_ATTR_NDEV_INDEX' ], [ 'HAVE_MLX5_DR_FLOW_DUMP', 'infiniband/mlx5dv.h', diff --git a/drivers/common/mlx5/linux/mlx5_nl.c b/drivers/common/mlx5/linux/mlx5_nl.c index 9120a697fd5..4b762850941 100644 --- a/drivers/common/mlx5/linux/mlx5_nl.c +++ b/drivers/common/mlx5/linux/mlx5_nl.c @@ -78,6 +78,9 @@ #ifndef HAVE_RDMA_NLDEV_ATTR_PORT_INDEX #define RDMA_NLDEV_ATTR_PORT_INDEX 3 #endif +#ifndef HAVE_RDMA_NLDEV_ATTR_PORT_STATE +#define RDMA_NLDEV_ATTR_PORT_STATE 12 +#endif #ifndef HAVE_RDMA_NLDEV_ATTR_NDEV_INDEX #define RDMA_NLDEV_ATTR_NDEV_INDEX 50 #endif @@ -160,14 +163,16 @@ struct mlx5_nl_mac_addr { #define MLX5_NL_CMD_GET_IB_INDEX (1 << 1) #define MLX5_NL_CMD_GET_NET_INDEX (1 << 2) #define MLX5_NL_CMD_GET_PORT_INDEX (1 << 3) +#define MLX5_NL_CMD_GET_PORT_STATE (1 << 4) /** Data structure used by mlx5_nl_cmdget_cb(). */ -struct mlx5_nl_ifindex_data { +struct mlx5_nl_port_info { const char *name; /**< IB device name (in). */ uint32_t flags; /**< found attribute flags (out). */ uint32_t ibindex; /**< IB device index (out). */ uint32_t ifindex; /**< Network interface index (out). */ uint32_t portnum; /**< IB device max port number (out). */ + uint16_t state; /**< IB device port state (out). */ }; uint32_t atomic_sn; @@ -966,8 +971,8 @@ mlx5_nl_allmulti(int nlsk_fd, unsigned int iface_idx, int enable) static int mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) { - struct mlx5_nl_ifindex_data *data = arg; - struct mlx5_nl_ifindex_data local = { + struct mlx5_nl_port_info *data = arg; + struct mlx5_nl_port_info local = { .flags = 0, }; size_t off = NLMSG_HDRLEN; @@ -1000,6 +1005,10 @@ mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) local.portnum = *(uint32_t *)payload; local.flags |= MLX5_NL_CMD_GET_PORT_INDEX; break; + case RDMA_NLDEV_ATTR_PORT_STATE: + local.state = *(uint8_t *)payload; + local.flags |= MLX5_NL_CMD_GET_PORT_STATE; + break; default: break; } @@ -1016,6 +1025,7 @@ mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) data->ibindex = local.ibindex; data->ifindex = local.ifindex; data->portnum = local.portnum; + data->state = local.state; } return 0; error: @@ -1024,7 +1034,7 @@ mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) } /** - * Get index of network interface associated with some IB device. + * Get port info of network interface associated with some IB device. * * This is the only somewhat safe method to avoid resorting to heuristics * when faced with port representors. Unfortunately it requires at least @@ -1032,27 +1042,20 @@ mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) * * @param nl * Netlink socket of the RDMA kind (NETLINK_RDMA). - * @param[in] name - * IB device name. * @param[in] pindex * IB device port index, starting from 1 + * @param[out] data + * Pointer to port info. * @return - * A valid (nonzero) interface index on success, 0 otherwise and rte_errno - * is set. + * 0 on success, negative on error and rte_errno is set. */ -unsigned int -mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) +static int +mlx5_nl_port_info(int nl, uint32_t pindex, struct mlx5_nl_port_info *data) { - struct mlx5_nl_ifindex_data data = { - .name = name, - .flags = 0, - .ibindex = 0, /* Determined during first pass. */ - .ifindex = 0, /* Determined during second pass. */ - }; union { struct nlmsghdr nh; uint8_t buf[NLMSG_HDRLEN + - NLA_HDRLEN + NLA_ALIGN(sizeof(data.ibindex)) + + NLA_HDRLEN + NLA_ALIGN(sizeof(data->ibindex)) + NLA_HDRLEN + NLA_ALIGN(sizeof(pindex))]; } req = { .nh = { @@ -1068,24 +1071,24 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) ret = mlx5_nl_send(nl, &req.nh, sn); if (ret < 0) - return 0; - ret = mlx5_nl_recv(nl, sn, mlx5_nl_cmdget_cb, &data); + return ret; + ret = mlx5_nl_recv(nl, sn, mlx5_nl_cmdget_cb, data); if (ret < 0) - return 0; - if (!(data.flags & MLX5_NL_CMD_GET_IB_NAME) || - !(data.flags & MLX5_NL_CMD_GET_IB_INDEX)) + return ret; + if (!(data->flags & MLX5_NL_CMD_GET_IB_NAME) || + !(data->flags & MLX5_NL_CMD_GET_IB_INDEX)) goto error; - data.flags = 0; + data->flags = 0; sn = MLX5_NL_SN_GENERATE; req.nh.nlmsg_type = RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_PORT_GET); req.nh.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK; req.nh.nlmsg_len = NLMSG_LENGTH(sizeof(req.buf) - NLMSG_HDRLEN); na = (void *)((uintptr_t)req.buf + NLMSG_HDRLEN); - na->nla_len = NLA_HDRLEN + sizeof(data.ibindex); + na->nla_len = NLA_HDRLEN + sizeof(data->ibindex); na->nla_type = RDMA_NLDEV_ATTR_DEV_INDEX; memcpy((void *)((uintptr_t)na + NLA_HDRLEN), - &data.ibindex, sizeof(data.ibindex)); + &data->ibindex, sizeof(data->ibindex)); na = (void *)((uintptr_t)na + NLA_ALIGN(na->nla_len)); na->nla_len = NLA_HDRLEN + sizeof(pindex); na->nla_type = RDMA_NLDEV_ATTR_PORT_INDEX; @@ -1093,19 +1096,82 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) &pindex, sizeof(pindex)); ret = mlx5_nl_send(nl, &req.nh, sn); if (ret < 0) - return 0; - ret = mlx5_nl_recv(nl, sn, mlx5_nl_cmdget_cb, &data); + return ret; + ret = mlx5_nl_recv(nl, sn, mlx5_nl_cmdget_cb, data); if (ret < 0) - return 0; - if (!(data.flags & MLX5_NL_CMD_GET_IB_NAME) || - !(data.flags & MLX5_NL_CMD_GET_IB_INDEX) || - !(data.flags & MLX5_NL_CMD_GET_NET_INDEX) || - !data.ifindex) + return ret; + if (!(data->flags & MLX5_NL_CMD_GET_IB_NAME) || + !(data->flags & MLX5_NL_CMD_GET_IB_INDEX) || + !(data->flags & MLX5_NL_CMD_GET_NET_INDEX) || + !data->ifindex) goto error; - return data.ifindex; + return 1; error: rte_errno = ENODEV; - return 0; + return -rte_errno; +} + +/** + * Get index of network interface associated with some IB device. + * + * This is the only somewhat safe method to avoid resorting to heuristics + * when faced with port representors. Unfortunately it requires at least + * Linux 4.17. + * + * @param nl + * Netlink socket of the RDMA kind (NETLINK_RDMA). + * @param[in] name + * IB device name. + * @param[in] pindex + * IB device port index, starting from 1 + * @return + * A valid (nonzero) interface index on success, 0 otherwise and rte_errno + * is set. + */ +unsigned int +mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) +{ + struct mlx5_nl_port_info data = { + .ifindex = 0, + .name = name, + }; + + if (mlx5_nl_port_info(nl, pindex, &data) < 0) + return 0; + return data.ifindex; +} + +/** + * Get IB device port state. + * + * This is the only somewhat safe method to get info for port number >= 255. + * Unfortunately it requires at least Linux 4.17. + * + * @param nl + * Netlink socket of the RDMA kind (NETLINK_RDMA). + * @param[in] name + * IB device name. + * @param[in] pindex + * IB device port index, starting from 1 + * @return + * Port state (ibv_port_state) on success, negative on error + * and rte_errno is set. + */ +int +mlx5_nl_port_state(int nl, const char *name, uint32_t pindex) +{ + struct mlx5_nl_port_info data = { + .state = 0, + .name = name, + }; + + if (mlx5_nl_port_info(nl, pindex, &data) < 0) + return -rte_errno; + if ((data.flags & MLX5_NL_CMD_GET_PORT_STATE) == 0) { + rte_errno = ENOTSUP; + return -rte_errno; + } + return (int)data.state; } /** @@ -1123,7 +1189,7 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) unsigned int mlx5_nl_portnum(int nl, const char *name) { - struct mlx5_nl_ifindex_data data = { + struct mlx5_nl_port_info data = { .flags = 0, .name = name, .ifindex = 0, diff --git a/drivers/common/mlx5/linux/mlx5_nl.h b/drivers/common/mlx5/linux/mlx5_nl.h index 15129ffdc88..396f8f3f20a 100644 --- a/drivers/common/mlx5/linux/mlx5_nl.h +++ b/drivers/common/mlx5/linux/mlx5_nl.h @@ -54,6 +54,8 @@ unsigned int mlx5_nl_portnum(int nl, const char *name); __rte_internal unsigned int mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex); __rte_internal +int mlx5_nl_port_state(int nl, const char *name, uint32_t pindex); +__rte_internal int mlx5_nl_vf_mac_addr_modify(int nlsk_fd, unsigned int iface_idx, struct rte_ether_addr *mac, int vf_index); __rte_internal diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index d3c5040aac8..2a2c7e51ba5 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -134,6 +134,7 @@ INTERNAL { mlx5_nl_mac_addr_flush; # WINDOWS_NO_EXPORT mlx5_nl_mac_addr_remove; # WINDOWS_NO_EXPORT mlx5_nl_mac_addr_sync; # WINDOWS_NO_EXPORT + mlx5_nl_port_state; # WINDOWS_NO_EXPORT mlx5_nl_portnum; # WINDOWS_NO_EXPORT mlx5_nl_promisc; # WINDOWS_NO_EXPORT mlx5_nl_switch_info; # WINDOWS_NO_EXPORT -- 2.33.0