From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 93263A034F; Sat, 16 Oct 2021 10:08:07 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4BD43410DC; Sat, 16 Oct 2021 10:08:07 +0200 (CEST) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2066.outbound.protection.outlook.com [40.107.220.66]) by mails.dpdk.org (Postfix) with ESMTP id DAF524003F for ; Sat, 16 Oct 2021 10:08:04 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Af00QigO15F2wdL9XYyoPJYsi+9zqQHxRIiWCvHZfOlmZv7I9+rmhfQC58jDHax9PgyaXTKHZGEXyIrgo+B3OHn1Ch+0swxpJuJn4l6xST8exzsdf6N0FBYcC2gKl+HrVfYUGiNYvMyYsjPo3t0AR/UJIUa0oje3og5PWCNVXGNLJ1IrEpnnB7Ct3RPunhne7t1BC5v/BGnoTUnNwhx2B1ZsDg6VUoxgD96Trs4L72zo9Q/Xaq2stxCbweGtXT25jGlATCFIGED288xwK/krk+qZRaEt6yTOWlUz74GtjJZqTkAQEV0uUoJHe9MdGSLbTnGmuBaAi9BYOJnxRnBzIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eL226hXTkCx7jeYW4DR9UvT96OtcN0Ql7UaczP+CjG0=; b=Wc2VrIh6V4/KIMwSxBAt/BqgE09Pp3yyJsxANp3CwUTFbwYTNyf01nN9uGP7brLVtBSBEFebW6R7oUt2Di42w+s3wGKFHDbIMtUil1l+ASBAvYkDCfPmbb5WGyZ5SEFQtiZNkCL6TqMwHR/ut0LrGUfM2pbZ8jS223uOmnX8ha5MZ3qzU/RwuTxxg6Pzo/K5q2dniIizDBkeVEoTU5m8i9y3BkdTIEp5S/aWgsZAl2Hhm8x3LDhUt7SmRDfY6Cw0iyGTM8ufYnZzDgGPqqsjP8J8iCnKVxt9IGs2ZI6cjeha656ZO5QBH1vzJnq6bpoBeHvFxuLPt0MPk5tWzn6DVA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=ashroe.eu smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eL226hXTkCx7jeYW4DR9UvT96OtcN0Ql7UaczP+CjG0=; b=JjnYka6l4lNhgQhuN+XXeYTsuR6TTYO/2I4cCdYtUjyjyPNA+bcH0BxZ+rl0UAbA8XuhohkmU2seRIHL5Cs0ealZUtfR1U/OLeF9/C9U8sZHbqxSubUiJu1fFpcvLr4wTLXxSBtnM25ArOiZcc8kBm76iWV+0tivmqPcByTrRgfyFzJUvoOXuyN5hFFbJFrnkbfYbXE3ZKxdenxdb6y4/itcnu7beo4OKvXD+wrC7Ggwmh4/aRzK5qWSOHtcouiTYLgU3j53I48iOPF++Yxkkoyav8Vk/woOmOR6GIAKtM6LdzTwBmsYOr3b54o1UIVY7ezAxelgvVLPniGZFITmhA== Received: from MWHPR01CA0045.prod.exchangelabs.com (2603:10b6:300:101::31) by BN6PR12MB1283.namprd12.prod.outlook.com (2603:10b6:404:19::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.22; Sat, 16 Oct 2021 08:08:02 +0000 Received: from CO1NAM11FT011.eop-nam11.prod.protection.outlook.com (2603:10b6:300:101:cafe::84) by MWHPR01CA0045.outlook.office365.com (2603:10b6:300:101::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.15 via Frontend Transport; Sat, 16 Oct 2021 08:08:02 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; ashroe.eu; dkim=none (message not signed) header.d=none;ashroe.eu; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by CO1NAM11FT011.mail.protection.outlook.com (10.13.175.186) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Sat, 16 Oct 2021 08:08:01 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Sat, 16 Oct 2021 08:07:59 +0000 From: Xueming Li To: CC: , Viacheslav Ovsiienko , "Lior Margalit" , Matan Azrad , Ray Kinsella Date: Sat, 16 Oct 2021 16:07:25 +0800 Message-ID: <20211016080732.1781622-2-xuemingl@nvidia.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211016080732.1781622-1-xuemingl@nvidia.com> References: <20210927083256.337450-1-xuemingl@nvidia.com> <20211016080732.1781622-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL105.nvidia.com (172.20.187.12) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 166bf004-aef4-439d-35f9-08d9907c1310 X-MS-TrafficTypeDiagnostic: BN6PR12MB1283: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2512; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hk+396IjgxOTY29/0l3LROqdoAX8mUYSidWbCiuZh2rY39IcdUzGM2c3OePL1DoeNaRMDXKCzLdcOS7dmS2zR/PZt2rDbfkVtZ5XKoPZa66o6vpd0ENQRD+Es9U0vQGM9phr/CJau6szRn1X/IZfr+beFUo2hUFzNeEcVsR1xIvrsqd3SdNIA9Fu7txESnEPdZUl2EpphImdxqpZtWbTDm3CF0RhfbGp1zSGcIRG3ykXLStuOOv7x9T9GR5kBr3XHTWy0xZPc03Wuyhhsv6r595WA5Mg71b8iUG1h46fy2AOYUNhslU1kY6h8NV9YMBsfOnfTNJXTameWnw2yoVUIzuxAk7kFW91cOrYFXnnCUA28e2YXUoqze4sOTDGDl1OFNohERhSreA2cXoaDOhcDspBp6gYe9+YF2X7URK3E5ejCt8LqOxM/U5wb+Mj6nnZtIszgeHy6EnOg7fhG1NRgd8TGQn5faIq8eQfYL7hMt+Qm69xL0elyudlg2biJDqGwOOQqZTbVb1GRo5jv+Qv68qG8H5oNvkXh3s/Fjb5Dd5nljT5ib97CJ+eEFTTYOB2zFQP8/kVA93Ni9M5GNkFY3rS4pzoeagove6ZBbx48qXcwt5usuOtXkLiFeGeVtc7dp2bk8cmni0ktlkGYsma/cQAbagjXSEAJSlikcZYsAwW/T821VfNsgSWkOsn5gbRPaglZQZGGBDK7SuNzZKusw== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(336012)(1076003)(54906003)(316002)(2906002)(6666004)(2616005)(6286002)(186003)(16526019)(36906005)(47076005)(426003)(5660300002)(55016002)(83380400001)(4326008)(86362001)(70206006)(82310400003)(36860700001)(26005)(7696005)(8676002)(6916009)(356005)(508600001)(36756003)(70586007)(8936002)(7636003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Oct 2021 08:08:01.8270 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 166bf004-aef4-439d-35f9-08d9907c1310 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT011.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR12MB1283 Subject: [dpdk-dev] [PATCH v2 1/8] common/mlx5: add netlink API to get RDMA port state X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Introduce netlink API to get rdma port state. Port state is restrieved based on RDMA device name and port index. Signed-off-by: Xueming Li --- drivers/common/mlx5/linux/meson.build | 2 + drivers/common/mlx5/linux/mlx5_nl.c | 136 +++++++++++++++++++------- drivers/common/mlx5/linux/mlx5_nl.h | 2 + drivers/common/mlx5/version.map | 1 + 4 files changed, 106 insertions(+), 35 deletions(-) diff --git a/drivers/common/mlx5/linux/meson.build b/drivers/common/mlx5/linux/meson.build index cbea58f557d..2dcd27b7786 100644 --- a/drivers/common/mlx5/linux/meson.build +++ b/drivers/common/mlx5/linux/meson.build @@ -175,6 +175,8 @@ has_sym_args = [ 'RDMA_NLDEV_ATTR_DEV_NAME' ], [ 'HAVE_RDMA_NLDEV_ATTR_PORT_INDEX', 'rdma/rdma_netlink.h', 'RDMA_NLDEV_ATTR_PORT_INDEX' ], + [ 'HAVE_RDMA_NLDEV_ATTR_PORT_STATE', 'rdma/rdma_netlink.h', + 'RDMA_NLDEV_ATTR_PORT_STATE' ], [ 'HAVE_RDMA_NLDEV_ATTR_NDEV_INDEX', 'rdma/rdma_netlink.h', 'RDMA_NLDEV_ATTR_NDEV_INDEX' ], [ 'HAVE_MLX5_DR_FLOW_DUMP', 'infiniband/mlx5dv.h', diff --git a/drivers/common/mlx5/linux/mlx5_nl.c b/drivers/common/mlx5/linux/mlx5_nl.c index 9120a697fd5..4b762850941 100644 --- a/drivers/common/mlx5/linux/mlx5_nl.c +++ b/drivers/common/mlx5/linux/mlx5_nl.c @@ -78,6 +78,9 @@ #ifndef HAVE_RDMA_NLDEV_ATTR_PORT_INDEX #define RDMA_NLDEV_ATTR_PORT_INDEX 3 #endif +#ifndef HAVE_RDMA_NLDEV_ATTR_PORT_STATE +#define RDMA_NLDEV_ATTR_PORT_STATE 12 +#endif #ifndef HAVE_RDMA_NLDEV_ATTR_NDEV_INDEX #define RDMA_NLDEV_ATTR_NDEV_INDEX 50 #endif @@ -160,14 +163,16 @@ struct mlx5_nl_mac_addr { #define MLX5_NL_CMD_GET_IB_INDEX (1 << 1) #define MLX5_NL_CMD_GET_NET_INDEX (1 << 2) #define MLX5_NL_CMD_GET_PORT_INDEX (1 << 3) +#define MLX5_NL_CMD_GET_PORT_STATE (1 << 4) /** Data structure used by mlx5_nl_cmdget_cb(). */ -struct mlx5_nl_ifindex_data { +struct mlx5_nl_port_info { const char *name; /**< IB device name (in). */ uint32_t flags; /**< found attribute flags (out). */ uint32_t ibindex; /**< IB device index (out). */ uint32_t ifindex; /**< Network interface index (out). */ uint32_t portnum; /**< IB device max port number (out). */ + uint16_t state; /**< IB device port state (out). */ }; uint32_t atomic_sn; @@ -966,8 +971,8 @@ mlx5_nl_allmulti(int nlsk_fd, unsigned int iface_idx, int enable) static int mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) { - struct mlx5_nl_ifindex_data *data = arg; - struct mlx5_nl_ifindex_data local = { + struct mlx5_nl_port_info *data = arg; + struct mlx5_nl_port_info local = { .flags = 0, }; size_t off = NLMSG_HDRLEN; @@ -1000,6 +1005,10 @@ mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) local.portnum = *(uint32_t *)payload; local.flags |= MLX5_NL_CMD_GET_PORT_INDEX; break; + case RDMA_NLDEV_ATTR_PORT_STATE: + local.state = *(uint8_t *)payload; + local.flags |= MLX5_NL_CMD_GET_PORT_STATE; + break; default: break; } @@ -1016,6 +1025,7 @@ mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) data->ibindex = local.ibindex; data->ifindex = local.ifindex; data->portnum = local.portnum; + data->state = local.state; } return 0; error: @@ -1024,7 +1034,7 @@ mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) } /** - * Get index of network interface associated with some IB device. + * Get port info of network interface associated with some IB device. * * This is the only somewhat safe method to avoid resorting to heuristics * when faced with port representors. Unfortunately it requires at least @@ -1032,27 +1042,20 @@ mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) * * @param nl * Netlink socket of the RDMA kind (NETLINK_RDMA). - * @param[in] name - * IB device name. * @param[in] pindex * IB device port index, starting from 1 + * @param[out] data + * Pointer to port info. * @return - * A valid (nonzero) interface index on success, 0 otherwise and rte_errno - * is set. + * 0 on success, negative on error and rte_errno is set. */ -unsigned int -mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) +static int +mlx5_nl_port_info(int nl, uint32_t pindex, struct mlx5_nl_port_info *data) { - struct mlx5_nl_ifindex_data data = { - .name = name, - .flags = 0, - .ibindex = 0, /* Determined during first pass. */ - .ifindex = 0, /* Determined during second pass. */ - }; union { struct nlmsghdr nh; uint8_t buf[NLMSG_HDRLEN + - NLA_HDRLEN + NLA_ALIGN(sizeof(data.ibindex)) + + NLA_HDRLEN + NLA_ALIGN(sizeof(data->ibindex)) + NLA_HDRLEN + NLA_ALIGN(sizeof(pindex))]; } req = { .nh = { @@ -1068,24 +1071,24 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) ret = mlx5_nl_send(nl, &req.nh, sn); if (ret < 0) - return 0; - ret = mlx5_nl_recv(nl, sn, mlx5_nl_cmdget_cb, &data); + return ret; + ret = mlx5_nl_recv(nl, sn, mlx5_nl_cmdget_cb, data); if (ret < 0) - return 0; - if (!(data.flags & MLX5_NL_CMD_GET_IB_NAME) || - !(data.flags & MLX5_NL_CMD_GET_IB_INDEX)) + return ret; + if (!(data->flags & MLX5_NL_CMD_GET_IB_NAME) || + !(data->flags & MLX5_NL_CMD_GET_IB_INDEX)) goto error; - data.flags = 0; + data->flags = 0; sn = MLX5_NL_SN_GENERATE; req.nh.nlmsg_type = RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_PORT_GET); req.nh.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK; req.nh.nlmsg_len = NLMSG_LENGTH(sizeof(req.buf) - NLMSG_HDRLEN); na = (void *)((uintptr_t)req.buf + NLMSG_HDRLEN); - na->nla_len = NLA_HDRLEN + sizeof(data.ibindex); + na->nla_len = NLA_HDRLEN + sizeof(data->ibindex); na->nla_type = RDMA_NLDEV_ATTR_DEV_INDEX; memcpy((void *)((uintptr_t)na + NLA_HDRLEN), - &data.ibindex, sizeof(data.ibindex)); + &data->ibindex, sizeof(data->ibindex)); na = (void *)((uintptr_t)na + NLA_ALIGN(na->nla_len)); na->nla_len = NLA_HDRLEN + sizeof(pindex); na->nla_type = RDMA_NLDEV_ATTR_PORT_INDEX; @@ -1093,19 +1096,82 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) &pindex, sizeof(pindex)); ret = mlx5_nl_send(nl, &req.nh, sn); if (ret < 0) - return 0; - ret = mlx5_nl_recv(nl, sn, mlx5_nl_cmdget_cb, &data); + return ret; + ret = mlx5_nl_recv(nl, sn, mlx5_nl_cmdget_cb, data); if (ret < 0) - return 0; - if (!(data.flags & MLX5_NL_CMD_GET_IB_NAME) || - !(data.flags & MLX5_NL_CMD_GET_IB_INDEX) || - !(data.flags & MLX5_NL_CMD_GET_NET_INDEX) || - !data.ifindex) + return ret; + if (!(data->flags & MLX5_NL_CMD_GET_IB_NAME) || + !(data->flags & MLX5_NL_CMD_GET_IB_INDEX) || + !(data->flags & MLX5_NL_CMD_GET_NET_INDEX) || + !data->ifindex) goto error; - return data.ifindex; + return 1; error: rte_errno = ENODEV; - return 0; + return -rte_errno; +} + +/** + * Get index of network interface associated with some IB device. + * + * This is the only somewhat safe method to avoid resorting to heuristics + * when faced with port representors. Unfortunately it requires at least + * Linux 4.17. + * + * @param nl + * Netlink socket of the RDMA kind (NETLINK_RDMA). + * @param[in] name + * IB device name. + * @param[in] pindex + * IB device port index, starting from 1 + * @return + * A valid (nonzero) interface index on success, 0 otherwise and rte_errno + * is set. + */ +unsigned int +mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) +{ + struct mlx5_nl_port_info data = { + .ifindex = 0, + .name = name, + }; + + if (mlx5_nl_port_info(nl, pindex, &data) < 0) + return 0; + return data.ifindex; +} + +/** + * Get IB device port state. + * + * This is the only somewhat safe method to get info for port number >= 255. + * Unfortunately it requires at least Linux 4.17. + * + * @param nl + * Netlink socket of the RDMA kind (NETLINK_RDMA). + * @param[in] name + * IB device name. + * @param[in] pindex + * IB device port index, starting from 1 + * @return + * Port state (ibv_port_state) on success, negative on error + * and rte_errno is set. + */ +int +mlx5_nl_port_state(int nl, const char *name, uint32_t pindex) +{ + struct mlx5_nl_port_info data = { + .state = 0, + .name = name, + }; + + if (mlx5_nl_port_info(nl, pindex, &data) < 0) + return -rte_errno; + if ((data.flags & MLX5_NL_CMD_GET_PORT_STATE) == 0) { + rte_errno = ENOTSUP; + return -rte_errno; + } + return (int)data.state; } /** @@ -1123,7 +1189,7 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) unsigned int mlx5_nl_portnum(int nl, const char *name) { - struct mlx5_nl_ifindex_data data = { + struct mlx5_nl_port_info data = { .flags = 0, .name = name, .ifindex = 0, diff --git a/drivers/common/mlx5/linux/mlx5_nl.h b/drivers/common/mlx5/linux/mlx5_nl.h index 15129ffdc88..396f8f3f20a 100644 --- a/drivers/common/mlx5/linux/mlx5_nl.h +++ b/drivers/common/mlx5/linux/mlx5_nl.h @@ -54,6 +54,8 @@ unsigned int mlx5_nl_portnum(int nl, const char *name); __rte_internal unsigned int mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex); __rte_internal +int mlx5_nl_port_state(int nl, const char *name, uint32_t pindex); +__rte_internal int mlx5_nl_vf_mac_addr_modify(int nlsk_fd, unsigned int iface_idx, struct rte_ether_addr *mac, int vf_index); __rte_internal diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index d3c5040aac8..2a2c7e51ba5 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -134,6 +134,7 @@ INTERNAL { mlx5_nl_mac_addr_flush; # WINDOWS_NO_EXPORT mlx5_nl_mac_addr_remove; # WINDOWS_NO_EXPORT mlx5_nl_mac_addr_sync; # WINDOWS_NO_EXPORT + mlx5_nl_port_state; # WINDOWS_NO_EXPORT mlx5_nl_portnum; # WINDOWS_NO_EXPORT mlx5_nl_promisc; # WINDOWS_NO_EXPORT mlx5_nl_switch_info; # WINDOWS_NO_EXPORT -- 2.33.0