From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id ADA47A0548 for ; Mon, 16 Aug 2021 18:30:22 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A7BAD41153; Mon, 16 Aug 2021 18:30:22 +0200 (CEST) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2077.outbound.protection.outlook.com [40.107.236.77]) by mails.dpdk.org (Postfix) with ESMTP id 4846C41153 for ; Mon, 16 Aug 2021 18:30:21 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dHH+5iJrJ3H1AG9JCrnLJ9AlL5zaPfUuemXB6OuqN6dJod2hKK6RclS6uVU1x/VeFjzCXcorUTfnWe4UbxSIOPe6FILJZYa9yb80+uM3t8SXG6eGNMawpC8bzTVZ2ZCXBtKKCIPzKyi99+NKAFoHNuvlxElpRglRAXvSy7boxz4bVRwlCO2N90Kvqcv7Mmqxo2jJjZfn2guxoBAE5LYEoFKgRU9qI7SgfNZoZ2tkwhDBm3dNW4MUX0/OXFBl8MH0/tRm2lluhuB0DnXcwHkGXaXWJLAscYw2O+6jMfhTwWJgKaPU4qHhhgi/1UHPSD/TOImgXza9YGVswQegZ0hl/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GHD+bASa2Vi7VyJO9DDQdUvE0Cad74f03ApoH9pumGU=; b=YJQZF6hm1rXlz2jZ37it16rXD8PkaSbwe+T1k0leOrjaqomvTWiFJrTmC4b5kZLqQcJYrtwSFQUN4kVwspBM5pSAnyWo9aAAkGxAfSo6tvHSrGHdpHxZXhPhJT0rNcUM0HOoPMeMd5BcTF3xZryDv41ShNyCmVdRghQFlOUm5yLpm39+I7GMKXqMxeXs33z6xBsYpnHUGfIkaIYJiW+tKSSDY7ut6HfZTj8i3Qm6N1plpLv+dO5nkLJisYI5/n+CexdHiQeEk06t73s0Zi5KY3CiqHXmbSTREU2hL3iV3iO6zEIn42ySsEkVIgoA1q6/VYKoH+SV6XHDQdgu2xkutw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.35) smtp.rcpttodomain=canonical.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GHD+bASa2Vi7VyJO9DDQdUvE0Cad74f03ApoH9pumGU=; b=N1e792SB4X0EjFLMwBtMwa3wed4cm5c+c7ZtucKlQv440euYHku7InvSfQwWN69t+zlJnT7ehLL3ACP/EERJUfikiCWpoMgTN75sdD7IUQHDM1av+eMTKqfV1Ift0kA6x5UxJlBVow1NpHube4BLN5HnTYMzuN2ED7TL+B31dgOTbXm9z816oULb0h1qU3g5vKGuIY+C5+3PHTEdYABvZEF+Nfn1GtWDsf2TH3VqhXJ+f/lIf5mZE0m9IyJWPpnlRfLHNaS/w+FKNDg/q03AWTYsP8AU0hqh+bvDXxAY4ij1ssusULhUU3ejq0xx4dH22ZAO8pUoECQtWF6Hxcod0g== Received: from DS7PR03CA0121.namprd03.prod.outlook.com (2603:10b6:5:3b4::6) by DM5PR1201MB0219.namprd12.prod.outlook.com (2603:10b6:4:56::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.19; Mon, 16 Aug 2021 16:30:19 +0000 Received: from DM6NAM11FT047.eop-nam11.prod.protection.outlook.com (2603:10b6:5:3b4:cafe::ad) by DS7PR03CA0121.outlook.office365.com (2603:10b6:5:3b4::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.17 via Frontend Transport; Mon, 16 Aug 2021 16:30:19 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.35) smtp.mailfrom=nvidia.com; canonical.com; dkim=none (message not signed) header.d=none;canonical.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.35 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.35; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.35) by DM6NAM11FT047.mail.protection.outlook.com (10.13.172.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:30:19 +0000 Received: from DRHQMAIL107.nvidia.com (10.27.9.16) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 16 Aug 2021 16:30:18 +0000 Received: from nvidia.com (172.20.187.6) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 16 Aug 2021 16:30:16 +0000 From: Bing Zhao To: , CC: , Date: Mon, 16 Aug 2021 19:29:51 +0300 Message-ID: <20210816162952.1931473-6-bingz@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210816162952.1931473-1-bingz@nvidia.com> References: <20210816162952.1931473-1-bingz@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL101.nvidia.com (172.20.187.10) To DRHQMAIL107.nvidia.com (10.27.9.16) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 23b4f050-48a3-4dd9-898a-08d960d32301 X-MS-TrafficTypeDiagnostic: DM5PR1201MB0219: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:597; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 57J2h3pr0v5hunYxO/ItZwUX9yMmlhPyZutE8FmcafSqzIAaRGISexqzPRJ4zj2HSsGOAz2Km7u75vH1wuEKzPj/EfvrI6FLRKDAgU2ClqELSNNOPjlaI7TlGWLya0Uyu7yep8Z5M7hGzVfeg1CcZ0hywGxYLS5c2bltTLRQnumJys1isW5h6BayTN5RMj8bsagQUpbAzeHTbZMvNjzEsJDOtNqB33ypAfUr6MoaeQLIuBJFPIv/8u2s3iUl5Aa/N2k3DvWlWIWgEz3UEVkuY3tE5tybye1KTeWua2AYZgG35eVltRCGfku/9oObO0yVEvoh+F1KiELeFRETOxPSSIA4WRIQ9eZI6ELetIwpS+nc+6Q4/ZpaVPfoNBDGQ2rRqqQDhtfbX9C3tuCNZQfKC/PPZsWqain5q1Wo3IaByY9oaBP6Z0Ire0eYucW4tsZjnTv1JxAAE0UXC/2uy8PpyFDAiMSU/E2Fg7ohaVIZKt+paA+l5+R15ATHUgrvJLRPwF4uD9YCaeVuq1zo9uf5tvXntJaAQv2qCkF6URdkoI9Yyruw00PmAYPNFp9+CU8H6LNSQytT4VvIizT0S9qu91wKbdK+CqhR7e/S66LZTyYW7EF5S71h7ctqTCRrDchSjfiHF8R2BQ8Mx9/cGbbIoPMpSAJJaGjxlK7ebegny2qMEAJ9KVFXeK5YIowNWaU382K8FORu+PhOVPpz013h7A== X-Forefront-Antispam-Report: CIP:216.228.112.35; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid04.nvidia.com; CAT:NONE; SFS:(4636009)(376002)(39860400002)(346002)(396003)(136003)(36840700001)(46966006)(2616005)(70206006)(7696005)(26005)(70586007)(30864003)(83380400001)(7636003)(356005)(478600001)(6286002)(36756003)(16526019)(5660300002)(55016002)(6666004)(186003)(110136005)(316002)(36860700001)(86362001)(1076003)(8936002)(8676002)(4326008)(47076005)(107886003)(82310400003)(336012)(2906002)(426003)(54906003)(82740400003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:30:19.0265 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 23b4f050-48a3-4dd9-898a-08d960d32301 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.35]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT047.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR1201MB0219 Subject: [dpdk-stable] [PATCH 19.11 5/6] common/mlx5: use new port query API if available X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Sender: "stable" From: Viacheslav Ovsiienko [ upstream commit d0cf77e8c2b64319057f5f629a7a595ce6e8b556 ] In order to get E-Switch vport identifiers the mlx5 PMD relies on two approaches: [a] use port query API if it is provided by rdma-core library [b] otherwise, deduce vport ids from the related VF index The latter is not reliable and may not work with newer kernel drivers and in some configurations (LAG), causing E-Switch malfunction. Hence, engaging the port query API is highly desirable. Depending on rdma-core version the port query API is: - very old OFED versions have no query API (approach [b]) - rdma-core OFED < 5.5 provides mlx5dv_query_devx_port, HAVE_MLX5DV_DR_DEVX_PORT flag is defined (approach [a]) - rdma-core OFED >= 5.5 has mlx5dv_query_port, flag HAVE_MLX5DV_DR_DEVX_PORT_V35 is defined (approach [a]) - future OFED versions might remove mlx5dv_query_devx_port and HAVE_MLX5DV_DR_DEVX_PORT will not be defined - Upstream rdma-core < v35 has no port query API (approach [b]) - Upstream rdma-core >= v35 has mlx5dv_query_port, flag HAVE_MLX5DV_DR_DEVX_PORT_V35 is defined (approach [a]) In order to support the new mlx5dv_query_port routine, the conditional compilation flag HAVE_MLX5DV_DR_DEVX_PORT_V35 is introduced by this patch. The flag HAVE_MLX5DV_DR_DEVX_PORT is kept for compatibility with previous rdma-core versions. Despite this patch is not a bugfix (it follows the introduced API variation in underlying library), it resolves the compatibility issue and is highly desired to be ported to DPDK LTS. Cc: stable@dpdk.org Signed-off-by: Viacheslav Ovsiienko Acked-by: Matan Azrad Signed-off-by: Bing Zhao --- drivers/net/mlx5/Makefile | 5 +++ drivers/net/mlx5/meson.build | 2 ++ drivers/net/mlx5/mlx5.c | 60 ++++++++++++++++-------------------- drivers/net/mlx5/mlx5_glue.c | 55 +++++++++++++++++++++++++++------ drivers/net/mlx5/mlx5_glue.h | 16 +++++++++- 5 files changed, 94 insertions(+), 44 deletions(-) diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 3719f0f11e..d62bfd6875 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -178,6 +178,11 @@ mlx5_autoconf.h.new: $(RTE_SDK)/buildtools/auto-config-h.sh infiniband/mlx5dv.h \ func mlx5dv_query_devx_port \ $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_MLX5DV_DR_DEVX_PORT_V35 \ + infiniband/mlx5dv.h \ + func mlx5dv_query_port \ + $(AUTOCONF_OUTPUT) $Q sh -- '$<' '$@' \ HAVE_MLX5DV_DR_CREATE_DEST_IB_PORT \ infiniband/mlx5dv.h \ diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build index 3336c58ce5..81126d6a1b 100644 --- a/drivers/net/mlx5/meson.build +++ b/drivers/net/mlx5/meson.build @@ -138,6 +138,8 @@ if build 'IBV_WQ_FLAG_RX_END_PADDING' ], [ 'HAVE_MLX5DV_DR_DEVX_PORT', 'infiniband/mlx5dv.h', 'mlx5dv_query_devx_port' ], + [ 'HAVE_MLX5DV_DR_DEVX_PORT_V35', 'infiniband/mlx5dv.h', + 'mlx5dv_query_port' ], [ 'HAVE_MLX5DV_DR_CREATE_DEST_IB_PORT', 'infiniband/mlx5dv.h', 'mlx5dv_dr_action_create_dest_ib_port' ], [ 'HAVE_IBV_DEVX_OBJ', 'infiniband/mlx5dv.h', diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 4696a1f2d1..18fccecf3d 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -2175,9 +2175,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, int own_domain_id = 0; uint16_t port_id; unsigned int i; -#ifdef HAVE_MLX5DV_DR_DEVX_PORT - struct mlx5dv_devx_port devx_port = { .comp_mask = 0 }; -#endif + struct mlx5_port_info vport_info = { .query_flags = 0 }; /* Determine if this port representor is supposed to be spawned. */ if (switch_info->representor && dpdk_dev->devargs) { @@ -2411,28 +2409,26 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, priv->vport_meta_tag = 0; priv->vport_meta_mask = 0; priv->pf_bond = spawn->pf_bond; -#ifdef HAVE_MLX5DV_DR_DEVX_PORT /* - * The DevX port query API is implemented. E-Switch may use - * either vport or reg_c[0] metadata register to match on - * vport index. The engaged part of metadata register is - * defined by mask. + * If we have E-Switch we should determine the vport attributes. + * E-Switch may use either source vport field or reg_c[0] metadata + * register to match on vport index. The engaged part of metadata + * register is defined by mask. */ if (switch_info->representor || switch_info->master) { - devx_port.comp_mask = MLX5DV_DEVX_PORT_VPORT | - MLX5DV_DEVX_PORT_MATCH_REG_C_0; - err = mlx5_glue->devx_port_query(sh->ctx, spawn->ibv_port, - &devx_port); + err = mlx5_glue->devx_port_query(sh->ctx, + spawn->ibv_port, + &vport_info); if (err) { DRV_LOG(WARNING, "can't query devx port %d on device %s", spawn->ibv_port, spawn->ibv_dev->name); - devx_port.comp_mask = 0; + vport_info.query_flags = 0; } } - if (devx_port.comp_mask & MLX5DV_DEVX_PORT_MATCH_REG_C_0) { - priv->vport_meta_tag = devx_port.reg_c_0.value; - priv->vport_meta_mask = devx_port.reg_c_0.mask; + if (vport_info.query_flags & MLX5_PORT_QUERY_REG_C0) { + priv->vport_meta_tag = vport_info.vport_meta_tag; + priv->vport_meta_mask = vport_info.vport_meta_mask; if (!priv->vport_meta_mask) { DRV_LOG(ERR, "vport zero mask for port %d" " on bonding device %s", @@ -2448,8 +2444,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, goto error; } } - if (devx_port.comp_mask & MLX5DV_DEVX_PORT_VPORT) { - priv->vport_id = devx_port.vport_num; + if (vport_info.query_flags & MLX5_PORT_QUERY_VPORT) { + priv->vport_id = vport_info.vport_id; } else if (spawn->pf_bond >= 0 && (switch_info->representor || switch_info->master)) { DRV_LOG(ERR, "can't deduce vport index for port %d" @@ -2458,25 +2454,21 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, err = ENOTSUP; goto error; } else { - /* Suppose vport index in compatible way. */ + /* + * Suppose vport index in compatible way. Kernel/rdma_core + * support single E-Switch per PF configurations only and + * vport_id field contains the vport index for associated VF, + * which is deduced from representor port name. + * For example, let's have the IB device port 10, it has + * attached network device eth0, which has port name attribute + * pf0vf2, we can deduce the VF number as 2, and set vport index + * as 3 (2+1). This assigning schema should be changed if the + * multiple E-Switch instances per PF configurations or/and PCI + * subfunctions are added. + */ priv->vport_id = switch_info->representor ? switch_info->port_name + 1 : -1; } -#else - /* - * Kernel/rdma_core support single E-Switch per PF configurations - * only and vport_id field contains the vport index for - * associated VF, which is deduced from representor port name. - * For example, let's have the IB device port 10, it has - * attached network device eth0, which has port name attribute - * pf0vf2, we can deduce the VF number as 2, and set vport index - * as 3 (2+1). This assigning schema should be changed if the - * multiple E-Switch instances per PF configurations or/and PCI - * subfunctions are added. - */ - priv->vport_id = switch_info->representor ? - switch_info->port_name + 1 : -1; -#endif /* representor_id field keeps the unmodified VF index. */ priv->representor_id = switch_info->representor ? switch_info->port_name : -1; diff --git a/drivers/net/mlx5/mlx5_glue.c b/drivers/net/mlx5/mlx5_glue.c index 1553a9b41c..33f604364a 100644 --- a/drivers/net/mlx5/mlx5_glue.c +++ b/drivers/net/mlx5/mlx5_glue.c @@ -1025,17 +1025,54 @@ mlx5_glue_devx_qp_query(struct ibv_qp *qp, static int mlx5_glue_devx_port_query(struct ibv_context *ctx, uint32_t port_num, - struct mlx5dv_devx_port *mlx5_devx_port) -{ + struct mlx5_port_info *info) +{ + int err = 0; + + info->query_flags = 0; +#ifdef HAVE_MLX5DV_DR_DEVX_PORT_V35 + /* The DevX port query API is implemented (rdma-core v35 and above). */ + struct mlx5_ib_uapi_query_port devx_port; + + memset(&devx_port, 0, sizeof(devx_port)); + err = mlx5dv_query_port(ctx, port_num, &devx_port); + if (err) + return err; + if (devx_port.flags & MLX5DV_QUERY_PORT_VPORT_REG_C0) { + info->vport_meta_tag = devx_port.reg_c0.value; + info->vport_meta_mask = devx_port.reg_c0.mask; + info->query_flags |= MLX5_PORT_QUERY_REG_C0; + } + if (devx_port.flags & MLX5DV_QUERY_PORT_VPORT) { + info->vport_id = devx_port.vport; + info->query_flags |= MLX5_PORT_QUERY_VPORT; + } +#else #ifdef HAVE_MLX5DV_DR_DEVX_PORT - return mlx5dv_query_devx_port(ctx, port_num, mlx5_devx_port); + /* The legacy DevX port query API is implemented (prior v35). */ + struct mlx5dv_devx_port devx_port = { + .comp_mask = MLX5DV_DEVX_PORT_VPORT | + MLX5DV_DEVX_PORT_MATCH_REG_C_0 + }; + + err = mlx5dv_query_devx_port(ctx, port_num, &devx_port); + if (err) + return err; + if (devx_port.comp_mask & MLX5DV_DEVX_PORT_MATCH_REG_C_0) { + info->vport_meta_tag = devx_port.reg_c_0.value; + info->vport_meta_mask = devx_port.reg_c_0.mask; + info->query_flags |= MLX5_PORT_QUERY_REG_C0; + } + if (devx_port.comp_mask & MLX5DV_DEVX_PORT_VPORT) { + info->vport_id = devx_port.vport_num; + info->query_flags |= MLX5_PORT_QUERY_VPORT; + } #else - (void)ctx; - (void)port_num; - (void)mlx5_devx_port; - errno = ENOTSUP; - return errno; -#endif + RTE_SET_USED(ctx); + RTE_SET_USED(port_num); +#endif /* HAVE_MLX5DV_DR_DEVX_PORT */ +#endif /* HAVE_MLX5DV_DR_DEVX_PORT_V35 */ + return err; } alignas(RTE_CACHE_LINE_SIZE) diff --git a/drivers/net/mlx5/mlx5_glue.h b/drivers/net/mlx5/mlx5_glue.h index 9895e55974..a639ab28b0 100644 --- a/drivers/net/mlx5/mlx5_glue.h +++ b/drivers/net/mlx5/mlx5_glue.h @@ -81,6 +81,20 @@ struct mlx5dv_dr_domain; struct mlx5dv_devx_port; #endif +#ifndef HAVE_MLX5DV_DR_DEVX_PORT_V35 +struct mlx5dv_port; +#endif + +#define MLX5_PORT_QUERY_VPORT (1u << 0) +#define MLX5_PORT_QUERY_REG_C0 (1u << 1) + +struct mlx5_port_info { + uint16_t query_flags; + uint16_t vport_id; /* Associated VF vport index (if any). */ + uint32_t vport_meta_tag; /* Used for vport index match ove VF LAG. */ + uint32_t vport_meta_mask; /* Used for vport index field match mask. */ +}; + #ifndef HAVE_MLX5_DR_CREATE_ACTION_FLOW_METER struct mlx5dv_dr_flow_meter_attr; #endif @@ -255,7 +269,7 @@ struct mlx5_glue { void *out, size_t outlen); int (*devx_port_query)(struct ibv_context *ctx, uint32_t port_num, - struct mlx5dv_devx_port *mlx5_devx_port); + struct mlx5_port_info *info); }; extern const struct mlx5_glue *mlx5_glue; -- 2.21.0