From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D9080A0548; Thu, 27 May 2021 16:02:42 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DE22F41104; Thu, 27 May 2021 16:02:33 +0200 (CEST) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2087.outbound.protection.outlook.com [40.107.94.87]) by mails.dpdk.org (Postfix) with ESMTP id 87AE8410F5 for ; Thu, 27 May 2021 16:02:32 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KseoOUDi/STLg1iNtJUBLsR3yzdNphdTFk4AuwHo5LNMC67ictG2vOUfTfO0YzgGJCb7RVulsvdhRWbV/L9Ho904jT95/YgHraHkM/s4bDQEYNXK6qSAxiJDWAnRj/Ob7rNtRnqyhYDxsq85iAZzf5kEQZ52TKShpIw5u3X2Z25hqFX+A8P5QIXDhV8fGkeRvADRCtYh0FfiMQ5H4AWkTK+DeQpOqBi+iCRvW/Ty2L7w5WN2kRCBPfUyiC5Bcz9x+ObTBtvz0bPE1uGEImATUJ3ZIfKen6IbIOtirLcA+23ck4HQgGefd1TBwvKQ4eN1ydKk86ptNFdRRqFIOv7H8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DwSiLrJzpvEkEGLRsG6xR5DZJwMQCybhN/AeQg3Ykrk=; b=RYdKWYcabFv0F5uleeReLCSZGOmZK6fZ+sKUX1KBVZgr3qoKcML8sbROK2kX77I0f9cQVk/TLLvIDjsjSTqFKazhjOior3+L66pfyMITwUDf4l0kF6LYklK1PKMhmFQ1/UJCAP94bk0mTtLvABzIgYoaChjP7Vb4SJn4P5pvezwdpBVXYh2bnsIYsfCqhBaTx6yOLh2M6iDPWXCeFqqZj8j6AxqDiLymceuyq5s3O18VaGZbM3P7fZd+beBhMDe7p1D8acHsoUhY1cQcA3hIl7IiRC+ThEOmL21M3/jNp2mX6yVL6ZReQyfqAcLND1CGccH5nJaODluDgx6ULY79AQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=tuxdriver.com smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DwSiLrJzpvEkEGLRsG6xR5DZJwMQCybhN/AeQg3Ykrk=; b=Hw0u+3uBl4aHJkR90kLa85bR9KEBXcgtFVwtANN1Viyy/lToYZL/XH/cWRiPKh9RITVKN29dQCIEVkDQly123agKQpoqCECerxfaUgVFG1oAbKrlNgFUB1Sq4nr4AFC39xjQOQd4tbMiZ+QrXglqyJiTGx5kuTZx1RqFIHgTOx4og4Op5a3gDJRV4khHBGpXX7sHFfU2Fj86gYpVFJH25zVFp6D79dRIBoQX8tgN5lGleVY7mws06pv6qnNaTQAiSzymnMVhh2RjhCDuUI9j0Y2hQwId2EvH4gcfs2KvsBQfpH7iyXBOFP4C/iMaFl28ZYsr1OP/B6XBCA/lqqtvRw== Received: from BN1PR10CA0008.namprd10.prod.outlook.com (2603:10b6:408:e0::13) by DM5PR1201MB2474.namprd12.prod.outlook.com (2603:10b6:3:e3::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4150.26; Thu, 27 May 2021 14:02:30 +0000 Received: from BN8NAM11FT039.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e0:cafe::bf) by BN1PR10CA0008.outlook.office365.com (2603:10b6:408:e0::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4173.20 via Frontend Transport; Thu, 27 May 2021 14:02:30 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; tuxdriver.com; dkim=none (message not signed) header.d=none;tuxdriver.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT039.mail.protection.outlook.com (10.13.177.169) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4129.25 via Frontend Transport; Thu, 27 May 2021 14:02:30 +0000 Received: from nvidia.com (172.20.145.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 27 May 2021 14:02:19 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: , , Matan Azrad , Shahaf Shuler , Ray Kinsella , Neil Horman Date: Thu, 27 May 2021 17:01:59 +0300 Message-ID: <20210527140202.19377-2-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210527140202.19377-1-xuemingl@nvidia.com> References: <20210527133759.17401-1-xuemingl@nvidia.com> <20210527140202.19377-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.145.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b6fd0571-37ac-4b36-0ac2-08d92118118f X-MS-TrafficTypeDiagnostic: DM5PR1201MB2474: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:147; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LQU4OJOBDQlNxJ0/jSv2XtufZt9BNYgsMIip17/EU7IeFbv/6gbKodyLdBXDVhA7ryiVHPV3Orz2y2wsPMjJojyJ9Y3mlO288pJKR3RtF8lKU1efyLokBtaMTcVKyShjOOqkIghbK+7IV2I+42TTjyjqxUFPGE4ACHyFQ4mTW3i5ao1tWHdU5C9hqwxwWUtUvIhRn0xwDg4baVD9wptUi8K1LJtaHT1gfPBo+2xcsvuahM+jjAQm7+ps0mwrRjIMI/fTJL0RypkLPvvHn3losADTmKftZShi6pbZ0UbxVx8K/bGIQ9ohyo+IAqP9Z4pToQMrGy+fASNeL9uc+u3UVt3x650g7h1pSZ3o1sa2rRDBq3wNrLAN9mmDkl9RRO770tmFDV0mlpy4CKcgB1c50z6N0t0CIKA5kALGxaNdqdHBDXJv/T56OpFNKyh/eus1rlnGmHt7Jud7Cu9rY0tYEhYNbGv/4r3Kbnk1tNrlKoaUTLsvIj0lbWH/pB0RcX5RtXtf1NasRSwJeRyKrIXQQbPqzRaPhswQDhFu8YpMv7roL4NmO9UWxFMJPY80WBxUHDBXNTxNdcr+A4pV1HEsw4NbhZVWusIOKLy6BI3jsDkv0/HBXl3+r6LFekH2ln9abkrhewzW3mXMaDviOWMlSngVgY2Tc9rFHCZjR0SdHkI= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(346002)(39860400002)(136003)(376002)(396003)(36840700001)(46966006)(70206006)(1076003)(4326008)(6636002)(8676002)(426003)(83380400001)(70586007)(5660300002)(316002)(6666004)(54906003)(2616005)(478600001)(36756003)(26005)(82310400003)(7696005)(36906005)(8936002)(356005)(336012)(55016002)(7636003)(86362001)(82740400003)(6862004)(36860700001)(47076005)(186003)(16526019)(37006003)(6286002)(2906002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 May 2021 14:02:30.5245 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b6fd0571-37ac-4b36-0ac2-08d92118118f X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT039.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR1201MB2474 Subject: [dpdk-dev] [RFC 11/14] common/mlx5: support auxiliary bus X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch adds auxiliary bus driver and delegate to registered internal mlx5 common device drivers, i.e. eth, vdpa... Current major target is to support SubFunction on auxiliary bus. As a limitation of current driver, numa node of device is detected from PCI bus of device symbol link, will remove once numa node file available on sysfs. Signed-off-by: Xueming Li --- drivers/common/mlx5/linux/meson.build | 3 + .../common/mlx5/linux/mlx5_common_auxiliary.c | 173 ++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_verbs.c | 5 +- drivers/common/mlx5/mlx5_common.c | 3 + drivers/common/mlx5/mlx5_common.h | 5 + drivers/common/mlx5/mlx5_common_private.h | 5 + drivers/common/mlx5/version.map | 2 + 7 files changed, 195 insertions(+), 1 deletion(-) create mode 100644 drivers/common/mlx5/linux/mlx5_common_auxiliary.c diff --git a/drivers/common/mlx5/linux/meson.build b/drivers/common/mlx5/linux/meson.build index 007834a49b..a1070acb77 100644 --- a/drivers/common/mlx5/linux/meson.build +++ b/drivers/common/mlx5/linux/meson.build @@ -48,10 +48,13 @@ endif sources += files('mlx5_nl.c') sources += files('mlx5_common_os.c') sources += files('mlx5_common_verbs.c') +sources += files('mlx5_common_auxiliary.c') if not dlopen_ibverbs sources += files('mlx5_glue.c') endif +deps += ['bus_auxiliary'] + # To maintain the compatibility with the make build system # mlx5_autoconf.h file is still generated. # input array for meson member search: diff --git a/drivers/common/mlx5/linux/mlx5_common_auxiliary.c b/drivers/common/mlx5/linux/mlx5_common_auxiliary.c new file mode 100644 index 0000000000..f16fd2ee37 --- /dev/null +++ b/drivers/common/mlx5/linux/mlx5_common_auxiliary.c @@ -0,0 +1,173 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies Ltd + */ + +#include +#include +#include +#include +#include +#include +#include "eal_filesystem.h" + +#include "mlx5_common_utils.h" +#include "mlx5_common_private.h" + +#define AUXILIARY_SYSFS_PATH "/sys/bus/auxiliary/devices" +#define MLX5_AUXILIARY_PREFIX "mlx5_core.sf." + +int +mlx5_auxiliary_get_child_name(const char *dev, const char *node, + char *child, size_t size) +{ + DIR *dir; + struct dirent *dent; + MKSTR(path, "%s/%s%s", AUXILIARY_SYSFS_PATH, dev, node); + + dir = opendir(path); + if (dir == NULL) { + rte_errno = errno; + return -rte_errno; + } + /* Get the first file name. */ + while ((dent = readdir(dir)) != NULL) { + if (dent->d_name[0] != '.') + break; + } + closedir(dir); + if (dent == NULL) { + rte_errno = ENOENT; + return -rte_errno; + } + if (rte_strscpy(child, dent->d_name, size) < 0) + return -rte_errno; + return 0; +} + +static int +mlx5_auxiliary_get_pci_path(const struct rte_auxiliary_device *dev, + char *sysfs_pci, size_t size) +{ + char sysfs_real[PATH_MAX]; + char *last_slash; + MKSTR(sysfs_aux, "%s/%s", AUXILIARY_SYSFS_PATH, dev->name); + + if (realpath(sysfs_aux, sysfs_real) == NULL) { + rte_errno = errno; + return -rte_errno; + } + last_slash = strrchr(sysfs_real, '/'); + if (last_slash == NULL) { + rte_errno = EINVAL; + return -rte_errno; + } + *last_slash = '\0'; + if (rte_strscpy(sysfs_pci, sysfs_real, size) < 0) + return -rte_errno; + return 0; +} + +static int +mlx5_auxiliary_get_numa(const struct rte_auxiliary_device *dev) +{ + unsigned long numa; + char numa_path[PATH_MAX]; + + if (mlx5_auxiliary_get_pci_path(dev, numa_path, sizeof(numa_path)) != 0) + return SOCKET_ID_ANY; + if (strcat(numa_path, "/numa_node") == NULL) { + rte_errno = ENAMETOOLONG; + return SOCKET_ID_ANY; + } + if (eal_parse_sysfs_value(numa_path, &numa) != 0) { + rte_errno = EINVAL; + return SOCKET_ID_ANY; + } + return (int)numa; +} + +struct ibv_device * +mlx5_get_aux_ibv_device(const struct rte_auxiliary_device *dev) +{ + int n; + char ib_name[64]; + struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n); + struct ibv_device *ibv_match = NULL; + + if (!ibv_list) { + rte_errno = ENOSYS; + return NULL; + } + if (mlx5_auxiliary_get_child_name(dev->name, "/infiniband", + ib_name, sizeof(ib_name)) != 0) + return NULL; + while (n-- > 0) { + if (strcmp(ibv_list[n]->name, ib_name) != 0) + continue; + ibv_match = ibv_list[n]; + break; + } + if (ibv_match == NULL) + rte_errno = ENOENT; + mlx5_glue->free_device_list(ibv_list); + return ibv_match; +} + +static bool +mlx5_common_auxiliary_match(const char *name) +{ + return strncmp(name, MLX5_AUXILIARY_PREFIX, + strlen(MLX5_AUXILIARY_PREFIX)) == 0; +} + +static int +mlx5_common_auxiliary_probe(struct rte_auxiliary_driver *drv __rte_unused, + struct rte_auxiliary_device *dev) +{ + dev->device.numa_node = mlx5_auxiliary_get_numa(dev); + return mlx5_common_dev_probe(&dev->device); +} + +static int +mlx5_common_auxiliary_remove(struct rte_auxiliary_device *auxiliary_dev) +{ + return mlx5_common_dev_remove(&auxiliary_dev->device); +} + +static int +mlx5_common_auxiliary_dma_map(struct rte_auxiliary_device *auxiliary_dev, + void *addr, uint64_t iova, size_t len) +{ + return mlx5_common_dev_dma_map(&auxiliary_dev->device, addr, iova, len); +} + +static int +mlx5_common_auxiliary_dma_unmap(struct rte_auxiliary_device *auxiliary_dev, + void *addr, uint64_t iova, size_t len) +{ + return mlx5_common_dev_dma_unmap(&auxiliary_dev->device, addr, iova, + len); +} + +static struct rte_auxiliary_driver mlx5_auxiliary_driver = { + .driver = { + .name = MLX5_AUXILIARY_DRIVER_NAME, + }, + .match = mlx5_common_auxiliary_match, + .probe = mlx5_common_auxiliary_probe, + .remove = mlx5_common_auxiliary_remove, + .dma_map = mlx5_common_auxiliary_dma_map, + .dma_unmap = mlx5_common_auxiliary_dma_unmap, +}; + +void mlx5_common_auxiliary_init(void) +{ + if (mlx5_auxiliary_driver.bus == NULL) + rte_auxiliary_register(&mlx5_auxiliary_driver); +} + +RTE_FINI(mlx5_common_auxiliary_driver_finish) +{ + if (mlx5_auxiliary_driver.bus != NULL) + rte_auxiliary_unregister(&mlx5_auxiliary_driver); +} diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c index a49440ef72..856e782878 100644 --- a/drivers/common/mlx5/linux/mlx5_common_verbs.c +++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c @@ -12,6 +12,7 @@ #include #include +#include #include "mlx5_common_log.h" #include "mlx5_common_utils.h" @@ -24,10 +25,12 @@ struct ibv_device * mlx5_get_ibv_device(const struct rte_device *dev) { - struct ibv_device *ibv = NULL; + struct ibv_device *ibv; if (mlx5_dev_is_pci(dev)) ibv = mlx5_get_pci_ibv_device(&RTE_DEV_TO_PCI_CONST(dev)->addr); + else + ibv = mlx5_get_aux_ibv_device(RTE_DEV_TO_AUXILIARY_CONST(dev)); if (ibv == NULL) { rte_errno = ENODEV; DRV_LOG(ERR, "Verbs device not found: %s", dev->name); diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 875668d72b..b7be713cbe 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -356,6 +356,9 @@ mlx5_class_driver_register(struct mlx5_class_driver *driver) static void mlx5_common_driver_init(void) { mlx5_common_pci_init(); +#ifdef RTE_EXEC_ENV_LINUX + mlx5_common_auxiliary_init(); +#endif } static bool mlx5_common_initialized; diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 62a0dc4bad..c9c77ce540 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -23,6 +23,7 @@ /* Reported driver name. */ #define MLX5_PCI_DRIVER_NAME "mlx5_pci" +#define MLX5_AUXILIARY_DRIVER_NAME "mlx5_auxiliary" /* Bit-field manipulation. */ #define BITFIELD_DECLARE(bf, type, size) \ @@ -140,6 +141,10 @@ struct ibv_device; __rte_internal struct ibv_device *mlx5_get_ibv_device(const struct rte_device *dev); +__rte_internal +int mlx5_auxiliary_get_child_name(const char *dev, const char *node, + char *child, size_t size); + /* Maximum number of simultaneous unicast MAC addresses. */ #define MLX5_MAX_UC_MAC_ADDRESSES 128 /* Maximum number of simultaneous Multicast MAC addresses. */ diff --git a/drivers/common/mlx5/mlx5_common_private.h b/drivers/common/mlx5/mlx5_common_private.h index 1beeaae50e..d1ab15ac43 100644 --- a/drivers/common/mlx5/mlx5_common_private.h +++ b/drivers/common/mlx5/mlx5_common_private.h @@ -6,6 +6,7 @@ #define _MLX5_COMMON_PRIVATE_H_ #include +#include #include "mlx5_common.h" @@ -36,6 +37,10 @@ bool mlx5_dev_pci_match(const struct mlx5_class_driver *drv, const struct rte_device *dev); struct ibv_device *mlx5_get_pci_ibv_device(const struct rte_pci_addr *); +/* Common auxiliary bus driver: */ +void mlx5_common_auxiliary_init(void); +struct ibv_device *mlx5_get_aux_ibv_device(const struct rte_auxiliary_device *); + #ifdef __cplusplus } #endif /* __cplusplus */ diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index b10e1c4646..33eb7f09bc 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -3,6 +3,8 @@ INTERNAL { haswell_broadwell_cpu; + mlx5_auxiliary_get_child_name; + mlx5_class_driver_register; mlx5_common_init; -- 2.25.1