From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A6C39A0C51; Wed, 21 Jul 2021 16:38:58 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3B7A44111E; Wed, 21 Jul 2021 16:38:46 +0200 (CEST) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2066.outbound.protection.outlook.com [40.107.244.66]) by mails.dpdk.org (Postfix) with ESMTP id F3BAA40DF8 for ; Wed, 21 Jul 2021 16:38:42 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nCKFeKLuDdnZH9gLQ2dHTRKrLFjVgMJRJnHQs8oPei7Ut1iLoyEP2gaGSXZziHK86FKYv8N+OvZkoFh4HkwpmOjn6B+HQjnXejKvnVGCYMCZgSLLITBLvQwIh5hITCWpgnD0+jgr9IZ8EzxO5I3PuRR1TJoT6RagMgU8PF3eBvFO7HXkIBk1lrvq7ZdNjPozRkzV88dpVfraR4ebpq5h5NsRKiQQltjzxP6K6vBZKXS8Jvr6zA9mu7ZshZ3o3gk2yqsUPYcs+hsXIztFrfsrZD9BQA4sf0kkGH+gG+kCLL0G8+fu1YmgMNNy36K0OgIpXk6i+AVbRL1NlmHu7ftvhA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VIPU6E8K1nfAfHxs8kHhuvhM2tbvFEDNvDBaZ0dGVCk=; b=Alx2bmOj42HoDVNMzByhf0MOMyAeJJzXq1m4p0BY007aJVOBMIpXAk3rhRrUIci7CFiI2mjf0Nzp5gFsUdFPF48LKASGU1XoUAPKteqQbk5RtqQAT40DD+aqrceKAKbdKejlHWMgIAmdfiX4Kdo0wOAkCJLmEMzMfKjO7uqhDjq8kGw6uhmIIopTcq67FJ5DIqSVtTdffjeKg2FEGTpRQBaIb2M1mB8MFiL8fRx16R7zKLfBqA3Inq111YDkLq1w0y1OjDz0ddQ5i6JPr1fmAhOmDNnYjTPDEpntvtqsg6503EPV1vgyaP/HWPhMNoT4uQalhNGWLEnHXBx5KXAoMg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=ashroe.eu smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VIPU6E8K1nfAfHxs8kHhuvhM2tbvFEDNvDBaZ0dGVCk=; b=nJZAszytakTmueMvdfG7kS8ICvgzUQNVLCWD0woojrid9bwqyrP0VcR3+2AAM8LSOPsnG26wBk9lCr+Sk7YHSj9re37SKZX35gtzeMmsZvrpXEHG2yUAZRJD9PS6JqbsfaMd65sgv8n45c0EqhwNSAhEMJKarwaZePCnvluy4ForoWYuoyulRHqRQCem+p0UdPyYEiJtj5wrho7TynUNEKTnwgFw7VQmXGW44stqaV1jlt3udu9o+XhNi3v4DmHCBtvLOTfogYEb8gXy2SJw2062a5azMAc6k/4aJQ+El7SpPgq30eyP0vMEowWLJO3a1rIBgYcsfv5bkN3bUNHs1Q== Received: from BN9PR03CA0658.namprd03.prod.outlook.com (2603:10b6:408:13b::33) by CY4PR1201MB2503.namprd12.prod.outlook.com (2603:10b6:903:d1::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.29; Wed, 21 Jul 2021 14:38:40 +0000 Received: from BN8NAM11FT019.eop-nam11.prod.protection.outlook.com (2603:10b6:408:13b:cafe::73) by BN9PR03CA0658.outlook.office365.com (2603:10b6:408:13b::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4352.24 via Frontend Transport; Wed, 21 Jul 2021 14:38:40 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; ashroe.eu; dkim=none (message not signed) header.d=none;ashroe.eu; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT019.mail.protection.outlook.com (10.13.176.158) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Wed, 21 Jul 2021 14:38:40 +0000 Received: from nvidia.com (172.20.187.5) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 21 Jul 2021 14:38:38 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: , , Matan Azrad , Shahaf Shuler , Ray Kinsella Date: Wed, 21 Jul 2021 22:37:31 +0800 Message-ID: <20210721143743.24626-5-xuemingl@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210721143743.24626-1-xuemingl@nvidia.com> References: <20210616040935.311733-1-xuemingl@nvidia.com> <20210721143743.24626-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL105.nvidia.com (172.20.187.12) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3f8a77e6-44a3-4c04-8a94-08d94c553b9a X-MS-TrafficTypeDiagnostic: CY4PR1201MB2503: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:147; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: uYXNVxIWqlxMNg3heN5WrrqfhSkHQP+d+Ztl7ZoSK98NboZ8gI3zUAItMOr06GlUuTESzsEMs1vFAS5HfNbpxVbdzzxYGMnvnm8AqkitOO/mup1PTTHPzmiq0bchaVCQbmW/IZwYRc9cEyzYLEN2M6I3P3zaYG/6oZzeRTNB1TCnx4rJaYHtAzVChfv4SPIm9RGU85NP3X0ltGTjw/HXeNwPw0H5P7MJYr1GLlG/acpfnoSRxUuMaCF3g17OzZFteHzAKlRxQ8VnOWOhFRdaXrksWdrIP0QiCaEUHsXzga5qzu7TOFPVvmN704p8XuRx5NnWyO4hV/Hz4rAIiJhA8ZEueplM8Fy6LZlEYjgSeqD3hZ3xL6qh3DAM+Jzoe/Kn/0TrLFoutXl/VtSyAEtU0X7BrIDd3nTUw5Q9nQp44o9YHr0IOoFSCivzwtk4RJ4dYSlYS7J3DANgD2Os+giwj5KmuFpfRqo4u5OfGzB72Kd21ePis3jPJG1UkIEUTOFqOb83D3gsJdTcDw8fTJCS5vDWYpts2OwzNQf+328hyXUFSblxLXv3TY+iGCxssXsdLpF6lk7O+Sln62FmyXLqHq7Y8ZeRTypEKfo2iPdBtuLlE4oCwO5HEsjaO0DCbsSR0oAup9pMulHtQAFrb+5psZpf2Cm7gyYVH5vnru8FZOVvc8XuxmzPGN3MGXtbGnLYwqOIqcZlthZ90iNSEi1hCw== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(346002)(396003)(136003)(376002)(39860400002)(46966006)(36840700001)(2616005)(36860700001)(70206006)(16526019)(82740400003)(7696005)(7636003)(8676002)(26005)(336012)(8936002)(6666004)(37006003)(82310400003)(186003)(2906002)(86362001)(356005)(478600001)(6286002)(47076005)(4326008)(36906005)(6862004)(70586007)(426003)(5660300002)(55016002)(36756003)(1076003)(316002)(54906003)(83380400001)(6636002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jul 2021 14:38:40.3931 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3f8a77e6-44a3-4c04-8a94-08d94c553b9a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT019.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR1201MB2503 Subject: [dpdk-dev] [PATCH v4 04/16] common/mlx5: support auxiliary bus X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch adds auxiliary bus driver and delegate to registered internal mlx5 common device drivers, i.e. eth, vdpa... Current major target is to support SubFunction on auxiliary bus. As a limitation of current driver, numa node of device is detected from PCI bus of device symbol link, will remove once numa node file available on sysfs. Signed-off-by: Xueming Li Acked-by: Viacheslav Ovsiienko --- drivers/common/mlx5/linux/meson.build | 3 + .../common/mlx5/linux/mlx5_common_auxiliary.c | 173 ++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_verbs.c | 5 +- drivers/common/mlx5/meson.build | 2 +- drivers/common/mlx5/mlx5_common.c | 3 + drivers/common/mlx5/mlx5_common.h | 6 + drivers/common/mlx5/mlx5_common_private.h | 6 + drivers/common/mlx5/version.map | 2 + 8 files changed, 198 insertions(+), 2 deletions(-) create mode 100644 drivers/common/mlx5/linux/mlx5_common_auxiliary.c diff --git a/drivers/common/mlx5/linux/meson.build b/drivers/common/mlx5/linux/meson.build index 686df26909..44446e2743 100644 --- a/drivers/common/mlx5/linux/meson.build +++ b/drivers/common/mlx5/linux/meson.build @@ -48,10 +48,13 @@ endif sources += files('mlx5_nl.c') sources += files('mlx5_common_os.c') sources += files('mlx5_common_verbs.c') +sources += files('mlx5_common_auxiliary.c') if not dlopen_ibverbs sources += files('mlx5_glue.c') endif +deps += ['bus_auxiliary'] + # To maintain the compatibility with the make build system # mlx5_autoconf.h file is still generated. # input array for meson member search: diff --git a/drivers/common/mlx5/linux/mlx5_common_auxiliary.c b/drivers/common/mlx5/linux/mlx5_common_auxiliary.c new file mode 100644 index 0000000000..4ca27cd281 --- /dev/null +++ b/drivers/common/mlx5/linux/mlx5_common_auxiliary.c @@ -0,0 +1,173 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies Ltd + */ + +#include +#include +#include +#include +#include +#include +#include "eal_filesystem.h" + +#include "mlx5_common_utils.h" +#include "mlx5_common_private.h" + +#define AUXILIARY_SYSFS_PATH "/sys/bus/auxiliary/devices" +#define MLX5_AUXILIARY_PREFIX "mlx5_core.sf." + +int +mlx5_auxiliary_get_child_name(const char *dev, const char *node, + char *child, size_t size) +{ + DIR *dir; + struct dirent *dent; + MKSTR(path, "%s/%s%s", AUXILIARY_SYSFS_PATH, dev, node); + + dir = opendir(path); + if (dir == NULL) { + rte_errno = errno; + return -rte_errno; + } + /* Get the first file name. */ + while ((dent = readdir(dir)) != NULL) { + if (dent->d_name[0] != '.') + break; + } + closedir(dir); + if (dent == NULL) { + rte_errno = ENOENT; + return -rte_errno; + } + if (rte_strscpy(child, dent->d_name, size) < 0) + return -rte_errno; + return 0; +} + +static int +mlx5_auxiliary_get_pci_path(const struct rte_auxiliary_device *dev, + char *sysfs_pci, size_t size) +{ + char sysfs_real[PATH_MAX] = { 0 }; + MKSTR(sysfs_aux, "%s/%s", AUXILIARY_SYSFS_PATH, dev->name); + char *dir; + + if (realpath(sysfs_aux, sysfs_real) == NULL) { + rte_errno = errno; + return -rte_errno; + } + dir = dirname(sysfs_real); + if (dir == NULL) { + rte_errno = errno; + return -rte_errno; + } + if (rte_strscpy(sysfs_pci, dir, size) < 0) + return -rte_errno; + return 0; +} + +static int +mlx5_auxiliary_get_numa(const struct rte_auxiliary_device *dev) +{ + unsigned long numa; + char numa_path[PATH_MAX]; + + if (mlx5_auxiliary_get_pci_path(dev, numa_path, sizeof(numa_path)) != 0) + return SOCKET_ID_ANY; + if (strcat(numa_path, "/numa_node") == NULL) { + rte_errno = ENAMETOOLONG; + return SOCKET_ID_ANY; + } + if (eal_parse_sysfs_value(numa_path, &numa) != 0) { + rte_errno = EINVAL; + return SOCKET_ID_ANY; + } + return (int)numa; +} + +struct ibv_device * +mlx5_get_aux_ibv_device(const struct rte_auxiliary_device *dev) +{ + int n; + char ib_name[64] = { 0 }; + struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n); + struct ibv_device *ibv_match = NULL; + + if (!ibv_list) { + rte_errno = ENOSYS; + return NULL; + } + if (mlx5_auxiliary_get_child_name(dev->name, "/infiniband", + ib_name, sizeof(ib_name)) != 0) + goto out; + while (n-- > 0) { + if (strcmp(ibv_list[n]->name, ib_name) != 0) + continue; + ibv_match = ibv_list[n]; + break; + } + if (ibv_match == NULL) + rte_errno = ENOENT; +out: + mlx5_glue->free_device_list(ibv_list); + return ibv_match; +} + +static bool +mlx5_common_auxiliary_match(const char *name) +{ + return strncmp(name, MLX5_AUXILIARY_PREFIX, + strlen(MLX5_AUXILIARY_PREFIX)) == 0; +} + +static int +mlx5_common_auxiliary_probe(struct rte_auxiliary_driver *drv __rte_unused, + struct rte_auxiliary_device *dev) +{ + dev->device.numa_node = mlx5_auxiliary_get_numa(dev); + return mlx5_common_dev_probe(&dev->device); +} + +static int +mlx5_common_auxiliary_remove(struct rte_auxiliary_device *auxiliary_dev) +{ + return mlx5_common_dev_remove(&auxiliary_dev->device); +} + +static int +mlx5_common_auxiliary_dma_map(struct rte_auxiliary_device *auxiliary_dev, + void *addr, uint64_t iova, size_t len) +{ + return mlx5_common_dev_dma_map(&auxiliary_dev->device, addr, iova, len); +} + +static int +mlx5_common_auxiliary_dma_unmap(struct rte_auxiliary_device *auxiliary_dev, + void *addr, uint64_t iova, size_t len) +{ + return mlx5_common_dev_dma_unmap(&auxiliary_dev->device, addr, iova, + len); +} + +static struct rte_auxiliary_driver mlx5_auxiliary_driver = { + .driver = { + .name = MLX5_AUXILIARY_DRIVER_NAME, + }, + .match = mlx5_common_auxiliary_match, + .probe = mlx5_common_auxiliary_probe, + .remove = mlx5_common_auxiliary_remove, + .dma_map = mlx5_common_auxiliary_dma_map, + .dma_unmap = mlx5_common_auxiliary_dma_unmap, +}; + +void mlx5_common_auxiliary_init(void) +{ + if (mlx5_auxiliary_driver.bus == NULL) + rte_auxiliary_register(&mlx5_auxiliary_driver); +} + +RTE_FINI(mlx5_common_auxiliary_driver_finish) +{ + if (mlx5_auxiliary_driver.bus != NULL) + rte_auxiliary_unregister(&mlx5_auxiliary_driver); +} diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c index 6a6ab7a7a2..9080bd3e87 100644 --- a/drivers/common/mlx5/linux/mlx5_common_verbs.c +++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c @@ -12,6 +12,7 @@ #include #include +#include #include "mlx5_common_utils.h" #include "mlx5_common_log.h" @@ -24,10 +25,12 @@ struct ibv_device * mlx5_os_get_ibv_dev(const struct rte_device *dev) { - struct ibv_device *ibv = NULL; + struct ibv_device *ibv; if (mlx5_dev_is_pci(dev)) ibv = mlx5_os_get_ibv_device(&RTE_DEV_TO_PCI_CONST(dev)->addr); + else + ibv = mlx5_get_aux_ibv_device(RTE_DEV_TO_AUXILIARY_CONST(dev)); if (ibv == NULL) { rte_errno = ENODEV; DRV_LOG(ERR, "Verbs device not found: %s", dev->name); diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build index fa2b8b9834..6ddbde7e8f 100644 --- a/drivers/common/mlx5/meson.build +++ b/drivers/common/mlx5/meson.build @@ -7,7 +7,7 @@ if not (is_linux or (is_windows and is_ms_linker)) subdir_done() endif -deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs'] +deps += ['hash', 'pci', 'bus_pci', 'bus_auxiliary', 'net', 'eal', 'kvargs'] sources += files( 'mlx5_devx_cmds.c', 'mlx5_common.c', diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 97d3e3e60e..94f858a9bd 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -398,6 +398,9 @@ mlx5_class_driver_register(struct mlx5_class_driver *driver) static void mlx5_common_driver_init(void) { mlx5_common_pci_init(); +#ifdef RTE_EXEC_ENV_LINUX + mlx5_common_auxiliary_init(); +#endif } static bool mlx5_common_initialized; diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 1b811f2509..3cb961f495 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -24,6 +24,7 @@ /* Reported driver name. */ #define MLX5_PCI_DRIVER_NAME "mlx5_pci" +#define MLX5_AUXILIARY_DRIVER_NAME "mlx5_auxiliary" /* Bit-field manipulation. */ #define BITFIELD_DECLARE(bf, type, size) \ @@ -109,6 +110,7 @@ pmd_drv_log_basename(const char *s) int mkstr_size_##name = snprintf(NULL, 0, "" __VA_ARGS__); \ char name[mkstr_size_##name + 1]; \ \ + memset(name, 0, mkstr_size_##name + 1); \ snprintf(name, sizeof(name), "" __VA_ARGS__) enum { @@ -136,6 +138,10 @@ enum { PCI_DEVICE_ID_MELLANOX_CONNECTX7BF = 0Xa2dc, }; + +__rte_internal +int mlx5_auxiliary_get_child_name(const char *dev, const char *node, + char *child, size_t size); /* Maximum number of simultaneous unicast MAC addresses. */ #define MLX5_MAX_UC_MAC_ADDRESSES 128 /* Maximum number of simultaneous Multicast MAC addresses. */ diff --git a/drivers/common/mlx5/mlx5_common_private.h b/drivers/common/mlx5/mlx5_common_private.h index 791eb3cd77..9f00a6c54d 100644 --- a/drivers/common/mlx5/mlx5_common_private.h +++ b/drivers/common/mlx5/mlx5_common_private.h @@ -6,6 +6,7 @@ #define _MLX5_COMMON_PRIVATE_H_ #include +#include #include "mlx5_common.h" @@ -34,6 +35,11 @@ void mlx5_common_driver_on_register_pci(struct mlx5_class_driver *driver); bool mlx5_dev_pci_match(const struct mlx5_class_driver *drv, const struct rte_device *dev); +/* Common auxiliary bus driver: */ +void mlx5_common_auxiliary_init(void); +struct ibv_device *mlx5_get_aux_ibv_device( + const struct rte_auxiliary_device *dev); + #ifdef __cplusplus } #endif /* __cplusplus */ diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index e9d43dc1e5..b47d73b425 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -3,6 +3,8 @@ INTERNAL { haswell_broadwell_cpu; + mlx5_auxiliary_get_child_name; # WINDOWS_NO_EXPORT + mlx5_class_driver_register; mlx5_common_init; -- 2.25.1