From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 63E3BA0C45; Wed, 16 Jun 2021 06:10:17 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BCBD5410F7; Wed, 16 Jun 2021 06:10:00 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2081.outbound.protection.outlook.com [40.107.237.81]) by mails.dpdk.org (Postfix) with ESMTP id F0C07410F3 for ; Wed, 16 Jun 2021 06:09:58 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=odL4aswcbWcd2HFbRP4xD3iBIoalvvjl3qGXb3xTmhWbhugUUgALcYI1CEmHBSnvl+nxLwF60HeMqrFVLePwB9GRJFeDWdEe5/x88nKVjwuXttZAFtEMn7zI7dmMyRSqLEK7KrmzjDR/qS1SUrxOdlM8I4BMYtBNqClzWzPHbOwz5kM4DEV6h3pm85Bg8A7zv7y/8KQXSoDv/w9CQuizxHZ7EKDM7cr5nQ+KEXaCnxi0+7vkNSxkAOqsmqykSlBvpEBgTXmpY1o0Q8ohIC6+q251ey9nov9AspgXTa624jXdjDrnC96tSOn/0mlSIGzNjGe4ffbWDcMtzZDTKieOFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8zbOHQZXtEbLucqxUiAW1JBIwij/Sh/H6oE9ZbQgRk8=; b=k/PaX7wJp0fNw0WNGYaKJqlVfMGaTq3IweorB+MV2RH1OP91yP3bEjLzcLy6eCAQ+rQR05KC+HFdXeYIB3R7SJdIMYFGZSzGHTUThTlPT8tJV/jHo0QK/v6OWSZntq5yp3LUyJt6/wkS8J3/9aJlOEvLv1s2oNb2frN1/d1CVFxR9geRr0gMI+2GepLBjaGg5CU4oMJCV3hvOL6JHwsyWyMCRyTCodZeLQ7RBVVBdsmTaW0tHWAKnSTU4oIGm7VSeDgeK2CvOE9VDINe/4giL7fU019oDvR9D6cRdvBNp/EhjMXcPQzhkZ4HoFyobRa4rKOtNGYI4+TCAOmnXpVLuQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=tuxdriver.com smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8zbOHQZXtEbLucqxUiAW1JBIwij/Sh/H6oE9ZbQgRk8=; b=qXALKLBZcgfiQEYwqMGxXT8f8+pyLt2baqHiFrLHdA/X6RhdLWTgeHuNpkGKpnI2uj8fiite3IQW7u3zfhWlUdl5J40WJu5Ym2Mmn/OVDl7anzFvcw6KumuF+prmzNLSzsJrKj2A5M7hiHJ/jB3sGpwLw67Uql3kOOjRO/tvbUYVt0LPUaemZbTnHoDjjQtQU0pxea21JsTOI+u4KLBMFdD6XfsPTjS+cGxEi4pZlWVy6hUmXX74L7D0RBwCS7G3w+czPhUb3cpqESBEUtLNB9+3lOzN18kQxfNspmEMOEz0P06QuhZPR3OcIIRZMB4rK0uaHc2v0BhH/RpNfE0yVw== Received: from DM6PR03CA0027.namprd03.prod.outlook.com (2603:10b6:5:40::40) by DM6PR12MB4942.namprd12.prod.outlook.com (2603:10b6:5:1be::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.24; Wed, 16 Jun 2021 04:09:57 +0000 Received: from DM6NAM11FT003.eop-nam11.prod.protection.outlook.com (2603:10b6:5:40:cafe::37) by DM6PR03CA0027.outlook.office365.com (2603:10b6:5:40::40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.18 via Frontend Transport; Wed, 16 Jun 2021 04:09:57 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; tuxdriver.com; dkim=none (message not signed) header.d=none;tuxdriver.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT003.mail.protection.outlook.com (10.13.173.162) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:09:57 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 16 Jun 2021 04:09:55 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: , , Matan Azrad , Shahaf Shuler , Ray Kinsella , Neil Horman Date: Wed, 16 Jun 2021 07:09:24 +0300 Message-ID: <20210616040935.311733-4-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210527133759.17401-2-xuemingl@nvidia.com> References: <20210527133759.17401-2-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e3d9d1dd-e260-4374-8c1b-08d9307c9a67 X-MS-TrafficTypeDiagnostic: DM6PR12MB4942: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:147; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: noE8ItcpBQjXmv45YYSXwbtkNoT7zfvCwSXo803gHFQAZPuS/vGvWR7w1kH5MQf5EOfD3Lm3B4Zhy/xPeXabMjO+RlxjWEDJUSDN4RGK7Me7g09TJWcxg/orssKYhRFpXMJWd1UrUlW7r85BnK5I9AJ4+tlEWEbIGZWeFhLY4NZMZy79YyWtoG9fpJUKbBspORYOSIxv6IjWL7eUbSJvEPWvMMzTehf52sbfvZNpLSIKQWQJ6iXWh86Q4/IEtD5ZeiDUQtVWQIGns3ucnBAgzJOuyLWxWD4xZnD6+fl24FyjxSh2ndYxDiNRqIrLKYZjr06xhbIpi+JeZiRtN6hKIof+mlI5awEPtc8BKRTnIaFbztx2Qn6k/rs/1P8w02tLOND7ofGe8XlpMNib1x3uEft6nK1K4WmjZX3pWrR8INSMxuXk183GfqQ9AtXXxTlVDB0Lk0yyVVdnVCn2LJLhi6RVsjV8f58c1Smt9Hiz2omWLJPmF5ywhbkujksEVexQKxMfNhpVCKrbzNzMb9OQ09B8qC9JVSFSWUSOEbT2/SqncXV/jhwz8W+t2/PjljtQEEvvL51ehJBd4GVoMnYZ6FJsIv/QJ95GA1ESyGEIEAHpSsqjmXody69p9bUzScOBgaV1DxUd7IdOYcNQdq+QtY0IVCzJp2SBt2KM6YeI3OI= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(376002)(39860400002)(346002)(136003)(46966006)(36840700001)(83380400001)(5660300002)(36860700001)(82740400003)(6666004)(8936002)(6636002)(54906003)(356005)(36906005)(1076003)(7696005)(316002)(37006003)(70206006)(4326008)(70586007)(7636003)(36756003)(2906002)(47076005)(2616005)(6862004)(86362001)(8676002)(426003)(478600001)(16526019)(82310400003)(186003)(336012)(55016002)(6286002)(26005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2021 04:09:57.2651 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e3d9d1dd-e260-4374-8c1b-08d9307c9a67 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT003.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4942 Subject: [dpdk-dev] [PATCH v1 03/14] common/mlx5: support auxiliary bus X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch adds auxiliary bus driver and delegate to registered internal mlx5 common device drivers, i.e. eth, vdpa... Current major target is to support SubFunction on auxiliary bus. As a limitation of current driver, numa node of device is detected from PCI bus of device symbol link, will remove once numa node file available on sysfs. Signed-off-by: Xueming Li --- drivers/common/mlx5/linux/meson.build | 3 + .../common/mlx5/linux/mlx5_common_auxiliary.c | 172 ++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_verbs.c | 5 +- drivers/common/mlx5/meson.build | 2 +- drivers/common/mlx5/mlx5_common.c | 3 + drivers/common/mlx5/mlx5_common.h | 6 + drivers/common/mlx5/mlx5_common_private.h | 6 + drivers/common/mlx5/version.map | 2 + 8 files changed, 197 insertions(+), 2 deletions(-) create mode 100644 drivers/common/mlx5/linux/mlx5_common_auxiliary.c diff --git a/drivers/common/mlx5/linux/meson.build b/drivers/common/mlx5/linux/meson.build index 5cea1b44d7..676fd05014 100644 --- a/drivers/common/mlx5/linux/meson.build +++ b/drivers/common/mlx5/linux/meson.build @@ -48,10 +48,13 @@ endif sources += files('mlx5_nl.c') sources += files('mlx5_common_os.c') sources += files('mlx5_common_verbs.c') +sources += files('mlx5_common_auxiliary.c') if not dlopen_ibverbs sources += files('mlx5_glue.c') endif +deps += ['bus_auxiliary'] + # To maintain the compatibility with the make build system # mlx5_autoconf.h file is still generated. # input array for meson member search: diff --git a/drivers/common/mlx5/linux/mlx5_common_auxiliary.c b/drivers/common/mlx5/linux/mlx5_common_auxiliary.c new file mode 100644 index 0000000000..79d567087c --- /dev/null +++ b/drivers/common/mlx5/linux/mlx5_common_auxiliary.c @@ -0,0 +1,172 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies Ltd + */ + +#include +#include +#include +#include +#include +#include +#include "eal_filesystem.h" + +#include "mlx5_common_utils.h" +#include "mlx5_common_private.h" + +#define AUXILIARY_SYSFS_PATH "/sys/bus/auxiliary/devices" +#define MLX5_AUXILIARY_PREFIX "mlx5_core.sf." + +int +mlx5_auxiliary_get_child_name(const char *dev, const char *node, + char *child, size_t size) +{ + DIR *dir; + struct dirent *dent; + MKSTR(path, "%s/%s%s", AUXILIARY_SYSFS_PATH, dev, node); + + dir = opendir(path); + if (dir == NULL) { + rte_errno = errno; + return -rte_errno; + } + /* Get the first file name. */ + while ((dent = readdir(dir)) != NULL) { + if (dent->d_name[0] != '.') + break; + } + closedir(dir); + if (dent == NULL) { + rte_errno = ENOENT; + return -rte_errno; + } + if (rte_strscpy(child, dent->d_name, size) < 0) + return -rte_errno; + return 0; +} + +static int +mlx5_auxiliary_get_pci_path(const struct rte_auxiliary_device *dev, + char *sysfs_pci, size_t size) +{ + char sysfs_real[PATH_MAX] = { 0 }; + MKSTR(sysfs_aux, "%s/%s", AUXILIARY_SYSFS_PATH, dev->name); + char *dir; + + if (realpath(sysfs_aux, sysfs_real) == NULL) { + rte_errno = errno; + return -rte_errno; + } + dir = dirname(sysfs_real); + if (dir == NULL) { + rte_errno = errno; + return -rte_errno; + } + if (rte_strscpy(sysfs_pci, dir, size) < 0) + return -rte_errno; + return 0; +} + +static int +mlx5_auxiliary_get_numa(const struct rte_auxiliary_device *dev) +{ + unsigned long numa; + char numa_path[PATH_MAX]; + + if (mlx5_auxiliary_get_pci_path(dev, numa_path, sizeof(numa_path)) != 0) + return SOCKET_ID_ANY; + if (strcat(numa_path, "/numa_node") == NULL) { + rte_errno = ENAMETOOLONG; + return SOCKET_ID_ANY; + } + if (eal_parse_sysfs_value(numa_path, &numa) != 0) { + rte_errno = EINVAL; + return SOCKET_ID_ANY; + } + return (int)numa; +} + +struct ibv_device * +mlx5_get_aux_ibv_device(const struct rte_auxiliary_device *dev) +{ + int n; + char ib_name[64] = { 0 }; + struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n); + struct ibv_device *ibv_match = NULL; + + if (!ibv_list) { + rte_errno = ENOSYS; + return NULL; + } + if (mlx5_auxiliary_get_child_name(dev->name, "/infiniband", + ib_name, sizeof(ib_name)) != 0) + return NULL; + while (n-- > 0) { + if (strcmp(ibv_list[n]->name, ib_name) != 0) + continue; + ibv_match = ibv_list[n]; + break; + } + if (ibv_match == NULL) + rte_errno = ENOENT; + mlx5_glue->free_device_list(ibv_list); + return ibv_match; +} + +static bool +mlx5_common_auxiliary_match(const char *name) +{ + return strncmp(name, MLX5_AUXILIARY_PREFIX, + strlen(MLX5_AUXILIARY_PREFIX)) == 0; +} + +static int +mlx5_common_auxiliary_probe(struct rte_auxiliary_driver *drv __rte_unused, + struct rte_auxiliary_device *dev) +{ + dev->device.numa_node = mlx5_auxiliary_get_numa(dev); + return mlx5_common_dev_probe(&dev->device); +} + +static int +mlx5_common_auxiliary_remove(struct rte_auxiliary_device *auxiliary_dev) +{ + return mlx5_common_dev_remove(&auxiliary_dev->device); +} + +static int +mlx5_common_auxiliary_dma_map(struct rte_auxiliary_device *auxiliary_dev, + void *addr, uint64_t iova, size_t len) +{ + return mlx5_common_dev_dma_map(&auxiliary_dev->device, addr, iova, len); +} + +static int +mlx5_common_auxiliary_dma_unmap(struct rte_auxiliary_device *auxiliary_dev, + void *addr, uint64_t iova, size_t len) +{ + return mlx5_common_dev_dma_unmap(&auxiliary_dev->device, addr, iova, + len); +} + +static struct rte_auxiliary_driver mlx5_auxiliary_driver = { + .driver = { + .name = MLX5_AUXILIARY_DRIVER_NAME, + }, + .match = mlx5_common_auxiliary_match, + .probe = mlx5_common_auxiliary_probe, + .remove = mlx5_common_auxiliary_remove, + .dma_map = mlx5_common_auxiliary_dma_map, + .dma_unmap = mlx5_common_auxiliary_dma_unmap, +}; + +void mlx5_common_auxiliary_init(void) +{ + if (mlx5_auxiliary_driver.bus == NULL) + rte_auxiliary_register(&mlx5_auxiliary_driver); +} + +RTE_FINI(mlx5_common_auxiliary_driver_finish) +{ + if (mlx5_auxiliary_driver.bus != NULL) + rte_auxiliary_unregister(&mlx5_auxiliary_driver); +} diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c index 6a6ab7a7a2..9080bd3e87 100644 --- a/drivers/common/mlx5/linux/mlx5_common_verbs.c +++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c @@ -12,6 +12,7 @@ #include #include +#include #include "mlx5_common_utils.h" #include "mlx5_common_log.h" @@ -24,10 +25,12 @@ struct ibv_device * mlx5_os_get_ibv_dev(const struct rte_device *dev) { - struct ibv_device *ibv = NULL; + struct ibv_device *ibv; if (mlx5_dev_is_pci(dev)) ibv = mlx5_os_get_ibv_device(&RTE_DEV_TO_PCI_CONST(dev)->addr); + else + ibv = mlx5_get_aux_ibv_device(RTE_DEV_TO_AUXILIARY_CONST(dev)); if (ibv == NULL) { rte_errno = ENODEV; DRV_LOG(ERR, "Verbs device not found: %s", dev->name); diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build index fa2b8b9834..6ddbde7e8f 100644 --- a/drivers/common/mlx5/meson.build +++ b/drivers/common/mlx5/meson.build @@ -7,7 +7,7 @@ if not (is_linux or (is_windows and is_ms_linker)) subdir_done() endif -deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs'] +deps += ['hash', 'pci', 'bus_pci', 'bus_auxiliary', 'net', 'eal', 'kvargs'] sources += files( 'mlx5_devx_cmds.c', 'mlx5_common.c', diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index dc32a7bd90..b4ca0a9fe1 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -399,6 +399,9 @@ mlx5_class_driver_register(struct mlx5_class_driver *driver) static void mlx5_common_driver_init(void) { mlx5_common_pci_init(); +#ifdef RTE_EXEC_ENV_LINUX + mlx5_common_auxiliary_init(); +#endif } static bool mlx5_common_initialized; diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index c0c950f8f9..439f16cde1 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -22,6 +22,7 @@ /* Reported driver name. */ #define MLX5_PCI_DRIVER_NAME "mlx5_pci" +#define MLX5_AUXILIARY_DRIVER_NAME "mlx5_auxiliary" /* Bit-field manipulation. */ #define BITFIELD_DECLARE(bf, type, size) \ @@ -107,6 +108,7 @@ pmd_drv_log_basename(const char *s) int mkstr_size_##name = snprintf(NULL, 0, "" __VA_ARGS__); \ char name[mkstr_size_##name + 1]; \ \ + memset(name, 0, mkstr_size_##name + 1); \ snprintf(name, sizeof(name), "" __VA_ARGS__) enum { @@ -134,6 +136,10 @@ enum { PCI_DEVICE_ID_MELLANOX_CONNECTX7BF = 0Xa2dc, }; + +__rte_internal +int mlx5_auxiliary_get_child_name(const char *dev, const char *node, + char *child, size_t size); /* Maximum number of simultaneous unicast MAC addresses. */ #define MLX5_MAX_UC_MAC_ADDRESSES 128 /* Maximum number of simultaneous Multicast MAC addresses. */ diff --git a/drivers/common/mlx5/mlx5_common_private.h b/drivers/common/mlx5/mlx5_common_private.h index 791eb3cd77..9f00a6c54d 100644 --- a/drivers/common/mlx5/mlx5_common_private.h +++ b/drivers/common/mlx5/mlx5_common_private.h @@ -6,6 +6,7 @@ #define _MLX5_COMMON_PRIVATE_H_ #include +#include #include "mlx5_common.h" @@ -34,6 +35,11 @@ void mlx5_common_driver_on_register_pci(struct mlx5_class_driver *driver); bool mlx5_dev_pci_match(const struct mlx5_class_driver *drv, const struct rte_device *dev); +/* Common auxiliary bus driver: */ +void mlx5_common_auxiliary_init(void); +struct ibv_device *mlx5_get_aux_ibv_device( + const struct rte_auxiliary_device *dev); + #ifdef __cplusplus } #endif /* __cplusplus */ diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index 6be882e98c..edfd3792e0 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -3,6 +3,8 @@ INTERNAL { haswell_broadwell_cpu; + mlx5_auxiliary_get_child_name; # WINDOWS_NO_EXPORT + mlx5_class_driver_register; mlx5_common_init; -- 2.25.1