From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 053B4A0350; Sun, 21 Jun 2020 21:12:57 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 5C3081C1A9; Sun, 21 Jun 2020 21:12:31 +0200 (CEST) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (mail-am6eur05on2081.outbound.protection.outlook.com [40.107.22.81]) by dpdk.org (Postfix) with ESMTP id DC07A1C199 for ; Sun, 21 Jun 2020 21:12:25 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KzTXTF/B+3NEGz6A/NzoEJmwLWd5XKsStrSukl9/q2niTHD0/YUdyr3lRCaf1XOkPrUML1QLNlUjR5glOEoVGJBVBCHuyRwn1ozIQogIRINAAN30bIBhHe+aj0UXEJtqEkF3MlJ/SlR48viDtziVicEoKyHfShNVnIQWY4OfxxeLibKme2yxU3pa3YAkMbnOMcGG1XHXgI6Nx5ZXq/gqQB9FlwNOztZZzGQvVQ47bn1DrIPPSUcbzOjaL7WlUrY1LVjSkQuku0zRe8E/0jbVNZRFFdjl9ibXwMwDQNslBTTdcqFMXRQkFOUvciSNXISkSBj2ZOGFR5rQ/uUfNhJL8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qvdMYKddghyYnHDDKT9MxmhSRsEWw5t+y8tKQ6YN5zM=; b=KSMHhcaE+FHwOz0Dm8VYDEVJEXTapoQR61KPkutJJ8Fu2cAXc+13UM4QR2j5oXoBUn0hE3X6u+e0gaCaLE6bw9tzHRSF5aYjypxlZzwIGLcg3kqtpgV3HOlZQiAoEKYBVDwcskF/G/AFs+F7EEkYigkuWYQ9Cx0cdbSJtVdLxu9x9NmqUBXWTXnHvOnSDt2cyKrOKx0AnrSlPkO6VbBUNZSUpnrsq387VNDbjDZoBdsXdleKNnb8lCm2apyZ62kF985IMI/iBhAph0mW9jp64N4pGw1WDiArpS8KMuRs54la9YSBcCrWWb7otdNmRmi0dQvwK692cm1+dvYvnLUSig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qvdMYKddghyYnHDDKT9MxmhSRsEWw5t+y8tKQ6YN5zM=; b=ewVPYHKskV9PuUIMCy4uMRfy65I6ccJfqguY4CykXXW08BmBDGLErqVayfPIeM8CrOSdEcy2YezqcT8bJSALwTtFQC97dGsAKsQYZR4Dt5ZjMphs+GCVtzF/aL/JqgdYCWNmDhkYFx5o91XBhk8hBQod8mNoEhZhgawcW9+pmZk= Authentication-Results: u256.net; dkim=none (message not signed) header.d=none;u256.net; dmarc=none action=none header.from=mellanox.com; Received: from AM0PR05MB4866.eurprd05.prod.outlook.com (2603:10a6:208:c0::32) by AM0PR05MB6307.eurprd05.prod.outlook.com (2603:10a6:208:145::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3109.21; Sun, 21 Jun 2020 19:12:25 +0000 Received: from AM0PR05MB4866.eurprd05.prod.outlook.com ([fe80::d44d:a804:c730:d2b7]) by AM0PR05MB4866.eurprd05.prod.outlook.com ([fe80::d44d:a804:c730:d2b7%2]) with mapi id 15.20.3109.026; Sun, 21 Jun 2020 19:12:25 +0000 From: Parav Pandit To: grive@u256.net, ferruh.yigit@intel.com, thomas@monjalon.net, dev@dpdk.org Cc: orika@mellanox.com, matan@mellanox.com, Parav Pandit Date: Sun, 21 Jun 2020 19:11:58 +0000 Message-Id: <20200621191200.28120-5-parav@mellanox.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200621191200.28120-1-parav@mellanox.com> References: <20200610171728.89-2-parav@mellanox.com> <20200621191200.28120-1-parav@mellanox.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: DM6PR02CA0105.namprd02.prod.outlook.com (2603:10b6:5:1f4::46) To AM0PR05MB4866.eurprd05.prod.outlook.com (2603:10a6:208:c0::32) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from sw-mtx-036.mtx.labs.mlnx (208.176.44.194) by DM6PR02CA0105.namprd02.prod.outlook.com (2603:10b6:5:1f4::46) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3109.22 via Frontend Transport; Sun, 21 Jun 2020 19:12:23 +0000 X-Mailer: git-send-email 2.25.4 X-Originating-IP: [208.176.44.194] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: a2274d04-ed6a-4a18-ac73-08d816170819 X-MS-TrafficTypeDiagnostic: AM0PR05MB6307: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5516; X-Forefront-PRVS: 04410E544A X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: nLgO8ULZfuMUSBtS8/GruYN//h47qjkYhvkTMgHHCev71pf74gJLe7Ls14Lp7PajBLFbm7ii/Ncbd/MvSPnIkNUDCtxE7Jc/fQ2JUvwo3IrmDuudrNt1Co+N4k3DncnQDdlXbNF4OKsL4AWQMMQqrQlQRnxmsiPfbcFG2dzoVyErwGj3uEJXGvsWIw2KC1Ee1UryBeyGBs0GVx6LbQLR9rHgzSNpcv/e+yodbrSd2NPz3Ik6NCISuQvnZlb/o9C0cqv3agjq+zq3j8DrmdKOyxb/CBVIiocgzDhqtuovFnkMm3tUKRf6lDSoBQd6wQ5C X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM0PR05MB4866.eurprd05.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(376002)(366004)(39860400002)(396003)(346002)(136003)(36756003)(66556008)(66476007)(6506007)(4326008)(5660300002)(6486002)(66946007)(26005)(186003)(107886003)(16526019)(316002)(86362001)(6512007)(1076003)(478600001)(2616005)(6666004)(8676002)(83380400001)(956004)(52116002)(2906002)(8936002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: jXel2MQtPK6fS+KEEzYpglZLmC6pJETdy6GU+PBqV2PRNuX6G+VLva6REaPzzYfz2qIxl/gbmkMajPrmiY90Y1rkyX4mq09LTPTuZAw91c5eTWnANmQnWw1nKiQ0VakmXnHPhm9HMoU4MkQAoBrGKyT11ZNpYAHWWWVrOQ8gSLKtPt5dBKI8W/PlzK7OvNj7pvwUVR/xSma/IsyogVquq/ylOGUhrDJs+K7+lS/qraZJwJzhtJ+EkfIqWAnlWri+Vx/rYrmRCCOoE+xR75oOY6/MaRosfEXcjrbuU1fvxfNvcYWGX1XRQt14c+2UwKpcSGtZHijGMEIDAAwqFMHIc0J9JQRhY6+0IFC165f9bvGNvgTQUmY0TYzgC/lvgrRV22UPVCoidV8MbhzElMxyW//pN6OqpMWAXnzO0u5Y159UsdG5zKu2gmnoTjPArEvi2o/D5Q/oGj/xsA89dFIz8EcCUEUeXdBFM20YwUB7wyWii6Qcd5Umd4ehYw7JN7Wr X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: a2274d04-ed6a-4a18-ac73-08d816170819 X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jun 2020 19:12:25.0803 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /E/VHfxZPHoGn1kty5zlmOWOZ150zQlCwyO/WThXjJnlIpR5k0XXGXqgpWIOIAI2Li6sIxapSlPmlNO0RbKR7A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR05MB6307 Subject: [dpdk-dev] [PATCH v2 4/6] bus/mlx5_pci: register a PCI driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Create a mlx5 bus driver framework for invoking drivers of multiple classes who have registered with the mlx5_pci bus driver. Validate user class arguments for supported class combinations. Signed-off-by: Parav Pandit --- Changelog: v1->v2: - Address comments from Thomas and Gaetan - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - USe uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options --- drivers/bus/mlx5_pci/Makefile | 2 + drivers/bus/mlx5_pci/meson.build | 2 +- drivers/bus/mlx5_pci/mlx5_pci_bus.c | 290 ++++++++++++++++++++++++ drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 1 + 4 files changed, 294 insertions(+), 1 deletion(-) diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile index 7db977ba8..e53ed8856 100644 --- a/drivers/bus/mlx5_pci/Makefile +++ b/drivers/bus/mlx5_pci/Makefile @@ -13,7 +13,9 @@ CFLAGS += $(WERROR_FLAGS) CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 CFLAGS += -I$(RTE_SDK)/drivers/bus/pci +CFLAGS += -D_DEFAULT_SOURCE LDLIBS += -lrte_eal +LDLIBS += -lrte_kvargs LDLIBS += -lrte_common_mlx5 LDLIBS += -lrte_pci -lrte_bus_pci diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build index cc4a84e23..5111baa4e 100644 --- a/drivers/bus/mlx5_pci/meson.build +++ b/drivers/bus/mlx5_pci/meson.build @@ -1,6 +1,6 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2020 Mellanox Technologies Ltd -deps += ['pci', 'bus_pci', 'common_mlx5'] +deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs'] install_headers('rte_bus_mlx5_pci.h') sources = files('mlx5_pci_bus.c') diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c index 66db3c7b0..e8f1649a3 100644 --- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c @@ -3,12 +3,302 @@ */ #include "rte_bus_mlx5_pci.h" +#include static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list = TAILQ_HEAD_INITIALIZER(drv_list); +static const struct { + const char *name; + unsigned int dev_class; +} mlx5_classes[] = { + { .name = "vdpa", .dev_class = MLX5_CLASS_VDPA }, + { .name = "net", .dev_class = MLX5_CLASS_NET }, +}; + +static const unsigned int mlx5_valid_class_combo[] = { + MLX5_CLASS_NET, + MLX5_CLASS_VDPA, + /* New class combination should be added here */ +}; + +static int class_name_to_val(const char *class_name) +{ + unsigned int i; + + for (i = 0; i < RTE_DIM(mlx5_classes); i++) { + if (strcmp(class_name, mlx5_classes[i].name) == 0) + return mlx5_classes[i].dev_class; + + } + return -EINVAL; +} + +static int +mlx5_bus_opt_handler(__rte_unused const char *key, const char *class_names, + void *opaque) +{ + int *ret = opaque; + char *nstr_org; + int class_val; + char *found; + char *nstr; + + *ret = 0; + nstr = strdup(class_names); + if (!nstr) { + *ret = -ENOMEM; + return *ret; + } + + nstr_org = nstr; + while (nstr) { + /* Extract each individual class name */ + found = strsep(&nstr, ":"); + if (!found) + continue; + + /* Check if its a valid class */ + class_val = class_name_to_val(found); + if (class_val < 0) { + *ret = -EINVAL; + goto err; + } + + *ret |= class_val; + } +err: + free(nstr_org); + if (*ret < 0) + DRV_LOG(ERR, "Invalid mlx5 class options %s. Maybe typo in device class argument setting?", + class_names); + return *ret; +} + +static int +parse_class_options(const struct rte_devargs *devargs) +{ + const char *key = MLX5_CLASS_ARG_NAME; + struct rte_kvargs *kvlist; + int ret = 0; + + if (devargs == NULL) + return 0; + kvlist = rte_kvargs_parse(devargs->args, NULL); + if (kvlist == NULL) + return 0; + if (rte_kvargs_count(kvlist, key)) + rte_kvargs_process(kvlist, key, mlx5_bus_opt_handler, &ret); + rte_kvargs_free(kvlist); + return ret; +} + void rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver) { TAILQ_INSERT_TAIL(&drv_list, driver, next); } + +static bool +mlx5_bus_match(const struct rte_mlx5_pci_driver *drv, + const struct rte_pci_device *pci_dev) +{ + const struct rte_pci_id *id_table; + + for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0; + id_table++) { + /* check if device's ids match the class driver's ones */ + if (id_table->vendor_id != pci_dev->id.vendor_id && + id_table->vendor_id != PCI_ANY_ID) + continue; + if (id_table->device_id != pci_dev->id.device_id && + id_table->device_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_vendor_id != + pci_dev->id.subsystem_vendor_id && + id_table->subsystem_vendor_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_device_id != + pci_dev->id.subsystem_device_id && + id_table->subsystem_device_id != PCI_ANY_ID) + continue; + if (id_table->class_id != pci_dev->id.class_id && + id_table->class_id != RTE_CLASS_ANY_ID) + continue; + + return true; + } + return false; +} + +static int is_valid_class_combo(uint32_t user_classes) +{ + unsigned int i; + + /* Verify if user specified valid supported combination */ + for (i = 0; i < RTE_DIM(mlx5_valid_class_combo); i++) { + if (mlx5_valid_class_combo[i] == user_classes) + return 0; + } + /* Not found any valid class combination */ + return -EINVAL; +} + +static int validate_single_class_dma_ops(void) +{ + struct rte_mlx5_pci_driver *class; + int dma_map_classes = 0; + + TAILQ_FOREACH(class, &drv_list, next) { + if (class->pci_driver.dma_map) + dma_map_classes++; + } + if (dma_map_classes > 1) { + DRV_LOG(ERR, "Multiple classes with DMA ops is unsupported"); + return -EINVAL; + } + return 0; +} + +/** + * DPDK callback to register to probe multiple PCI class devices. + * + * @param[in] pci_drv + * PCI driver structure. + * @param[in] dev + * PCI device information. + * + * @return + * 0 on success, 1 to skip this driver, a negative errno value otherwise + * and rte_errno is set. + */ +static int +mlx5_bus_pci_probe(struct rte_pci_driver *drv __rte_unused, + struct rte_pci_device *dev) +{ + struct rte_mlx5_pci_driver *class; + uint32_t user_classes = 0; + int ret; + + ret = validate_single_class_dma_ops(); + if (ret) + return ret; + + ret = parse_class_options(dev->device.devargs); + if (ret < 0) + return ret; + + user_classes = ret; + if (user_classes) { + /* Validate combination here */ + ret = is_valid_class_combo(user_classes); + if (ret) { + DRV_LOG(ERR, "Unsupported mlx5 classes supplied"); + return ret; + } + } + + /* Default to net class */ + if (user_classes == 0) + user_classes = MLX5_CLASS_NET; + + TAILQ_FOREACH(class, &drv_list, next) { + if (!mlx5_bus_match(class, dev)) + continue; + + if ((class->dev_class & user_classes) == 0) + continue; + + ret = -EINVAL; + if (class->loaded) { + /* If already loaded and class driver can handle + * reprobe, probe such class driver again. + */ + if (class->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN) + ret = class->pci_driver.probe(drv, dev); + } else { + ret = class->pci_driver.probe(drv, dev); + } + if (!ret) + class->loaded = true; + } + return 0; +} + +/** + * DPDK callback to remove one or more class devices for a PCI device. + * + * This function removes all class devices belong to a given PCI device. + * + * @param[in] pci_dev + * Pointer to the PCI device. + * + * @return + * 0 on success, the function cannot fail. + */ +static int +mlx5_bus_pci_remove(struct rte_pci_device *dev) +{ + struct rte_mlx5_pci_driver *class; + + /* Remove each class driver in reverse order */ + TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) { + if (class->loaded) + class->pci_driver.remove(dev); + } + return 0; +} + +static int +mlx5_bus_pci_dma_map(struct rte_pci_device *dev, void *addr, + uint64_t iova, size_t len) +{ + struct rte_mlx5_pci_driver *class; + int ret = -EINVAL; + + TAILQ_FOREACH(class, &drv_list, next) { + if (!class->pci_driver.dma_map) + continue; + + return class->pci_driver.dma_map(dev, addr, iova, len); + } + return ret; +} + +static int +mlx5_bus_pci_dma_unmap(struct rte_pci_device *dev, void *addr, + uint64_t iova, size_t len) +{ + struct rte_mlx5_pci_driver *class; + int ret = -EINVAL; + + TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) { + if (!class->pci_driver.dma_unmap) + continue; + + return class->pci_driver.dma_unmap(dev, addr, iova, len); + } + return ret; +} + +static const struct rte_pci_id mlx5_bus_pci_id_map[] = { + { + .vendor_id = 0 + } +}; + +static struct rte_pci_driver mlx5_bus_driver = { + .driver = { + .name = "mlx5_bus_pci", + }, + .id_table = mlx5_bus_pci_id_map, + .probe = mlx5_bus_pci_probe, + .remove = mlx5_bus_pci_remove, + .dma_map = mlx5_bus_pci_dma_map, + .dma_unmap = mlx5_bus_pci_dma_unmap, + .drv_flags = RTE_PCI_DRV_INTR_LSC | RTE_PCI_DRV_INTR_RMV | + RTE_PCI_DRV_PROBE_AGAIN, +}; + +RTE_PMD_REGISTER_PCI(mlx5_bus, mlx5_bus_driver); +RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map); diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h index 571f7dfd6..c8cd7187b 100644 --- a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h @@ -55,6 +55,7 @@ struct rte_mlx5_pci_driver { enum mlx5_class dev_class; /**< Class of this driver */ struct rte_pci_driver pci_driver; /**< Inherit core pci driver. */ TAILQ_ENTRY(rte_mlx5_pci_driver) next; + bool loaded; }; /** -- 2.25.4