From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id B61BCA051E; Wed, 10 Jun 2020 19:18:17 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 734FF1BF94; Wed, 10 Jun 2020 19:18:11 +0200 (CEST) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-eopbgr140053.outbound.protection.outlook.com [40.107.14.53]) by dpdk.org (Postfix) with ESMTP id 709CE1BF74 for ; Wed, 10 Jun 2020 19:18:08 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Sbf8KXYEvQzhtwME0TUsa866jyM9AT2Y4xmJu8BF8qsa+2c+Cv7ILsWM8eZCmaPk10JUnHVcFvvK3l9qu/ljsU29B9JoHJCRCfMFTXc2ajHhv1F1j9v7HlR4Xgi1p089yEDk2shYHXxX42bg6tXc4F/0RlBc+6LZ2ZaYb+9NduQdz5RHu59JY7KpEY4c/kTm1RZJabxinymSi0GAN3bFxXB1dFeYNgF3GvC0koWTtfKji/u02he8mfgdNsQ7qxor/pqGsnTKW4RKzXKjbCYr1dbOBDAjVwuPpDmOIqBQPr7uiRDJp+o+6YNu40bQo/oM1AWgpSGq1vob/BXpaBdKzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IibuB48oDc3MrsxV4PZIOarFIVxAqpfDZxqui42K1zk=; b=IoKm6og4VFI7eK1X/QrXZN5FOSg1v3tI0KfMVTj8pFihCQyGgCOgtUjbDL8sXOgjUcMxtM4h0TPRkGfCneqUfg4KUKuqLgfNsqIQArlt8J8jFqJlrRb+8Mx+kCrABC//V555bzNGGIOm7d/oB2pCnX1ABa3VgdczCMoiqpkO1CD+QILavbb+0vtrfKzKiq9vI7sSfhfkuAg27d9OD0UHpeWWSgOGJfsMeqnu3B0hHf5ybzW2wY9hYuRpwhaC+4Qv0bW1oYfy21szK8etrac9W81MuX2m7RVCZBht7F3tTKj+vwby3j7WAhsAQCp/YtwLnwoC/dCWaSOosFX+4Yr1RA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IibuB48oDc3MrsxV4PZIOarFIVxAqpfDZxqui42K1zk=; b=jbvYRLNh0Z60ozv1v4miGdcw63KiSXXDnzP8vq9QeicQeZS9Tio7CVXgjBhZeVgd4NhhgnL/VX4S1YsphaBGtCK3LD+xIHcAID8aYdj1HvHU/a72P2N4X78WeO3AKJMPGt+F+f6shF1G6kgycaANlTa07Vg2++TdyVSZmAqDORU= Authentication-Results: dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=none action=none header.from=mellanox.com; Received: from AM0PR05MB4866.eurprd05.prod.outlook.com (2603:10a6:208:c0::32) by AM0PR05MB5346.eurprd05.prod.outlook.com (2603:10a6:208:f5::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3066.18; Wed, 10 Jun 2020 17:18:04 +0000 Received: from AM0PR05MB4866.eurprd05.prod.outlook.com ([fe80::d44d:a804:c730:d2b7]) by AM0PR05MB4866.eurprd05.prod.outlook.com ([fe80::d44d:a804:c730:d2b7%2]) with mapi id 15.20.3066.023; Wed, 10 Jun 2020 17:18:04 +0000 From: Parav Pandit To: dev@dpdk.org Cc: ferruh.yigit@intel.com, thomasm@mellanox.com, orika@mellanox.com, matan@mellanox.com, Parav Pandit Date: Wed, 10 Jun 2020 17:17:22 +0000 Message-Id: <20200610171728.89-1-parav@mellanox.com> X-Mailer: git-send-email 2.25.4 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: SN6PR16CA0060.namprd16.prod.outlook.com (2603:10b6:805:ca::37) To AM0PR05MB4866.eurprd05.prod.outlook.com (2603:10a6:208:c0::32) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from sw-mtx-036.mtx.labs.mlnx (208.176.44.194) by SN6PR16CA0060.namprd16.prod.outlook.com (2603:10b6:805:ca::37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3088.21 via Frontend Transport; Wed, 10 Jun 2020 17:18:02 +0000 X-Mailer: git-send-email 2.25.4 X-Originating-IP: [208.176.44.194] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: e7bc4592-f548-4f5a-064e-08d80d623bc5 X-MS-TrafficTypeDiagnostic: AM0PR05MB5346: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:369; X-Forefront-PRVS: 0430FA5CB7 X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 45u52+iS6v9HjR6kJYpSRqI0PoPYIb0IfWCj97yytvDf50q7oYqF8B2c/jo4z81MV9VzPb84+YZPdNentznIO8KeSmwStWVt0ClKeUkwvpIBTDJluPIh6ZbLg/gF5Gcf1JoKkuHAS///zWC9EaFleY2ya0Q7MI5HWjvdXVCO1kd+gEReusEIOhSPY8ER91krduFStX6rSMpxepsHvqeDEmbS0ZZEuLFfmPeSCQKrle8zNEk9hH5HmK8rkXgd9r8M96XGNEpQGxQKgVFYHJlVPceo2XRcirS0w3lpluP53iGSKMlo4hfml1EN3f0lpBrC X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM0PR05MB4866.eurprd05.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(366004)(346002)(396003)(136003)(39860400002)(376002)(66476007)(6506007)(36756003)(16526019)(6486002)(6916009)(2906002)(316002)(26005)(5660300002)(186003)(66556008)(66946007)(6512007)(6666004)(8936002)(8676002)(4326008)(2616005)(83380400001)(956004)(478600001)(1076003)(86362001)(52116002)(107886003); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: ss7IAIZlTRcfSJYi67vkf06Ll2E9XSbpzvTxxFhlpLTGMRxYoqmC+eGmMO6mofO8QYtrMOX/wcZkKMEkZmktJkNJJmBoyW1+bRvB+MxiEaR+7YkF2U0Pzp+7wtFT16X1jgQS8Ytw2PmhCJCLwMMqJxhm5n4iueNzKV13atkRDzkVfjMCycq6BBoMyvlsKqmZLU2NR1t7sU3z5Ldy2H/oe3per/KqOrNjdQt/hPfy3zVKVSOLwQvyRPyjD39ENz8LHXgbS1dlyuvbACO18N5u67joTMMgA0FTAjp4x51tyD6vPAjMg7GzPwNYjaJu/+icBETENLLit/yejbQJdHsOv8gwNrW+6jmMN0axOdX/Yczy0lshZW/sZkZh49q2pf6C6wkxVmL8JTQiuPO/ZzGtiIqRN7T3Nh2xPg5P5vmvmGl1RsO79IV6Zi6hcVaTRk6YBJ5bpOSWTEYBWu6fv7kOY5Ge8gah6UFbXIkg88PjsnlP0rXmeOYvlRgbGwBgO54p X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: e7bc4592-f548-4f5a-064e-08d80d623bc5 X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jun 2020 17:18:04.8410 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3pUJMtyz/UCbHyhnF1OQNTnoS4o1wC/mulQrTDxt8pQlkDjvfi67a98AqbtshzeOFqtYPsDgBIkyA2aXZFaQfQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR05MB5346 Subject: [dpdk-dev] [RFC PATCH 0/6] Improve mlx5 PMD common driver framework for multiple classes X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit introduces mlx5 bus to support multiple class of devices for a single PCI device. Motivation and example ---------------------- mlx5 PCI device supports multiple class of devices such as net, vdpa and regex devices. Currently only one pmd (either net or vdpa) can bind to this device. This design limits use of PCI device only for single device class. To support multiple classes simultaneously for a mlx5 PCI device, a new mlx5 PCI bus is created. This bus allows binding multiple class drivers (such as net, vdpa, regex(future)) to bind to the mlx5 PCI bus driver. Change description ------------------ Patch-1 prepares the code to have RTE_BIT() macro defined in a common header. Patch-2 Prepares the code to check for enabled class Patch-3 Changes class value to a bit field Patch-4 Exposes mlx5_pci class driver registration APIs PAtch-5 Implements mlx5 PCI bus Patch-6 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus API instead of rte PCI bus API Design overview --------------- ----------- ------------ ------------- | mlx5 | | mlx5 | | mlx5 | | net pmd | | vdpa pmd | | regex pmd | ----------- ------------ ------------- \ | / \ | / \ ------------- / \______| mlx5 |_____ / | pci bus | ------------- | ----------- | mlx5 | | pci dev | ----------- - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI ID table of all related mlx5 PCI devices. - mlx5 class driver such as net, vdpa, regex PMD defines its specific PCI ID table and mlx5 bus driver probes matching class drivers. - mlx5 pci bus driver is cental place that validates supported class combinations. - In future as code evolves, more device setup/cleanup and resource creation code moves to mlx5 PCI bus driver. Alternatives considered ----------------------- 1. Instead of creating mlx5 pci bus, a common driver is implemented which exposes class registration API. However, bus model fits better with existing DPDK design similar to ifpga driver. Class registration API need to create a new callbacks and ID signature; instead it is better to utilize current well defined methods. 2. Enhance pci core to allow multiple driver binding to single rte PCI device. This approach is not taken, because peer drivers using one PCI device won't be aware of other's presence. This requires cross-driver syncronization of who initializes common resources (such as irq, eq and more). This also requires refcounting common objects etc among peer drivers. Instead of layered approach delivers and allows putting common resource sharing, setup code in common bus driver. It also eliminates peer blind zone problem as bottom pci bus layer provides necessary setup without any reference counting. 3. In future mlx5 prefers to use RDMA MR cache of the mbuf used between net and regex pmd so that same mbuf use across multiple device can be possible. Examples: -------- A user who wish to use a specific class(es) provides list of classes at command line such as, ./testpmd -w ,class=net:vdpa ./testpmd -w ,class=vdpa In future, ./testpmd -w ,class=net:regex Parav Pandit (6): eal: introduce macros for getting value for bit common/mlx5: use class enable check helper function common/mlx5: change mlx5 class enum values as bits bus/mlx5_pci: add mlx5 PCI bus bus/mlx5_pci: register a PCI driver bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver config/common_base | 6 + config/defconfig_arm64-bluefield-linuxapp-gcc | 6 + drivers/bus/Makefile | 3 + drivers/bus/meson.build | 2 +- drivers/bus/mlx5_pci/Makefile | 49 +++ drivers/bus/mlx5_pci/meson.build | 6 + drivers/bus/mlx5_pci/mlx5_pci_bus.c | 327 ++++++++++++++++++ drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 66 ++++ .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 + drivers/common/mlx5/mlx5_common.c | 12 +- drivers/common/mlx5/mlx5_common.h | 8 +- .../common/mlx5/rte_common_mlx5_version.map | 2 +- drivers/net/mlx5/Makefile | 3 +- drivers/net/mlx5/linux/mlx5_os.c | 4 +- drivers/net/mlx5/linux/mlx5_os.h | 3 - drivers/net/mlx5/meson.build | 2 +- drivers/net/mlx5/mlx5.c | 7 +- drivers/net/mlx5/mlx5.h | 1 - drivers/vdpa/mlx5/Makefile | 3 +- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 12 +- lib/librte_eal/include/rte_bits.h | 10 + mk/rte.app.mk | 1 + 23 files changed, 511 insertions(+), 29 deletions(-) create mode 100644 drivers/bus/mlx5_pci/Makefile create mode 100644 drivers/bus/mlx5_pci/meson.build create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map create mode 100644 lib/librte_eal/include/rte_bits.h -- 2.25.4