From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D7B6E43186; Tue, 17 Oct 2023 07:48:47 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EC86640EDC; Tue, 17 Oct 2023 07:47:15 +0200 (CEST) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2108.outbound.protection.outlook.com [40.107.244.108]) by mails.dpdk.org (Postfix) with ESMTP id 1B0D742D8C for ; Tue, 17 Oct 2023 07:47:14 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mEn8yf1MwQ2ZcpzgsmD27FxCGpmIrDnaRivr9bZrI57lVT9hDUWyCcW6X6I9SBhUhMIdVHcq/bzChZE9owTN59EhR8cXsAFSnt4wfycH7MeVN1bGExrr9BOok1i58zOANUAihGPvXyec28mJy/EdrL3BEK9oBq6laDZ65/XjcwqDhBoMDhv83M8CGUeRQ7CvX8MQ3m95A7Hk9d492xGvqNdO2fOmxm6FY9xheW25LIUbc8OilXGIXNKSo0bIPis4islOTp5KLfNsuhx7soMduAYSIBVEhT6JuSVvldRjunFHd7XAWsfRYJ3H/J/Mq+sfvQwHjAPuvhiJl0Ns2HHVRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jk/CbMQztnVDf2I4hFCWS4AwJwyfhhq7mTjnyIunuEA=; b=Tu/h6K8fxiCMSSRgpHLYyWKnVPrcUlaPhNc2lM6+RzxUa0DnswL/2RTnji4imRighZtydNKkPnWi5KhPkHa5sz6crh+qZWkfXiLXD3T7XgHr6/FvDnGC5/fegNxPiOwqpqcxon7SrpKCGzpltLDcAlqmAkH6JAwcRdRicUCv7cNT8PGnNRzAba9M5350n5ttfMiH8Oh2dRwzu/fbLO4fGhLEuOYNdUL7JysqjmtQJ+EmN7sqGw0XZSzBcUgCRqNKdvn6fAcFD5ykhyN99sWEBqkIXsAyzjTuo33mX75heLk2DkXKj61qIWMQtJljllGRBMAk9063X4D90l5zRKpN/w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=corigine.com; dmarc=pass action=none header.from=corigine.com; dkim=pass header.d=corigine.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corigine.onmicrosoft.com; s=selector2-corigine-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jk/CbMQztnVDf2I4hFCWS4AwJwyfhhq7mTjnyIunuEA=; b=ggg4GAnJnkeHfeqN8Ia14uIBUIAccgncJzkNM35JBvO8UWAj1ySLfW4f9LWe3WpEVE9iYEi21KGPP8LBgnhHYglNSJNgOeitF4CgHSAG4lWGI579xcLZ+hEAOkhEfjvjX5g6QkaCSyybaT8EOEKMgs7DzZ4+QffPLD5p7H4Y89M= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=corigine.com; Received: from PH0PR13MB5568.namprd13.prod.outlook.com (2603:10b6:510:12b::16) by MW5PR13MB5904.namprd13.prod.outlook.com (2603:10b6:303:1ce::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.45; Tue, 17 Oct 2023 05:47:12 +0000 Received: from PH0PR13MB5568.namprd13.prod.outlook.com ([fe80::b070:92e1:931e:fee7]) by PH0PR13MB5568.namprd13.prod.outlook.com ([fe80::b070:92e1:931e:fee7%4]) with mapi id 15.20.6863.047; Tue, 17 Oct 2023 05:47:12 +0000 From: Chaoyong He To: dev@dpdk.org Cc: oss-drivers@corigine.com, Chaoyong He , Shujing Dong , Long Wu , Peng Zhang Subject: [PATCH 20/25] vdpa/nfp: add the logic of remap PCI memory Date: Tue, 17 Oct 2023 13:45:40 +0800 Message-Id: <20231017054545.1692509-21-chaoyong.he@corigine.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20231017054545.1692509-1-chaoyong.he@corigine.com> References: <20231017054545.1692509-1-chaoyong.he@corigine.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: PH7PR17CA0069.namprd17.prod.outlook.com (2603:10b6:510:325::29) To PH0PR13MB5568.namprd13.prod.outlook.com (2603:10b6:510:12b::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH0PR13MB5568:EE_|MW5PR13MB5904:EE_ X-MS-Office365-Filtering-Correlation-Id: 61e8597c-70dd-4fbb-9ed3-08dbced4828c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: CQl7vIVqZkZyvPMKWp/qORwJJFA5VI3kpyDy8dzyvZsdQ8/9aBXIuRHKPO8SqKW4grysok31uhBYsfQQIkO4b40RCJPRV7V+hb9Y7VKvS/OrVdPya2dcTQzsdoyE4vM7ZOhuUPj7ILiqokaEjLKjrxSwHHOzdzZ4eH03gjPaEq2FxDutSbH2IDcavXIIrCvQrVzNvlnwlIDB8zuKNXFBhqjCkflrymoiZAtQ03EK+tczi//QLjBxBYxgBcLvkQEeKPITYmsb7IMTYcNbojBQKhDxcYewJqJGGinF3DeewOLkJOzFgELHvx8WqbDbszt6ZecUlFnBWJbYVhoUo5VimKSDJ9veuVUBEkHlQqN4Frtspsp3yNDbib+wJFJAbE3loAJly1BpgGUQPXeo8wwGOewXF7xszEEltCezMT1zX2bx9OB81xKg7qJxZJHNmKsCAN7yXvBxbX/vifGGe+xp/3Azo6RbXD2C7Y/26lQa0SinZUvDxE2tbkvU9L/bFwOATF+DXUS32pf7JtVuemWpy6dNEUe6/ScDSpSSWSGtqUZxiClGTxiB2H41pyyf+cyX2UNiTLX5kQYz7GDZZqSgJG5THnX7I8Cagqg+MhRuZxH9Z9rZR/2zg+uE1gWszkHPI14GCRG3au4haHUnzY5l/X0Cx4xWVVP3rBVJG+KaDHE= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR13MB5568.namprd13.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(136003)(366004)(376002)(39830400003)(346002)(396003)(230922051799003)(186009)(1800799009)(451199024)(64100799003)(2906002)(5660300002)(8936002)(8676002)(83380400001)(66556008)(66476007)(6916009)(316002)(66946007)(54906003)(41300700001)(478600001)(4326008)(6486002)(44832011)(6666004)(6512007)(6506007)(107886003)(38350700005)(2616005)(1076003)(26005)(36756003)(38100700002)(52116002)(86362001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?ggaTRAEdSV6cgGVTgHo9fL3jCeFiwX5RrbUoikI84w13LVhUeFNiw24b1PpV?= =?us-ascii?Q?55dWEHov/penMtp1ICPBxJP0wKdvRKlTS3K6w+33AeUcmTOkz3gMBDveWTFO?= =?us-ascii?Q?P4wlLBmxxottdHyAMBRny+z2JewPlxtgCixaq0JRJ33ZOw94HpFaY5T9AFSz?= =?us-ascii?Q?QxL4fkmWaTK4K+XaQMrlYXTXPTsxlJ6ulnkoyH5VMYTB5ESrhvyqDd0pY8Ja?= =?us-ascii?Q?KWlOYeJl9HVANKo2bbBIu5PrLmDMKgazKy+FjlqQAV8i81+8n3oeR3sFXpGY?= =?us-ascii?Q?Mtbz8i1J0+97t0m3V12EQtDbsYvlQzQByzdal0oo3/+4LmZAoqYf413xPWru?= =?us-ascii?Q?PHM1Ye+snUsV7bc5FBOPDTkjv08zNHonEah3tj7Ap7u1RD9ROWEMhpLYrhxf?= =?us-ascii?Q?SnbE40eNjROBkae5/TXoLUnQuIyU9d3x8jvovuf21BeYJbpWi1ie0X2XNoAy?= =?us-ascii?Q?SBUUH6LDx6FI7G/A04jbqTNXJKC2LexoY5i2dNJGy7jmIezJEXkn+Ghe+Wkh?= =?us-ascii?Q?w4bQE/cJaw5cZCsayyL4ULJ/0Cn4U9sBbIZliSmLK/c5vI00dhbVQnBdqFtv?= =?us-ascii?Q?qyHes6ZirScVFmztRF04JQA4fxn301RZuqNHf7XiXSFoJoHAWj/3EhLkida4?= =?us-ascii?Q?R8XOY75xzO8G75qQFRgxL9/fOXZDJJ77zrYjPR5tKU52lS9914gBTMlYqStS?= =?us-ascii?Q?Ykaq9FWPWlZf7+0sBrOUDO3NWSc8KeqiWh4S7oLQXUv4iG4T5pCv+BIGIK4X?= =?us-ascii?Q?OdqCJVI1dd2YEbJfY+6t4Aedr+pK3bpSxH9y22jluNTac/D2KmRIFJsIgtdw?= =?us-ascii?Q?z/1mpBnQqxoGg6vb8992eAZenaClNqO7tCxtcPTVJxtrDpXTOXoavY+oDzVO?= =?us-ascii?Q?djquSyU8a5ubodXBaklCCFqvF5zSAOrMxaZfQHHw3r91YKWYRszsPVqAExiZ?= =?us-ascii?Q?ZdmwHH558IA/aTFTt8XXF7NoW+3NTj/oy4dXBF6oF2w6CbS6V5d7ir2+MNv/?= =?us-ascii?Q?o+NfH0xMa78ndw/1TgX/VbpdgxkVYaprqAMFG5wVsv75JURnpFEneQEDkY+S?= =?us-ascii?Q?af6jmQIfHhU3NKR3/JMqbK+XkZOWNLkuwdYBi1muBGQEh0Y5x5w7M0O7SfC4?= =?us-ascii?Q?M5/UXThwjprtwZKWqKrZNWumublY38hrD4hI36+dbJhd9WBiOTNiyW65I/7b?= =?us-ascii?Q?vkqC7Cxc/05Uzd2CHUmAYdEWQCejUenVeXd1M1s4AA2kvDl9RPQCbRivcEH7?= =?us-ascii?Q?3f1xws8xxUZKOxWvzM/asJrbno9mkbW6+W9K3rRsUlAOt3WvVnfr47euIqBC?= =?us-ascii?Q?onPNGhWvTBpBreaugAqap6cTHw0pZV2cp49CFQntwhyuhBoz9SJ0ULvGL1gw?= =?us-ascii?Q?cGQcxOaWTWdCdG7L5eW77BTPE7iYFpMIyVztQOQ7xhFDDQ9KeCRUiBBldQcp?= =?us-ascii?Q?oxt3DcVTtwIMPDVw8MFJvbmkyDuornO+wLfddMOGQdN+vOvJuH4MdcxMVQ2b?= =?us-ascii?Q?xOeZqTnCYDbqkEiKxxqGxOZ0LFiI30ShKEqsr42VoFdKBHsdvhRZM9Nrcxvw?= =?us-ascii?Q?5HfNBCZR0dMtNnPGjAho0abNZOIKCoVI4uJC5zqPW/EisVGYqEBslQxxpRfr?= =?us-ascii?Q?cA=3D=3D?= X-OriginatorOrg: corigine.com X-MS-Exchange-CrossTenant-Network-Message-Id: 61e8597c-70dd-4fbb-9ed3-08dbced4828c X-MS-Exchange-CrossTenant-AuthSource: PH0PR13MB5568.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2023 05:47:12.4758 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: fe128f2c-073b-4c20-818e-7246a585940c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rXB6DjJkWSAUtaGUSpCukFdHtN385NgJSBiZ3hhVnTuo5iZRrYvNyJ1wq9TDNJRXRgrYXlDAV/3XqEx5yDi9YkyWpf1eTMEXEiL41N56NpM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW5PR13MB5904 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add the logic of remap PCI memory. Signed-off-by: Chaoyong He Signed-off-by: Shujing Dong Reviewed-by: Long Wu Reviewed-by: Peng Zhang --- drivers/vdpa/nfp/nfp_vdpa.c | 65 ++++++++++++++++++++++++++++++++++++- 1 file changed, 64 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c index ef4ae4946a..824b0d681a 100644 --- a/drivers/vdpa/nfp/nfp_vdpa.c +++ b/drivers/vdpa/nfp/nfp_vdpa.c @@ -7,6 +7,7 @@ #include #include +#include #include #include "nfp_vdpa_log.h" @@ -16,6 +17,11 @@ struct nfp_vdpa_dev { struct rte_pci_device *pci_dev; struct rte_vdpa_device *vdev; + + int vfio_container_fd; + int vfio_group_fd; + int vfio_dev_fd; + int iommu_group; }; struct nfp_vdpa_dev_node { @@ -53,12 +59,62 @@ nfp_vdpa_find_node_by_pdev(struct rte_pci_device *pdev) return NULL; } +static int +nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device) +{ + int ret; + char dev_name[RTE_DEV_NAME_MAX_LEN] = {0}; + struct rte_pci_device *pci_dev = device->pci_dev; + + rte_pci_unmap_device(pci_dev); + + rte_pci_device_name(&pci_dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN); + rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name, + &device->iommu_group); + + device->vfio_container_fd = rte_vfio_container_create(); + if (device->vfio_container_fd < 0) + return -1; + + device->vfio_group_fd = rte_vfio_container_group_bind( + device->vfio_container_fd, device->iommu_group); + if (device->vfio_group_fd < 0) + goto container_destroy; + + DRV_VDPA_LOG(DEBUG, "container_fd=%d, group_fd=%d,\n", + device->vfio_container_fd, device->vfio_group_fd); + + ret = rte_pci_map_device(pci_dev); + if (ret != 0) + goto group_unbind; + + device->vfio_dev_fd = rte_intr_dev_fd_get(pci_dev->intr_handle); + + return 0; + +group_unbind: + rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group); +container_destroy: + rte_vfio_container_destroy(device->vfio_container_fd); + + return -1; +} + +static void +nfp_vdpa_vfio_teardown(struct nfp_vdpa_dev *device) +{ + rte_pci_unmap_device(device->pci_dev); + rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group); + rte_vfio_container_destroy(device->vfio_container_fd); +} + struct rte_vdpa_dev_ops nfp_vdpa_ops = { }; static int nfp_vdpa_pci_probe(struct rte_pci_device *pci_dev) { + int ret; struct nfp_vdpa_dev *device; struct nfp_vdpa_dev_node *node; @@ -75,10 +131,14 @@ nfp_vdpa_pci_probe(struct rte_pci_device *pci_dev) device->pci_dev = pci_dev; + ret = nfp_vdpa_vfio_setup(device); + if (ret != 0) + goto free_device; + device->vdev = rte_vdpa_register_device(&pci_dev->device, &nfp_vdpa_ops); if (device->vdev == NULL) { DRV_VDPA_LOG(ERR, "Failed to register device %s", pci_dev->name); - goto free_device; + goto vfio_teardown; } node->device = device; @@ -88,6 +148,8 @@ nfp_vdpa_pci_probe(struct rte_pci_device *pci_dev) return 0; +vfio_teardown: + nfp_vdpa_vfio_teardown(device); free_device: free(device); free_node: @@ -118,6 +180,7 @@ nfp_vdpa_pci_remove(struct rte_pci_device *pci_dev) pthread_mutex_unlock(&vdpa_list_lock); rte_vdpa_unregister_device(device->vdev); + nfp_vdpa_vfio_teardown(device); free(device); free(node); -- 2.39.1