From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AF1EB43204; Thu, 26 Oct 2023 08:47:20 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5AC7C42E03; Thu, 26 Oct 2023 08:44:32 +0200 (CEST) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2139.outbound.protection.outlook.com [40.107.93.139]) by mails.dpdk.org (Postfix) with ESMTP id 84F3C42E00 for ; Thu, 26 Oct 2023 08:44:24 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EP4POKJx5xMEYR/B5AIv16xrW5kyPUeKfRWoesxniGUzER5tuP92pXUS2mHke7nXiF3rIPTr7S0AG8sEEjadxa0FmRWtrKlQQs0dsk1IaW6Z0CaKkYPdPqGlOMf7bDoTJM6864UjiOo/wC/Kx6vVQjwaQOkTIxRjU8z8hxJgQyRX1SpeId1BsSNPUrNk0ICDEGnJjYxjnS/a8mdeRkKBXRnWIQY0eBoTZrGN1ynXb1jvlH0XXCcBqxVBEXhhd05HBMNU5/TZwbg/JDgR1bygkA2n/2B3Vi50GhqOQJChGmG3RpYUreor1R3FKPZaZEhTwNf/y76wdtYLsg1jP+s/zg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jk/CbMQztnVDf2I4hFCWS4AwJwyfhhq7mTjnyIunuEA=; b=Sh3NaOAdPEX40v6BWBW4voue4Ks2p6HaRR8seTAcK8pb+fsihywRLa1LJiJYB18lkHLGr48rb1QomoTfkQb3lLGVICifo93xiP/8pOEtQ3ikip249h4znnfjkBM27uDeG75zc5Kgd9X4KLfcdR+gdiVHkcaCXxEnExP3bhMHNb0iJz18rQ/ytHO9VSGMVGv2MPUQcXG1dDzQzy1phs0uUYac0JlqgGyW/k22MyAkjab2W0lmHEs31ztFDi8y8Qjqj51dQab0x5/EcJ3z5BE2QvuHM2PJZYIZc32IFtcSepQP3dTXzipJRse59tIqj/AxhajDkgcOdMCIe8Im7+dwCQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=corigine.com; dmarc=pass action=none header.from=corigine.com; dkim=pass header.d=corigine.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corigine.onmicrosoft.com; s=selector2-corigine-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jk/CbMQztnVDf2I4hFCWS4AwJwyfhhq7mTjnyIunuEA=; b=L5vIL6Ac0HaTA+cFJyWNvNCcrNiiXf45N3Z33ZrsdH2yjzNEZ3iK/2yI5FgpS4uWpE+7OQ48gowVfKI7Vk83ScZvUlWQx7g00jDDfdncz+L0UY53hWTt1i4vfsZRtmQJsThYaGH8YpP3nzYAOnYAIanytuYYAR1aBZLcMEBHcLI= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=corigine.com; Received: from SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) by DM4PR13MB5908.namprd13.prod.outlook.com (2603:10b6:8:50::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.19; Thu, 26 Oct 2023 06:44:23 +0000 Received: from SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::45b3:d83c:cff2:4a1b]) by SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::45b3:d83c:cff2:4a1b%6]) with mapi id 15.20.6933.022; Thu, 26 Oct 2023 06:44:22 +0000 From: Chaoyong He To: dev@dpdk.org Cc: oss-drivers@corigine.com, Chaoyong He , Shujing Dong , Long Wu , Peng Zhang Subject: [PATCH v3 20/25] vdpa/nfp: add the logic of remap PCI memory Date: Thu, 26 Oct 2023 14:43:19 +0800 Message-Id: <20231026064324.177531-21-chaoyong.he@corigine.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20231026064324.177531-1-chaoyong.he@corigine.com> References: <20231024022849.3092506-1-chaoyong.he@corigine.com> <20231026064324.177531-1-chaoyong.he@corigine.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: BYAPR07CA0034.namprd07.prod.outlook.com (2603:10b6:a02:bc::47) To SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR13MB5545:EE_|DM4PR13MB5908:EE_ X-MS-Office365-Filtering-Correlation-Id: 6f55e652-ddd2-4191-09db-08dbd5eefcf7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5wahKDUPpnmwdF3thPTVvlAGB0/kMuU8I5+MhqbmdRXbmwXbeANmIgMIrMQA+RO3rsAAvcicPIhinwK887KyFeQwlBlmvNQKy9o4QezC3TuJAH6PanKK8qVrDpg19MA6B8Y3ZXFzk7Jdn6xNXj+MlrjuvYFg1Eao9vYtEMNtDK4iPsff3Fa6SEp8y731P42hugOfYsLMkXp1HvqnGwgi/8kArh+83LPME4VPclsMkfplNHfMnIseAy53k+nMky7K0QaWV2a5ZPquPvTC8lNp+scR2B0arcGK27hVnVWqCnAd2g3V0blwt3/gMzOccgl1qe+N+a4QGrEV9XvH3F+xtxLPQ7V9vlT+FcYPALM4xvhtt/ph38LL6EL94/Jya1IZW7uOcP5BsF/ZSaJM0cyzCYpgPWGBywHmYH0d0yHxUqLqov3vtqhaEgZPfweNSbmxkNJUhoGvEnFLBTPgeeBIqhf2T8pEEZsRt2frXiFN/eUhTxTZlOXH69zRDryV7UP0wrBFOAKsEic3MWuB2NFLiu2O7xEdEIXKNENUwne+v9Cxw0Cyl263+fWVXY4zU4/7si2D9VpYvhw/alcm2iywZnwsbJeHz6i/nDDWA9O5OUGjrPUfroeYTDfD+HhpY31VUtUZPJdd+6SnB6jc6EiSprOgYkEThf2MXIJltKR+naw= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SJ0PR13MB5545.namprd13.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(136003)(376002)(366004)(396003)(346002)(39840400004)(230922051799003)(186009)(1800799009)(451199024)(64100799003)(26005)(478600001)(1076003)(316002)(6666004)(6512007)(86362001)(6506007)(6916009)(66946007)(66556008)(38100700002)(66476007)(54906003)(36756003)(38350700005)(2616005)(6486002)(107886003)(44832011)(2906002)(83380400001)(4326008)(8676002)(8936002)(5660300002)(41300700001)(52116002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?62NA5BLV1x0rkWBA8usOyVgLdiSxuVcbpxZiOnYccAZohtOlmUfVAKUk6j5s?= =?us-ascii?Q?98Huhi3X8FALwdndqFRB+ZZ0QSB1HD42oiqawH3fjpjqMo6PnFkFX7EbjThG?= =?us-ascii?Q?0crUX9h0w3yMtlbSpKvfsyAU8T3Z6gNra5qmTh6x6bCbYVPnGOY8fOJKYvbz?= =?us-ascii?Q?X6RV2IWJyZYYjGSVFqWH787PuwlmTqmcoTZYk/1/NpltJxF0nTvvdBlg1rd0?= =?us-ascii?Q?au3+i4aC/Oxn2IrtWQKdz3J6cwm6y8c4ml0Ms+U2aCgVqVSbcuC2BBwBB55T?= =?us-ascii?Q?EwdtnoAf48x/cuCaVvZss0i50uvG8LDNzVmKLgZF4slGiy6QHxSQlo20KqqG?= =?us-ascii?Q?P4a06tchc1NT3cUBNULgQ00IoE0nWcEctRO6r6KaTmC081wTDa+ZemBfdk+X?= =?us-ascii?Q?NeftopqtSYdcIP5D//rNEtmGODrJTXycHDYND+uSSCbMnQvifjQD6Mxet36C?= =?us-ascii?Q?+B80BoUaSH4sZzeJXO8eA3xeQsBYzbceFDuzh3wlrOJVu+okjGqYqqbThlHG?= =?us-ascii?Q?SgaLTBC8kzsFs6Y16mvdAAy1fdhOnrcsADTt1Hx5y8YzOj4lEYuhEzHowi2Z?= =?us-ascii?Q?C6gEYBfarnSpVycZnn+S4yo/0h1GvK2gdGMwDHVD9TgAaDr3/f/lSbwFrktY?= =?us-ascii?Q?LiDtaEzQyWuFxgoLZFABFz/bbDdGlj0RTXAe2AOKAqhdZqWw86DLgB4AgKUN?= =?us-ascii?Q?TxAPWwase8HaJnCoVu9xGFcMN6lUualKradc6tEImJqc+GyEUjyPWRps+47r?= =?us-ascii?Q?AO8WitHZKZD+g26N6npmuKRWGn/WS5Y5X/r0SKiq18dV2ocbHfM+pk+IK60p?= =?us-ascii?Q?7o/KfszIKVG3z1pLMaJex1V1MtJTkN8sv3Ph595udIh2A7wcLUSoOZlA0cfD?= =?us-ascii?Q?/jkdo5//ySIb0tmrDLJ3X5pBBsfjloq9h2dHMWgSSiBL8eAT+4QNr6NIIOeC?= =?us-ascii?Q?KC7pexFVqrrao6Gx7rQTF5EFclwznLuFhgTd4SO6bPreJ/BUyUjdml6iWGlF?= =?us-ascii?Q?0fJD5qrVB2ho0A2lCFtLbpTxvY2FaWl4da3VHxaIg4QYKnvYo7Qh4md+ppED?= =?us-ascii?Q?Yvyl/1HrjrVAbp+sM2UYRbWO8V/Cq1dQH9QSERF948o3nd4oo1jitR7T+e6O?= =?us-ascii?Q?gaNtdV+F4++KrHaj0dD588o0dqlDP2vc5uvRIsVx7NogU/aQoA/gZXTln5Kz?= =?us-ascii?Q?FKMiBuRL/+pUoqflLlWGKfpD4HjtY5Vu+YTtms7Z59Z9iU4p09NAz4kVFKLI?= =?us-ascii?Q?0ZwjDhacq53lEdlRiYNlJBJ2frZ/xYneFpbRWvL5xM0nM2sxVuFfmXuY+snH?= =?us-ascii?Q?AP+zwHw5YU5zXA04o1nAMX+hUtdiY3aQ5m8gbADRzItMWVY8UNZYCoZ2fc7f?= =?us-ascii?Q?XyFxAZ+1NCjL0YYVKS1VXXNxPYFJzsacu6tip/nQihmdjJhFU407zDNxyl8Q?= =?us-ascii?Q?hPMdi1v4bJrrH2/S8NaU3NaOwo9VdW+a6CYvXHVXlXYIkF1z292QblDCFw0g?= =?us-ascii?Q?FW6GKuumausKk1NPVNOUzSZqT+waBlWoHjEl3svVf/cxHn1WNo6ziocwtX5I?= =?us-ascii?Q?NGqC3ql+HYLDKa+hfEaA+UbAmNkxy2+M8c4sbhLyHuJfx2h57KSGMbUwC1P0?= =?us-ascii?Q?qw=3D=3D?= X-OriginatorOrg: corigine.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6f55e652-ddd2-4191-09db-08dbd5eefcf7 X-MS-Exchange-CrossTenant-AuthSource: SJ0PR13MB5545.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Oct 2023 06:44:22.8818 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: fe128f2c-073b-4c20-818e-7246a585940c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: EKsgyJGCi/20tME3BtkWlxUynxhoo9M8qduklB9GidvVCvQ1pJfp4CyDkvrPCjnsh5xQQMA4kVWsJMmhnf/dtFarjukxjuG1u4wOacRTkTM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR13MB5908 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add the logic of remap PCI memory. Signed-off-by: Chaoyong He Signed-off-by: Shujing Dong Reviewed-by: Long Wu Reviewed-by: Peng Zhang --- drivers/vdpa/nfp/nfp_vdpa.c | 65 ++++++++++++++++++++++++++++++++++++- 1 file changed, 64 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c index ef4ae4946a..824b0d681a 100644 --- a/drivers/vdpa/nfp/nfp_vdpa.c +++ b/drivers/vdpa/nfp/nfp_vdpa.c @@ -7,6 +7,7 @@ #include #include +#include #include #include "nfp_vdpa_log.h" @@ -16,6 +17,11 @@ struct nfp_vdpa_dev { struct rte_pci_device *pci_dev; struct rte_vdpa_device *vdev; + + int vfio_container_fd; + int vfio_group_fd; + int vfio_dev_fd; + int iommu_group; }; struct nfp_vdpa_dev_node { @@ -53,12 +59,62 @@ nfp_vdpa_find_node_by_pdev(struct rte_pci_device *pdev) return NULL; } +static int +nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device) +{ + int ret; + char dev_name[RTE_DEV_NAME_MAX_LEN] = {0}; + struct rte_pci_device *pci_dev = device->pci_dev; + + rte_pci_unmap_device(pci_dev); + + rte_pci_device_name(&pci_dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN); + rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name, + &device->iommu_group); + + device->vfio_container_fd = rte_vfio_container_create(); + if (device->vfio_container_fd < 0) + return -1; + + device->vfio_group_fd = rte_vfio_container_group_bind( + device->vfio_container_fd, device->iommu_group); + if (device->vfio_group_fd < 0) + goto container_destroy; + + DRV_VDPA_LOG(DEBUG, "container_fd=%d, group_fd=%d,\n", + device->vfio_container_fd, device->vfio_group_fd); + + ret = rte_pci_map_device(pci_dev); + if (ret != 0) + goto group_unbind; + + device->vfio_dev_fd = rte_intr_dev_fd_get(pci_dev->intr_handle); + + return 0; + +group_unbind: + rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group); +container_destroy: + rte_vfio_container_destroy(device->vfio_container_fd); + + return -1; +} + +static void +nfp_vdpa_vfio_teardown(struct nfp_vdpa_dev *device) +{ + rte_pci_unmap_device(device->pci_dev); + rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group); + rte_vfio_container_destroy(device->vfio_container_fd); +} + struct rte_vdpa_dev_ops nfp_vdpa_ops = { }; static int nfp_vdpa_pci_probe(struct rte_pci_device *pci_dev) { + int ret; struct nfp_vdpa_dev *device; struct nfp_vdpa_dev_node *node; @@ -75,10 +131,14 @@ nfp_vdpa_pci_probe(struct rte_pci_device *pci_dev) device->pci_dev = pci_dev; + ret = nfp_vdpa_vfio_setup(device); + if (ret != 0) + goto free_device; + device->vdev = rte_vdpa_register_device(&pci_dev->device, &nfp_vdpa_ops); if (device->vdev == NULL) { DRV_VDPA_LOG(ERR, "Failed to register device %s", pci_dev->name); - goto free_device; + goto vfio_teardown; } node->device = device; @@ -88,6 +148,8 @@ nfp_vdpa_pci_probe(struct rte_pci_device *pci_dev) return 0; +vfio_teardown: + nfp_vdpa_vfio_teardown(device); free_device: free(device); free_node: @@ -118,6 +180,7 @@ nfp_vdpa_pci_remove(struct rte_pci_device *pci_dev) pthread_mutex_unlock(&vdpa_list_lock); rte_vdpa_unregister_device(device->vdev); + nfp_vdpa_vfio_teardown(device); free(device); free(node); -- 2.39.1