From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A47024320F; Fri, 27 Oct 2023 05:04:21 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3E53142E26; Fri, 27 Oct 2023 05:01:32 +0200 (CEST) Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on2118.outbound.protection.outlook.com [40.107.102.118]) by mails.dpdk.org (Postfix) with ESMTP id 21C0842E45 for ; Fri, 27 Oct 2023 05:01:10 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LZ3DSWa8qrtnKtasFgdrLCl/m+Tn1mbzl1lSA2EOdKWSGca3a5fBhEWW0vZMnDu7yp6WqAxIWWkkUi9wuR5DpxUGW2zFqrgFlEcyXAsK3roM1ouZnpnTMv3j5+rrIIi/Q9hr5jDQhjXcAo59lok5ihHRDSz83TarwCthrmERwnoKFJ88TTFNMRtb4a//MuNSW8XD0lEy7zVBrwUqyUygPhTFph1AD/15ls78QbvpfrdP1SyJy3Jm23xy1DBCoCtICrh73vEBIv6jUWCW4C01IJwdNejn213yQXSbJ10P+pKcADEq1uHQG6RhksNyBNSBMdnH3B0nESY1NDSXb71pgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YB/LcqT05BvqehMbYYSOhkaz8aoA2CxEadovUYwjKU8=; b=ObEEexpq4F508bnyNG6q24YuYM3U8iUCJ4HAMlAbgdjZ8YdTZZ35LJmU/HLIugq1rSF7bX8Clx5jDokty6mQdmBP/TskZsz1N6NtD2XacSoTmz3yRo7YM8ovW1igTQHbhtsJmhFNdX4xuqaAEtnOdRbXQ0ZTTBl2gfOPJVslIhCC4AC26Rl63BpT50Mg+lJAf3zG8CVDGHOCfSH065z9Fd429bwcI6/zIHeKwnmteXT3ICLmSy5qGJeH15o9fTxg4Old0zlLFFws5f1RMjGYD9uQwSkBdkQ5Rb5FclZWq9Pbk2wR0psUAtv1CuZ0dMiM3nOx3YrczYqfyO5IfSvRBA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=corigine.com; dmarc=pass action=none header.from=corigine.com; dkim=pass header.d=corigine.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corigine.onmicrosoft.com; s=selector2-corigine-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YB/LcqT05BvqehMbYYSOhkaz8aoA2CxEadovUYwjKU8=; b=Bwud7/ECKlIQGUiSJSdd2CjWD0vmaT0TyWLhn8hnqfrzOLtbs9PhCdTIxBF8XhwjpB53oDRGIDAAndNihWnPE4uMB1NVgWMNqwXdL/eNAnkKmb5CbXsqusjW56H1gWYgdaYKsn5sHBEqP61RxBw4fQSQCcrUYsWqqOgSUTJ4zgY= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=corigine.com; Received: from SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) by SA0PR13MB4048.namprd13.prod.outlook.com (2603:10b6:806:97::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.24; Fri, 27 Oct 2023 03:01:08 +0000 Received: from SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::45b3:d83c:cff2:4a1b]) by SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::45b3:d83c:cff2:4a1b%6]) with mapi id 15.20.6933.022; Fri, 27 Oct 2023 03:01:08 +0000 From: Chaoyong He To: dev@dpdk.org Cc: oss-drivers@corigine.com, Chaoyong He , Shujing Dong , Long Wu , Peng Zhang Subject: [PATCH v4 22/24] vdpa/nfp: add datapath update Date: Fri, 27 Oct 2023 10:59:59 +0800 Message-Id: <20231027030001.602639-23-chaoyong.he@corigine.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20231027030001.602639-1-chaoyong.he@corigine.com> References: <20231026064324.177531-1-chaoyong.he@corigine.com> <20231027030001.602639-1-chaoyong.he@corigine.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: BYAPR08CA0065.namprd08.prod.outlook.com (2603:10b6:a03:117::42) To SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR13MB5545:EE_|SA0PR13MB4048:EE_ X-MS-Office365-Filtering-Correlation-Id: 696327f6-43af-49c3-437b-08dbd698f781 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: CSWIMOWh++udAOOl9M4ZLW2ph2IzQEiNag4br4z80YrrqprgC76XEGJTHEUMG7mjsyTHQI65RNYfSxb9WiEGCwuDF6aYCuid4JvybYg8X4cS3tzzEek69GEH+Lvcloxq8n/E6qWXy+OKFuUSebPIp62iDl/8FUFrKpkooYR64Ksrs92kK449hw+PVUt5BIWch7ib3UDzP4mh065tIZ0JFpFhBNf0OSdXh6G6gu6lKUXc6FZAtrVY41Ihu9waJwlCGuj+cP77e7RVogCaBBM0Rq/cJuwc/l4zTTGC52JeTwG439l7EuqzwMTVaupDyVfJ6AJ4eS+Bc1wnJdtpxCJinOD1yZZrM3BLqUzsN7XSsDgYbJO6vyGs9ORBBQA4Qxf+S7sNF0wf2pq4n7a0ynrCr0KvKfc2oEgVGN3es0n6PIBWlsBpQlDrvPuSVS16D1gvboEH3L+f+7vZvJIxxyueFToq3IRJ5WEFr2wNNPWFcoACXg54xeqMvOqo/T2acQjQS/nX3dubrMpNQ643DHEuoqbdP7E5xkuZpo8ceTbkDfqGOKn55GPU/9gy7JQ2jPvuEQX3Rt+A3q5rd3CWn2UTX5GZV6OdeMsmyiGTIbRqqdpKMI871+LUKj/HCqhWTDFYHl0jQ99kygH4NWtsjgDovsY80axrF+G/WMShVNKCK9Q= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SJ0PR13MB5545.namprd13.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(376002)(396003)(39840400004)(366004)(346002)(136003)(230922051799003)(64100799003)(186009)(1800799009)(451199024)(6512007)(6666004)(52116002)(6506007)(6486002)(86362001)(36756003)(478600001)(38100700002)(15650500001)(30864003)(2906002)(6916009)(41300700001)(83380400001)(1076003)(107886003)(26005)(2616005)(38350700005)(66946007)(316002)(66556008)(66476007)(4326008)(8936002)(8676002)(44832011)(5660300002)(54906003); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?DBK6HKUkBAK6+x3tvkwQpRx3BNKoh/Mu3txoYiokJ3M6apRDr0Pr5PN4589v?= =?us-ascii?Q?SXsQLDY8baX+NBDSS9bmaalD7U52jNRSPJSRq93Qp5fIwp4s1U+fnUvHiOCz?= =?us-ascii?Q?lH+tV7zHi0C3z2ddRw/ooxtE7aIdCloCO82P1lp3Ti5xMnIguFznjgJhaPiG?= =?us-ascii?Q?YN2Hjam66co/IMtjIfZgPZbhNjtLsFNUh5gHmCa/wNPAmJKcicmVREHmGHSP?= =?us-ascii?Q?4J+lnyLNu0QPvIXd0gs0BhpiR2T0SzTU1P3JlWwOOXHsNvOJb4bYRBt80rdD?= =?us-ascii?Q?9SaJyal5iZOx2ftqoVFj5nsQzlWDUEBtSkWHQYhvzW16YgjA+cunNrbxq1im?= =?us-ascii?Q?wuUDJHoh/flCQzJ4bVndzc8yRHAzz0fLfoUP9aLw2lOUlaLZl1GCgm721wfh?= =?us-ascii?Q?zeL0YBJGK83HSKaZShAH4J7wUY1Ps18DInZqwomH7oXDdZ5nk9lUoBU85uX1?= =?us-ascii?Q?TU6twClvX7lXkjVw70qVYcDICM26Ic3iO9lRmmEeZVJmwK0q+Anm5NuIus6t?= =?us-ascii?Q?vHk1LhFqx+e0BV7BDdeSXM5bjHlh1BOcmbaj6FkzgBixLcY1S8hEUoy6CLED?= =?us-ascii?Q?J7JXoLF6b7LqQaRgBRtkMDs4XuVVDzqsHzaNBREm0uUO0N1cUr8jzE9TM5+T?= =?us-ascii?Q?GXBrgpg8ZS87nDU5yyeyY+D1aXRMh2g3M/dfguRiIW7xf4M61i9vmKnvM7+q?= =?us-ascii?Q?LyZODuKqe+OFhZkzAjirboVJKeDu91gXgjhrEUNpbhBOcZlqkVG+AsOK29FK?= =?us-ascii?Q?P7swKTaFgPIcZdF7PyFD9GSNCrlL80o/OIGNycAur0CvyrUHJbdII11bHKRO?= =?us-ascii?Q?+DONpswemR1WbmFUQ40jld3kgiRziWjbyPJiyOAQuYaMTi2mRSvGBbK/OL1Q?= =?us-ascii?Q?8vJAx52K32KL2KYC25sKG+UbyJe4ix+moZ3JubwhQ2VPCZ5FczUOYbj2UJ9S?= =?us-ascii?Q?cnbmPVzonGN1D5m/y7y/30oNqWtRfBQ/P053OLrMzkERJd2lHF8RPahpIUww?= =?us-ascii?Q?bxijqT+v8yuxws+3aVzzp+HvJg43LVnEco8xLCK+eJg4cz+wVV0+dKbXsaou?= =?us-ascii?Q?aKg/HbcgycTz+Pm6wKxydJW4wc6sZ+BoJZ0/GBEsVVn314jMSnP2Kv2aalTy?= =?us-ascii?Q?WPq5+8UOn/u9AV63z9Seif6Zyg552V0Zj+2xPfjerk5HsUxZaLsyBFq5LNZH?= =?us-ascii?Q?A7O5xc8qmkeagqs9PpsLSkSeeJXVwEB22DYrQFoOkLmmNJUIpLBcs8RTpgDQ?= =?us-ascii?Q?OoiHusEvYQeXYlqMWCZu5Bxb8uuL5ZW+A4wawZsx7wlF4oixc9Jcugkib3FC?= =?us-ascii?Q?XUKhIpz1w0YXmIRRpxFjn7riavpqZOBZGoryzKHIADwmE2kea5Whrcal68is?= =?us-ascii?Q?FVY26UlY0ndRzWieDLzKKotUtBKmMMit21I7G9C/4G9qqD/oHY7kAsiPl63h?= =?us-ascii?Q?SRIymEie4Mb2OiAsg9IvEKx5k3RRDY0h4bCWQqAe58A9XH7YlTDrfoNoHPXY?= =?us-ascii?Q?/uz48sPdEFLE3NDt/9bmvYzHmmQB+sUuY/dSsrwzli0qV0d72OtTvRCNSour?= =?us-ascii?Q?MWtJUIg2LM3jhfiU5aSEie6+4TjF8Va/5S1N+DzoRSHpucok1gomyXDXYJ8Q?= =?us-ascii?Q?QQ=3D=3D?= X-OriginatorOrg: corigine.com X-MS-Exchange-CrossTenant-Network-Message-Id: 696327f6-43af-49c3-437b-08dbd698f781 X-MS-Exchange-CrossTenant-AuthSource: SJ0PR13MB5545.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Oct 2023 03:01:08.1734 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: fe128f2c-073b-4c20-818e-7246a585940c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: RQNQNjdqg2D1Ja4G+EBM48nTA7mRNcWxrOZW9+ytgrwAPUah/GHN7gz0SFgGFAT8nE/evyPZ4OD8rvTw38wCUBHyN9ILKyD/YPETuZU4iHI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR13MB4048 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add the vDPA datapath update logic. Signed-off-by: Chaoyong He Signed-off-by: Shujing Dong Reviewed-by: Long Wu Reviewed-by: Peng Zhang --- drivers/common/nfp/nfp_common_ctrl.h | 1 + drivers/vdpa/nfp/nfp_vdpa.c | 318 +++++++++++++++++++++++++++ drivers/vdpa/nfp/nfp_vdpa_core.c | 78 +++++++ drivers/vdpa/nfp/nfp_vdpa_core.h | 15 ++ 4 files changed, 412 insertions(+) diff --git a/drivers/common/nfp/nfp_common_ctrl.h b/drivers/common/nfp/nfp_common_ctrl.h index 3c8cd916cf..f92ce50fc0 100644 --- a/drivers/common/nfp/nfp_common_ctrl.h +++ b/drivers/common/nfp/nfp_common_ctrl.h @@ -238,6 +238,7 @@ struct nfp_net_fw_ver { #define NFP_NET_CFG_CTRL_IPSEC (0x1 << 1) /**< IPsec offload */ #define NFP_NET_CFG_CTRL_IPSEC_SM_LOOKUP (0x1 << 3) /**< SA short match lookup */ #define NFP_NET_CFG_CTRL_IPSEC_LM_LOOKUP (0x1 << 4) /**< SA long match lookup */ +#define NFP_NET_CFG_CTRL_IN_ORDER (0x1 << 11) /**< Virtio in-order flag */ #define NFP_NET_CFG_CAP_WORD1 0x00a4 diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c index 00d8f7e007..04dd29092b 100644 --- a/drivers/vdpa/nfp/nfp_vdpa.c +++ b/drivers/vdpa/nfp/nfp_vdpa.c @@ -4,6 +4,7 @@ */ #include +#include #include #include @@ -15,6 +16,9 @@ #define NFP_VDPA_DRIVER_NAME nfp_vdpa +#define MSIX_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \ + sizeof(int) * (NFP_VDPA_MAX_QUEUES * 2 + 1)) + struct nfp_vdpa_dev { struct rte_pci_device *pci_dev; struct rte_vdpa_device *vdev; @@ -25,7 +29,15 @@ struct nfp_vdpa_dev { int vfio_dev_fd; int iommu_group; + int vid; uint16_t max_queues; + RTE_ATOMIC(uint32_t) started; + RTE_ATOMIC(uint32_t) dev_attached; + RTE_ATOMIC(uint32_t) running; + rte_spinlock_t lock; + + /** Eventfd for used ring interrupt */ + int intr_fd[NFP_VDPA_MAX_QUEUES * 2]; }; struct nfp_vdpa_dev_node { @@ -112,6 +124,305 @@ nfp_vdpa_vfio_teardown(struct nfp_vdpa_dev *device) rte_vfio_container_destroy(device->vfio_container_fd); } +static int +nfp_vdpa_dma_do_unmap(struct rte_vhost_memory *mem, + uint32_t times, + int vfio_container_fd) +{ + uint32_t i; + int ret = 0; + struct rte_vhost_mem_region *region; + + for (i = 0; i < times; i++) { + region = &mem->regions[i]; + + ret = rte_vfio_container_dma_unmap(vfio_container_fd, + region->host_user_addr, region->guest_phys_addr, + region->size); + if (ret < 0) { + /* Here should not return, even error happened. */ + DRV_VDPA_LOG(ERR, "DMA unmap failed. Times: %u", i); + } + } + + return ret; +} + +static int +nfp_vdpa_dma_do_map(struct rte_vhost_memory *mem, + uint32_t times, + int vfio_container_fd) +{ + int ret; + uint32_t i; + struct rte_vhost_mem_region *region; + + for (i = 0; i < times; i++) { + region = &mem->regions[i]; + + ret = rte_vfio_container_dma_map(vfio_container_fd, + region->host_user_addr, region->guest_phys_addr, + region->size); + if (ret < 0) { + DRV_VDPA_LOG(ERR, "DMA map failed."); + nfp_vdpa_dma_do_unmap(mem, i, vfio_container_fd); + return ret; + } + } + + return 0; +} + +static int +nfp_vdpa_dma_map(struct nfp_vdpa_dev *device, + bool do_map) +{ + int ret; + int vfio_container_fd; + struct rte_vhost_memory *mem = NULL; + + ret = rte_vhost_get_mem_table(device->vid, &mem); + if (ret < 0) { + DRV_VDPA_LOG(ERR, "Failed to get memory layout."); + return ret; + } + + vfio_container_fd = device->vfio_container_fd; + DRV_VDPA_LOG(DEBUG, "vfio_container_fd %d", vfio_container_fd); + + if (do_map) + ret = nfp_vdpa_dma_do_map(mem, mem->nregions, vfio_container_fd); + else + ret = nfp_vdpa_dma_do_unmap(mem, mem->nregions, vfio_container_fd); + + free(mem); + + return ret; +} + +static uint64_t +nfp_vdpa_qva_to_gpa(int vid, + uint64_t qva) +{ + int ret; + uint32_t i; + uint64_t gpa = 0; + struct rte_vhost_memory *mem = NULL; + struct rte_vhost_mem_region *region; + + ret = rte_vhost_get_mem_table(vid, &mem); + if (ret < 0) { + DRV_VDPA_LOG(ERR, "Failed to get memory layout."); + return gpa; + } + + for (i = 0; i < mem->nregions; i++) { + region = &mem->regions[i]; + + if (qva >= region->host_user_addr && + qva < region->host_user_addr + region->size) { + gpa = qva - region->host_user_addr + region->guest_phys_addr; + break; + } + } + + free(mem); + + return gpa; +} + +static int +nfp_vdpa_start(struct nfp_vdpa_dev *device) +{ + int ret; + int vid; + uint16_t i; + uint64_t gpa; + struct rte_vhost_vring vring; + struct nfp_vdpa_hw *vdpa_hw = &device->hw; + + vid = device->vid; + vdpa_hw->nr_vring = rte_vhost_get_vring_num(vid); + + ret = rte_vhost_get_negotiated_features(vid, &vdpa_hw->req_features); + if (ret != 0) + return ret; + + for (i = 0; i < vdpa_hw->nr_vring; i++) { + ret = rte_vhost_get_vhost_vring(vid, i, &vring); + if (ret != 0) + return ret; + + gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.desc); + if (gpa == 0) { + DRV_VDPA_LOG(ERR, "Fail to get GPA for descriptor ring."); + return -1; + } + + vdpa_hw->vring[i].desc = gpa; + + gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.avail); + if (gpa == 0) { + DRV_VDPA_LOG(ERR, "Fail to get GPA for available ring."); + return -1; + } + + vdpa_hw->vring[i].avail = gpa; + + gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.used); + if (gpa == 0) { + DRV_VDPA_LOG(ERR, "Fail to get GPA for used ring."); + return -1; + } + + vdpa_hw->vring[i].used = gpa; + + vdpa_hw->vring[i].size = vring.size; + + ret = rte_vhost_get_vring_base(vid, i, + &vdpa_hw->vring[i].last_avail_idx, + &vdpa_hw->vring[i].last_used_idx); + if (ret != 0) + return ret; + } + + return nfp_vdpa_hw_start(&device->hw, vid); +} + +static void +nfp_vdpa_stop(struct nfp_vdpa_dev *device) +{ + int vid; + uint32_t i; + struct nfp_vdpa_hw *vdpa_hw = &device->hw; + + nfp_vdpa_hw_stop(vdpa_hw); + + vid = device->vid; + for (i = 0; i < vdpa_hw->nr_vring; i++) + rte_vhost_set_vring_base(vid, i, + vdpa_hw->vring[i].last_avail_idx, + vdpa_hw->vring[i].last_used_idx); +} + +static int +nfp_vdpa_enable_vfio_intr(struct nfp_vdpa_dev *device) +{ + int ret; + uint16_t i; + int *fd_ptr; + uint16_t nr_vring; + struct vfio_irq_set *irq_set; + struct rte_vhost_vring vring; + char irq_set_buf[MSIX_IRQ_SET_BUF_LEN]; + + nr_vring = rte_vhost_get_vring_num(device->vid); + + irq_set = (struct vfio_irq_set *)irq_set_buf; + irq_set->argsz = sizeof(irq_set_buf); + irq_set->count = nr_vring + 1; + irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER; + irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX; + irq_set->start = 0; + + fd_ptr = (int *)&irq_set->data; + fd_ptr[RTE_INTR_VEC_ZERO_OFFSET] = rte_intr_fd_get(device->pci_dev->intr_handle); + + for (i = 0; i < nr_vring; i++) + device->intr_fd[i] = -1; + + for (i = 0; i < nr_vring; i++) { + rte_vhost_get_vhost_vring(device->vid, i, &vring); + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; + } + + ret = ioctl(device->vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set); + if (ret != 0) { + DRV_VDPA_LOG(ERR, "Error enabling MSI-X interrupts."); + return -EIO; + } + + return 0; +} + +static int +nfp_vdpa_disable_vfio_intr(struct nfp_vdpa_dev *device) +{ + int ret; + struct vfio_irq_set *irq_set; + char irq_set_buf[MSIX_IRQ_SET_BUF_LEN]; + + irq_set = (struct vfio_irq_set *)irq_set_buf; + irq_set->argsz = sizeof(irq_set_buf); + irq_set->count = 0; + irq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER; + irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX; + irq_set->start = 0; + + ret = ioctl(device->vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set); + if (ret != 0) { + DRV_VDPA_LOG(ERR, "Error disabling MSI-X interrupts."); + return -EIO; + } + + return 0; +} + +static int +update_datapath(struct nfp_vdpa_dev *device) +{ + int ret; + + rte_spinlock_lock(&device->lock); + + if ((rte_atomic_load_explicit(&device->running, rte_memory_order_relaxed) == 0) && + (rte_atomic_load_explicit(&device->started, + rte_memory_order_relaxed) != 0) && + (rte_atomic_load_explicit(&device->dev_attached, + rte_memory_order_relaxed) != 0)) { + ret = nfp_vdpa_dma_map(device, true); + if (ret != 0) + goto unlock_exit; + + ret = nfp_vdpa_enable_vfio_intr(device); + if (ret != 0) + goto dma_map_rollback; + + ret = nfp_vdpa_start(device); + if (ret != 0) + goto disable_vfio_intr; + + rte_atomic_store_explicit(&device->running, 1, rte_memory_order_relaxed); + } else if ((rte_atomic_load_explicit(&device->running, rte_memory_order_relaxed) != 0) && + ((rte_atomic_load_explicit(&device->started, + rte_memory_order_relaxed) != 0) || + (rte_atomic_load_explicit(&device->dev_attached, + rte_memory_order_relaxed) != 0))) { + nfp_vdpa_stop(device); + + ret = nfp_vdpa_disable_vfio_intr(device); + if (ret != 0) + goto unlock_exit; + + ret = nfp_vdpa_dma_map(device, false); + if (ret != 0) + goto unlock_exit; + + rte_atomic_store_explicit(&device->running, 0, rte_memory_order_relaxed); + } + + rte_spinlock_unlock(&device->lock); + return 0; + +disable_vfio_intr: + nfp_vdpa_disable_vfio_intr(device); +dma_map_rollback: + nfp_vdpa_dma_map(device, false); +unlock_exit: + rte_spinlock_unlock(&device->lock); + return ret; +} + struct rte_vdpa_dev_ops nfp_vdpa_ops = { }; @@ -156,6 +467,10 @@ nfp_vdpa_pci_probe(struct rte_pci_device *pci_dev) TAILQ_INSERT_TAIL(&vdpa_dev_list, node, next); pthread_mutex_unlock(&vdpa_list_lock); + rte_spinlock_init(&device->lock); + rte_atomic_store_explicit(&device->started, 1, rte_memory_order_relaxed); + update_datapath(device); + return 0; vfio_teardown: @@ -185,6 +500,9 @@ nfp_vdpa_pci_remove(struct rte_pci_device *pci_dev) device = node->device; + rte_atomic_store_explicit(&device->started, 0, rte_memory_order_relaxed); + update_datapath(device); + pthread_mutex_lock(&vdpa_list_lock); TAILQ_REMOVE(&vdpa_dev_list, node, next); pthread_mutex_unlock(&vdpa_list_lock); diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c index a7e15fa88a..db9b8462b4 100644 --- a/drivers/vdpa/nfp/nfp_vdpa_core.c +++ b/drivers/vdpa/nfp/nfp_vdpa_core.c @@ -5,6 +5,7 @@ #include "nfp_vdpa_core.h" +#include #include #include "nfp_vdpa_log.h" @@ -52,3 +53,80 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw, return 0; } + +static uint32_t +nfp_vdpa_check_offloads(void) +{ + return NFP_NET_CFG_CTRL_SCATTER | + NFP_NET_CFG_CTRL_IN_ORDER; +} + +int +nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw, + int vid) +{ + int ret; + uint32_t update; + uint32_t new_ctrl; + struct timespec wait_tst; + struct nfp_hw *hw = &vdpa_hw->super; + uint8_t mac_addr[RTE_ETHER_ADDR_LEN]; + + nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc); + nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0), rte_log2_u32(vdpa_hw->vring[1].size)); + nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail); + nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used); + + nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc); + nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0), rte_log2_u32(vdpa_hw->vring[0].size)); + nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail); + nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used); + + rte_wmb(); + + nfp_disable_queues(hw); + nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES); + + new_ctrl = nfp_vdpa_check_offloads(); + + nn_cfg_writel(hw, NFP_NET_CFG_MTU, 9216); + nn_cfg_writel(hw, NFP_NET_CFG_FLBUFSZ, 10240); + + /* TODO: Temporary set MAC to fixed value fe:1b:ac:05:a5:22 */ + mac_addr[0] = 0xfe; + mac_addr[1] = 0x1b; + mac_addr[2] = 0xac; + mac_addr[3] = 0x05; + mac_addr[4] = 0xa5; + mac_addr[5] = (0x22 + vid); + + /* Writing new MAC to the specific port BAR address */ + nfp_write_mac(hw, (uint8_t *)mac_addr); + + /* Enable device */ + new_ctrl |= NFP_NET_CFG_CTRL_ENABLE; + + /* Signal the NIC about the change */ + update = NFP_NET_CFG_UPDATE_MACADDR | + NFP_NET_CFG_UPDATE_GEN | + NFP_NET_CFG_UPDATE_RING; + + ret = nfp_reconfig(hw, new_ctrl, update); + if (ret < 0) + return -EIO; + + hw->ctrl = new_ctrl; + + DRV_CORE_LOG(DEBUG, "Enabling the device, sleep 1 seconds..."); + wait_tst.tv_sec = 1; + wait_tst.tv_nsec = 0; + nanosleep(&wait_tst, 0); + + return 0; +} + +void +nfp_vdpa_hw_stop(struct nfp_vdpa_hw *vdpa_hw) +{ + nfp_disable_queues(&vdpa_hw->super); +} diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h b/drivers/vdpa/nfp/nfp_vdpa_core.h index c9403e0ea4..a88de768dd 100644 --- a/drivers/vdpa/nfp/nfp_vdpa_core.h +++ b/drivers/vdpa/nfp/nfp_vdpa_core.h @@ -15,6 +15,15 @@ #define NFP_VDPA_NOTIFY_ADDR_BASE 0x4000 #define NFP_VDPA_NOTIFY_ADDR_INTERVAL 0x1000 +struct nfp_vdpa_vring { + uint64_t desc; + uint64_t avail; + uint64_t used; + uint16_t size; + uint16_t last_avail_idx; + uint16_t last_used_idx; +}; + struct nfp_vdpa_hw { struct nfp_hw super; @@ -22,11 +31,17 @@ struct nfp_vdpa_hw { uint64_t req_features; uint8_t *notify_addr[NFP_VDPA_MAX_QUEUES * 2]; + struct nfp_vdpa_vring vring[NFP_VDPA_MAX_QUEUES * 2]; uint8_t mac_addr[RTE_ETHER_ADDR_LEN]; uint8_t notify_region; + uint8_t nr_vring; }; int nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw, struct rte_pci_device *dev); +int nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw, int vid); + +void nfp_vdpa_hw_stop(struct nfp_vdpa_hw *vdpa_hw); + #endif /* __NFP_VDPA_CORE_H__ */ -- 2.39.1