From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 541054560F;
	Fri, 12 Jul 2024 11:48:49 +0200 (CEST)
Received: from mails.dpdk.org (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 1F0F942EB2;
	Fri, 12 Jul 2024 11:48:45 +0200 (CEST)
Received: from egress-ip42a.ess.de.barracuda.com
 (egress-ip42a.ess.de.barracuda.com [18.185.115.201])
 by mails.dpdk.org (Postfix) with ESMTP id 0E622402AE
 for <dev@dpdk.org>; Fri, 12 Jul 2024 11:48:42 +0200 (CEST)
Received: from EUR05-VI1-obe.outbound.protection.outlook.com
 (mail-vi1eur05lp2168.outbound.protection.outlook.com [104.47.17.168]) by
 mx-outbound8-179.eu-central-1a.ess.aws.cudaops.com (version=TLSv1.2
 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
 Fri, 12 Jul 2024 09:48:40 +0000
ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;
 b=Y2eYoScIA8rz5ZY+m4KcFCvOZ7GnxrqDxiV1lnGToy8591hvQwdUG/PF82cl0dONDra/mXGfarbYaRbXg+/NfijddzJJYu1K6Upb2nZm0IWmPWzfUGI9MYpzeWJkA6gbPMwaQCQ0n9m6C3SUzFtMFkz4oZ/QBykBrfKiMOtLKg3eJfXpZIAm0bhGB1gr/SoMjmwmKdwzzvKlncgyJeAZ8Z6eskSOE4ay8xRp/nPDm1Cj31tqmYRJtW1JkoVhZG3hql1zaOF8ivZksbDjKYAlBzHQUHhnS11vAR12KC1Yj1xBbfqLL5pzT0dKiTTjyWZwhToXa3Mg4Ym2OqhM8+ahjA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
 s=arcselector10001;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=Q/GDHnPIgKGUE2u1gYyI6r6H0jG3S2SEkX5vaXGPaPg=;
 b=nA+nxS+PCyO08XeJ1P2xzv56WNXRbVx7i3+IokL1/tnIGuikH8gG2Nllz1oYjBxfHaqsFZlMzRAxG8K34iFqzmRMLxewzF7Ft+ZuUMiB5kpkW2H3YhlaixfIeaSuV/pq7prerBTxfJ6bqXUvf+z1BTFYnzQM03jriwBCPm+tnjp7MV00lN3Qxhb7X2aONZ/rdyHgaENv/PnhSB+WDUG1i0c9WR6qgWq2V8arq4u9S65WvpzYVUkQD+EszJp8u2hTBNan6fB3EWUrF8RLiLiAA0Llfgx8C5JG229YSzwd18vX2c8m1YFSRFFVf83zSrMSl/JAcUKaYbrm8JqxoxmnrA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=fail (sender ip is
 178.72.21.4) smtp.rcpttodomain=dpdk.org smtp.mailfrom=napatech.com;
 dmarc=fail (p=reject sp=reject pct=100) action=oreject
 header.from=napatech.com; dkim=none (message not signed); arc=none (0)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=napatech.com;
 s=selector1;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=Q/GDHnPIgKGUE2u1gYyI6r6H0jG3S2SEkX5vaXGPaPg=;
 b=ZKAhsrS5XZTGa1NPLXnPF/bFGxb6QimqEFK1dQeJkks4La4Lafce/mzA/QgkRxZ9majhFbjz4Sn6MKbzQQK+QG9e7SpRt1Do3dzoBQ4mrXe7wuB0LDqnLaMzHLW/ofo0UPwj3IFL8w/mIAb0wlogogy1Rkc60+l5dmNURfxo7po=
Received: from AS9PR06CA0518.eurprd06.prod.outlook.com (2603:10a6:20b:49d::13)
 by AM8P190MB0866.EURP190.PROD.OUTLOOK.COM (2603:10a6:20b:1d9::6) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7762.20; Fri, 12 Jul
 2024 09:48:39 +0000
Received: from AM1PEPF000252DD.eurprd07.prod.outlook.com
 (2603:10a6:20b:49d:cafe::64) by AS9PR06CA0518.outlook.office365.com
 (2603:10a6:20b:49d::13) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7762.22 via Frontend
 Transport; Fri, 12 Jul 2024 09:48:39 +0000
X-MS-Exchange-Authentication-Results: spf=fail (sender IP is 178.72.21.4)
 smtp.mailfrom=napatech.com; dkim=none (message not signed)
 header.d=none;dmarc=fail action=oreject header.from=napatech.com;
Received-SPF: Fail (protection.outlook.com: domain of napatech.com does not
 designate 178.72.21.4 as permitted sender) receiver=protection.outlook.com;
 client-ip=178.72.21.4; helo=localhost.localdomain;
Received: from localhost.localdomain (178.72.21.4) by
 AM1PEPF000252DD.mail.protection.outlook.com (10.167.16.55) with Microsoft
 SMTP Server id 15.20.7762.17 via Frontend Transport; Fri, 12 Jul 2024
 09:48:39 +0000
From: Serhii Iliushyk <sil-plv@napatech.com>
To: dev@dpdk.org
Cc: mko-plv@napatech.com, sil-plv@napatech.com, ckm@napatech.com,
 andrew.rybchenko@oktetlabs.ru, ferruh.yigit@amd.com
Subject: [PATCH v7 05/21] net/ntnic: add VFIO module
Date: Fri, 12 Jul 2024 11:48:10 +0200
Message-ID: <20240712094831.861260-5-sil-plv@napatech.com>
X-Mailer: git-send-email 2.45.0
In-Reply-To: <20240712094831.861260-1-sil-plv@napatech.com>
References: <20240530144929.4127931-1-sil-plv@napatech.com>
 <20240712094831.861260-1-sil-plv@napatech.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-EOPAttributedMessage: 0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: AM1PEPF000252DD:EE_|AM8P190MB0866:EE_
Content-Type: text/plain
X-MS-Office365-Filtering-Correlation-Id: c9a52a8f-9e90-43d9-2329-08dca257ce99
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
 ARA:13230040|376014|36860700013|82310400026|1800799024; 
X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?HiPSii9gScprdsbqiMffettlC83HfAqK4Xc3uR1PFABMruQCfEWCUpOk/BTN?=
 =?us-ascii?Q?cXGtg7ireXE1FV3+yBwBB7k8tZpI3zgxA81O/Ejf+HVdn1WznlNJVEFsEHzM?=
 =?us-ascii?Q?yjJWQrCzWLfEYPp/tcaVq82w4gH+LoaRPojJhhGbTZ3oni/bPSWz101EWNEL?=
 =?us-ascii?Q?WpYmxyHabCwYhGW45/5wOn+JGrtbO4rJTw/sm4kMuYq28sjdLsuIEn94GQ3z?=
 =?us-ascii?Q?V+EYpqSpCZ6welwgJKzOcRhR19z1xouAYAZqsl7ErdQ7rNmznFDMLQA7FUpR?=
 =?us-ascii?Q?gFKbk+axT2V6gLwkEyOjfj8s0yKd8hPt0WQdCQRAzdXdUaB03ydgJQayEUon?=
 =?us-ascii?Q?LTDmnoMPAjGHT0X4RlM+AM/iqKSJ3t6AaXdcXPiuHegfcMNgCTuTq/UMcfzY?=
 =?us-ascii?Q?nSEqrSZ1aiUOPQrDFkU8T6R9VWvOXL4nzCYBE3eFDNsUBHI8hEA6wW9VS7v1?=
 =?us-ascii?Q?T4EJZ684XZ5KBtlwVgHSYtumL9Ohl5/Fq0lNmQOlN/UzIBtRi0AzxyGhzouT?=
 =?us-ascii?Q?ud40XO7LWL/ZCeux+Bux8/fofg6qqPtL/LOkaACeSYFuLEsRLtN26wFftNtO?=
 =?us-ascii?Q?3WJHDTgH7zMlJ95oAeRJsysxWiw1btYp+nTsQ4ipKavb6kcwhcXQXSW3knG3?=
 =?us-ascii?Q?g6VVNZqLtO/745PXSTMoz+OEGS+EsFgOq2Tg106XGHGy1mz1S0BPmCg6nplz?=
 =?us-ascii?Q?iNJKd7g9vNKjmHibf8irVzQK94QDpltoeQ0o0NPQ56BaZelLjRfMTQq5ePSg?=
 =?us-ascii?Q?j/ZiRCuRlewcqZyZKmufzeNuJ8aQPAWTYhvMKZ7VNDo2uR9WSqCfOjYYukgg?=
 =?us-ascii?Q?NiCjHX8Rm+gJquG8yqSzy1jIZrQfdHZMJFiocL1MIoljtWvoDS4qfCc9tkj7?=
 =?us-ascii?Q?rmpjVSeRhbgYfN7KZjFuoHNubmMOajTcRtWRXFCozveIAC3c+psrv5plWdsc?=
 =?us-ascii?Q?81hFCa9dxnHAv3wYVZRA+nlE9X8smV0sJ4H1LTZ0aH8tJssfEwiM8Sbxb64e?=
 =?us-ascii?Q?H0HjVT0z2mC7jKhDLhc95nmwIo11/aaPrdgTSK4OCNhuuLRqQzsRHahr7gH0?=
 =?us-ascii?Q?rDjKqZ66M/5014gP4b7nEpkNI7yj1a/UUit646NxspZeCQp1KPynGVkjFI6m?=
 =?us-ascii?Q?tY8qs/LYFVXWc4x7FRJcJFY48PZcwgyMNP5JNeYnkMNAwR+dMf9KvfQ+8xOa?=
 =?us-ascii?Q?5WolL5QDJr8mJWvkrYg0KWdd1k5ohgoxK1q2QWEKmptAMl01hyMt4DQrJ3jn?=
 =?us-ascii?Q?6pc098D4DQR75HUHPLPpbmiK5+OeI2RExhx68geFCgjoXAQa/RJskhKx19iF?=
 =?us-ascii?Q?F/KJKOAX0sfVa6AbMQMnpZ/LY0z592G0Q7VU75VKDSrSJQmGnJEwQ8xve6ks?=
 =?us-ascii?Q?G5fVsEK7rqEAPfBTYBwy8O2F9iaFF/pQNxqY4TyrRD/tB90XeOtKI6jnFrsc?=
 =?us-ascii?Q?U1yoF4PiVua33vVLik6S4sf+gWaEVQgt?=
X-Forefront-Antispam-Report: CIP:178.72.21.4; CTRY:DK; LANG:en; SCL:1; SRV:;
 IPV:NLI; SFV:NSPM; H:localhost.localdomain; PTR:InfoDomainNonexistent;
 CAT:NONE; SFS:(13230040)(376014)(36860700013)(82310400026)(1800799024);
 DIR:OUT; SFP:1102; 
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: uiJjeftdqLeud4GgOaqg0CVrEbbXOc0QEYoche5tCvftPE0ILgTJIGCwfOqs9afD03HDcbcTOaG1H+/4WYZRBUb3C2ufAYfzXJ4NSQm6aplMah0P0aZi0+8XZ8l/1fw9auP3g5HA54PZ1z7Xb+ZNAj0tkLR5P1BW4w+jRj0clfox9QZXuPjqwYYVoHJLT+E9h61Su0ZUrMh7oyuaCNJMgXgJ+gX9CYpxCmfUpVRh4U/t7VWG15xQ+VNajqpV6ce3y6myGsjtxxW9gTxuLom35HrKQjIB4/ZnIZm1StPzA05P2+epZn9+6uR31a/soyy2K5+IPAnDL6bA8dGUjsxV1OUqOmOTjoOyVEaa+Zi/Soq6Oa8ZD8jkTUdoDoBSMB1Zy9s04ajr9sTKJFnq1McaiWXvZK2aNuj7cfnKwNMjiecXyWCXwV7Aw0GUkPM9U1ysc8OXLosU+gXTA5i4Kow42HSUkoNwdbtLE71OUSDGldV7s0CRXpiTFr4mmOpzMYGUVue7pjournMgBBWlLZFqqW/4p+CF8uQ6vb928gJoBf6gDLH7xIVf3UeRWioTi1bOKEknwml8aQyGWETMHUyId/0TRW74xwbci+AIxTtd+wUnd5JhGWOIkEo9XJiQScL+lAtGAwmdbvWGOECspvTWWw==
X-OriginatorOrg: napatech.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jul 2024 09:48:39.0358 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: c9a52a8f-9e90-43d9-2329-08dca257ce99
X-MS-Exchange-CrossTenant-Id: c4540d0b-728a-4233-9da5-9ea30c7ec3ed
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=c4540d0b-728a-4233-9da5-9ea30c7ec3ed; Ip=[178.72.21.4];
 Helo=[localhost.localdomain]
X-MS-Exchange-CrossTenant-AuthSource: AM1PEPF000252DD.eurprd07.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8P190MB0866
X-BESS-ID: 1720777720-302227-12691-8266-1
X-BESS-VER: 2019.1_20240702.1505
X-BESS-Apparent-Source-IP: 104.47.17.168
X-BESS-Parts: H4sIAAAAAAACA4uuVkqtKFGyUioBkjpK+cVKVoaGBkbGQGYGUDQ5NS3RzDTJxN
 jI0sLIzMDQLCk1LcncMs3YwMLU2NjCVKk2FgDLrVYUQgAAAA==
X-BESS-Outbound-Spam-Score: 0.00
X-BESS-Outbound-Spam-Report: Code version 3.2,
 rules version 3.2.2.257571 [from 
 cloudscan21-76.eu-central-1b.ess.aws.cudaops.com]
 Rule breakdown below
 pts rule name              description
 ---- ---------------------- --------------------------------
 0.00 BSF_BESS_OUTBOUND      META: BESS Outbound 
X-BESS-Outbound-Spam-Status: SCORE=0.00 using account:ESS113687 scores of
 KILL_LEVEL=7.0 tests=BSF_BESS_OUTBOUND
X-BESS-BRTS-Status: 1
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org

Adds VFIO functionality and the DMA it requires.
The VFIO context is initialized during ntnic ethdev startup.

Signed-off-by: Serhii Iliushyk <sil-plv@napatech.com>
---
 drivers/net/ntnic/meson.build      |   1 +
 drivers/net/ntnic/ntnic_ethdev.c   |  23 +++
 drivers/net/ntnic/ntnic_vfio.c     | 235 +++++++++++++++++++++++++++++
 drivers/net/ntnic/ntnic_vfio.h     |  29 ++++
 drivers/net/ntnic/ntutil/nt_util.c |  65 ++++++++
 5 files changed, 353 insertions(+)
 create mode 100644 drivers/net/ntnic/ntnic_vfio.c
 create mode 100644 drivers/net/ntnic/ntnic_vfio.h

diff --git a/drivers/net/ntnic/meson.build b/drivers/net/ntnic/meson.build
index c7f5a4efc7..bf86a0a098 100644
--- a/drivers/net/ntnic/meson.build
+++ b/drivers/net/ntnic/meson.build
@@ -18,5 +18,6 @@ includes = [
 sources = files(
     'ntlog/ntlog.c',
     'ntutil/nt_util.c',
+    'ntnic_vfio.c',
     'ntnic_ethdev.c',
 )
diff --git a/drivers/net/ntnic/ntnic_ethdev.c b/drivers/net/ntnic/ntnic_ethdev.c
index 26700eba6c..184bb43d92 100644
--- a/drivers/net/ntnic/ntnic_ethdev.c
+++ b/drivers/net/ntnic/ntnic_ethdev.c
@@ -8,10 +8,18 @@
 #include <rte_bus_pci.h>
 #include <ethdev_pci.h>
 
+#include <rte_kvargs.h>
+#include <rte_eal.h>
+#include <rte_dev.h>
+#include <rte_vfio.h>
+
 #include "ntlog.h"
 
+#include "ntnic_vfio.h"
 #include "nt_util.h"
 
+#define EXCEPTION_PATH_HID 0
+
 static const struct rte_pci_id nthw_pci_id_map[] = {
 	{
 		.vendor_id = 0,
@@ -21,12 +29,24 @@ static const struct rte_pci_id nthw_pci_id_map[] = {
 static int
 nthw_pci_dev_init(struct rte_pci_device *pci_dev)
 {
+	nt_vfio_init();
+
 	uint32_t n_port_mask = -1;	/* All ports enabled by default */
 	int n_phy_ports;
 	NT_LOG_DBGX(DEBUG, NTNIC, "Dev %s PF #%i Init : %02x:%02x:%i\n", pci_dev->name,
 		pci_dev->addr.function, pci_dev->addr.bus, pci_dev->addr.devid,
 		pci_dev->addr.function);
 
+
+	/* Setup VFIO context */
+	int vfio = nt_vfio_setup(pci_dev);
+
+	if (vfio < 0) {
+		NT_LOG_DBGX(ERR, TNIC, "%s: vfio_setup error %d\n",
+			(pci_dev->name[0] ? pci_dev->name : "NA"), -1);
+		return -1;
+	}
+
 	n_phy_ports = 0;
 
 	for (int n_intf_no = 0; n_intf_no < n_phy_ports; n_intf_no++) {
@@ -86,6 +106,8 @@ nthw_pci_dev_deinit(struct rte_eth_dev *eth_dev __rte_unused)
 				continue; /* port already released */
 			rte_eth_dev_release_port(eth_dev);
 		}
+
+		nt_vfio_remove(EXCEPTION_PATH_HID);
 		return 0;
 }
 
@@ -150,3 +172,4 @@ static struct rte_pci_driver rte_nthw_pmd = {
 };
 
 RTE_PMD_REGISTER_PCI(net_ntnic, rte_nthw_pmd);
+RTE_PMD_REGISTER_KMOD_DEP(net_ntnic, "* vfio-pci");
diff --git a/drivers/net/ntnic/ntnic_vfio.c b/drivers/net/ntnic/ntnic_vfio.c
new file mode 100644
index 0000000000..f4433152b7
--- /dev/null
+++ b/drivers/net/ntnic/ntnic_vfio.c
@@ -0,0 +1,235 @@
+/*
+ * SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Napatech A/S
+ */
+
+#include <sys/ioctl.h>
+
+#include <rte_malloc.h>
+#include <rte_memory.h>
+#include <rte_vfio.h>
+#include <rte_dev.h>
+#include <rte_bus_pci.h>
+#include <rte_spinlock.h>
+
+#include <ntlog.h>
+#include <nt_util.h>
+#include "ntnic_vfio.h"
+
+#define ONE_G_SIZE 0x40000000
+#define ONE_G_MASK (ONE_G_SIZE - 1)
+#define START_VF_IOVA 0x220000000000
+
+int
+nt_vfio_vf_num(const struct rte_pci_device *pdev)
+{
+	return ((pdev->addr.devid & 0x1f) << 3) + ((pdev->addr.function) & 0x7);
+}
+
+/* Internal API */
+struct vfio_dev {
+	int container_fd;
+	int group_fd;
+	int dev_fd;
+	uint64_t iova_addr;
+};
+
+static struct vfio_dev vfio_list[256];
+
+static struct vfio_dev *
+vfio_get(int vf_num)
+{
+	if (vf_num < 0 || vf_num > 255)
+		return NULL;
+
+	return &vfio_list[vf_num];
+}
+
+/* External API */
+int
+nt_vfio_setup(struct rte_pci_device *dev)
+{
+	char devname[RTE_DEV_NAME_MAX_LEN] = { 0 };
+	int iommu_group_num;
+	int vf_num;
+	struct vfio_dev *vfio;
+
+	NT_LOG(INF, NTNIC, "NT VFIO device setup %s\n", dev->name);
+
+	vf_num = nt_vfio_vf_num(dev);
+
+	vfio = vfio_get(vf_num);
+
+	if (vfio == NULL) {
+		NT_LOG(ERR, NTNIC, "VFIO device setup failed. Illegal device id\n");
+		return -1;
+	}
+
+	vfio->dev_fd = -1;
+	vfio->group_fd = -1;
+	vfio->container_fd = -1;
+	vfio->iova_addr = START_VF_IOVA;
+
+	rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
+	rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname, &iommu_group_num);
+
+	if (vf_num == 0) {
+		/* use default container for pf0 */
+		vfio->container_fd = RTE_VFIO_DEFAULT_CONTAINER_FD;
+
+	} else {
+		vfio->container_fd = rte_vfio_container_create();
+
+		if (vfio->container_fd < 0) {
+			NT_LOG(ERR, NTNIC,
+				"VFIO device setup failed. VFIO container creation failed.\n");
+			return -1;
+		}
+	}
+
+	vfio->group_fd = rte_vfio_container_group_bind(vfio->container_fd, iommu_group_num);
+
+	if (vfio->group_fd < 0) {
+		NT_LOG(ERR, NTNIC,
+			"VFIO device setup failed. VFIO container group bind failed.\n");
+		goto err;
+	}
+
+	if (vf_num > 0) {
+		if (rte_pci_map_device(dev)) {
+			NT_LOG(ERR, NTNIC,
+				"Map VFIO device failed. is the vfio-pci driver loaded?\n");
+			goto err;
+		}
+	}
+
+	vfio->dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+
+	NT_LOG(DBG, NTNIC,
+		"%s: VFIO id=%d, dev_fd=%d, container_fd=%d, group_fd=%d, iommu_group_num=%d\n",
+		dev->name, vf_num, vfio->dev_fd, vfio->container_fd, vfio->group_fd,
+		iommu_group_num);
+
+	return vf_num;
+
+err:
+
+	if (vfio->container_fd != RTE_VFIO_DEFAULT_CONTAINER_FD)
+		rte_vfio_container_destroy(vfio->container_fd);
+
+	return -1;
+}
+
+int
+nt_vfio_remove(int vf_num)
+{
+	struct vfio_dev *vfio;
+
+	NT_LOG(DBG, NTNIC, "NT VFIO device remove VF=%d\n", vf_num);
+
+	vfio = vfio_get(vf_num);
+
+	if (!vfio) {
+		NT_LOG(ERR, NTNIC, "VFIO device remove failed. Illegal device id\n");
+		return -1;
+	}
+
+	rte_vfio_container_destroy(vfio->container_fd);
+	return 0;
+}
+
+int
+nt_vfio_dma_map(int vf_num, void *virt_addr, uint64_t *iova_addr, uint64_t size)
+{
+	uint64_t gp_virt_base;
+	uint64_t gp_offset;
+
+	if (size == ONE_G_SIZE) {
+		gp_virt_base = (uint64_t)virt_addr & ~ONE_G_MASK;
+		gp_offset = (uint64_t)virt_addr & ONE_G_MASK;
+
+	} else {
+		gp_virt_base = (uint64_t)virt_addr;
+		gp_offset = 0;
+	}
+
+	struct vfio_dev *vfio;
+
+	vfio = vfio_get(vf_num);
+
+	if (vfio == NULL) {
+		NT_LOG(ERR, NTNIC, "VFIO MAP: VF number %d invalid\n", vf_num);
+		return -1;
+	}
+
+	NT_LOG(DBG, NTNIC,
+		"VFIO MMAP VF=%d VirtAddr=%p HPA=%" PRIX64 " VirtBase=%" PRIX64
+		" IOVA Addr=%" PRIX64 " size=%" PRIX64 "\n",
+		vf_num, virt_addr, rte_malloc_virt2iova(virt_addr), gp_virt_base, vfio->iova_addr,
+		size);
+
+	int res = rte_vfio_container_dma_map(vfio->container_fd, gp_virt_base, vfio->iova_addr,
+			size);
+
+	NT_LOG(DBG, NTNIC, "VFIO MMAP res %i, container_fd %i, vf_num %i\n", res,
+		vfio->container_fd, vf_num);
+
+	if (res) {
+		NT_LOG(ERR, NTNIC, "rte_vfio_container_dma_map failed: res %d\n", res);
+		return -1;
+	}
+
+	*iova_addr = vfio->iova_addr + gp_offset;
+
+	vfio->iova_addr += ONE_G_SIZE;
+
+	return 0;
+}
+
+int
+nt_vfio_dma_unmap(int vf_num, void *virt_addr, uint64_t iova_addr, uint64_t size)
+{
+	uint64_t gp_virt_base;
+	struct vfio_dev *vfio;
+
+	if (size == ONE_G_SIZE) {
+		uint64_t gp_offset;
+		gp_virt_base = (uint64_t)virt_addr & ~ONE_G_MASK;
+		gp_offset = (uint64_t)virt_addr & ONE_G_MASK;
+		iova_addr -= gp_offset;
+
+	} else {
+		gp_virt_base = (uint64_t)virt_addr;
+	}
+
+	vfio = vfio_get(vf_num);
+
+	if (vfio == NULL) {
+		NT_LOG(ERR, NTNIC, "VFIO UNMAP: VF number %d invalid\n", vf_num);
+		return -1;
+	}
+
+	if (vfio->container_fd == -1)
+		return 0;
+
+	int res = rte_vfio_container_dma_unmap(vfio->container_fd, gp_virt_base, iova_addr, size);
+
+	if (res != 0) {
+		NT_LOG(ERR, NTNIC,
+			"VFIO UNMMAP FAILED! res %i, container_fd %i, vf_num %i, virt_base=%" PRIX64
+			", IOVA=%" PRIX64 ", size=%" PRIX64 "\n",
+			res, vfio->container_fd, vf_num, gp_virt_base, iova_addr, size);
+		return -1;
+	}
+
+	return 0;
+}
+
+void
+nt_vfio_init(void)
+{
+	struct nt_util_vfio_impl s = { .vfio_dma_map = nt_vfio_dma_map,
+		       .vfio_dma_unmap = nt_vfio_dma_unmap
+	};
+	nt_util_vfio_init(&s);
+}
diff --git a/drivers/net/ntnic/ntnic_vfio.h b/drivers/net/ntnic/ntnic_vfio.h
new file mode 100644
index 0000000000..69fef7923d
--- /dev/null
+++ b/drivers/net/ntnic/ntnic_vfio.h
@@ -0,0 +1,29 @@
+/*
+ * SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Napatech A/S
+ */
+
+#ifndef _NTNIC_VFIO_H_
+#define _NTNIC_VFIO_H_
+
+#include <rte_dev.h>
+#include <rte_bus_pci.h>
+#include <ethdev_pci.h>
+
+void
+nt_vfio_init(void);
+
+int
+nt_vfio_setup(struct rte_pci_device *dev);
+int
+nt_vfio_remove(int vf_num);
+
+int
+nt_vfio_dma_map(int vf_num, void *virt_addr, uint64_t *iova_addr, uint64_t size);
+int
+nt_vfio_dma_unmap(int vf_num, void *virt_addr, uint64_t iova_addr, uint64_t size);
+
+/* Find device (PF/VF) number from device address */
+int
+nt_vfio_vf_num(const struct rte_pci_device *dev);
+#endif	/* _NTNIC_VFIO_H_ */
diff --git a/drivers/net/ntnic/ntutil/nt_util.c b/drivers/net/ntnic/ntutil/nt_util.c
index 5395bf6993..53c39ef112 100644
--- a/drivers/net/ntnic/ntutil/nt_util.c
+++ b/drivers/net/ntnic/ntutil/nt_util.c
@@ -15,6 +15,8 @@
 #include "ntlog.h"
 #include "nt_util.h"
 
+static struct nt_util_vfio_impl vfio_cb;
+
 /* uses usleep which schedules out the calling thread */
 void nt_os_wait_usec(int val)
 {
@@ -31,3 +33,66 @@ uint64_t nt_util_align_size(uint64_t size)
 {
 	return 1 << rte_log2_u64(size);
 }
+
+void nt_util_vfio_init(struct nt_util_vfio_impl *impl)
+{
+	vfio_cb = *impl;
+}
+
+struct nt_dma_s *nt_dma_alloc(uint64_t size, uint64_t align, int numa)
+{
+	int res;
+	struct nt_dma_s *vfio_addr;
+
+	vfio_addr = rte_malloc(NULL, sizeof(struct nt_dma_s), 0);
+
+	if (!vfio_addr) {
+		NT_LOG(ERR, GENERAL, "VFIO rte_malloc failed\n");
+		return NULL;
+	}
+
+	void *addr = rte_malloc_socket(NULL, size, align, numa);
+
+	if (!addr) {
+		rte_free(vfio_addr);
+		NT_LOG(ERR, GENERAL, "VFIO rte_malloc_socket failed\n");
+		return NULL;
+	}
+
+	res = vfio_cb.vfio_dma_map(0, addr, &vfio_addr->iova, nt_util_align_size(size));
+
+	if (res != 0) {
+		rte_free(addr);
+		rte_free(vfio_addr);
+		NT_LOG(ERR, GENERAL, "VFIO nt_dma_map failed\n");
+		return NULL;
+	}
+
+	vfio_addr->addr = (uint64_t)addr;
+	vfio_addr->size = nt_util_align_size(size);
+
+	NT_LOG(DBG, GENERAL,
+		"VFIO DMA alloc addr=%" PRIX64 ", iova=%" PRIX64
+		", size=%" PRIX64 "align=0x%" PRIX64 "\n",
+		vfio_addr->addr, vfio_addr->iova, vfio_addr->size, align);
+
+	return vfio_addr;
+}
+
+void nt_dma_free(struct nt_dma_s *vfio_addr)
+{
+	NT_LOG(DBG, GENERAL, "VFIO DMA free addr=%" PRIX64 ", iova=%" PRIX64 ", size=%" PRIX64 "\n",
+		vfio_addr->addr, vfio_addr->iova, vfio_addr->size);
+
+	int res = vfio_cb.vfio_dma_unmap(0, (void *)vfio_addr->addr, vfio_addr->iova,
+			vfio_addr->size);
+
+	if (res != 0) {
+		NT_LOG(WRN, GENERAL,
+			"VFIO DMA free FAILED addr=%" PRIX64 ", iova=%" PRIX64 ", size=%" PRIX64 "\n",
+			vfio_addr->addr, vfio_addr->iova, vfio_addr->size);
+	}
+
+	rte_free((void *)(vfio_addr->addr));
+	rte_free(vfio_addr);
+}
-- 
2.45.0