From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id C9A5BA0C43;
	Thu, 30 Sep 2021 19:41:41 +0200 (CEST)
Received: from [217.70.189.124] (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 1785E4117A;
	Thu, 30 Sep 2021 19:40:04 +0200 (CEST)
Received: from NAM11-DM6-obe.outbound.protection.outlook.com
 (mail-dm6nam11on2066.outbound.protection.outlook.com [40.107.223.66])
 by mails.dpdk.org (Postfix) with ESMTP id 643714067E
 for <dev@dpdk.org>; Thu, 30 Sep 2021 19:29:18 +0200 (CEST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=I1mFT5+riwgcILHHBYf2t+3oJUyRZwnhvCZmWxeMGjJIc2t995m2Lc+J4bWMBgHCOXLQC6AF9kPJADKkgrna0KcUW7LK3/nkN42evisO0L2VXpRZvUzeCcb4if6bc4Wzf2amKhol97sFjFrZtMKfTyEakHvgl7S2vi5EFqqwuz2sO6/c9Rrx2zksbVfDrcXdzMw0/0dlmU0sK8RcUg7LR01yBpX3V7DuVugx7zTdi849r5ve9ZBeOXBkHg5dR2jyDWqQeldiUlWJNZG2niWe7aOVcT4i3f4rl/aqwA/ImxXrTfOXqM4hSM3MwOFKz/OtGkdO/rIBq38ehIdyDiH3JQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; 
 bh=MsmuG4KRgijvQq9rusL9PSH54KRLJvk+H5bmN+m5cDA=;
 b=L4Lit16pFB/uPAMma0/+tT2G0nTFEVr6dvoNw+ppnbc3+yEBIMosN+DmPB4bYfDLqVixoIX7F4f7aYBuovpC1weixkL+aKdY5Soi4EKgyynKY7kBAu9Qb3jtyqvK35Awl+1s7duR5ZcBpnA31TGW3LaUuLkPw6p0RwLZTv3edQRMWaXKiFcenXJvdzVARddHwHLi8JJhEXmGpKq6138EbHFIvC2rmFyt8JOhVEWnGmcJGvRBfzsvpvzdnE3S8VGxkCQFvEFpr9l0cBwzGN3hC3rO+akKIz7+TO0SGlnXjC/vCZQ7gdLw8zq3YTx9xTGz+3oIhygIORKX0mrQlNzhlw==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is
 216.228.112.34) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com;
 dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; 
 dkim=none (message not signed); arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=MsmuG4KRgijvQq9rusL9PSH54KRLJvk+H5bmN+m5cDA=;
 b=bHLCC1oC137WpVZBcJEOhigFvlzff7KuA1buEUuSyFKqoQJyAOUEJw8hHYss40J8i9AokOdbzGvkB1vlAKp2FwIw9vUlnk14Ntsz5hayWlkLCnRwmnY1O8hKUJHh2wR54u1GUg8Xj1lZzKPB64y14l3MPEK6Uw8sJq9+8ZmG6LmLu0VCTF0ASnzTcKMGgsVMm3qXWEH51nwPSNf4/+fT3P5jvPjDDoqdotraQ0nb4JcyBaEy3QGQErrF5uK+OebmTM/uZszqrtiAFCjAPoH6/h5zeg4kHq+4QCzQNFa5vGlCCsFb/BBx6dmF/nrQcEHYp/1XymSxTAz4yzCaUzQmYQ==
Received: from BN6PR22CA0066.namprd22.prod.outlook.com (2603:10b6:404:ca::28)
 by MN2PR12MB2973.namprd12.prod.outlook.com (2603:10b6:208:cc::19)
 with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4566.15; Thu, 30 Sep
 2021 17:29:16 +0000
Received: from BN8NAM11FT036.eop-nam11.prod.protection.outlook.com
 (2603:10b6:404:ca:cafe::d1) by BN6PR22CA0066.outlook.office365.com
 (2603:10b6:404:ca::28) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4566.14 via Frontend
 Transport; Thu, 30 Sep 2021 17:29:16 +0000
X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34)
 smtp.mailfrom=nvidia.com; monjalon.net; dkim=none (message not signed)
 header.d=none;monjalon.net; dmarc=pass action=none header.from=nvidia.com;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.112.34 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.112.34; helo=mail.nvidia.com;
Received: from mail.nvidia.com (216.228.112.34) by
 BN8NAM11FT036.mail.protection.outlook.com (10.13.177.168) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id
 15.20.4566.14 via Frontend Transport; Thu, 30 Sep 2021 17:29:15 +0000
Received: from nvidia.com (172.20.187.5) by HQMAIL107.nvidia.com
 (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Thu, 30 Sep
 2021 17:29:08 +0000
From: <michaelba@nvidia.com>
To: <dev@dpdk.org>
CC: Matan Azrad <matan@nvidia.com>, Thomas Monjalon <thomas@monjalon.net>,
 Michael Baum <michaelba@oss.nvidia.com>
Date: Thu, 30 Sep 2021 20:28:21 +0300
Message-ID: <20210930172822.1949969-18-michaelba@nvidia.com>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20210930172822.1949969-1-michaelba@nvidia.com>
References: <20210930172822.1949969-1-michaelba@nvidia.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
X-Originating-IP: [172.20.187.5]
X-ClientProxiedBy: HQMAIL107.nvidia.com (172.20.187.13) To
 HQMAIL107.nvidia.com (172.20.187.13)
X-EOPAttributedMessage: 0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 0dc54a87-22c1-467f-8b76-08d98437d3ad
X-MS-TrafficTypeDiagnostic: MN2PR12MB2973:
X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr
X-Microsoft-Antispam-PRVS: <MN2PR12MB2973BE2998A442628F13449BCCAA9@MN2PR12MB2973.namprd12.prod.outlook.com>
X-MS-Oob-TLC-OOBClassifiers: OLM:2201;
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: 9B6lpBzUAK8BF2WdhoP0mflJZcsWv9M2hPIXOv/lD/iSNn11Y54fqa/2Xm7XmyA9yDWfdas7zCa4oQ+OjJRhEeNpXyYDAUX3RcpY+7i1BKWysNc7k8Ir8gNF3a8RHk8jq6PbbTj+ueT8mmNSbphTZIzFmmEwWEog0zsG5lAx9llExQru2xqJfxOv2/6hdJRStLCsfRNpsu/Vsms/4uw7a7LU0FdIZ3blzn1937Cq/y+jkuP2/lx0gVCoC2soYQtO5nS5p9TgnK5Zh5ueyTLXZ/nevm6UO9/PIngUnSh3cHF8yJ5ns+8M37ZBSrNSDq6kEgv64XE/xGSU7sSrl5pyXNGWRe+EC/TTAU/BbgXHvXjefQ1YLIIO/Q4PqRrwl/3oPrp3bcs9KiKWEyjwlwQPjuG4qtg9xzorXBGZYQcFTDtyY6BTfjfotFr+D3YAf5T682hnSDf/5M42zbOqLbutcXnxlfMK5AkI5KpRNGYQlFVzUcnsrS7TFXETE5WB7GuCveBt5d5iklg3L7/WjghH0kN3ZfdWwrg4L4Besi/lPGs1bUlFU78VQGny5PvbJNVJlzZz1W5GQHo2K67XR4w4PUwGSFHzdrU9InOdfV+BnNZZ7Itu4HQQjaLLahowsRxJ6YHPC+WVoawa3LGoQB8Z6SWoSSBbA2NeZE0KfN8xDrIYAM7JkBg1x1QvxJcy7hcg32mrt1feEFK0Zh9sbErXcQ==
X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:;
 IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE;
 SFS:(4636009)(46966006)(36840700001)(7636003)(6916009)(8676002)(55016002)(2906002)(356005)(36756003)(2876002)(36860700001)(47076005)(70206006)(82310400003)(30864003)(107886003)(316002)(86362001)(6286002)(83380400001)(54906003)(1076003)(5660300002)(2616005)(4326008)(8936002)(336012)(26005)(426003)(7696005)(186003)(16526019)(70586007)(508600001)(6666004);
 DIR:OUT; SFP:1101; 
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Sep 2021 17:29:15.7370 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 0dc54a87-22c1-467f-8b76-08d98437d3ad
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34];
 Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT036.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB2973
X-Mailman-Approved-At: Thu, 30 Sep 2021 19:39:43 +0200
Subject: [dpdk-dev] [PATCH 17/18] common/mlx5: support device DMA map and
 unmap
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>

From: Michael Baum <michaelba@oss.nvidia.com>

Since MR management has moved to the common area, there is no longer a
need for the DMA map and unmap function for each driver.
This patch share those functions. For most drivers it supports these
operations for the first time.

Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/common/mlx5/mlx5_common.c    | 144 +++++++++++++++++----------
 drivers/common/mlx5/mlx5_common.h    |  41 --------
 drivers/common/mlx5/mlx5_common_mr.c |   2 +-
 drivers/common/mlx5/mlx5_common_mr.h |  25 ++---
 drivers/common/mlx5/version.map      |   9 --
 drivers/net/mlx5/mlx5.c              |   2 -
 drivers/net/mlx5/mlx5_mr.c           | 132 ------------------------
 7 files changed, 100 insertions(+), 255 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index d6acf87493..0ed1477eb8 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -258,12 +258,6 @@ is_valid_class_combination(uint32_t user_classes)
 	return 0;
 }
 
-static bool
-device_class_enabled(const struct mlx5_common_device *device, uint32_t class)
-{
-	return (device->classes_loaded & class) > 0;
-}
-
 static bool
 mlx5_bus_match(const struct mlx5_class_driver *drv,
 	       const struct rte_device *dev)
@@ -597,62 +591,106 @@ mlx5_common_dev_remove(struct rte_device *eal_dev)
 	return ret;
 }
 
+/**
+ * Callback to DMA map external memory to a device.
+ *
+ * @param rte_dev
+ *   Pointer to the generic device.
+ * @param addr
+ *   Starting virtual address of memory to be mapped.
+ * @param iova
+ *   Starting IOVA address of memory to be mapped.
+ * @param len
+ *   Length of memory segment being mapped.
+ *
+ * @return
+ *   0 on success, negative value on error.
+ */
 int
-mlx5_common_dev_dma_map(struct rte_device *dev, void *addr, uint64_t iova,
-			size_t len)
+mlx5_common_dev_dma_map(struct rte_device *rte_dev, void *addr,
+			uint64_t iova __rte_unused, size_t len)
 {
-	struct mlx5_class_driver *driver = NULL;
-	struct mlx5_class_driver *temp;
-	struct mlx5_common_device *mdev;
-	int ret = -EINVAL;
-
-	mdev = to_mlx5_device(dev);
-	if (!mdev)
-		return -ENODEV;
-	TAILQ_FOREACH(driver, &drivers_list, next) {
-		if (!device_class_enabled(mdev, driver->drv_class) ||
-		    driver->dma_map == NULL)
-			continue;
-		ret = driver->dma_map(dev, addr, iova, len);
-		if (ret)
-			goto map_err;
+	struct mlx5_common_device *dev;
+	struct mlx5_mr *mr;
+
+	dev = to_mlx5_device(rte_dev);
+	if (!dev) {
+		DRV_LOG(WARNING,
+			"Unable to find matching mlx5 device to device %s",
+			rte_dev->name);
+		rte_errno = ENODEV;
+		return -1;
 	}
-	return ret;
-map_err:
-	TAILQ_FOREACH(temp, &drivers_list, next) {
-		if (temp == driver)
-			break;
-		if (device_class_enabled(mdev, temp->drv_class) &&
-		    temp->dma_map && temp->dma_unmap)
-			temp->dma_unmap(dev, addr, iova, len);
+	mr = mlx5_create_mr_ext(dev->pd, (uintptr_t)addr, len,
+				SOCKET_ID_ANY, dev->mr_scache.reg_mr_cb);
+	if (!mr) {
+		DRV_LOG(WARNING, "Device %s unable to DMA map", rte_dev->name);
+		rte_errno = EINVAL;
+		return -1;
 	}
-	return ret;
+	rte_rwlock_write_lock(&dev->mr_scache.rwlock);
+	LIST_INSERT_HEAD(&dev->mr_scache.mr_list, mr, mr);
+	/* Insert to the global cache table. */
+	mlx5_mr_insert_cache(&dev->mr_scache, mr);
+	rte_rwlock_write_unlock(&dev->mr_scache.rwlock);
+	return 0;
 }
 
+/**
+ * Callback to DMA unmap external memory to a device.
+ *
+ * @param rte_dev
+ *   Pointer to the generic device.
+ * @param addr
+ *   Starting virtual address of memory to be unmapped.
+ * @param iova
+ *   Starting IOVA address of memory to be unmapped.
+ * @param len
+ *   Length of memory segment being unmapped.
+ *
+ * @return
+ *   0 on success, negative value on error.
+ */
 int
-mlx5_common_dev_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova,
-			  size_t len)
+mlx5_common_dev_dma_unmap(struct rte_device *rte_dev, void *addr,
+			  uint64_t iova __rte_unused, size_t len __rte_unused)
 {
-	struct mlx5_class_driver *driver;
-	struct mlx5_common_device *mdev;
-	int local_ret = -EINVAL;
-	int ret = 0;
-
-	mdev = to_mlx5_device(dev);
-	if (!mdev)
-		return -ENODEV;
-	/* There is no unmap error recovery in current implementation. */
-	TAILQ_FOREACH_REVERSE(driver, &drivers_list, mlx5_drivers, next) {
-		if (!device_class_enabled(mdev, driver->drv_class) ||
-		    driver->dma_unmap == NULL)
-			continue;
-		local_ret = driver->dma_unmap(dev, addr, iova, len);
-		if (local_ret && (ret == 0))
-			ret = local_ret;
+	struct mlx5_common_device *dev;
+	struct mr_cache_entry entry;
+	struct mlx5_mr *mr;
+
+	dev = to_mlx5_device(rte_dev);
+	if (!dev) {
+		DRV_LOG(WARNING,
+			"Unable to find matching mlx5 device to device %s.",
+			rte_dev->name);
+		rte_errno = ENODEV;
+		return -1;
 	}
-	if (local_ret)
-		ret = local_ret;
-	return ret;
+	rte_rwlock_read_lock(&dev->mr_scache.rwlock);
+	mr = mlx5_mr_lookup_list(&dev->mr_scache, &entry, (uintptr_t)addr);
+	if (!mr) {
+		rte_rwlock_read_unlock(&dev->mr_scache.rwlock);
+		DRV_LOG(WARNING,
+			"Address 0x%" PRIxPTR " wasn't registered to device %s",
+			(uintptr_t)addr, rte_dev->name);
+		rte_errno = EINVAL;
+		return -1;
+	}
+	LIST_REMOVE(mr, mr);
+	DRV_LOG(DEBUG, "MR(%p) is removed from list.", (void *)mr);
+	mlx5_mr_free(mr, dev->mr_scache.dereg_mr_cb);
+	mlx5_mr_rebuild_cache(&dev->mr_scache);
+	/*
+	 * No explicit wmb is needed after updating dev_gen due to
+	 * store-release ordering in unlock that provides the
+	 * implicit barrier at the software visible level.
+	 */
+	++dev->mr_scache.dev_gen;
+	DRV_LOG(DEBUG, "Broadcasting local cache flush, gen=%d.",
+		dev->mr_scache.dev_gen);
+	rte_rwlock_read_unlock(&dev->mr_scache.rwlock);
+	return 0;
 }
 
 void
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 1a6b8c0f52..72ff0ff809 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -364,44 +364,6 @@ typedef int (mlx5_class_driver_probe_t)(struct mlx5_common_device *dev);
  */
 typedef int (mlx5_class_driver_remove_t)(struct mlx5_common_device *dev);
 
-/**
- * Driver-specific DMA mapping. After a successful call the device
- * will be able to read/write from/to this segment.
- *
- * @param dev
- *   Pointer to the device.
- * @param addr
- *   Starting virtual address of memory to be mapped.
- * @param iova
- *   Starting IOVA address of memory to be mapped.
- * @param len
- *   Length of memory segment being mapped.
- * @return
- *   - 0 On success.
- *   - Negative value and rte_errno is set otherwise.
- */
-typedef int (mlx5_class_driver_dma_map_t)(struct rte_device *dev, void *addr,
-					  uint64_t iova, size_t len);
-
-/**
- * Driver-specific DMA un-mapping. After a successful call the device
- * will not be able to read/write from/to this segment.
- *
- * @param dev
- *   Pointer to the device.
- * @param addr
- *   Starting virtual address of memory to be unmapped.
- * @param iova
- *   Starting IOVA address of memory to be unmapped.
- * @param len
- *   Length of memory segment being unmapped.
- * @return
- *   - 0 On success.
- *   - Negative value and rte_errno is set otherwise.
- */
-typedef int (mlx5_class_driver_dma_unmap_t)(struct rte_device *dev, void *addr,
-					    uint64_t iova, size_t len);
-
 /** Device already probed can be probed again to check for new ports. */
 #define MLX5_DRV_PROBE_AGAIN 0x0004
 
@@ -414,9 +376,6 @@ struct mlx5_class_driver {
 	const char *name;                     /**< Driver name. */
 	mlx5_class_driver_probe_t *probe;     /**< Device probe function. */
 	mlx5_class_driver_remove_t *remove;   /**< Device remove function. */
-	mlx5_class_driver_dma_map_t *dma_map; /**< Device DMA map function. */
-	mlx5_class_driver_dma_unmap_t *dma_unmap;
-	/**< Device DMA unmap function. */
 	const struct rte_pci_id *id_table;    /**< ID table, NULL terminated. */
 	uint32_t probe_again:1;
 	/**< Device already probed can be probed again to check new device. */
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index d63e973b60..5bfddac08e 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -455,7 +455,7 @@ mlx5_mr_lookup_list(struct mlx5_mr_share_cache *share_cache,
  * @return
  *   Searched LKey on success, UINT32_MAX on failure and rte_errno is set.
  */
-uint32_t
+static uint32_t
 mlx5_mr_lookup_cache(struct mlx5_mr_share_cache *share_cache,
 		     struct mr_cache_entry *entry, uintptr_t addr)
 {
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 0bc3519fd9..8a7af05ca5 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -124,12 +124,13 @@ mlx5_mr_lookup_lkey(struct mr_cache_entry *lkp_tbl, uint16_t *cached_idx,
 	return UINT32_MAX;
 }
 
+/* mlx5_common_mr.c */
+
 __rte_internal
 int mlx5_mr_ctrl_init(struct mlx5_mr_ctrl *mr_ctrl, uint32_t *dev_gen_ptr,
 		      int socket);
 __rte_internal
 void mlx5_mr_btree_free(struct mlx5_mr_btree *bt);
-__rte_internal
 void mlx5_mr_btree_dump(struct mlx5_mr_btree *bt __rte_unused);
 __rte_internal
 uint32_t mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id,
@@ -142,36 +143,30 @@ uint32_t mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
 			       struct rte_mempool *mp, uintptr_t addr);
 void mlx5_mr_release_cache(struct mlx5_mr_share_cache *mr_cache);
 int mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket);
-__rte_internal
 void mlx5_mr_dump_cache(struct mlx5_mr_share_cache *share_cache __rte_unused);
-__rte_internal
 void mlx5_mr_rebuild_cache(struct mlx5_mr_share_cache *share_cache);
 __rte_internal
 void mlx5_mr_flush_local_cache(struct mlx5_mr_ctrl *mr_ctrl);
 void mlx5_free_mr_by_addr(struct mlx5_mr_share_cache *share_cache,
 			  const char *ibdev_name, const void *addr, size_t len);
-__rte_internal
-int
-mlx5_mr_insert_cache(struct mlx5_mr_share_cache *share_cache,
-		     struct mlx5_mr *mr);
-__rte_internal
-uint32_t
-mlx5_mr_lookup_cache(struct mlx5_mr_share_cache *share_cache,
-		     struct mr_cache_entry *entry, uintptr_t addr);
-__rte_internal
+int mlx5_mr_insert_cache(struct mlx5_mr_share_cache *share_cache,
+			 struct mlx5_mr *mr);
 struct mlx5_mr *
 mlx5_mr_lookup_list(struct mlx5_mr_share_cache *share_cache,
 		    struct mr_cache_entry *entry, uintptr_t addr);
-__rte_internal
 struct mlx5_mr *
 mlx5_create_mr_ext(void *pd, uintptr_t addr, size_t len, int socket_id,
 		   mlx5_reg_mr_t reg_mr_cb);
+void mlx5_mr_free(struct mlx5_mr *mr, mlx5_dereg_mr_t dereg_mr_cb);
 __rte_internal
 uint32_t
 mlx5_mr_create_primary(void *pd,
 		       struct mlx5_mr_share_cache *share_cache,
 		       struct mr_cache_entry *entry, uintptr_t addr,
 		       unsigned int mr_ext_memseg_en);
+
+/* mlx5_common_verbs.c */
+
 __rte_internal
 int
 mlx5_common_verbs_reg_mr(void *pd, void *addr, size_t length,
@@ -183,10 +178,6 @@ mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr);
 void
 mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb);
 
-__rte_internal
-void
-mlx5_mr_free(struct mlx5_mr *mr, mlx5_dereg_mr_t dereg_mr_cb);
-
 __rte_internal
 int
 mlx5_mr_mempool_register(struct mlx5_mr_share_cache *share_cache, void *pd,
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 5d9d247dc8..b41fdb883d 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -12,8 +12,6 @@ INTERNAL {
 	mlx5_common_verbs_reg_mr; # WINDOWS_NO_EXPORT
 	mlx5_common_verbs_dereg_mr; # WINDOWS_NO_EXPORT
 
-	mlx5_create_mr_ext;
-
 	mlx5_dev_is_pci;
 
 	mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -104,18 +102,11 @@ INTERNAL {
 	mlx5_mp_uninit_secondary; # WINDOWS_NO_EXPORT
 
 	mlx5_mr_addr2mr_bh;
-	mlx5_mr_btree_dump;
 	mlx5_mr_btree_free;
 	mlx5_mr_create_primary;
     mlx5_mr_ctrl_init;
-	mlx5_mr_dump_cache;
 	mlx5_mr_flush_local_cache;
-	mlx5_mr_free;
-	mlx5_mr_insert_cache;
-	mlx5_mr_lookup_cache;
-	mlx5_mr_lookup_list;
 	mlx5_mr_mb2mr;
-	mlx5_mr_rebuild_cache;
 
 	mlx5_nl_allmulti; # WINDOWS_NO_EXPORT
 	mlx5_nl_ifindex; # WINDOWS_NO_EXPORT
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 91aa5c0c75..17113be873 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -2589,8 +2589,6 @@ static struct mlx5_class_driver mlx5_net_driver = {
 	.id_table = mlx5_pci_id_map,
 	.probe = mlx5_os_net_probe,
 	.remove = mlx5_net_remove,
-	.dma_map = mlx5_net_dma_map,
-	.dma_unmap = mlx5_net_dma_unmap,
 	.probe_again = 1,
 	.intr_lsc = 1,
 	.intr_rmv = 1,
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
index 38780202dc..ac3d8e2492 100644
--- a/drivers/net/mlx5/mlx5_mr.c
+++ b/drivers/net/mlx5/mlx5_mr.c
@@ -87,135 +87,3 @@ mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 	}
 	return mlx5_tx_addr2mr_bh(txq, addr);
 }
-
-/**
- * Finds the first ethdev that match the device.
- * The existence of multiple ethdev per pci device is only with representors.
- * On such case, it is enough to get only one of the ports as they all share
- * the same ibv context.
- *
- * @param dev
- *   Pointer to the device.
- *
- * @return
- *   Pointer to the ethdev if found, NULL otherwise.
- */
-static struct rte_eth_dev *
-dev_to_eth_dev(struct rte_device *dev)
-{
-	uint16_t port_id;
-
-	port_id = rte_eth_find_next_of(0, dev);
-	if (port_id == RTE_MAX_ETHPORTS)
-		return NULL;
-	return &rte_eth_devices[port_id];
-}
-
-/**
- * Callback to DMA map external memory to a device.
- *
- * @param rte_dev
- *   Pointer to the generic device.
- * @param addr
- *   Starting virtual address of memory to be mapped.
- * @param iova
- *   Starting IOVA address of memory to be mapped.
- * @param len
- *   Length of memory segment being mapped.
- *
- * @return
- *   0 on success, negative value on error.
- */
-int
-mlx5_net_dma_map(struct rte_device *rte_dev, void *addr,
-		 uint64_t iova __rte_unused, size_t len)
-{
-	struct rte_eth_dev *dev;
-	struct mlx5_mr *mr;
-	struct mlx5_priv *priv;
-	struct mlx5_common_device *cdev;
-
-	dev = dev_to_eth_dev(rte_dev);
-	if (!dev) {
-		DRV_LOG(WARNING, "unable to find matching ethdev "
-				 "to device %s", rte_dev->name);
-		rte_errno = ENODEV;
-		return -1;
-	}
-	priv = dev->data->dev_private;
-	cdev = priv->sh->cdev;
-	mr = mlx5_create_mr_ext(cdev->pd, (uintptr_t)addr, len,
-				SOCKET_ID_ANY, cdev->mr_scache.reg_mr_cb);
-	if (!mr) {
-		DRV_LOG(WARNING,
-			"port %u unable to dma map", dev->data->port_id);
-		rte_errno = EINVAL;
-		return -1;
-	}
-	rte_rwlock_write_lock(&cdev->mr_scache.rwlock);
-	LIST_INSERT_HEAD(&cdev->mr_scache.mr_list, mr, mr);
-	/* Insert to the global cache table. */
-	mlx5_mr_insert_cache(&cdev->mr_scache, mr);
-	rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
-	return 0;
-}
-
-/**
- * Callback to DMA unmap external memory to a device.
- *
- * @param rte_dev
- *   Pointer to the generic device.
- * @param addr
- *   Starting virtual address of memory to be unmapped.
- * @param iova
- *   Starting IOVA address of memory to be unmapped.
- * @param len
- *   Length of memory segment being unmapped.
- *
- * @return
- *   0 on success, negative value on error.
- */
-int
-mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr,
-		   uint64_t iova __rte_unused, size_t len __rte_unused)
-{
-	struct rte_eth_dev *dev;
-	struct mlx5_priv *priv;
-	struct mlx5_common_device *cdev;
-	struct mlx5_mr *mr;
-	struct mr_cache_entry entry;
-
-	dev = dev_to_eth_dev(rte_dev);
-	if (!dev) {
-		DRV_LOG(WARNING, "unable to find matching ethdev to device %s",
-			rte_dev->name);
-		rte_errno = ENODEV;
-		return -1;
-	}
-	priv = dev->data->dev_private;
-	cdev = priv->sh->cdev;
-	rte_rwlock_write_lock(&cdev->mr_scache.rwlock);
-	mr = mlx5_mr_lookup_list(&cdev->mr_scache, &entry, (uintptr_t)addr);
-	if (!mr) {
-		rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
-		DRV_LOG(WARNING, "address 0x%" PRIxPTR " wasn't registered to device %s",
-			(uintptr_t)addr, rte_dev->name);
-		rte_errno = EINVAL;
-		return -1;
-	}
-	LIST_REMOVE(mr, mr);
-	DRV_LOG(DEBUG, "port %u remove MR(%p) from list", dev->data->port_id,
-	      (void *)mr);
-	mlx5_mr_free(mr, cdev->mr_scache.dereg_mr_cb);
-	mlx5_mr_rebuild_cache(&cdev->mr_scache);
-	/*
-	 * No explicit wmb is needed after updating dev_gen due to
-	 * store-release ordering in unlock that provides the
-	 * implicit barrier at the software visible level.
-	 */
-	++cdev->mr_scache.dev_gen;
-	DRV_LOG(DEBUG, "broadcasting local cache flush, gen=%d",
-	      cdev->mr_scache.dev_gen);
-	rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
-	return 0;
-}
-- 
2.25.1