DPDK patches and discussions
 help / color / mirror / Atom feed
From: Matan Azrad <matan@nvidia.com>
To: dev@dpdk.org
Cc: Thomas Monjalon <thomas@monjalon.net>,
	Ashish Gupta <ashish.gupta@marvell.com>,
	Fiona Trahe <fiona.trahe@intel.com>,
	akhil.goyal@nxp.com
Subject: [dpdk-dev] [PATCH v2 07/10] compress/mlx5: add memory region management
Date: Wed, 13 Jan 2021 16:18:07 +0000
Message-ID: <1610554690-411627-8-git-send-email-matan@nvidia.com> (raw)
In-Reply-To: <1610554690-411627-1-git-send-email-matan@nvidia.com>

Mellanox user space drivers don't deal with physical addresses, that's
why any mbuf virtual address moved directly to the HW descriptor(WQE).

The mapping between the virtual address to the physical address is saved
in MR configured by the kernel to the HW.

Each MR has a key that should also be moved to the WQE by the SW.

When the SW see address which is not mapped, it extends the address
range and creates a MR using a system call.

Add memory region cache management:
2 level cache per queue-pair - no locks.
1 shared cache between all the queues using a lock.

Using this way, the MR key search per data-path address is optimized.

Signed-off-by: Matan Azrad <matan@nvidia.com>
---
 drivers/compress/mlx5/mlx5_compress.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 132837e..ab24a84 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -46,6 +46,7 @@ struct mlx5_compress_priv {
 	struct rte_compressdev_config dev_config;
 	SLIST_HEAD(xform_list, mlx5_compress_xform) xform_list;
 	rte_spinlock_t xform_sl;
+	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 };
 
 struct mlx5_compress_qp {
@@ -54,6 +55,7 @@ struct mlx5_compress_qp {
 	uint16_t pi;
 	uint16_t ci;
 	volatile uint64_t *uar_addr;
+	struct mlx5_mr_ctrl mr_ctrl;
 	int socket_id;
 	struct mlx5_devx_cq cq;
 	struct mlx5_devx_sq sq;
@@ -118,6 +120,7 @@ struct mlx5_compress_qp {
 		if (opaq != NULL)
 			rte_free(opaq);
 	}
+	mlx5_mr_btree_free(&qp->mr_ctrl.cache_bh);
 	rte_free(qp);
 	dev->data->queue_pairs[qp_id] = NULL;
 	return 0;
@@ -184,6 +187,13 @@ struct mlx5_compress_qp {
 		rte_errno = ENOMEM;
 		goto err;
 	}
+	if (mlx5_mr_btree_init(&qp->mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N,
+			       priv->dev_config.socket_id)) {
+		DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
+			(uint32_t)qp_id);
+		rte_errno = ENOMEM;
+		goto err;
+	}
 	qp->entries_n = 1 << log_ops_n;
 	qp->socket_id = socket_id;
 	qp->qp_id = qp_id;
@@ -513,6 +523,17 @@ struct mlx5_compress_qp {
 		claim_zero(mlx5_glue->close_device(priv->ctx));
 		return -1;
 	}
+	if (mlx5_mr_btree_init(&priv->mr_scache.cache,
+			     MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
+		DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
+		mlx5_compress_hw_global_release(priv);
+		rte_compressdev_pmd_destroy(priv->cdev);
+		claim_zero(mlx5_glue->close_device(priv->ctx));
+		rte_errno = ENOMEM;
+		return -rte_errno;
+	}
+	priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
+	priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
 	pthread_mutex_lock(&priv_list_lock);
 	TAILQ_INSERT_TAIL(&mlx5_compress_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
@@ -547,6 +568,7 @@ struct mlx5_compress_qp {
 		TAILQ_REMOVE(&mlx5_compress_priv_list, priv, next);
 	pthread_mutex_unlock(&priv_list_lock);
 	if (found != 0) {
+		mlx5_mr_release_cache(&priv->mr_scache);
 		mlx5_compress_hw_global_release(priv);
 		rte_compressdev_pmd_destroy(priv->cdev);
 		claim_zero(mlx5_glue->close_device(priv->ctx));
-- 
1.8.3.1


  parent reply	other threads:[~2021-01-13 16:19 UTC|newest]

Thread overview: 41+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-11 13:59 [dpdk-dev] [PATCH 00/10] add mlx5 compress PMD Matan Azrad
2021-01-11 13:59 ` [dpdk-dev] [PATCH 01/10] common/mlx5: add DevX attributes for compress Matan Azrad
2021-01-11 13:59 ` [dpdk-dev] [PATCH 02/10] drivers: introduce mlx5 compress PMD Matan Azrad
2021-01-11 13:59 ` [dpdk-dev] [PATCH 03/10] compress/mlx5: support basic control operations Matan Azrad
2021-01-11 13:59 ` [dpdk-dev] [PATCH 04/10] common/mlx5: add compress primitives Matan Azrad
2021-01-11 13:59 ` [dpdk-dev] [PATCH 05/10] compress/mlx5: support queue pair operations Matan Azrad
2021-01-11 13:59 ` [dpdk-dev] [PATCH 06/10] compress/mlx5: add transformation operations Matan Azrad
2021-01-11 13:59 ` [dpdk-dev] [PATCH 07/10] compress/mlx5: add memory region management Matan Azrad
2021-01-11 13:59 ` [dpdk-dev] [PATCH 08/10] compress/mlx5: add data-path functions Matan Azrad
2021-01-11 13:59 ` [dpdk-dev] [PATCH 09/10] compress/mlx5: add statistics operations Matan Azrad
2021-01-11 13:59 ` [dpdk-dev] [PATCH 10/10] compress/mlx5: add the supported capabilities Matan Azrad
2021-01-12 13:08 ` [dpdk-dev] [PATCH 00/10] add mlx5 compress PMD Asaf Penso
2021-01-13 16:18 ` [dpdk-dev] [PATCH v2 " Matan Azrad
2021-01-13 16:18   ` [dpdk-dev] [PATCH v2 01/10] common/mlx5: add DevX attributes for compress Matan Azrad
2021-01-18 15:08     ` Slava Ovsiienko
2021-01-13 16:18   ` [dpdk-dev] [PATCH v2 02/10] drivers: introduce mlx5 compress PMD Matan Azrad
2021-01-13 16:18   ` [dpdk-dev] [PATCH v2 03/10] compress/mlx5: support basic control operations Matan Azrad
2021-01-13 16:18   ` [dpdk-dev] [PATCH v2 04/10] common/mlx5: add compress primitives Matan Azrad
2021-01-13 16:18   ` [dpdk-dev] [PATCH v2 05/10] compress/mlx5: support queue pair operations Matan Azrad
2021-01-13 16:18   ` [dpdk-dev] [PATCH v2 06/10] compress/mlx5: add transformation operations Matan Azrad
2021-01-13 16:18   ` Matan Azrad [this message]
2021-01-13 16:18   ` [dpdk-dev] [PATCH v2 08/10] compress/mlx5: add data-path functions Matan Azrad
2021-01-13 16:18   ` [dpdk-dev] [PATCH v2 09/10] compress/mlx5: add statistics operations Matan Azrad
2021-01-13 16:18   ` [dpdk-dev] [PATCH v2 10/10] compress/mlx5: add the supported capabilities Matan Azrad
2021-01-19 17:54     ` Akhil Goyal
2021-01-20  7:19       ` Matan Azrad
2021-01-20 11:29   ` [dpdk-dev] [PATCH v3 00/11] add mlx5 compress PMD Matan Azrad
2021-01-20 11:29     ` [dpdk-dev] [PATCH v3 01/11] common/mlx5: add DevX attributes for compress Matan Azrad
2021-01-21 16:52       ` Tal Shnaiderman
2021-01-21 17:10         ` Matan Azrad
2021-01-20 11:29     ` [dpdk-dev] [PATCH v3 02/11] drivers: introduce mlx5 compress PMD Matan Azrad
2021-01-20 11:29     ` [dpdk-dev] [PATCH v3 03/11] compress/mlx5: support basic control operations Matan Azrad
2021-01-20 11:29     ` [dpdk-dev] [PATCH v3 04/11] common/mlx5: add compress primitives Matan Azrad
2021-01-20 11:29     ` [dpdk-dev] [PATCH v3 05/11] compress/mlx5: support queue pair operations Matan Azrad
2021-01-20 11:29     ` [dpdk-dev] [PATCH v3 06/11] compress/mlx5: add transformation operations Matan Azrad
2021-01-20 11:29     ` [dpdk-dev] [PATCH v3 07/11] compress/mlx5: add memory region management Matan Azrad
2021-01-20 11:29     ` [dpdk-dev] [PATCH v3 08/11] compress/mlx5: add data-path functions Matan Azrad
2021-01-20 11:29     ` [dpdk-dev] [PATCH v3 09/11] compress/mlx5: add statistics operations Matan Azrad
2021-01-20 11:29     ` [dpdk-dev] [PATCH v3 10/11] compress/mlx5: support 32-bit systems Matan Azrad
2021-01-20 11:29     ` [dpdk-dev] [PATCH v3 11/11] compress/mlx5: add the supported capabilities Matan Azrad
2021-01-27 19:38     ` [dpdk-dev] [PATCH v3 00/11] add mlx5 compress PMD Akhil Goyal

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1610554690-411627-8-git-send-email-matan@nvidia.com \
    --to=matan@nvidia.com \
    --cc=akhil.goyal@nxp.com \
    --cc=ashish.gupta@marvell.com \
    --cc=dev@dpdk.org \
    --cc=fiona.trahe@intel.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ http://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git