DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] [PATCH 0/4] vdpa/mlx5: support direct notification
@ 2020-03-24 14:24 Asaf Penso
  2020-03-24 14:24 ` [dpdk-dev] [PATCH 1/4] vdpa/mlx5: move virtual doorbell alloc to probe Asaf Penso
                   ` (4 more replies)
  0 siblings, 5 replies; 13+ messages in thread
From: Asaf Penso @ 2020-03-24 14:24 UTC (permalink / raw)
  To: dev; +Cc: Viacheslav Ovsiienko, Shahaf Shuler, Maxime Coquelin

In order to save CPU usage in host and even make it 0 when guest work with poll mode,
add support for direct doorbell ringing.

Now, when the feature is supported in QEMU, the guest doorbell address is mapped directly to the HW doorbell space and the host SW relay is not used. 

Matan Azrad (4):
  vdpa/mlx5: move virtual doorbell alloc to probe
  vdpa/mlx5: support direct HW notifications
  vdpa/mlx5: validate notifier configuration
  vdpa/mlx5: add log prints

 drivers/vdpa/mlx5/mlx5_vdpa.c       | 90 +++++++++++++++++++++++++++++++------
 drivers/vdpa/mlx5/mlx5_vdpa.h       |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa_steer.c |  2 +
 drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 39 ++++++++--------
 4 files changed, 100 insertions(+), 32 deletions(-)

-- 
1.8.3.1


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [dpdk-dev] [PATCH 1/4] vdpa/mlx5: move virtual doorbell alloc to probe
  2020-03-24 14:24 [dpdk-dev] [PATCH 0/4] vdpa/mlx5: support direct notification Asaf Penso
@ 2020-03-24 14:24 ` Asaf Penso
  2020-04-15  9:44   ` Maxime Coquelin
  2020-03-24 14:24 ` [dpdk-dev] [PATCH 2/4] vdpa/mlx5: support direct HW notifications Asaf Penso
                   ` (3 subsequent siblings)
  4 siblings, 1 reply; 13+ messages in thread
From: Asaf Penso @ 2020-03-24 14:24 UTC (permalink / raw)
  To: dev; +Cc: Viacheslav Ovsiienko, Shahaf Shuler, Maxime Coquelin, Matan Azrad

From: Matan Azrad <matan@mellanox.com>

The configure and close operations may be called a lot of time by vhost
library according to the virtio connections in the guest.

VAR is the device memory space for the virtio queues doorbells.
Each VAR page can be shared for more than one queue while its owner must
synchronize the writes to it.

The mlx5 driver allocates single VAR page for all its queues.

Therefore, it is better to allocate it in probe device level instead of
creating and destroying it per new connection.

Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
---
 drivers/vdpa/mlx5/mlx5_vdpa.c       | 14 +++++++++++++-
 drivers/vdpa/mlx5/mlx5_vdpa_virtq.c |  9 ---------
 2 files changed, 13 insertions(+), 10 deletions(-)

diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 97d914a..5542c29 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -447,6 +447,11 @@
 	priv->ctx = ctx;
 	priv->dev_addr.pci_addr = pci_dev->addr;
 	priv->dev_addr.type = PCI_ADDR;
+	priv->var = mlx5_glue->dv_alloc_var(ctx, 0);
+	if (!priv->var) {
+		DRV_LOG(ERR, "Failed to allocate VAR %u.\n", errno);
+		goto error;
+	}
 	priv->id = rte_vdpa_register_device(&priv->dev_addr, &mlx5_vdpa_ops);
 	if (priv->id < 0) {
 		DRV_LOG(ERR, "Failed to register vDPA device.");
@@ -461,8 +466,11 @@
 	return 0;
 
 error:
-	if (priv)
+	if (priv) {
+		if (priv->var)
+			mlx5_glue->dv_free_var(priv->var);
 		rte_free(priv);
+	}
 	if (ctx)
 		mlx5_glue->close_device(ctx);
 	return -rte_errno;
@@ -499,6 +507,10 @@
 	if (found) {
 		if (priv->configured)
 			mlx5_vdpa_dev_close(priv->vid);
+		if (priv->var) {
+			mlx5_glue->dv_free_var(priv->var);
+			priv->var = NULL;
+		}
 		mlx5_glue->close_device(priv->ctx);
 		rte_free(priv);
 	}
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
index 2312331..6390385 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
@@ -105,10 +105,6 @@
 		claim_zero(munmap(priv->virtq_db_addr, priv->var->length));
 		priv->virtq_db_addr = NULL;
 	}
-	if (priv->var) {
-		mlx5_glue->dv_free_var(priv->var);
-		priv->var = NULL;
-	}
 	priv->features = 0;
 }
 
@@ -343,11 +339,6 @@
 		DRV_LOG(ERR, "Failed to configure negotiated features.");
 		return -1;
 	}
-	priv->var = mlx5_glue->dv_alloc_var(priv->ctx, 0);
-	if (!priv->var) {
-		DRV_LOG(ERR, "Failed to allocate VAR %u.\n", errno);
-		return -1;
-	}
 	/* Always map the entire page. */
 	priv->virtq_db_addr = mmap(NULL, priv->var->length, PROT_READ |
 				   PROT_WRITE, MAP_SHARED, priv->ctx->cmd_fd,
-- 
1.8.3.1


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [dpdk-dev] [PATCH 2/4] vdpa/mlx5: support direct HW notifications
  2020-03-24 14:24 [dpdk-dev] [PATCH 0/4] vdpa/mlx5: support direct notification Asaf Penso
  2020-03-24 14:24 ` [dpdk-dev] [PATCH 1/4] vdpa/mlx5: move virtual doorbell alloc to probe Asaf Penso
@ 2020-03-24 14:24 ` Asaf Penso
  2020-04-15  9:47   ` Maxime Coquelin
  2020-03-24 14:24 ` [dpdk-dev] [PATCH 3/4] vdpa/mlx5: validate notifier configuration Asaf Penso
                   ` (2 subsequent siblings)
  4 siblings, 1 reply; 13+ messages in thread
From: Asaf Penso @ 2020-03-24 14:24 UTC (permalink / raw)
  To: dev; +Cc: Viacheslav Ovsiienko, Shahaf Shuler, Maxime Coquelin, Matan Azrad

From: Matan Azrad <matan@mellanox.com>

Add support for the next 2 callbacks:
get_vfio_device_fd and get_notify_area.

This will allow direct HW doorbell ringing from guest and will save CPU
usage in host.

By this patch, the QEMU will map the physical address of the virtio
device in guest directly to the physical address of the HW device
doorbell.

The guest doorbell write is 2 bytes transaction while some Mellanox nics
support only 4 bytes transactions.

Remove ConnectX-5 and BF1 devices support which don't support 2B
doorbell writes for HW triggering.

Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
---
 drivers/vdpa/mlx5/mlx5_vdpa.c | 74 ++++++++++++++++++++++++++++++++++++-------
 drivers/vdpa/mlx5/mlx5_vdpa.h |  1 +
 2 files changed, 63 insertions(+), 12 deletions(-)

diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 5542c29..4eb6abf 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -133,6 +133,29 @@
 }
 
 static int
+mlx5_vdpa_direct_db_prepare(struct mlx5_vdpa_priv *priv)
+{
+	int ret;
+
+	if (priv->direct_notifier) {
+		ret = rte_vhost_host_notifier_ctrl(priv->vid, false);
+		if (ret != 0) {
+			DRV_LOG(INFO, "Direct HW notifier FD cannot be "
+				"destroyed for device %d: %d.", priv->vid, ret);
+			return -1;
+		}
+		priv->direct_notifier = 0;
+	}
+	ret = rte_vhost_host_notifier_ctrl(priv->vid, true);
+	if (ret != 0)
+		DRV_LOG(INFO, "Direct HW notifier FD cannot be configured for"
+			" device %d: %d.", priv->vid, ret);
+	else
+		priv->direct_notifier = 1;
+	return 0;
+}
+
+static int
 mlx5_vdpa_features_set(int vid)
 {
 	int did = rte_vhost_get_vdpa_device_id(vid);
@@ -209,8 +232,9 @@
 		return -1;
 	}
 	priv->vid = vid;
-	if (mlx5_vdpa_mem_register(priv) || mlx5_vdpa_virtqs_prepare(priv) ||
-	    mlx5_vdpa_steer_setup(priv) || mlx5_vdpa_cqe_event_setup(priv)) {
+	if (mlx5_vdpa_mem_register(priv) || mlx5_vdpa_direct_db_prepare(priv) ||
+	    mlx5_vdpa_virtqs_prepare(priv) || mlx5_vdpa_steer_setup(priv) ||
+	    mlx5_vdpa_cqe_event_setup(priv)) {
 		mlx5_vdpa_dev_close(vid);
 		return -1;
 	}
@@ -218,6 +242,40 @@
 	return 0;
 }
 
+static int
+mlx5_vdpa_get_device_fd(int vid)
+{
+	int did = rte_vhost_get_vdpa_device_id(vid);
+	struct mlx5_vdpa_priv *priv = mlx5_vdpa_find_priv_resource_by_did(did);
+
+	if (priv == NULL) {
+		DRV_LOG(ERR, "Invalid device id: %d.", did);
+		return -EINVAL;
+	}
+	return priv->ctx->cmd_fd;
+}
+
+static int
+mlx5_vdpa_get_notify_area(int vid, int qid, uint64_t *offset, uint64_t *size)
+{
+	int did = rte_vhost_get_vdpa_device_id(vid);
+	struct mlx5_vdpa_priv *priv = mlx5_vdpa_find_priv_resource_by_did(did);
+
+	RTE_SET_USED(qid);
+	if (priv == NULL) {
+		DRV_LOG(ERR, "Invalid device id: %d.", did);
+		return -EINVAL;
+	}
+	if (!priv->var) {
+		DRV_LOG(ERR, "VAR was not created for device %d, is the device"
+			" configured?.", did);
+		return -EINVAL;
+	}
+	*offset = priv->var->mmap_off;
+	*size = priv->var->length;
+	return 0;
+}
+
 static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
 	.get_queue_num = mlx5_vdpa_get_queue_num,
 	.get_features = mlx5_vdpa_get_vdpa_features,
@@ -228,8 +286,8 @@
 	.set_features = mlx5_vdpa_features_set,
 	.migration_done = NULL,
 	.get_vfio_group_fd = NULL,
-	.get_vfio_device_fd = NULL,
-	.get_notify_area = NULL,
+	.get_vfio_device_fd = mlx5_vdpa_get_device_fd,
+	.get_notify_area = mlx5_vdpa_get_notify_area,
 };
 
 static struct ibv_device *
@@ -520,14 +578,6 @@
 static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = {
 	{
 		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
-			       PCI_DEVICE_ID_MELLANOX_CONNECTX5BF)
-	},
-	{
-		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
-			       PCI_DEVICE_ID_MELLANOX_CONNECTX5BFVF)
-	},
-	{
-		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
 				PCI_DEVICE_ID_MELLANOX_CONNECTX6)
 	},
 	{
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
index 3324c9d..75af410 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
@@ -100,6 +100,7 @@ struct mlx5_vdpa_steer {
 struct mlx5_vdpa_priv {
 	TAILQ_ENTRY(mlx5_vdpa_priv) next;
 	uint8_t configured;
+	uint8_t direct_notifier; /* Whether direct notifier is on or off. */
 	int id; /* vDPA device id. */
 	int vid; /* vhost device id. */
 	struct ibv_context *ctx; /* Device context. */
-- 
1.8.3.1


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [dpdk-dev] [PATCH 3/4] vdpa/mlx5: validate notifier configuration
  2020-03-24 14:24 [dpdk-dev] [PATCH 0/4] vdpa/mlx5: support direct notification Asaf Penso
  2020-03-24 14:24 ` [dpdk-dev] [PATCH 1/4] vdpa/mlx5: move virtual doorbell alloc to probe Asaf Penso
  2020-03-24 14:24 ` [dpdk-dev] [PATCH 2/4] vdpa/mlx5: support direct HW notifications Asaf Penso
@ 2020-03-24 14:24 ` Asaf Penso
  2020-04-15  9:54   ` Maxime Coquelin
  2020-03-24 14:24 ` [dpdk-dev] [PATCH 4/4] vdpa/mlx5: add log prints Asaf Penso
  2020-04-17 17:14 ` [dpdk-dev] [PATCH 0/4] vdpa/mlx5: support direct notification Maxime Coquelin
  4 siblings, 1 reply; 13+ messages in thread
From: Asaf Penso @ 2020-03-24 14:24 UTC (permalink / raw)
  To: dev; +Cc: Viacheslav Ovsiienko, Shahaf Shuler, Maxime Coquelin, Matan Azrad

From: Matan Azrad <matan@mellanox.com>

When both, direct and indirect notifier management cannot be
configured, return an error.

Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
---
 drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)

diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
index 6390385..8bebb92 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
@@ -46,7 +46,7 @@
 	int retries = MLX5_VDPA_INTR_RETRIES;
 	int ret = -EAGAIN;
 
-	if (virtq->intr_handle.fd) {
+	if (virtq->intr_handle.fd != -1) {
 		while (retries-- && ret == -EAGAIN) {
 			ret = rte_intr_callback_unregister(&virtq->intr_handle,
 							mlx5_vdpa_virtq_handler,
@@ -59,7 +59,7 @@
 				usleep(MLX5_VDPA_INTR_RETRIES_USEC);
 			}
 		}
-		memset(&virtq->intr_handle, 0, sizeof(virtq->intr_handle));
+		virtq->intr_handle.fd = -1;
 	}
 	if (virtq->virtq) {
 		claim_zero(mlx5_devx_cmd_destroy(virtq->virtq));
@@ -255,15 +255,25 @@
 	rte_write32(virtq->index, priv->virtq_db_addr);
 	/* Setup doorbell mapping. */
 	virtq->intr_handle.fd = vq.kickfd;
-	virtq->intr_handle.type = RTE_INTR_HANDLE_EXT;
-	if (rte_intr_callback_register(&virtq->intr_handle,
-				       mlx5_vdpa_virtq_handler, virtq)) {
-		virtq->intr_handle.fd = 0;
-		DRV_LOG(ERR, "Failed to register virtq %d interrupt.", index);
-		goto error;
+	if (virtq->intr_handle.fd == -1) {
+		DRV_LOG(WARNING, "Virtq %d kickfd is invalid.", index);
+		if (!priv->direct_notifier) {
+			DRV_LOG(ERR, "Virtq %d cannot be notified.", index);
+			goto error;
+		}
 	} else {
-		DRV_LOG(DEBUG, "Register fd %d interrupt for virtq %d.",
-			virtq->intr_handle.fd, index);
+		virtq->intr_handle.type = RTE_INTR_HANDLE_EXT;
+		if (rte_intr_callback_register(&virtq->intr_handle,
+					       mlx5_vdpa_virtq_handler,
+					       virtq)) {
+			virtq->intr_handle.fd = -1;
+			DRV_LOG(ERR, "Failed to register virtq %d interrupt.",
+				index);
+			goto error;
+		} else {
+			DRV_LOG(DEBUG, "Register fd %d interrupt for virtq %d.",
+				virtq->intr_handle.fd, index);
+		}
 	}
 	return 0;
 error:
-- 
1.8.3.1


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [dpdk-dev] [PATCH 4/4] vdpa/mlx5: add log prints
  2020-03-24 14:24 [dpdk-dev] [PATCH 0/4] vdpa/mlx5: support direct notification Asaf Penso
                   ` (2 preceding siblings ...)
  2020-03-24 14:24 ` [dpdk-dev] [PATCH 3/4] vdpa/mlx5: validate notifier configuration Asaf Penso
@ 2020-03-24 14:24 ` Asaf Penso
  2020-04-17 11:54   ` Maxime Coquelin
  2020-04-17 17:14 ` [dpdk-dev] [PATCH 0/4] vdpa/mlx5: support direct notification Maxime Coquelin
  4 siblings, 1 reply; 13+ messages in thread
From: Asaf Penso @ 2020-03-24 14:24 UTC (permalink / raw)
  To: dev; +Cc: Viacheslav Ovsiienko, Shahaf Shuler, Maxime Coquelin, Matan Azrad

From: Matan Azrad <matan@mellanox.com>

Add log prints to improve driver status following.

Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
---
 drivers/vdpa/mlx5/mlx5_vdpa.c       | 2 ++
 drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 4eb6abf..f10647b 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -214,6 +214,7 @@
 	mlx5_vdpa_mem_dereg(priv);
 	priv->configured = 0;
 	priv->vid = 0;
+	DRV_LOG(INFO, "vDPA device %d was closed.", vid);
 	return ret;
 }
 
@@ -239,6 +240,7 @@
 		return -1;
 	}
 	priv->configured = 1;
+	DRV_LOG(INFO, "vDPA device %d was configured.", vid);
 	return 0;
 }
 
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_steer.c b/drivers/vdpa/mlx5/mlx5_vdpa_steer.c
index 36017f1..8e1cf0a 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_steer.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_steer.c
@@ -123,6 +123,8 @@
 	struct mlx5_vdpa_priv *priv = virtq->priv;
 	int ret = 0;
 
+	DRV_LOG(INFO, "Update virtq %d status %sable -> %sable.", virtq->index,
+		virtq->enable ? "en" : "dis", enable ? "en" : "dis");
 	if (virtq->enable == !!enable)
 		return 0;
 	virtq->enable = !!enable;
-- 
1.8.3.1


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [dpdk-dev] [PATCH 1/4] vdpa/mlx5: move virtual doorbell alloc to probe
  2020-03-24 14:24 ` [dpdk-dev] [PATCH 1/4] vdpa/mlx5: move virtual doorbell alloc to probe Asaf Penso
@ 2020-04-15  9:44   ` Maxime Coquelin
  0 siblings, 0 replies; 13+ messages in thread
From: Maxime Coquelin @ 2020-04-15  9:44 UTC (permalink / raw)
  To: Asaf Penso, dev; +Cc: Viacheslav Ovsiienko, Shahaf Shuler, Matan Azrad



On 3/24/20 3:24 PM, Asaf Penso wrote:
> From: Matan Azrad <matan@mellanox.com>
> 
> The configure and close operations may be called a lot of time by vhost
> library according to the virtio connections in the guest.
> 
> VAR is the device memory space for the virtio queues doorbells.
> Each VAR page can be shared for more than one queue while its owner must
> synchronize the writes to it.
> 
> The mlx5 driver allocates single VAR page for all its queues.
> 
> Therefore, it is better to allocate it in probe device level instead of
> creating and destroying it per new connection.
> 
> Signed-off-by: Matan Azrad <matan@mellanox.com>
> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
>  drivers/vdpa/mlx5/mlx5_vdpa.c       | 14 +++++++++++++-
>  drivers/vdpa/mlx5/mlx5_vdpa_virtq.c |  9 ---------
>  2 files changed, 13 insertions(+), 10 deletions(-)

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [dpdk-dev] [PATCH 2/4] vdpa/mlx5: support direct HW notifications
  2020-03-24 14:24 ` [dpdk-dev] [PATCH 2/4] vdpa/mlx5: support direct HW notifications Asaf Penso
@ 2020-04-15  9:47   ` Maxime Coquelin
  2020-04-17 11:54     ` Maxime Coquelin
  0 siblings, 1 reply; 13+ messages in thread
From: Maxime Coquelin @ 2020-04-15  9:47 UTC (permalink / raw)
  To: Asaf Penso, dev; +Cc: Viacheslav Ovsiienko, Shahaf Shuler, Matan Azrad



On 3/24/20 3:24 PM, Asaf Penso wrote:
> From: Matan Azrad <matan@mellanox.com>
> 
> Add support for the next 2 callbacks:
> get_vfio_device_fd and get_notify_area.
> 
> This will allow direct HW doorbell ringing from guest and will save CPU
> usage in host.
> 
> By this patch, the QEMU will map the physical address of the virtio
> device in guest directly to the physical address of the HW device
> doorbell.
> 
> The guest doorbell write is 2 bytes transaction while some Mellanox nics
> support only 4 bytes transactions.
> 
> Remove ConnectX-5 and BF1 devices support which don't support 2B
> doorbell writes for HW triggering.

Couldn't we have different rte_vdpa_dev_ops depending on whether
doorbell write support?

> Signed-off-by: Matan Azrad <matan@mellanox.com>
> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
>  drivers/vdpa/mlx5/mlx5_vdpa.c | 74 ++++++++++++++++++++++++++++++++++++-------
>  drivers/vdpa/mlx5/mlx5_vdpa.h |  1 +
>  2 files changed, 63 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
> index 5542c29..4eb6abf 100644
> --- a/drivers/vdpa/mlx5/mlx5_vdpa.c
> +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
> @@ -133,6 +133,29 @@
>  }
>  
>  static int
> +mlx5_vdpa_direct_db_prepare(struct mlx5_vdpa_priv *priv)
> +{
> +	int ret;
> +
> +	if (priv->direct_notifier) {
> +		ret = rte_vhost_host_notifier_ctrl(priv->vid, false);
> +		if (ret != 0) {
> +			DRV_LOG(INFO, "Direct HW notifier FD cannot be "
> +				"destroyed for device %d: %d.", priv->vid, ret);
> +			return -1;
> +		}
> +		priv->direct_notifier = 0;
> +	}
> +	ret = rte_vhost_host_notifier_ctrl(priv->vid, true);
> +	if (ret != 0)
> +		DRV_LOG(INFO, "Direct HW notifier FD cannot be configured for"
> +			" device %d: %d.", priv->vid, ret);
> +	else
> +		priv->direct_notifier = 1;
> +	return 0;
> +}
> +
> +static int
>  mlx5_vdpa_features_set(int vid)
>  {
>  	int did = rte_vhost_get_vdpa_device_id(vid);
> @@ -209,8 +232,9 @@
>  		return -1;
>  	}
>  	priv->vid = vid;
> -	if (mlx5_vdpa_mem_register(priv) || mlx5_vdpa_virtqs_prepare(priv) ||
> -	    mlx5_vdpa_steer_setup(priv) || mlx5_vdpa_cqe_event_setup(priv)) {
> +	if (mlx5_vdpa_mem_register(priv) || mlx5_vdpa_direct_db_prepare(priv) ||
> +	    mlx5_vdpa_virtqs_prepare(priv) || mlx5_vdpa_steer_setup(priv) ||
> +	    mlx5_vdpa_cqe_event_setup(priv)) {
>  		mlx5_vdpa_dev_close(vid);
>  		return -1;
>  	}
> @@ -218,6 +242,40 @@
>  	return 0;
>  }
>  
> +static int
> +mlx5_vdpa_get_device_fd(int vid)
> +{
> +	int did = rte_vhost_get_vdpa_device_id(vid);
> +	struct mlx5_vdpa_priv *priv = mlx5_vdpa_find_priv_resource_by_did(did);
> +
> +	if (priv == NULL) {
> +		DRV_LOG(ERR, "Invalid device id: %d.", did);
> +		return -EINVAL;
> +	}
> +	return priv->ctx->cmd_fd;
> +}
> +
> +static int
> +mlx5_vdpa_get_notify_area(int vid, int qid, uint64_t *offset, uint64_t *size)
> +{
> +	int did = rte_vhost_get_vdpa_device_id(vid);
> +	struct mlx5_vdpa_priv *priv = mlx5_vdpa_find_priv_resource_by_did(did);
> +
> +	RTE_SET_USED(qid);
> +	if (priv == NULL) {
> +		DRV_LOG(ERR, "Invalid device id: %d.", did);
> +		return -EINVAL;
> +	}
> +	if (!priv->var) {
> +		DRV_LOG(ERR, "VAR was not created for device %d, is the device"
> +			" configured?.", did);
> +		return -EINVAL;
> +	}
> +	*offset = priv->var->mmap_off;
> +	*size = priv->var->length;
> +	return 0;
> +}
> +
>  static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
>  	.get_queue_num = mlx5_vdpa_get_queue_num,
>  	.get_features = mlx5_vdpa_get_vdpa_features,
> @@ -228,8 +286,8 @@
>  	.set_features = mlx5_vdpa_features_set,
>  	.migration_done = NULL,
>  	.get_vfio_group_fd = NULL,
> -	.get_vfio_device_fd = NULL,
> -	.get_notify_area = NULL,
> +	.get_vfio_device_fd = mlx5_vdpa_get_device_fd,
> +	.get_notify_area = mlx5_vdpa_get_notify_area,
>  };
>  
>  static struct ibv_device *
> @@ -520,14 +578,6 @@
>  static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = {
>  	{
>  		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
> -			       PCI_DEVICE_ID_MELLANOX_CONNECTX5BF)
> -	},
> -	{
> -		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
> -			       PCI_DEVICE_ID_MELLANOX_CONNECTX5BFVF)
> -	},
> -	{
> -		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
>  				PCI_DEVICE_ID_MELLANOX_CONNECTX6)
>  	},
>  	{
> diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
> index 3324c9d..75af410 100644
> --- a/drivers/vdpa/mlx5/mlx5_vdpa.h
> +++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
> @@ -100,6 +100,7 @@ struct mlx5_vdpa_steer {
>  struct mlx5_vdpa_priv {
>  	TAILQ_ENTRY(mlx5_vdpa_priv) next;
>  	uint8_t configured;
> +	uint8_t direct_notifier; /* Whether direct notifier is on or off. */
>  	int id; /* vDPA device id. */
>  	int vid; /* vhost device id. */
>  	struct ibv_context *ctx; /* Device context. */
> 


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [dpdk-dev] [PATCH 3/4] vdpa/mlx5: validate notifier configuration
  2020-03-24 14:24 ` [dpdk-dev] [PATCH 3/4] vdpa/mlx5: validate notifier configuration Asaf Penso
@ 2020-04-15  9:54   ` Maxime Coquelin
  0 siblings, 0 replies; 13+ messages in thread
From: Maxime Coquelin @ 2020-04-15  9:54 UTC (permalink / raw)
  To: Asaf Penso, dev; +Cc: Viacheslav Ovsiienko, Shahaf Shuler, Matan Azrad



On 3/24/20 3:24 PM, Asaf Penso wrote:
> From: Matan Azrad <matan@mellanox.com>
> 
> When both, direct and indirect notifier management cannot be
> configured, return an error.
> 
> Signed-off-by: Matan Azrad <matan@mellanox.com>
> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
>  drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 30 ++++++++++++++++++++----------
>  1 file changed, 20 insertions(+), 10 deletions(-)

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [dpdk-dev] [PATCH 2/4] vdpa/mlx5: support direct HW notifications
  2020-04-15  9:47   ` Maxime Coquelin
@ 2020-04-17 11:54     ` Maxime Coquelin
  2020-04-26  7:06       ` Matan Azrad
  0 siblings, 1 reply; 13+ messages in thread
From: Maxime Coquelin @ 2020-04-17 11:54 UTC (permalink / raw)
  To: Asaf Penso, dev; +Cc: Viacheslav Ovsiienko, Shahaf Shuler, Matan Azrad



On 4/15/20 11:47 AM, Maxime Coquelin wrote:
> 
> 
> On 3/24/20 3:24 PM, Asaf Penso wrote:
>> From: Matan Azrad <matan@mellanox.com>
>>
>> Add support for the next 2 callbacks:
>> get_vfio_device_fd and get_notify_area.
>>
>> This will allow direct HW doorbell ringing from guest and will save CPU
>> usage in host.
>>
>> By this patch, the QEMU will map the physical address of the virtio
>> device in guest directly to the physical address of the HW device
>> doorbell.
>>
>> The guest doorbell write is 2 bytes transaction while some Mellanox nics
>> support only 4 bytes transactions.
>>
>> Remove ConnectX-5 and BF1 devices support which don't support 2B
>> doorbell writes for HW triggering.
> 
> Couldn't we have different rte_vdpa_dev_ops depending on whether
> doorbell write support?

I'll take the patch so that it lands into -rc1.
But I would like to discuss the opportunity to have different dev_ops
based on the device IDs, so that BF can still be supported. Maybe that
could be done in -rc2?

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime



^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [dpdk-dev] [PATCH 4/4] vdpa/mlx5: add log prints
  2020-03-24 14:24 ` [dpdk-dev] [PATCH 4/4] vdpa/mlx5: add log prints Asaf Penso
@ 2020-04-17 11:54   ` Maxime Coquelin
  0 siblings, 0 replies; 13+ messages in thread
From: Maxime Coquelin @ 2020-04-17 11:54 UTC (permalink / raw)
  To: Asaf Penso, dev; +Cc: Viacheslav Ovsiienko, Shahaf Shuler, Matan Azrad



On 3/24/20 3:24 PM, Asaf Penso wrote:
> From: Matan Azrad <matan@mellanox.com>
> 
> Add log prints to improve driver status following.
> 
> Signed-off-by: Matan Azrad <matan@mellanox.com>
> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
>  drivers/vdpa/mlx5/mlx5_vdpa.c       | 2 ++
>  drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 2 ++
>  2 files changed, 4 insertions(+)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [dpdk-dev] [PATCH 0/4] vdpa/mlx5: support direct notification
  2020-03-24 14:24 [dpdk-dev] [PATCH 0/4] vdpa/mlx5: support direct notification Asaf Penso
                   ` (3 preceding siblings ...)
  2020-03-24 14:24 ` [dpdk-dev] [PATCH 4/4] vdpa/mlx5: add log prints Asaf Penso
@ 2020-04-17 17:14 ` Maxime Coquelin
  4 siblings, 0 replies; 13+ messages in thread
From: Maxime Coquelin @ 2020-04-17 17:14 UTC (permalink / raw)
  To: Asaf Penso, dev; +Cc: Viacheslav Ovsiienko, Shahaf Shuler



On 3/24/20 3:24 PM, Asaf Penso wrote:
> In order to save CPU usage in host and even make it 0 when guest work with poll mode,
> add support for direct doorbell ringing.
> 
> Now, when the feature is supported in QEMU, the guest doorbell address is mapped directly to the HW doorbell space and the host SW relay is not used. 
> 
> Matan Azrad (4):
>   vdpa/mlx5: move virtual doorbell alloc to probe
>   vdpa/mlx5: support direct HW notifications
>   vdpa/mlx5: validate notifier configuration
>   vdpa/mlx5: add log prints
> 
>  drivers/vdpa/mlx5/mlx5_vdpa.c       | 90 +++++++++++++++++++++++++++++++------
>  drivers/vdpa/mlx5/mlx5_vdpa.h       |  1 +
>  drivers/vdpa/mlx5/mlx5_vdpa_steer.c |  2 +
>  drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 39 ++++++++--------
>  4 files changed, 100 insertions(+), 32 deletions(-)
> 

Applied to dpdk-next-virtio/master

Thanks,
Maxime


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [dpdk-dev] [PATCH 2/4] vdpa/mlx5: support direct HW notifications
  2020-04-17 11:54     ` Maxime Coquelin
@ 2020-04-26  7:06       ` Matan Azrad
  2020-04-27  7:45         ` Maxime Coquelin
  0 siblings, 1 reply; 13+ messages in thread
From: Matan Azrad @ 2020-04-26  7:06 UTC (permalink / raw)
  To: Maxime Coquelin, Asaf Penso, dev; +Cc: Slava Ovsiienko, Shahaf Shuler



From: Maxime Coquelin <maxime.coquelin@redhat.com>
> On 4/15/20 11:47 AM, Maxime Coquelin wrote:
> >
> >
> > On 3/24/20 3:24 PM, Asaf Penso wrote:
> >> From: Matan Azrad <matan@mellanox.com>
> >>
> >> Add support for the next 2 callbacks:
> >> get_vfio_device_fd and get_notify_area.
> >>
> >> This will allow direct HW doorbell ringing from guest and will save
> >> CPU usage in host.
> >>
> >> By this patch, the QEMU will map the physical address of the virtio
> >> device in guest directly to the physical address of the HW device
> >> doorbell.
> >>
> >> The guest doorbell write is 2 bytes transaction while some Mellanox
> >> nics support only 4 bytes transactions.
> >>
> >> Remove ConnectX-5 and BF1 devices support which don't support 2B
> >> doorbell writes for HW triggering.
> >
> > Couldn't we have different rte_vdpa_dev_ops depending on whether
> > doorbell write support?
> 
> I'll take the patch so that it lands into -rc1.
> But I would like to discuss the opportunity to have different dev_ops based
> on the device IDs, so that BF can still be supported. Maybe that could be
> done in -rc2?

Mellanox decided to remove BF1 from GA and from upstream support.
GA devices are CX6dx + BF2.

> 
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> 
> Thanks,
> Maxime
> 


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [dpdk-dev] [PATCH 2/4] vdpa/mlx5: support direct HW notifications
  2020-04-26  7:06       ` Matan Azrad
@ 2020-04-27  7:45         ` Maxime Coquelin
  0 siblings, 0 replies; 13+ messages in thread
From: Maxime Coquelin @ 2020-04-27  7:45 UTC (permalink / raw)
  To: Matan Azrad, Asaf Penso, dev; +Cc: Slava Ovsiienko, Shahaf Shuler



On 4/26/20 9:06 AM, Matan Azrad wrote:
> 
> 
> From: Maxime Coquelin <maxime.coquelin@redhat.com>
>> On 4/15/20 11:47 AM, Maxime Coquelin wrote:
>>>
>>>
>>> On 3/24/20 3:24 PM, Asaf Penso wrote:
>>>> From: Matan Azrad <matan@mellanox.com>
>>>>
>>>> Add support for the next 2 callbacks:
>>>> get_vfio_device_fd and get_notify_area.
>>>>
>>>> This will allow direct HW doorbell ringing from guest and will save
>>>> CPU usage in host.
>>>>
>>>> By this patch, the QEMU will map the physical address of the virtio
>>>> device in guest directly to the physical address of the HW device
>>>> doorbell.
>>>>
>>>> The guest doorbell write is 2 bytes transaction while some Mellanox
>>>> nics support only 4 bytes transactions.
>>>>
>>>> Remove ConnectX-5 and BF1 devices support which don't support 2B
>>>> doorbell writes for HW triggering.
>>>
>>> Couldn't we have different rte_vdpa_dev_ops depending on whether
>>> doorbell write support?
>>
>> I'll take the patch so that it lands into -rc1.
>> But I would like to discuss the opportunity to have different dev_ops based
>> on the device IDs, so that BF can still be supported. Maybe that could be
>> done in -rc2?
> 
> Mellanox decided to remove BF1 from GA and from upstream support.
> GA devices are CX6dx + BF2.

Thanks for the clarification.

Maxime

>>
>> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
>>
>> Thanks,
>> Maxime
>>
> 


^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2020-04-27  7:45 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-03-24 14:24 [dpdk-dev] [PATCH 0/4] vdpa/mlx5: support direct notification Asaf Penso
2020-03-24 14:24 ` [dpdk-dev] [PATCH 1/4] vdpa/mlx5: move virtual doorbell alloc to probe Asaf Penso
2020-04-15  9:44   ` Maxime Coquelin
2020-03-24 14:24 ` [dpdk-dev] [PATCH 2/4] vdpa/mlx5: support direct HW notifications Asaf Penso
2020-04-15  9:47   ` Maxime Coquelin
2020-04-17 11:54     ` Maxime Coquelin
2020-04-26  7:06       ` Matan Azrad
2020-04-27  7:45         ` Maxime Coquelin
2020-03-24 14:24 ` [dpdk-dev] [PATCH 3/4] vdpa/mlx5: validate notifier configuration Asaf Penso
2020-04-15  9:54   ` Maxime Coquelin
2020-03-24 14:24 ` [dpdk-dev] [PATCH 4/4] vdpa/mlx5: add log prints Asaf Penso
2020-04-17 11:54   ` Maxime Coquelin
2020-04-17 17:14 ` [dpdk-dev] [PATCH 0/4] vdpa/mlx5: support direct notification Maxime Coquelin

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).