[dpdk-dev] [PATCH 0/4] vdpa/mlx5: support direct notification

[dpdk-dev] [PATCH 0/4] vdpa/mlx5: support direct notification

[Asaf Penso]

In order to save CPU usage in host and even make it 0 when guest work with poll mode,
add support for direct doorbell ringing.

Now, when the feature is supported in QEMU, the guest doorbell address is mapped directly to the HW doorbell space and the host SW relay is not used. 

Matan Azrad (4):
  vdpa/mlx5: move virtual doorbell alloc to probe
  vdpa/mlx5: support direct HW notifications
  vdpa/mlx5: validate notifier configuration
  vdpa/mlx5: add log prints

 drivers/vdpa/mlx5/mlx5_vdpa.c       | 90 +++++++++++++++++++++++++++++++------
 drivers/vdpa/mlx5/mlx5_vdpa.h       |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa_steer.c |  2 +
 drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 39 ++++++++--------
 4 files changed, 100 insertions(+), 32 deletions(-)

-- 
1.8.3.1

[dpdk-dev] [PATCH 1/4] vdpa/mlx5: move virtual doorbell alloc to probe

[Asaf Penso]

From: Matan Azrad <matan@mellanox.com>

The configure and close operations may be called a lot of time by vhost
library according to the virtio connections in the guest.

VAR is the device memory space for the virtio queues doorbells.
Each VAR page can be shared for more than one queue while its owner must
synchronize the writes to it.

The mlx5 driver allocates single VAR page for all its queues.

Therefore, it is better to allocate it in probe device level instead of
creating and destroying it per new connection.

Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
---
 drivers/vdpa/mlx5/mlx5_vdpa.c       | 14 +++++++++++++-
 drivers/vdpa/mlx5/mlx5_vdpa_virtq.c |  9 ---------
 2 files changed, 13 insertions(+), 10 deletions(-)

diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 97d914a..5542c29 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -447,6 +447,11 @@
 	priv->ctx = ctx;
 	priv->dev_addr.pci_addr = pci_dev->addr;
 	priv->dev_addr.type = PCI_ADDR;
+	priv->var = mlx5_glue->dv_alloc_var(ctx, 0);
+	if (!priv->var) {
+		DRV_LOG(ERR, "Failed to allocate VAR %u.\n", errno);
+		goto error;
+	}
 	priv->id = rte_vdpa_register_device(&priv->dev_addr, &mlx5_vdpa_ops);
 	if (priv->id < 0) {
 		DRV_LOG(ERR, "Failed to register vDPA device.");
@@ -461,8 +466,11 @@
 	return 0;
 
 error:
-	if (priv)
+	if (priv) {
+		if (priv->var)
+			mlx5_glue->dv_free_var(priv->var);
 		rte_free(priv);
+	}
 	if (ctx)
 		mlx5_glue->close_device(ctx);
 	return -rte_errno;
@@ -499,6 +507,10 @@
 	if (found) {
 		if (priv->configured)
 			mlx5_vdpa_dev_close(priv->vid);
+		if (priv->var) {
+			mlx5_glue->dv_free_var(priv->var);
+			priv->var = NULL;
+		}
 		mlx5_glue->close_device(priv->ctx);
 		rte_free(priv);
 	}
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
index 2312331..6390385 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
@@ -105,10 +105,6 @@
 		claim_zero(munmap(priv->virtq_db_addr, priv->var->length));
 		priv->virtq_db_addr = NULL;
 	}
-	if (priv->var) {
-		mlx5_glue->dv_free_var(priv->var);
-		priv->var = NULL;
-	}
 	priv->features = 0;
 }
 
@@ -343,11 +339,6 @@
 		DRV_LOG(ERR, "Failed to configure negotiated features.");
 		return -1;
 	}
-	priv->var = mlx5_glue->dv_alloc_var(priv->ctx, 0);
-	if (!priv->var) {
-		DRV_LOG(ERR, "Failed to allocate VAR %u.\n", errno);
-		return -1;
-	}
 	/* Always map the entire page. */
 	priv->virtq_db_addr = mmap(NULL, priv->var->length, PROT_READ |
 				   PROT_WRITE, MAP_SHARED, priv->ctx->cmd_fd,
-- 
1.8.3.1

Re: [dpdk-dev] [PATCH 1/4] vdpa/mlx5: move virtual doorbell alloc to probe

[Maxime Coquelin]

On 3/24/20 3:24 PM, Asaf Penso wrote:
> From: Matan Azrad <matan@mellanox.com>
> 
> The configure and close operations may be called a lot of time by vhost
> library according to the virtio connections in the guest.
> 
> VAR is the device memory space for the virtio queues doorbells.
> Each VAR page can be shared for more than one queue while its owner must
> synchronize the writes to it.
> 
> The mlx5 driver allocates single VAR page for all its queues.
> 
> Therefore, it is better to allocate it in probe device level instead of
> creating and destroying it per new connection.
> 
> Signed-off-by: Matan Azrad <matan@mellanox.com>
> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
>  drivers/vdpa/mlx5/mlx5_vdpa.c       | 14 +++++++++++++-
>  drivers/vdpa/mlx5/mlx5_vdpa_virtq.c |  9 ---------
>  2 files changed, 13 insertions(+), 10 deletions(-)

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[dpdk-dev] [PATCH 2/4] vdpa/mlx5: support direct HW notifications

[Asaf Penso]

From: Matan Azrad <matan@mellanox.com>

Add support for the next 2 callbacks:
get_vfio_device_fd and get_notify_area.

This will allow direct HW doorbell ringing from guest and will save CPU
usage in host.

By this patch, the QEMU will map the physical address of the virtio
device in guest directly to the physical address of the HW device
doorbell.

The guest doorbell write is 2 bytes transaction while some Mellanox nics
support only 4 bytes transactions.

Remove ConnectX-5 and BF1 devices support which don't support 2B
doorbell writes for HW triggering.

Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
---
 drivers/vdpa/mlx5/mlx5_vdpa.c | 74 ++++++++++++++++++++++++++++++++++++-------
 drivers/vdpa/mlx5/mlx5_vdpa.h |  1 +
 2 files changed, 63 insertions(+), 12 deletions(-)

diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 5542c29..4eb6abf 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -133,6 +133,29 @@
 }
 
 static int
+mlx5_vdpa_direct_db_prepare(struct mlx5_vdpa_priv *priv)
+{
+	int ret;
+
+	if (priv->direct_notifier) {
+		ret = rte_vhost_host_notifier_ctrl(priv->vid, false);
+		if (ret != 0) {
+			DRV_LOG(INFO, "Direct HW notifier FD cannot be "
+				"destroyed for device %d: %d.", priv->vid, ret);
+			return -1;
+		}
+		priv->direct_notifier = 0;
+	}
+	ret = rte_vhost_host_notifier_ctrl(priv->vid, true);
+	if (ret != 0)
+		DRV_LOG(INFO, "Direct HW notifier FD cannot be configured for"
+			" device %d: %d.", priv->vid, ret);
+	else
+		priv->direct_notifier = 1;
+	return 0;
+}
+
+static int
 mlx5_vdpa_features_set(int vid)
 {
 	int did = rte_vhost_get_vdpa_device_id(vid);
@@ -209,8 +232,9 @@
 		return -1;
 	}
 	priv->vid = vid;
-	if (mlx5_vdpa_mem_register(priv) || mlx5_vdpa_virtqs_prepare(priv) ||
-	    mlx5_vdpa_steer_setup(priv) || mlx5_vdpa_cqe_event_setup(priv)) {
+	if (mlx5_vdpa_mem_register(priv) || mlx5_vdpa_direct_db_prepare(priv) ||
+	    mlx5_vdpa_virtqs_prepare(priv) || mlx5_vdpa_steer_setup(priv) ||
+	    mlx5_vdpa_cqe_event_setup(priv)) {
 		mlx5_vdpa_dev_close(vid);
 		return -1;
 	}
@@ -218,6 +242,40 @@
 	return 0;
 }
 
+static int
+mlx5_vdpa_get_device_fd(int vid)
+{
+	int did = rte_vhost_get_vdpa_device_id(vid);
+	struct mlx5_vdpa_priv *priv = mlx5_vdpa_find_priv_resource_by_did(did);
+
+	if (priv == NULL) {
+		DRV_LOG(ERR, "Invalid device id: %d.", did);
+		return -EINVAL;
+	}
+	return priv->ctx->cmd_fd;
+}
+
+static int
+mlx5_vdpa_get_notify_area(int vid, int qid, uint64_t *offset, uint64_t *size)
+{
+	int did = rte_vhost_get_vdpa_device_id(vid);
+	struct mlx5_vdpa_priv *priv = mlx5_vdpa_find_priv_resource_by_did(did);
+
+	RTE_SET_USED(qid);
+	if (priv == NULL) {
+		DRV_LOG(ERR, "Invalid device id: %d.", did);
+		return -EINVAL;
+	}
+	if (!priv->var) {
+		DRV_LOG(ERR, "VAR was not created for device %d, is the device"
+			" configured?.", did);
+		return -EINVAL;
+	}
+	*offset = priv->var->mmap_off;
+	*size = priv->var->length;
+	return 0;
+}
+
 static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
 	.get_queue_num = mlx5_vdpa_get_queue_num,
 	.get_features = mlx5_vdpa_get_vdpa_features,
@@ -228,8 +286,8 @@
 	.set_features = mlx5_vdpa_features_set,
 	.migration_done = NULL,
 	.get_vfio_group_fd = NULL,
-	.get_vfio_device_fd = NULL,
-	.get_notify_area = NULL,
+	.get_vfio_device_fd = mlx5_vdpa_get_device_fd,
+	.get_notify_area = mlx5_vdpa_get_notify_area,
 };
 
 static struct ibv_device *
@@ -520,14 +578,6 @@
 static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = {
 	{
 		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
-			       PCI_DEVICE_ID_MELLANOX_CONNECTX5BF)
-	},
-	{
-		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
-			       PCI_DEVICE_ID_MELLANOX_CONNECTX5BFVF)
-	},
-	{
-		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
 				PCI_DEVICE_ID_MELLANOX_CONNECTX6)
 	},
 	{
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
index 3324c9d..75af410 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
@@ -100,6 +100,7 @@ struct mlx5_vdpa_steer {
 struct mlx5_vdpa_priv {
 	TAILQ_ENTRY(mlx5_vdpa_priv) next;
 	uint8_t configured;
+	uint8_t direct_notifier; /* Whether direct notifier is on or off. */
 	int id; /* vDPA device id. */
 	int vid; /* vhost device id. */
 	struct ibv_context *ctx; /* Device context. */
-- 
1.8.3.1

Re: [dpdk-dev] [PATCH 2/4] vdpa/mlx5: support direct HW notifications

[Maxime Coquelin]

On 3/24/20 3:24 PM, Asaf Penso wrote:
> From: Matan Azrad <matan@mellanox.com>
> 
> Add support for the next 2 callbacks:
> get_vfio_device_fd and get_notify_area.
> 
> This will allow direct HW doorbell ringing from guest and will save CPU
> usage in host.
> 
> By this patch, the QEMU will map the physical address of the virtio
> device in guest directly to the physical address of the HW device
> doorbell.
> 
> The guest doorbell write is 2 bytes transaction while some Mellanox nics
> support only 4 bytes transactions.
> 
> Remove ConnectX-5 and BF1 devices support which don't support 2B
> doorbell writes for HW triggering.

Couldn't we have different rte_vdpa_dev_ops depending on whether
doorbell write support?

> Signed-off-by: Matan Azrad <matan@mellanox.com>
> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
>  drivers/vdpa/mlx5/mlx5_vdpa.c | 74 ++++++++++++++++++++++++++++++++++++-------
>  drivers/vdpa/mlx5/mlx5_vdpa.h |  1 +
>  2 files changed, 63 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
> index 5542c29..4eb6abf 100644
> --- a/drivers/vdpa/mlx5/mlx5_vdpa.c
> +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
> @@ -133,6 +133,29 @@
>  }
>  
>  static int
> +mlx5_vdpa_direct_db_prepare(struct mlx5_vdpa_priv *priv)
> +{
> +	int ret;
> +
> +	if (priv->direct_notifier) {
> +		ret = rte_vhost_host_notifier_ctrl(priv->vid, false);
> +		if (ret != 0) {
> +			DRV_LOG(INFO, "Direct HW notifier FD cannot be "
> +				"destroyed for device %d: %d.", priv->vid, ret);
> +			return -1;
> +		}
> +		priv->direct_notifier = 0;
> +	}
> +	ret = rte_vhost_host_notifier_ctrl(priv->vid, true);
> +	if (ret != 0)
> +		DRV_LOG(INFO, "Direct HW notifier FD cannot be configured for"
> +			" device %d: %d.", priv->vid, ret);
> +	else
> +		priv->direct_notifier = 1;
> +	return 0;
> +}
> +
> +static int
>  mlx5_vdpa_features_set(int vid)
>  {
>  	int did = rte_vhost_get_vdpa_device_id(vid);
> @@ -209,8 +232,9 @@
>  		return -1;
>  	}
>  	priv->vid = vid;
> -	if (mlx5_vdpa_mem_register(priv) || mlx5_vdpa_virtqs_prepare(priv) ||
> -	    mlx5_vdpa_steer_setup(priv) || mlx5_vdpa_cqe_event_setup(priv)) {
> +	if (mlx5_vdpa_mem_register(priv) || mlx5_vdpa_direct_db_prepare(priv) ||
> +	    mlx5_vdpa_virtqs_prepare(priv) || mlx5_vdpa_steer_setup(priv) ||
> +	    mlx5_vdpa_cqe_event_setup(priv)) {
>  		mlx5_vdpa_dev_close(vid);
>  		return -1;
>  	}
> @@ -218,6 +242,40 @@
>  	return 0;
>  }
>  
> +static int
> +mlx5_vdpa_get_device_fd(int vid)
> +{
> +	int did = rte_vhost_get_vdpa_device_id(vid);
> +	struct mlx5_vdpa_priv *priv = mlx5_vdpa_find_priv_resource_by_did(did);
> +
> +	if (priv == NULL) {
> +		DRV_LOG(ERR, "Invalid device id: %d.", did);
> +		return -EINVAL;
> +	}
> +	return priv->ctx->cmd_fd;
> +}
> +
> +static int
> +mlx5_vdpa_get_notify_area(int vid, int qid, uint64_t *offset, uint64_t *size)
> +{
> +	int did = rte_vhost_get_vdpa_device_id(vid);
> +	struct mlx5_vdpa_priv *priv = mlx5_vdpa_find_priv_resource_by_did(did);
> +
> +	RTE_SET_USED(qid);
> +	if (priv == NULL) {
> +		DRV_LOG(ERR, "Invalid device id: %d.", did);
> +		return -EINVAL;
> +	}
> +	if (!priv->var) {
> +		DRV_LOG(ERR, "VAR was not created for device %d, is the device"
> +			" configured?.", did);
> +		return -EINVAL;
> +	}
> +	*offset = priv->var->mmap_off;
> +	*size = priv->var->length;
> +	return 0;
> +}
> +
>  static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
>  	.get_queue_num = mlx5_vdpa_get_queue_num,
>  	.get_features = mlx5_vdpa_get_vdpa_features,
> @@ -228,8 +286,8 @@
>  	.set_features = mlx5_vdpa_features_set,
>  	.migration_done = NULL,
>  	.get_vfio_group_fd = NULL,
> -	.get_vfio_device_fd = NULL,
> -	.get_notify_area = NULL,
> +	.get_vfio_device_fd = mlx5_vdpa_get_device_fd,
> +	.get_notify_area = mlx5_vdpa_get_notify_area,
>  };
>  
>  static struct ibv_device *
> @@ -520,14 +578,6 @@
>  static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = {
>  	{
>  		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
> -			       PCI_DEVICE_ID_MELLANOX_CONNECTX5BF)
> -	},
> -	{
> -		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
> -			       PCI_DEVICE_ID_MELLANOX_CONNECTX5BFVF)
> -	},
> -	{
> -		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
>  				PCI_DEVICE_ID_MELLANOX_CONNECTX6)
>  	},
>  	{
> diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
> index 3324c9d..75af410 100644
> --- a/drivers/vdpa/mlx5/mlx5_vdpa.h
> +++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
> @@ -100,6 +100,7 @@ struct mlx5_vdpa_steer {
>  struct mlx5_vdpa_priv {
>  	TAILQ_ENTRY(mlx5_vdpa_priv) next;
>  	uint8_t configured;
> +	uint8_t direct_notifier; /* Whether direct notifier is on or off. */
>  	int id; /* vDPA device id. */
>  	int vid; /* vhost device id. */
>  	struct ibv_context *ctx; /* Device context. */
> 

Re: [dpdk-dev] [PATCH 2/4] vdpa/mlx5: support direct HW notifications

[Maxime Coquelin]

On 4/15/20 11:47 AM, Maxime Coquelin wrote:
> 
> 
> On 3/24/20 3:24 PM, Asaf Penso wrote:
>> From: Matan Azrad <matan@mellanox.com>
>>
>> Add support for the next 2 callbacks:
>> get_vfio_device_fd and get_notify_area.
>>
>> This will allow direct HW doorbell ringing from guest and will save CPU
>> usage in host.
>>
>> By this patch, the QEMU will map the physical address of the virtio
>> device in guest directly to the physical address of the HW device
>> doorbell.
>>
>> The guest doorbell write is 2 bytes transaction while some Mellanox nics
>> support only 4 bytes transactions.
>>
>> Remove ConnectX-5 and BF1 devices support which don't support 2B
>> doorbell writes for HW triggering.
> 
> Couldn't we have different rte_vdpa_dev_ops depending on whether
> doorbell write support?

I'll take the patch so that it lands into -rc1.
But I would like to discuss the opportunity to have different dev_ops
based on the device IDs, so that BF can still be supported. Maybe that
could be done in -rc2?

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

Re: [dpdk-dev] [PATCH 2/4] vdpa/mlx5: support direct HW notifications

[Matan Azrad]

From: Maxime Coquelin <maxime.coquelin@redhat.com>
> On 4/15/20 11:47 AM, Maxime Coquelin wrote:
> >
> >
> > On 3/24/20 3:24 PM, Asaf Penso wrote:
> >> From: Matan Azrad <matan@mellanox.com>
> >>
> >> Add support for the next 2 callbacks:
> >> get_vfio_device_fd and get_notify_area.
> >>
> >> This will allow direct HW doorbell ringing from guest and will save
> >> CPU usage in host.
> >>
> >> By this patch, the QEMU will map the physical address of the virtio
> >> device in guest directly to the physical address of the HW device
> >> doorbell.
> >>
> >> The guest doorbell write is 2 bytes transaction while some Mellanox
> >> nics support only 4 bytes transactions.
> >>
> >> Remove ConnectX-5 and BF1 devices support which don't support 2B
> >> doorbell writes for HW triggering.
> >
> > Couldn't we have different rte_vdpa_dev_ops depending on whether
> > doorbell write support?
> 
> I'll take the patch so that it lands into -rc1.
> But I would like to discuss the opportunity to have different dev_ops based
> on the device IDs, so that BF can still be supported. Maybe that could be
> done in -rc2?

Mellanox decided to remove BF1 from GA and from upstream support.
GA devices are CX6dx + BF2.

> 
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> 
> Thanks,
> Maxime
> 

Re: [dpdk-dev] [PATCH 2/4] vdpa/mlx5: support direct HW notifications

[Maxime Coquelin]

On 4/26/20 9:06 AM, Matan Azrad wrote:
> 
> 
> From: Maxime Coquelin <maxime.coquelin@redhat.com>
>> On 4/15/20 11:47 AM, Maxime Coquelin wrote:
>>>
>>>
>>> On 3/24/20 3:24 PM, Asaf Penso wrote:
>>>> From: Matan Azrad <matan@mellanox.com>
>>>>
>>>> Add support for the next 2 callbacks:
>>>> get_vfio_device_fd and get_notify_area.
>>>>
>>>> This will allow direct HW doorbell ringing from guest and will save
>>>> CPU usage in host.
>>>>
>>>> By this patch, the QEMU will map the physical address of the virtio
>>>> device in guest directly to the physical address of the HW device
>>>> doorbell.
>>>>
>>>> The guest doorbell write is 2 bytes transaction while some Mellanox
>>>> nics support only 4 bytes transactions.
>>>>
>>>> Remove ConnectX-5 and BF1 devices support which don't support 2B
>>>> doorbell writes for HW triggering.
>>>
>>> Couldn't we have different rte_vdpa_dev_ops depending on whether
>>> doorbell write support?
>>
>> I'll take the patch so that it lands into -rc1.
>> But I would like to discuss the opportunity to have different dev_ops based
>> on the device IDs, so that BF can still be supported. Maybe that could be
>> done in -rc2?
> 
> Mellanox decided to remove BF1 from GA and from upstream support.
> GA devices are CX6dx + BF2.

Thanks for the clarification.

Maxime

>>
>> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
>>
>> Thanks,
>> Maxime
>>
> 

[dpdk-dev] [PATCH 3/4] vdpa/mlx5: validate notifier configuration

[Asaf Penso]

From: Matan Azrad <matan@mellanox.com>

When both, direct and indirect notifier management cannot be
configured, return an error.

Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
---
 drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)

diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
index 6390385..8bebb92 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
@@ -46,7 +46,7 @@
 	int retries = MLX5_VDPA_INTR_RETRIES;
 	int ret = -EAGAIN;
 
-	if (virtq->intr_handle.fd) {
+	if (virtq->intr_handle.fd != -1) {
 		while (retries-- && ret == -EAGAIN) {
 			ret = rte_intr_callback_unregister(&virtq->intr_handle,
 							mlx5_vdpa_virtq_handler,
@@ -59,7 +59,7 @@
 				usleep(MLX5_VDPA_INTR_RETRIES_USEC);
 			}
 		}
-		memset(&virtq->intr_handle, 0, sizeof(virtq->intr_handle));
+		virtq->intr_handle.fd = -1;
 	}
 	if (virtq->virtq) {
 		claim_zero(mlx5_devx_cmd_destroy(virtq->virtq));
@@ -255,15 +255,25 @@
 	rte_write32(virtq->index, priv->virtq_db_addr);
 	/* Setup doorbell mapping. */
 	virtq->intr_handle.fd = vq.kickfd;
-	virtq->intr_handle.type = RTE_INTR_HANDLE_EXT;
-	if (rte_intr_callback_register(&virtq->intr_handle,
-				       mlx5_vdpa_virtq_handler, virtq)) {
-		virtq->intr_handle.fd = 0;
-		DRV_LOG(ERR, "Failed to register virtq %d interrupt.", index);
-		goto error;
+	if (virtq->intr_handle.fd == -1) {
+		DRV_LOG(WARNING, "Virtq %d kickfd is invalid.", index);
+		if (!priv->direct_notifier) {
+			DRV_LOG(ERR, "Virtq %d cannot be notified.", index);
+			goto error;
+		}
 	} else {
-		DRV_LOG(DEBUG, "Register fd %d interrupt for virtq %d.",
-			virtq->intr_handle.fd, index);
+		virtq->intr_handle.type = RTE_INTR_HANDLE_EXT;
+		if (rte_intr_callback_register(&virtq->intr_handle,
+					       mlx5_vdpa_virtq_handler,
+					       virtq)) {
+			virtq->intr_handle.fd = -1;
+			DRV_LOG(ERR, "Failed to register virtq %d interrupt.",
+				index);
+			goto error;
+		} else {
+			DRV_LOG(DEBUG, "Register fd %d interrupt for virtq %d.",
+				virtq->intr_handle.fd, index);
+		}
 	}
 	return 0;
 error:
-- 
1.8.3.1

Re: [dpdk-dev] [PATCH 3/4] vdpa/mlx5: validate notifier configuration

[Maxime Coquelin]

On 3/24/20 3:24 PM, Asaf Penso wrote:
> From: Matan Azrad <matan@mellanox.com>
> 
> When both, direct and indirect notifier management cannot be
> configured, return an error.
> 
> Signed-off-by: Matan Azrad <matan@mellanox.com>
> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
>  drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 30 ++++++++++++++++++++----------
>  1 file changed, 20 insertions(+), 10 deletions(-)

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[dpdk-dev] [PATCH 4/4] vdpa/mlx5: add log prints

[Asaf Penso]

From: Matan Azrad <matan@mellanox.com>

Add log prints to improve driver status following.

Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
---
 drivers/vdpa/mlx5/mlx5_vdpa.c       | 2 ++
 drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 4eb6abf..f10647b 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -214,6 +214,7 @@
 	mlx5_vdpa_mem_dereg(priv);
 	priv->configured = 0;
 	priv->vid = 0;
+	DRV_LOG(INFO, "vDPA device %d was closed.", vid);
 	return ret;
 }
 
@@ -239,6 +240,7 @@
 		return -1;
 	}
 	priv->configured = 1;
+	DRV_LOG(INFO, "vDPA device %d was configured.", vid);
 	return 0;
 }
 
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_steer.c b/drivers/vdpa/mlx5/mlx5_vdpa_steer.c
index 36017f1..8e1cf0a 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_steer.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_steer.c
@@ -123,6 +123,8 @@
 	struct mlx5_vdpa_priv *priv = virtq->priv;
 	int ret = 0;
 
+	DRV_LOG(INFO, "Update virtq %d status %sable -> %sable.", virtq->index,
+		virtq->enable ? "en" : "dis", enable ? "en" : "dis");
 	if (virtq->enable == !!enable)
 		return 0;
 	virtq->enable = !!enable;
-- 
1.8.3.1

Re: [dpdk-dev] [PATCH 4/4] vdpa/mlx5: add log prints

[Maxime Coquelin]

On 3/24/20 3:24 PM, Asaf Penso wrote:
> From: Matan Azrad <matan@mellanox.com>
> 
> Add log prints to improve driver status following.
> 
> Signed-off-by: Matan Azrad <matan@mellanox.com>
> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> ---
>  drivers/vdpa/mlx5/mlx5_vdpa.c       | 2 ++
>  drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 2 ++
>  2 files changed, 4 insertions(+)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

Re: [dpdk-dev] [PATCH 0/4] vdpa/mlx5: support direct notification

[Maxime Coquelin]

On 3/24/20 3:24 PM, Asaf Penso wrote:
> In order to save CPU usage in host and even make it 0 when guest work with poll mode,
> add support for direct doorbell ringing.
> 
> Now, when the feature is supported in QEMU, the guest doorbell address is mapped directly to the HW doorbell space and the host SW relay is not used. 
> 
> Matan Azrad (4):
>   vdpa/mlx5: move virtual doorbell alloc to probe
>   vdpa/mlx5: support direct HW notifications
>   vdpa/mlx5: validate notifier configuration
>   vdpa/mlx5: add log prints
> 
>  drivers/vdpa/mlx5/mlx5_vdpa.c       | 90 +++++++++++++++++++++++++++++++------
>  drivers/vdpa/mlx5/mlx5_vdpa.h       |  1 +
>  drivers/vdpa/mlx5/mlx5_vdpa_steer.c |  2 +
>  drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 39 ++++++++--------
>  4 files changed, 100 insertions(+), 32 deletions(-)
> 

Applied to dpdk-next-virtio/master

Thanks,
Maxime