[dpdk-dev] [PATCH] vhost: add unsafe API to check inflight packets

[dpdk-dev] [PATCH] vhost: add unsafe API to check inflight packets

[Xuan Ding]

In async data path, when vring state changes, it is necessary to
know the number of inflight packets in DMA engine. This patch
provides a thread unsafe API to return the number of inflight
packets without using any lock.

Signed-off-by: Xuan Ding <xuan.ding@intel.com>
---
 doc/guides/prog_guide/vhost_lib.rst    |  5 +++++
 doc/guides/rel_notes/release_21_11.rst |  5 +++++
 lib/vhost/rte_vhost_async.h            | 14 ++++++++++++++
 lib/vhost/version.map                  |  3 +++
 lib/vhost/vhost.c                      | 25 +++++++++++++++++++++++++
 5 files changed, 52 insertions(+)

diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
index 8874033165..b4b1134f54 100644
--- a/doc/guides/prog_guide/vhost_lib.rst
+++ b/doc/guides/prog_guide/vhost_lib.rst
@@ -305,6 +305,11 @@ The following is an overview of some key Vhost API functions:
   This function returns the amount of in-flight packets for the vhost
   queue using async acceleration.
 
+    ``rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id)``
+
+  Get the number of inflight packets for a vhost queue without
+  performing any locking.
+
 * ``rte_vhost_clear_queue_thread_unsafe(vid, queue_id, **pkts, count)``
 
   Clear inflight packets which are submitted to DMA engine in vhost async data
diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index 675b573834..db080e9490 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added vhost API to get the number of inflight packets.**
+
+  Added an API which can get the number of inflight packets in
+  vhost async data path.
+
 * **Enabled new devargs parser.**
 
   * Enabled devargs syntax
diff --git a/lib/vhost/rte_vhost_async.h b/lib/vhost/rte_vhost_async.h
index b25ff446f7..0af414bf78 100644
--- a/lib/vhost/rte_vhost_async.h
+++ b/lib/vhost/rte_vhost_async.h
@@ -246,6 +246,20 @@ uint16_t rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id,
 __rte_experimental
 int rte_vhost_async_get_inflight(int vid, uint16_t queue_id);
 
+/**
+ * This function is lock-free version to return the amount of in-flight
+ * packets for the vhost queue which uses async channel acceleration.
+ *
+ * @param vid
+ *  id of vhost device to enqueue data
+ * @param queue_id
+ *  queue id to enqueue data
+ * @return
+ *  the amount of in-flight packets on success; -1 on failure
+ */
+__rte_experimental
+int rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id);
+
 /**
  * This function checks async completion status and clear packets for
  * a specific vhost device queue. Packets which are inflight will be
diff --git a/lib/vhost/version.map b/lib/vhost/version.map
index c92a9d4962..b150dc408d 100644
--- a/lib/vhost/version.map
+++ b/lib/vhost/version.map
@@ -85,4 +85,7 @@ EXPERIMENTAL {
 	rte_vhost_async_channel_register_thread_unsafe;
 	rte_vhost_async_channel_unregister_thread_unsafe;
 	rte_vhost_clear_queue_thread_unsafe;
+
+	#added in 21.11
+	rte_vhost_async_get_inflight_thread_unsafe;
 };
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 355ff37651..df96f84873 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -1886,5 +1886,30 @@ int rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
 	return ret;
 }
 
+int rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id)
+{
+	struct vhost_virtqueue *vq;
+	struct virtio_net *dev = get_device(vid);
+	int ret = -1;
+
+	if (dev == NULL)
+		return ret;
+
+	if (queue_id >= VHOST_MAX_VRING)
+		return ret;
+
+	vq = dev->virtqueue[queue_id];
+
+	if (vq == NULL)
+		return ret;
+
+	if (!vq->async_registered)
+		return ret;
+
+	ret = vq->async_pkts_inflight_n;
+
+	return ret;
+}
+
 RTE_LOG_REGISTER_SUFFIX(vhost_config_log_level, config, INFO);
 RTE_LOG_REGISTER_SUFFIX(vhost_data_log_level, data, WARNING);
-- 
2.17.1

Re: [dpdk-dev] [PATCH] vhost: add unsafe API to check inflight packets

[Xia, Chenbo]

Hi Xuan,

> -----Original Message-----
> From: Ding, Xuan <xuan.ding@intel.com>
> Sent: Thursday, September 9, 2021 1:58 PM
> To: dev@dpdk.org; maxime.coquelin@redhat.com; Xia, Chenbo
> <chenbo.xia@intel.com>
> Cc: Hu, Jiayu <jiayu.hu@intel.com>; cheng.jiang@intel.com; Richardson, Bruce
> <bruce.richardson@intel.com>; Pai G, Sunil <sunil.pai.g@intel.com>; Ding, Xuan
> <xuan.ding@intel.com>
> Subject: [PATCH] vhost: add unsafe API to check inflight packets
> 
> In async data path, when vring state changes, it is necessary to
> know the number of inflight packets in DMA engine. This patch
> provides a thread unsafe API to return the number of inflight
> packets without using any lock.
> 
> Signed-off-by: Xuan Ding <xuan.ding@intel.com>
> ---
>  doc/guides/prog_guide/vhost_lib.rst    |  5 +++++
>  doc/guides/rel_notes/release_21_11.rst |  5 +++++
>  lib/vhost/rte_vhost_async.h            | 14 ++++++++++++++
>  lib/vhost/version.map                  |  3 +++
>  lib/vhost/vhost.c                      | 25 +++++++++++++++++++++++++
>  5 files changed, 52 insertions(+)
> 
> diff --git a/doc/guides/prog_guide/vhost_lib.rst
> b/doc/guides/prog_guide/vhost_lib.rst
> index 8874033165..b4b1134f54 100644
> --- a/doc/guides/prog_guide/vhost_lib.rst
> +++ b/doc/guides/prog_guide/vhost_lib.rst
> @@ -305,6 +305,11 @@ The following is an overview of some key Vhost API
> functions:
>    This function returns the amount of in-flight packets for the vhost
>    queue using async acceleration.
> 
> +    ``rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id)``
> +
> +  Get the number of inflight packets for a vhost queue without
> +  performing any locking.
> +
>  * ``rte_vhost_clear_queue_thread_unsafe(vid, queue_id, **pkts, count)``

This does not align with others. Please check.

> 
>    Clear inflight packets which are submitted to DMA engine in vhost async
> data
> diff --git a/doc/guides/rel_notes/release_21_11.rst
> b/doc/guides/rel_notes/release_21_11.rst
> index 675b573834..db080e9490 100644
> --- a/doc/guides/rel_notes/release_21_11.rst
> +++ b/doc/guides/rel_notes/release_21_11.rst
> @@ -55,6 +55,11 @@ New Features
>       Also, make sure to start the actual text at the margin.
>       =======================================================
> 
> +* **Added vhost API to get the number of inflight packets.**
> +
> +  Added an API which can get the number of inflight packets in
> +  vhost async data path.
> +

Please add 'without lock' or something similar as we already have a lock version.

>  * **Enabled new devargs parser.**
> 
>    * Enabled devargs syntax
> diff --git a/lib/vhost/rte_vhost_async.h b/lib/vhost/rte_vhost_async.h
> index b25ff446f7..0af414bf78 100644
> --- a/lib/vhost/rte_vhost_async.h
> +++ b/lib/vhost/rte_vhost_async.h
> @@ -246,6 +246,20 @@ uint16_t rte_vhost_poll_enqueue_completed(int vid,
> uint16_t queue_id,
>  __rte_experimental
>  int rte_vhost_async_get_inflight(int vid, uint16_t queue_id);
> 
> +/**
> + * This function is lock-free version to return the amount of in-flight
> + * packets for the vhost queue which uses async channel acceleration.
> + *
> + * @param vid
> + *  id of vhost device to enqueue data
> + * @param queue_id
> + *  queue id to enqueue data

You can also check dequeue inflight packets, right?

> + * @return
> + *  the amount of in-flight packets on success; -1 on failure
> + */
> +__rte_experimental
> +int rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id);
> +
>  /**
>   * This function checks async completion status and clear packets for
>   * a specific vhost device queue. Packets which are inflight will be
> diff --git a/lib/vhost/version.map b/lib/vhost/version.map
> index c92a9d4962..b150dc408d 100644
> --- a/lib/vhost/version.map
> +++ b/lib/vhost/version.map
> @@ -85,4 +85,7 @@ EXPERIMENTAL {
>  	rte_vhost_async_channel_register_thread_unsafe;
>  	rte_vhost_async_channel_unregister_thread_unsafe;
>  	rte_vhost_clear_queue_thread_unsafe;
> +
> +	#added in 21.11
> +	rte_vhost_async_get_inflight_thread_unsafe;
>  };
> diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
> index 355ff37651..df96f84873 100644
> --- a/lib/vhost/vhost.c
> +++ b/lib/vhost/vhost.c
> @@ -1886,5 +1886,30 @@ int rte_vhost_async_get_inflight(int vid, uint16_t
> queue_id)
>  	return ret;
>  }
> 
> +int rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id)

According to DPDK coding style, return type and func name should be in different
lines. I notice there are also some left to clean. May clean in another patch.

Thanks,
Chenbo 

> +{
> +	struct vhost_virtqueue *vq;
> +	struct virtio_net *dev = get_device(vid);
> +	int ret = -1;
> +
> +	if (dev == NULL)
> +		return ret;
> +
> +	if (queue_id >= VHOST_MAX_VRING)
> +		return ret;
> +
> +	vq = dev->virtqueue[queue_id];
> +
> +	if (vq == NULL)
> +		return ret;
> +
> +	if (!vq->async_registered)
> +		return ret;
> +
> +	ret = vq->async_pkts_inflight_n;
> +
> +	return ret;
> +}
> +
>  RTE_LOG_REGISTER_SUFFIX(vhost_config_log_level, config, INFO);
>  RTE_LOG_REGISTER_SUFFIX(vhost_data_log_level, data, WARNING);
> --
> 2.17.1

Re: [dpdk-dev] [PATCH] vhost: add unsafe API to check inflight packets

[Ding, Xuan]

Hi Chenbo,

> -----Original Message-----
> From: Xia, Chenbo <chenbo.xia@intel.com>
> Sent: Wednesday, September 15, 2021 2:49 PM
> To: Ding, Xuan <xuan.ding@intel.com>; dev@dpdk.org;
> maxime.coquelin@redhat.com
> Cc: Hu, Jiayu <jiayu.hu@intel.com>; cheng.jiang@intel.com; Richardson, Bruce
> <bruce.richardson@intel.com>; Pai G, Sunil <sunil.pai.g@intel.com>
> Subject: RE: [PATCH] vhost: add unsafe API to check inflight packets
> 
> Hi Xuan,
> 
> > -----Original Message-----
> > From: Ding, Xuan <xuan.ding@intel.com>
> > Sent: Thursday, September 9, 2021 1:58 PM
> > To: dev@dpdk.org; maxime.coquelin@redhat.com; Xia, Chenbo
> > <chenbo.xia@intel.com>
> > Cc: Hu, Jiayu <jiayu.hu@intel.com>; cheng.jiang@intel.com; Richardson, Bruce
> > <bruce.richardson@intel.com>; Pai G, Sunil <sunil.pai.g@intel.com>; Ding,
> Xuan
> > <xuan.ding@intel.com>
> > Subject: [PATCH] vhost: add unsafe API to check inflight packets
> >
> > In async data path, when vring state changes, it is necessary to
> > know the number of inflight packets in DMA engine. This patch
> > provides a thread unsafe API to return the number of inflight
> > packets without using any lock.
> >
> > Signed-off-by: Xuan Ding <xuan.ding@intel.com>
> > ---
> >  doc/guides/prog_guide/vhost_lib.rst    |  5 +++++
> >  doc/guides/rel_notes/release_21_11.rst |  5 +++++
> >  lib/vhost/rte_vhost_async.h            | 14 ++++++++++++++
> >  lib/vhost/version.map                  |  3 +++
> >  lib/vhost/vhost.c                      | 25 +++++++++++++++++++++++++
> >  5 files changed, 52 insertions(+)
> >
> > diff --git a/doc/guides/prog_guide/vhost_lib.rst
> > b/doc/guides/prog_guide/vhost_lib.rst
> > index 8874033165..b4b1134f54 100644
> > --- a/doc/guides/prog_guide/vhost_lib.rst
> > +++ b/doc/guides/prog_guide/vhost_lib.rst
> > @@ -305,6 +305,11 @@ The following is an overview of some key Vhost API
> > functions:
> >    This function returns the amount of in-flight packets for the vhost
> >    queue using async acceleration.
> >
> > +    ``rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id)``
> > +
> > +  Get the number of inflight packets for a vhost queue without
> > +  performing any locking.
> > +
> >  * ``rte_vhost_clear_queue_thread_unsafe(vid, queue_id, **pkts, count)``
> 
> This does not align with others. Please check.

Thanks, will fix it in next version.

> 
> >
> >    Clear inflight packets which are submitted to DMA engine in vhost async
> > data
> > diff --git a/doc/guides/rel_notes/release_21_11.rst
> > b/doc/guides/rel_notes/release_21_11.rst
> > index 675b573834..db080e9490 100644
> > --- a/doc/guides/rel_notes/release_21_11.rst
> > +++ b/doc/guides/rel_notes/release_21_11.rst
> > @@ -55,6 +55,11 @@ New Features
> >       Also, make sure to start the actual text at the margin.
> >       =======================================================
> >
> > +* **Added vhost API to get the number of inflight packets.**
> > +
> > +  Added an API which can get the number of inflight packets in
> > +  vhost async data path.
> > +
> 
> Please add 'without lock' or something similar as we already have a lock version.

You are right, add "without lock" is more accuracy.

> 
> >  * **Enabled new devargs parser.**
> >
> >    * Enabled devargs syntax
> > diff --git a/lib/vhost/rte_vhost_async.h b/lib/vhost/rte_vhost_async.h
> > index b25ff446f7..0af414bf78 100644
> > --- a/lib/vhost/rte_vhost_async.h
> > +++ b/lib/vhost/rte_vhost_async.h
> > @@ -246,6 +246,20 @@ uint16_t rte_vhost_poll_enqueue_completed(int vid,
> > uint16_t queue_id,
> >  __rte_experimental
> >  int rte_vhost_async_get_inflight(int vid, uint16_t queue_id);
> >
> > +/**
> > + * This function is lock-free version to return the amount of in-flight
> > + * packets for the vhost queue which uses async channel acceleration.
> > + *
> > + * @param vid
> > + *  id of vhost device to enqueue data
> > + * @param queue_id
> > + *  queue id to enqueue data
> 
> You can also check dequeue inflight packets, right?

Yes, this API applies to both enqueue and dequeue directions.

> 
> > + * @return
> > + *  the amount of in-flight packets on success; -1 on failure
> > + */
> > +__rte_experimental
> > +int rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id);
> > +
> >  /**
> >   * This function checks async completion status and clear packets for
> >   * a specific vhost device queue. Packets which are inflight will be
> > diff --git a/lib/vhost/version.map b/lib/vhost/version.map
> > index c92a9d4962..b150dc408d 100644
> > --- a/lib/vhost/version.map
> > +++ b/lib/vhost/version.map
> > @@ -85,4 +85,7 @@ EXPERIMENTAL {
> >  	rte_vhost_async_channel_register_thread_unsafe;
> >  	rte_vhost_async_channel_unregister_thread_unsafe;
> >  	rte_vhost_clear_queue_thread_unsafe;
> > +
> > +	#added in 21.11
> > +	rte_vhost_async_get_inflight_thread_unsafe;
> >  };
> > diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
> > index 355ff37651..df96f84873 100644
> > --- a/lib/vhost/vhost.c
> > +++ b/lib/vhost/vhost.c
> > @@ -1886,5 +1886,30 @@ int rte_vhost_async_get_inflight(int vid, uint16_t
> > queue_id)
> >  	return ret;
> >  }
> >
> > +int rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id)
> 
> According to DPDK coding style, return type and func name should be in
> different
> lines. I notice there are also some left to clean. May clean in another patch.

Thanks, notice the rte_vhost_async_get_inflight() API also has this problem.
Will fix both in next version.

Regards,
Xuan

> 
> Thanks,
> Chenbo
> 
> > +{
> > +	struct vhost_virtqueue *vq;
> > +	struct virtio_net *dev = get_device(vid);
> > +	int ret = -1;
> > +
> > +	if (dev == NULL)
> > +		return ret;
> > +
> > +	if (queue_id >= VHOST_MAX_VRING)
> > +		return ret;
> > +
> > +	vq = dev->virtqueue[queue_id];
> > +
> > +	if (vq == NULL)
> > +		return ret;
> > +
> > +	if (!vq->async_registered)
> > +		return ret;
> > +
> > +	ret = vq->async_pkts_inflight_n;
> > +
> > +	return ret;
> > +}
> > +
> >  RTE_LOG_REGISTER_SUFFIX(vhost_config_log_level, config, INFO);
> >  RTE_LOG_REGISTER_SUFFIX(vhost_data_log_level, data, WARNING);
> > --
> > 2.17.1

[dpdk-dev] [PATCH v2 v2] vhost: add unsafe API to check inflight packets

[Xuan Ding]

In async data path, when vring state changes, it is necessary to
know the number of inflight packets in DMA engine. This patch
provides a thread unsafe API to return the number of inflight
packets without using any lock.

Signed-off-by: Xuan Ding <xuan.ding@intel.com>
---

v2:
* Fixed some format issues.
---
 doc/guides/prog_guide/vhost_lib.rst    |  5 ++++
 doc/guides/rel_notes/release_21_11.rst |  5 ++++
 lib/vhost/rte_vhost_async.h            | 14 +++++++++
 lib/vhost/version.map                  |  3 ++
 lib/vhost/vhost.c                      | 41 ++++++++++++++++++++++----
 5 files changed, 63 insertions(+), 5 deletions(-)

diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
index 8874033165..0c4fb9ea91 100644
--- a/doc/guides/prog_guide/vhost_lib.rst
+++ b/doc/guides/prog_guide/vhost_lib.rst
@@ -305,6 +305,11 @@ The following is an overview of some key Vhost API functions:
   This function returns the amount of in-flight packets for the vhost
   queue using async acceleration.
 
+* ``rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id)``
+
+  Get the number of inflight packets for a vhost queue without
+  performing any locking.
+
 * ``rte_vhost_clear_queue_thread_unsafe(vid, queue_id, **pkts, count)``
 
   Clear inflight packets which are submitted to DMA engine in vhost async data
diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index 675b573834..c08e2b80c4 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added vhost API to get the number of inflight packets.**
+
+  Added an API which can get the number of inflight packets in
+  vhost async data path without lock.
+
 * **Enabled new devargs parser.**
 
   * Enabled devargs syntax
diff --git a/lib/vhost/rte_vhost_async.h b/lib/vhost/rte_vhost_async.h
index b25ff446f7..0af414bf78 100644
--- a/lib/vhost/rte_vhost_async.h
+++ b/lib/vhost/rte_vhost_async.h
@@ -246,6 +246,20 @@ uint16_t rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id,
 __rte_experimental
 int rte_vhost_async_get_inflight(int vid, uint16_t queue_id);
 
+/**
+ * This function is lock-free version to return the amount of in-flight
+ * packets for the vhost queue which uses async channel acceleration.
+ *
+ * @param vid
+ *  id of vhost device to enqueue data
+ * @param queue_id
+ *  queue id to enqueue data
+ * @return
+ *  the amount of in-flight packets on success; -1 on failure
+ */
+__rte_experimental
+int rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id);
+
 /**
  * This function checks async completion status and clear packets for
  * a specific vhost device queue. Packets which are inflight will be
diff --git a/lib/vhost/version.map b/lib/vhost/version.map
index c92a9d4962..b150dc408d 100644
--- a/lib/vhost/version.map
+++ b/lib/vhost/version.map
@@ -85,4 +85,7 @@ EXPERIMENTAL {
 	rte_vhost_async_channel_register_thread_unsafe;
 	rte_vhost_async_channel_unregister_thread_unsafe;
 	rte_vhost_clear_queue_thread_unsafe;
+
+	#added in 21.11
+	rte_vhost_async_get_inflight_thread_unsafe;
 };
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 355ff37651..69e9d229af 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -1500,7 +1500,8 @@ rte_vhost_get_vdpa_device(int vid)
 	return dev->vdpa_dev;
 }
 
-int rte_vhost_get_log_base(int vid, uint64_t *log_base,
+int
+rte_vhost_get_log_base(int vid, uint64_t *log_base,
 		uint64_t *log_size)
 {
 	struct virtio_net *dev = get_device(vid);
@@ -1514,7 +1515,8 @@ int rte_vhost_get_log_base(int vid, uint64_t *log_base,
 	return 0;
 }
 
-int rte_vhost_get_vring_base(int vid, uint16_t queue_id,
+int
+rte_vhost_get_vring_base(int vid, uint16_t queue_id,
 		uint16_t *last_avail_idx, uint16_t *last_used_idx)
 {
 	struct vhost_virtqueue *vq;
@@ -1543,7 +1545,8 @@ int rte_vhost_get_vring_base(int vid, uint16_t queue_id,
 	return 0;
 }
 
-int rte_vhost_set_vring_base(int vid, uint16_t queue_id,
+int
+rte_vhost_set_vring_base(int vid, uint16_t queue_id,
 		uint16_t last_avail_idx, uint16_t last_used_idx)
 {
 	struct vhost_virtqueue *vq;
@@ -1606,7 +1609,8 @@ rte_vhost_get_vring_base_from_inflight(int vid,
 	return 0;
 }
 
-int rte_vhost_extern_callback_register(int vid,
+int
+rte_vhost_extern_callback_register(int vid,
 		struct rte_vhost_user_extern_ops const * const ops, void *ctx)
 {
 	struct virtio_net *dev = get_device(vid);
@@ -1854,7 +1858,8 @@ rte_vhost_async_channel_unregister_thread_unsafe(int vid, uint16_t queue_id)
 	return 0;
 }
 
-int rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
+int
+rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
 {
 	struct vhost_virtqueue *vq;
 	struct virtio_net *dev = get_device(vid);
@@ -1886,5 +1891,31 @@ int rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
 	return ret;
 }
 
+int
+rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id)
+{
+	struct vhost_virtqueue *vq;
+	struct virtio_net *dev = get_device(vid);
+	int ret = -1;
+
+	if (dev == NULL)
+		return ret;
+
+	if (queue_id >= VHOST_MAX_VRING)
+		return ret;
+
+	vq = dev->virtqueue[queue_id];
+
+	if (vq == NULL)
+		return ret;
+
+	if (!vq->async_registered)
+		return ret;
+
+	ret = vq->async_pkts_inflight_n;
+
+	return ret;
+}
+
 RTE_LOG_REGISTER_SUFFIX(vhost_config_log_level, config, INFO);
 RTE_LOG_REGISTER_SUFFIX(vhost_data_log_level, data, WARNING);
-- 
2.17.1

[dpdk-dev] [PATCH v3] vhost: add unsafe API to check inflight packets

[Xuan Ding]

In async data path, when vring state changes, it is necessary to
know the number of inflight packets in DMA engine. This patch
provides a thread unsafe API to return the number of inflight
packets without using any lock.

Signed-off-by: Xuan Ding <xuan.ding@intel.com>
---
v3:
* Fixed one typo.
* Revised the doc to be more accuracy.

v2:
* Fixed some format issues.
---
 doc/guides/prog_guide/vhost_lib.rst    |  5 +++++
 doc/guides/rel_notes/release_21_11.rst |  5 +++++
 lib/vhost/rte_vhost_async.h            | 14 ++++++++++++++
 lib/vhost/version.map                  |  3 +++
 lib/vhost/vhost.c                      | 26 ++++++++++++++++++++++++++
 5 files changed, 53 insertions(+)

diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
index 8874033165..0c4fb9ea91 100644
--- a/doc/guides/prog_guide/vhost_lib.rst
+++ b/doc/guides/prog_guide/vhost_lib.rst
@@ -305,6 +305,11 @@ The following is an overview of some key Vhost API functions:
   This function returns the amount of in-flight packets for the vhost
   queue using async acceleration.
 
+* ``rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id)``
+
+  Get the number of inflight packets for a vhost queue without
+  performing any locking.
+
 * ``rte_vhost_clear_queue_thread_unsafe(vid, queue_id, **pkts, count)``
 
   Clear inflight packets which are submitted to DMA engine in vhost async data
diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index 675b573834..c9814c68df 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added vhost API to get the number of inflight packets.**
+
+  Added an API which can get the number of inflight packets in
+  vhost async data path without using lock.
+
 * **Enabled new devargs parser.**
 
   * Enabled devargs syntax
diff --git a/lib/vhost/rte_vhost_async.h b/lib/vhost/rte_vhost_async.h
index b25ff446f7..0af414bf78 100644
--- a/lib/vhost/rte_vhost_async.h
+++ b/lib/vhost/rte_vhost_async.h
@@ -246,6 +246,20 @@ uint16_t rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id,
 __rte_experimental
 int rte_vhost_async_get_inflight(int vid, uint16_t queue_id);
 
+/**
+ * This function is lock-free version to return the amount of in-flight
+ * packets for the vhost queue which uses async channel acceleration.
+ *
+ * @param vid
+ *  id of vhost device to enqueue data
+ * @param queue_id
+ *  queue id to enqueue data
+ * @return
+ *  the amount of in-flight packets on success; -1 on failure
+ */
+__rte_experimental
+int rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id);
+
 /**
  * This function checks async completion status and clear packets for
  * a specific vhost device queue. Packets which are inflight will be
diff --git a/lib/vhost/version.map b/lib/vhost/version.map
index c92a9d4962..b150dc408d 100644
--- a/lib/vhost/version.map
+++ b/lib/vhost/version.map
@@ -85,4 +85,7 @@ EXPERIMENTAL {
 	rte_vhost_async_channel_register_thread_unsafe;
 	rte_vhost_async_channel_unregister_thread_unsafe;
 	rte_vhost_clear_queue_thread_unsafe;
+
+	#added in 21.11
+	rte_vhost_async_get_inflight_thread_unsafe;
 };
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 355ff37651..24ae1025ea 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -1886,5 +1886,31 @@ int rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
 	return ret;
 }
 
+int
+rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id)
+{
+	struct vhost_virtqueue *vq;
+	struct virtio_net *dev = get_device(vid);
+	int ret = -1;
+
+	if (dev == NULL)
+		return ret;
+
+	if (queue_id >= VHOST_MAX_VRING)
+		return ret;
+
+	vq = dev->virtqueue[queue_id];
+
+	if (vq == NULL)
+		return ret;
+
+	if (!vq->async_registered)
+		return ret;
+
+	ret = vq->async_pkts_inflight_n;
+
+	return ret;
+}
+
 RTE_LOG_REGISTER_SUFFIX(vhost_config_log_level, config, INFO);
 RTE_LOG_REGISTER_SUFFIX(vhost_data_log_level, data, WARNING);
-- 
2.17.1

Re: [dpdk-dev] [PATCH v3] vhost: add unsafe API to check inflight packets

[Kevin Traynor]

On 16/09/2021 03:58, Xuan Ding wrote:
> In async data path, when vring state changes, it is necessary to
> know the number of inflight packets in DMA engine. This patch
> provides a thread unsafe API to return the number of inflight
> packets without using any lock.
> 
> Signed-off-by: Xuan Ding <xuan.ding@intel.com>
> ---
> v3:
> * Fixed one typo.
> * Revised the doc to be more accuracy.
> 
> v2:
> * Fixed some format issues.
> ---
>   doc/guides/prog_guide/vhost_lib.rst    |  5 +++++
>   doc/guides/rel_notes/release_21_11.rst |  5 +++++
>   lib/vhost/rte_vhost_async.h            | 14 ++++++++++++++
>   lib/vhost/version.map                  |  3 +++
>   lib/vhost/vhost.c                      | 26 ++++++++++++++++++++++++++
>   5 files changed, 53 insertions(+)
> 

<snip>

Should there be a change to the vring_state_changed() in vhost example 
app to go along with this patch? It would help to understand the 
operation if this API was used.

Re: [dpdk-dev] [PATCH v3] vhost: add unsafe API to check inflight packets

[Ding, Xuan]

Hi,

> -----Original Message-----
> From: Kevin Traynor <ktraynor@redhat.com>
> Sent: Wednesday, September 22, 2021 1:28 AM
> To: Ding, Xuan <xuan.ding@intel.com>; dev@dpdk.org;
> maxime.coquelin@redhat.com; Xia, Chenbo <chenbo.xia@intel.com>
> Cc: Hu, Jiayu <jiayu.hu@intel.com>; Jiang, Cheng1 <cheng1.jiang@intel.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; Pai G, Sunil
> <sunil.pai.g@intel.com>; Yang, YvonneX <yvonnex.yang@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v3] vhost: add unsafe API to check inflight
> packets
> 
> On 16/09/2021 03:58, Xuan Ding wrote:
> > In async data path, when vring state changes, it is necessary to
> > know the number of inflight packets in DMA engine. This patch
> > provides a thread unsafe API to return the number of inflight
> > packets without using any lock.
> >
> > Signed-off-by: Xuan Ding <xuan.ding@intel.com>
> > ---
> > v3:
> > * Fixed one typo.
> > * Revised the doc to be more accuracy.
> >
> > v2:
> > * Fixed some format issues.
> > ---
> >   doc/guides/prog_guide/vhost_lib.rst    |  5 +++++
> >   doc/guides/rel_notes/release_21_11.rst |  5 +++++
> >   lib/vhost/rte_vhost_async.h            | 14 ++++++++++++++
> >   lib/vhost/version.map                  |  3 +++
> >   lib/vhost/vhost.c                      | 26 ++++++++++++++++++++++++++
> >   5 files changed, 53 insertions(+)
> >
> 
> <snip>
> 
> Should there be a change to the vring_state_changed() in vhost example
> app to go along with this patch? It would help to understand the
> operation if this API was used.

Thanks Kevin, will add it in vhost example in next version.

Regards,
Xuan

[dpdk-dev] [PATCH v4 0/2] add unsafe API to get inflight packets

[Xuan Ding]

This patchset introduces an unsafe API to get the number of inflight
packets in DMA engine in some situations. Like vring state changes or
device is destroyed. Compared with rte_vhost_async_get_inflight(),
this is a lock free version.

v4:
* Added use case for API in vhost example.

v3:
* Fixed one typo.
* Revised the doc to be more accuracy.

v2:
* Fixed some format issues.

Xuan Ding (2):
  vhost: add unsafe API to check inflight packets
  examples/vhost: use API to check inflight packets

 doc/guides/prog_guide/vhost_lib.rst    |  5 +++++
 doc/guides/rel_notes/release_21_11.rst |  5 +++++
 examples/vhost/main.c                  | 26 +++++++++++---------------
 examples/vhost/main.h                  |  1 -
 lib/vhost/rte_vhost_async.h            | 14 ++++++++++++++
 lib/vhost/version.map                  |  3 +++
 lib/vhost/vhost.c                      | 26 ++++++++++++++++++++++++++
 7 files changed, 64 insertions(+), 16 deletions(-)

-- 
2.17.1

[dpdk-dev] [PATCH v4 1/2] vhost: add unsafe API to check inflight packets

[Xuan Ding]

In async data path, when vring state changes or device is destroyed,
it is necessary to know the number of inflight packets in DMA engine.
This patch provides a thread unsafe API to return the number of
inflight packets for a vhost queue without using any lock.

Signed-off-by: Xuan Ding <xuan.ding@intel.com>
---
 doc/guides/prog_guide/vhost_lib.rst    |  5 +++++
 doc/guides/rel_notes/release_21_11.rst |  5 +++++
 lib/vhost/rte_vhost_async.h            | 14 ++++++++++++++
 lib/vhost/version.map                  |  3 +++
 lib/vhost/vhost.c                      | 26 ++++++++++++++++++++++++++
 5 files changed, 53 insertions(+)

diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
index 8874033165..0c4fb9ea91 100644
--- a/doc/guides/prog_guide/vhost_lib.rst
+++ b/doc/guides/prog_guide/vhost_lib.rst
@@ -305,6 +305,11 @@ The following is an overview of some key Vhost API functions:
   This function returns the amount of in-flight packets for the vhost
   queue using async acceleration.
 
+* ``rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id)``
+
+  Get the number of inflight packets for a vhost queue without
+  performing any locking.
+
 * ``rte_vhost_clear_queue_thread_unsafe(vid, queue_id, **pkts, count)``
 
   Clear inflight packets which are submitted to DMA engine in vhost async data
diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index 675b573834..c9814c68df 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added vhost API to get the number of inflight packets.**
+
+  Added an API which can get the number of inflight packets in
+  vhost async data path without using lock.
+
 * **Enabled new devargs parser.**
 
   * Enabled devargs syntax
diff --git a/lib/vhost/rte_vhost_async.h b/lib/vhost/rte_vhost_async.h
index b25ff446f7..0af414bf78 100644
--- a/lib/vhost/rte_vhost_async.h
+++ b/lib/vhost/rte_vhost_async.h
@@ -246,6 +246,20 @@ uint16_t rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id,
 __rte_experimental
 int rte_vhost_async_get_inflight(int vid, uint16_t queue_id);
 
+/**
+ * This function is lock-free version to return the amount of in-flight
+ * packets for the vhost queue which uses async channel acceleration.
+ *
+ * @param vid
+ *  id of vhost device to enqueue data
+ * @param queue_id
+ *  queue id to enqueue data
+ * @return
+ *  the amount of in-flight packets on success; -1 on failure
+ */
+__rte_experimental
+int rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id);
+
 /**
  * This function checks async completion status and clear packets for
  * a specific vhost device queue. Packets which are inflight will be
diff --git a/lib/vhost/version.map b/lib/vhost/version.map
index c92a9d4962..b150dc408d 100644
--- a/lib/vhost/version.map
+++ b/lib/vhost/version.map
@@ -85,4 +85,7 @@ EXPERIMENTAL {
 	rte_vhost_async_channel_register_thread_unsafe;
 	rte_vhost_async_channel_unregister_thread_unsafe;
 	rte_vhost_clear_queue_thread_unsafe;
+
+	#added in 21.11
+	rte_vhost_async_get_inflight_thread_unsafe;
 };
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 355ff37651..24ae1025ea 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -1886,5 +1886,31 @@ int rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
 	return ret;
 }
 
+int
+rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id)
+{
+	struct vhost_virtqueue *vq;
+	struct virtio_net *dev = get_device(vid);
+	int ret = -1;
+
+	if (dev == NULL)
+		return ret;
+
+	if (queue_id >= VHOST_MAX_VRING)
+		return ret;
+
+	vq = dev->virtqueue[queue_id];
+
+	if (vq == NULL)
+		return ret;
+
+	if (!vq->async_registered)
+		return ret;
+
+	ret = vq->async_pkts_inflight_n;
+
+	return ret;
+}
+
 RTE_LOG_REGISTER_SUFFIX(vhost_config_log_level, config, INFO);
 RTE_LOG_REGISTER_SUFFIX(vhost_data_log_level, data, WARNING);
-- 
2.17.1

[dpdk-dev] [PATCH v4 2/2] examples/vhost: use API to check inflight packets

[Xuan Ding]

In async data path, call rte_vhost_async_get_inflight_thread_unsafe()
API to directly return the number of inflight packets instead of
maintaining a local variable.

Signed-off-by: Xuan Ding <xuan.ding@intel.com>
---
 examples/vhost/main.c | 26 +++++++++++---------------
 examples/vhost/main.h |  1 -
 2 files changed, 11 insertions(+), 16 deletions(-)

diff --git a/examples/vhost/main.c b/examples/vhost/main.c
index bc3d71c898..f0b74b5086 100644
--- a/examples/vhost/main.c
+++ b/examples/vhost/main.c
@@ -851,11 +851,8 @@ complete_async_pkts(struct vhost_dev *vdev)
 
 	complete_count = rte_vhost_poll_enqueue_completed(vdev->vid,
 					VIRTIO_RXQ, p_cpl, MAX_PKT_BURST);
-	if (complete_count) {
+	if (complete_count)
 		free_pkts(p_cpl, complete_count);
-		__atomic_sub_fetch(&vdev->pkts_inflight, complete_count, __ATOMIC_SEQ_CST);
-	}
-
 }
 
 static __rte_always_inline void
@@ -898,7 +895,6 @@ drain_vhost(struct vhost_dev *vdev)
 		complete_async_pkts(vdev);
 		ret = rte_vhost_submit_enqueue_burst(vdev->vid, VIRTIO_RXQ,
 					m, nr_xmit, m_cpu_cpl, &cpu_cpl_nr);
-		__atomic_add_fetch(&vdev->pkts_inflight, ret - cpu_cpl_nr, __ATOMIC_SEQ_CST);
 
 		if (cpu_cpl_nr)
 			free_pkts(m_cpu_cpl, cpu_cpl_nr);
@@ -1230,8 +1226,6 @@ drain_eth_rx(struct vhost_dev *vdev)
 		enqueue_count = rte_vhost_submit_enqueue_burst(vdev->vid,
 					VIRTIO_RXQ, pkts, rx_count,
 					m_cpu_cpl, &cpu_cpl_nr);
-		__atomic_add_fetch(&vdev->pkts_inflight, enqueue_count - cpu_cpl_nr,
-					__ATOMIC_SEQ_CST);
 
 		if (cpu_cpl_nr)
 			free_pkts(m_cpu_cpl, cpu_cpl_nr);
@@ -1360,6 +1354,7 @@ destroy_device(int vid)
 	struct vhost_dev *vdev = NULL;
 	int lcore;
 	uint16_t i;
+	int pkts_inflight;
 
 	TAILQ_FOREACH(vdev, &vhost_dev_list, global_vdev_entry) {
 		if (vdev->vid == vid)
@@ -1406,13 +1401,13 @@ destroy_device(int vid)
 
 	if (async_vhost_driver) {
 		uint16_t n_pkt = 0;
-		struct rte_mbuf *m_cpl[vdev->pkts_inflight];
+		pkts_inflight = rte_vhost_async_get_inflight_thread_unsafe(vid, VIRTIO_RXQ);
+		struct rte_mbuf *m_cpl[pkts_inflight];
 
-		while (vdev->pkts_inflight) {
+		while (pkts_inflight) {
 			n_pkt = rte_vhost_clear_queue_thread_unsafe(vid, VIRTIO_RXQ,
-						m_cpl, vdev->pkts_inflight);
+						m_cpl, pkts_inflight);
 			free_pkts(m_cpl, n_pkt);
-			__atomic_sub_fetch(&vdev->pkts_inflight, n_pkt, __ATOMIC_SEQ_CST);
 		}
 
 		rte_vhost_async_channel_unregister(vid, VIRTIO_RXQ);
@@ -1509,6 +1504,7 @@ static int
 vring_state_changed(int vid, uint16_t queue_id, int enable)
 {
 	struct vhost_dev *vdev = NULL;
+	int pkts_inflight;
 
 	TAILQ_FOREACH(vdev, &vhost_dev_list, global_vdev_entry) {
 		if (vdev->vid == vid)
@@ -1523,13 +1519,13 @@ vring_state_changed(int vid, uint16_t queue_id, int enable)
 	if (async_vhost_driver) {
 		if (!enable) {
 			uint16_t n_pkt = 0;
-			struct rte_mbuf *m_cpl[vdev->pkts_inflight];
+			pkts_inflight = rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id);
+			struct rte_mbuf *m_cpl[pkts_inflight];
 
-			while (vdev->pkts_inflight) {
+			while (pkts_inflight) {
 				n_pkt = rte_vhost_clear_queue_thread_unsafe(vid, queue_id,
-							m_cpl, vdev->pkts_inflight);
+							m_cpl, pkts_inflight);
 				free_pkts(m_cpl, n_pkt);
-				__atomic_sub_fetch(&vdev->pkts_inflight, n_pkt, __ATOMIC_SEQ_CST);
 			}
 		}
 	}
diff --git a/examples/vhost/main.h b/examples/vhost/main.h
index e7b1ac60a6..0ccdce4b4a 100644
--- a/examples/vhost/main.h
+++ b/examples/vhost/main.h
@@ -51,7 +51,6 @@ struct vhost_dev {
 	uint64_t features;
 	size_t hdr_len;
 	uint16_t nr_vrings;
-	uint16_t pkts_inflight;
 	struct rte_vhost_memory *mem;
 	struct device_statistics stats;
 	TAILQ_ENTRY(vhost_dev) global_vdev_entry;
-- 
2.17.1

[dpdk-dev] [PATCH v5 0/2] add unsafe API to get inflight packets

[Xuan Ding]

This patchset introduces an unsafe API to get the number of inflight
packets in DMA engine in some situations. Like vring state changes or
device is destroyed. Compared with rte_vhost_async_get_inflight(),
this is a lock free version.

v5:
* Rebased to the lastest branch.

v4:
* Added use case for API in vhost example.

v3:
* Fixed one typo.
* Revised the doc to be more accuracy.

v2:
* Fixed some format issues.

Xuan Ding (2):
  vhost: add unsafe API to check inflight packets
  examples/vhost: use API to check inflight packets

 doc/guides/prog_guide/vhost_lib.rst    |  5 +++++
 doc/guides/rel_notes/release_21_11.rst |  5 +++++
 examples/vhost/main.c                  | 25 +++++++++++--------------
 examples/vhost/main.h                  |  1 -
 lib/vhost/rte_vhost_async.h            | 14 ++++++++++++++
 lib/vhost/version.map                  |  3 +++
 lib/vhost/vhost.c                      | 26 ++++++++++++++++++++++++++
 7 files changed, 64 insertions(+), 15 deletions(-)

-- 
2.17.1

[dpdk-dev] [PATCH v5 1/2] vhost: add unsafe API to check inflight packets

[Xuan Ding]

In async data path, when vring state changes or device is destroyed,
it is necessary to know the number of inflight packets in DMA engine.
This patch provides a thread unsafe API to return the number of
inflight packets for a vhost queue without using any lock.

Signed-off-by: Xuan Ding <xuan.ding@intel.com>
---
 doc/guides/prog_guide/vhost_lib.rst    |  5 +++++
 doc/guides/rel_notes/release_21_11.rst |  5 +++++
 lib/vhost/rte_vhost_async.h            | 14 ++++++++++++++
 lib/vhost/version.map                  |  3 +++
 lib/vhost/vhost.c                      | 26 ++++++++++++++++++++++++++
 5 files changed, 53 insertions(+)

diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
index 171e0096f6..73a4e24640 100644
--- a/doc/guides/prog_guide/vhost_lib.rst
+++ b/doc/guides/prog_guide/vhost_lib.rst
@@ -298,6 +298,11 @@ The following is an overview of some key Vhost API functions:
   This function returns the amount of in-flight packets for the vhost
   queue using async acceleration.
 
+* ``rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id)``
+
+  Get the number of inflight packets for a vhost queue without
+  performing any locking.
+
 * ``rte_vhost_clear_queue_thread_unsafe(vid, queue_id, **pkts, count)``
 
   Clear inflight packets which are submitted to DMA engine in vhost async data
diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index ad7c1afec0..fd821b29d3 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added vhost API to get the number of inflight packets.**
+
+  Added an API which can get the number of inflight packets in
+  vhost async data path without using lock.
+
 * **Enabled new devargs parser.**
 
   * Enabled devargs syntax
diff --git a/lib/vhost/rte_vhost_async.h b/lib/vhost/rte_vhost_async.h
index ad71555a7f..1f0708378b 100644
--- a/lib/vhost/rte_vhost_async.h
+++ b/lib/vhost/rte_vhost_async.h
@@ -234,6 +234,20 @@ uint16_t rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id,
 __rte_experimental
 int rte_vhost_async_get_inflight(int vid, uint16_t queue_id);
 
+/**
+ * This function is lock-free version to return the amount of in-flight
+ * packets for the vhost queue which uses async channel acceleration.
+ *
+ * @param vid
+ *  id of vhost device to enqueue data
+ * @param queue_id
+ *  queue id to enqueue data
+ * @return
+ *  the amount of in-flight packets on success; -1 on failure
+ */
+__rte_experimental
+int rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id);
+
 /**
  * This function checks async completion status and clear packets for
  * a specific vhost device queue. Packets which are inflight will be
diff --git a/lib/vhost/version.map b/lib/vhost/version.map
index 8ebde3f694..ed6cb53a99 100644
--- a/lib/vhost/version.map
+++ b/lib/vhost/version.map
@@ -85,4 +85,7 @@ EXPERIMENTAL {
 	rte_vhost_async_channel_register_thread_unsafe;
 	rte_vhost_async_channel_unregister_thread_unsafe;
 	rte_vhost_clear_queue_thread_unsafe;
+
+	#added in 21.11
+	rte_vhost_async_get_inflight_thread_unsafe;
 };
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 996287c3c2..c16c6bc2bb 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -1884,5 +1884,31 @@ int rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
 	return ret;
 }
 
+int
+rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id)
+{
+	struct vhost_virtqueue *vq;
+	struct virtio_net *dev = get_device(vid);
+	int ret = -1;
+
+	if (dev == NULL)
+		return ret;
+
+	if (queue_id >= VHOST_MAX_VRING)
+		return ret;
+
+	vq = dev->virtqueue[queue_id];
+
+	if (vq == NULL)
+		return ret;
+
+	if (!vq->async_registered)
+		return ret;
+
+	ret = vq->async_pkts_inflight_n;
+
+	return ret;
+}
+
 RTE_LOG_REGISTER_SUFFIX(vhost_config_log_level, config, INFO);
 RTE_LOG_REGISTER_SUFFIX(vhost_data_log_level, data, WARNING);
-- 
2.17.1

[dpdk-dev] [PATCH v5 2/2] examples/vhost: use API to check inflight packets

[Xuan Ding]

In async data path, call rte_vhost_async_get_inflight_thread_unsafe()
API to directly return the number of inflight packets instead of
maintaining a local variable.

Signed-off-by: Xuan Ding <xuan.ding@intel.com>
---
 examples/vhost/main.c | 25 +++++++++++--------------
 examples/vhost/main.h |  1 -
 2 files changed, 11 insertions(+), 15 deletions(-)

diff --git a/examples/vhost/main.c b/examples/vhost/main.c
index d0bf1f31e3..3faac6d053 100644
--- a/examples/vhost/main.c
+++ b/examples/vhost/main.c
@@ -842,11 +842,8 @@ complete_async_pkts(struct vhost_dev *vdev)
 
 	complete_count = rte_vhost_poll_enqueue_completed(vdev->vid,
 					VIRTIO_RXQ, p_cpl, MAX_PKT_BURST);
-	if (complete_count) {
+	if (complete_count)
 		free_pkts(p_cpl, complete_count);
-		__atomic_sub_fetch(&vdev->pkts_inflight, complete_count, __ATOMIC_SEQ_CST);
-	}
-
 }
 
 static __rte_always_inline void
@@ -886,7 +883,6 @@ drain_vhost(struct vhost_dev *vdev)
 
 		complete_async_pkts(vdev);
 		ret = rte_vhost_submit_enqueue_burst(vdev->vid, VIRTIO_RXQ, m, nr_xmit);
-		__atomic_add_fetch(&vdev->pkts_inflight, ret, __ATOMIC_SEQ_CST);
 
 		enqueue_fail = nr_xmit - ret;
 		if (enqueue_fail)
@@ -1212,7 +1208,6 @@ drain_eth_rx(struct vhost_dev *vdev)
 		complete_async_pkts(vdev);
 		enqueue_count = rte_vhost_submit_enqueue_burst(vdev->vid,
 					VIRTIO_RXQ, pkts, rx_count);
-		__atomic_add_fetch(&vdev->pkts_inflight, enqueue_count, __ATOMIC_SEQ_CST);
 
 		enqueue_fail = rx_count - enqueue_count;
 		if (enqueue_fail)
@@ -1338,6 +1333,7 @@ destroy_device(int vid)
 	struct vhost_dev *vdev = NULL;
 	int lcore;
 	uint16_t i;
+	int pkts_inflight;
 
 	TAILQ_FOREACH(vdev, &vhost_dev_list, global_vdev_entry) {
 		if (vdev->vid == vid)
@@ -1384,13 +1380,13 @@ destroy_device(int vid)
 
 	if (async_vhost_driver) {
 		uint16_t n_pkt = 0;
-		struct rte_mbuf *m_cpl[vdev->pkts_inflight];
+		pkts_inflight = rte_vhost_async_get_inflight_thread_unsafe(vid, VIRTIO_RXQ);
+		struct rte_mbuf *m_cpl[pkts_inflight];
 
-		while (vdev->pkts_inflight) {
+		while (pkts_inflight) {
 			n_pkt = rte_vhost_clear_queue_thread_unsafe(vid, VIRTIO_RXQ,
-						m_cpl, vdev->pkts_inflight);
+						m_cpl, pkts_inflight);
 			free_pkts(m_cpl, n_pkt);
-			__atomic_sub_fetch(&vdev->pkts_inflight, n_pkt, __ATOMIC_SEQ_CST);
 		}
 
 		rte_vhost_async_channel_unregister(vid, VIRTIO_RXQ);
@@ -1486,6 +1482,7 @@ static int
 vring_state_changed(int vid, uint16_t queue_id, int enable)
 {
 	struct vhost_dev *vdev = NULL;
+	int pkts_inflight;
 
 	TAILQ_FOREACH(vdev, &vhost_dev_list, global_vdev_entry) {
 		if (vdev->vid == vid)
@@ -1500,13 +1497,13 @@ vring_state_changed(int vid, uint16_t queue_id, int enable)
 	if (async_vhost_driver) {
 		if (!enable) {
 			uint16_t n_pkt = 0;
-			struct rte_mbuf *m_cpl[vdev->pkts_inflight];
+			pkts_inflight = rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id);
+			struct rte_mbuf *m_cpl[pkts_inflight];
 
-			while (vdev->pkts_inflight) {
+			while (pkts_inflight) {
 				n_pkt = rte_vhost_clear_queue_thread_unsafe(vid, queue_id,
-							m_cpl, vdev->pkts_inflight);
+							m_cpl, pkts_inflight);
 				free_pkts(m_cpl, n_pkt);
-				__atomic_sub_fetch(&vdev->pkts_inflight, n_pkt, __ATOMIC_SEQ_CST);
 			}
 		}
 	}
diff --git a/examples/vhost/main.h b/examples/vhost/main.h
index e7b1ac60a6..0ccdce4b4a 100644
--- a/examples/vhost/main.h
+++ b/examples/vhost/main.h
@@ -51,7 +51,6 @@ struct vhost_dev {
 	uint64_t features;
 	size_t hdr_len;
 	uint16_t nr_vrings;
-	uint16_t pkts_inflight;
 	struct rte_vhost_memory *mem;
 	struct device_statistics stats;
 	TAILQ_ENTRY(vhost_dev) global_vdev_entry;
-- 
2.17.1

Re: [dpdk-dev] [PATCH v5 2/2] examples/vhost: use API to check inflight packets

[Kevin Traynor]

On 28/09/2021 07:24, Xuan Ding wrote:
> In async data path, call rte_vhost_async_get_inflight_thread_unsafe()
> API to directly return the number of inflight packets instead of
> maintaining a local variable.
> 
> Signed-off-by: Xuan Ding <xuan.ding@intel.com>
> ---
>   examples/vhost/main.c | 25 +++++++++++--------------
>   examples/vhost/main.h |  1 -
>   2 files changed, 11 insertions(+), 15 deletions(-)
> 
> diff --git a/examples/vhost/main.c b/examples/vhost/main.c
> index d0bf1f31e3..3faac6d053 100644
> --- a/examples/vhost/main.c
> +++ b/examples/vhost/main.c
> @@ -842,11 +842,8 @@ complete_async_pkts(struct vhost_dev *vdev)
>   
>   	complete_count = rte_vhost_poll_enqueue_completed(vdev->vid,
>   					VIRTIO_RXQ, p_cpl, MAX_PKT_BURST);
> -	if (complete_count) {
> +	if (complete_count)
>   		free_pkts(p_cpl, complete_count);
> -		__atomic_sub_fetch(&vdev->pkts_inflight, complete_count, __ATOMIC_SEQ_CST);
> -	}
> -
>   }
>   
>   static __rte_always_inline void
> @@ -886,7 +883,6 @@ drain_vhost(struct vhost_dev *vdev)
>   
>   		complete_async_pkts(vdev);
>   		ret = rte_vhost_submit_enqueue_burst(vdev->vid, VIRTIO_RXQ, m, nr_xmit);
> -		__atomic_add_fetch(&vdev->pkts_inflight, ret, __ATOMIC_SEQ_CST);
>   
>   		enqueue_fail = nr_xmit - ret;
>   		if (enqueue_fail)
> @@ -1212,7 +1208,6 @@ drain_eth_rx(struct vhost_dev *vdev)
>   		complete_async_pkts(vdev);
>   		enqueue_count = rte_vhost_submit_enqueue_burst(vdev->vid,
>   					VIRTIO_RXQ, pkts, rx_count);
> -		__atomic_add_fetch(&vdev->pkts_inflight, enqueue_count, __ATOMIC_SEQ_CST);
>   
>   		enqueue_fail = rx_count - enqueue_count;
>   		if (enqueue_fail)
> @@ -1338,6 +1333,7 @@ destroy_device(int vid)
>   	struct vhost_dev *vdev = NULL;
>   	int lcore;
>   	uint16_t i;

> +	int pkts_inflight;

You can move this down to the block it is used in

>   
>   	TAILQ_FOREACH(vdev, &vhost_dev_list, global_vdev_entry) {
>   		if (vdev->vid == vid)
> @@ -1384,13 +1380,13 @@ destroy_device(int vid)
>   
>   	if (async_vhost_driver) {
>   		uint16_t n_pkt = 0;
> -		struct rte_mbuf *m_cpl[vdev->pkts_inflight];
> +		pkts_inflight = rte_vhost_async_get_inflight_thread_unsafe(vid, VIRTIO_RXQ);
> +		struct rte_mbuf *m_cpl[pkts_inflight];
>   
> -		while (vdev->pkts_inflight) {
> +		while (pkts_inflight) {
>   			n_pkt = rte_vhost_clear_queue_thread_unsafe(vid, VIRTIO_RXQ,
> -						m_cpl, vdev->pkts_inflight);
> +						m_cpl, pkts_inflight);
>   			free_pkts(m_cpl, n_pkt);
> -			__atomic_sub_fetch(&vdev->pkts_inflight, n_pkt, __ATOMIC_SEQ_CST);

This is an infinite loop if there are pkts_inflight, need to recheck 
pkts_inflight in the loop.

>   		}
>   
>   		rte_vhost_async_channel_unregister(vid, VIRTIO_RXQ);
> @@ -1486,6 +1482,7 @@ static int
>   vring_state_changed(int vid, uint16_t queue_id, int enable)
>   {
>   	struct vhost_dev *vdev = NULL;
> +	int pkts_inflight;
>   
>   	TAILQ_FOREACH(vdev, &vhost_dev_list, global_vdev_entry) {
>   		if (vdev->vid == vid)
> @@ -1500,13 +1497,13 @@ vring_state_changed(int vid, uint16_t queue_id, int enable)
>   	if (async_vhost_driver) {
>   		if (!enable) {
>   			uint16_t n_pkt = 0;
> -			struct rte_mbuf *m_cpl[vdev->pkts_inflight];
> +			pkts_inflight = rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id);
> +			struct rte_mbuf *m_cpl[pkts_inflight];
>   
> -			while (vdev->pkts_inflight) {
> +			while (pkts_inflight) {
>   				n_pkt = rte_vhost_clear_queue_thread_unsafe(vid, queue_id,
> -							m_cpl, vdev->pkts_inflight);
> +							m_cpl, pkts_inflight);
>   				free_pkts(m_cpl, n_pkt);
> -				__atomic_sub_fetch(&vdev->pkts_inflight, n_pkt, __ATOMIC_SEQ_CST);

Same comments as destroy_device

>   			}
>   		}
>   	}
> diff --git a/examples/vhost/main.h b/examples/vhost/main.h
> index e7b1ac60a6..0ccdce4b4a 100644
> --- a/examples/vhost/main.h
> +++ b/examples/vhost/main.h
> @@ -51,7 +51,6 @@ struct vhost_dev {
>   	uint64_t features;
>   	size_t hdr_len;
>   	uint16_t nr_vrings;
> -	uint16_t pkts_inflight;
>   	struct rte_vhost_memory *mem;
>   	struct device_statistics stats;
>   	TAILQ_ENTRY(vhost_dev) global_vdev_entry;
> 

Re: [dpdk-dev] [PATCH v5 2/2] examples/vhost: use API to check inflight packets

[Ding, Xuan]

Hi Kevin,

> -----Original Message-----
> From: Kevin Traynor <ktraynor@redhat.com>
> Sent: Tuesday, September 28, 2021 5:18 PM
> To: Ding, Xuan <xuan.ding@intel.com>; dev@dpdk.org;
> maxime.coquelin@redhat.com; Xia, Chenbo <chenbo.xia@intel.com>
> Cc: Hu, Jiayu <jiayu.hu@intel.com>; Jiang, Cheng1 <cheng1.jiang@intel.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; Pai G, Sunil
> <sunil.pai.g@intel.com>; Wang, Yinan <yinan.wang@intel.com>; Yang, YvonneX
> <yvonnex.yang@intel.com>
> Subject: Re: [PATCH v5 2/2] examples/vhost: use API to check inflight packets
> 
> On 28/09/2021 07:24, Xuan Ding wrote:
> > In async data path, call rte_vhost_async_get_inflight_thread_unsafe()
> > API to directly return the number of inflight packets instead of
> > maintaining a local variable.
> >
> > Signed-off-by: Xuan Ding <xuan.ding@intel.com>
> > ---
> >   examples/vhost/main.c | 25 +++++++++++--------------
> >   examples/vhost/main.h |  1 -
> >   2 files changed, 11 insertions(+), 15 deletions(-)
> >
> > diff --git a/examples/vhost/main.c b/examples/vhost/main.c
> > index d0bf1f31e3..3faac6d053 100644
> > --- a/examples/vhost/main.c
> > +++ b/examples/vhost/main.c
> > @@ -842,11 +842,8 @@ complete_async_pkts(struct vhost_dev *vdev)
> >
> >   	complete_count = rte_vhost_poll_enqueue_completed(vdev->vid,
> >   					VIRTIO_RXQ, p_cpl, MAX_PKT_BURST);
> > -	if (complete_count) {
> > +	if (complete_count)
> >   		free_pkts(p_cpl, complete_count);
> > -		__atomic_sub_fetch(&vdev->pkts_inflight, complete_count,
> __ATOMIC_SEQ_CST);
> > -	}
> > -
> >   }
> >
> >   static __rte_always_inline void
> > @@ -886,7 +883,6 @@ drain_vhost(struct vhost_dev *vdev)
> >
> >   		complete_async_pkts(vdev);
> >   		ret = rte_vhost_submit_enqueue_burst(vdev->vid, VIRTIO_RXQ,
> m, nr_xmit);
> > -		__atomic_add_fetch(&vdev->pkts_inflight, ret,
> __ATOMIC_SEQ_CST);
> >
> >   		enqueue_fail = nr_xmit - ret;
> >   		if (enqueue_fail)
> > @@ -1212,7 +1208,6 @@ drain_eth_rx(struct vhost_dev *vdev)
> >   		complete_async_pkts(vdev);
> >   		enqueue_count = rte_vhost_submit_enqueue_burst(vdev->vid,
> >   					VIRTIO_RXQ, pkts, rx_count);
> > -		__atomic_add_fetch(&vdev->pkts_inflight, enqueue_count,
> __ATOMIC_SEQ_CST);
> >
> >   		enqueue_fail = rx_count - enqueue_count;
> >   		if (enqueue_fail)
> > @@ -1338,6 +1333,7 @@ destroy_device(int vid)
> >   	struct vhost_dev *vdev = NULL;
> >   	int lcore;
> >   	uint16_t i;
> 
> > +	int pkts_inflight;
> 
> You can move this down to the block it is used in

Thanks for the suggestion.
I consider calling the unsafe API in while (condition), and there is no need to define this variable.

> 
> >
> >   	TAILQ_FOREACH(vdev, &vhost_dev_list, global_vdev_entry) {
> >   		if (vdev->vid == vid)
> > @@ -1384,13 +1380,13 @@ destroy_device(int vid)
> >
> >   	if (async_vhost_driver) {
> >   		uint16_t n_pkt = 0;
> > -		struct rte_mbuf *m_cpl[vdev->pkts_inflight];
> > +		pkts_inflight = rte_vhost_async_get_inflight_thread_unsafe(vid,
> VIRTIO_RXQ);
> > +		struct rte_mbuf *m_cpl[pkts_inflight];
> >
> > -		while (vdev->pkts_inflight) {
> > +		while (pkts_inflight) {
> >   			n_pkt = rte_vhost_clear_queue_thread_unsafe(vid,
> VIRTIO_RXQ,
> > -						m_cpl, vdev->pkts_inflight);
> > +						m_cpl, pkts_inflight);
> >   			free_pkts(m_cpl, n_pkt);
> > -			__atomic_sub_fetch(&vdev->pkts_inflight, n_pkt,
> __ATOMIC_SEQ_CST);
> 
> This is an infinite loop if there are pkts_inflight, need to recheck
> pkts_inflight in the loop.

Thanks for the catch, will call the unsafe API directly in the while (condition).

> 
> >   		}
> >
> >   		rte_vhost_async_channel_unregister(vid, VIRTIO_RXQ);
> > @@ -1486,6 +1482,7 @@ static int
> >   vring_state_changed(int vid, uint16_t queue_id, int enable)
> >   {
> >   	struct vhost_dev *vdev = NULL;
> > +	int pkts_inflight;
> >
> >   	TAILQ_FOREACH(vdev, &vhost_dev_list, global_vdev_entry) {
> >   		if (vdev->vid == vid)
> > @@ -1500,13 +1497,13 @@ vring_state_changed(int vid, uint16_t queue_id,
> int enable)
> >   	if (async_vhost_driver) {
> >   		if (!enable) {
> >   			uint16_t n_pkt = 0;
> > -			struct rte_mbuf *m_cpl[vdev->pkts_inflight];
> > +			pkts_inflight =
> rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id);
> > +			struct rte_mbuf *m_cpl[pkts_inflight];
> >
> > -			while (vdev->pkts_inflight) {
> > +			while (pkts_inflight) {
> >   				n_pkt =
> rte_vhost_clear_queue_thread_unsafe(vid, queue_id,
> > -							m_cpl, vdev-
> >pkts_inflight);
> > +							m_cpl, pkts_inflight);
> >   				free_pkts(m_cpl, n_pkt);
> > -				__atomic_sub_fetch(&vdev->pkts_inflight,
> n_pkt, __ATOMIC_SEQ_CST);
> 
> Same comments as destroy_device

Same as vring_state_changed.

Regards,
Xuan

> 
> >   			}
> >   		}
> >   	}
> > diff --git a/examples/vhost/main.h b/examples/vhost/main.h
> > index e7b1ac60a6..0ccdce4b4a 100644
> > --- a/examples/vhost/main.h
> > +++ b/examples/vhost/main.h
> > @@ -51,7 +51,6 @@ struct vhost_dev {
> >   	uint64_t features;
> >   	size_t hdr_len;
> >   	uint16_t nr_vrings;
> > -	uint16_t pkts_inflight;
> >   	struct rte_vhost_memory *mem;
> >   	struct device_statistics stats;
> >   	TAILQ_ENTRY(vhost_dev) global_vdev_entry;
> >

Re: [dpdk-dev] [PATCH v5 2/2] examples/vhost: use API to check inflight packets

[Ding, Xuan]

Hi,

> -----Original Message-----
> From: Ding, Xuan
> Sent: Tuesday, September 28, 2021 7:51 PM
> To: Kevin Traynor <ktraynor@redhat.com>; dev@dpdk.org;
> maxime.coquelin@redhat.com; Xia, Chenbo <Chenbo.Xia@intel.com>
> Cc: Hu, Jiayu <Jiayu.Hu@intel.com>; Jiang, Cheng1 <Cheng1.Jiang@intel.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; Pai G, Sunil
> <Sunil.Pai.G@intel.com>; Wang, Yinan <yinan.wang@intel.com>; Yang,
> YvonneX <YvonneX.Yang@intel.com>
> Subject: RE: [PATCH v5 2/2] examples/vhost: use API to check inflight packets
> 
> Hi Kevin,
> 
> > -----Original Message-----
> > From: Kevin Traynor <ktraynor@redhat.com>
> > Sent: Tuesday, September 28, 2021 5:18 PM
> > To: Ding, Xuan <xuan.ding@intel.com>; dev@dpdk.org;
> > maxime.coquelin@redhat.com; Xia, Chenbo <chenbo.xia@intel.com>
> > Cc: Hu, Jiayu <jiayu.hu@intel.com>; Jiang, Cheng1 <cheng1.jiang@intel.com>;
> > Richardson, Bruce <bruce.richardson@intel.com>; Pai G, Sunil
> > <sunil.pai.g@intel.com>; Wang, Yinan <yinan.wang@intel.com>; Yang,
> YvonneX
> > <yvonnex.yang@intel.com>
> > Subject: Re: [PATCH v5 2/2] examples/vhost: use API to check inflight packets
> >
> > On 28/09/2021 07:24, Xuan Ding wrote:
> > > In async data path, call rte_vhost_async_get_inflight_thread_unsafe()
> > > API to directly return the number of inflight packets instead of
> > > maintaining a local variable.
> > >
> > > Signed-off-by: Xuan Ding <xuan.ding@intel.com>
> > > ---
> > >   examples/vhost/main.c | 25 +++++++++++--------------
> > >   examples/vhost/main.h |  1 -
> > >   2 files changed, 11 insertions(+), 15 deletions(-)
> > >
> > > diff --git a/examples/vhost/main.c b/examples/vhost/main.c
> > > index d0bf1f31e3..3faac6d053 100644
> > > --- a/examples/vhost/main.c
> > > +++ b/examples/vhost/main.c
> > > @@ -842,11 +842,8 @@ complete_async_pkts(struct vhost_dev *vdev)
> > >
> > >   	complete_count = rte_vhost_poll_enqueue_completed(vdev->vid,
> > >   					VIRTIO_RXQ, p_cpl, MAX_PKT_BURST);
> > > -	if (complete_count) {
> > > +	if (complete_count)
> > >   		free_pkts(p_cpl, complete_count);
> > > -		__atomic_sub_fetch(&vdev->pkts_inflight, complete_count,
> > __ATOMIC_SEQ_CST);
> > > -	}
> > > -
> > >   }
> > >
> > >   static __rte_always_inline void
> > > @@ -886,7 +883,6 @@ drain_vhost(struct vhost_dev *vdev)
> > >
> > >   		complete_async_pkts(vdev);
> > >   		ret = rte_vhost_submit_enqueue_burst(vdev->vid, VIRTIO_RXQ,
> > m, nr_xmit);
> > > -		__atomic_add_fetch(&vdev->pkts_inflight, ret,
> > __ATOMIC_SEQ_CST);
> > >
> > >   		enqueue_fail = nr_xmit - ret;
> > >   		if (enqueue_fail)
> > > @@ -1212,7 +1208,6 @@ drain_eth_rx(struct vhost_dev *vdev)
> > >   		complete_async_pkts(vdev);
> > >   		enqueue_count = rte_vhost_submit_enqueue_burst(vdev->vid,
> > >   					VIRTIO_RXQ, pkts, rx_count);
> > > -		__atomic_add_fetch(&vdev->pkts_inflight, enqueue_count,
> > __ATOMIC_SEQ_CST);
> > >
> > >   		enqueue_fail = rx_count - enqueue_count;
> > >   		if (enqueue_fail)
> > > @@ -1338,6 +1333,7 @@ destroy_device(int vid)
> > >   	struct vhost_dev *vdev = NULL;
> > >   	int lcore;
> > >   	uint16_t i;
> >
> > > +	int pkts_inflight;
> >
> > You can move this down to the block it is used in
> 
> Thanks for the suggestion.
> I consider calling the unsafe API in while (condition), and there is no need to
> define this variable.
> 
> >
> > >
> > >   	TAILQ_FOREACH(vdev, &vhost_dev_list, global_vdev_entry) {
> > >   		if (vdev->vid == vid)
> > > @@ -1384,13 +1380,13 @@ destroy_device(int vid)
> > >
> > >   	if (async_vhost_driver) {
> > >   		uint16_t n_pkt = 0;
> > > -		struct rte_mbuf *m_cpl[vdev->pkts_inflight];
> > > +		pkts_inflight = rte_vhost_async_get_inflight_thread_unsafe(vid,
> > VIRTIO_RXQ);
> > > +		struct rte_mbuf *m_cpl[pkts_inflight];
> > >
> > > -		while (vdev->pkts_inflight) {
> > > +		while (pkts_inflight) {
> > >   			n_pkt = rte_vhost_clear_queue_thread_unsafe(vid,
> > VIRTIO_RXQ,
> > > -						m_cpl, vdev->pkts_inflight);
> > > +						m_cpl, pkts_inflight);
> > >   			free_pkts(m_cpl, n_pkt);
> > > -			__atomic_sub_fetch(&vdev->pkts_inflight, n_pkt,
> > __ATOMIC_SEQ_CST);
> >
> > This is an infinite loop if there are pkts_inflight, need to recheck
> > pkts_inflight in the loop.
> 
> Thanks for the catch, will call the unsafe API directly in the while (condition).

Sorry for replying myself, as rte_mbuf *m_cpl also needs pkts_inflight here.
Will follow your suggestion, see next version, thanks!

Regards,
Xuan

> 
> >
> > >   		}
> > >
> > >   		rte_vhost_async_channel_unregister(vid, VIRTIO_RXQ);
> > > @@ -1486,6 +1482,7 @@ static int
> > >   vring_state_changed(int vid, uint16_t queue_id, int enable)
> > >   {
> > >   	struct vhost_dev *vdev = NULL;
> > > +	int pkts_inflight;
> > >
> > >   	TAILQ_FOREACH(vdev, &vhost_dev_list, global_vdev_entry) {
> > >   		if (vdev->vid == vid)
> > > @@ -1500,13 +1497,13 @@ vring_state_changed(int vid, uint16_t queue_id,
> > int enable)
> > >   	if (async_vhost_driver) {
> > >   		if (!enable) {
> > >   			uint16_t n_pkt = 0;
> > > -			struct rte_mbuf *m_cpl[vdev->pkts_inflight];
> > > +			pkts_inflight =
> > rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id);
> > > +			struct rte_mbuf *m_cpl[pkts_inflight];
> > >
> > > -			while (vdev->pkts_inflight) {
> > > +			while (pkts_inflight) {
> > >   				n_pkt =
> > rte_vhost_clear_queue_thread_unsafe(vid, queue_id,
> > > -							m_cpl, vdev-
> > >pkts_inflight);
> > > +							m_cpl, pkts_inflight);
> > >   				free_pkts(m_cpl, n_pkt);
> > > -				__atomic_sub_fetch(&vdev->pkts_inflight,
> > n_pkt, __ATOMIC_SEQ_CST);
> >
> > Same comments as destroy_device
> 
> Same as vring_state_changed.
> 
> Regards,
> Xuan
> 
> >
> > >   			}
> > >   		}
> > >   	}
> > > diff --git a/examples/vhost/main.h b/examples/vhost/main.h
> > > index e7b1ac60a6..0ccdce4b4a 100644
> > > --- a/examples/vhost/main.h
> > > +++ b/examples/vhost/main.h
> > > @@ -51,7 +51,6 @@ struct vhost_dev {
> > >   	uint64_t features;
> > >   	size_t hdr_len;
> > >   	uint16_t nr_vrings;
> > > -	uint16_t pkts_inflight;
> > >   	struct rte_vhost_memory *mem;
> > >   	struct device_statistics stats;
> > >   	TAILQ_ENTRY(vhost_dev) global_vdev_entry;
> > >

[dpdk-dev] [PATCH v6 0/2] add unsafe API to get inflight packets

[Xuan Ding]

This patchset introduces an unsafe API to get the number of inflight
packets in DMA engine in some situations. Like vring state changes or
device is destroyed. Compared with rte_vhost_async_get_inflight(),
this is a lock free version.

v6:
* Fixed an issue of infinite loop.

v5:
* Rebased to the lastest branch.

v4:
* Added use case for API in vhost example.

v3:
* Fixed one typo.
* Revised the doc to be more accuracy.

v2:
* Fixed some format issues.

Xuan Ding (2):
  vhost: add unsafe API to check inflight packets
  examples/vhost: use API to check inflight packets

 doc/guides/prog_guide/vhost_lib.rst    |  5 +++++
 doc/guides/rel_notes/release_21_11.rst |  5 +++++
 examples/vhost/main.c                  | 31 ++++++++++++++------------
 examples/vhost/main.h                  |  1 -
 lib/vhost/rte_vhost_async.h            | 14 ++++++++++++
 lib/vhost/version.map                  |  3 +++
 lib/vhost/vhost.c                      | 26 +++++++++++++++++++++
 7 files changed, 70 insertions(+), 15 deletions(-)

-- 
2.17.1

[dpdk-dev] [PATCH v6 1/2] vhost: add unsafe API to check inflight packets

[Xuan Ding]

In async data path, when vring state changes or device is destroyed,
it is necessary to know the number of inflight packets in DMA engine.
This patch provides a thread unsafe API to return the number of
inflight packets for a vhost queue without using any lock.

Signed-off-by: Xuan Ding <xuan.ding@intel.com>
---
 doc/guides/prog_guide/vhost_lib.rst    |  5 +++++
 doc/guides/rel_notes/release_21_11.rst |  5 +++++
 lib/vhost/rte_vhost_async.h            | 14 ++++++++++++++
 lib/vhost/version.map                  |  3 +++
 lib/vhost/vhost.c                      | 26 ++++++++++++++++++++++++++
 5 files changed, 53 insertions(+)

diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
index 171e0096f6..73a4e24640 100644
--- a/doc/guides/prog_guide/vhost_lib.rst
+++ b/doc/guides/prog_guide/vhost_lib.rst
@@ -298,6 +298,11 @@ The following is an overview of some key Vhost API functions:
   This function returns the amount of in-flight packets for the vhost
   queue using async acceleration.
 
+* ``rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id)``
+
+  Get the number of inflight packets for a vhost queue without
+  performing any locking.
+
 * ``rte_vhost_clear_queue_thread_unsafe(vid, queue_id, **pkts, count)``
 
   Clear inflight packets which are submitted to DMA engine in vhost async data
diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index ad7c1afec0..fd821b29d3 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added vhost API to get the number of inflight packets.**
+
+  Added an API which can get the number of inflight packets in
+  vhost async data path without using lock.
+
 * **Enabled new devargs parser.**
 
   * Enabled devargs syntax
diff --git a/lib/vhost/rte_vhost_async.h b/lib/vhost/rte_vhost_async.h
index ad71555a7f..1f0708378b 100644
--- a/lib/vhost/rte_vhost_async.h
+++ b/lib/vhost/rte_vhost_async.h
@@ -234,6 +234,20 @@ uint16_t rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id,
 __rte_experimental
 int rte_vhost_async_get_inflight(int vid, uint16_t queue_id);
 
+/**
+ * This function is lock-free version to return the amount of in-flight
+ * packets for the vhost queue which uses async channel acceleration.
+ *
+ * @param vid
+ *  id of vhost device to enqueue data
+ * @param queue_id
+ *  queue id to enqueue data
+ * @return
+ *  the amount of in-flight packets on success; -1 on failure
+ */
+__rte_experimental
+int rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id);
+
 /**
  * This function checks async completion status and clear packets for
  * a specific vhost device queue. Packets which are inflight will be
diff --git a/lib/vhost/version.map b/lib/vhost/version.map
index 8ebde3f694..ed6cb53a99 100644
--- a/lib/vhost/version.map
+++ b/lib/vhost/version.map
@@ -85,4 +85,7 @@ EXPERIMENTAL {
 	rte_vhost_async_channel_register_thread_unsafe;
 	rte_vhost_async_channel_unregister_thread_unsafe;
 	rte_vhost_clear_queue_thread_unsafe;
+
+	#added in 21.11
+	rte_vhost_async_get_inflight_thread_unsafe;
 };
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 996287c3c2..c16c6bc2bb 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -1884,5 +1884,31 @@ int rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
 	return ret;
 }
 
+int
+rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id)
+{
+	struct vhost_virtqueue *vq;
+	struct virtio_net *dev = get_device(vid);
+	int ret = -1;
+
+	if (dev == NULL)
+		return ret;
+
+	if (queue_id >= VHOST_MAX_VRING)
+		return ret;
+
+	vq = dev->virtqueue[queue_id];
+
+	if (vq == NULL)
+		return ret;
+
+	if (!vq->async_registered)
+		return ret;
+
+	ret = vq->async_pkts_inflight_n;
+
+	return ret;
+}
+
 RTE_LOG_REGISTER_SUFFIX(vhost_config_log_level, config, INFO);
 RTE_LOG_REGISTER_SUFFIX(vhost_data_log_level, data, WARNING);
-- 
2.17.1

[dpdk-dev] [PATCH v6 2/2] examples/vhost: use API to check inflight packets

[Xuan Ding]

In async data path, call rte_vhost_async_get_inflight_thread_unsafe()
API to directly return the number of inflight packets instead of
maintaining a local variable.

Signed-off-by: Xuan Ding <xuan.ding@intel.com>
---
 examples/vhost/main.c | 31 +++++++++++++++++--------------
 examples/vhost/main.h |  1 -
 2 files changed, 17 insertions(+), 15 deletions(-)

diff --git a/examples/vhost/main.c b/examples/vhost/main.c
index d0bf1f31e3..7a5d97cef6 100644
--- a/examples/vhost/main.c
+++ b/examples/vhost/main.c
@@ -842,11 +842,8 @@ complete_async_pkts(struct vhost_dev *vdev)
 
 	complete_count = rte_vhost_poll_enqueue_completed(vdev->vid,
 					VIRTIO_RXQ, p_cpl, MAX_PKT_BURST);
-	if (complete_count) {
+	if (complete_count)
 		free_pkts(p_cpl, complete_count);
-		__atomic_sub_fetch(&vdev->pkts_inflight, complete_count, __ATOMIC_SEQ_CST);
-	}
-
 }
 
 static __rte_always_inline void
@@ -886,7 +883,6 @@ drain_vhost(struct vhost_dev *vdev)
 
 		complete_async_pkts(vdev);
 		ret = rte_vhost_submit_enqueue_burst(vdev->vid, VIRTIO_RXQ, m, nr_xmit);
-		__atomic_add_fetch(&vdev->pkts_inflight, ret, __ATOMIC_SEQ_CST);
 
 		enqueue_fail = nr_xmit - ret;
 		if (enqueue_fail)
@@ -1212,7 +1208,6 @@ drain_eth_rx(struct vhost_dev *vdev)
 		complete_async_pkts(vdev);
 		enqueue_count = rte_vhost_submit_enqueue_burst(vdev->vid,
 					VIRTIO_RXQ, pkts, rx_count);
-		__atomic_add_fetch(&vdev->pkts_inflight, enqueue_count, __ATOMIC_SEQ_CST);
 
 		enqueue_fail = rx_count - enqueue_count;
 		if (enqueue_fail)
@@ -1384,13 +1379,17 @@ destroy_device(int vid)
 
 	if (async_vhost_driver) {
 		uint16_t n_pkt = 0;
-		struct rte_mbuf *m_cpl[vdev->pkts_inflight];
+		int pkts_inflight;
 
-		while (vdev->pkts_inflight) {
+		pkts_inflight = rte_vhost_async_get_inflight_thread_unsafe(vid, VIRTIO_RXQ);
+		struct rte_mbuf *m_cpl[pkts_inflight];
+
+		while (pkts_inflight) {
 			n_pkt = rte_vhost_clear_queue_thread_unsafe(vid, VIRTIO_RXQ,
-						m_cpl, vdev->pkts_inflight);
+						m_cpl, pkts_inflight);
 			free_pkts(m_cpl, n_pkt);
-			__atomic_sub_fetch(&vdev->pkts_inflight, n_pkt, __ATOMIC_SEQ_CST);
+			pkts_inflight = rte_vhost_async_get_inflight_thread_unsafe(vid,
+									VIRTIO_RXQ);
 		}
 
 		rte_vhost_async_channel_unregister(vid, VIRTIO_RXQ);
@@ -1500,13 +1499,17 @@ vring_state_changed(int vid, uint16_t queue_id, int enable)
 	if (async_vhost_driver) {
 		if (!enable) {
 			uint16_t n_pkt = 0;
-			struct rte_mbuf *m_cpl[vdev->pkts_inflight];
+			int pkts_inflight;
+
+			pkts_inflight = rte_vhost_async_get_inflight_thread_unsafe(vid, queue_id);
+			struct rte_mbuf *m_cpl[pkts_inflight];
 
-			while (vdev->pkts_inflight) {
+			while (pkts_inflight) {
 				n_pkt = rte_vhost_clear_queue_thread_unsafe(vid, queue_id,
-							m_cpl, vdev->pkts_inflight);
+							m_cpl, pkts_inflight);
 				free_pkts(m_cpl, n_pkt);
-				__atomic_sub_fetch(&vdev->pkts_inflight, n_pkt, __ATOMIC_SEQ_CST);
+				pkts_inflight = rte_vhost_async_get_inflight_thread_unsafe(vid,
+										queue_id);
 			}
 		}
 	}
diff --git a/examples/vhost/main.h b/examples/vhost/main.h
index e7b1ac60a6..0ccdce4b4a 100644
--- a/examples/vhost/main.h
+++ b/examples/vhost/main.h
@@ -51,7 +51,6 @@ struct vhost_dev {
 	uint64_t features;
 	size_t hdr_len;
 	uint16_t nr_vrings;
-	uint16_t pkts_inflight;
 	struct rte_vhost_memory *mem;
 	struct device_statistics stats;
 	TAILQ_ENTRY(vhost_dev) global_vdev_entry;
-- 
2.17.1