[dpdk-dev] [PATCH 0/4] Some fixes for vhost zero copy

[dpdk-dev] [PATCH 0/4] Some fixes for vhost zero copy

[Tiwei Bie]

Tiwei Bie (4):
  vhost: restore mbuf first when freeing zmbuf
  vhost: fix potential use-after-free for zero copy mbuf
  vhost: fix potential use-after-free for memory region
  doc: improve vhost zero copy guide

 doc/guides/prog_guide/vhost_lib.rst |  3 +++
 lib/librte_vhost/vhost.h            | 34 +++++++++++++++++++++++
 lib/librte_vhost/vhost_user.c       | 42 ++++++++++++++++++++++-------
 lib/librte_vhost/virtio_net.c       | 34 -----------------------
 4 files changed, 70 insertions(+), 43 deletions(-)

-- 
2.17.1

[dpdk-dev] [PATCH 1/4] vhost: restore mbuf first when freeing zmbuf

[Tiwei Bie]

The mbufs should also be restored in free_zmbufs().

Fixes: b0a985d1f340 ("vhost: add dequeue zero copy")
Fixes: 3ebd930588b7 ("vhost: fix mbuf free")
Cc: stable@dpdk.org

Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
---
 lib/librte_vhost/vhost.h      | 16 ++++++++++++++++
 lib/librte_vhost/vhost_user.c |  1 +
 lib/librte_vhost/virtio_net.c | 16 ----------------
 3 files changed, 17 insertions(+), 16 deletions(-)

diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
index fc31796bf..bcfce274b 100644
--- a/lib/librte_vhost/vhost.h
+++ b/lib/librte_vhost/vhost.h
@@ -741,4 +741,20 @@ free_ind_table(void *idesc)
 	rte_free(idesc);
 }
 
+static __rte_always_inline void
+restore_mbuf(struct rte_mbuf *m)
+{
+	uint32_t mbuf_size, priv_size;
+
+	while (m) {
+		priv_size = rte_pktmbuf_priv_size(m->pool);
+		mbuf_size = sizeof(struct rte_mbuf) + priv_size;
+		/* start of buffer is after mbuf structure and priv data */
+
+		m->buf_addr = (char *)m + mbuf_size;
+		m->buf_iova = rte_mempool_virt2iova(m) + mbuf_size;
+		m = m->next;
+	}
+}
+
 #endif /* _VHOST_NET_CDEV_H_ */
diff --git a/lib/librte_vhost/vhost_user.c b/lib/librte_vhost/vhost_user.c
index b086ad95f..e3ddf2589 100644
--- a/lib/librte_vhost/vhost_user.c
+++ b/lib/librte_vhost/vhost_user.c
@@ -1218,6 +1218,7 @@ free_zmbufs(struct vhost_virtqueue *vq)
 	     zmbuf != NULL; zmbuf = next) {
 		next = TAILQ_NEXT(zmbuf, next);
 
+		restore_mbuf(zmbuf->mbuf);
 		rte_pktmbuf_free(zmbuf->mbuf);
 		TAILQ_REMOVE(&vq->zmbuf_list, zmbuf, next);
 	}
diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c
index 37a4c00d2..862ca5e1a 100644
--- a/lib/librte_vhost/virtio_net.c
+++ b/lib/librte_vhost/virtio_net.c
@@ -1318,22 +1318,6 @@ mbuf_is_consumed(struct rte_mbuf *m)
 	return true;
 }
 
-static __rte_always_inline void
-restore_mbuf(struct rte_mbuf *m)
-{
-	uint32_t mbuf_size, priv_size;
-
-	while (m) {
-		priv_size = rte_pktmbuf_priv_size(m->pool);
-		mbuf_size = sizeof(struct rte_mbuf) + priv_size;
-		/* start of buffer is after mbuf structure and priv data */
-
-		m->buf_addr = (char *)m + mbuf_size;
-		m->buf_iova = rte_mempool_virt2iova(m) + mbuf_size;
-		m = m->next;
-	}
-}
-
 static __rte_always_inline uint16_t
 virtio_dev_tx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
 	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count)
-- 
2.17.1

Re: [dpdk-dev] [PATCH 1/4] vhost: restore mbuf first when freeing zmbuf

[Maxime Coquelin]

On 2/22/19 3:42 AM, Tiwei Bie wrote:
> The mbufs should also be restored in free_zmbufs().
> 
> Fixes: b0a985d1f340 ("vhost: add dequeue zero copy")
> Fixes: 3ebd930588b7 ("vhost: fix mbuf free")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
> ---
>   lib/librte_vhost/vhost.h      | 16 ++++++++++++++++
>   lib/librte_vhost/vhost_user.c |  1 +
>   lib/librte_vhost/virtio_net.c | 16 ----------------
>   3 files changed, 17 insertions(+), 16 deletions(-)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[dpdk-dev] [PATCH 2/4] vhost: fix potential use-after-free for zero copy mbuf

[Tiwei Bie]

Don't free the zero copy mbufs before they have been consumed,
otherwise there could be use-after-free.

Fixes: b0a985d1f340 ("vhost: add dequeue zero copy")
Cc: stable@dpdk.org

Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
---
 lib/librte_vhost/vhost.h      | 12 ++++++++++++
 lib/librte_vhost/vhost_user.c |  3 +++
 lib/librte_vhost/virtio_net.c | 12 ------------
 3 files changed, 15 insertions(+), 12 deletions(-)

diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
index bcfce274b..044651b19 100644
--- a/lib/librte_vhost/vhost.h
+++ b/lib/librte_vhost/vhost.h
@@ -757,4 +757,16 @@ restore_mbuf(struct rte_mbuf *m)
 	}
 }
 
+static __rte_always_inline bool
+mbuf_is_consumed(struct rte_mbuf *m)
+{
+	while (m) {
+		if (rte_mbuf_refcnt_read(m) > 1)
+			return false;
+		m = m->next;
+	}
+
+	return true;
+}
+
 #endif /* _VHOST_NET_CDEV_H_ */
diff --git a/lib/librte_vhost/vhost_user.c b/lib/librte_vhost/vhost_user.c
index e3ddf2589..6d8253514 100644
--- a/lib/librte_vhost/vhost_user.c
+++ b/lib/librte_vhost/vhost_user.c
@@ -1218,6 +1218,9 @@ free_zmbufs(struct vhost_virtqueue *vq)
 	     zmbuf != NULL; zmbuf = next) {
 		next = TAILQ_NEXT(zmbuf, next);
 
+		while (!mbuf_is_consumed(zmbuf->mbuf))
+			usleep(1000);
+
 		restore_mbuf(zmbuf->mbuf);
 		rte_pktmbuf_free(zmbuf->mbuf);
 		TAILQ_REMOVE(&vq->zmbuf_list, zmbuf, next);
diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c
index 862ca5e1a..40a292364 100644
--- a/lib/librte_vhost/virtio_net.c
+++ b/lib/librte_vhost/virtio_net.c
@@ -1306,18 +1306,6 @@ get_zmbuf(struct vhost_virtqueue *vq)
 	return NULL;
 }
 
-static __rte_always_inline bool
-mbuf_is_consumed(struct rte_mbuf *m)
-{
-	while (m) {
-		if (rte_mbuf_refcnt_read(m) > 1)
-			return false;
-		m = m->next;
-	}
-
-	return true;
-}
-
 static __rte_always_inline uint16_t
 virtio_dev_tx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
 	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count)
-- 
2.17.1

Re: [dpdk-dev] [PATCH 2/4] vhost: fix potential use-after-free for zero copy mbuf

[Maxime Coquelin]

On 2/22/19 3:42 AM, Tiwei Bie wrote:
> Don't free the zero copy mbufs before they have been consumed,
> otherwise there could be use-after-free.
> 
> Fixes: b0a985d1f340 ("vhost: add dequeue zero copy")
> Cc:stable@dpdk.org
> 
> Signed-off-by: Tiwei Bie<tiwei.bie@intel.com>
> ---
>   lib/librte_vhost/vhost.h      | 12 ++++++++++++
>   lib/librte_vhost/vhost_user.c |  3 +++
>   lib/librte_vhost/virtio_net.c | 12 ------------
>   3 files changed, 15 insertions(+), 12 deletions(-)


Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[dpdk-dev] [PATCH 3/4] vhost: fix potential use-after-free for memory region

[Tiwei Bie]

Reclaim outstanding zmbufs first before freeing memory regions,
otherwise there could be use-after-free.

Fixes: b0a985d1f340 ("vhost: add dequeue zero copy")
Cc: stable@dpdk.org

Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
---
 lib/librte_vhost/vhost.h      |  6 +++++
 lib/librte_vhost/vhost_user.c | 46 +++++++++++++++++++++++++----------
 lib/librte_vhost/virtio_net.c |  6 -----
 3 files changed, 39 insertions(+), 19 deletions(-)

diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
index 044651b19..f008ec43b 100644
--- a/lib/librte_vhost/vhost.h
+++ b/lib/librte_vhost/vhost.h
@@ -769,4 +769,10 @@ mbuf_is_consumed(struct rte_mbuf *m)
 	return true;
 }
 
+static __rte_always_inline void
+put_zmbuf(struct zcopy_mbuf *zmbuf)
+{
+	zmbuf->in_use = 0;
+}
+
 #endif /* _VHOST_NET_CDEV_H_ */
diff --git a/lib/librte_vhost/vhost_user.c b/lib/librte_vhost/vhost_user.c
index 6d8253514..36c0c676d 100644
--- a/lib/librte_vhost/vhost_user.c
+++ b/lib/librte_vhost/vhost_user.c
@@ -93,15 +93,47 @@ get_blk_size(int fd)
 	return ret == -1 ? (uint64_t)-1 : (uint64_t)stat.st_blksize;
 }
 
+/*
+ * Reclaim all the outstanding zmbufs for a virtqueue.
+ */
+static void
+drain_zmbuf_list(struct vhost_virtqueue *vq)
+{
+	struct zcopy_mbuf *zmbuf, *next;
+
+	for (zmbuf = TAILQ_FIRST(&vq->zmbuf_list);
+	     zmbuf != NULL; zmbuf = next) {
+		next = TAILQ_NEXT(zmbuf, next);
+
+		while (!mbuf_is_consumed(zmbuf->mbuf))
+			usleep(1000);
+
+		TAILQ_REMOVE(&vq->zmbuf_list, zmbuf, next);
+		restore_mbuf(zmbuf->mbuf);
+		rte_pktmbuf_free(zmbuf->mbuf);
+		put_zmbuf(zmbuf);
+		vq->nr_zmbuf -= 1;
+	}
+}
+
 static void
 free_mem_region(struct virtio_net *dev)
 {
 	uint32_t i;
 	struct rte_vhost_mem_region *reg;
+	struct vhost_virtqueue *vq;
 
 	if (!dev || !dev->mem)
 		return;
 
+	if (dev->dequeue_zero_copy) {
+		for (i = 0; i < dev->nr_vring; i++) {
+			vq = dev->virtqueue[i];
+			if (vq)
+				drain_zmbuf_list(vq);
+		}
+	}
+
 	for (i = 0; i < dev->mem->nregions; i++) {
 		reg = &dev->mem->regions[i];
 		if (reg->host_user_addr) {
@@ -1212,19 +1244,7 @@ vhost_user_set_vring_kick(struct virtio_net **pdev, struct VhostUserMsg *msg,
 static void
 free_zmbufs(struct vhost_virtqueue *vq)
 {
-	struct zcopy_mbuf *zmbuf, *next;
-
-	for (zmbuf = TAILQ_FIRST(&vq->zmbuf_list);
-	     zmbuf != NULL; zmbuf = next) {
-		next = TAILQ_NEXT(zmbuf, next);
-
-		while (!mbuf_is_consumed(zmbuf->mbuf))
-			usleep(1000);
-
-		restore_mbuf(zmbuf->mbuf);
-		rte_pktmbuf_free(zmbuf->mbuf);
-		TAILQ_REMOVE(&vq->zmbuf_list, zmbuf, next);
-	}
+	drain_zmbuf_list(vq);
 
 	rte_free(vq->zmbufs);
 }
diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c
index 40a292364..a6a33a101 100644
--- a/lib/librte_vhost/virtio_net.c
+++ b/lib/librte_vhost/virtio_net.c
@@ -1063,12 +1063,6 @@ vhost_dequeue_offload(struct virtio_net_hdr *hdr, struct rte_mbuf *m)
 	}
 }
 
-static __rte_always_inline void
-put_zmbuf(struct zcopy_mbuf *zmbuf)
-{
-	zmbuf->in_use = 0;
-}
-
 static __rte_always_inline int
 copy_desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq,
 		  struct buf_vector *buf_vec, uint16_t nr_vec,
-- 
2.17.1

Re: [dpdk-dev] [PATCH 3/4] vhost: fix potential use-after-free for memory region

[Maxime Coquelin]

On 2/22/19 3:42 AM, Tiwei Bie wrote:
> Reclaim outstanding zmbufs first before freeing memory regions,
> otherwise there could be use-after-free.
> 
> Fixes: b0a985d1f340 ("vhost: add dequeue zero copy")
> Cc:stable@dpdk.org
> 
> Signed-off-by: Tiwei Bie<tiwei.bie@intel.com>
> ---
>   lib/librte_vhost/vhost.h      |  6 +++++
>   lib/librte_vhost/vhost_user.c | 46 +++++++++++++++++++++++++----------
>   lib/librte_vhost/virtio_net.c |  6 -----
>   3 files changed, 39 insertions(+), 19 deletions(-)


Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[dpdk-dev] [PATCH 4/4] doc: improve vhost zero copy guide

[Tiwei Bie]

Highlight that vhost zero copy mbufs should be consumed
as soon as possible.

Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
---
 doc/guides/prog_guide/vhost_lib.rst | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
index c77df338f..a86c07a62 100644
--- a/doc/guides/prog_guide/vhost_lib.rst
+++ b/doc/guides/prog_guide/vhost_lib.rst
@@ -92,6 +92,9 @@ The following is an overview of some key Vhost API functions:
       to use vfio-pci driver, please insert vfio-pci kernel module in noiommu
       mode.
 
+    * The consumer of zero copy mbufs should consume these mbufs as soon as
+      possible, otherwise it may block the operations in vhost.
+
   - ``RTE_VHOST_USER_IOMMU_SUPPORT``
 
     IOMMU support will be enabled when this flag is set. It is disabled by
-- 
2.17.1

Re: [dpdk-dev] [PATCH 4/4] doc: improve vhost zero copy guide

[Maxime Coquelin]

On 2/22/19 3:42 AM, Tiwei Bie wrote:
> Highlight that vhost zero copy mbufs should be consumed
> as soon as possible.
> 
> Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
> ---
>   doc/guides/prog_guide/vhost_lib.rst | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
> index c77df338f..a86c07a62 100644
> --- a/doc/guides/prog_guide/vhost_lib.rst
> +++ b/doc/guides/prog_guide/vhost_lib.rst
> @@ -92,6 +92,9 @@ The following is an overview of some key Vhost API functions:
>         to use vfio-pci driver, please insert vfio-pci kernel module in noiommu
>         mode.
>   
> +    * The consumer of zero copy mbufs should consume these mbufs as soon as
> +      possible, otherwise it may block the operations in vhost.
> +
>     - ``RTE_VHOST_USER_IOMMU_SUPPORT``
>   
>       IOMMU support will be enabled when this flag is set. It is disabled by
> 

Indeed, and it may also block QEMU. That's what I find really fragile
with dequeue zero-copy.

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

Re: [dpdk-dev] [PATCH 0/4] Some fixes for vhost zero copy

[Maxime Coquelin]

On 2/22/19 3:42 AM, Tiwei Bie wrote:
> Tiwei Bie (4):
>    vhost: restore mbuf first when freeing zmbuf
>    vhost: fix potential use-after-free for zero copy mbuf
>    vhost: fix potential use-after-free for memory region
>    doc: improve vhost zero copy guide
> 
>   doc/guides/prog_guide/vhost_lib.rst |  3 +++
>   lib/librte_vhost/vhost.h            | 34 +++++++++++++++++++++++
>   lib/librte_vhost/vhost_user.c       | 42 ++++++++++++++++++++++-------
>   lib/librte_vhost/virtio_net.c       | 34 -----------------------
>   4 files changed, 70 insertions(+), 43 deletions(-)
> 

Looking at the spec, I think we may need also to drain zmbufs in the
VHOST_USER_SET_VRING_ENABLE for the disable case:

""
If VHOST_USER_F_PROTOCOL_FEATURES has been negotiated, the ring is 
initialized
in a disabled state. Client must not pass data to/from the backend until 
ring is enabled by
VHOST_USER_SET_VRING_ENABLE with parameter 1, or after it has been 
disabled by
VHOST_USER_SET_VRING_ENABLE with parameter 0.

Each ring is initialized in a stopped state, client must not process it 
until
ring is started, or *after it has been stopped*.
""

Do you take care of this or I send a patch on top?

Thanks,
Maxime

Re: [dpdk-dev] [PATCH 0/4] Some fixes for vhost zero copy

[Tiwei Bie]

On Tue, Feb 26, 2019 at 03:46:41PM +0100, Maxime Coquelin wrote:
> On 2/22/19 3:42 AM, Tiwei Bie wrote:
> > Tiwei Bie (4):
> >    vhost: restore mbuf first when freeing zmbuf
> >    vhost: fix potential use-after-free for zero copy mbuf
> >    vhost: fix potential use-after-free for memory region
> >    doc: improve vhost zero copy guide
> > 
> >   doc/guides/prog_guide/vhost_lib.rst |  3 +++
> >   lib/librte_vhost/vhost.h            | 34 +++++++++++++++++++++++
> >   lib/librte_vhost/vhost_user.c       | 42 ++++++++++++++++++++++-------
> >   lib/librte_vhost/virtio_net.c       | 34 -----------------------
> >   4 files changed, 70 insertions(+), 43 deletions(-)
> > 
> 
> Looking at the spec, I think we may need also to drain zmbufs in the
> VHOST_USER_SET_VRING_ENABLE for the disable case:
> 
> ""
> If VHOST_USER_F_PROTOCOL_FEATURES has been negotiated, the ring is
> initialized
> in a disabled state. Client must not pass data to/from the backend until
> ring is enabled by
> VHOST_USER_SET_VRING_ENABLE with parameter 1, or after it has been disabled
> by
> VHOST_USER_SET_VRING_ENABLE with parameter 0.
> 
> Each ring is initialized in a stopped state, client must not process it
> until
> ring is started, or *after it has been stopped*.
> ""
> 
> Do you take care of this or I send a patch on top?

Agree. Please feel free to send any patch on top.

Thanks!
Tiwei

> 
> Thanks,
> Maxime

Re: [dpdk-dev] [PATCH 0/4] Some fixes for vhost zero copy

[Maxime Coquelin]

On 2/27/19 2:52 AM, Tiwei Bie wrote:
> On Tue, Feb 26, 2019 at 03:46:41PM +0100, Maxime Coquelin wrote:
>> On 2/22/19 3:42 AM, Tiwei Bie wrote:
>>> Tiwei Bie (4):
>>>     vhost: restore mbuf first when freeing zmbuf
>>>     vhost: fix potential use-after-free for zero copy mbuf
>>>     vhost: fix potential use-after-free for memory region
>>>     doc: improve vhost zero copy guide
>>>
>>>    doc/guides/prog_guide/vhost_lib.rst |  3 +++
>>>    lib/librte_vhost/vhost.h            | 34 +++++++++++++++++++++++
>>>    lib/librte_vhost/vhost_user.c       | 42 ++++++++++++++++++++++-------
>>>    lib/librte_vhost/virtio_net.c       | 34 -----------------------
>>>    4 files changed, 70 insertions(+), 43 deletions(-)
>>>
>>
>> Looking at the spec, I think we may need also to drain zmbufs in the
>> VHOST_USER_SET_VRING_ENABLE for the disable case:
>>
>> ""
>> If VHOST_USER_F_PROTOCOL_FEATURES has been negotiated, the ring is
>> initialized
>> in a disabled state. Client must not pass data to/from the backend until
>> ring is enabled by
>> VHOST_USER_SET_VRING_ENABLE with parameter 1, or after it has been disabled
>> by
>> VHOST_USER_SET_VRING_ENABLE with parameter 0.
>>
>> Each ring is initialized in a stopped state, client must not process it
>> until
>> ring is started, or *after it has been stopped*.
>> ""
>>
>> Do you take care of this or I send a patch on top?
> 
> Agree. Please feel free to send any patch on top.

Good, I'll do the patch now.

Maxime

> Thanks!
> Tiwei
> 
>>
>> Thanks,
>> Maxime

Re: [dpdk-dev] [PATCH 0/4] Some fixes for vhost zero copy

[Maxime Coquelin]

On 2/22/19 3:42 AM, Tiwei Bie wrote:
> Tiwei Bie (4):
>    vhost: restore mbuf first when freeing zmbuf
>    vhost: fix potential use-after-free for zero copy mbuf
>    vhost: fix potential use-after-free for memory region
>    doc: improve vhost zero copy guide
> 
>   doc/guides/prog_guide/vhost_lib.rst |  3 +++
>   lib/librte_vhost/vhost.h            | 34 +++++++++++++++++++++++
>   lib/librte_vhost/vhost_user.c       | 42 ++++++++++++++++++++++-------
>   lib/librte_vhost/virtio_net.c       | 34 -----------------------
>   4 files changed, 70 insertions(+), 43 deletions(-)
> 

Applied to dpdk-next-virtio/master.

Thanks,
Maxime