From: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru> There is a family of cleanup from completed transmits functions. Fix packed virtqueues cleanup functions to have the same signature as split virtqueues have. This lets all functions of the family to match the same callback prototype. Fixes: 892dc798fa9 ("net/virtio: implement Tx path for packed queues") Cc: stable@dpdk.org Signed-off-by: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru> Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru> --- drivers/net/virtio/virtqueue.h | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h index 03957b2bd0..d0c48ca415 100644 --- a/drivers/net/virtio/virtqueue.h +++ b/drivers/net/virtio/virtqueue.h @@ -803,25 +803,26 @@ vq_ring_free_id_packed(struct virtqueue *vq, uint16_t id) } static void -virtio_xmit_cleanup_inorder_packed(struct virtqueue *vq, int num) +virtio_xmit_cleanup_inorder_packed(struct virtqueue *vq, uint16_t num) { uint16_t used_idx, id, curr_id, free_cnt = 0; uint16_t size = vq->vq_nentries; struct vring_packed_desc *desc = vq->vq_packed.ring.desc; struct vq_desc_extra *dxp; + int nb = num; used_idx = vq->vq_used_cons_idx; /* desc_is_used has a load-acquire or rte_io_rmb inside * and wait for used desc in virtqueue. */ - while (num > 0 && desc_is_used(&desc[used_idx], vq)) { + while (nb > 0 && desc_is_used(&desc[used_idx], vq)) { id = desc[used_idx].id; do { curr_id = used_idx; dxp = &vq->vq_descx[used_idx]; used_idx += dxp->ndescs; free_cnt += dxp->ndescs; - num -= dxp->ndescs; + nb -= dxp->ndescs; if (used_idx >= size) { used_idx -= size; vq->vq_packed.used_wrap_counter ^= 1; @@ -837,7 +838,7 @@ virtio_xmit_cleanup_inorder_packed(struct virtqueue *vq, int num) } static void -virtio_xmit_cleanup_normal_packed(struct virtqueue *vq, int num) +virtio_xmit_cleanup_normal_packed(struct virtqueue *vq, uint16_t num) { uint16_t used_idx, id; uint16_t size = vq->vq_nentries; @@ -867,7 +868,7 @@ virtio_xmit_cleanup_normal_packed(struct virtqueue *vq, int num) /* Cleanup from completed transmits. */ static inline void -virtio_xmit_cleanup_packed(struct virtqueue *vq, int num, int in_order) +virtio_xmit_cleanup_packed(struct virtqueue *vq, uint16_t num, int in_order) { if (in_order) virtio_xmit_cleanup_inorder_packed(vq, num); -- 2.30.2
From: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru> Free Tx completed mbufs on device stop. Not completed Tx mbufs cannot be freed since they are still in use. Fixes: c1f86306a02 ("virtio: add new driver") Cc: stable@dpdk.org Signed-off-by: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru> Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru> --- drivers/net/virtio/virtio_ethdev.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c index e58085a2c9..ed3fefee7c 100644 --- a/drivers/net/virtio/virtio_ethdev.c +++ b/drivers/net/virtio/virtio_ethdev.c @@ -2393,6 +2393,34 @@ static void virtio_dev_free_mbufs(struct rte_eth_dev *dev) PMD_INIT_LOG(DEBUG, "%d mbufs freed", mbuf_num); } +static void +virtio_tx_completed_cleanup(struct rte_eth_dev *dev) +{ + struct virtio_hw *hw = dev->data->dev_private; + struct virtqueue *vq; + int qidx; + void (*xmit_cleanup)(struct virtqueue *vq, uint16_t nb_used); + + if (virtio_with_packed_queue(hw)) { + if (hw->use_vec_tx) + xmit_cleanup = &virtio_xmit_cleanup_inorder_packed; + else if (virtio_with_feature(hw, VIRTIO_F_IN_ORDER)) + xmit_cleanup = &virtio_xmit_cleanup_inorder_packed; + else + xmit_cleanup = &virtio_xmit_cleanup_normal_packed; + } else { + if (hw->use_inorder_tx) + xmit_cleanup = &virtio_xmit_cleanup_inorder; + else + xmit_cleanup = &virtio_xmit_cleanup; + } + + for (qidx = 0; qidx < hw->max_queue_pairs; qidx++) { + vq = hw->vqs[2 * qidx + VTNET_SQ_TQ_QUEUE_IDX]; + xmit_cleanup(vq, virtqueue_nused(vq)); + } +} + /* * Stop device: disable interrupt and mark link down */ @@ -2411,6 +2439,8 @@ virtio_dev_stop(struct rte_eth_dev *dev) goto out_unlock; hw->started = 0; + virtio_tx_completed_cleanup(dev); + if (intr_conf->lsc || intr_conf->rxq) { virtio_intr_disable(dev); -- 2.30.2
On 8/18/21 4:13 PM, Andrew Rybchenko wrote:
> From: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru>
>
> There is a family of cleanup from completed transmits functions.
> Fix packed virtqueues cleanup functions to have the same signature
> as split virtqueues have. This lets all functions of the family to
> match the same callback prototype.
>
> Fixes: 892dc798fa9 ("net/virtio: implement Tx path for packed queues")
> Cc: stable@dpdk.org
>
> Signed-off-by: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru>
> Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
> ---
> drivers/net/virtio/virtqueue.h | 11 ++++++-----
> 1 file changed, 6 insertions(+), 5 deletions(-)
>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
On 8/18/21 4:13 PM, Andrew Rybchenko wrote: > From: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru> > > Free Tx completed mbufs on device stop. Not completed Tx mbufs cannot be > freed since they are still in use. > > Fixes: c1f86306a02 ("virtio: add new driver") > Cc: stable@dpdk.org > > Signed-off-by: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru> > Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru> > --- > drivers/net/virtio/virtio_ethdev.c | 30 ++++++++++++++++++++++++++++++ > 1 file changed, 30 insertions(+) > > diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c > index e58085a2c9..ed3fefee7c 100644 > --- a/drivers/net/virtio/virtio_ethdev.c > +++ b/drivers/net/virtio/virtio_ethdev.c > @@ -2393,6 +2393,34 @@ static void virtio_dev_free_mbufs(struct rte_eth_dev *dev) > PMD_INIT_LOG(DEBUG, "%d mbufs freed", mbuf_num); > } > > +static void > +virtio_tx_completed_cleanup(struct rte_eth_dev *dev) > +{ > + struct virtio_hw *hw = dev->data->dev_private; > + struct virtqueue *vq; > + int qidx; > + void (*xmit_cleanup)(struct virtqueue *vq, uint16_t nb_used); > + > + if (virtio_with_packed_queue(hw)) { > + if (hw->use_vec_tx) > + xmit_cleanup = &virtio_xmit_cleanup_inorder_packed; > + else if (virtio_with_feature(hw, VIRTIO_F_IN_ORDER)) > + xmit_cleanup = &virtio_xmit_cleanup_inorder_packed; > + else > + xmit_cleanup = &virtio_xmit_cleanup_normal_packed; > + } else { > + if (hw->use_inorder_tx) > + xmit_cleanup = &virtio_xmit_cleanup_inorder; > + else > + xmit_cleanup = &virtio_xmit_cleanup; > + } > + > + for (qidx = 0; qidx < hw->max_queue_pairs; qidx++) { > + vq = hw->vqs[2 * qidx + VTNET_SQ_TQ_QUEUE_IDX]; Maybe add a check to ensure that vq is non-NULL since it is dereferenced later without checking. > + xmit_cleanup(vq, virtqueue_nused(vq)); > + } > +} > + > /* > * Stop device: disable interrupt and mark link down > */ > @@ -2411,6 +2439,8 @@ virtio_dev_stop(struct rte_eth_dev *dev) > goto out_unlock; > hw->started = 0; > > + virtio_tx_completed_cleanup(dev); > + > if (intr_conf->lsc || intr_conf->rxq) { > virtio_intr_disable(dev); > >
On 9/13/21 6:41 PM, Maxime Coquelin wrote: > > > On 8/18/21 4:13 PM, Andrew Rybchenko wrote: >> From: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru> >> >> Free Tx completed mbufs on device stop. Not completed Tx mbufs cannot be >> freed since they are still in use. >> >> Fixes: c1f86306a02 ("virtio: add new driver") >> Cc: stable@dpdk.org >> >> Signed-off-by: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru> >> Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru> >> --- >> drivers/net/virtio/virtio_ethdev.c | 30 ++++++++++++++++++++++++++++++ >> 1 file changed, 30 insertions(+) >> >> diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c >> index e58085a2c9..ed3fefee7c 100644 >> --- a/drivers/net/virtio/virtio_ethdev.c >> +++ b/drivers/net/virtio/virtio_ethdev.c >> @@ -2393,6 +2393,34 @@ static void virtio_dev_free_mbufs(struct rte_eth_dev *dev) >> PMD_INIT_LOG(DEBUG, "%d mbufs freed", mbuf_num); >> } >> >> +static void >> +virtio_tx_completed_cleanup(struct rte_eth_dev *dev) >> +{ >> + struct virtio_hw *hw = dev->data->dev_private; >> + struct virtqueue *vq; >> + int qidx; >> + void (*xmit_cleanup)(struct virtqueue *vq, uint16_t nb_used); >> + >> + if (virtio_with_packed_queue(hw)) { >> + if (hw->use_vec_tx) >> + xmit_cleanup = &virtio_xmit_cleanup_inorder_packed; >> + else if (virtio_with_feature(hw, VIRTIO_F_IN_ORDER)) >> + xmit_cleanup = &virtio_xmit_cleanup_inorder_packed; >> + else >> + xmit_cleanup = &virtio_xmit_cleanup_normal_packed; >> + } else { >> + if (hw->use_inorder_tx) >> + xmit_cleanup = &virtio_xmit_cleanup_inorder; >> + else >> + xmit_cleanup = &virtio_xmit_cleanup; >> + } >> + >> + for (qidx = 0; qidx < hw->max_queue_pairs; qidx++) { >> + vq = hw->vqs[2 * qidx + VTNET_SQ_TQ_QUEUE_IDX]; > > Maybe add a check to ensure that vq is non-NULL since it is dereferenced > later without checking. Good idea, I'll send v2. > >> + xmit_cleanup(vq, virtqueue_nused(vq)); >> + } >> +} >> + >> /* >> * Stop device: disable interrupt and mark link down >> */ >> @@ -2411,6 +2439,8 @@ virtio_dev_stop(struct rte_eth_dev *dev) >> goto out_unlock; >> hw->started = 0; >> >> + virtio_tx_completed_cleanup(dev); >> + >> if (intr_conf->lsc || intr_conf->rxq) { >> virtio_intr_disable(dev); >> >>
From: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru> There is a family of cleanup from completed transmits functions. Fix packed virtqueues cleanup functions to have the same signature as split virtqueues have. This lets all functions of the family to match the same callback prototype. Fixes: 892dc798fa9 ("net/virtio: implement Tx path for packed queues") Cc: stable@dpdk.org Signed-off-by: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru> Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru> --- drivers/net/virtio/virtqueue.h | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h index 03957b2bd0..d0c48ca415 100644 --- a/drivers/net/virtio/virtqueue.h +++ b/drivers/net/virtio/virtqueue.h @@ -803,25 +803,26 @@ vq_ring_free_id_packed(struct virtqueue *vq, uint16_t id) } static void -virtio_xmit_cleanup_inorder_packed(struct virtqueue *vq, int num) +virtio_xmit_cleanup_inorder_packed(struct virtqueue *vq, uint16_t num) { uint16_t used_idx, id, curr_id, free_cnt = 0; uint16_t size = vq->vq_nentries; struct vring_packed_desc *desc = vq->vq_packed.ring.desc; struct vq_desc_extra *dxp; + int nb = num; used_idx = vq->vq_used_cons_idx; /* desc_is_used has a load-acquire or rte_io_rmb inside * and wait for used desc in virtqueue. */ - while (num > 0 && desc_is_used(&desc[used_idx], vq)) { + while (nb > 0 && desc_is_used(&desc[used_idx], vq)) { id = desc[used_idx].id; do { curr_id = used_idx; dxp = &vq->vq_descx[used_idx]; used_idx += dxp->ndescs; free_cnt += dxp->ndescs; - num -= dxp->ndescs; + nb -= dxp->ndescs; if (used_idx >= size) { used_idx -= size; vq->vq_packed.used_wrap_counter ^= 1; @@ -837,7 +838,7 @@ virtio_xmit_cleanup_inorder_packed(struct virtqueue *vq, int num) } static void -virtio_xmit_cleanup_normal_packed(struct virtqueue *vq, int num) +virtio_xmit_cleanup_normal_packed(struct virtqueue *vq, uint16_t num) { uint16_t used_idx, id; uint16_t size = vq->vq_nentries; @@ -867,7 +868,7 @@ virtio_xmit_cleanup_normal_packed(struct virtqueue *vq, int num) /* Cleanup from completed transmits. */ static inline void -virtio_xmit_cleanup_packed(struct virtqueue *vq, int num, int in_order) +virtio_xmit_cleanup_packed(struct virtqueue *vq, uint16_t num, int in_order) { if (in_order) virtio_xmit_cleanup_inorder_packed(vq, num); -- 2.30.2
From: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru> Free Tx completed mbufs on device stop. Not completed Tx mbufs cannot be freed since they are still in use. Fixes: c1f86306a02 ("virtio: add new driver") Cc: stable@dpdk.org Signed-off-by: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru> Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru> --- v2: - check vq pointer vs NULL before calling cleanup function drivers/net/virtio/virtio_ethdev.c | 31 ++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c index da1633d77e..3e3b42eaf6 100644 --- a/drivers/net/virtio/virtio_ethdev.c +++ b/drivers/net/virtio/virtio_ethdev.c @@ -2401,6 +2401,35 @@ static void virtio_dev_free_mbufs(struct rte_eth_dev *dev) PMD_INIT_LOG(DEBUG, "%d mbufs freed", mbuf_num); } +static void +virtio_tx_completed_cleanup(struct rte_eth_dev *dev) +{ + struct virtio_hw *hw = dev->data->dev_private; + struct virtqueue *vq; + int qidx; + void (*xmit_cleanup)(struct virtqueue *vq, uint16_t nb_used); + + if (virtio_with_packed_queue(hw)) { + if (hw->use_vec_tx) + xmit_cleanup = &virtio_xmit_cleanup_inorder_packed; + else if (virtio_with_feature(hw, VIRTIO_F_IN_ORDER)) + xmit_cleanup = &virtio_xmit_cleanup_inorder_packed; + else + xmit_cleanup = &virtio_xmit_cleanup_normal_packed; + } else { + if (hw->use_inorder_tx) + xmit_cleanup = &virtio_xmit_cleanup_inorder; + else + xmit_cleanup = &virtio_xmit_cleanup; + } + + for (qidx = 0; qidx < hw->max_queue_pairs; qidx++) { + vq = hw->vqs[2 * qidx + VTNET_SQ_TQ_QUEUE_IDX]; + if (vq != NULL) + xmit_cleanup(vq, virtqueue_nused(vq)); + } +} + /* * Stop device: disable interrupt and mark link down */ @@ -2419,6 +2448,8 @@ virtio_dev_stop(struct rte_eth_dev *dev) goto out_unlock; hw->started = 0; + virtio_tx_completed_cleanup(dev); + if (intr_conf->lsc || intr_conf->rxq) { virtio_intr_disable(dev); -- 2.30.2
On 9/15/21 11:19, Andrew Rybchenko wrote:
> From: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru>
>
> There is a family of cleanup from completed transmits functions.
> Fix packed virtqueues cleanup functions to have the same signature
> as split virtqueues have. This lets all functions of the family to
> match the same callback prototype.
>
> Fixes: 892dc798fa9 ("net/virtio: implement Tx path for packed queues")
> Cc: stable@dpdk.org
>
> Signed-off-by: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru>
> Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
> ---
> drivers/net/virtio/virtqueue.h | 11 ++++++-----
> 1 file changed, 6 insertions(+), 5 deletions(-)
>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
On 9/15/21 11:19, Andrew Rybchenko wrote:
> From: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru>
>
> Free Tx completed mbufs on device stop. Not completed Tx mbufs cannot be
> freed since they are still in use.
>
> Fixes: c1f86306a02 ("virtio: add new driver")
> Cc: stable@dpdk.org
>
> Signed-off-by: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru>
> Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
> ---
> v2:
> - check vq pointer vs NULL before calling cleanup function
>
> drivers/net/virtio/virtio_ethdev.c | 31 ++++++++++++++++++++++++++++++
> 1 file changed, 31 insertions(+)
>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
On 9/15/21 11:19, Andrew Rybchenko wrote:
> From: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru>
>
> There is a family of cleanup from completed transmits functions.
> Fix packed virtqueues cleanup functions to have the same signature
> as split virtqueues have. This lets all functions of the family to
> match the same callback prototype.
>
> Fixes: 892dc798fa9 ("net/virtio: implement Tx path for packed queues")
> Cc: stable@dpdk.org
>
> Signed-off-by: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru>
> Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
> ---
> drivers/net/virtio/virtqueue.h | 11 ++++++-----
> 1 file changed, 6 insertions(+), 5 deletions(-)
>
Applied to dpdk-next-virtio/main.
Thanks,
Maxime
On 9/15/21 11:19, Andrew Rybchenko wrote:
> From: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru>
>
> Free Tx completed mbufs on device stop. Not completed Tx mbufs cannot be
> freed since they are still in use.
>
> Fixes: c1f86306a02 ("virtio: add new driver")
> Cc: stable@dpdk.org
>
> Signed-off-by: Ivan Ilchenko <ivan.ilchenko@oktetlabs.ru>
> Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
> ---
> v2:
> - check vq pointer vs NULL before calling cleanup function
>
> drivers/net/virtio/virtio_ethdev.c | 31 ++++++++++++++++++++++++++++++
> 1 file changed, 31 insertions(+)
>
Applied to dpdk-next-virtio/main.
Thanks,
Maxime