DPDK patches and discussions
 help / color / mirror / Atom feed
* [PATCH 00/21] Add control queue & MQ support to Virtio-user vDPA
@ 2023-02-07 15:15 Maxime Coquelin
  2023-02-07 15:15 ` [PATCH 01/21] net/virtio: move CVQ code into a dedicated file Maxime Coquelin
                   ` (4 more replies)
  0 siblings, 5 replies; 7+ messages in thread
From: Maxime Coquelin @ 2023-02-07 15:15 UTC (permalink / raw)
  To: dev, chenbo.xia, david.marchand, eperezma, stephen; +Cc: Maxime Coquelin

This series introduces control queue support for Vhost-vDPA
backend. This is a requirement to support multiqueue, but
be usefull for other features like RSS for example.

Since the Virtio-user layer of the Virtio PMD must handle
some control messages, like the number of queue pairs to
be used by the device, a shadow control queue is created
at Virtio-user layer.

Control messages from the regular Virtio control queue
are still dequeues and handled if needed by the Virtio-user
layer, and are then forwarded to the shadow control queue
so that the physical vDPA device can handle them.

This model is similar to the one adopted by the QEMU
project.

In order to avoid code duplication, virtqueue allocation
and control queue message sending has been factored out
of the Virtio layer to be reusable by the Virtio-user
layer.

Finally, in order to support vDPA hardware which may
support large number of queues, last patch removes the
8 queue pairs limitation by dynamically allocating
vring metadata.

The series has been tested with Nvidia Cx-6 DX NIC
with up to 16 queue pairs:

# echo 0 > /sys/bus/pci/devices/0000\:3b\:00.0/sriov_numvfs
# echo 0 > /sys/bus/pci/devices/0000\:3b\:00.1/sriov_numvfs
# modprobe vhost_vdpa
# modprobe mlx5_vdpa
# echo 1 > /sys/bus/pci/devices/0000\:3b\:00.0/sriov_numvfs
# echo 0000:3b:00.2 >/sys/bus/pci/drivers/mlx5_core/unbind
# devlink dev eswitch set pci/0000:3b:00.0 mode switchdev
# echo 0000:3b:00.2 >/sys/bus/pci/drivers/mlx5_core/bind
# vdpa dev add name vdpa0 mgmtdev pci/0000:3b:00.2 mac 00:11:22:33:44:03  max_vqp 16
# ulimit -l unlimited
# dpdk-testpmd -l 0,2,4,6 --socket-mem 1024,0  --vdev 'virtio_user0,path=/dev/vhost-vdpa-0' --no-pci -n 3 -- --nb-cores=3 -i --rxq=16 --txq=16

Changes in v2:
==============
- Fix double spaces (Chenbo)
- Get rid of uneeded gotos (Stephen)
- Only allocate packed ring metadata if supported (Chenbo)
- Rebased on top of main

Maxime Coquelin (21):
  net/virtio: move CVQ code into a dedicated file
  net/virtio: introduce notify callback for control queue
  net/virtio: virtqueue headers alloc refactoring
  net/virtio: remove port ID info from Rx queue
  net/virtio: remove unused fields in Tx queue struct
  net/virtio: remove unused queue ID field in Rx queue
  net/virtio: remove unused Port ID in control queue
  net/virtio: move vring memzone to virtqueue struct
  net/virtio: refactor indirect desc headers init
  net/virtio: alloc Rx SW ring only if vectorized path
  net/virtio: extract virtqueue init from virtio queue init
  net/virtio-user: fix device starting failure handling
  net/virtio-user: simplify queues setup
  net/virtio-user: use proper type for number of queue pairs
  net/virtio-user: get max number of queue pairs from device
  net/virtio-user: allocate shadow control queue
  net/virtio-user: send shadow virtqueue info to the backend
  net/virtio-user: add new callback to enable control queue
  net/virtio-user: forward control messages to shadow queue
  net/virtio-user: advertize control VQ support with vDPA
  net/virtio-user: remove max queues limitation

 drivers/net/virtio/meson.build                |   1 +
 drivers/net/virtio/virtio.h                   |   6 -
 drivers/net/virtio/virtio_cvq.c               | 229 +++++++++
 drivers/net/virtio/virtio_cvq.h               | 127 +++++
 drivers/net/virtio/virtio_ethdev.c            | 472 +-----------------
 drivers/net/virtio/virtio_rxtx.c              |  47 +-
 drivers/net/virtio/virtio_rxtx.h              |  31 +-
 drivers/net/virtio/virtio_rxtx_packed.c       |   3 +-
 drivers/net/virtio/virtio_rxtx_simple.c       |   3 +-
 drivers/net/virtio/virtio_rxtx_simple.h       |   7 +-
 .../net/virtio/virtio_rxtx_simple_altivec.c   |   4 +-
 drivers/net/virtio/virtio_rxtx_simple_neon.c  |   4 +-
 drivers/net/virtio/virtio_rxtx_simple_sse.c   |   4 +-
 drivers/net/virtio/virtio_user/vhost.h        |   1 +
 drivers/net/virtio/virtio_user/vhost_vdpa.c   |  19 +-
 .../net/virtio/virtio_user/virtio_user_dev.c  | 305 +++++++++--
 .../net/virtio/virtio_user/virtio_user_dev.h  |  30 +-
 drivers/net/virtio/virtio_user_ethdev.c       |  49 +-
 drivers/net/virtio/virtqueue.c                | 346 ++++++++++++-
 drivers/net/virtio/virtqueue.h                | 127 +----
 20 files changed, 1069 insertions(+), 746 deletions(-)
 create mode 100644 drivers/net/virtio/virtio_cvq.c
 create mode 100644 drivers/net/virtio/virtio_cvq.h

-- 
2.39.1


^ permalink raw reply	[flat|nested] 7+ messages in thread
* [PATCH 00/21] Add control queue & MQ support to Virtio-user vDPA
@ 2023-02-09  9:16 Maxime Coquelin
  2023-02-09  9:16 ` [PATCH 02/21] net/virtio: introduce notify callback for control queue Maxime Coquelin
  0 siblings, 1 reply; 7+ messages in thread
From: Maxime Coquelin @ 2023-02-09  9:16 UTC (permalink / raw)
  To: dev, chenbo.xia, david.marchand, eperezma, stephen; +Cc: Maxime Coquelin

This series introduces control queue support for Vhost-vDPA
backend. This is a requirement to support multiqueue, but
be usefull for other features like RSS for example.

Since the Virtio-user layer of the Virtio PMD must handle
some control messages, like the number of queue pairs to
be used by the device, a shadow control queue is created
at Virtio-user layer.

Control messages from the regular Virtio control queue
are still dequeues and handled if needed by the Virtio-user
layer, and are then forwarded to the shadow control queue
so that the physical vDPA device can handle them.

This model is similar to the one adopted by the QEMU
project.

In order to avoid code duplication, virtqueue allocation
and control queue message sending has been factored out
of the Virtio layer to be reusable by the Virtio-user
layer.

Finally, in order to support vDPA hardware which may
support large number of queues, last patch removes the
8 queue pairs limitation by dynamically allocating
vring metadata.

The series has been tested with Nvidia Cx-6 DX NIC
with up to 16 queue pairs:

# echo 0 > /sys/bus/pci/devices/0000\:3b\:00.0/sriov_numvfs
# echo 0 > /sys/bus/pci/devices/0000\:3b\:00.1/sriov_numvfs
# modprobe vhost_vdpa
# modprobe mlx5_vdpa
# echo 1 > /sys/bus/pci/devices/0000\:3b\:00.0/sriov_numvfs
# echo 0000:3b:00.2 >/sys/bus/pci/drivers/mlx5_core/unbind
# devlink dev eswitch set pci/0000:3b:00.0 mode switchdev
# echo 0000:3b:00.2 >/sys/bus/pci/drivers/mlx5_core/bind
# vdpa dev add name vdpa0 mgmtdev pci/0000:3b:00.2 mac 00:11:22:33:44:03  max_vqp 16
# ulimit -l unlimited
# dpdk-testpmd -l 0,2,4,6 --socket-mem 1024,0  --vdev 'virtio_user0,path=/dev/vhost-vdpa-0' --no-pci -n 3 -- --nb-cores=3 -i --rxq=16 --txq=16

Changes in v3:
==============
- Trivial code simplifications (Eugenio)

Changes in v2:
==============
- Fix double spaces (Chenbo)
- Get rid of uneeded gotos (Stephen)
- Only allocate packed ring metadata if supported (Chenbo)
- Rebased on top of main

Maxime Coquelin (21):
  net/virtio: move CVQ code into a dedicated file
  net/virtio: introduce notify callback for control queue
  net/virtio: virtqueue headers alloc refactoring
  net/virtio: remove port ID info from Rx queue
  net/virtio: remove unused fields in Tx queue struct
  net/virtio: remove unused queue ID field in Rx queue
  net/virtio: remove unused Port ID in control queue
  net/virtio: move vring memzone to virtqueue struct
  net/virtio: refactor indirect desc headers init
  net/virtio: alloc Rx SW ring only if vectorized path
  net/virtio: extract virtqueue init from virtio queue init
  net/virtio-user: fix device starting failure handling
  net/virtio-user: simplify queues setup
  net/virtio-user: use proper type for number of queue pairs
  net/virtio-user: get max number of queue pairs from device
  net/virtio-user: allocate shadow control queue
  net/virtio-user: send shadow virtqueue info to the backend
  net/virtio-user: add new callback to enable control queue
  net/virtio-user: forward control messages to shadow queue
  net/virtio-user: advertize control VQ support with vDPA
  net/virtio-user: remove max queues limitation

 drivers/net/virtio/meson.build                |   1 +
 drivers/net/virtio/virtio.h                   |   6 -
 drivers/net/virtio/virtio_cvq.c               | 229 +++++++++
 drivers/net/virtio/virtio_cvq.h               | 127 +++++
 drivers/net/virtio/virtio_ethdev.c            | 472 +-----------------
 drivers/net/virtio/virtio_rxtx.c              |  47 +-
 drivers/net/virtio/virtio_rxtx.h              |  31 +-
 drivers/net/virtio/virtio_rxtx_packed.c       |   3 +-
 drivers/net/virtio/virtio_rxtx_simple.c       |   3 +-
 drivers/net/virtio/virtio_rxtx_simple.h       |   7 +-
 .../net/virtio/virtio_rxtx_simple_altivec.c   |   4 +-
 drivers/net/virtio/virtio_rxtx_simple_neon.c  |   4 +-
 drivers/net/virtio/virtio_rxtx_simple_sse.c   |   4 +-
 drivers/net/virtio/virtio_user/vhost.h        |   1 +
 drivers/net/virtio/virtio_user/vhost_vdpa.c   |  16 +-
 .../net/virtio/virtio_user/virtio_user_dev.c  | 305 +++++++++--
 .../net/virtio/virtio_user/virtio_user_dev.h  |  30 +-
 drivers/net/virtio/virtio_user_ethdev.c       |  49 +-
 drivers/net/virtio/virtqueue.c                | 346 ++++++++++++-
 drivers/net/virtio/virtqueue.h                | 127 +----
 20 files changed, 1066 insertions(+), 746 deletions(-)
 create mode 100644 drivers/net/virtio/virtio_cvq.c
 create mode 100644 drivers/net/virtio/virtio_cvq.h

-- 
2.39.1


^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2023-02-09  9:17 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-02-07 15:15 [PATCH 00/21] Add control queue & MQ support to Virtio-user vDPA Maxime Coquelin
2023-02-07 15:15 ` [PATCH 01/21] net/virtio: move CVQ code into a dedicated file Maxime Coquelin
2023-02-07 15:15 ` [PATCH 02/21] net/virtio: introduce notify callback for control queue Maxime Coquelin
2023-02-07 15:15 ` [PATCH 03/21] net/virtio: virtqueue headers alloc refactoring Maxime Coquelin
2023-02-07 15:15 ` [PATCH 04/21] net/virtio: remove port ID info from Rx queue Maxime Coquelin
2023-02-07 15:17 ` [PATCH 00/21] Add control queue & MQ support to Virtio-user vDPA Maxime Coquelin
2023-02-09  9:16 Maxime Coquelin
2023-02-09  9:16 ` [PATCH 02/21] net/virtio: introduce notify callback for control queue Maxime Coquelin

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).