[dpdk-dev] [RFC 00/24] vhost: add virtio-vhost-user transport

[Stefan Hajnoczi <stefanha@redhat.com>]

This patch series implements the virtio-vhost-user device, which tunnels
vhost-user protocol messages over virtio.  This lets guests act as vhost device
backends for other guests.

The virtio-vhost-user device is the result of discussion about Wei Wang and
Zhiyong Yang's vhost-pci device.  This patch series demonstrates that vhost
device backends, such as the vhost vdev driver, can work over both AF_UNIX and
virtio-vhost-user without significant modifications.  This allows a lot of code
to be shared between traditional AF_UNIX vhost-user and virtio-vhost-user.  The
vhost-pci patches duplicated the vhost net device backend and didn't reuse
librte_vhost:

  http://dpdk.org/ml/archives/dev/2017-November/082615.html

User-visible changes
--------------------
The vhost vdev can now be used when DPDK runs inside a guest with a
virtio-vhost-user PCI device:

  --vdev net_vhost0,iface="0000:00:04.0",virtio-transport=1

The vhost-scsi example has also been extended to support virtio-vhost-user:

  ./vhost-scsi ... -- --virtio-vhost-user-pci "0000:00:04.0"

For more information (including instructions for running the code), see
https://wiki.qemu.org/Features/VirtioVhostUser

Virtio device design
--------------------
The virtio-vhost-user device is a new virtio device type.  It acts as a
vhost-user transport and is an alternative for the traditional AF_UNIX
transport.

You can find the virtio-vhost-user VIRTIO device specification here:
https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007

librte_vhost API changes
------------------------
This patch series extends librte_vhost so that it now accepts:

  rte_vhost_driver_register("0000:00:04.0",
                            RTE_VHOST_USER_VIRTIO_TRANSPORT);

All other librte_vhost API usage remains unchanged, except that the file
descriptors exposed in some <rte_vhost.h> structs will be -1 since there is no
file descriptor passing involved.

This makes it extremely easy to support virtio-vhost-user in existing vhost
device backends!  I have extended the vhost vdev driver and the vhost-scsi
example application in this patch series.

Patch series overview
---------------------
This series is based on commit 814339ba7eea13d132508af2cccec2f73568e2d0 from
dpdk-next-virtio/master.  You can also get my git branch here:

  https://github.com/stefanha/dpdk/tree/virtio-vhost-user

The initial patches refactor librte_vhost so that AF_UNIX-specific code is moved
to a new trans_af_unix.c file.  This also introduces a struct
vhost_transport_ops interface that all transports will implement:

  adf412c3c vhost: move vring_call() into trans_af_unix.c
  04079a077 vhost: move AF_UNIX code from socket.c to trans_af_unix.c
  885642091 vhost: allocate per-socket transport state
  9c48377df vhost: move socket_fd and un sockaddr into trans_af_unix.c
  c646e6292 vhost: move start_server/client() calls to trans_af_unix.c
  db07ef7a8 vhost: move vhost_user_connection to trans_af_unix.c
  d10b80163 vhost: move vhost_user_reconnect_init() into trans_af_unix.c
  1258bcd68 vhost: move vhost_user.fdset to trans_af_unix.c
  bc7f6d7ab vhost: pass vhost_transport_ops through vhost_new_device()
  b82187a29 vhost: embed struct virtio_net inside struct vhost_user_connection
  abbd544f2 vhost: extract vhost_user.c socket I/O into transport
  0658d711b vhost: move slave_req_fd field to AF_UNIX transport
  e2ecf78ed vhost: move mmap/munmap to AF_UNIX transport

I was about to add the virtio-vhost-user PCI driver when I realized that
lib/librte_vhost/ cannot have a dependency on rte_bus_pci.h (it lives in
drivers/).  The solution I chose is to move all of librte_vhost to
drivers/librte_vhost/, but I'm open to suggestions if this is undesirable.

  8615c2140 vhost: move librte_vhost to drivers/

Since virtio-vhost-user is a virtio device it's necessary to perform a sequence
of device initialization steps and set up virtqueues.  I didn't see a virtio
API in DPDK and didn't have time to create one myself.  I copied the virtio
code from drivers/net/virtio/ as a quick hack, but really there needs to be a
librte_virtio that is shared.

  c26937c66 vhost: add virtio pci framework

Next the virtio-vhost-user transport is added along with the
RTE_VHOST_USER_VIRTIO_TRANSPORT flag:

  7fad5adc4 vhost: remember a vhost_virtqueue's queue index
  d26922892 vhost: add virtio-vhost-user transport
  f562a70ab vhost: add RTE_VHOST_USER_VIRTIO_TRANSPORT flag

Then I extended the vhost vdev and vhost-scsi example application to support
virtio-vhost-user:

  47434f2a2 net/vhost: add virtio-vhost-user support
  22ca05d1b examples/vhost_scsi: add --socket-file argument
  db3b391dc examples/vhost_scsi: add virtio-vhost-user support

It was also necessary to tweak dpdk-devbind.py to support virtio-vhost-user:

  4aee6f653 usertools: add virtio-vhost-user devices to dpdk-devbind.py

Finally, vhost-scsi seems broken to me so two workarounds were needed so it can
be tested again:

  b9d17bfaf WORKAROUND revert virtio-net mq vring deletion
  cadb25e7d WORKAROUND examples/vhost_scsi: avoid broken EVENT_IDX

Stefan Hajnoczi (24):
  vhost: move vring_call() into trans_af_unix.c
  vhost: move AF_UNIX code from socket.c to trans_af_unix.c
  vhost: allocate per-socket transport state
  vhost: move socket_fd and un sockaddr into trans_af_unix.c
  vhost: move start_server/client() calls to trans_af_unix.c
  vhost: move vhost_user_connection to trans_af_unix.c
  vhost: move vhost_user_reconnect_init() into trans_af_unix.c
  vhost: move vhost_user.fdset to trans_af_unix.c
  vhost: pass vhost_transport_ops through vhost_new_device()
  vhost: embed struct virtio_net inside struct vhost_user_connection
  vhost: extract vhost_user.c socket I/O into transport
  vhost: move slave_req_fd field to AF_UNIX transport
  vhost: move mmap/munmap to AF_UNIX transport
  vhost: move librte_vhost to drivers/
  vhost: add virtio pci framework
  vhost: remember a vhost_virtqueue's queue index
  vhost: add virtio-vhost-user transport
  vhost: add RTE_VHOST_USER_VIRTIO_TRANSPORT flag
  net/vhost: add virtio-vhost-user support
  examples/vhost_scsi: add --socket-file argument
  examples/vhost_scsi: add virtio-vhost-user support
  usertools: add virtio-vhost-user devices to dpdk-devbind.py
  WORKAROUND revert virtio-net mq vring deletion
  WORKAROUND examples/vhost_scsi: avoid broken EVENT_IDX

 drivers/Makefile                                   |    2 +
 {lib => drivers}/librte_vhost/Makefile             |    5 +-
 lib/Makefile                                       |    3 -
 {lib => drivers}/librte_vhost/fd_man.h             |    0
 {lib => drivers}/librte_vhost/iotlb.h              |    0
 {lib => drivers}/librte_vhost/rte_vhost.h          |    1 +
 {lib => drivers}/librte_vhost/vhost.h              |  193 +++-
 {lib => drivers}/librte_vhost/vhost_user.h         |    9 +-
 drivers/librte_vhost/virtio_pci.h                  |  267 +++++
 drivers/librte_vhost/virtio_vhost_user.h           |   18 +
 drivers/librte_vhost/virtqueue.h                   |  181 ++++
 {lib => drivers}/librte_vhost/fd_man.c             |    0
 {lib => drivers}/librte_vhost/iotlb.c              |    0
 drivers/librte_vhost/socket.c                      |  282 ++++++
 drivers/librte_vhost/trans_af_unix.c               |  795 +++++++++++++++
 drivers/librte_vhost/trans_virtio_vhost_user.c     | 1050 ++++++++++++++++++++
 {lib => drivers}/librte_vhost/vhost.c              |   18 +-
 {lib => drivers}/librte_vhost/vhost_user.c         |  200 +---
 {lib => drivers}/librte_vhost/virtio_net.c         |    0
 drivers/librte_vhost/virtio_pci.c                  |  504 ++++++++++
 drivers/net/vhost/rte_eth_vhost.c                  |   13 +
 examples/vhost_scsi/vhost_scsi.c                   |  104 +-
 lib/librte_vhost/socket.c                          |  828 ---------------
 .../librte_vhost/rte_vhost_version.map             |    0
 usertools/dpdk-devbind.py                          |    8 +
 25 files changed, 3455 insertions(+), 1026 deletions(-)
 rename {lib => drivers}/librte_vhost/Makefile (86%)
 rename {lib => drivers}/librte_vhost/fd_man.h (100%)
 rename {lib => drivers}/librte_vhost/iotlb.h (100%)
 rename {lib => drivers}/librte_vhost/rte_vhost.h (99%)
 rename {lib => drivers}/librte_vhost/vhost.h (68%)
 rename {lib => drivers}/librte_vhost/vhost_user.h (93%)
 create mode 100644 drivers/librte_vhost/virtio_pci.h
 create mode 100644 drivers/librte_vhost/virtio_vhost_user.h
 create mode 100644 drivers/librte_vhost/virtqueue.h
 rename {lib => drivers}/librte_vhost/fd_man.c (100%)
 rename {lib => drivers}/librte_vhost/iotlb.c (100%)
 create mode 100644 drivers/librte_vhost/socket.c
 create mode 100644 drivers/librte_vhost/trans_af_unix.c
 create mode 100644 drivers/librte_vhost/trans_virtio_vhost_user.c
 rename {lib => drivers}/librte_vhost/vhost.c (97%)
 rename {lib => drivers}/librte_vhost/vhost_user.c (89%)
 rename {lib => drivers}/librte_vhost/virtio_net.c (100%)
 create mode 100644 drivers/librte_vhost/virtio_pci.c
 delete mode 100644 lib/librte_vhost/socket.c
 rename {lib => drivers}/librte_vhost/rte_vhost_version.map (100%)

-- 
2.14.3