Re: [dpdk-dev] [RFC 0/4] SocketPair Broker support for vhost and virtio-user.

[Stephen Hemminger <stephen@networkplumber.org>]

On Wed, 17 Mar 2021 21:25:26 +0100
Ilya Maximets <i.maximets@ovn.org> wrote:

> TL;DR;
>   Managing socket files is too much fun. :)  And here is how this
>   could be improved:
>     https://github.com/igsilya/one-socket
>     https://github.com/igsilya/one-socket/blob/main/doc/socketpair-broker.rst
>   In particular for vhost-user case.
> 
> In modern virtualization setups there are tens or hundreds of different
> socket files for different purposes.  Sockets to manage various
> daemons, vhost-user sockets for various virtual devices, memif sockets
> for memif network interfaces and so on.
> 
> In order to make things work in containerized environments software
> systems has to share these sockets with containers.  In most cases
> this sharing is implemented as a shared directory mounted inside the
> container, because socket files could be re-created in runtime or even
> not be available at the container startup.  For example, if they are
> created by the application inside the container.
> 
> Even more configuration tricks required in order to share some sockets
> between different containers and not only with the host, e.g. to
> create service chains.
> And some housekeeping usually required for applications in case the
> socket server terminated abnormally and socket files left on a file
> system:
>  "failed to bind to vhu: Address already in use; remove it and try again"
> 
> Additionally, all applications (system and user's!) should follow
> naming conventions and place socket files in particular location on a
> file system to make things work.
> 
> In particular, this applies to vhost-user sockets.
> 
> This patch-set aims to eliminate most of the inconveniences by
> leveraging an infrastructure service provided by a SocketPair Broker.
> 
> *SocketPair Broker* is a daemon that mediates establishment of direct
> socket-based connections between clients.
> 
> *One Socket* is a reference implementation of a SocketPair Broker
> Daemon, SocketPair Broker Protocol and a helper library for client
> applications (libspbroker):
> 
>   https://github.com/igsilya/one-socket
> 
> It's fully functional, but not completely ready for production use
> for now.  See 'todo' section in README.rst in one-socket repository.
> 
> Basically, it's a daemon that listens on a single unix socket
> (broker socket) and accepts clients.  Client connects and provides a
> 'key'.  If two clients provided the same 'key', One Socket daemon
> creates a pair of connected sockets with socketpair() and sends
> sides of this pair to these two clients.  At this point two clients
> have a direct communication channel between them.  They will disconnect
> from the broker and continue to operate and communicate normally.
> 
> Workflow overview with pictures available here:
> 
>   https://github.com/igsilya/one-socket/blob/main/doc/socketpair-broker.rst
> 
> Communication with a broker based on a SocketPair Broker Protocol:
> 
>   https://github.com/igsilya/one-socket/blob/main/doc/socketpair-broker-proto-spec.rst
> 
> 
> This patch-set extends vhost library, vhost pmd and virtio-user pmd to
> support SocketPair Broker as one of the connection methods.
> Usage example:
> 
>   # Starting a One Socket daemon with socket './one.socket':
>   $ ONE_SOCKET_PATH=./one.socket ./one-socket
> 
>   # Starting testpmd #1 with virtio-user device in server mode:
>   $ dpdk-testpmd --no-pci --in-memory --single-file-segments \
>       --vdev="net_virtio_user,path=./one.socket,broker-key=MY-KEY,server=1"
> 
>   # Starting testpmd #2 with vhost pmd in client mode:
>   $ dpdk-testpmd --no-pci --in-memory --single-file-segments \
>       --vdev="eth_vhost0,iface=./one.socket,broker-key=MY-KEY,client=1"
> 
> Details how to build and install One Socket are in README.rst in
> one-socket repository.
> 
> DPDK side is the first step of implementation.  Once available in DPDK,
> support could be easily added to Open vSwith or VPP or any DPDK-based
> application.  Same support could be added to QEMU (found a volunteer
> for this part).
> 
> Since SocketPair Broker is completely independent from the purposes
> connection will be used for, it has a potential to unify and replace
> all one-to-one unix socket connections on a host.  This one persistent
> broker socket could be passed to any containers and can be used by
> any application greatly simplifying system management.
> 
> Any feedback or suggestions on any component of this solution including
> this patch-set, One Socket Daemon, SocketPair Broker Protocol or
> libspbroker library are very welcome.
> 
> *Note* about the patch set:
> 
> First patch in a series is a *bug* fix, so it should be considered even
> outside of this series.  It basically fixes unregistering of a
> listening socket that never happens in current code.
> 
> The virtio-user part of the series heavily depends on this bug fix
> since broker connection unlike listening socket will not persist and
> will generate lots of interrupts if not unregistered.
> 
> Ilya Maximets (4):
>   net/virtio: fix interrupt unregistering for listening socket
>   vhost: add support for SocketPair Broker
>   net/vhost: add support for SocketPair Broker
>   net/virtio: add support for SocketPair Broker
> 
>  doc/guides/nics/vhost.rst                     |   5 +
>  doc/guides/nics/virtio.rst                    |   5 +
>  doc/guides/prog_guide/vhost_lib.rst           |  10 +
>  drivers/net/vhost/rte_eth_vhost.c             |  42 ++-
>  drivers/net/virtio/meson.build                |   6 +
>  drivers/net/virtio/virtio_user/vhost_user.c   | 122 ++++++++-
>  .../net/virtio/virtio_user/virtio_user_dev.c  | 142 +++++++---
>  .../net/virtio/virtio_user/virtio_user_dev.h  |   6 +-
>  drivers/net/virtio/virtio_user_ethdev.c       |  30 ++-
>  lib/librte_vhost/meson.build                  |   7 +
>  lib/librte_vhost/rte_vhost.h                  |   1 +
>  lib/librte_vhost/socket.c                     | 245 ++++++++++++++++--
>  12 files changed, 550 insertions(+), 71 deletions(-)
> 

Adding yet another dependency in DPDK for something that really should
be part of the OS (systemd?) seems like something that is not going
to be widely adopted or appreciated.

If this is a real problem, this should be solved outside of DPDK
with a common infrastructure available in almost all distros.