[dpdk-dev] [RFC v1 0/2] Add device emulation support in DPDK

[Chenbo Xia <chenbo.xia@intel.com>]

This series enables DPDK to be an alternative I/O device emulation library of
building virtualized devices in separate processes outside QEMU. It introduces
a new library (librte_vfio_user), a new device class (emudev) and one pilot
device provider (avf_emudev) with its backend of Ethdev PMD (avfbe_ethdev).

*librte_vfio_user* is a server implementation of VFIO-over-socket[1] (also
known as vfio-user) which is a protocol that allows a device to be virtualized
in a separate process outside of QEMU. 

*emudev* is a device type for emulated devices. It is up to device provider to
choose the transport. In avf_emudev case, it uses vfio-user as transport
communicate with its client (e.g., QEMU).

*avf_emudev* is the emudev provider of AVF which is a device specification for
Intel Virtual Function cross generation. It’s implemented by an AVF emudev
driver which offers a few APIs for avfbe_ethdev or app logic to operate. 

*avfbe_ethdev* is a normal ethdev PMD to supply the basic I/O as backend data
path of avf_emudev. One simple usage of avfbe_ethdev could be a para-virtualized
backend connected with network application logic.

Background & Motivation
-----------------------
In order to reduce the attack surface, QEMU community is disaggregating QEMU by
removing part of device emulation from it. The disaggregated/multi-process QEMU
is using VFIO-over-socket/vfio-user as the main transport mechanism to disaggregate
I/O services from QEMU[2]. Vfio-user essentially implements the VFIO device model
presented to the user process by a set of messages over a unix-domain socket. The
main difference between application using vfio-user and application using vfio
kernel module is that device manipulation is based on socket messages for vfio-user
but system calls for vfio kernel module. The vfio-user devices consist of a generic
VFIO device type, living in QEMU, which is called the client[3], and the core device
implementation (emulated device), living outside of QEMU, which is called the server. 

With the introduction and support of vfio-user in QEMU, QEMU is explicitly adding
support for external emulated device and data path. We are trying to leverage that
and introducing vfio-user support in DPDK. By doing so, DPDK is enabled to be an
alternative I/O device emulation library of building virtualized devices along with
high-performance data path in separate processes outside QEMU. It will be easy for
hardware vendors to provide virtualized solutions of their hardware devices by
implementing emulated device in DPDK.

Except for vfio-user introduced in DPDK, this series also introduces the first
emulated device implementation. That is emulated AVF device (avf_emudev) implemented
by AVF emulation driver (avf_emudev driver). Emulated AVF device demos how emulated
device could be implemented in DPDK. SPDK is also investigating to implement use case
for NVMe. 

Design overview
---------------

                    +------------------------------------------------------+
                    |   +---------------+      +---------------+           |
                    |   |  avf_emudev   |      |  avfbe_ethdev |           |
                    |   |    driver     |      |     driver    |           |
                    |   +---------------+      +---------------+           |
                    |           |                       |                  |
                    | ------------------------------------------- VDEV BUS |
                    |           |                       |                  |
                    |   +---------------+       +--------------+           |
+--------------+    |   | vdev:         |       | vdev:        |           |
| +----------+ |    |   | /path/to/vfio |       | avf_emudev_# |           |
| | Generic  | |    |   +---------------+       +--------------+           |
| | vfio-dev | |    |           |                                          |
| +----------+ |    |           |                                          |
| +----------+ |    |      +----------+                                    |
| | vfio-user| |    |      | vfio-user|                                    |
| | client   | |<---|----->| server   |                                    |
| +----------+ |    |      +----------+                                    |
| QEMU         |    | DPDK                                                 |
+--------------+    +------------------------------------------------------+

- vfio-user. Vfio-user in DPDK is referred to the vfio-user protocol implementation
playing server role. It provides transport between emulated device and generic VFIO
device in QEMU. Emulated device in DPDK and generic VFIO device in QEMU are working
together to present VFIO device model to VM. This series introduces vfio-user
implementation as a library called librte_vfio_user which is under lib/librte_vfio_user.

- vdev:/path/to/vfio. It defines the emudev device and binds to vdev bus driver. The
emudev device is defined by DPDK applications through command line as '--vdev=emu_iavf,
path=/path/to/socket, id=#' in avf_emudev case. Parameters in command line include device
name (emu_iavf) which is used to identify corresponding driver (in this case, avf_emudev
driver which implements emudev device of AVF), path=/path/to/socket which is used to open
the transport interface to vfio-user client in QEMU, and id which is the index of emudev
device.

- avf_emudev driver. It implements emulated AVF device which is the emudev provider of
AVF. The avf_emudev_driver offers a few APIs implementation exposed by emudev device APIs
for avfbe_ethdev_pmd or application logic to operate. These APIs are described in
lib/librte_emudev/rte_emudev.h.

- vdev: avf_emudev_#. The vdev device is defined by DPDK application through command line
as '--vdev=net_avfbe,id=#,avf_emu_id=#'.It is associated with emudev provider of AVF by
'avf_emu_id=#'.
     
- avfbe_ethdev driver. It is a normal ethdev PMD to supply the basic I/O as backend data
path of avf_emudev.

Why not rawdev for emulated device
----------------------------------
Instead of introducing new class emudev, emulated device could be presented as rawdev.
However, existing rawdev APIs cannot meet the requirements of emulated device. There are
three API categories for emudev. They are emudev device lifecycle management, backend
facing APIs, and emudev device provider facing APIs respectively. Existing rawdev APIs
could only cover lifecycle management APIs and some of backend facing APIs. Other APIs,
even if added to rawdev API are not required by other rawdev applications.
 
References
----------
[1]: https://patchew.org/QEMU/1594913503-52271-1-git-send-email-thanos.makatos@nutanix.com/
[2]: https://wiki.qemu.org/Features/MultiProcessQEMU
[3]: https://github.com/elmarco/qemu/blob/wip/vfio-user/hw/vfio/libvfio-user.c

Chenbo Xia (2):
  vfio_user: Add library for vfio over socket
  emudev: Add library for emulated device

 lib/librte_emudev/rte_emudev.h       | 315 +++++++++++++++++++++++++
 lib/librte_vfio_user/rte_vfio_user.h | 335 +++++++++++++++++++++++++++
 2 files changed, 650 insertions(+)
 create mode 100644 lib/librte_emudev/rte_emudev.h
 create mode 100644 lib/librte_vfio_user/rte_vfio_user.h

-- 
2.17.1