Re: [dpdk-dev] [RFC PATCH 0/2] Virtio-net PMD Extension to work on host.

[Rich Lane <rich.lane@bigswitch.com>]

What's the reason for using qemu as a middleman? Couldn't the new PMD
itself open /dev/vhost-net or the vhost-user socket and send the commands
to set up virtqueues? That was the approach taken by Jianfeng's earlier RFC.

On Thu, Nov 19, 2015 at 2:57 AM, Tetsuya Mukawa <mukawa@igel.co.jp> wrote:

> THIS IS A PoC IMPLEMENATION.
>
> [Abstraction]
>
> Normally, virtio-net PMD only works on VM, because there is no virtio-net
> device on host.
> This RFC patch extends virtio-net PMD to be able to work on host as
> virtual PMD.
> But we didn't implement virtio-net device as a part of virtio-net PMD.
> To prepare virtio-net device for the PMD, start QEMU process with special
> QTest mode, then connect it from virtio-net PMD through unix domain socket.
>
> The PMD can connect to anywhere QEMU virtio-net device can.
> For example, the PMD can connects to vhost-net kernel module and
> vhost-user backend application.
> Similar to virtio-net PMD on QEMU, application memory that uses virtio-net
> PMD will be shared between vhost backend application.
> But vhost backend application memory will not be shared.
>
> Main target of this PMD is container like docker, rkt, lxc and etc.
> We can isolate related processes(virtio-net PMD process, QEMU and
> vhost-user backend process) by container.
> But, to communicate through unix domain socket, shared directory will be
> needed.
>
>
> [How to use]
>
> So far, we need QEMU patch to connect to vhost-user backend.
> Please check known issue in later section.
> Because of this, I will describe example of using vhost-net kernel module.
>
>  - Compile
>  Set "CONFIG_RTE_VIRTIO_VDEV=y" in config/common_linux.
>  Then compile it.
>
>  - Start QEMU like below.
>  $ sudo qemu-system-x86_64 -qtest unix:/tmp/qtest0,server -machine
> accel=qtest \
>                            -display none -qtest-log /dev/null \
>                            -netdev
> type=tap,script=/etc/qemu-ifup,id=net0,vhost=on \
>                            -device virtio-net-pci,netdev=net0 \
>                            -chardev
> socket,id=chr1,path=/tmp/ivshmem0,server \
>                            -device ivshmem,size=1G,chardev=chr1,vectors=1
>
>  - Start DPDK application like below
>  $ sudo ./testpmd -c f -n 1 -m 1024 --shm \
>
> --vdev="eth_cvio0,qtest=/tmp/qtest0,ivshmem=/tmp/ivshmem0" -- \
>                   --disable-hw-vlan --txqflags=0xf00 -i
>
>  - Check created tap device.
>
> (*1) Please Specify same memory size in QEMU and DPDK command line.
>
>
> [Detailed Description]
>
>  - virtio-net device implementation
> The PMD uses QEMU virtio-net device. To do that, QEMU QTest functionality
> is used.
> QTest is a test framework of QEMU devices. It allows us to implement a
> device driver outside of QEMU.
> With QTest, we can implement DPDK application and virtio-net PMD as
> standalone process on host.
> When QEMU is invoked as QTest mode, any guest code will not run.
> To know more about QTest, see below.
> http://wiki.qemu.org/Features/QTest
>
>  - probing devices
> QTest provides a unix domain socket. Through this socket, driver process
> can access to I/O port and memory of QEMU virtual machine.
> The PMD will send I/O port accesses to probe pci devices.
> If we can find virtio-net and ivshmem device, initialize the devices.
> Also, I/O port accesses of virtio-net PMD will be sent through socket, and
> virtio-net PMD can initialize vitio-net device on QEMU correctly.
>
>  - ivshmem device to share memory
> To share memory that virtio-net PMD process uses, ivshmem device will be
> used.
> Because ivshmem device can only handle one file descriptor, shared memory
> should be consist of one file.
> To allocate such a memory, EAL has new option called "--shm".
> If the option is specified, EAL will open a file and allocate memory from
> hugepages.
> While initializing ivshmem device, we can set BAR(Base Address Register).
> It represents which memory QEMU vcpu can access to this shared memory.
> We will specify host physical address of shared memory as this address.
> It is very useful because we don't need to apply patch to QEMU to
> calculate address offset.
> (For example, if virtio-net PMD process will allocate memory from shared
> memory, then specify the physical address of it to virtio-net register,
> QEMU virtio-net device can understand it without calculating address
> offset.)
>
>  - Known limitation
> So far, the PMD doesn't handle interrupts from QEMU devices.
> Because of this, VIRTIO_NET_F_STATUS functionality is dropped.
> But without it, we can use all virtio-net functions.
>
>  - Known issues
> So far, to use vhost-user, we need to apply vhost-user patch to QEMU and
> DPDK vhost library.
> This is because, QEMU will not send memory information and file descriptor
> of ivshmem device to vhost-user backend.
> (Anyway, vhost-net kernel module can receive the information. So
> vhost-user behavior will not be correct. I will submit the patch to QEMU
> soon)
> Also, we may have an issue in DPDK vhost library to handle kickfd and
> callfd. The patch for it is needed.
> (Let me check it more)
> If someone wants to check vhost-user behavior, I will describe it more in
> later email.
>
>
> [Addition]
>
> We can apply same manner to handle any kind of QEMU devices from DPDK
> application.
> So far, I don't have any ideas except for virtio-net device. But someone
> would have.
>
>
> Tetsuya Mukawa (2):
>   EAL: Add new EAL "--shm" option.
>   virtio: Extend virtio-net PMD to support container environment
>
>  config/common_linuxapp                     |   5 +
>  drivers/net/virtio/Makefile                |   4 +
>  drivers/net/virtio/qtest.c                 | 590
> +++++++++++++++++++++++++++++
>  drivers/net/virtio/virtio_ethdev.c         | 214 ++++++++++-
>  drivers/net/virtio/virtio_ethdev.h         |  16 +
>  drivers/net/virtio/virtio_pci.h            |  25 ++
>  lib/librte_eal/common/eal_common_options.c |   5 +
>  lib/librte_eal/common/eal_internal_cfg.h   |   1 +
>  lib/librte_eal/common/eal_options.h        |   2 +
>  lib/librte_eal/common/include/rte_memory.h |   5 +
>  lib/librte_eal/linuxapp/eal/eal_memory.c   |  71 ++++
>  11 files changed, 917 insertions(+), 21 deletions(-)
>  create mode 100644 drivers/net/virtio/qtest.c
>
> --
> 2.1.4
>
>