Re: [RFC 00/27] Add VDUSE support to Vhost library

[Maxime Coquelin <maxime.coquelin@redhat.com>]

Hi Yongji,

On 4/6/23 05:44, Yongji Xie wrote:
> Hi Maxime,
> 
> On Fri, Mar 31, 2023 at 11:43 PM Maxime Coquelin
> <maxime.coquelin@redhat.com> wrote:
>>
>> This series introduces a new type of backend, VDUSE,
>> to the Vhost library.
>>
>> VDUSE stands for vDPA device in Userspace, it enables
>> implementing a Virtio device in userspace and have it
>> attached to the Kernel vDPA bus.
>>
>> Once attached to the vDPA bus, it can be used either by
>> Kernel Virtio drivers, like virtio-net in our case, via
>> the virtio-vdpa driver. Doing that, the device is visible
>> to the Kernel networking stack and is exposed to userspace
>> as a regular netdev.
>>
>> It can also be exposed to userspace thanks to the
>> vhost-vdpa driver, via a vhost-vdpa chardev that can be
>> passed to QEMU or Virtio-user PMD.
>>
>> While VDUSE support is already available in upstream
>> Kernel, a couple of patches are required to support
>> network device type:
>>
>> https://gitlab.com/mcoquelin/linux/-/tree/vduse_networking_poc
>>
>> In order to attach the created VDUSE device to the vDPA
>> bus, a recent iproute2 version containing the vdpa tool is
>> required.
>>
>> Usage:
>> ======
>>
>> 1. Probe required Kernel modules
>> # modprobe vdpa
>> # modprobe vduse
>> # modprobe virtio-vdpa
>>
>> 2. Build (require vduse kernel headers to be available)
>> # meson build
>> # ninja -C build
>>
>> 3. Create a VDUSE device (vduse0) using Vhost PMD with
>> testpmd (with 4 queue pairs in this example)
>> # ./build/app/dpdk-testpmd --no-pci --vdev=net_vhost0,iface=/dev/vduse/vduse0,queues=4 --log-level=*:9  -- -i --txq=4 --rxq=4
>>
>> 4. Attach the VDUSE device to the vDPA bus
>> # vdpa dev add name vduse0 mgmtdev vduse
>> => The virtio-net netdev shows up (eth0 here)
>> # ip l show eth0
>> 21: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
>>      link/ether c2:73:ea:a7:68:6d brd ff:ff:ff:ff:ff:ff
>>
>> 5. Start/stop traffic in testpmd
>> testpmd> start
>> testpmd> show port stats 0
>>    ######################## NIC statistics for port 0  ########################
>>    RX-packets: 11         RX-missed: 0          RX-bytes:  1482
>>    RX-errors: 0
>>    RX-nombuf:  0
>>    TX-packets: 1          TX-errors: 0          TX-bytes:  62
>>
>>    Throughput (since last show)
>>    Rx-pps:            0          Rx-bps:            0
>>    Tx-pps:            0          Tx-bps:            0
>>    ############################################################################
>> testpmd> stop
>>
>> 6. Detach the VDUSE device from the vDPA bus
>> # vdpa dev del vduse0
>>
>> 7. Quit testpmd
>> testpmd> quit
>>
>> Known issues & remaining work:
>> ==============================
>> - Fix issue in FD manager (still polling while FD has been removed)
>> - Add Netlink support in Vhost library
>> - Support device reconnection
>> - Support packed ring
>> - Enable & test more Virtio features
>> - Provide performance benchmark results
>>
> 
> Nice work! Thanks for bringing VDUSE to the network area. I wonder if
> you have some plan to support userspace memory registration [1]? I
> think this feature can benefit the performance since an extra data
> copy could be eliminated in our case.

I plan to have a closer look later, once VDUSE support will be added.
I think it will be difficult to support it in the case of DPDK for
networking:

  - For dequeue path it would be basically re-introducing dequeue zero-
copy support that we removed some time ago. It was a hack where we
replaced the regular mbuf buffer with the descriptor one, increased the
reference counter, and at next dequeue API calls checked if the former
mbufs ref counter is 1 and restore the mbuf. Issue is that physical NIC
drivers usually release sent mbufs by pool, once a certain threshold is
met. So it can cause draining of the virtqueue as the descs are not
written back into the used ring for quite some time, depending on the
NIC/traffic/...

- For enqueue path, I don't think this is possible with virtual switches
by design, as when a mbuf is received on a physical port, we don't know
in which Vhost/VDUSE port it will be switched to. And for VM to VM
communication, should it use the src VM buffer or the dest VM one?

Only case it could work is if you had a simple forwarder between a VDUSE
device and a physical port. But I don't think there is much interest in
such use-case.

Any thoughts?

Thanks,
Maxime

> [1] https://lwn.net/Articles/902809/
> 
> Thanks,
> Yongji
>