Re: [dpdk-dev] [PATCH 0/4] Support DMA-accelerated Tx operations for vhost-user PMD

[Maxime Coquelin <maxime.coquelin@redhat.com>]

Hi Jiayu,

On 3/17/20 10:21 AM, Jiayu Hu wrote:
> In vhost-user PMD's Tx operations, where data movement is heavily involved,
> performing large memory copies usually takes up a major part of CPU cycles
> and becomes the hot spot. To offload expensive memory operations from the
> CPU, this patch set proposes to leverage DMA engines, e.g., I/OAT, a DMA
> engine in the Intel's processor, to accelerate large copies for vhost-user.
> 
> Large copies are offloaded from the CPU to the DMA in an asynchronous
> manner. The CPU just submits copy jobs to the DMA but without waiting
> for its copy completion. Thus, there is no CPU intervention during data
> transfer; we can save precious CPU cycles and improve the overall
> throughput for vhost-user PMD based applications, like OVS. During
> packet transmission, it offloads large copies to the DMA and performs
> small copies by the CPU, due to startup overheads associated with the DMA.
> 
> vhost-user PMD is able to support various DMA engines, but it just
> supports I/OAT devices currently. In addition, I/OAT acceleration is only
> enabled for Tx operations of split rings. Users can explicitly assign a
> I/OAT device to a queue by the parameter 'dmas'. However, one I/OAT device
> can only be used by one queue, and a queue can use one I/OAT device at a
> time.
> 
> We measure the performance in testpmd. With 1024 bytes packets, compared
> with the original SW data path, DMA-enabled vhost-user PMD can improve
> the throughput around 20%~30% in the VM2VM and PVP cases. Furthermore,
> with larger packets, the throughput improvement will be higher.


I'm not sure it should be done like that for several reasons.

First, it seems really complex for the user to get the command line
right. There is no mention in the doc patch on how to bind the DMAs to
the DPDK application. Are all the DMAs on the system capable of doing
it?
I think it should be made transparent to the user, who should not have
to specify the DMA device address in command line. The user should just
pass a devarg specifying he wants to use DMAs, if available.

Second, it looks too much vendor-specific. IMHO, we should have a DMA
framework, so that the driver can request DMA channels based on
capabilities.

Also, I don't think implementing ring processing in the Vhost PMD is
welcome, Vhost PMD should just be a wrapper for the Vhost library. Doing
that in Vhost PMD causes code duplication, and will be a maintenance
burden on the long run.

As IOAT is a kind of acceleration, why not implement it through the vDPA
framework? vDPA framework should be extended to support this kind of
acceleration which requires some CPU processing, as opposed to full
offload of the ring processing as it only supports today.

Kind regards,
Maxime

> Jiayu Hu (4):
>   vhost: populate guest memory for DMA-accelerated vhost-user
>   net/vhost: setup vrings for DMA-accelerated datapath
>   net/vhost: leverage DMA engines to accelerate Tx operations
>   doc: add I/OAT acceleration support for vhost-user PMD
> 
>  doc/guides/nics/vhost.rst         |  14 +
>  drivers/Makefile                  |   2 +-
>  drivers/net/vhost/Makefile        |   6 +-
>  drivers/net/vhost/internal.h      | 160 +++++++
>  drivers/net/vhost/meson.build     |   5 +-
>  drivers/net/vhost/rte_eth_vhost.c | 308 +++++++++++---
>  drivers/net/vhost/virtio_net.c    | 861 ++++++++++++++++++++++++++++++++++++++
>  drivers/net/vhost/virtio_net.h    | 288 +++++++++++++
>  lib/librte_vhost/rte_vhost.h      |   1 +
>  lib/librte_vhost/socket.c         |  20 +
>  lib/librte_vhost/vhost.h          |   2 +
>  lib/librte_vhost/vhost_user.c     |   3 +-
>  12 files changed, 1597 insertions(+), 73 deletions(-)
>  create mode 100644 drivers/net/vhost/internal.h
>  create mode 100644 drivers/net/vhost/virtio_net.c
>  create mode 100644 drivers/net/vhost/virtio_net.h
>