Re: [dpdk-dev] [PATCH v7 00/19] vhost: add postcopy live-migration support

[Maxime Coquelin <maxime.coquelin@redhat.com>]

On 10/12/2018 02:40 PM, Maxime Coquelin wrote:
> In this v7:
> - Move vh_result enum declaration to vhost.h (Ilya)
> - Remove check in read_fd_message() for more fds received
>    than requested as already checked with MSG_CTRUNC (Ilya)
> - Don't return an error in vhost-crypto post message handler
>    if not a crypto message as all messages go through it.
> 
> With classic live-migration, the VM runs on source while its
> content is being migrated to destination. When pages already
> migrated to destination are dirtied by the source, they get
> copied until both source and destination memory converge.
> At that time, the source is stopped and destination is
> started.
> 
> With postcopy live-migration, the VM is started on destination
> before all the memory has been migrated. When the VM tries to
> access a page that haven't been migrated yet, a pagefault is
> triggered, handled by userfaultfd which pauses the thread.
> A Qemu thread in charge of postcopy request the source for
> the missing page. Once received and mapped, the paused thread
> gets resumed.
> 
> Userfaultfd supports handling faults from a different process,
> and Qemu supports postcopy with vhost-user backends since
> v2.12.
> 
> One problem encountered with classic live-migration for VMs
> relying on vhost-user backends is that when the traffic is
> high (e.g. PVP), it happens that it never converges as
> pages gets dirtied at a faster rate than they are copied
> to the destination.
> It is expected this problem sould be solved with using
> postcopy, as rings memory and buffers will be copied once,
> when destination will pagefault on them.
> 
> Note that it will certainly require a rebase to apply on top
> of Nikolay's vhost-user message handling rework.
> 
> Steps to test postcopy:
> 1. Run DPDK's Testpmd application on source:
> ./install/bin/testpmd -m 512 --file-prefix=src -l 0,2 -n 4 \
>    --vdev 'net_vhost0,iface=/tmp/vu-src' -- --portmask=1 -i \
>    --rxq=1 --txq=1 --nb-cores=1 --eth-peer=0,52:54:00:11:22:12 \
>    --no-mlockall
> 
> 2. Run DPDK's Testpmd application on destination:
> ./install/bin/testpmd -m 512 --file-prefix=dst -l 0,2 -n 4 \
>    --vdev 'net_vhost0,iface=/tmp/vu-dst,postcopy-support=1' -- --portmask=1 -i \
>    --rxq=1 --txq=1 --nb-cores=1 --eth-peer=0,52:54:00:11:22:12 \
>    --no-mlockall
> 
> 3. Launch VM on source:
> ./x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 3G -smp 2 -cpu host \
>    -object memory-backend-file,id=mem,size=3G,mem-path=/dev/shm,share=on \
>    -numa node,memdev=mem -mem-prealloc \
>    -chardev socket,id=char0,path=/tmp/vu-src \
>    -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
>    -device virtio-net-pci,netdev=mynet1 /home/virt/rhel7.6-1-clone.qcow2 \
>    -net none -vnc :0 -monitor stdio
> 
> 4. Launch VM on destination:
> ./x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 3G -smp 2 -cpu host \
>    -object memory-backend-file,id=mem,size=3G,mem-path=/dev/shm,share=on \
>    -numa node,memdev=mem -mem-prealloc \
>    -chardev socket,id=char0,path=/tmp/vu-dst \
>    -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
>    -device virtio-net-pci,netdev=mynet1 /home/virt/rhel7.6-1-clone.qcow2 \
>    -net none -vnc :1 -monitor stdio -incoming tcp::8888
> 
> 5. In both testpmd prompts, start flooding the virtio-net device:
> testpmd> set fwd txonly
> testpmd> start
> 
> 6. In destination's Qemu monitor, enable postcopy:
> (qemu) migrate_set_capability postcopy-ram on
> 
> 7. In source's Qemu monitor, enable postcopy and launch migration:
> (qemu) migrate_set_capability postcopy-ram on
> (qemu) migrate -d tcp:0:8888
> (qemu) migrate_start_postcopy
> 
> Maxime Coquelin (19):
>    vhost: fix messages results handling
>    vhost: fix return code of messages requiring replies
>    vhost: clarify reply-ack in case a reply was already sent
>    vhost: fix payload size of reply
>    vhost: fix error handling when mem table gets updated
>    vhost: define postcopy protocol flag
>    vhost: add number of fds to vhost-user messages and use it
>    vhost: pass socket fd to message handling callbacks
>    vhost: enable fds passing when sending vhost-user messages
>    vhost: add config flag for postcopy feature
>    vhost: introduce postcopy's advise message
>    vhost: add support for postcopy's listen message
>    vhost: register new regions with userfaultfd
>    vhost: avoid useless VhostUserMemory copy
>    vhost: send userfault range addresses back to qemu
>    vhost: add support to postcopy's end request
>    vhost: restrict postcopy live-migration enablement
>    net/vhost: add parameter to enable postcopy support
>    vhost: enable postcopy protocol feature
> 
>   config/common_linuxapp              |   1 +
>   doc/guides/nics/vhost.rst           |   5 +
>   doc/guides/prog_guide/vhost_lib.rst |   8 +
>   drivers/net/vhost/rte_eth_vhost.c   |  13 ++
>   lib/librte_vhost/meson.build        |   2 +
>   lib/librte_vhost/rte_vhost.h        |   5 +
>   lib/librte_vhost/socket.c           |  46 +++-
>   lib/librte_vhost/vhost.h            |  32 ++-
>   lib/librte_vhost/vhost_crypto.c     |  25 +-
>   lib/librte_vhost/vhost_user.c       | 344 +++++++++++++++++++++-------
>   lib/librte_vhost/vhost_user.h       |  12 +-
>   11 files changed, 377 insertions(+), 116 deletions(-)
> 

Applied to dpdk-next-virtio/master.

Maxime