From: Nikos Dragazis <ndragazis@arrikto.com>
To: Maxime Coquelin <maxime.coquelin@redhat.com>
Cc: dev@dpdk.org, Tiwei Bie <tiwei.bie@intel.com>,
Zhihong Wang <zhihong.wang@intel.com>,
Stefan Hajnoczi <stefanha@redhat.com>,
Wei Wang <wei.w.wang@intel.com>,
Stojaczyk Dariusz <dariusz.stojaczyk@intel.com>,
Vangelis Koukis <vkoukis@arrikto.com>
Subject: Re: [dpdk-dev] [PATCH 00/28] vhost: add virtio-vhost-user transport
Date: Sat, 22 Jun 2019 23:26:31 +0300 [thread overview]
Message-ID: <aa302511-0f31-ae2d-b2cd-f7c87573ec65@arrikto.com> (raw)
In-Reply-To: <d9a8eb71-b417-342b-7afb-5063b807cafb@redhat.com>
On 20/6/19 2:35 μ.μ., Maxime Coquelin wrote:
> Hi Nikos,
>
> On 6/19/19 5:14 PM, Nikos Dragazis wrote:
>> Hi everyone,
>>
>> this patch series introduces the concept of the virtio-vhost-user
>> transport. This is actually a revised version of an earlier RFC
>> implementation that has been proposed by Stefan Hajnoczi [1]. Though
>> this is a great feature, it seems to have been stalled, so I’d like to
>> restart the conversation on this and hopefully get it merged with your
>> help. Let me give you an overview.
>
> Thanks for taking over the series!
>
> I think you are already aware of that, but it arrives too late to
> consider it for v19.08, as the proposal deadline is over by almost 3
> weeks.
>
> That said, it is good that you sent it early, so that we can work to
> make it in for v19.11.
That's totally fine.
>
>> The virtio-vhost-user transport is a vhost-user transport implementation
>> that is based on the virtio-vhost-user device. Its key difference with
>> the existing transport is that it allows deploying vhost-user targets
>> inside dedicated Storage Appliance VMs instead of host user space. In
>> other words, it allows having guests that act as vhost-user backends for
>> other guests.
>>
>> The virtio-vhost-user device implements the vhost-user control plane
>> (master-slave communication) as follows:
>>
>> 1. it parses the vhost-user messages from the vhost-user unix domain
>> socket and forwards them to the slave guest through virtqueues
>>
>> 2. it maps the vhost memory regions in QEMU’s process address space and
>> exposes them to the slave guest as a RAM-backed PCI MMIO region
>>
>> 3. it hooks up doorbells to the callfds. The slave guest can use these
>> doorbells to interrupt the master guest driver
>>
>> The device code has not yet been merged into upstream QEMU, but this is
>> definitely the end goal.
>
> Could you provide a pointer to the QEMU series, and instructions to test
> this new device?
Of course. Please have a look at the following step-by-step guide:
https://github.com/ndragazis/ndragazis.github.io/blob/master/dpdk-vhost-vvu-demo.md
If you face any problems or you find that something breaks (I hope not
:) ) please let me know. I haven't done thorough testing so I may have
missed something. Please, also bear in mind that the device code is not
finished. There are still some things that need to be done.
>
>> The current state is that we are awaiting for
>> the approval of the virtio spec.
>
> Ditto, a link to the spec patches would be useful.
You can find the patches here:
https://lists.oasis-open.org/archives/virtio-dev/201905/msg00022.html
and an HTML version is available here:
https://ndragazis.github.io/virtio-v1.1-wd02.html#x1-41000011
>
>> I have Cced Darek from the SPDK community who has helped me a lot by
>> reviewing this series. Note that any device type could be implemented
>> over this new transport. So, adding the virtio-vhost-user transport in
>> DPDK would allow using it from SPDK as well.
>>
>> Getting into the code internals, this patch series makes the following
>> changes:
>>
>> 1. introduce a generic interface for the transport-specific operations.
>> Each of the two available transports, the pre-existing AF_UNIX
>> transport and the virtio-vhost-user transport, is going to implement
>> this interface. The AF_UNIX-specific code has been extracted from the
>> core vhost-user code and is now part of the AF_UNIX transport
>> implementation in trans_af_unix.c.
>>
>> 2. introduce the virtio-vhost-user transport. The virtio-vhost-user
>> transport requires a driver for the virtio-vhost-user devices. The
>> driver along with the transport implementation have been packed into
>> a separate library in `drivers/virtio_vhost_user/`. The necessary
>> virtio-pci code has been copied from `drivers/net/virtio/`. Some
>> additional changes have been made so that the driver can utilize the
>> additional resources of the virtio-vhost-user device.
>>
>> 3. update librte_vhost public API to enable choosing transport for each
>> new vhost device. Extend the vhost net driver and vhost-scsi example
>> application to export this new API to the end user.
>>
>> The primary changes I did to Stefan’s RFC implementation are the
>> following:
>>
>> 1. moved postcopy live migration code into trans_af_unix.c. Postcopy
>> live migration relies on the userfault fd mechanism, which cannot be
>> supported by virtio-vhost-user.
>>
>> 2. moved setup of the log memory region into trans_af_unix.c. Setting up
>> the log memory region involves mapping/unmapping guest memory. This
>> is an AF_UNIX transport-specific operation.
>>
>> 3. introduced a vhost transport operation for
>> process_slave_message_reply()
>>
>> 4. moved the virtio-vhost-user transport/driver into a separate library
>> in `drivers/virtio_vhost_user/`. This required making vhost.h and
>> vhost_user.h part of librte_vhost public API and exporting some
>> private symbols via the version script. This looks better to me that
>> just moving the entire librte_vhost into `drivers/`. I am not sure if
>> this is the most appropriate solution. I am looking forward to your
>> suggestions on this.
>
> I'm not sure this is the right place to put it.
Okay. Is there something specific that you think that doesn't fit
nicely? Do you have something else in mind?
>
>> 5. made use of the virtio PCI capabilities for the additional device
>> resources (doorbells, shared memory). This required changes in
>> virtio_pci.c and trans_virtio_vhost_user.c.
>>
>> 6. [minor] changed some commit headlines to comply with
>> check-git-log.sh.
>>
>> Please, have a look and let me know about your thoughts. Any
>> reviews/pointers/suggestions are welcome.
>
> Maxime
prev parent reply other threads:[~2019-06-22 20:26 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-06-19 15:14 Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 01/28] vhost: introduce vhost transport operations structure Nikos Dragazis
2019-06-19 20:14 ` Aaron Conole
2019-06-20 10:30 ` Bruce Richardson
2019-06-20 18:24 ` Nikos Dragazis
2019-06-20 18:19 ` Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 02/28] vhost: move socket management code Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 03/28] vhost: allocate per-socket transport state Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 04/28] vhost: move socket fd and un sockaddr Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 05/28] vhost: move start server/client calls Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 06/28] vhost: move vhost-user connection Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 07/28] vhost: move vhost-user reconnection Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 08/28] vhost: move vhost-user fdset Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 09/28] vhost: propagate vhost transport operations Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 10/28] vhost: use a single structure for the device state Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 11/28] vhost: extract socket I/O into transport Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 12/28] vhost: move slave request fd and lock Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 13/28] vhost: move mmap/munmap Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 14/28] vhost: move setup of the log memory region Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 15/28] vhost: remove main fd parameter from msg handlers Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 16/28] vhost: move postcopy live migration code Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 17/28] vhost: support registering additional vhost-user transports Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 18/28] drivers/virtio_vhost_user: add virtio PCI framework Nikos Dragazis
2019-09-05 16:34 ` Maxime Coquelin
2019-09-09 8:42 ` Nikos Dragazis
2019-09-09 8:44 ` Maxime Coquelin
2019-06-19 15:14 ` [dpdk-dev] [PATCH 19/28] vhost: add index field in vhost virtqueues Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 20/28] drivers: add virtio-vhost-user transport Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 21/28] drivers/virtio_vhost_user: use additional device resources Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 22/28] vhost: add flag for choosing vhost-user transport Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 23/28] net/vhost: add virtio-vhost-user support Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 24/28] examples/vhost_scsi: add --socket-file argument Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 25/28] examples/vhost_scsi: add virtio-vhost-user support Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 26/28] mk: link apps with virtio-vhost-user driver Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 27/28] config: add option for the virtio-vhost-user transport Nikos Dragazis
2019-06-19 15:14 ` [dpdk-dev] [PATCH 28/28] usertools: add virtio-vhost-user devices to dpdk-devbind.py Nikos Dragazis
[not found] ` <CGME20190620113240eucas1p22ca4faa64a36bbb7aec38a81298ade56@eucas1p2.samsung.com>
2019-06-20 11:32 ` [dpdk-dev] [PATCH 00/28] vhost: add virtio-vhost-user transport Ilya Maximets
2019-06-20 23:44 ` Nikos Dragazis
2019-06-20 11:35 ` Maxime Coquelin
2019-06-22 20:26 ` Nikos Dragazis [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=aa302511-0f31-ae2d-b2cd-f7c87573ec65@arrikto.com \
--to=ndragazis@arrikto.com \
--cc=dariusz.stojaczyk@intel.com \
--cc=dev@dpdk.org \
--cc=maxime.coquelin@redhat.com \
--cc=stefanha@redhat.com \
--cc=tiwei.bie@intel.com \
--cc=vkoukis@arrikto.com \
--cc=wei.w.wang@intel.com \
--cc=zhihong.wang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).