From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id 183AFA0471 for ; Wed, 19 Jun 2019 17:15:37 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E45AF1C33E; Wed, 19 Jun 2019 17:15:36 +0200 (CEST) Received: from mx0.arrikto.com (mx0.arrikto.com [212.71.252.59]) by dpdk.org (Postfix) with ESMTP id A8F672BEA for ; Wed, 19 Jun 2019 17:15:35 +0200 (CEST) Received: from troi.prod.arr (mail.arr [10.99.0.5]) by mx0.arrikto.com (Postfix) with ESMTP id 67424182004; Wed, 19 Jun 2019 18:15:35 +0300 (EEST) Received: from localhost.localdomain (unknown [10.89.50.133]) by troi.prod.arr (Postfix) with ESMTPSA id C07DD2B2; Wed, 19 Jun 2019 18:15:34 +0300 (EEST) From: Nikos Dragazis To: dev@dpdk.org Cc: Maxime Coquelin , Tiwei Bie , Zhihong Wang , Stefan Hajnoczi , Wei Wang , Stojaczyk Dariusz , Vangelis Koukis Date: Wed, 19 Jun 2019 18:14:25 +0300 Message-Id: <1560957293-17294-1-git-send-email-ndragazis@arrikto.com> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH 00/28] vhost: add virtio-vhost-user transport X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi everyone, this patch series introduces the concept of the virtio-vhost-user transport. This is actually a revised version of an earlier RFC implementation that has been proposed by Stefan Hajnoczi [1]. Though this is a great feature, it seems to have been stalled, so I’d like to restart the conversation on this and hopefully get it merged with your help. Let me give you an overview. The virtio-vhost-user transport is a vhost-user transport implementation that is based on the virtio-vhost-user device. Its key difference with the existing transport is that it allows deploying vhost-user targets inside dedicated Storage Appliance VMs instead of host user space. In other words, it allows having guests that act as vhost-user backends for other guests. The virtio-vhost-user device implements the vhost-user control plane (master-slave communication) as follows: 1. it parses the vhost-user messages from the vhost-user unix domain socket and forwards them to the slave guest through virtqueues 2. it maps the vhost memory regions in QEMU’s process address space and exposes them to the slave guest as a RAM-backed PCI MMIO region 3. it hooks up doorbells to the callfds. The slave guest can use these doorbells to interrupt the master guest driver The device code has not yet been merged into upstream QEMU, but this is definitely the end goal. The current state is that we are awaiting for the approval of the virtio spec. I have Cced Darek from the SPDK community who has helped me a lot by reviewing this series. Note that any device type could be implemented over this new transport. So, adding the virtio-vhost-user transport in DPDK would allow using it from SPDK as well. Getting into the code internals, this patch series makes the following changes: 1. introduce a generic interface for the transport-specific operations. Each of the two available transports, the pre-existing AF_UNIX transport and the virtio-vhost-user transport, is going to implement this interface. The AF_UNIX-specific code has been extracted from the core vhost-user code and is now part of the AF_UNIX transport implementation in trans_af_unix.c. 2. introduce the virtio-vhost-user transport. The virtio-vhost-user transport requires a driver for the virtio-vhost-user devices. The driver along with the transport implementation have been packed into a separate library in `drivers/virtio_vhost_user/`. The necessary virtio-pci code has been copied from `drivers/net/virtio/`. Some additional changes have been made so that the driver can utilize the additional resources of the virtio-vhost-user device. 3. update librte_vhost public API to enable choosing transport for each new vhost device. Extend the vhost net driver and vhost-scsi example application to export this new API to the end user. The primary changes I did to Stefan’s RFC implementation are the following: 1. moved postcopy live migration code into trans_af_unix.c. Postcopy live migration relies on the userfault fd mechanism, which cannot be supported by virtio-vhost-user. 2. moved setup of the log memory region into trans_af_unix.c. Setting up the log memory region involves mapping/unmapping guest memory. This is an AF_UNIX transport-specific operation. 3. introduced a vhost transport operation for process_slave_message_reply() 4. moved the virtio-vhost-user transport/driver into a separate library in `drivers/virtio_vhost_user/`. This required making vhost.h and vhost_user.h part of librte_vhost public API and exporting some private symbols via the version script. This looks better to me that just moving the entire librte_vhost into `drivers/`. I am not sure if this is the most appropriate solution. I am looking forward to your suggestions on this. 5. made use of the virtio PCI capabilities for the additional device resources (doorbells, shared memory). This required changes in virtio_pci.c and trans_virtio_vhost_user.c. 6. [minor] changed some commit headlines to comply with check-git-log.sh. Please, have a look and let me know about your thoughts. Any reviews/pointers/suggestions are welcome. Best regards, Nikos [1] http://mails.dpdk.org/archives/dev/2018-January/088155.html Nikos Dragazis (23): vhost: introduce vhost transport operations structure vhost: move socket management code vhost: move socket fd and un sockaddr vhost: move vhost-user connection vhost: move vhost-user reconnection vhost: move vhost-user fdset vhost: propagate vhost transport operations vhost: use a single structure for the device state vhost: extract socket I/O into transport vhost: move slave request fd and lock vhost: move mmap/munmap vhost: move setup of the log memory region vhost: remove main fd parameter from msg handlers vhost: move postcopy live migration code vhost: support registering additional vhost-user transports drivers/virtio_vhost_user: add virtio PCI framework drivers: add virtio-vhost-user transport drivers/virtio_vhost_user: use additional device resources vhost: add flag for choosing vhost-user transport net/vhost: add virtio-vhost-user support mk: link apps with virtio-vhost-user driver config: add option for the virtio-vhost-user transport usertools: add virtio-vhost-user devices to dpdk-devbind.py Stefan Hajnoczi (5): vhost: allocate per-socket transport state vhost: move start server/client calls vhost: add index field in vhost virtqueues examples/vhost_scsi: add --socket-file argument examples/vhost_scsi: add virtio-vhost-user support config/common_base | 6 + config/common_linux | 1 + drivers/Makefile | 5 + drivers/net/vhost/rte_eth_vhost.c | 13 + drivers/virtio_vhost_user/Makefile | 27 + .../rte_virtio_vhost_user_version.map | 4 + .../virtio_vhost_user/trans_virtio_vhost_user.c | 1077 +++++++++++++++++++ drivers/virtio_vhost_user/virtio_pci.c | 520 ++++++++++ drivers/virtio_vhost_user/virtio_pci.h | 289 ++++++ drivers/virtio_vhost_user/virtio_vhost_user.h | 18 + drivers/virtio_vhost_user/virtqueue.h | 181 ++++ examples/vhost_scsi/vhost_scsi.c | 103 +- lib/librte_vhost/Makefile | 4 +- lib/librte_vhost/rte_vhost.h | 1 + lib/librte_vhost/rte_vhost_version.map | 11 + lib/librte_vhost/socket.c | 685 +----------- lib/librte_vhost/trans_af_unix.c | 1094 ++++++++++++++++++++ lib/librte_vhost/vhost.c | 22 +- lib/librte_vhost/vhost.h | 298 +++++- lib/librte_vhost/vhost_user.c | 474 ++------- lib/librte_vhost/vhost_user.h | 10 +- mk/rte.app.mk | 6 + usertools/dpdk-devbind.py | 7 + 23 files changed, 3764 insertions(+), 1092 deletions(-) create mode 100644 drivers/virtio_vhost_user/Makefile create mode 100644 drivers/virtio_vhost_user/rte_virtio_vhost_user_version.map create mode 100644 drivers/virtio_vhost_user/trans_virtio_vhost_user.c create mode 100644 drivers/virtio_vhost_user/virtio_pci.c create mode 100644 drivers/virtio_vhost_user/virtio_pci.h create mode 100644 drivers/virtio_vhost_user/virtio_vhost_user.h create mode 100644 drivers/virtio_vhost_user/virtqueue.h create mode 100644 lib/librte_vhost/trans_af_unix.c -- 2.7.4