From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by dpdk.org (Postfix) with ESMTP id B39B81B455 for ; Thu, 27 Sep 2018 18:37:31 +0200 (CEST) Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E4809308625E; Thu, 27 Sep 2018 16:37:30 +0000 (UTC) Received: from [10.36.112.65] (ovpn-112-65.ams2.redhat.com [10.36.112.65]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 26C1F3091882; Thu, 27 Sep 2018 16:37:27 +0000 (UTC) To: Xiaolong Ye , dev@dpdk.org, Tiwei Bie , Zhihong Wang Cc: xiao.w.wang@intel.com, Rami Rosen , Wang Haiyue References: <20180925120716.102144-1-xiaolong.ye@intel.com> <20180926090638.4705-1-xiaolong.ye@intel.com> <20180926090638.4705-3-xiaolong.ye@intel.com> From: Maxime Coquelin Message-ID: <2404f5d8-beed-08fe-06bf-451868e3e9d3@redhat.com> Date: Thu, 27 Sep 2018 18:37:26 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <20180926090638.4705-3-xiaolong.ye@intel.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.84 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.49]); Thu, 27 Sep 2018 16:37:31 +0000 (UTC) Subject: Re: [dpdk-dev] [PATCH v6 2/2] examples/vdpa: introduce a new sample for vDPA X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Sep 2018 16:37:32 -0000 On 09/26/2018 11:06 AM, Xiaolong Ye wrote: > The vdpa sample application creates vhost-user sockets by using the > vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes > virtio ring compatible devices to serve virtio driver directly to enable > datapath acceleration. As vDPA driver can help to set up vhost datapath, > this application doesn't need to launch dedicated worker threads for vhost > enqueue/dequeue operations. > > Signed-off-by: Xiao Wang > Signed-off-by: Xiaolong Ye > --- > MAINTAINERS | 2 + > doc/guides/rel_notes/release_18_11.rst | 8 + > doc/guides/sample_app_ug/index.rst | 1 + > doc/guides/sample_app_ug/vdpa.rst | 120 +++++++ > examples/Makefile | 2 +- > examples/vdpa/Makefile | 32 ++ > examples/vdpa/main.c | 466 +++++++++++++++++++++++++ > examples/vdpa/meson.build | 16 + > 8 files changed, 646 insertions(+), 1 deletion(-) > create mode 100644 doc/guides/sample_app_ug/vdpa.rst > create mode 100644 examples/vdpa/Makefile > create mode 100644 examples/vdpa/main.c > create mode 100644 examples/vdpa/meson.build > > diff --git a/MAINTAINERS b/MAINTAINERS > index 5967c1dd3..5656f18e8 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst > F: examples/vhost_scsi/ > F: doc/guides/sample_app_ug/vhost_scsi.rst > F: examples/vhost_crypto/ > +F: examples/vdpa/ > +F: doc/guides/sample_app_ug/vdpa.rst > > Vhost PMD > M: Maxime Coquelin > diff --git a/doc/guides/rel_notes/release_18_11.rst b/doc/guides/rel_notes/release_18_11.rst > index bc9b74ec4..dd53a9ecf 100644 > --- a/doc/guides/rel_notes/release_18_11.rst > +++ b/doc/guides/rel_notes/release_18_11.rst > @@ -67,6 +67,14 @@ New Features > SR-IOV option in Hyper-V and Azure. This is an alternative to the previous > vdev_netvsc, tap, and failsafe drivers combination. > > +* **Add a new sample for vDPA** > + > + The vdpa sample application creates vhost-user sockets by using the > + vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes > + virtio ring compatible devices to serve virtio driver directly to enable > + datapath acceleration. As vDPA driver can help to set up vhost datapath, > + this application doesn't need to launch dedicated worker threads for vhost > + enqueue/dequeue operations. > > API Changes > ----------- > diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst > index 5bedf4f6f..74b12af85 100644 > --- a/doc/guides/sample_app_ug/index.rst > +++ b/doc/guides/sample_app_ug/index.rst > @@ -45,6 +45,7 @@ Sample Applications User Guides > vhost > vhost_scsi > vhost_crypto > + vdpa > netmap_compatibility > ip_pipeline > test_pipeline > diff --git a/doc/guides/sample_app_ug/vdpa.rst b/doc/guides/sample_app_ug/vdpa.rst > new file mode 100644 > index 000000000..745f196ca > --- /dev/null > +++ b/doc/guides/sample_app_ug/vdpa.rst > @@ -0,0 +1,120 @@ > +.. SPDX-License-Identifier: BSD-3-Clause > + Copyright(c) 2018 Intel Corporation. > + > +Vdpa Sample Application > +======================= > + > +The vdpa sample application creates vhost-user sockets by using the > +vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes > +virtio ring compatible devices to serve virtio driver directly to enable > +datapath acceleration. As vDPA driver can help to set up vhost datapath, > +this application doesn't need to launch dedicated worker threads for vhost > +enqueue/dequeue operations. > + > +Testing steps > +------------- > + > +This section shows the steps of how to start VMs with vDPA vhost-user > +backend and verify network connection & live migration. > + > +Build > +~~~~~ > + > +To compile the sample application see :doc:`compiling`. > + > +The application is located in the ``vdpa`` sub-directory. > + > +Start the vdpa example > +~~~~~~~~~~~~~~~~~~~~~~ > + > +.. code-block:: console > + > + ./vdpa [EAL options] -- [--client] [--interactive|-i] or [--iface SOCKET_PATH] > + > +where > + > +* --client means running vdpa app in client mode, in the client mode, QEMU needs > + to run as the server mode and take charge of socket file creation. > +* --iface specifies the path prefix of the UNIX domain socket file, e.g. > + /tmp/vhost-user-, then the socket files will be named as /tmp/vhost-user- > + (n starts from 0). > +* --interactive means run the vdpa sample in interactive mode, currently 4 > + internal cmds are supported: > + > + 1. help: show help message > + 2. list: list all available vdpa devices > + 3. create: create a new vdpa port with socket file and vdpa device address > + 4. quit: unregister vhost driver and exit the application > + > +Take IFCVF driver for example: > + > +.. code-block:: console > + > + ./vdpa -c 0x2 -n 4 --socket-mem 1024,1024 \ > + -w 0000:06:00.3,vdpa=1 -w 0000:06:00.4,vdpa=1 \ > + -- --interactive > + > +.. note:: > + Here 0000:06:00.3 and 0000:06:00.4 refer to virtio ring compatible devices, > + and we need to bind vfio-pci to them before running vdpa sample. > + > + * modprobe vfio-pci > + * ./usertools/dpdk-devbind.py -b vfio-pci 06:00.3 06:00.4 > + > +Then we can create 2 vdpa ports in interactive cmdline. > + > +.. code-block:: console > + > + vdpa> list > + device id device address queue num supported features > + 0 0000:06:00.3 1 0x14c238020 > + 1 0000:06:00.4 1 0x14c238020 > + 2 0000:06:00.5 1 0x14c238020 > + > + vdpa> create /tmp/vdpa-socket0 0000:06:00.3 > + vdpa> create /tmp/vdpa-socket1 0000:06:00.4 > + > +.. _vdpa_app_run_vm: > + > +Start the VMs > +~~~~~~~~~~~~~ > + > +.. code-block:: console > + > + qemu-system-x86_64 -cpu host -enable-kvm \ > + > + -mem-prealloc \ > + -chardev socket,id=char0,path= \ > + -netdev type=vhost-user,id=vdpa,chardev=char0 \ > + -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee,page-per-vq=on \ > + > +After the VMs launches, we can login the VMs and configure the ip, verify the > +network connection via ping or netperf. > + > +.. note:: > + Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA. > + > +Live Migration > +~~~~~~~~~~~~~~ > +vDPA supports cross-backend live migration, user can migrate SW vhost backend > +VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume A is > +the source host with SW vhost VM and B is the destination host with vDPA. > + > +1. Start vdpa sample and launch a VM with exact same parameters as the VM on A, > + in migration-listen mode: > + > +.. code-block:: console > + > + B: -incoming tcp:0:4444 (or other PORT)) > + > +2. Start the migration (on source host): > + > +.. code-block:: console > + > + A: (qemu) migrate -d tcp::4444 (or other PORT) > + > +3. Check the status (on source host): > + > +.. code-block:: console > + > + A: (qemu) info migrate > diff --git a/examples/Makefile b/examples/Makefile > index 481720cb6..356fcb1cd 100644 > --- a/examples/Makefile > +++ b/examples/Makefile > @@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y) > DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination > endif > DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer > -DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi > +DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi vdpa > ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y) > DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_crypto > endif > diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile > new file mode 100644 > index 000000000..42672a2bc > --- /dev/null > +++ b/examples/vdpa/Makefile > @@ -0,0 +1,32 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright(c) 2018 Intel Corporation > + > +ifeq ($(RTE_SDK),) > +$(error "Please define RTE_SDK environment variable") > +endif > + > +# Default target, can be overridden by command line or environment > +RTE_TARGET ?= x86_64-native-linuxapp-gcc > + > +include $(RTE_SDK)/mk/rte.vars.mk > + > +ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp") > +$(info This application can only operate in a linuxapp environment, \ > +please change the definition of the RTE_TARGET environment variable) > +all: > +else > + > +# binary name > +APP = vdpa > + > +# all source are stored in SRCS-y > +SRCS-y := main.c > + > +CFLAGS += -O2 -D_FILE_OFFSET_BITS=64 > +CFLAGS += $(WERROR_FLAGS) > +CFLAGS += -D_GNU_SOURCE > +CFLAGS += -DALLOW_EXPERIMENTAL_API > + > +include $(RTE_SDK)/mk/rte.extapp.mk > + > +endif > diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c > new file mode 100644 > index 000000000..898286e30 > --- /dev/null > +++ b/examples/vdpa/main.c > @@ -0,0 +1,466 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright(c) 2018 Intel Corporation > + */ > + > +#include > +#include > +#include > +#include > +#include > + > +#include > +#include > +#include > +#include > +#include > + > +#include > +#include > +#include > +#include > + > +#define MAX_PATH_LEN 128 > +#define MAX_VDPA_SAMPLE_PORTS 1024 > +#define RTE_LOGTYPE_VDPA RTE_LOGTYPE_USER1 > + > +struct vdpa_port { > + char ifname[MAX_PATH_LEN]; > + int did; > + int vid; > + uint64_t flags; > +}; > + > +struct vdpa_port vports[MAX_VDPA_SAMPLE_PORTS]; I guess it can be static? > + > +static char iface[MAX_PATH_LEN]; > +static int dev_total; > +static int devcnt; > +static int interactive; > +static int client_mode; > + > +/* display usage */ > +static void > +vdpa_usage(const char *prgname) > +{ > + printf("Usage: %s [EAL options] -- " > + " --interactive|-i: run in interactive mode.\n" > + " --iface : specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n" > + " --client: register a vhost-user socket as client mode.\n", > + prgname); > +} > + > +static int > +parse_args(int argc, char **argv) > +{ > + static const char *short_option = "i"; > + static struct option long_option[] = { > + {"iface", required_argument, NULL, 0}, > + {"interactive", no_argument, &interactive, 1}, > + {"client", no_argument, &client_mode, 1}, > + {NULL, 0, 0, 0}, > + }; > + int opt, idx; > + char *prgname = argv[0]; > + > + while ((opt = getopt_long(argc, argv, short_option, long_option, &idx)) > + != EOF) { > + switch (opt) { > + case 'i': > + printf("Interactive-mode selected\n"); > + interactive = 1; > + break; > + /* long options */ > + case 0: > + if (strncmp(long_option[idx].name, "iface", > + MAX_PATH_LEN) == 0) { > + strncpy(iface, optarg, MAX_PATH_LEN); > + printf("iface %s\n", iface); > + } > + if (!strcmp(long_option[idx].name, "interactive")) { > + printf("Interactive-mode selected\n"); > + interactive = 1; > + } > + break; > + > + default: > + vdpa_usage(prgname); > + return -1; > + } > + } > + > + if (iface[0] == '\0' && interactive == 0) { > + vdpa_usage(prgname); > + return -1; > + } > + > + return 0; > +} > + > +static int > +data_init(void) > +{ > + dev_total = rte_vdpa_get_device_num(); > + if (dev_total <= 0) { > + printf("No available vdpa device found\n"); > + return -1; > + } > + devcnt = 0; This isn't needed as it is static. > + interactive = 0; Ditto > + memset(iface, 0, sizeof(iface)); Ditto > + memset(vports, 0, sizeof(vports)); And this one too when it will be made static. Appart these small comments, it looks good to me. When fixed, please apply my: Reviewed-by: Maxime coquelin Thanks, Maxime