Re: [dpdk-dev] [PATCH v6 2/2] examples/vdpa: introduce a new sample for vDPA

[Maxime Coquelin <maxime.coquelin@redhat.com>]

On 09/26/2018 11:06 AM, Xiaolong Ye wrote:
> The vdpa sample application creates vhost-user sockets by using the
> vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> virtio ring compatible devices to serve virtio driver directly to enable
> datapath acceleration. As vDPA driver can help to set up vhost datapath,
> this application doesn't need to launch dedicated worker threads for vhost
> enqueue/dequeue operations.
> 
> Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> ---
>   MAINTAINERS                            |   2 +
>   doc/guides/rel_notes/release_18_11.rst |   8 +
>   doc/guides/sample_app_ug/index.rst     |   1 +
>   doc/guides/sample_app_ug/vdpa.rst      | 120 +++++++
>   examples/Makefile                      |   2 +-
>   examples/vdpa/Makefile                 |  32 ++
>   examples/vdpa/main.c                   | 466 +++++++++++++++++++++++++
>   examples/vdpa/meson.build              |  16 +
>   8 files changed, 646 insertions(+), 1 deletion(-)
>   create mode 100644 doc/guides/sample_app_ug/vdpa.rst
>   create mode 100644 examples/vdpa/Makefile
>   create mode 100644 examples/vdpa/main.c
>   create mode 100644 examples/vdpa/meson.build
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 5967c1dd3..5656f18e8 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst
>   F: examples/vhost_scsi/
>   F: doc/guides/sample_app_ug/vhost_scsi.rst
>   F: examples/vhost_crypto/
> +F: examples/vdpa/
> +F: doc/guides/sample_app_ug/vdpa.rst
>   
>   Vhost PMD
>   M: Maxime Coquelin <maxime.coquelin@redhat.com>
> diff --git a/doc/guides/rel_notes/release_18_11.rst b/doc/guides/rel_notes/release_18_11.rst
> index bc9b74ec4..dd53a9ecf 100644
> --- a/doc/guides/rel_notes/release_18_11.rst
> +++ b/doc/guides/rel_notes/release_18_11.rst
> @@ -67,6 +67,14 @@ New Features
>     SR-IOV option in Hyper-V and Azure. This is an alternative to the previous
>     vdev_netvsc, tap, and failsafe drivers combination.
>   
> +* **Add a new sample for vDPA**
> +
> +  The vdpa sample application creates vhost-user sockets by using the
> +  vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> +  virtio ring compatible devices to serve virtio driver directly to enable
> +  datapath acceleration. As vDPA driver can help to set up vhost datapath,
> +  this application doesn't need to launch dedicated worker threads for vhost
> +  enqueue/dequeue operations.
>   
>   API Changes
>   -----------
> diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
> index 5bedf4f6f..74b12af85 100644
> --- a/doc/guides/sample_app_ug/index.rst
> +++ b/doc/guides/sample_app_ug/index.rst
> @@ -45,6 +45,7 @@ Sample Applications User Guides
>       vhost
>       vhost_scsi
>       vhost_crypto
> +    vdpa
>       netmap_compatibility
>       ip_pipeline
>       test_pipeline
> diff --git a/doc/guides/sample_app_ug/vdpa.rst b/doc/guides/sample_app_ug/vdpa.rst
> new file mode 100644
> index 000000000..745f196ca
> --- /dev/null
> +++ b/doc/guides/sample_app_ug/vdpa.rst
> @@ -0,0 +1,120 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2018 Intel Corporation.
> +
> +Vdpa Sample Application
> +=======================
> +
> +The vdpa sample application creates vhost-user sockets by using the
> +vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> +virtio ring compatible devices to serve virtio driver directly to enable
> +datapath acceleration. As vDPA driver can help to set up vhost datapath,
> +this application doesn't need to launch dedicated worker threads for vhost
> +enqueue/dequeue operations.
> +
> +Testing steps
> +-------------
> +
> +This section shows the steps of how to start VMs with vDPA vhost-user
> +backend and verify network connection & live migration.
> +
> +Build
> +~~~~~
> +
> +To compile the sample application see :doc:`compiling`.
> +
> +The application is located in the ``vdpa`` sub-directory.
> +
> +Start the vdpa example
> +~~~~~~~~~~~~~~~~~~~~~~
> +
> +.. code-block:: console
> +
> +        ./vdpa [EAL options]  -- [--client] [--interactive|-i] or [--iface SOCKET_PATH]
> +
> +where
> +
> +* --client means running vdpa app in client mode, in the client mode, QEMU needs
> +  to run as the server mode and take charge of socket file creation.
> +* --iface specifies the path prefix of the UNIX domain socket file, e.g.
> +  /tmp/vhost-user-, then the socket files will be named as /tmp/vhost-user-<n>
> +  (n starts from 0).
> +* --interactive means run the vdpa sample in interactive mode, currently 4
> +  internal cmds are supported:
> +
> +  1. help: show help message
> +  2. list: list all available vdpa devices
> +  3. create: create a new vdpa port with socket file and vdpa device address
> +  4. quit: unregister vhost driver and exit the application
> +
> +Take IFCVF driver for example:
> +
> +.. code-block:: console
> +
> +        ./vdpa -c 0x2 -n 4 --socket-mem 1024,1024 \
> +                -w 0000:06:00.3,vdpa=1 -w 0000:06:00.4,vdpa=1 \
> +                -- --interactive
> +
> +.. note::
> +    Here 0000:06:00.3 and 0000:06:00.4 refer to virtio ring compatible devices,
> +    and we need to bind vfio-pci to them before running vdpa sample.
> +
> +    * modprobe vfio-pci
> +    * ./usertools/dpdk-devbind.py -b vfio-pci 06:00.3 06:00.4
> +
> +Then we can create 2 vdpa ports in interactive cmdline.
> +
> +.. code-block:: console
> +
> +        vdpa> list
> +        device id       device address  queue num       supported features
> +        0               0000:06:00.3    1               0x14c238020
> +        1               0000:06:00.4    1               0x14c238020
> +        2               0000:06:00.5    1               0x14c238020
> +
> +        vdpa> create /tmp/vdpa-socket0 0000:06:00.3
> +        vdpa> create /tmp/vdpa-socket1 0000:06:00.4
> +
> +.. _vdpa_app_run_vm:
> +
> +Start the VMs
> +~~~~~~~~~~~~~
> +
> +.. code-block:: console
> +
> +       qemu-system-x86_64 -cpu host -enable-kvm \
> +       <snip>
> +       -mem-prealloc \
> +       -chardev socket,id=char0,path=<socket_file created in above steps> \
> +       -netdev type=vhost-user,id=vdpa,chardev=char0 \
> +       -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee,page-per-vq=on \
> +
> +After the VMs launches, we can login the VMs and configure the ip, verify the
> +network connection via ping or netperf.
> +
> +.. note::
> +    Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA.
> +
> +Live Migration
> +~~~~~~~~~~~~~~
> +vDPA supports cross-backend live migration, user can migrate SW vhost backend
> +VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume A is
> +the source host with SW vhost VM and B is the destination host with vDPA.
> +
> +1. Start vdpa sample and launch a VM with exact same parameters as the VM on A,
> +   in migration-listen mode:
> +
> +.. code-block:: console
> +
> +        B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
> +
> +2. Start the migration (on source host):
> +
> +.. code-block:: console
> +
> +        A: (qemu) migrate -d tcp:<B ip>:4444 (or other PORT)
> +
> +3. Check the status (on source host):
> +
> +.. code-block:: console
> +
> +        A: (qemu) info migrate
> diff --git a/examples/Makefile b/examples/Makefile
> index 481720cb6..356fcb1cd 100644
> --- a/examples/Makefile
> +++ b/examples/Makefile
> @@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y)
>   DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
>   endif
>   DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
> -DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi
> +DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi vdpa
>   ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y)
>   DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_crypto
>   endif
> diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
> new file mode 100644
> index 000000000..42672a2bc
> --- /dev/null
> +++ b/examples/vdpa/Makefile
> @@ -0,0 +1,32 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2018 Intel Corporation
> +
> +ifeq ($(RTE_SDK),)
> +$(error "Please define RTE_SDK environment variable")
> +endif
> +
> +# Default target, can be overridden by command line or environment
> +RTE_TARGET ?= x86_64-native-linuxapp-gcc
> +
> +include $(RTE_SDK)/mk/rte.vars.mk
> +
> +ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp")
> +$(info This application can only operate in a linuxapp environment, \
> +please change the definition of the RTE_TARGET environment variable)
> +all:
> +else
> +
> +# binary name
> +APP = vdpa
> +
> +# all source are stored in SRCS-y
> +SRCS-y := main.c
> +
> +CFLAGS += -O2 -D_FILE_OFFSET_BITS=64
> +CFLAGS += $(WERROR_FLAGS)
> +CFLAGS += -D_GNU_SOURCE
> +CFLAGS += -DALLOW_EXPERIMENTAL_API
> +
> +include $(RTE_SDK)/mk/rte.extapp.mk
> +
> +endif
> diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c
> new file mode 100644
> index 000000000..898286e30
> --- /dev/null
> +++ b/examples/vdpa/main.c
> @@ -0,0 +1,466 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2018 Intel Corporation
> + */
> +
> +#include <getopt.h>
> +#include <signal.h>
> +#include <stdint.h>
> +#include <string.h>
> +#include <unistd.h>
> +
> +#include <rte_ethdev.h>
> +#include <rte_malloc.h>
> +#include <rte_vhost.h>
> +#include <rte_vdpa.h>
> +#include <rte_pci.h>
> +
> +#include <cmdline_parse.h>
> +#include <cmdline_socket.h>
> +#include <cmdline_parse_string.h>
> +#include <cmdline.h>
> +
> +#define MAX_PATH_LEN 128
> +#define MAX_VDPA_SAMPLE_PORTS 1024
> +#define RTE_LOGTYPE_VDPA RTE_LOGTYPE_USER1
> +
> +struct vdpa_port {
> +	char ifname[MAX_PATH_LEN];
> +	int did;
> +	int vid;
> +	uint64_t flags;
> +};
> +
> +struct vdpa_port vports[MAX_VDPA_SAMPLE_PORTS];

I guess it can be static?

> +
> +static char iface[MAX_PATH_LEN];
> +static int dev_total;
> +static int devcnt;
> +static int interactive;
> +static int client_mode;
> +
> +/* display usage */
> +static void
> +vdpa_usage(const char *prgname)
> +{
> +	printf("Usage: %s [EAL options] -- "
> +				 "	--interactive|-i: run in interactive mode.\n"
> +				 "	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
> +				 "	--client: register a vhost-user socket as client mode.\n",
> +				 prgname);
> +}
> +
> +static int
> +parse_args(int argc, char **argv)
> +{
> +	static const char *short_option = "i";
> +	static struct option long_option[] = {
> +		{"iface", required_argument, NULL, 0},
> +		{"interactive", no_argument, &interactive, 1},
> +		{"client", no_argument, &client_mode, 1},
> +		{NULL, 0, 0, 0},
> +	};
> +	int opt, idx;
> +	char *prgname = argv[0];
> +
> +	while ((opt = getopt_long(argc, argv, short_option, long_option, &idx))
> +			!= EOF) {
> +		switch (opt) {
> +		case 'i':
> +			printf("Interactive-mode selected\n");
> +			interactive = 1;
> +			break;
> +		/* long options */
> +		case 0:
> +			if (strncmp(long_option[idx].name, "iface",
> +						MAX_PATH_LEN) == 0) {
> +				strncpy(iface, optarg, MAX_PATH_LEN);
> +				printf("iface %s\n", iface);
> +			}
> +			if (!strcmp(long_option[idx].name, "interactive")) {
> +				printf("Interactive-mode selected\n");
> +				interactive = 1;
> +			}
> +			break;
> +
> +		default:
> +			vdpa_usage(prgname);
> +			return -1;
> +		}
> +	}
> +
> +	if (iface[0] == '\0' && interactive == 0) {
> +		vdpa_usage(prgname);
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +data_init(void)
> +{
> +	dev_total = rte_vdpa_get_device_num();
> +	if (dev_total <= 0) {
> +		printf("No available vdpa device found\n");
> +		return -1;
> +	}
> +	devcnt = 0;
This isn't needed as it is static.

> +	interactive = 0;
Ditto

> +	memset(iface, 0, sizeof(iface));
Ditto

> +	memset(vports, 0, sizeof(vports));
And this one too when it will be made static.


Appart these small comments, it looks good to me.
When fixed, please apply my:

Reviewed-by: Maxime coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime