From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id DCABEF04 for ; Fri, 21 Sep 2018 07:07:12 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Sep 2018 22:07:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,283,1534834800"; d="scan'208";a="91996495" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by fmsmga001.fm.intel.com with ESMTP; 20 Sep 2018 22:07:11 -0700 Received: from fmsmsx114.amr.corp.intel.com (10.18.116.8) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.319.2; Thu, 20 Sep 2018 22:07:11 -0700 Received: from shsmsx104.ccr.corp.intel.com (10.239.4.70) by FMSMSX114.amr.corp.intel.com (10.18.116.8) with Microsoft SMTP Server (TLS) id 14.3.319.2; Thu, 20 Sep 2018 22:07:10 -0700 Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.39]) by SHSMSX104.ccr.corp.intel.com ([169.254.5.227]) with mapi id 14.03.0319.002; Fri, 21 Sep 2018 13:07:08 +0800 From: "Wang, Xiao W" To: "Ye, Xiaolong" , "dev@dpdk.org" , Maxime Coquelin , "Bie, Tiwei" , "Wang, Zhihong" CC: Rami Rosen , "Wang, Haiyue" Thread-Topic: [PATCH v3 2/2] examples/vdpa: introduce a new sample for vDPA Thread-Index: AQHUUPg+Jwr+PzLBrE+CEAlWVKbTGaT6IiEg Date: Fri, 21 Sep 2018 05:07:08 +0000 Message-ID: References: <20180920222817.45829-1-xiaolong.ye@intel.com> <20180920222817.45829-3-xiaolong.ye@intel.com> In-Reply-To: <20180920222817.45829-3-xiaolong.ye@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ctpclassification: CTP_NT x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMDY4ZDVkZGEtODNkYi00OWExLTk2ZDAtOTZjZWNhOTY3NDkzIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiQXZZMVBncCtnNVNyd05laXZYbWFYN210bXVNZWtlNU96TzNMREFvcEZROEdBN2lsNnJKUjFtYVNSRUtvS0xCZyJ9 dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v3 2/2] examples/vdpa: introduce a new sample for vDPA X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Sep 2018 05:07:13 -0000 Hi Xiaolong, > -----Original Message----- > From: Ye, Xiaolong > Sent: Friday, September 21, 2018 6:28 AM > To: dev@dpdk.org; Maxime Coquelin ; Bie, > Tiwei ; Wang, Zhihong > Cc: Wang, Xiao W ; Rami Rosen > ; Wang, Haiyue ; Ye, > Xiaolong > Subject: [PATCH v3 2/2] examples/vdpa: introduce a new sample for vDPA >=20 > The vdpa sample application creates vhost-user sockets by using the > vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes > virtio ring compatible devices to serve virtio driver directly to enable > datapath acceleration. As vDPA driver can help to set up vhost datapath, > this application doesn't need to launch dedicated worker threads for vhos= t > enqueue/dequeue operations. >=20 > Signed-off-by: Xiao Wang > Signed-off-by: Xiaolong Ye > --- > MAINTAINERS | 2 + > doc/guides/sample_app_ug/index.rst | 1 + > doc/guides/sample_app_ug/vdpa.rst | 115 ++++++++ > examples/Makefile | 2 +- > examples/vdpa/Makefile | 32 ++ > examples/vdpa/main.c | 458 +++++++++++++++++++++++++++++ > examples/vdpa/meson.build | 16 + > 7 files changed, 625 insertions(+), 1 deletion(-) > create mode 100644 doc/guides/sample_app_ug/vdpa.rst > create mode 100644 examples/vdpa/Makefile > create mode 100644 examples/vdpa/main.c > create mode 100644 examples/vdpa/meson.build >=20 > diff --git a/MAINTAINERS b/MAINTAINERS > index 5967c1dd3..5656f18e8 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst > F: examples/vhost_scsi/ > F: doc/guides/sample_app_ug/vhost_scsi.rst > F: examples/vhost_crypto/ > +F: examples/vdpa/ > +F: doc/guides/sample_app_ug/vdpa.rst >=20 > Vhost PMD > M: Maxime Coquelin > diff --git a/doc/guides/sample_app_ug/index.rst > b/doc/guides/sample_app_ug/index.rst > index 5bedf4f6f..74b12af85 100644 > --- a/doc/guides/sample_app_ug/index.rst > +++ b/doc/guides/sample_app_ug/index.rst > @@ -45,6 +45,7 @@ Sample Applications User Guides > vhost > vhost_scsi > vhost_crypto > + vdpa > netmap_compatibility > ip_pipeline > test_pipeline > diff --git a/doc/guides/sample_app_ug/vdpa.rst > b/doc/guides/sample_app_ug/vdpa.rst > new file mode 100644 > index 000000000..44fe6736d > --- /dev/null > +++ b/doc/guides/sample_app_ug/vdpa.rst > @@ -0,0 +1,115 @@ > +.. SPDX-License-Identifier: BSD-3-Clause > + Copyright(c) 2018 Intel Corporation. > + > +Vdpa Sample Application > +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > + > +The vdpa sample application creates vhost-user sockets by using the > +vDPA backend. vDPA stands for vhost Data Path Acceleration which utilize= s > +virtio ring compatible devices to serve virtio driver directly to enable > +datapath acceleration. As vDPA driver can help to set up vhost datapath, > +this application doesn't need to launch dedicated worker threads for vho= st > +enqueue/dequeue operations. > + > +Testing steps [...] > + > +Then we can create 2 vdpa ports in interactive cmdline. > + > +.. code-block:: console > + > + vdpa> list > + device id device address > + 0 0000:06:00.2 > + 1 0000:06:00.3 The features and queue numbers of each vDPA device could also be shown now,= the doc should reflect this. > + vdpa> create /tmp/vdpa-socket0 0000:06:00.2 > + vdpa> create /tmp/vdpa-socket1 0000:06:00.3 > + > +.. _vdpa_app_run_vm: > + > +Start the VMs > +~~~~~~~~~~~~~ > + > +.. code-block:: console > + > + qemu-system-x86_64 -cpu host -enable-kvm \ > + > + -mem-prealloc \ > + -chardev socket,id=3Dchar0,path=3D \ > + -netdev type=3Dvhost-user,id=3Dvdpa,chardev=3Dchar0 \ > + -device virtio-net-pci,netdev=3Dvdpa,mac=3D00:aa:bb:cc:dd:ee \ > + > +After the VMs launches, we can login the VMs and configure the ip, verif= y the > +network connection via ping or netperf. > + > +.. note:: > + Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA. > + > +Live Migration > +~~~~~~~~~~~~~~ > +vDPA supports cross-backend live migration, user can migrate SW vhost > backend > +VM to vDPA backend VM and vice versa. Here are the detailed steps. Assum= e > A is > +the source host with SW vhost VM and B is the destination host with vDPA= . > + > +1. Start vdpa sample and launch a VM with exact same parameters as the V= M > on A, > + in migration-listen mode: > + > +.. code-block:: console > + > + B: -incoming tcp:0:4444 (or other PORT)) > + > +2. Start the migration (on source host): > + > +.. code-block:: console > + > + A: (qemu) migrate -d tcp::4444 (or other PORT) > + > +3. Check the status (on source host): > + > +.. code-block:: console > + > + A: (qemu) info migrate > diff --git a/examples/Makefile b/examples/Makefile > index 481720cb6..356fcb1cd 100644 > --- a/examples/Makefile > +++ b/examples/Makefile > @@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y) > DIRS-$(CONFIG_RTE_LIBRTE_VHOST) +=3D tep_termination > endif > DIRS-$(CONFIG_RTE_LIBRTE_TIMER) +=3D timer > -DIRS-$(CONFIG_RTE_LIBRTE_VHOST) +=3D vhost vhost_scsi > +DIRS-$(CONFIG_RTE_LIBRTE_VHOST) +=3D vhost vhost_scsi vdpa > ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y) > DIRS-$(CONFIG_RTE_LIBRTE_VHOST) +=3D vhost_crypto > endif > diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile > new file mode 100644 > index 000000000..42672a2bc > --- /dev/null > +++ b/examples/vdpa/Makefile > @@ -0,0 +1,32 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright(c) 2018 Intel Corporation > + > +ifeq ($(RTE_SDK),) > +$(error "Please define RTE_SDK environment variable") > +endif > + > +# Default target, can be overridden by command line or environment > +RTE_TARGET ?=3D x86_64-native-linuxapp-gcc > + > +include $(RTE_SDK)/mk/rte.vars.mk > + > +ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp") > +$(info This application can only operate in a linuxapp environment, \ > +please change the definition of the RTE_TARGET environment variable) > +all: > +else > + > +# binary name > +APP =3D vdpa > + > +# all source are stored in SRCS-y > +SRCS-y :=3D main.c > + > +CFLAGS +=3D -O2 -D_FILE_OFFSET_BITS=3D64 > +CFLAGS +=3D $(WERROR_FLAGS) > +CFLAGS +=3D -D_GNU_SOURCE > +CFLAGS +=3D -DALLOW_EXPERIMENTAL_API > + > +include $(RTE_SDK)/mk/rte.extapp.mk > + [...] > + > +static int > +new_device(int vid) > +{ > + char ifname[MAX_PATH_LEN]; > + int i; > + > + rte_vhost_get_ifname(vid, ifname, sizeof(ifname)); > + for (i =3D 0; i < MAX_VDPA_SAMPLE_PORTS; i++) { > + if (strncmp(ifname, vports[i].ifname, MAX_PATH_LEN) =3D=3D 0) { > + printf("\nnew port %s, did: %d\n", > + ifname, vports[i].did); > + vports[i].vid =3D vid; > + break; > + } > + } > + > + if (i >=3D MAX_VDPA_SAMPLE_PORTS) > + return -1; > + > + return 0; > +} > + > +static void > +destroy_device(int vid) > +{ > + char ifname[MAX_PATH_LEN]; > + int i; > + > + rte_vhost_get_ifname(vid, ifname, sizeof(ifname)); > + for (i =3D 0; i < MAX_VDPA_SAMPLE_PORTS; i++) { > + if (strcmp(ifname, vports[i].ifname) =3D=3D 0) { > + printf("\ndestroy port %s, did: %d\n", > + ifname, vports[i].did); > + break; > + } > + } > +} > + > +static const struct vhost_device_ops vdpa_sample_devops =3D { > + .new_device =3D new_device, > + .destroy_device =3D destroy_device, > +}; > + > +static int > +start_vdpa(struct vdpa_port *vport) > +{ > + int ret; > + char *socket_path =3D vport->ifname; > + int did =3D vport->did; > + > + if (access(socket_path, F_OK) !=3D -1) { > + RTE_LOG(ERR, VDPA, > + "%s exists, please remove it or specify another file and > try again.\n", > + socket_path); > + return -1; > + } > + ret =3D rte_vhost_driver_register(socket_path, vport->flags); It would be great if we can provide "client mode" option for this sample to= create a client mode vdpa. > + if (ret !=3D 0) > + rte_exit(EXIT_FAILURE, > + "register driver failed: %s\n", > + socket_path); > + > + ret =3D rte_vhost_driver_callback_register(socket_path, > + &vdpa_sample_devops); > + if (ret !=3D 0) > + rte_exit(EXIT_FAILURE, > + "register driver ops failed: %s\n", > + socket_path); > + > + ret =3D rte_vhost_driver_attach_vdpa_device(socket_path, did); > + if (ret !=3D 0) > + rte_exit(EXIT_FAILURE, > + "attach vdpa device failed: %s\n", > + socket_path); > + > + if (rte_vhost_driver_start(socket_path) < 0) > + rte_exit(EXIT_FAILURE, > + "start vhost driver failed: %s\n", > + socket_path); > + return 0; > +} > + > +static void > +close_vdpa(struct vdpa_port *vport) > +{ > + int ret; > + char *socket_path =3D vport->ifname; > + > + ret =3D rte_vhost_driver_detach_vdpa_device(socket_path); > + if (ret !=3D 0) > + RTE_LOG(ERR, VDPA, > + "attach vdpa device failed: %s\n", > + socket_path); > + > + ret =3D rte_vhost_driver_unregister(socket_path); > + if (ret !=3D 0) > + RTE_LOG(ERR, VDPA, > + "Fail to unregister vhost driver for %s.\n", > + socket_path); > +} > + > +static void > +vdpa_sample_quit(void) > +{ > + int i; > + for (i =3D 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total); i++) { > + if (vports[i].ifname[0] !=3D '\0') > + close_vdpa(&vports[i]); > + } > +} > + > +static void > +signal_handler(int signum) > +{ > + if (signum =3D=3D SIGINT || signum =3D=3D SIGTERM) { > + printf("\nSignal %d received, preparing to exit...\n", signum); > + vdpa_sample_quit(); > + exit(0); > + } > +} > + > +/* interactive cmds */ > + > +/* *** Help command with introduction. *** */ > +struct cmd_help_result { > + cmdline_fixed_string_t help; > +}; > + > +static void cmd_help_parsed(__attribute__((unused)) void *parsed_result, > + struct cmdline *cl, > + __attribute__((unused)) void *data) > +{ > + cmdline_printf( > + cl, > + "\n" > + "The following commands are currently available:\n\n" > + "Control:\n" > + " help : Show interactive inst= ructions.\n" > + " list : list all available vd= pa devices.\n" > + " create : create a new vdpa > port.\n" > + " quit : exit vdpa sample app.= \n" > + ); > +} > + > +cmdline_parse_token_string_t cmd_help_help =3D > + TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help"); > + > +cmdline_parse_inst_t cmd_help =3D { > + .f =3D cmd_help_parsed, > + .data =3D NULL, > + .help_str =3D "show help", > + .tokens =3D { > + (void *)&cmd_help_help, > + NULL, > + }, > +}; > + > +/* *** List all available vdpa devices *** */ > +struct cmd_list_result { > + cmdline_fixed_string_t action; > +}; > + > +static void cmd_list_vdpa_devices_parsed( > + __attribute__((unused)) void *parsed_result, > + struct cmdline *cl, > + __attribute__((unused)) void *data) > +{ > + int did; > + uint32_t queue_num; > + uint64_t features; > + struct rte_vdpa_device *vdev; > + struct rte_pci_addr addr; > + > + cmdline_printf(cl, "device id\tdevice address\tqueue num\tsupported > features\n"); > + for (did =3D 0; did < dev_total; did++) { > + vdev =3D rte_vdpa_get_device(did); > + if (!vdev) > + continue; > + if (vdev->ops->get_queue_num(did, &queue_num) < 0) { > + RTE_LOG(ERR, VDPA, > + "failed to get vdpa queue number " > + "for device id %d.\n", did); > + continue; > + } > + if (vdev->ops->get_features(did, &features) < 0) { > + RTE_LOG(ERR, VDPA, > + "failed to get vdpa features " > + "for device id %d.\n", did); > + continue; > + } > + addr =3D vdev->addr.pci_addr; > + cmdline_printf(cl, "%d\t\t"PCI_PRI_FMT"\t%d\t\t0x%lx\n", did, > + addr.domain, addr.bus, addr.devid, > + addr.function, queue_num, features); Use "%PRIu32"for queue_num, and use "PRIx64" instead of "%lx", otherwise yo= u may have compile error on 32-bit machine. > + } > +} > + > +cmdline_parse_token_string_t cmd_action_list =3D > + TOKEN_STRING_INITIALIZER(struct cmd_list_result, action, "list"); > + > +cmdline_parse_inst_t cmd_list_vdpa_devices =3D { > + .f =3D cmd_list_vdpa_devices_parsed, /* function to call */ > + .data =3D NULL, /* 2nd arg of func */ This comment doesn't provide much info. Remove the comment for all the cmdline_parse_inst_t instances to keep align= ment. > + .help_str =3D "list all available vdpa devices", > + .tokens =3D { /* token list, NULL terminated */ > + (void *)&cmd_action_list, > + NULL, > + }, > +}; > + > +/* *** Create new vdpa port *** */ > +struct cmd_create_result { > + cmdline_fixed_string_t action; > + cmdline_fixed_string_t socket_path; > + cmdline_fixed_string_t bdf; > +}; > + > +static void cmd_create_vdpa_port_parsed(void *parsed_result, > + struct cmdline *cl, > + __attribute__((unused)) void *data) > +{ > + int did; > + struct cmd_create_result *res =3D parsed_result; > + struct rte_vdpa_dev_addr addr; > + > + strncpy(vports[devcnt].ifname, res->socket_path, MAX_PATH_LEN); > + if (rte_pci_addr_parse(res->bdf, &addr.pci_addr) !=3D 0) { > + cmdline_printf(cl, "Unable to parse the given bdf.\n"); > + return; > + } > + addr.type =3D PCI_ADDR; > + did =3D rte_vdpa_find_device_id(&addr); > + if (did < 0) { > + cmdline_printf(cl, "Unable to find vdpa devide id.\n"); > + return; > + } > + > + vports[devcnt].did =3D did; > + > + if (start_vdpa(&vports[devcnt]) =3D=3D 0) > + devcnt++; > +} > + > +cmdline_parse_token_string_t cmd_action_create =3D > + TOKEN_STRING_INITIALIZER(struct cmd_create_result, action, > "create"); > +cmdline_parse_token_string_t cmd_socket_path =3D > + TOKEN_STRING_INITIALIZER(struct cmd_create_result, socket_path, > NULL); > +cmdline_parse_token_string_t cmd_bdf =3D > + TOKEN_STRING_INITIALIZER(struct cmd_create_result, bdf, NULL); > + > +cmdline_parse_inst_t cmd_create_vdpa_port =3D { > + .f =3D cmd_create_vdpa_port_parsed, /* function to call */ > + .data =3D NULL, /* 2nd arg of func */ > + .help_str =3D "create a new vdpa port", > + .tokens =3D { /* token list, NULL terminated */ > + (void *)&cmd_action_create, > + (void *)&cmd_socket_path, > + (void *)&cmd_bdf, > + NULL, > + }, > +}; > + > +/* *** QUIT *** */ > +struct cmd_quit_result { > + cmdline_fixed_string_t quit; > +}; > + > +static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result, > + struct cmdline *cl, > + __attribute__((unused)) void *data) > +{ > + vdpa_sample_quit(); > + cmdline_quit(cl); > +} > + > +cmdline_parse_token_string_t cmd_quit_quit =3D > + TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit"); > + > +cmdline_parse_inst_t cmd_quit =3D { > + .f =3D cmd_quit_parsed, > + .data =3D NULL, > + .help_str =3D "quit: exit application", > + .tokens =3D { > + (void *)&cmd_quit_quit, > + NULL, > + }, > +}; > +cmdline_parse_ctx_t main_ctx[] =3D { > + (cmdline_parse_inst_t *)&cmd_help, > + (cmdline_parse_inst_t *)&cmd_list_vdpa_devices, > + (cmdline_parse_inst_t *)&cmd_create_vdpa_port, > + (cmdline_parse_inst_t *)&cmd_quit, > + NULL, > +}; > + > +int > +main(int argc, char *argv[]) > +{ > + char ch; > + int i; > + int ret; > + struct cmdline *cl; > + > + ret =3D rte_eal_init(argc, argv); > + if (ret < 0) > + rte_exit(EXIT_FAILURE, "eal init failed\n"); > + argc -=3D ret; > + argv +=3D ret; > + > + data_init(); > + > + signal(SIGINT, signal_handler); > + signal(SIGTERM, signal_handler); > + > + ret =3D parse_args(argc, argv); > + if (ret < 0) > + rte_exit(EXIT_FAILURE, "invalid argument\n"); > + > + if (interactive =3D=3D 1) { > + cl =3D cmdline_stdin_new(main_ctx, "vdpa> "); > + if (cl =3D=3D NULL) > + rte_panic("Cannot create cmdline instance\n"); > + cmdline_interact(cl); > + cmdline_stdin_exit(cl); > + } else { > + for (i =3D 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total); > + i++) { > + vports[i].did =3D i; > + snprintf(vports[i].ifname, MAX_PATH_LEN, "%s%d", > + iface, i); > + > + start_vdpa(&vports[i]); > + } > + > + printf("enter \'q\' to quit\n"); > + while (scanf("%c", &ch)) { > + if (ch =3D=3D 'q') > + break; > + while (ch !=3D '\n') { > + if (scanf("%c", &ch)) > + printf("%c", ch); > + } > + printf("enter \'q\' to quit\n"); > + } Do vdpa_sample_quit here to stop the vhost session and delete the vhost soc= kets. > + } > + > + return 0; > +} > diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build > new file mode 100644 > index 000000000..2e38a069f > --- /dev/null > +++ b/examples/vdpa/meson.build > @@ -0,0 +1,16 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright(c) 2018 Intel Corporation > + > +# meson file, for building this example as part of a main DPDK build. > +# > +# To build this example as a standalone application with an already-inst= alled > +# DPDK instance, use 'make' > + > +if host_machine.system() !=3D 'linux' > + build =3D false > +endif > +deps +=3D 'vhost' > +allow_experimental_apis =3D true > +sources =3D files( > + 'main.c' > +) > \ No newline at end of file > -- > 2.17.1 Do not forget to add this into 18.11 release_note. BRs, Xiao