From: Yuanhan Liu <yuanhan.liu@linux.intel.com>
To: dev@dpdk.org
Cc: Maxime Coquelin <maxime.coquelin@redhat.com>,
Harris James R <james.r.harris@intel.com>,
Liu Changpeng <changpeng.liu@intel.com>,
Yuanhan Liu <yuanhan.liu@linux.intel.com>
Subject: [dpdk-dev] [PATCH v2 00/22] vhost: generic vhost API
Date: Thu, 23 Mar 2017 15:10:37 +0800 [thread overview]
Message-ID: <1490253059-28112-1-git-send-email-yuanhan.liu@linux.intel.com> (raw)
In-Reply-To: <1488534682-3494-1-git-send-email-yuanhan.liu@linux.intel.com>
This patchset makes DPDK vhost library be generic enough, so that we could
build other vhost-user drivers on top of it. For example, SPDK (Storage
Performance Development Kit) is trying to enable vhost-user SCSI.
The basic idea is, let DPDK vhost be a vhost-user agent. It stores all
the info about the virtio device (i.e. vring address, negotiated features,
etc) and let the specific vhost-user driver to fetch them (by the API
provided by DPDK vhost lib). With those info being provided, the vhost-user
driver then could get/put vring entries, thus, it could exchange data
between the guest and host.
The last patch demonstrates how to use these new APIs to implement a
very simple vhost-user net driver, without any fancy features enabled.
Major API/ABI Changes summary
=============================
- some renames
* "struct virtio_net_device_ops" ==> "struct vhost_device_ops"
* "rte_virtio_net.h" ==> "rte_vhost.h"
- driver related APIs are bond with the socket file
* rte_vhost_driver_set_features(socket_file, features);
* rte_vhost_driver_get_features(socket_file, features);
* rte_vhost_driver_enable_features(socket_file, features)
* rte_vhost_driver_disable_features(socket_file, features)
* rte_vhost_driver_callback_register(socket_file, notify_ops);
* rte_vhost_driver_start(socket_file);
This function replaces rte_vhost_driver_session_start(). Check patch
18 for more information.
- new APIs to fetch guest and vring info
* rte_vhost_get_mem_table(vid, mem);
* rte_vhost_get_negotiated_features(vid);
* rte_vhost_get_vhost_vring(vid, vring_idx, vring);
- new exported structures
* struct rte_vhost_vring
* struct rte_vhost_mem_region
* struct rte_vhost_memory
- a new device ops callback: features_changed().
Change log
==========
v2: - rebase
- updated release note
- updated API comments
- renamed rte_vhost_get_vhost_memory to rte_vhost_get_mem_table
- added a new device callback: features_changed(), bascially for live
migration support
- introduced rte_vhost_driver_start() to start a specific driver
- misc fixes
Some design choices
===================
While making this patchset, I met quite few design choices and here are
two of them, with the issue and the reason I made such choices provided.
Please let me know if you have any comments (or better ideas).
Export public structures or not
-------------------------------
I made an ABI refactor last time (v16.07): move all the structures
internally and let applications use a "vid" to reference the internal
struct. With that, I hope we could never worry about the annoying ABI
issues.
It works great (and as expected) since then, as far as we only support
virito-net, as far as we can handle all the descs inside vhost lib. It
becomes problematic when a user wants to implement a vhost-user driver
somewhere. For example, it needs do the GPA to VVA translation. Without
any structs exported, some functions like gpa_to_vva() can't be inlined.
Calling it would be costly, especially it's a function we have to invoke
for processing each vring desc.
For that reason, the guest memory regions are exported. With that, the
gpa_to_vva could be inlined.
Add helper functions to fetch/update descs or not
-------------------------------------------------
I intended to do it like this way: introduce one function to get @count
of descs from a specific vring and another one to update the used descs.
It's something like
rte_vhost_vring_get_descs(vid, vring_idx, count, offset, iov, descs);
rte_vhost_vring_update_used_descs(vid, vring_idx, count, offset, descs);
With that, vhost-user driver programmer's task would be easier, as he/she
doesn't have to parse the descs any more (such as to handle indirect desc).
But judging that virtio 1.1 is just emerged and it proposes a completely
ring layout, and most importantly, the vring desc structure is also changed,
I'd like to hold the introducation of such two functions. Otherwise, it's
very likely the two will be invalid when virtio 1.1 is out. Though I think
it may could be addressed with a care design, something like making the IOV
generic enough:
struct rte_vhost_iov {
uint64_t gpa;
uint64_t vva;
uint64_t len;
};
Instead, I go with the other way: introduce few APIs to export all the vring
infos (vring size, vring addr, callfd, etc), and let the vhost-user driver
read and update the descs. Those info could be passed to vhost-user driver
by introducing one API for each, but for saving few APIs and reducing few
calls for the programmer, I packed few key fields into a new structure, so
that it can be fetched with one call:
struct rte_vhost_vring {
struct vring_desc *desc;
struct vring_avail *avail;
struct vring_used *used;
uint64_t log_guest_addr;
int callfd;
int kickfd;
uint16_t size;
};
When virtio 1.1 comes out, likely a simple change like following would
just work:
struct rte_vhost_vring {
union {
struct {
struct vring_desc *desc;
struct vring_avail *avail;
struct vring_used *used;
uint64_t log_guest_addr;
};
struct desc *desc_1_1; /* vring addr for virtio 1.1 */
};
int callfd;
int kickfd;
uint16_t size;
};
AFAIK, it's not an ABI breakage. Even if it does, we could introduce a new
API to get the virtio 1.1 ring address.
Those fields are the minimum set I got for a specific vring, with the mind
it would bring the minimum chance to break ABI for future extension. If we
need more info, we could introduce a new API.
OTOH, for getting the best performance, the two functions also have to be
inlined ("vid + vring_idx" combo is replaced with "vring"):
rte_vhost_vring_get_descs(vring, count, offset, iov, descs);
rte_vhost_vring_update_used_descs(vring, count, offset, descs);
That said, one way or another, we have to export rte_vhost_vring struct.
For this reason, I didn't rush into introducing the two APIs.
--yliu
---
Yuanhan Liu (22):
vhost: introduce driver features related APIs
net/vhost: remove feature related APIs
vhost: use new APIs to handle features
vhost: make notify ops per vhost driver
vhost: export guest memory regions
vhost: introduce API to fetch negotiated features
vhost: export vhost vring info
vhost: export API to translate gpa to vva
vhost: turn queue pair to vring
vhost: export the number of vrings
vhost: move the device ready check at proper place
vhost: drop the Rx and Tx queue macro
vhost: do not include net specific headers
vhost: rename device ops struct
vhost: rename virtio-net to vhost
vhost: add features changed callback
vhost: export APIs for live migration support
vhost: introduce API to start a specific driver
vhost: rename header file
vhost: workaround the build dependency on mbuf header
vhost: do not destroy device on repeat mem table message
examples/vhost: demonstrate the new generic vhost APIs
doc/guides/prog_guide/vhost_lib.rst | 42 +--
doc/guides/rel_notes/deprecation.rst | 9 -
doc/guides/rel_notes/release_17_05.rst | 40 +++
drivers/net/vhost/rte_eth_vhost.c | 101 ++-----
drivers/net/vhost/rte_eth_vhost.h | 32 +--
drivers/net/vhost/rte_pmd_vhost_version.map | 3 -
examples/tep_termination/main.c | 23 +-
examples/tep_termination/main.h | 2 +
examples/tep_termination/vxlan_setup.c | 2 +-
examples/vhost/Makefile | 2 +-
examples/vhost/main.c | 100 +++++--
examples/vhost/main.h | 33 ++-
examples/vhost/virtio_net.c | 405 ++++++++++++++++++++++++++
lib/librte_vhost/Makefile | 4 +-
lib/librte_vhost/fd_man.c | 9 +-
lib/librte_vhost/fd_man.h | 2 +-
lib/librte_vhost/rte_vhost.h | 423 ++++++++++++++++++++++++++++
lib/librte_vhost/rte_vhost_version.map | 17 +-
lib/librte_vhost/rte_virtio_net.h | 208 --------------
lib/librte_vhost/socket.c | 222 ++++++++++++---
lib/librte_vhost/vhost.c | 229 ++++++++-------
lib/librte_vhost/vhost.h | 113 +++++---
lib/librte_vhost/vhost_user.c | 115 ++++----
lib/librte_vhost/vhost_user.h | 2 +-
lib/librte_vhost/virtio_net.c | 71 ++---
25 files changed, 1523 insertions(+), 686 deletions(-)
create mode 100644 examples/vhost/virtio_net.c
create mode 100644 lib/librte_vhost/rte_vhost.h
delete mode 100644 lib/librte_vhost/rte_virtio_net.h
--
1.9.0
next prev parent reply other threads:[~2017-03-23 7:13 UTC|newest]
Thread overview: 135+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-03-03 9:51 [dpdk-dev] [PATCH 00/17] " Yuanhan Liu
2017-03-03 9:51 ` [dpdk-dev] [PATCH 01/17] vhost: introduce driver features related APIs Yuanhan Liu
2017-03-14 9:46 ` Maxime Coquelin
2017-03-14 9:53 ` Maxime Coquelin
2017-03-16 7:08 ` Yuanhan Liu
2017-03-16 9:18 ` Maxime Coquelin
2017-03-17 5:50 ` Yuanhan Liu
2017-03-03 9:51 ` [dpdk-dev] [PATCH 02/17] net/vhost: remove feature " Yuanhan Liu
2017-03-14 10:15 ` Maxime Coquelin
2017-03-03 9:51 ` [dpdk-dev] [PATCH 03/17] vhost: use new APIs to handle features Yuanhan Liu
2017-03-14 10:43 ` Maxime Coquelin
2017-03-16 7:43 ` Yuanhan Liu
2017-03-16 9:39 ` Maxime Coquelin
2017-03-17 5:48 ` Yuanhan Liu
2017-03-03 9:51 ` [dpdk-dev] [PATCH 04/17] vhost: make notify ops per vhost driver Yuanhan Liu
2017-03-14 10:55 ` Maxime Coquelin
2017-03-16 7:50 ` Yuanhan Liu
2017-03-03 9:51 ` [dpdk-dev] [PATCH 05/17] vhost: export guest memory regions Yuanhan Liu
2017-03-14 11:00 ` Maxime Coquelin
2017-03-03 9:51 ` [dpdk-dev] [PATCH 06/17] vhost: introduce API to fetch negotiated features Yuanhan Liu
2017-03-14 11:02 ` Maxime Coquelin
2017-03-16 7:35 ` Yuanhan Liu
2017-03-16 9:22 ` Maxime Coquelin
2017-03-17 5:49 ` Yuanhan Liu
2017-03-03 9:51 ` [dpdk-dev] [PATCH 07/17] vhost: export vhost vring info Yuanhan Liu
2017-03-14 12:11 ` Maxime Coquelin
2017-03-16 7:24 ` Yuanhan Liu
2017-03-16 9:20 ` Maxime Coquelin
2017-03-03 9:51 ` [dpdk-dev] [PATCH 08/17] vhost: export API to translate gpa to vva Yuanhan Liu
2017-03-14 12:24 ` Maxime Coquelin
2017-03-03 9:51 ` [dpdk-dev] [PATCH 09/17] vhost: turn queue pair to vring Yuanhan Liu
2017-03-14 12:31 ` Maxime Coquelin
2017-03-03 9:51 ` [dpdk-dev] [PATCH 10/17] vhost: export the number of vrings Yuanhan Liu
2017-03-14 12:33 ` Maxime Coquelin
2017-03-03 9:51 ` [dpdk-dev] [PATCH 11/17] vhost: move the device ready check at proper place Yuanhan Liu
2017-03-14 12:37 ` Maxime Coquelin
2017-03-03 9:51 ` [dpdk-dev] [PATCH 12/17] vhost: drop the Rx and Tx queue macro Yuanhan Liu
2017-03-14 12:42 ` Maxime Coquelin
2017-03-03 9:51 ` [dpdk-dev] [PATCH 13/17] vhost: do not include net specific headers Yuanhan Liu
2017-03-14 12:46 ` Maxime Coquelin
2017-03-20 7:32 ` Liu, Changpeng
2017-03-22 6:21 ` Yuanhan Liu
2017-03-03 9:51 ` [dpdk-dev] [PATCH 14/17] vhost: rename device ops struct Yuanhan Liu
2017-03-14 12:48 ` Maxime Coquelin
2017-03-03 9:51 ` [dpdk-dev] [PATCH 15/17] vhost: rename virtio-net to vhost Yuanhan Liu
2017-03-14 12:50 ` Maxime Coquelin
2017-03-03 9:51 ` [dpdk-dev] [PATCH 16/17] vhost: rename header file Yuanhan Liu
2017-03-14 12:59 ` Maxime Coquelin
2017-03-20 5:35 ` Yuanhan Liu
2017-03-03 9:51 ` [dpdk-dev] [PATCH 17/17] examples/vhost: demonstrate the new generic vhost APIs Yuanhan Liu
2017-03-23 7:10 ` Yuanhan Liu [this message]
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 01/22] vhost: introduce driver features related APIs Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 02/22] net/vhost: remove feature " Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 03/22] vhost: use new APIs to handle features Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 04/22] vhost: make notify ops per vhost driver Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 05/22] vhost: export guest memory regions Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 06/22] vhost: introduce API to fetch negotiated features Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 07/22] vhost: export vhost vring info Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 08/22] vhost: export API to translate gpa to vva Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 09/22] vhost: turn queue pair to vring Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 10/22] vhost: export the number of vrings Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 11/22] vhost: move the device ready check at proper place Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 12/22] vhost: drop the Rx and Tx queue macro Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 13/22] vhost: do not include net specific headers Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 14/22] vhost: rename device ops struct Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 15/22] vhost: rename virtio-net to vhost Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 16/22] vhost: add features changed callback Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 17/22] vhost: export APIs for live migration support Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 18/22] vhost: introduce API to start a specific driver Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 19/22] vhost: rename header file Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 20/22] vhost: workaround the build dependency on mbuf header Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 21/22] vhost: do not destroy device on repeat mem table message Yuanhan Liu
2017-03-23 7:10 ` [dpdk-dev] [PATCH v2 22/22] examples/vhost: demonstrate the new generic vhost APIs Yuanhan Liu
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 00/22] vhost: generic vhost API Yuanhan Liu
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 01/22] vhost: introduce driver features related APIs Yuanhan Liu
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 02/22] net/vhost: remove feature " Yuanhan Liu
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 03/22] vhost: use new APIs to handle features Yuanhan Liu
2017-03-29 14:57 ` Maxime Coquelin
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 04/22] vhost: make notify ops per vhost driver Yuanhan Liu
2017-03-29 15:03 ` Maxime Coquelin
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 05/22] vhost: export guest memory regions Yuanhan Liu
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 06/22] vhost: introduce API to fetch negotiated features Yuanhan Liu
2017-03-31 7:45 ` Maxime Coquelin
2017-03-31 8:51 ` Yuanhan Liu
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 07/22] vhost: export vhost vring info Yuanhan Liu
2017-03-31 7:48 ` Maxime Coquelin
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 08/22] vhost: export API to translate gpa to vva Yuanhan Liu
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 09/22] vhost: turn queue pair to vring Yuanhan Liu
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 10/22] vhost: export the number of vrings Yuanhan Liu
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 11/22] vhost: move the device ready check at proper place Yuanhan Liu
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 12/22] vhost: drop the Rx and Tx queue macro Yuanhan Liu
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 13/22] vhost: do not include net specific headers Yuanhan Liu
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 14/22] vhost: rename device ops struct Yuanhan Liu
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 15/22] vhost: rename virtio-net to vhost Yuanhan Liu
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 16/22] vhost: add features changed callback Yuanhan Liu
2017-03-31 7:50 ` Maxime Coquelin
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 17/22] vhost: export APIs for live migration support Yuanhan Liu
2017-03-31 8:05 ` Maxime Coquelin
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 18/22] vhost: introduce API to start a specific driver Yuanhan Liu
2017-03-31 9:11 ` Maxime Coquelin
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 19/22] vhost: rename header file Yuanhan Liu
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 20/22] vhost: workaround the build dependency on mbuf header Yuanhan Liu
2017-03-31 9:13 ` Maxime Coquelin
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 21/22] vhost: do not destroy device on repeat mem table message Yuanhan Liu
2017-03-31 9:26 ` Maxime Coquelin
2017-03-28 12:45 ` [dpdk-dev] [PATCH v3 22/22] examples/vhost: demonstrate the new generic vhost APIs Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 00/22] vhost: generic vhost API Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 01/22] vhost: introduce driver features related APIs Yuanhan Liu
2017-04-05 0:01 ` Thomas Monjalon
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 02/22] net/vhost: remove feature " Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 03/22] vhost: use new APIs to handle features Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 04/22] vhost: make notify ops per vhost driver Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 05/22] vhost: export guest memory regions Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 06/22] vhost: introduce API to fetch negotiated features Yuanhan Liu
2017-04-01 8:28 ` Maxime Coquelin
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 07/22] vhost: export vhost vring info Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 08/22] vhost: export API to translate gpa to vva Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 09/22] vhost: turn queue pair to vring Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 10/22] vhost: export the number of vrings Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 11/22] vhost: move the device ready check at proper place Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 12/22] vhost: drop the Rx and Tx queue macro Yuanhan Liu
2017-04-05 0:17 ` Thomas Monjalon
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 13/22] vhost: do not include net specific headers Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 14/22] vhost: rename device ops struct Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 15/22] vhost: rename virtio-net to vhost Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 16/22] vhost: add features changed callback Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 17/22] vhost: export APIs for live migration support Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 18/22] vhost: introduce API to start a specific driver Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 19/22] vhost: rename header file Yuanhan Liu
2017-04-05 0:26 ` Thomas Monjalon
2017-04-05 5:16 ` Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 20/22] vhost: workaround the build dependency on mbuf header Yuanhan Liu
2017-04-01 7:22 ` [dpdk-dev] [PATCH v4 21/22] vhost: do not destroy device on repeat mem table message Yuanhan Liu
2017-04-01 7:23 ` [dpdk-dev] [PATCH v4 22/22] examples/vhost: demonstrate the new generic vhost APIs Yuanhan Liu
2017-04-01 8:44 ` [dpdk-dev] [PATCH v4 00/22] vhost: generic vhost API Yuanhan Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1490253059-28112-1-git-send-email-yuanhan.liu@linux.intel.com \
--to=yuanhan.liu@linux.intel.com \
--cc=changpeng.liu@intel.com \
--cc=dev@dpdk.org \
--cc=james.r.harris@intel.com \
--cc=maxime.coquelin@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).