DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] [PATCH 00/21] Vhost-user: Implement device IOTLB support
@ 2017-08-31  9:50 Maxime Coquelin
  2017-08-31  9:50 ` [dpdk-dev] [PATCH 01/21] Revert "vhost: workaround MQ fails to startup" Maxime Coquelin
                   ` (21 more replies)
  0 siblings, 22 replies; 59+ messages in thread
From: Maxime Coquelin @ 2017-08-31  9:50 UTC (permalink / raw)
  To: dev, yliu, jfreiman, tiwei.bie; +Cc: mst, vkaplans, jasowang, Maxime Coquelin

This first non-RFC, which targets v17.11, adds support for
VIRTIO_F_IOMMU_PLATFORM feature, by implementing device IOTLB in the
vhost-user backend. It improves the guest safety by enabling the
possibility to isolate the Virtio device.

It makes possible to use Virtio PMD in guest with using VFIO driver
without enable_unsafe_noiommu_mode parameter set, so that the DPDK
application on guest can only access memory its has been allowed to,
and preventing malicious/buggy DPDK application in guest to make
vhost-user backend write random guest memory. Note that Virtio-net
Kernel driver also support IOMMU.

The series depends on Qemu's "vhost-user: Specify and implement
device IOTLB support" [0], available upstream and which will be part
of Qemu v2.10 release.

Performance-wise, even if this RFC has still room for optimizations,
no performance degradation is noticed with static mappings (i.e. DPDK
on guest) with PVP benchmark:
	Traffic Generator: Moongen (lua-trafficgen)
	Acceptable Loss: 0.005%
	Validation run time: 1 min
	Guest DPDK version/commit: v17.05
	QEMU version/commit: master (6db174aed1fd)
	Virtio features: default
	CPU: Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
	NIC: 2 x X710
	Page size: 1G host/1G guest
	Results (bidirectional, total of the two flows):
	 - base: 18.8Mpps
	 - base + IOTLB series, IOMMU OFF: 18.8Mpps
	 - base + IOTLB series, IOMMU ON: 18.8Mpps (14.5Mpps w/o PATCH 21/21)

This is explained because IOTLB misses, which are very costly, only
happen at startup time. Indeed, once used, the buffers are not
invalidated, so if the IOTLB cache is large enough, there will be only
cache hits. Also, the use of 1G huge pages improves the IOTLB cache
searching time by reducing the number of entries.

With 2M hugepages, a performance degradation is seen with IOMMU on:
	Traffic Generator: Moongen (lua-trafficgen)
	Acceptable Loss: 0.005%
	Validation run time: 1 min
	Guest DPDK version/commit: v17.05
	QEMU version/commit: master (6db174aed1fd)
	Virtio features: default
	CPU: Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
	NIC: 2 x X710
	Page size: 2M host/2M guest
	Results (bidirectional, total of the two flows):
	 - base: 18.8Mpps
	 - base + IOTLB series, IOMMU OFF: 18.8Mpps
	 - base + IOTLB series, IOMMU ON: 13.5Mpps (12.4Mpps wo PATCH 21/21)

A possible improvement would be to merge contiguous IOTLB entries sharing
the same permissions. A very rough patch implementing this idea fixes
the performance degradation (18.8Mpps), but the required work to clean
it would delay this series after v17.11.

With dynamic mappings (i.e. Virtio-net kernel driver), this is another
story. The performance is so poor it makes it almost unusable. Indeed,
since the Kernel driver unmaps the buffers as soon as they are handled,
almost all descriptors buffers addresses translations result in an IOTLB
miss. There is not much that can be done on DPDK side, except maybe
batching IOTLB miss requests no to break bursts, but it would require
a big rework. In Qemu, we may consider enabling IOMMU MAP notifications,
so that DPDK receives the IOTLB updates without having to send IOTLB miss
request.

Regarding the design choices:
 - I initially intended to use userspace RCU library[1] for the cache
implementation, but it would have added an external dependency, and the
lib is not available in all distros. Qemu for example got rid of this
dependency by copying some of the userspace RCU lib parts into Qemu tree,
but this is not possible with DPDK due to licensing issues (RCU lib is
LGPL v2). Thanks to Jason advice, I implemented the cache using rd/wr
locks.
 - I initially implemented a per-device IOTLB cache, but the concurrent
acccesses on the IOTLB lock had huge impact on performance (~-40% in
bidirectionnal, expect even worse with multiqueue). I move to a per-
virtqueue IOTLB design, which prevents this concurrency.
 - The slave IOTLB miss request supports reply-ack feature in spec, but
this version doesn't block or busy-wait for the corresponding update so
that other queues sharing the same lcore can be processed in the meantime.

For those who would like to test the series, I made it available on
gitlab[2] (vhost_user_iotlb_v1 tag). The guest kernel command line requires
the intel_iommu=on parameter, and the guest should be started with and
iommu device attached to the virtio-net device. For example:

./qemu-system-x86_64 \
  -enable-kvm -m 4096 -smp 2 \
  -M q35,kernel-irqchip=split \
  -cpu host \
  -device intel-iommu,device-iotlb=on,intremap \
  -device ioh3420,id=root.1,chassis=1 \
  -chardev socket,id=char0,path=/tmp/vhost-user1 \
  -netdev type=vhost-user,id=hn2,chardev=char0 \
  -device virtio-net-pci,netdev=hn2,id=v0,mq=off,mac=$MAC,bus=root.1,disable-modern=off,disable-legacy=on,iommu_platform=on,ats=on \
...

[0]: https://lists.gnu.org/archive/html/qemu-devel/2017-06/msg00520.html
[1]: http://liburcu.org/
[2]: https://gitlab.com/mcoquelin/dpdk-next-virtio/commits/vhost_user_iotlb_v1

Changes since RFC:
==================
- Fix memory leak in error patch reported by Jens
- Rework wait for IOTLB update by stopping the burst to let other
  queues to be processed, if any. It implies the introduction an
  iotlb_pending_list, so that iotlb miss requests aren't sent multiple
  times for a same address.
- Optimize iotlb lock usage to recover to same as IOMMU off performance
- Fix device locking issue in rte_vhost_dequeue_burst() error path
- Change virtio_dev_rx error handling for consistency with mergeable rx,
  and to ease returning in case of IOTLB misses.
- Fix checkpatch warnings reported by checkpatch@dpdk.org

Maxime Coquelin (21):
  Revert "vhost: workaround MQ fails to startup"
  vhost: make error handling consistent in rx path
  vhost: protect virtio_net device struct
  vhost: prepare send_vhost_message() to slave requests
  vhost: add support to slave requests channel
  vhost: declare missing IOMMU-related definitions for old kernels
  vhost: add iotlb helper functions
  vhost: iotlb: add pending miss request list and helpers
  vhost-user: add support to IOTLB miss slave requests
  vhost: initialize vrings IOTLB caches
  vhost-user: handle IOTLB update and invalidate requests
  vhost: introduce guest IOVA to backend VA helper
  vhost: use the guest IOVA to host VA helper
  vhost: enable rings at the right time
  vhost: don't dereference invalid dev pointer after its reallocation
  vhost: postpone rings addresses translation
  vhost-user: translate ring addresses when IOMMU enabled
  vhost-user: iommu: postpone device creation until ring are mapped
  vhost: iommu: Invalidate vring in case of matching IOTLB invalidate
  vhost: enable IOMMU support
  vhost: iotlb: reduce iotlb read lock usage

 lib/librte_vhost/Makefile     |   4 +-
 lib/librte_vhost/iotlb.c      | 315 +++++++++++++++++++++++++++++++++++
 lib/librte_vhost/iotlb.h      |  64 +++++++
 lib/librte_vhost/vhost.c      | 340 ++++++++++++++++++++++++++++++++------
 lib/librte_vhost/vhost.h      |  53 +++++-
 lib/librte_vhost/vhost_user.c | 376 ++++++++++++++++++++++++++++++++----------
 lib/librte_vhost/vhost_user.h |  20 ++-
 lib/librte_vhost/virtio_net.c | 129 +++++++++++----
 8 files changed, 1130 insertions(+), 171 deletions(-)
 create mode 100644 lib/librte_vhost/iotlb.c
 create mode 100644 lib/librte_vhost/iotlb.h

-- 
2.13.3

^ permalink raw reply	[flat|nested] 59+ messages in thread

end of thread, other threads:[~2017-09-24 10:42 UTC | newest]

Thread overview: 59+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-08-31  9:50 [dpdk-dev] [PATCH 00/21] Vhost-user: Implement device IOTLB support Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 01/21] Revert "vhost: workaround MQ fails to startup" Maxime Coquelin
2017-09-07 11:54   ` Yuanhan Liu
2017-09-07 12:59     ` Maxime Coquelin
2017-09-24 10:41       ` Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 02/21] vhost: make error handling consistent in rx path Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 03/21] vhost: protect virtio_net device struct Maxime Coquelin
2017-09-05  4:45   ` Tiwei Bie
2017-09-05  9:24     ` Maxime Coquelin
2017-09-05 10:07       ` Tiwei Bie
2017-09-05 11:00         ` Maxime Coquelin
2017-09-06  1:15           ` Tiwei Bie
2017-09-06  2:59             ` Stephen Hemminger
2017-09-06  7:50               ` Maxime Coquelin
2017-09-06  7:15             ` Maxime Coquelin
2017-09-06  7:30               ` Tiwei Bie
2017-09-06 20:02                 ` Maxime Coquelin
2017-09-07  5:08                   ` Tiwei Bie
2017-09-07 13:44   ` Yuanhan Liu
2017-09-07 14:01     ` Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 04/21] vhost: prepare send_vhost_message() to slave requests Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 05/21] vhost: add support to slave requests channel Maxime Coquelin
2017-09-05  4:19   ` Tiwei Bie
2017-09-05  8:18     ` Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 06/21] vhost: declare missing IOMMU-related definitions for old kernels Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 07/21] vhost: add iotlb helper functions Maxime Coquelin
2017-09-05  6:02   ` Tiwei Bie
2017-09-05 15:16     ` Maxime Coquelin
2017-09-08  8:08   ` Yuanhan Liu
2017-09-08  8:24     ` Maxime Coquelin
2017-09-08  8:36       ` Yuanhan Liu
2017-09-08  8:50         ` Maxime Coquelin
2017-09-08  9:21           ` Yuanhan Liu
2017-09-08  9:28             ` Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 08/21] vhost: iotlb: add pending miss request list and helpers Maxime Coquelin
2017-09-05  7:11   ` Tiwei Bie
2017-09-05 15:18     ` Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 09/21] vhost-user: add support to IOTLB miss slave requests Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 10/21] vhost: initialize vrings IOTLB caches Maxime Coquelin
2017-09-04 13:57   ` Remy Horton
2017-09-04 15:45     ` Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 11/21] vhost-user: handle IOTLB update and invalidate requests Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 12/21] vhost: introduce guest IOVA to backend VA helper Maxime Coquelin
2017-09-05  4:14   ` Tiwei Bie
2017-09-05  7:05     ` Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 13/21] vhost: use the guest IOVA to host " Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 14/21] vhost: enable rings at the right time Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 15/21] vhost: don't dereference invalid dev pointer after its reallocation Maxime Coquelin
2017-09-04 13:58   ` Remy Horton
2017-08-31  9:50 ` [dpdk-dev] [PATCH 16/21] vhost: postpone rings addresses translation Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 17/21] vhost-user: translate ring addresses when IOMMU enabled Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 18/21] vhost-user: iommu: postpone device creation until ring are mapped Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 19/21] vhost: iommu: Invalidate vring in case of matching IOTLB invalidate Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 20/21] vhost: enable IOMMU support Maxime Coquelin
2017-08-31  9:50 ` [dpdk-dev] [PATCH 21/21] vhost: iotlb: reduce iotlb read lock usage Maxime Coquelin
2017-09-11  4:18   ` Yuanhan Liu
2017-09-11  7:34     ` Maxime Coquelin
2017-09-11  9:39       ` Yuanhan Liu
2017-09-04 13:58 ` [dpdk-dev] [PATCH 00/21] Vhost-user: Implement device IOTLB support Remy Horton

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).