From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id 0393E1B1A8 for ; Sun, 4 Feb 2018 15:56:46 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 04 Feb 2018 06:56:44 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,458,1511856000"; d="scan'208";a="28692302" Received: from dpdk-xiao-1.sh.intel.com ([10.67.110.153]) by orsmga001.jf.intel.com with ESMTP; 04 Feb 2018 06:56:42 -0800 From: Xiao Wang To: dev@dpdk.org Cc: jianfeng.tan@intel.com, tiwei.bie@intel.com, maxime.coquelin@redhat.com, yliu@fridaylinux.org, cunming.liang@intel.com, dan.daly@intel.com, zhihong.wang@intel.com, Xiao Wang Date: Sun, 4 Feb 2018 22:55:39 +0800 Message-Id: <20180204145542.38345-1-xiao.w.wang@intel.com> X-Mailer: git-send-email 2.15.1 Subject: [dpdk-dev] [PATCH 0/3] add vDPA sample driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Feb 2018 14:56:55 -0000 This patch set has dependency on the vhost lib patch: http://dpdk.org/dev/patchwork/patch/34872/ This patch set shows a reference sample of making vDPA device driver , this driver uses a QEMU-emulated virtio-net PCI device as vDPA device, and make it serve as a backend for a virtio-net pci device in nested VM. The key driver ops implemented are: * vdpa_virtio_eng_init Mapping virtio pci device with VFIO into userspace, and read device capability and intialize internal data. * vdpa_virtio_eng_uninit Release the mapped device. * vdpa_virtio_info_query Device capability reporting, e.g. queue number, features. * vdpa_virtio_dev_config With the guest virtio information provideed by vhost lib, this function configures device and IOMMU to set up vhost datapath, which includes: Rx/Tx vring, VFIO interrupt, kick relay. * vdpa_virtio_dev_close Unset the stuff that are configured previously by dev_conf. This driver requires the virtio device supports VIRTIO_F_IOMMU_PLATFORM , because the buffer address written in desc is IOVA. Because vDPA driver needs to set up MSI-X vector to interrupt the guest, only vfio-pci is supported currently. Below are setup steps for your reference: 1. Make sure your kernnel vhost module and QEMU support vIOMMU. - OS: CentOS 7.4 - QEMU: 2.10.1 - Guest OS: CentOS 7.2 - Nested VM OS: CentOS 7.2 2. enable VT-x feature for vCPU in VM. modprobe kvm_intel nested=1 3. Start a VM with a virtio-net-pci device. ./qemu-2.10.1/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host \ -machine q35 \ -device intel-iommu \ -netdev tap,id=mytap,ifname=vdpa,vhostforce=on \ -device virtio-net-pci,netdev=mytap,mac=00:aa:bb:cc:dd:ee,\ disable-modern=off,disable-legacy=on,iommu_platform=on \ 4. Bind VFIO-pci to virtio_net_pci device a) login to VM; b) modprobe vfio-pci c) rmmod vfio_iommu_type1 d) modprobe vfio_iommu_type1 allow_unsafe_interrupts=1 e) ./usertools/dpdk-devbind.py -b vfio-pci 00:03.0 5. Start vdpa sample ./examples/vdpa/build/vdpa -c 0x2 -n 4 --socket-mem 1024 --no-pci \ --vdev "net_vdpa_virtio_pci0,bdf=0000:00:03.0" -- --bdf 0000:00:03.0 \ --iface /tmp/vhost-user- --devcnt 1 --queue 1 6. Start nested VM ./qemu-2.10.1/x86_64-softmmu/qemu-system-x86_64 -cpu host -enable-kvm \ -mem-prealloc \ -chardev socket,id=char0,path=/tmp/vhost-user-0 \ -netdev type=vhost-user,id=vdpa,chardev=char0,vhostforce \ -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee \ 7. Login the nested VM, and verify the virtio in nested VM can communicate with tap device on host. Xiao Wang (3): bus/pci: expose API for vDPA net/vdpa_virtio_pci: introduce vdpa sample driver examples/vdpa: add a new sample for vdpa config/common_base | 6 + config/common_linuxapp | 1 + drivers/bus/pci/Makefile | 1 + drivers/bus/pci/linux/pci.c | 10 +- drivers/bus/pci/linux/pci_init.h | 22 +- drivers/bus/pci/linux/pci_vfio.c | 5 +- drivers/bus/pci/rte_bus_pci_version.map | 13 + drivers/net/Makefile | 1 + drivers/net/vdpa_virtio_pci/Makefile | 31 + .../net/vdpa_virtio_pci/rte_eth_vdpa_virtio_pci.c | 1527 ++++++++++++++++++++ .../rte_vdpa_virtio_pci_version.map | 4 + examples/vdpa/Makefile | 32 + examples/vdpa/main.c | 387 +++++ mk/rte.app.mk | 1 + 14 files changed, 2030 insertions(+), 11 deletions(-) create mode 100644 drivers/net/vdpa_virtio_pci/Makefile create mode 100644 drivers/net/vdpa_virtio_pci/rte_eth_vdpa_virtio_pci.c create mode 100644 drivers/net/vdpa_virtio_pci/rte_vdpa_virtio_pci_version.map create mode 100644 examples/vdpa/Makefile create mode 100644 examples/vdpa/main.c -- 2.15.1