From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 1FB35106B for ; Mon, 15 Jun 2015 09:56:55 +0200 (CEST) Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga102.jf.intel.com with ESMTP; 15 Jun 2015 00:56:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.13,617,1427785200"; d="scan'208";a="508393456" Received: from shvmail01.sh.intel.com ([10.239.29.42]) by FMSMGA003.fm.intel.com with ESMTP; 15 Jun 2015 00:56:52 -0700 Received: from shecgisg004.sh.intel.com (shecgisg004.sh.intel.com [10.239.29.89]) by shvmail01.sh.intel.com with ESMTP id t5F7uo6m005329; Mon, 15 Jun 2015 15:56:50 +0800 Received: from shecgisg004.sh.intel.com (localhost [127.0.0.1]) by shecgisg004.sh.intel.com (8.13.6/8.13.6/SuSE Linux 0.8) with ESMTP id t5F7ukfd030618; Mon, 15 Jun 2015 15:56:48 +0800 Received: (from couyang@localhost) by shecgisg004.sh.intel.com (8.13.6/8.13.6/Submit) id t5F7ukO3030614; Mon, 15 Jun 2015 15:56:46 +0800 From: Ouyang Changchun To: dev@dpdk.org Date: Mon, 15 Jun 2015 15:56:37 +0800 Message-Id: <1434355006-30583-1-git-send-email-changchun.ouyang@intel.com> X-Mailer: git-send-email 1.7.12.2 In-Reply-To: <1433915549-18571-1-git-send-email-changchun.ouyang@intel.com> References: <1433915549-18571-1-git-send-email-changchun.ouyang@intel.com> Subject: [dpdk-dev] [PATCH v3 0/9] Support multiple queues in vhost X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jun 2015 07:56:56 -0000 This patch set supports the multiple queues for each virtio device in vhost. The vhost-user is used to enable the multiple queues feature, It's not ready for vhost-cuse. The QEMU patch of enabling vhost-use multiple queues has already merged into upstream sub-tree in QEMU community and it will be put in QEMU 2.4. If using QEMU 2.3, it requires applying the same patch onto QEMU 2.3 and rebuild the QEMU before running vhost multiple queues: http://patchwork.ozlabs.org/patch/477461/ Basically vhost sample leverages the VMDq+RSS in HW to receive packets and distribute them into different queue in the pool according to their 5 tuples. On the other hand, the vhost will get the queue pair number based on the communication message with QEMU. HW queue numbers in pool is strongly recommended to set as identical with the queue number to start the QMEU guest and identical with the queue number to start with virtio port on guest. E.g. use '--rxq 4' to set the queue number as 4, it means there are 4 HW queues in each VMDq pool, and 4 queues in each vhost device/port, every queue in pool maps to one queue in vhost device. ========================================= ==================| |==================| vport0 | | vport1 | --- --- --- ---| |--- --- --- ---| q0 | q1 | q2 | q3 | |q0 | q1 | q2 | q3 | /\= =/\= =/\= =/\=| |/\= =/\= =/\= =/\=| || || || || || || || || || || || || || || || || ||= =||= =||= =||=| =||== ||== ||== ||=| q0 | q1 | q2 | q3 | |q0 | q1 | q2 | q3 | ------------------| |------------------| VMDq pool0 | | VMDq pool1 | ==================| |==================| In RX side, it firstly polls each queue of the pool and gets the packets from it and enqueue them into its corresponding virtqueue in virtio device/port. In TX side, it dequeue packets from each virtqueue of virtio device/port and send to either physical port or another virtio device according to its destination MAC address. Here is some test guidance. 1. On host, firstly mount hugepage, and insmod uio, igb_uio, bind one nic on igb_uio; and then run vhost sample, key steps as follows: sudo mount -t hugetlbfs nodev /mnt/huge sudo modprobe uio sudo insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio.ko $RTE_SDK/tools/dpdk_nic_bind.py --bind igb_uio 0000:08:00.0 sudo $RTE_SDK/examples/vhost/build/vhost-switch -c 0xf0 -n 4 --huge-dir /mnt/huge --socket-mem 1024,0 -- -p 1 --vm2vm 0 --dev-basename usvhost --rxq 2 use '--stats 1' to enable the stats dumping on screen for vhost. 2. After step 1, on host, modprobe kvm and kvm_intel, and use qemu command line to start one guest: modprobe kvm modprobe kvm_intel sudo mount -t hugetlbfs nodev /dev/hugepages -o pagesize=1G $QEMU_PATH/qemu-system-x86_64 -enable-kvm -m 4096 -object memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc -smp 10 -cpu core2duo,+sse3,+sse4.1,+sse4.2 -name -drive file=/vm.img -chardev socket,id=char0,path=/usvhost -netdev type=vhost-user,id=hostnet2,chardev=char0,vhostforce=on,queues=2 -device virtio-net-pci,mq=on,vectors=6,netdev=hostnet2,id=net2,mac=52:54:00:12:34:56,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off -chardev socket,id=char1,path=/usvhost -netdev type=vhost-user,id=hostnet3,chardev=char1,vhostforce=on,queues=2 -device virtio-net-pci,mq=on,vectors=6,netdev=hostnet3,id=net3,mac=52:54:00:12:34:57,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off 3. Log on guest, use testpmd(dpdk based) to test, use multiple virtio queues to rx and tx packets. modprobe uio insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio.ko echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages ./tools/dpdk_nic_bind.py --bind igb_uio 00:03.0 00:04.0 $RTE_SDK/$RTE_TARGET/app/testpmd -c 1f -n 4 -- --rxq=2 --txq=2 --nb-cores=4 --rx-queue-stats-mapping="(0,0,0),(0,1,1),(1,0,2),(1,1,3)" --tx-queue-stats-mapping="(0,0,0),(0,1,1),(1,0,2),(1,1,3)" -i --disable-hw-vlan --txqflags 0xf00 set fwd mac start tx_first 4. Use packet generator to send packets with dest MAC:52 54 00 12 34 57 VLAN tag:1001, select IPv4 as protocols and continuous incremental IP address. 5. Testpmd on guest can display packets received/transmitted in both queues of each virtio port. Changchun Ouyang (9): ixgbe: Support VMDq RSS in non-SRIOV environment lib_vhost: Support multiple queues in virtio dev lib_vhost: Set memory layout for multiple queues mode lib_vhost: Check the virtqueue address's validity vhost: Add new command line option: rxq vhost: Support multiple queues virtio: Resolve for control queue vhost: Add per queue stats info doc: Update doc for vhost multiple queues doc/guides/prog_guide/vhost_lib.rst | 35 +++ doc/guides/sample_app_ug/vhost.rst | 110 +++++++++ drivers/net/ixgbe/ixgbe_rxtx.c | 86 +++++-- drivers/net/virtio/virtio_ethdev.c | 15 +- examples/vhost/main.c | 324 +++++++++++++++++--------- lib/librte_ether/rte_ethdev.c | 40 ++++ lib/librte_vhost/rte_virtio_net.h | 20 +- lib/librte_vhost/vhost-net.h | 1 + lib/librte_vhost/vhost_cuse/virtio-net-cdev.c | 57 +++-- lib/librte_vhost/vhost_rxtx.c | 70 ++++-- lib/librte_vhost/vhost_user/vhost-net-user.c | 15 +- lib/librte_vhost/vhost_user/virtio-net-user.c | 135 +++++++---- lib/librte_vhost/vhost_user/virtio-net-user.h | 2 + lib/librte_vhost/virtio-net.c | 205 +++++++++++----- 14 files changed, 824 insertions(+), 291 deletions(-) -- 1.8.4.2