From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id 4E66D5952 for ; Thu, 21 May 2015 09:49:48 +0200 (CEST) Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga101.jf.intel.com with ESMTP; 21 May 2015 00:49:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.13,467,1427785200"; d="scan'208";a="496438940" Received: from shvmail01.sh.intel.com ([10.239.29.42]) by FMSMGA003.fm.intel.com with ESMTP; 21 May 2015 00:49:46 -0700 Received: from shecgisg004.sh.intel.com (shecgisg004.sh.intel.com [10.239.29.89]) by shvmail01.sh.intel.com with ESMTP id t4L7niYH025407; Thu, 21 May 2015 15:49:44 +0800 Received: from shecgisg004.sh.intel.com (localhost [127.0.0.1]) by shecgisg004.sh.intel.com (8.13.6/8.13.6/SuSE Linux 0.8) with ESMTP id t4L7nf0v015334; Thu, 21 May 2015 15:49:43 +0800 Received: (from couyang@localhost) by shecgisg004.sh.intel.com (8.13.6/8.13.6/Submit) id t4L7nfkK015330; Thu, 21 May 2015 15:49:41 +0800 From: Ouyang Changchun To: dev@dpdk.org Date: Thu, 21 May 2015 15:49:35 +0800 Message-Id: <1432194581-15301-1-git-send-email-changchun.ouyang@intel.com> X-Mailer: git-send-email 1.7.12.2 Subject: [dpdk-dev] [PATCH 0/6] Support multiple queues in vhost X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 May 2015 07:49:49 -0000 This patch set supports the multiple queues for each virtio device in vhost. The vhost-user is used to enable the multiple queues feature, It's not ready for vhost-cuse. One prerequisite to enable this feature is that a QEMU patch plus a fix is required to apply on QEMU2.2/2.3, pls refer to this link for the details of the patch and the fix: http://lists.nongnu.org/archive/html/qemu-devel/2015-04/msg00917.html A formal v3 patch for the code change and the fix will be sent to qemu community soon. Basicaly vhost sample leverages the VMDq+RSS in HW to receive packets and distribute them into different queue in the pool according to their 5 tuples. On the other hand, it enables multiple queues mode in vhost/virtio layer by setting the queue number as the value larger than 1. HW queue numbers in pool is required to be exactly same with the queue number in each virtio device, e.g. rxq = 4, the queue number is 4, it means there are 4 HW queues in each VMDq pool, and 4 queues in each virtio device/port, every queue in pool maps to one qeueu in virtio device. ========================================= ==================| |==================| vport0 | | vport1 | --- --- --- ---| |--- --- --- ---| q0 | q1 | q2 | q3 | |q0 | q1 | q2 | q3 | /\= =/\= =/\= =/\=| |/\= =/\= =/\= =/\=| || || || || || || || || || || || || || || || || ||= =||= =||= =||=| =||== ||== ||== ||=| q0 | q1 | q2 | q3 | |q0 | q1 | q2 | q3 | ------------------| |------------------| VMDq pool0 | | VMDq pool1 | ==================| |==================| In RX side, it firstly polls each queue of the pool and gets the packets from it and enqueue them into its corresponding queue in virtio device/port. In TX side, it dequeue packets from each queue of virtio device/port and send to either physical port or another virtio device according to its destination MAC address. It includes a workaround here in virtio as control queue not work for vhost-user multiple queues. It needs further investigate to root the cause, hopefully it could be addressed in next version. Here is some test guidance. 1. On host, firstly mount hugepage, and insmod uio, igb_uio, bind one nic on igb_uio; and then run vhost sample, key steps as follows: sudo mount -t hugetlbfs nodev /mnt/huge sudo modprobe uio sudo insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio.ko $RTE_SDK/tools/dpdk_nic_bind.py --bind igb_uio 0000:08:00.0 sudo $RTE_SDK/examples/vhost/build/vhost-switch -c 0xf0 -n 4 --huge-dir /mnt/huge --socket-mem 1024,0 -- -p 1 --vm2vm 0 --dev-basename usvhost --rxq 2 2. After step 1, on host, modprobe kvm and kvm_intel, and use qemu command line to start one guest: modprobe kvm modprobe kvm_intel sudo mount -t hugetlbfs nodev /dev/hugepages -o pagesize=1G $QEMU_PATH/qemu-system-x86_64 -enable-kvm -m 4096 -object memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc -smp 10 -cpu core2duo,+sse3,+sse4.1,+sse4.2 -name -drive file=/vm.img -chardev socket,id=char0,path=/usvhost -netdev type=vhost-user,id=hostnet2,chardev=char0,vhostforce=on,queues=2 -device virtio-net-pci,mq=on,vectors=6,netdev=hostnet2,id=net2,mac=52:54:00:12:34:56,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off -chardev socket,id=char1,path=/usvhost -netdev type=vhost-user,id=hostnet3,chardev=char1,vhostforce=on,queues=2 -device virtio-net-pci,mq=on,vectors=6,netdev=hostnet3,id=net3,mac=52:54:00:12:34:57,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off 3. Log on guest, use testpmd(dpdk based) to test, use multiple virtio queues to rx and tx packets. modprobe uio insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio.ko echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages ./tools/dpdk_nic_bind.py --bind igb_uio 00:03.0 00:04.0 $RTE_SDK/$RTE_TARGET/app/testpmd -c 1f -n 4 -- --rxq=2 --txq=2 --nb-cores=4 --rx-queue-stats-mapping="(0,0,0),(0,1,1),(1,0,2),(1,1,3)" --tx-queue-stats-mapping="(0,0,0),(0,1,1),(1,0,2),(1,1,3)" -i --disable-hw-vlan --txqflags 0xf00 4. Use packet generator to send packets with dest MAC:52 54 00 12 34 57 VLAN tag:1001, select IPv4 as protocols and continuous incremental IP address. 5. Testpmd on guest can display packets received/transmitted in both queues of each virtio port. Changchun Ouyang (6): ixgbe: Support VMDq RSS in non-SRIOV environment lib_vhost: Support multiple queues in virtio dev lib_vhost: Set memory layout for multiple queues mode vhost: Add new command line option: rxq vhost: Support multiple queues virtio: Resolve for control queue examples/vhost/main.c | 199 +++++++++++++++++--------- lib/librte_ether/rte_ethdev.c | 40 ++++++ lib/librte_pmd_ixgbe/ixgbe_rxtx.c | 82 +++++++++-- lib/librte_pmd_virtio/virtio_ethdev.c | 6 + lib/librte_vhost/rte_virtio_net.h | 25 +++- lib/librte_vhost/vhost_cuse/virtio-net-cdev.c | 57 ++++---- lib/librte_vhost/vhost_rxtx.c | 53 +++---- lib/librte_vhost/vhost_user/vhost-net-user.c | 4 +- lib/librte_vhost/vhost_user/virtio-net-user.c | 156 ++++++++++++++------ lib/librte_vhost/vhost_user/virtio-net-user.h | 2 + lib/librte_vhost/virtio-net.c | 158 ++++++++++++-------- 11 files changed, 545 insertions(+), 237 deletions(-) -- 1.8.4.2