From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by dpdk.org (Postfix) with ESMTP id 96F7D591A for ; Fri, 22 May 2015 03:13:11 +0200 (CEST) Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id t4M1DAuV021475 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Thu, 21 May 2015 21:13:10 -0400 Received: from tfherb.local (vpn-53-133.rdu2.redhat.com [10.10.53.133]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t4M1D9Fu029153; Thu, 21 May 2015 21:13:10 -0400 Message-ID: <555E82A5.4080608@redhat.com> Date: Thu, 21 May 2015 21:13:09 -0400 From: Thomas F Herbert Organization: Red Hat User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Ouyang Changchun , dev@dpdk.org References: <1432194581-15301-1-git-send-email-changchun.ouyang@intel.com> In-Reply-To: <1432194581-15301-1-git-send-email-changchun.ouyang@intel.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 Subject: Re: [dpdk-dev] [PATCH 0/6] Support multiple queues in vhost X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2015 01:13:12 -0000 On 5/21/15 3:49 AM, Ouyang Changchun wrote: > This patch set supports the multiple queues for each virtio device in vhost. > The vhost-user is used to enable the multiple queues feature, It's not ready for vhost-cuse. Thanks. I tried it and verified that this patch applies cleanly to master. Could you also notify the list when qemu patch is available. Thanks again! > > One prerequisite to enable this feature is that a QEMU patch plus a fix is required to apply > on QEMU2.2/2.3, pls refer to this link for the details of the patch and the fix: > http://lists.nongnu.org/archive/html/qemu-devel/2015-04/msg00917.html > > A formal v3 patch for the code change and the fix will be sent to qemu community soon. > > Basicaly vhost sample leverages the VMDq+RSS in HW to receive packets and distribute them > into different queue in the pool according to their 5 tuples. > > On the other hand, it enables multiple queues mode in vhost/virtio layer by setting the queue > number as the value larger than 1. > > HW queue numbers in pool is required to be exactly same with the queue number in each virtio > device, e.g. rxq = 4, the queue number is 4, it means there are 4 HW queues in each VMDq pool, > and 4 queues in each virtio device/port, every queue in pool maps to one qeueu in virtio device. > > ========================================= > ==================| |==================| > vport0 | | vport1 | > --- --- --- ---| |--- --- --- ---| > q0 | q1 | q2 | q3 | |q0 | q1 | q2 | q3 | > /\= =/\= =/\= =/\=| |/\= =/\= =/\= =/\=| > || || || || || || || || > || || || || || || || || > ||= =||= =||= =||=| =||== ||== ||== ||=| > q0 | q1 | q2 | q3 | |q0 | q1 | q2 | q3 | > > ------------------| |------------------| > VMDq pool0 | | VMDq pool1 | > ==================| |==================| > > In RX side, it firstly polls each queue of the pool and gets the packets from > it and enqueue them into its corresponding queue in virtio device/port. > In TX side, it dequeue packets from each queue of virtio device/port and send > to either physical port or another virtio device according to its destination > MAC address. > > It includes a workaround here in virtio as control queue not work for vhost-user > multiple queues. It needs further investigate to root the cause, hopefully it could > be addressed in next version. > > Here is some test guidance. > 1. On host, firstly mount hugepage, and insmod uio, igb_uio, bind one nic on igb_uio; > and then run vhost sample, key steps as follows: > sudo mount -t hugetlbfs nodev /mnt/huge > sudo modprobe uio > sudo insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio.ko > > $RTE_SDK/tools/dpdk_nic_bind.py --bind igb_uio 0000:08:00.0 > sudo $RTE_SDK/examples/vhost/build/vhost-switch -c 0xf0 -n 4 --huge-dir /mnt/huge --socket-mem 1024,0 -- -p 1 --vm2vm 0 --dev-basename usvhost --rxq 2 > > 2. After step 1, on host, modprobe kvm and kvm_intel, and use qemu command line to start one guest: > modprobe kvm > modprobe kvm_intel > sudo mount -t hugetlbfs nodev /dev/hugepages -o pagesize=1G > > $QEMU_PATH/qemu-system-x86_64 -enable-kvm -m 4096 -object memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc -smp 10 -cpu core2duo,+sse3,+sse4.1,+sse4.2 -name -drive file=/vm.img -chardev socket,id=char0,path=/usvhost -netdev type=vhost-user,id=hostnet2,chardev=char0,vhostforce=on,queues=2 -device virtio-net-pci,mq=on,vectors=6,netdev=hostnet2,id=net2,mac=52:54:00:12:34:56,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off -chardev socket,id=char1,path=/usvhost -netdev type=vhost-user,id=hostnet3,chardev=char1,vhostforce=on,queues=2 -device virtio-net-pci,mq=on,vectors=6,netdev=hostnet3,id=net3,mac=52:54:00:12:34:57,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off > > 3. Log on guest, use testpmd(dpdk based) to test, use multiple virtio queues to rx and tx packets. > modprobe uio > insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio.ko > echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages > ./tools/dpdk_nic_bind.py --bind igb_uio 00:03.0 00:04.0 > > $RTE_SDK/$RTE_TARGET/app/testpmd -c 1f -n 4 -- --rxq=2 --txq=2 --nb-cores=4 --rx-queue-stats-mapping="(0,0,0),(0,1,1),(1,0,2),(1,1,3)" --tx-queue-stats-mapping="(0,0,0),(0,1,1),(1,0,2),(1,1,3)" -i --disable-hw-vlan --txqflags 0xf00 > > 4. Use packet generator to send packets with dest MAC:52 54 00 12 34 57 VLAN tag:1001, > select IPv4 as protocols and continuous incremental IP address. > > 5. Testpmd on guest can display packets received/transmitted in both queues of each virtio port. > > Changchun Ouyang (6): > ixgbe: Support VMDq RSS in non-SRIOV environment > lib_vhost: Support multiple queues in virtio dev > lib_vhost: Set memory layout for multiple queues mode > vhost: Add new command line option: rxq > vhost: Support multiple queues > virtio: Resolve for control queue > > examples/vhost/main.c | 199 +++++++++++++++++--------- > lib/librte_ether/rte_ethdev.c | 40 ++++++ > lib/librte_pmd_ixgbe/ixgbe_rxtx.c | 82 +++++++++-- > lib/librte_pmd_virtio/virtio_ethdev.c | 6 + > lib/librte_vhost/rte_virtio_net.h | 25 +++- > lib/librte_vhost/vhost_cuse/virtio-net-cdev.c | 57 ++++---- > lib/librte_vhost/vhost_rxtx.c | 53 +++---- > lib/librte_vhost/vhost_user/vhost-net-user.c | 4 +- > lib/librte_vhost/vhost_user/virtio-net-user.c | 156 ++++++++++++++------ > lib/librte_vhost/vhost_user/virtio-net-user.h | 2 + > lib/librte_vhost/virtio-net.c | 158 ++++++++++++-------- > 11 files changed, 545 insertions(+), 237 deletions(-) >