From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from fncnmp04.fnc.fujitsu.com (fncnmp04.fnc.fujitsu.com [168.127.0.57]) by dpdk.org (Postfix) with ESMTP id AD6325697 for ; Fri, 15 May 2015 19:27:34 +0200 (CEST) X-IronPort-AV: E=Sophos;i="5.13,435,1427778000"; d="scan'208";a="65233168" Received: from rchexhcp2.fnc.net.local ([168.127.134.76]) by fncnmp02.fnc.fujitsu.com with ESMTP/TLS/AES256-SHA; 15 May 2015 12:27:33 -0500 Received: from RCHEXMBP1.fnc.net.local ([169.254.2.128]) by RCHEXHCP2.fnc.net.local ([168.127.134.76]) with mapi id 14.03.0181.006; Fri, 15 May 2015 12:27:32 -0500 From: "Rao, Ravi" To: "Traynor, Kevin" , "therbert@redhat.com" , "dev@dpdk.org" Thread-Topic: [dpdk-dev] performance issue with ovs + dpdk2.0 with vhost Thread-Index: AQHQjx1BCvlUGD4J0km0B26rhjgu5J19RjOAgABRKoD//7J5UA== Date: Fri, 15 May 2015 17:27:32 +0000 Message-ID: <4E5A07CF014F73489C60025F5BCB7834EBC16866@RCHEXMBP1.fnc.net.local> References: <4E5A07CF014F73489C60025F5BCB7834EBC15D9C@RCHEXMBP1.fnc.net.local> <5556059A.6000006@redhat.com> <55561ABF.8080205@us.fujitsu.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [168.127.136.253] x-tm-as-product-ver: SMEX-10.2.0.3176-7.500.1018-21548.001 x-tm-as-result: No--50.674200-0.000000-31 x-tm-as-user-approved-sender: Yes x-tm-as-user-blocked-sender: No Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] performance issue with ovs + dpdk2.0 with vhost X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 May 2015 17:27:35 -0000 Hi , If do not enable dpdk, plian ovs can pass about 120,000 pps but once I m= ake ovs to use dpdk the main reason in doing this is to get more performanc= e I am seeing that it is passing only 1000 pps. So I guess the drop from 120,000 to just 1000 means I am missing something. Regards, Ravi -----Original Message----- From: Traynor, Kevin [mailto:kevin.traynor@intel.com]=20 Sent: Friday, May 15, 2015 12:02 PM To: Rao, Ravi; therbert@redhat.com; dev@dpdk.org Subject: RE: [dpdk-dev] performance issue with ovs + dpdk2.0 with vhost > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ravi Rao > Sent: Friday, May 15, 2015 5:12 PM > To: therbert@redhat.com; dev@dpdk.org > Subject: Re: [dpdk-dev] performance issue with ovs + dpdk2.0 with=20 > vhost >=20 > Hi, > This is what I am trying to Do. > Below is the setup.. >=20 > | +----------------------+ | > | guest | | > | | | > | | | guest > | eth0 L3fwd eth1 | | > | | | | | > +---+--------------+---+ __| > ^ : > | | > : v __ > +-----------------+--------------+-----------------+ | > | | ovs-br0 | | > | +--------------+ | | > | ^ : | | > | +----------+ +---------+ | | host > | : v | | > | +--------------+ +--------------+ | | > | | dpdk0 | ovs-dpdk | dpdk1 | | | > +---+--------------+------------+--------------+---+ __| > ^ : > | | > : v > +--------------------------------------------------+ > | | > | traffic generator | > | | > +--------------------------------------------------+| >=20 >=20 > Step1: Use the latest ovs and dpdk2.0 to get the ovs running with 2=20 > dpdk interfaces that are bound to 2 10GB physical interfaces > #** Inser the required Modules > cd /root/dpdk-2.0.0 > modprobe uio > modprobe cuse > rmmod igb_uio > rmmod rte_kni > insmod x86_64-native-linuxapp-gcc/kmod/igb_uio.ko >=20 > #**** Assign the dpdk capable interfaces to igb_uio driver=20 > tools/dpdk_nic_bind.py --status tools/dpdk_nic_bind.py -b igb_uio=20 > 0000:02:00.0 tools/dpdk_nic_bind.py -b igb_uio 0000:02:00.1=20 > tools/dpdk_nic_bind.py --status >=20 > #--- Setup the openVswitch > cd /root/ovs > pkill -9 ovs > mkdir -p /usr/local/etc/openvswitch > mkdir -p /usr/local/var/run/openvswitch rm -rf=20 > /usr/local/etc/openvswitch/conf.db > ovsdb/ovsdb-tool create /usr/local/etc/openvswitch/conf.db > vswitchd/vswitch.ovsschema >=20 > #Start ovsdb-server > ovsdb/ovsdb-server=20 > --remote=3Dpunix:/usr/local/var/run/openvswitch/db.sock > --remote=3Ddb:Open_vSwitch,Open_vSwitch,manager_options --pidfile=20 > --detach utilities/ovs-vsctl --no-wait init >=20 > #Start vswitchd: > export DB_SOCK=3D/usr/local/var/run/openvswitch/db.sock > rm /dev/vhost-net > vswitchd/ovs-vswitchd --dpdk -c 0x3 -n 4 --socket-mem 1024,0 --=20 > unix:$DB_SOCK --pidfile --detach >=20 > #Add bridge & ports > utilities/ovs-vsctl add-br ovs-br0 -- set bridge ovs-br0=20 > datapath_type=3Dnetdev utilities/ovs-vsctl add-port ovs-br0 dpdk0 -- set= =20 > Interface dpdk0 type=3Ddpdk utilities/ovs-vsctl add-port ovs-br0 dpdk1=20 > -- set Interface dpdk1 type=3Ddpdk >=20 > Step2: Create the dpdkvhost interfaces and bring up the guestVM using=20 > QEMU export DPDK_DIR=3D/root/dpdk-2.0.0 insmod=20 > $DPDK_DIR/lib/librte_vhost/eventfd_link/eventfd_link.ko > cd /root/ovs > utilities/ovs-vsctl add-port ovs-br0 dpdkvhost0 -- set Interface > dpdkvhost0 type=3Ddpdkvhost > utilities/ovs-vsctl add-port ovs-br0 dpdkvhost1 -- set Interface > dpdkvhost1 type=3Ddpdkvhost >=20 > #**** Start the guest ubuntu VM1 from a terminal that is logged in as=20 > root > qemu-system-x86_64 --enable-kvm -k fr -m 1G \ > -cpu host -smp cores=3D2,threads=3D1,sockets=3D1 \ > -serial telnet::4444,server,nowait -monitor=20 > telnet::5555,server,nowait \ > -hda /root/VMs/images/ubuntu-14.04-template.qcow2 \ > -object > memory-backend-file,id=3Dmem,size=3D1G,mem-path=3D/mnt/huge_1GB,share=3Do= n \ > -numa node,memdev=3Dmem \ > -netdev > type=3Dtap,id=3Ddpdkvhost0,script=3Dno,downscript=3Dno,ifname=3Ddpdkvhost= 0,vhost=3Don \ > -device > virtio-net- > pci,netdev=3Ddpdkvhost0,mac=3D52:54:00:12:34:56,csum=3Doff,gso=3Doff,gues= t_tso > 4=3Doff,g > uest_tso6=3Doff,guest_ecn=3Doff > \ > -netdev > type=3Dtap,id=3Ddpdkvhost1,script=3Dno,downscript=3Dno,ifname=3Ddpdkvhost= 1,vhost=3Don \ > -device > virtio-net- > pci,netdev=3Ddpdkvhost1,mac=3D52:54:00:12:34:57,csum=3Doff,gso=3Doff,gues= t_tso > 4=3Doff,g > uest_tso6=3Doff,guest_ecn=3Doff > \ > -device ne2k_pci,mac=3DDE:AD:DE:01:02:03,netdev=3Duser.0 -netdev > user,id=3Duser.0,hostfwd=3Dtcp::2222-:22 & >=20 > # **** Add flows between ports > utilities/ovs-ofctl del-flows ovs-br0 > utilities/ovs-ofctl add-flow ovs-br0 in_port=3D1,action=3Doutput:3=20 > utilities/ovs-ofctl add-flow ovs-br0 in_port=3D2,action=3Doutput:4=20 > utilities/ovs-ofctl add-flow ovs-br0 in_port=3D3,action=3Doutput:1=20 > utilities/ovs-ofctl add-flow ovs-br0 in_port=3D4,action=3Doutput:2 >=20 > Once I complete the above settings. I log into the VM and enable ipv4=20 > forwarding so that it could do l3 fwd between eth0 and eth1 >=20 > The issue I am seeing is when I start pumping packets on IXIA port=20 > connected to physical port dpdk0 I see lots of tx_errors on dpdk0. > I can only pass about 1000 pps without getting any errors. Is there=20 > anything I am doing wrong or missing in the above setup. >>From what you are saying, I take it that it is functionally working but the= rates are low. In that case I would check/set core affinities with top -H = and taskset to see if there's contention occurring. You could also test wit= h multiple PMDs (hence cores) in OVS to spread the switching workload - if = you wanted to test with 4 PMDs you could use ovs-vsctl --timeout 10 set Ope= n_vSwitch . other_config:pmd-cpu-mask=3D3c >=20 > Thanks & Regards, > Ravi.. >=20 > On 05/15/15 09:41, Thomas F Herbert wrote: > > On 5/15/15 9:55 AM, Rao, Ravi wrote: > >> Hi All, > >> I am trying to get a Guest VM connected to the dpdkvhost=20 > >> interface on a host which has the ovs running from the latest ovs=20 > >> git and dpdk2.0. Looks like I am missing something as 95% of=20 > >> traffic is not getting to the VM. Can one of you please let me know=20 > >> which mailing list I should be posting the details for getting a=20 > >> resolution. Is it this dpdk list OR would it be the openvswitch list? > > You could also try the openvswitch discuss list but this list is a=20 > > good starting point. I am working with vhost/dpdk2.0 and ovs and=20 > > others are too. > > Could you please send more details about what you are seeing? Are=20 > > you using the Linux stack in the guest? Do you see packets on the=20 > > input IF in the guest? > >> Regards, > >> Ravi.. > > > >