From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 0A8952C10 for ; Thu, 21 Jul 2016 03:38:59 +0200 (CEST) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga103.jf.intel.com with ESMTP; 20 Jul 2016 18:38:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.28,397,1464678000"; d="scan'208";a="142910361" Received: from fmsmsx106.amr.corp.intel.com ([10.18.124.204]) by fmsmga004.fm.intel.com with ESMTP; 20 Jul 2016 18:38:58 -0700 Received: from FMSMSX109.amr.corp.intel.com (10.18.116.9) by FMSMSX106.amr.corp.intel.com (10.18.124.204) with Microsoft SMTP Server (TLS) id 14.3.248.2; Wed, 20 Jul 2016 18:38:58 -0700 Received: from shsmsx104.ccr.corp.intel.com (10.239.4.70) by fmsmsx109.amr.corp.intel.com (10.18.116.9) with Microsoft SMTP Server (TLS) id 14.3.248.2; Wed, 20 Jul 2016 18:38:58 -0700 Received: from shsmsx102.ccr.corp.intel.com ([169.254.2.147]) by SHSMSX104.ccr.corp.intel.com ([169.254.5.116]) with mapi id 14.03.0248.002; Thu, 21 Jul 2016 09:38:54 +0800 From: "Xu, Qian Q" To: Yuanhan Liu , "Tan, Jianfeng" CC: "dev@dpdk.org" , "Wang, Zhihong" Thread-Topic: [PATCH v3] examples/vhost: fix perf regression Thread-Index: AQHR4ujjNRrBbPDv8EujpiAsZctKtKAhlBoAgACHQQA= Date: Thu, 21 Jul 2016 01:38:53 +0000 Message-ID: <82F45D86ADE5454A95A89742C8D1410E032F0B66@shsmsx102.ccr.corp.intel.com> References: <1468936391-138371-1-git-send-email-jianfeng.tan@intel.com> <1469061765-50814-1-git-send-email-jianfeng.tan@intel.com> <20160721013426.GA28708@yliu-dev.sh.intel.com> In-Reply-To: <20160721013426.GA28708@yliu-dev.sh.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v3] examples/vhost: fix perf regression X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jul 2016 01:39:00 -0000 Add the tested-by:) Tested-by: Qian Xu - Test Commit: 608487f3fc96704271c624d0f3fe9d7fb2187aea - OS/Kernel: Fedora 21/4.1.13 - GCC: gcc (GCC) 4.9.2 20141101 (Red Hat 4.9.2-1) - CPU: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10 - NIC: Intel(R) Ethernet Controller X710 for 10GbE SFP+ - Total 2 cases, 2 passed, 0 failed.=20 Test Case1: Virtio-net IPV4 fwd performance with mergable=3Doff Summary:=20 Launch the vhost-switch sample, and launch VM with 2 virtio-net devices, le= t 2 virtio-net run IPV4 fwd, send traffic to the NIC port and let the traff= ic go through 2 virtio-net devices. Check the performance. Details:=20 1. Bind one port to igb_uio.=20 2. Run vhost switch sample with mergeable=3D0, disable mergeable.=20 taskset -c 18-19 ./examples/vhost/build/vhost-switch -c 0xc0000 -n 4 --huge= -dir /mnt/huge --socket-mem 1024,1024 -- -p 0x1 --mergeable 0 --vm2vm 0 3. = Launch VM:=20 taskset -c 22-23 \ /root/qemu-versions/qemu-2.6.0/x86_64-softmmu/qemu-system-x86_64 -name vm1 = \ -cpu host -enable-kvm -m 2048 -object memory-backend-file,id=3Dmem,size= =3D2048M,mem-path=3D/mnt/huge,share=3Don -numa node,memdev=3Dmem -mem-preal= loc \ -smp cores=3D4 -drive file=3D/home/img/vm1.img \ -chardev socket,id= =3Dchar0,path=3D./vhost-net \ -netdev type=3Dvhost-user,id=3Dmynet1,chardev= =3Dchar0,vhostforce \ -device virtio-net-pci,mac=3D52:54:00:00:00:01,netdev= =3Dmynet1,mrg_rxbuf=3Don \ -chardev socket,id=3Dchar1,path=3D./vhost-net \ = -netdev type=3Dvhost-user,id=3Dmynet2,chardev=3Dchar1,vhostforce \ -device = virtio-net-pci,mac=3D52:54:00:00:00:02,netdev=3Dmynet2,mrg_rxbuf=3Don \ -ne= tdev tap,id=3Dipvm1,ifname=3Dtap3,script=3D/etc/qemu-ifup -device rtl8139,n= etdev=3Dipvm1,id=3Dnet0,mac=3D00:00:00:00:10:01 \ -vnc :3 -daemonize 4. Set= IPV4 fwd rules in VM:=20 virtio1=3D$1 virtio2=3D$2 systemctl stop firewalld.service systemctl disable firewalld.service systemctl stop ip6tables.service systemctl disable ip6tables.service systemctl stop iptables.service systemctl disable iptables.service systemctl stop NetworkManager.service echo 1 >/proc/sys/net/ipv4/ip_forward ip addr add 192.168.1.2/24 dev $virtio1 ip neigh add 192.168.1.1 lladdr 00:00:10:00:24:00 dev $virtio1 ip link set = dev $virtio1 up ip addr add 192.168.2.2/24 dev $virtio2 ip neigh add 192.168.2.1 lladdr 00:00:10:00:24:01 dev $virtio2 ip link set = dev $virtio2 up 5. Send traffic to NIC and see the performance back from virtio2. The perfo= rmance is back with the patch.=20 Test Case2: Virtio-net IPV4 fwd performance with mergable=3Don Similar step= s, just one feature set is different, set mergable=3D1 in the vhost-switch = sample, then the performance is good as before. -----Original Message----- From: Yuanhan Liu [mailto:yuanhan.liu@linux.intel.com]=20 Sent: Thursday, July 21, 2016 9:34 AM To: Tan, Jianfeng Cc: dev@dpdk.org; Wang, Zhihong ; Xu, Qian Q Subject: Re: [PATCH v3] examples/vhost: fix perf regression On Thu, Jul 21, 2016 at 12:42:45AM +0000, Jianfeng Tan wrote: > We find significant perfermance drop introduced by below commit, when=20 > vhost example is started with --mergeable 0 and inside vm, kernel=20 > virtio-net driver is used to do ip based forwarding. >=20 > The commit, 859b480d5afd ("vhost: add guest offload setting"), adds=20 > support for VIRTIO_NET_F_GUEST_TSO4 and VIRTIO_NET_F_GUEST_TSO6, in=20 > vhost lib. But inside vhost example, the way to disable tso only=20 > excludes the direction from virtio to vhost, but not the opposite=20 > direction. When mergeable is disabled, it triggers big_packets path of=20 > virtio-net driver to prepare to receive possible big packets with size=20 > of 64K. Because mergeable is off, for each entry of avail ring, virtio=20 > driver uses 19 desc chained together, with one desc pointing to=20 > header, other 18 desc pointing to 4K-sized pages. But QEMU only=20 > creates 256 desc entries for each vq, which results in that only 13=20 > packets can be received. VM kernel can quickly handle those packets=20 > and go to sleep (HLT). >=20 > As QEMU has no option to set the desc entries of a vq, so here, we=20 > disable VIRTIO_NET_F_GUEST_TSO4 and VIRTIO_NET_F_GUEST_TSO6 with=20 > VIRTIO_NET_F_HOST_TSO4 and VIRTIO_NET_F_HOST_TSO6 when we disable tso=20 > of vhost example, to avoid VM kernel virtio driver go into big_packets=20 > path. >=20 > Fixes: 9fd72e3cbd29 ("examples/vhost: add virtio offload") >=20 > Reported-by: Qian Xu > Signed-off-by: Jianfeng Tan > --- > v3: reword commit log. Yes, much better. One minor nit: you forgot to carry the Tested-by from Qia= n. Acked-by: Yuanhan Liu Thanks. --yliu