From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id A6BF63B5 for ; Wed, 20 Jul 2016 05:52:42 +0200 (CEST) Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga101.jf.intel.com with ESMTP; 19 Jul 2016 20:52:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.28,392,1464678000"; d="scan'208";a="1020254344" Received: from fmsmsx106.amr.corp.intel.com ([10.18.124.204]) by orsmga002.jf.intel.com with ESMTP; 19 Jul 2016 20:52:43 -0700 Received: from fmsmsx102.amr.corp.intel.com (10.18.124.200) by FMSMSX106.amr.corp.intel.com (10.18.124.204) with Microsoft SMTP Server (TLS) id 14.3.248.2; Tue, 19 Jul 2016 20:52:41 -0700 Received: from shsmsx103.ccr.corp.intel.com (10.239.4.69) by FMSMSX102.amr.corp.intel.com (10.18.124.200) with Microsoft SMTP Server (TLS) id 14.3.248.2; Tue, 19 Jul 2016 20:52:40 -0700 Received: from shsmsx102.ccr.corp.intel.com ([169.254.2.147]) by SHSMSX103.ccr.corp.intel.com ([169.254.4.181]) with mapi id 14.03.0248.002; Wed, 20 Jul 2016 11:52:39 +0800 From: "Xu, Qian Q" To: "Tan, Jianfeng" , "dev@dpdk.org" CC: "yuanhan.liu@linux.intel.com" , "Wang, Zhihong" , "Tan, Jianfeng" Thread-Topic: [dpdk-dev] [PATCH] examples/vhost: fix perf regression Thread-Index: AQHR4cT9gmZYhJZfx0WS63xlyth1baAgrvpg Date: Wed, 20 Jul 2016 03:52:38 +0000 Message-ID: <82F45D86ADE5454A95A89742C8D1410E032EF10B@shsmsx102.ccr.corp.intel.com> References: <1468936391-138371-1-git-send-email-jianfeng.tan@intel.com> In-Reply-To: <1468936391-138371-1-git-send-email-jianfeng.tan@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH] examples/vhost: fix perf regression X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jul 2016 03:52:43 -0000 Tested-by: Qian Xu - Test Commit: 8f6f24342281f59de0df7bd976a32f714d39b9a9 - OS/Kernel: Fedora 21/4.1.13 - GCC: gcc (GCC) 4.9.2 20141101 (Red Hat 4.9.2-1) - CPU: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10 - NIC: Intel(R) Ethernet Controller X710 for 10GbE SFP+ - Total 2 cases, 2 passed, 0 failed.=20 Test Case1: Virtio-net IPV4 fwd performance with mergable=3Doff Summary:=20 Launch the vhost-switch sample, and launch VM with 2 virtio-net devices, le= t 2 virtio-net run IPV4 fwd, send traffic to the NIC port and let the traff= ic go through 2 virtio-net devices. Check the performance. Details:=20 1. Bind one port to igb_uio.=20 2. Run vhost switch sample with mergeable=3D0, disable mergeable.=20 taskset -c 18-19 ./examples/vhost/build/vhost-switch -c 0xc0000 -n 4 --huge= -dir /mnt/huge --socket-mem 1024,1024 -- -p 0x1 --mergeable 0 --vm2vm 0 3. Launch VM:=20 taskset -c 22-23 \ /root/qemu-versions/qemu-2.6.0/x86_64-softmmu/qemu-system-x86_64 -name vm1 = \ -cpu host -enable-kvm -m 2048 -object memory-backend-file,id=3Dmem,size=3D2= 048M,mem-path=3D/mnt/huge,share=3Don -numa node,memdev=3Dmem -mem-prealloc = \ -smp cores=3D4 -drive file=3D/home/img/vm1.img \ -chardev socket,id=3Dchar0,path=3D./vhost-net \ -netdev type=3Dvhost-user,id=3Dmynet1,chardev=3Dchar0,vhostforce \ -device virtio-net-pci,mac=3D52:54:00:00:00:01,netdev=3Dmynet1,mrg_rxbuf=3D= on \ -chardev socket,id=3Dchar1,path=3D./vhost-net \ -netdev type=3Dvhost-user,id=3Dmynet2,chardev=3Dchar1,vhostforce \ -device virtio-net-pci,mac=3D52:54:00:00:00:02,netdev=3Dmynet2,mrg_rxbuf=3D= on \ -netdev tap,id=3Dipvm1,ifname=3Dtap3,script=3D/etc/qemu-ifup -device rtl813= 9,netdev=3Dipvm1,id=3Dnet0,mac=3D00:00:00:00:10:01 \ -vnc :3 -daemonize 4. Set IPV4 fwd rules in VM:=20 virtio1=3D$1 virtio2=3D$2 systemctl stop firewalld.service systemctl disable firewalld.service systemctl stop ip6tables.service systemctl disable ip6tables.service systemctl stop iptables.service systemctl disable iptables.service systemctl stop NetworkManager.service echo 1 >/proc/sys/net/ipv4/ip_forward ip addr add 192.168.1.2/24 dev $virtio1 ip neigh add 192.168.1.1 lladdr 00:00:10:00:24:00 dev $virtio1 ip link set dev $virtio1 up ip addr add 192.168.2.2/24 dev $virtio2 ip neigh add 192.168.2.1 lladdr 00:00:10:00:24:01 dev $virtio2 ip link set dev $virtio2 up 5. Send traffic to NIC and see the performance back from virtio2. The perfo= rmance is back with the patch.=20 Test Case2: Virtio-net IPV4 fwd performance with mergable=3Don Similar steps, just one feature set is different, set mergable=3D1 in the v= host-switch sample, then the performance is good as before.=20 Thanks Qian -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jianfeng Tan Sent: Tuesday, July 19, 2016 9:53 PM To: dev@dpdk.org Cc: yuanhan.liu@linux.intel.com; Wang, Zhihong ; Ta= n, Jianfeng Subject: [dpdk-dev] [PATCH] examples/vhost: fix perf regression We find significant perfermance drop introduced by below commit, when vhost= example is started with --mergeable 0 and inside vm, kernel virtio-net dri= ver is used to do ip based forwarding. The root cause is that below commit adds support for VIRTIO_NET_F_GUEST_TSO4 and VIRTIO_NET_F_GUEST_TSO6, and when mergeable is = disabled, it triggers big_packets path of virtio-net driver. In this path, = virtio driver uses 19 desc with 18 4K-sized pages to receive each packet, s= o that it can receive a big packet with size of 64K. But QEMU only creates = 256 desc entries for each vq, which results in that only 13 packets can be = received. VM kernel can quickly handle those packets and go to sleep (HLT). As QEMU has no option to set the desc entries of a vq, so here, we disable = VIRTIO_NET_F_GUEST_TSO4 and VIRTIO_NET_F_GUEST_TSO6 with VIRTIO_NET_F_HOST_= TSO4 and VIRTIO_NET_F_HOST_TSO6 when we disable tso of vhost example, to av= oid VM kernel virtio driver go into big_packets path. Fixes: 859b480d5afd ("vhost: add guest offload setting") Reported-by: Qian Xu Signed-off-by: Jianfeng Tan --- examples/vhost/main.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/examples/vhost/main.c b/examples/vhost/main.c index 3b98f42..9= 2a9823 100644 --- a/examples/vhost/main.c +++ b/examples/vhost/main.c @@ -327,6 +327,8 @@ port_init(uint8_t port) if (enable_tso =3D=3D 0) { rte_vhost_feature_disable(1ULL << VIRTIO_NET_F_HOST_TSO4); rte_vhost_feature_disable(1ULL << VIRTIO_NET_F_HOST_TSO6); + rte_vhost_feature_disable(1ULL << VIRTIO_NET_F_GUEST_TSO4); + rte_vhost_feature_disable(1ULL << VIRTIO_NET_F_GUEST_TSO6); } =20 rx_rings =3D (uint16_t)dev_info.max_rx_queues; -- 2.7.4