From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from demumfd002.nsn-inter.net (demumfd002.nsn-inter.net [93.183.12.31]) by dpdk.org (Postfix) with ESMTP id BFFA8C742 for ; Thu, 25 Jun 2015 23:14:01 +0200 (CEST) Received: from demuprx017.emea.nsn-intra.net ([10.150.129.56]) by demumfd002.nsn-inter.net (8.15.1/8.15.1) with ESMTPS id t5PLE0l0030152 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 25 Jun 2015 21:14:01 GMT Received: from DEMUHTC001.nsn-intra.net ([10.159.42.32]) by demuprx017.emea.nsn-intra.net (8.12.11.20060308/8.12.11) with ESMTP id t5PLE0DA000365 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Thu, 25 Jun 2015 23:14:00 +0200 Received: from DEMUHTC006.nsn-intra.net (10.159.42.37) by DEMUHTC001.nsn-intra.net (10.159.42.32) with Microsoft SMTP Server (TLS) id 14.3.235.1; Thu, 25 Jun 2015 23:14:00 +0200 Received: from DEMUMBX003.nsn-intra.net ([169.254.2.58]) by DEMUHTC006.nsn-intra.net ([10.159.42.37]) with mapi id 14.03.0235.001; Thu, 25 Jun 2015 23:14:00 +0200 From: "Vass, Sandor (Nokia - HU/Budapest)" To: "ext Patel, Rashmin N" , Matthew Hall Thread-Topic: [dpdk-dev] VMXNET3 on vmware, ping delay Thread-Index: AdCvJ2To6UqiEGymSWSUJy1NN7xnKgAIgsQAAAvLRgAABDq5oA== Date: Thu, 25 Jun 2015 21:13:59 +0000 Message-ID: <792CF0A6B0883C45AF8C719B2ECA946E42B24BD9@DEMUMBX003.nsn-intra.net> References: <792CF0A6B0883C45AF8C719B2ECA946E42B2430F@DEMUMBX003.nsn-intra.net> <20150625151834.GA29296@mhcomputing.net> In-Reply-To: Accept-Language: hu-HU, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.159.42.156] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-purgate-type: clean X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de X-purgate: clean X-purgate: This mail is considered clean (visit http://www.eleven.de for further information) X-purgate-size: 4703 X-purgate-ID: 151667::1435266841-000076F8-43A0CC7A/0/0 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] VMXNET3 on vmware, ping delay X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jun 2015 21:14:02 -0000 It seems I have found the cause, but I still don't understand the reason. So, let me describe my setup a bit further. I installed the VMWare Workstat= ion onto my laptop. It has a mobile i5 CPU: 2 cores with hyperthreading, so= basically 4 cores. In VMWare I assigned to C1 and C2 nodes 1 CPU and one core, BR has one CPU = and 4 cores allocated (the possible maximum value). If I execute the 'basicfwd' or the multi-process master (and two clients) o= n any of the cores out of [2,3,4] then the ping is received immediately (le= ss than 0.5ms) and the transfer speed is immediately high (starting from ~3= 0MB and finishing at around 80-90MB/s with basicfwd and test-pdm also *).=20 If I allocate them on core 1 (the clients on any other cores), then the pin= g behaves as I originally described: 1sec delays. When I tried to transfer = a bigger file (I used scp) it started really slow (some 16-32KB/s), sometim= es even it was stalled. Then later on it get faster as Matthew wrote but it= didn't went upper than 20-30MB/s test-pmd worked originally.=20 This is because when executing test-pmd there had to be defined 2 cores and= I always passed '-c 3'. Checking with top it could be seen that it always = used the CPU#2 (top showed that the second CPU was utilized by 100%). Can anyone tell me the reason of this behavior? Using CPU 1 there are huge = latencies, using other CPUs everything work as expected... Checking on the laptop (windows task manager) it could be seen that none of= the VMs were utilizing one CPU's to 100% on my laptop. The dpdk processes = 100% utilization were somehow distributed amongst the physical CPU cores. S= o no single core were allocated exclusively by a VM. Why is it a different = situation when I use the first CPU on BR rather than the others? It doesn't= seem that C1 and C2 are blocking that CPU. Anyway, the HOST opsys already = uses all the cores (not heavily). Rashmin, thanks for the docs. I think I already saw that one but I didn't t= ake that as serious. I thought perf tuning about latency in VMWare ESXi mak= es point when one would like to go from 5ms to 0.5ms. But I had 1000ms late= ncy at low load... I will check those params if they apply to Workstation a= t all. *) Top speed of Multi process master-client example was around 20-30 MB/s, = immediately. I think this is a normal limitation because the processes have= to talk with each other through shared mem, so it is anyway slower. I didn= 't test its speed when the Master process was bound to core 1 Sandor -----Original Message----- From: ext Patel, Rashmin N [mailto:rashmin.n.patel@intel.com]=20 Sent: Thursday, June 25, 2015 10:56 PM To: Matthew Hall; Vass, Sandor (Nokia - HU/Budapest) Cc: dev@dpdk.org Subject: RE: [dpdk-dev] VMXNET3 on vmware, ping delay For tuning ESXi and vSwitch for latency sensitive workloads, I remember the= following paper published by VMware: https://www.vmware.com/files/pdf/tech= paper/VMW-Tuning-Latency-Sensitive-Workloads.pdf that you can try out. The overall latency in setup (vmware and dpdk-vm using vmxnet3) remains in = vmware-native-driver/vmkernel/vmxnet3-backend/vmx-emulation threads in ESXi= . So you can better tune ESXi (as explained in the above white paper) and/o= r make sure that these important threads are not starving to improve latenc= y and throughput in some cases of this setup. Thanks, Rashmin -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Matthew Hall Sent: Thursday, June 25, 2015 8:19 AM To: Vass, Sandor (Nokia - HU/Budapest) Cc: dev@dpdk.org Subject: Re: [dpdk-dev] VMXNET3 on vmware, ping delay On Thu, Jun 25, 2015 at 09:14:53AM +0000, Vass, Sandor (Nokia - HU/Budapest= ) wrote: > According to my understanding each packet should go through BR as fast=20 > as possible, but it seems that the rte_eth_rx_burst retrieves packets=20 > only when there are at least 2 packets on the RX queue of the NIC. At=20 > least most of the times as there are cases (rarely - according to my=20 > console log) when it can retrieve 1 packet also and sometimes only 3=20 > packets can be retrieved... By default DPDK is optimized for throughput not latency. Try a test with he= avier traffic. There is also some work going on now for DPDK interrupt-driven mode, which = will work more like traditional Ethernet drivers instead of polling mode Et= hernet drivers. Though I'm not an expert on it, there is also a series of ways to optimize = for latency, which hopefully some others could discuss... or maybe search t= he archives / web site / Intel tuning documentation. Matthew.