From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk0-f181.google.com (mail-qk0-f181.google.com [209.85.220.181]) by dpdk.org (Postfix) with ESMTP id AF61156A3 for ; Mon, 27 Aug 2018 18:05:31 +0200 (CEST) Received: by mail-qk0-f181.google.com with SMTP id 130-v6so10827442qkd.10 for ; Mon, 27 Aug 2018 09:05:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:subject:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Trv9puAka3cjNU/BnPFvmmQ163UgFPrBW169RIY+1jQ=; b=Iev4XveneWEmk6Uki5UGso2S/q7IXeaEmI5S1RTNP/vNB+c8jDdbYvGZeDNI1fgJYD fe0KzTWjAcLWIOj9tH0h0MZ/vTrf+eZFJOkqJ7wk7J77Euf8AG3taxvEvitLoas1Q2e4 q6cu2SL5G//O0lksC8Gl0Z3hLqqf7nDqgWdMNd4pnf7Gl1FLALyr7xbPdVPypKy7TOjy r8n8hHX9rtGh5bSNnYfFtFltcZAUJUB+88GqBjXveuYU8C/KJi4itR5wznXcTV9YvRJK Ug0FbWvWd/I1SC2UL4fWau0MQVDfLaR+5bHgwiUUI+uV/6SvsaZvT+tq15yCtaU96rlA aeow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Trv9puAka3cjNU/BnPFvmmQ163UgFPrBW169RIY+1jQ=; b=q8tc+G8XuOIy77FAXio/wdrkZ2XvUDLuB/+AKdcMmVqj/o2DdseS3SovKR7L/iB8NW WFGQ7NCXZV/U/vVT6JuGmTvzMZgzKccWdmlq+LswRXVNLzZdVjF3zfhK3VBT5Wjxg01Y SSAuLSQOuy+a5438JDMoRpuWxuTOKyqxFUYNErIwfrr2qnWxVVrXCyco1ClUb5xmFV1g 0SwICj8mluVXe8HIzoioRz70QFz0NXYsrAE1rHFlUPBVdw850j35XjPHsgpapHz05AUX KK6DfUJObLFbEyVw17xW3PD97A96AffH9jDI0LfE6GHfkcGl6WLCVlqsmZKEWYEmlWSp t4iw== X-Gm-Message-State: APzg51DSv3SkIvXQ2fwesUtF2+tDCFD5CCvvBY13sjvJ6Xcv+ZmWeMYj 2Pz+AXOmx5GxS2Gb/zvYh6tAQM6p7LI= X-Google-Smtp-Source: ANB0VdbIU1q+YP4FDyOon6nOWPF0ki5Mor+jlidSwRfc17essHaMYBJLFNjATVLc62kxPSLSeoD8OQ== X-Received: by 2002:a37:b246:: with SMTP id b67-v6mr13786573qkf.284.1535385930632; Mon, 27 Aug 2018 09:05:30 -0700 (PDT) Received: from shemminger-XPS-13-9360 ([167.220.63.219]) by smtp.gmail.com with ESMTPSA id o18-v6sm8384759qtm.84.2018.08.27.09.05.30 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 27 Aug 2018 09:05:30 -0700 (PDT) Date: Mon, 27 Aug 2018 09:05:27 -0700 From: Stephen Hemminger To: users@dpdk.org Message-ID: <20180827090527.5e7ef3f5@shemminger-XPS-13-9360> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-users] Explanation for poor performance of DPDK not found X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2018 16:05:31 -0000 On Mon, 27 Aug 2018 17:21:03 +0200 Victor Huertas wrote: > Dear colleagues, > > I am seeing a strange behaviour in terms of performance when I run the L3 > forwarding pipeline app example of DPDK. > > The diagram is as simple as this: > > PC1 <--------1 Gbps link----------> DPDK app (L3 forwarding) <--------1 > Gbps link--------> PC2 > > I have implemented a new pipeline which is performs ARP task in order to > configure the Routing type pipelines's table 1 (the one that performs MAC > translation from next-hop IP addr). > > The first strange thing I see is that when I ping from PC 1 to PC 2, the > ping works but it is reporting me a delay of 19,9 ms. And also every ping > report (1 per second) reports a decreasing delay in 1 ms like this: > PING 192.168.1.101 (192.168.1.101) 56(84) bytes of data. > 64 bytes from 192.168.1.101: icmp_seq=2 ttl=64 time=17.2 ms > 64 bytes from 192.168.1.101: icmp_seq=3 ttl=64 time=15.9 ms > 64 bytes from 192.168.1.101: icmp_seq=4 ttl=64 time=14.9 ms > 64 bytes from 192.168.1.101: icmp_seq=5 ttl=64 time=13.9 ms > 64 bytes from 192.168.1.101: icmp_seq=6 ttl=64 time=12.9 ms > 64 bytes from 192.168.1.101: icmp_seq=7 ttl=64 time=11.9 ms > 64 bytes from 192.168.1.101: icmp_seq=8 ttl=64 time=10.9 ms > 64 bytes from 192.168.1.101: icmp_seq=9 ttl=64 time=19.9 ms > 64 bytes from 192.168.1.101: icmp_seq=10 ttl=64 time=18.9 ms > 64 bytes from 192.168.1.101: icmp_seq=11 ttl=64 time=17.9 ms > > As you can see, there is a 1 ms decrease each ping report and suddenly > comes back to 19,9 ms > > The second issue comes up when I send an 700 Mbps UDP stream (using iperf > v2.0.5 at both sides) from PC1 to PC2. What I see is a slight packet loss > on reception. > [ 4] 0.0-509.3 sec 1 datagrams received out-of-order > [ 3] local 192.168.0.101 port 5001 connected with 192.168.1.101 port 60184 > [ 3] 0.0- 5.0 sec 437 MBytes 733 Mbits/sec 0.022 ms 39/311788 > (0.013%) > [ 3] 5.0-10.0 sec 437 MBytes 733 Mbits/sec 0.025 ms 166/311988 > (0.053%) > [ 3] 10.0-15.0 sec 437 MBytes 734 Mbits/sec 0.022 ms 0/312067 > (0%) > [ 3] 15.0-20.0 sec 437 MBytes 733 Mbits/sec 0.029 ms 151/311916 > (0.048%) > [ 3] 20.0-25.0 sec 437 MBytes 734 Mbits/sec 0.016 ms 30/311926 > (0.0096%) > [ 3] 25.0-30.0 sec 437 MBytes 734 Mbits/sec 0.022 ms 143/312118 > (0.046%) > [ 3] 30.0-35.0 sec 437 MBytes 733 Mbits/sec 0.022 ms 20/311801 > (0.0064%) > [ 3] 35.0-40.0 sec 437 MBytes 733 Mbits/sec 0.020 ms 202/311857 > (0.065%) > [ 3] 40.0-45.0 sec 437 MBytes 733 Mbits/sec 0.017 ms 242/311921 > (0.078%) > [ 3] 45.0-50.0 sec 437 MBytes 733 Mbits/sec 0.021 ms 280/311890 > (0.09%) > [ 3] 50.0-55.0 sec 438 MBytes 734 Mbits/sec 0.019 ms 0/312119 > (0%) > [ 3] 55.0-60.0 sec 436 MBytes 732 Mbits/sec 0.018 ms 152/311339 > (0.049%) > [ 3] 60.0-65.0 sec 437 MBytes 734 Mbits/sec 0.017 ms 113/312048 > (0.036%) > [ 3] 65.0-70.0 sec 437 MBytes 733 Mbits/sec 0.023 ms 180/311756 > (0.058%) > [ 3] 70.0-75.0 sec 437 MBytes 734 Mbits/sec 0.020 ms 0/311960 > (0%) > [ 3] 75.0-80.0 sec 437 MBytes 734 Mbits/sec 0.013 ms 118/312060 > (0.038%) > [ 3] 80.0-85.0 sec 437 MBytes 734 Mbits/sec 0.019 ms 122/312060 > (0.039%) > [ 3] 85.0-90.0 sec 437 MBytes 733 Mbits/sec 0.025 ms 55/311904 > (0.018%) > [ 3] 90.0-95.0 sec 437 MBytes 733 Mbits/sec 0.024 ms 259/312002 > (0.083%) > [ 3] 0.0-97.0 sec 8.28 GBytes 733 Mbits/sec 0.034 ms 2271/6053089 > (0.038%) > > Sometimes I even see packet disorder report from the iperf receipt part. > > I didn't expect such a performance in terms of delay and throughput and I > would link to find an explanation. That's why I need your help. > > Allow me to tell you some particularities of the machine that runs the DPDK > application and the environment which could help us explain this behaviour. > > > 1. When I run the application I running using the "Debug" environment of > the Eclipse in Linux OpenSuse 42.3 Leap. > 2. The hugepages size in this machine is 2 MB > 3. 1024 hugepages has been reserved for the application > 4. lscpu displayed subsequently > > cuda1@cuda1:~/eclipse-workspace/SimpleModelDPDK> lscpu > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 8 > On-line CPU(s) list: 0-7 > Thread(s) per core: 1 > Core(s) per socket: 4 > Socket(s): 2 > NUMA node(s): 2 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 26 > Model name: Intel(R) Xeon(R) CPU E5506 @ 2.13GHz > Stepping: 5 > CPU MHz: 2133.000 > CPU max MHz: 2133.0000 > CPU min MHz: 1600.0000 > BogoMIPS: 4267.10 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 256K > L3 cache: 4096K > NUMA node0 CPU(s): 0-3 > NUMA node1 CPU(s): 4-7 > Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge > mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall > nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology > nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 > xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm dtherm retpoline kaiser > tpr_shadow vnmi flexpriority ept vpid > > 5. Routing pipeline is executed using core 1. Master pipeline is executed > using core 0 and new ARP pipeline is executed using core 2. > > 6. The two NICs I am using seems not to be assigned to any NUMA node > > cuda1@cuda1:~/eclipse-workspace/SimpleModelDPDK> cat > /sys/bus/pci/devices/0000\:04\:00.0/numa_node > -1 > cuda1@cuda1:~/eclipse-workspace/SimpleModelDPDK> cat > /sys/bus/pci/devices/0000\:04\:00.1/numa_node > -1 > > 7. According to the pipeline ROUTING statistics (regarding table 0 and > table 1) very few miss drops at table 0 are reported and do not coincide at > all with the ones reported by iperf (iperf drops are much higher than the > table 0 and table 1 drops) and also the links used in the application do > not report any drop at all. > > So where are these packets dropped? > > Any of you have an idea if this particularities from my PC can justify this > behaviour? > > I need to find an answer to this because I expected a much better > performance according to the DPDK performance expectations. > > Thanks for your attention > > Victor > What NIC?