From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-la0-f46.google.com (mail-la0-f46.google.com [209.85.215.46]) by dpdk.org (Postfix) with ESMTP id D0C9C8D94 for ; Tue, 22 Sep 2015 07:36:21 +0200 (CEST) Received: by lagj9 with SMTP id j9so82251864lag.2 for ; Mon, 21 Sep 2015 22:36:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=+ot7jx9g53OoeYegSA9goUPn00Zwuf/AuI2RNRRY2ms=; b=GVNOSW12ZYtbmkW3xEo3SM9B986XH8pxaTCahjwiXE/G/jm3tpH64EmDAj887V1y3I fCRI+28Vq+mf5hITpqdvCh5gm9aWkynzZfTFztdnQ9/x4j4yLVZZWLNZ5/3/x+T23Aiq G2mDs0ND8NcD9zT55CRBwwbYryRCdWgX//0CXDzkcJXcGYkJspZq4py7ulxP7iaiVm3Y qzYfMC8r7aUfa3miRVD9+avZOIUM3omHiu1HyuFqWe9KHL61OTFkAKinRAgWkl5YBRdQ VPZAQsGR1BzWUDEawuDLMr/DM4uAItqzJloge24umbdxBojPV5Dhq1pcJtDr1+CErmu9 PB5A== MIME-Version: 1.0 X-Received: by 10.25.206.133 with SMTP id e127mr2205871lfg.57.1442900181240; Mon, 21 Sep 2015 22:36:21 -0700 (PDT) Received: by 10.112.13.103 with HTTP; Mon, 21 Sep 2015 22:36:21 -0700 (PDT) Date: Tue, 22 Sep 2015 14:36:21 +0900 Message-ID: From: Moon-Sang Lee To: dev@dpdk.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-dev] about poor KNI performance X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Sep 2015 05:36:22 -0000 I'm running NAT with iptables on server M2, and two linux PCs, M1 and M3, are directly connected to M2 as below. All interfaces are 1G NICs over 1G links. [ M1 ]--------p1p3-[ M2 ]-p1p4--------[ M3 ] Without KNI nor DPDK, native iptables in Linux supports 360Mbps for UDP and 545Mbps for TCP. With KNI, it drops almost every packet for UDP (i.e. 87% of received packets are dropped), and shows 42Mbps for TCP. If I use KNI with DPDK, it runs in poll-mode thus there should not be interrupts. In my experiment, something strange is ksoftirqd consumes 50% CPU loads with KNI. Why ksoftirqd works such hard? I'm using the KNI application in the example directory of DPDK 2.1.0 source tree with options -c 0xf0 -n 4 -- -p 0x3 -P --config="(0,4,6,8),(1,5,7,9)". (i.e. rte_kni.ko is loaded with kthread_mode=multiple option) Any comments is appreciated. ##### without KNI (i.e. with native iptables) ##### [mslee@localhost ~]$ iperf3 -s -i 10 ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.110.4, port 48099 [ 5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 54364 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-10.00 sec 376 MBytes 315 Mbits/sec 0.002 ms 471622/3548257 (13%) [ 5] 10.00-10.04 sec 1.53 MBytes 326 Mbits/sec 0.003 ms 1520/14071 (11%) - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-10.04 sec 435 MBytes 363 Mbits/sec 0.003 ms 473142/3562328 (13%) ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.110.4, port 48100 [ 5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 48101 [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 650 MBytes 545 Mbits/sec [ 5] 10.00-10.04 sec 2.53 MBytes 563 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 5] 0.00-10.04 sec 653 MBytes 546 Mbits/sec 97 sender [ 5] 0.00-10.04 sec 652 MBytes 545 Mbits/sec receiver ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- ##### with KNI (i.e. with KNI + native iptables) ##### [mslee@localhost ~]$ iperf3 -s -i 10 ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.110.4, port 48102 [ 5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 60867 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-10.00 sec 57.9 MBytes 48.6 Mbits/sec 0.021 ms 3239121/3713340 (87%) [ 5] 10.00-10.27 sec 278 KBytes 8.37 Mbits/sec 0.018 ms 16319/18541 (88%) - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-10.27 sec 456 MBytes 372 Mbits/sec 0.018 ms 3255440/3731881 (87%) ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.110.4, port 48103 [ 5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 48104 [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 50.6 MBytes 42.4 Mbits/sec [ 5] 10.00-10.04 sec 204 KBytes 44.2 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 5] 0.00-10.04 sec 52.0 MBytes 43.4 Mbits/sec 151 sender [ 5] 0.00-10.04 sec 50.8 MBytes 42.4 Mbits/sec receiver ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- ##### snapshot of top command ##### PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9519 root 20 0 8659492 20404 19948 R 400.0 0.1 10:41.09 unatd-kni 114 root 20 0 0 0 0 S 45.7 0.0 1:31.46 ksoftirqd/8 9525 root 20 0 0 0 0 S 17.2 0.0 0:07.89 kni_p1p3 9590 root 20 0 0 0 0 S 1.2 0.0 0:01.57 kni_p1p4 -- Moon-Sang Lee, SW Engineer Email: sang0627@gmail.com Wisdom begins in wonder. *Socrates*