From: Moon-Sang Lee <sang0627@gmail.com>
To: dev@dpdk.org
Subject: [dpdk-dev] about poor KNI performance
Date: Tue, 22 Sep 2015 14:36:21 +0900 [thread overview]
Message-ID: <CAACK=XeGtfALvfeDqCuTA6ET3CrA8R5ZjbOwmF-NUWVFZufr4w@mail.gmail.com> (raw)
I'm running NAT with iptables on server M2, and two linux PCs, M1 and M3,
are directly connected to M2 as below. All interfaces are 1G NICs over 1G
links.
[ M1 ]--------p1p3-[ M2 ]-p1p4--------[ M3 ]
Without KNI nor DPDK, native iptables in Linux supports 360Mbps for UDP and
545Mbps for TCP.
With KNI, it drops almost every packet for UDP (i.e. 87% of received
packets are dropped),
and shows 42Mbps for TCP.
If I use KNI with DPDK, it runs in poll-mode thus there should not be
interrupts.
In my experiment, something strange is ksoftirqd consumes 50% CPU loads
with KNI.
Why ksoftirqd works such hard?
I'm using the KNI application in the example directory of DPDK 2.1.0 source
tree with options
-c 0xf0 -n 4 -- -p 0x3 -P --config="(0,4,6,8),(1,5,7,9)".
(i.e. rte_kni.ko is loaded with kthread_mode=multiple option)
Any comments is appreciated.
##### without KNI (i.e. with native iptables) #####
[mslee@localhost ~]$ iperf3 -s -i 10
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.110.4, port 48099
[ 5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 54364
[ ID] Interval Transfer Bandwidth Jitter Lost/Total
Datagrams
[ 5] 0.00-10.00 sec 376 MBytes 315 Mbits/sec 0.002 ms
471622/3548257 (13%)
[ 5] 10.00-10.04 sec 1.53 MBytes 326 Mbits/sec 0.003 ms 1520/14071
(11%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Jitter Lost/Total
Datagrams
[ 5] 0.00-10.04 sec 435 MBytes 363 Mbits/sec 0.003 ms
473142/3562328 (13%)
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.110.4, port 48100
[ 5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 48101
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.00 sec 650 MBytes 545 Mbits/sec
[ 5] 10.00-10.04 sec 2.53 MBytes 563 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 5] 0.00-10.04 sec 653 MBytes 546 Mbits/sec 97 sender
[ 5] 0.00-10.04 sec 652 MBytes 545 Mbits/sec
receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
##### with KNI (i.e. with KNI + native iptables) #####
[mslee@localhost ~]$ iperf3 -s -i 10
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.110.4, port 48102
[ 5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 60867
[ ID] Interval Transfer Bandwidth Jitter Lost/Total
Datagrams
[ 5] 0.00-10.00 sec 57.9 MBytes 48.6 Mbits/sec 0.021 ms
3239121/3713340 (87%)
[ 5] 10.00-10.27 sec 278 KBytes 8.37 Mbits/sec 0.018 ms 16319/18541
(88%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Jitter Lost/Total
Datagrams
[ 5] 0.00-10.27 sec 456 MBytes 372 Mbits/sec 0.018 ms
3255440/3731881 (87%)
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.110.4, port 48103
[ 5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 48104
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.00 sec 50.6 MBytes 42.4 Mbits/sec
[ 5] 10.00-10.04 sec 204 KBytes 44.2 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 5] 0.00-10.04 sec 52.0 MBytes 43.4 Mbits/sec 151 sender
[ 5] 0.00-10.04 sec 50.8 MBytes 42.4 Mbits/sec
receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
##### snapshot of top command #####
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9519 root 20 0 8659492 20404 19948 R 400.0 0.1 10:41.09
unatd-kni
114 root 20 0 0 0 0 S 45.7 0.0
1:31.46 ksoftirqd/8
9525 root 20 0 0 0 0 S 17.2 0.0
0:07.89 kni_p1p3
9590 root 20 0 0 0 0 S 1.2 0.0
0:01.57 kni_p1p4
--
Moon-Sang Lee, SW Engineer
Email: sang0627@gmail.com
Wisdom begins in wonder. *Socrates*
reply other threads:[~2015-09-22 5:36 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAACK=XeGtfALvfeDqCuTA6ET3CrA8R5ZjbOwmF-NUWVFZufr4w@mail.gmail.com' \
--to=sang0627@gmail.com \
--cc=dev@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).