[dpdk-dev] about poor KNI performance

[dpdk-dev] about poor KNI performance

[Moon-Sang Lee]

I'm running NAT with iptables on server M2, and two linux PCs, M1 and M3,
are directly connected to M2 as below. All interfaces are 1G NICs over 1G
links.

[ M1 ]--------p1p3-[ M2 ]-p1p4--------[ M3 ]

Without KNI nor DPDK, native iptables in Linux supports 360Mbps for UDP and
545Mbps for TCP.
With KNI, it drops almost every packet for UDP (i.e. 87% of received
packets are dropped),
and shows 42Mbps for TCP.

If I use KNI with DPDK, it runs in poll-mode thus there should not be
interrupts.
In my experiment,  something strange is ksoftirqd consumes 50% CPU loads
with KNI.
Why ksoftirqd works such hard?

I'm using the KNI application in the example directory of DPDK 2.1.0 source
tree with options
-c 0xf0 -n 4 -- -p 0x3 -P --config="(0,4,6,8),(1,5,7,9)".
(i.e. rte_kni.ko is loaded with kthread_mode=multiple option)

Any comments is appreciated.



##### without KNI (i.e. with native iptables) #####

[mslee@localhost ~]$ iperf3 -s -i 10
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.110.4, port 48099
[  5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 54364
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total
Datagrams
[  5]   0.00-10.00  sec   376 MBytes   315 Mbits/sec  0.002 ms
 471622/3548257 (13%)
[  5]  10.00-10.04  sec  1.53 MBytes   326 Mbits/sec  0.003 ms  1520/14071
(11%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total
Datagrams
[  5]   0.00-10.04  sec   435 MBytes   363 Mbits/sec  0.003 ms
 473142/3562328 (13%)
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.110.4, port 48100
[  5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 48101
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec   650 MBytes   545 Mbits/sec
[  5]  10.00-10.04  sec  2.53 MBytes   563 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-10.04  sec   653 MBytes   546 Mbits/sec   97             sender
[  5]   0.00-10.04  sec   652 MBytes   545 Mbits/sec
 receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------





##### with KNI (i.e. with KNI + native iptables) #####

[mslee@localhost ~]$ iperf3 -s -i 10
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.110.4, port 48102
[  5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 60867
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total
Datagrams
[  5]   0.00-10.00  sec  57.9 MBytes  48.6 Mbits/sec  0.021 ms
 3239121/3713340 (87%)
[  5]  10.00-10.27  sec   278 KBytes  8.37 Mbits/sec  0.018 ms  16319/18541
(88%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total
Datagrams
[  5]   0.00-10.27  sec   456 MBytes   372 Mbits/sec  0.018 ms
 3255440/3731881 (87%)
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.110.4, port 48103
[  5] local 192.168.110.5 port 5201 connected to 192.168.110.4 port 48104
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  50.6 MBytes  42.4 Mbits/sec
[  5]  10.00-10.04  sec   204 KBytes  44.2 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-10.04  sec  52.0 MBytes  43.4 Mbits/sec  151             sender
[  5]   0.00-10.04  sec  50.8 MBytes  42.4 Mbits/sec
 receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------




##### snapshot of top command #####
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

 9519 root      20   0 8659492  20404  19948 R 400.0  0.1  10:41.09
unatd-kni
  114 root       20   0            0           0      0 S       45.7  0.0
1:31.46 ksoftirqd/8
 9525 root      20   0            0           0      0 S       17.2  0.0
0:07.89 kni_p1p3
 9590 root      20   0            0           0      0 S         1.2  0.0
0:01.57 kni_p1p4



-- 
Moon-Sang Lee, SW Engineer
Email: sang0627@gmail.com
Wisdom begins in wonder. *Socrates*