From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx.bisdn.de (mx.bisdn.de [185.27.182.31]) by dpdk.org (Postfix) with ESMTP id 327C5C32A for ; Fri, 5 Jun 2015 17:13:12 +0200 (CEST) Received: from [172.16.250.156] (unknown [172.16.250.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mx.bisdn.de (Postfix) with ESMTPSA id E498BA3376 for ; Fri, 5 Jun 2015 17:13:10 +0200 (CEST) Message-ID: <5571BC86.8060909@bisdn.de> Date: Fri, 05 Jun 2015 17:13:10 +0200 From: Marc Sune User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Icedove/31.6.0 MIME-Version: 1.0 To: dev@dpdk.org References: In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] KNI performance X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Jun 2015 15:13:12 -0000 On 05/06/15 17:06, Jay Rolette wrote: > The past few days I've been trying to chase down why operations over KNI > are so bloody slow. To give you an idea how bad it is, we did a simple test > over an NFS mount: > > # Mount over a non-KNI interface (eth0 on vanilla Ubuntu 14.04 LTS) > $ time $(ls -last -R /mnt/sfs2008 > /dev/null) > real 11m58.224s > user 0m10.758s > sys 0m25.050s > > # Reboot to make sure NFS cache is cleared and mount over a KNI interface > $ time $(ls -last -R /mnt/sfs2008 > /dev/null) > real 87m36.295s > user 0m14.552s > sys 0m25.949s > > Packet captures showed a pretty consistent ~4ms delay. Get a READDIRPLUS > reply from NFS server and the TCP stack on the DPDK/KNI system took about > 4ms to ACK the reply. It isn't just on ACK packets either. If there was no > ACK required, there would be a 4ms delay before the next call was sent > (ACCESS, LOOKUP, another READDIR, etc.). > > This is running on top of a real DPDK app, so there are various queues and > ring-buffers in the path between KNI and the wire, so I started there. Long > story short, worst case, those could only inject ~120us of latency into the > path. > > Next stop was KNI itself. Ignoring a few minor optos I found, nothing in > the code looked like it could account for 4ms of latency. That wasn't quite > right though... > > Here's the code for the processing loop in kni_thread_single(): > > while (!kthread_should_stop()) { > down_read(&kni_list_lock); > for (j = 0; j < KNI_RX_LOOP_NUM; j++) { > list_for_each_entry(dev, &kni_list_head, list) { > #ifdef RTE_KNI_VHOST > kni_chk_vhost_rx(dev); > #else > kni_net_rx(dev); > #endif > kni_net_poll_resp(dev); > } > } > up_read(&kni_list_lock); > /* reschedule out for a while */ > schedule_timeout_interruptible(usecs_to_jiffies( \ > KNI_KTHREAD_RESCHEDULE_INTERVAL)); > > Turns out the 4ms delay is due to the schedule_timeout() call. The code > specifies a 5us sleep, but the call only guarantees a sleep of *at least* > the time specified. > > The resolution of the sleep is controlled by the timer interrupt rate. If > you are using a kernel from one of the usual Linux distros, HZ = 250 on > x86. That works out nicely to a 4ms period. The KNI kernel thread was going > to sleep and frequently not getting woken up for nearly 4ms. > > We rebuilt the kernel with HZ = 1000 and things improved considerably: > > # Mount over a KNI interface, HZ=1000 > $ time $(ls -last -R /mnt/sfs2008 > /dev/null) > > real 21m8.478s > user 0m13.824s > sys 0m18.113s > > Still not where I'd like to get it, but much, much better. > > Hopefully my pain is your gain and this helps other KNI users. Jay, If you set CONFIG_RTE_KNI_PREEMPT_DEFAULT to 'n' you should see a reduced latency and delay since there is no preemption (though sacrifices 1 CPU for the kni kmod): http://patchwork.dpdk.org/dev/patchwork/patch/3304/ However, KNI is still pretty slow. Even considering that there will always be at least 1 copy involved, I still think is too slow. I didn't had time to look closer yet. Marc > > Jay