From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f170.google.com (mail-wi0-f170.google.com [209.85.212.170]) by dpdk.org (Postfix) with ESMTP id 454545A54 for ; Fri, 5 Jun 2015 17:06:55 +0200 (CEST) Received: by wiga1 with SMTP id a1so23984962wig.0 for ; Fri, 05 Jun 2015 08:06:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=+FCkFr2fPngJ4IuCoV9tke+Bi1NYN0eA7tHyMUVlnTA=; b=es3ys/7S0CRS9g+da/C+RhfglCoj43Hup1K4ssSWu0LH8CzoIRzv+v1JpJBH5NxY6A D/6qx/tCzGUnb4cBNM+poKo6MlmIaLWsCPn0K70Zx13vXflTREcJCs0Fj/ddBRiZtxrE QNaqwHhw6b50r8iGQxUe1PPF3wFh0AZ86XB7zQXWKvqikuMv29vesk/fkzTBQ5FUh1hF XAjib6b6ny8MN5bu1wF1EMqGnO9BcCHuIqFy4jiuohocR+yoPNU5HkuWt+O7HnsNKkAy /yhYsvaGNbVHUC6oV+aeMC/Z9fx6WEMkWVxEyEFf0ywPEIKb3arA+NZR1tQiEyKR8otP 9gzQ== X-Gm-Message-State: ALoCoQm3RVT8/UjUco00HhZBHut5+MwPEdj58uolghona6MWDfkyiVfex25hGcZvAdZT7qeamP3M MIME-Version: 1.0 X-Received: by 10.194.157.168 with SMTP id wn8mr7082479wjb.79.1433516815052; Fri, 05 Jun 2015 08:06:55 -0700 (PDT) Received: by 10.194.36.193 with HTTP; Fri, 5 Jun 2015 08:06:54 -0700 (PDT) Date: Fri, 5 Jun 2015 10:06:54 -0500 Message-ID: From: Jay Rolette To: DPDK Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-dev] KNI performance X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Jun 2015 15:06:55 -0000 The past few days I've been trying to chase down why operations over KNI are so bloody slow. To give you an idea how bad it is, we did a simple test over an NFS mount: # Mount over a non-KNI interface (eth0 on vanilla Ubuntu 14.04 LTS) $ time $(ls -last -R /mnt/sfs2008 > /dev/null) real 11m58.224s user 0m10.758s sys 0m25.050s # Reboot to make sure NFS cache is cleared and mount over a KNI interface $ time $(ls -last -R /mnt/sfs2008 > /dev/null) real 87m36.295s user 0m14.552s sys 0m25.949s Packet captures showed a pretty consistent ~4ms delay. Get a READDIRPLUS reply from NFS server and the TCP stack on the DPDK/KNI system took about 4ms to ACK the reply. It isn't just on ACK packets either. If there was no ACK required, there would be a 4ms delay before the next call was sent (ACCESS, LOOKUP, another READDIR, etc.). This is running on top of a real DPDK app, so there are various queues and ring-buffers in the path between KNI and the wire, so I started there. Long story short, worst case, those could only inject ~120us of latency into the path. Next stop was KNI itself. Ignoring a few minor optos I found, nothing in the code looked like it could account for 4ms of latency. That wasn't quite right though... Here's the code for the processing loop in kni_thread_single(): while (!kthread_should_stop()) { down_read(&kni_list_lock); for (j = 0; j < KNI_RX_LOOP_NUM; j++) { list_for_each_entry(dev, &kni_list_head, list) { #ifdef RTE_KNI_VHOST kni_chk_vhost_rx(dev); #else kni_net_rx(dev); #endif kni_net_poll_resp(dev); } } up_read(&kni_list_lock); /* reschedule out for a while */ schedule_timeout_interruptible(usecs_to_jiffies( \ KNI_KTHREAD_RESCHEDULE_INTERVAL)); Turns out the 4ms delay is due to the schedule_timeout() call. The code specifies a 5us sleep, but the call only guarantees a sleep of *at least* the time specified. The resolution of the sleep is controlled by the timer interrupt rate. If you are using a kernel from one of the usual Linux distros, HZ = 250 on x86. That works out nicely to a 4ms period. The KNI kernel thread was going to sleep and frequently not getting woken up for nearly 4ms. We rebuilt the kernel with HZ = 1000 and things improved considerably: # Mount over a KNI interface, HZ=1000 $ time $(ls -last -R /mnt/sfs2008 > /dev/null) real 21m8.478s user 0m13.824s sys 0m18.113s Still not where I'd like to get it, but much, much better. Hopefully my pain is your gain and this helps other KNI users. Jay