From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2099DA04A4 for ; Wed, 27 May 2020 07:12:02 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 2DFAE1D625; Wed, 27 May 2020 07:12:01 +0200 (CEST) Received: from mail-wr1-f68.google.com (mail-wr1-f68.google.com [209.85.221.68]) by dpdk.org (Postfix) with ESMTP id 620131D60C for ; Wed, 27 May 2020 07:11:59 +0200 (CEST) Received: by mail-wr1-f68.google.com with SMTP id i15so22629137wrx.10 for ; Tue, 26 May 2020 22:11:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=3ZfIF06gksU7/Max6AkDIMJCGnM0qm3gw/rAyY8IfG0=; b=gdvUtyIFJnz0y/REs8++/u4U9BU04cq7lmMaD3XcKwNGMAbYC0AhyI+o+WcZ4l8aDo hq+uQLx1+2N/mVkjcrh5nf49yTY1yzwbi6pnYMOzHm3a49bOdLbU9c4VDZFkDeRk1jXF qYPdLr9IPNSV+Lk1JbQuN4SBqXJvg6Hct36h3vFC8/jxd+OWDMruO2a8zYNAUIHGplOW FWM3P8WAAu4NcbtvF4vPENkWg9SdgWJFlZWLt1Rmf5sFNbanwTPvWvtO2llRk7B6h1Uq kPUKt0CIj423gVyMYlHndJ4NqQ3QtDhmkcTxHrnHJkMj5FgF3/OsqmGYDHw1I3Z5TSLT JJqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=3ZfIF06gksU7/Max6AkDIMJCGnM0qm3gw/rAyY8IfG0=; b=MKM/lLgorzlMrthelBXcr3VoJZbCr88nPcGCGAEdvkrjEwZpAdbrqHusiUp3n31Prs 2B4Pvz8VOuijahUGdFtMyj8jYwsG85wJBSTXMyOcwlSLj3Rvt0c4FYfdfWuZsltQRTVX FzAiXNvHbTQoFyVZhG2qNb8TxJYLYFZa/FzHrk0zRpk98Os2Fz4P6cHe0PlYqLFr5VD9 xxn26x5n2WUIOXSt8EJOvBpANZ+TnIhP1m07qRtiN7AZKCtyXIaUln4RdRy23t3EPm6l gXjyAksioX8BtntMBav1Z/QY6z3zuOtkaSZF9vckZd8/+rUcVZO8Kp7+NBqHUT/5ynfW C9tQ== X-Gm-Message-State: AOAM5330XPB69r4ULVhIzwyCSRbKimblK2pnAYFKa24/KoaHnZuPii1N +JKYnFG9bh6c4FQ+T+NwPALjVuCyF4pBSzY6kAQ= X-Google-Smtp-Source: ABdhPJy1g9qa7bkABAAn61AMrPgRucl6n2ivR8VVgxnTAszD4lxYzpmAlr1vNUxXs3BklZyHbPjiHR5IlHYm87/KbDQ= X-Received: by 2002:adf:fdc1:: with SMTP id i1mr25205006wrs.0.1590556319041; Tue, 26 May 2020 22:11:59 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Pavel Vajarov Date: Wed, 27 May 2020 08:11:44 +0300 Message-ID: To: Vincent Li Cc: users Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] Peformance troubleshouting of TCP/IP stack over DPDK. X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" > > > Hi there, > > > > We are trying to compare the performance of DPDK+FreeBSD networking stack > > vs standard Linux kernel and we have problems finding out why the former > is > > slower. The details are below. > > > > There is a project called F-Stack . > > It glues the networking stack from > > FreeBSD 11.01 over DPDK. We made a setup to test the performance of > > transparent > > TCP proxy based on F-Stack and another one running on Standard Linux > > kernel. > > I assume you wrote your own TCP proxy based on F-Stack library? > Yes, I wrote transparent TCP proxy based on the F-Stack library for the tests. The thing is that we have our transparent caching proxy running on Linux and now we try to find a ways to improve its performance and hardware requirements. > > > > Here are the test results: > > 1. The Linux based proxy was able to handle about 1.7-1.8 Gbps before it > > started to throttle the traffic. No visible CPU usage was observed on > core > > 0 during the tests, only core 1, where the application and the IRQs were > > pinned, took the load. > > 2. The DPDK+FreeBSD proxy was able to thandle 700-800 Mbps before it > > started to throttle the traffic. No visible CPU usage was observed on > core > > 0 during the tests only core 1, where the application was pinned, took > the > > load. In some of the latter tests I did some changes to the number of > read > > packets in one call from the network card and the number of handled > events > > in one call to epoll. With these changes I was able to increase the > > throughput > > to 900-1000 Mbps but couldn't increase it more. > > 3. We did another test with the DPDK+FreeBSD proxy just to give us some > > more info about the problem. We disabled the TCP proxy functionality and > > let the packets be simply ip forwarded by the FreeBSD stack. In this test > > we reached up to 5Gbps without being able to throttle the traffic. We > just > > don't have more traffic to redirect there at the moment. So the bottlneck > > seem to be either in the upper level of the network stack or in the > > application > > code. > > > > I once tested F-Stack ported Nginx and used Nginx TCP proxy, I could > achieve above 6Gbps with iperf. After seeing your email, I setup PCI > passthrough to KVM VM and ran F-Stack Nginx as webserver > with http load test, no proxy, I could achieve about 6.5Gbps > Can I ask on how many cores you run the Nginx? The results from our tests are from single core. We are trying to reach max performance on single core because we know that the F-stack soulution has linear scalability. We tested in on 3 cores and got around 3 Gbps which is 3 times the result on single core. Also we test with traffic from one internet service provider. We just redirect few ip pools to the test machine for the duration of the tests and see at which point the proxy will start choking the traffic and the switch the traffic back. > There is a huawei switch which redirects the traffic to this server. It > > regularly > > sends arping and if the server doesn't respond it stops the redirection. > > So we assumed that when the redirection stops it's because the server > > throttles the traffic and drops packets and can't respond to the arping > > because > > of the packets drop. > > I did have some weird issue with ARPing of F-Stack, I manually added > static ARP for F-Stack interface for each F-Stack process, not sure if it > is related to your ARPing, see > https://github.com/F-Stack/f-stack/issues/515 > Hmm, I've missed that. Thanks a lot for it because it may help for the tests and for the next stage.