From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 68F8BA0352 for ; Wed, 6 May 2020 07:14:36 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 03E3A1D662; Wed, 6 May 2020 07:14:36 +0200 (CEST) Received: from mail-wr1-f66.google.com (mail-wr1-f66.google.com [209.85.221.66]) by dpdk.org (Postfix) with ESMTP id B9A9B1D65A for ; Wed, 6 May 2020 07:14:34 +0200 (CEST) Received: by mail-wr1-f66.google.com with SMTP id k1so566992wrx.4 for ; Tue, 05 May 2020 22:14:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=0nuaG+BUUb9P5rETDjWBWkFe0c+wxDjbh/d3FqmIQdQ=; b=jUCDuvScH2Z3OCkrkHLSROsXH6tZKuDaTAY/HYoreBDZRFmBou8BYKbe1Hn07Tm698 GKrAs0ymm6+PLc+/t44HMJqAwooJlm7ngPRS42CPPkPjKC4EXHJ1RtAhj7H8Az8RfZYe +kB4EJgipg7wimfQIF85/Te+++GVSsMw09MtsQCKAv3uoHtcWBAs/qOGyUcUj6FI5ljR BdvEAAZT+HR4Q1BzpuL4Nie6ADyGR/vN3pi3MJmPzUMM0iSzfsQBXHj+ZCCxLSGs1pac FlzhJ0Ea6RcmgHWvLYhDeNDSu3WH00KqBWn70IeYalMp7kEnIjNUn/So2auypJ7T+KWz xvFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=0nuaG+BUUb9P5rETDjWBWkFe0c+wxDjbh/d3FqmIQdQ=; b=c86f/k9t110v7EiPbYKUzR4rsX+0ZDCsOq0ttHdkdXjq1d4AgIyy1uRljsYAcA+/YE Ffuo2En6WPNHWFU0LJKwndVYgP9gI++/OiuGAADgHrjYv5D28pBH5U4xmiL+UGwaJATn G/9ZdgxiQFY8jpWFHnJbQZO8vf3VJCYgQBXIB3yQKXySJwn/D92QRFdn3VVUj2Em5qi2 r/4I0YktEbqzqGjJ4X4l2vFt7Fo6tC7723Z2VHdNTmyG39aQJ4Hdmh3AqkgGxkh/IqAm ChbNmGQxcOR/wJf7t9bJaEBzZBQnzCnhxepbMpzT7/xJcEBhc6KZ6jZItlVPhsRA5zzf +GJg== X-Gm-Message-State: AGi0Pubsevg9rVgMdaPTbmGmmiiVrIPWcg+2iorDsVO6Ee1HyqBANB4C aktHkyeZJ4bRPc7lSVUcA4rqywUoJPaQ23j8y8c0gQ/AZmE= X-Google-Smtp-Source: APiQypJMSfxDfprtMqGy9dBM+6GhiRc29e8AKIoFm1RHJUt5aJ5EqLgZRyMTkxWGdgLDnmPUtsOUnmumDk74Y5gfAEo= X-Received: by 2002:a5d:664f:: with SMTP id f15mr7098177wrw.72.1588742074065; Tue, 05 May 2020 22:14:34 -0700 (PDT) MIME-Version: 1.0 From: Pavel Vajarov Date: Wed, 6 May 2020 08:14:20 +0300 Message-ID: To: users@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] Peformance troubleshouting of TCP/IP stack over DPDK. X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi there, We are trying to compare the performance of DPDK+FreeBSD networking stack vs standard Linux kernel and we have problems finding out why the former is slower. The details are below. There is a project called F-Stack . It glues the networking stack from FreeBSD 11.01 over DPDK. We made a setup to test the performance of transparent TCP proxy based on F-Stack and another one running on Standard Linux kernel. We did the tests on KVM with 2 cores (Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz) and 32GB RAM. 10Gbs NIC was attached in passthrough mode. The application level code, the one which handles epoll notifications and memcpy data between the sockets, of the both proxy applications is 100% the same. Both proxy applications are single threaded and in all tests we pinned the applications on core 1. The interrupts from the network card were pinned to the same core 1 for the test with the standard Linux application. Here are the test results: 1. The Linux based proxy was able to handle about 1.7-1.8 Gbps before it started to throttle the traffic. No visible CPU usage was observed on core 0 during the tests, only core 1, where the application and the IRQs were pinned, took the load. 2. The DPDK+FreeBSD proxy was able to thandle 700-800 Mbps before it started to throttle the traffic. No visible CPU usage was observed on core 0 during the tests only core 1, where the application was pinned, took the load. In some of the latter tests I did some changes to the number of read packets in one call from the network card and the number of handled events in one call to epoll. With these changes I was able to increase the throughput to 900-1000 Mbps but couldn't increase it more. 3. We did another test with the DPDK+FreeBSD proxy just to give us some more info about the problem. We disabled the TCP proxy functionality and let the packets be simply ip forwarded by the FreeBSD stack. In this test we reached up to 5Gbps without being able to throttle the traffic. We just don't have more traffic to redirect there at the moment. So the bottlneck seem to be either in the upper level of the network stack or in the application code. There is a huawei switch which redirects the traffic to this server. It regularly sends arping and if the server doesn't respond it stops the redirection. So we assumed that when the redirection stops it's because the server throttles the traffic and drops packets and can't respond to the arping because of the packets drop. The whole application can be very roughly represented in the following way: - Write pending outgoing packets to the network card - Read incoming packets from the network card - Push the incoming packets to the FreeBSD stack - Call epoll_wait/kevent without waiting - Handle the events - loop from the beginning According to the performance profiling that we did, aside from packet processing, about 25-30% of the application time seems to be spent in the epoll_wait/kevent even though the `timeout` parameter of this call is set to 0 i.e. it shouldn't block waiting for events if there is none. I can give you much more details and code for everything, if needed. My questions are: 1. Does somebody have observations or educated guesses about what amount of traffic should I expect the DPDK + FreeBSD stack + kevent to process in the above scenario? Are the numbers low or expected? We've expected to see better performance than the standard Linux kernel one but so far we can't get this performance. 2. Do you think the diffrence comes because of the time spending handling packets and handling epoll in both of the tests? What do I mean. For the standard Linux tests the interrupts handling has higher priority than the epoll handling and thus the application can spend much more time handling packets and processing them in the kernel than handling epoll events in the user space. For the DPDK+FreeBSD case the time for handling packets and the time for processing epolls is kind of equal. I think, that this was the reason why we were able to get more performance increasing the number of read packets at one go and decreasing the epoll events. However, we couldn't increase the throughput enough with these tweaks. 3. Can you suggest something else that we can test/measure/profile to get better idea what exactly is happening here and to improve the performance more? Any help is appreciated! Thanks in advance, Pavel.