From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io0-f169.google.com (mail-io0-f169.google.com [209.85.223.169]) by dpdk.org (Postfix) with ESMTP id 683BB2C27 for ; Thu, 13 Apr 2017 02:07:34 +0200 (CEST) Received: by mail-io0-f169.google.com with SMTP id l7so60838234ioe.3 for ; Wed, 12 Apr 2017 17:07:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=lVtLiy98Kwq34JE0C+QZQS2vZrDQv2Uti4Su7aYwr4c=; b=oO9R46iY4HgfIVafgdoCg2GrM+p0rSxMmMGFbASBW1SdMW8iw+OC3YHvXqLT0aDe28 aCnQ1jrXcRKmdASn5dFH2DdT03R+8Yji6AB4spR9nEeq/3sQ/P8+7yzBH1oCp0AiJgXu n6HsGFHTkj/t9mI9LHf95YITvfRTKV73IXKYf/GuMRv2VNOTAKk/39xnLUu3e/uCq00F pA282yqchaiKcuPTmpzxGgyFwiCoQOh73CowVt4c9WQWIIOHMSBeCV+e0gfxQQ0CFbTt Z7AwbUS67DBT/YBRhfc6XdMEIXnB2JO6EcfnMqU6v5h61tV78kQNLtbS9g+2NxfkrKje gtiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=lVtLiy98Kwq34JE0C+QZQS2vZrDQv2Uti4Su7aYwr4c=; b=SCtn8p1059Mt7Z7sTHabvaYLdBUREv/m6WC72vsCju/hhb9QbqKwXHcEA2QJ03cSCM 50x6XS5SF4nQ1NZDr6Eh1GtdNQkqE53OsWakycc3mG6505RnfozXkSa27cNIAH0jJYPo SHcdO/YT52YV5nwPvk9WHmiiJrbXZrd0LvqO8ZmmnS7yFjaADGlmfgyP7o9czCEFDNqr hkCt7QihCl2zigxXal/p4XnCBrZ6NsNz44r9zEUQySuvLFFldo4CC4X/wkQkvGcH/Gkh zxXo3cLzdbT1tYn05tMpsm1q1fEfTurGXwNmqmuw8pNlj8DTpS88/ncxp20DfIGgVvl+ AqPQ== X-Gm-Message-State: AN3rC/51hjO3FrAdcnGw/KBLt2VDCQKEgS4wth3AwFwVX6N7M06e0+g4 wfh/qys5I8QsLAedDaV3E+MT4XJv6A== X-Received: by 10.36.173.19 with SMTP id c19mr890622itf.76.1492042053763; Wed, 12 Apr 2017 17:07:33 -0700 (PDT) MIME-Version: 1.0 Received: by 10.79.145.19 with HTTP; Wed, 12 Apr 2017 17:06:53 -0700 (PDT) In-Reply-To: References: From: Shihabur Rahman Chowdhury Date: Wed, 12 Apr 2017 20:06:53 -0400 Message-ID: To: "Wiles, Keith" Cc: "users@dpdk.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] Low Rx throughput when using Mellanox ConnectX-3 card with DPDK X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Apr 2017 00:07:34 -0000 We've disabled the pause frames. That also disables flow control I assume. Correct me if I am wrong. On the pktgen side, the dropped field and overrun field for Rx keeps increasing for ifconfig. Btw, both overrun and dropped fields have the exact same value always. Our ConnectX3 NIC has just a single 10G port. So there is no way to create such loopback connection. Thanks Shihabur Rahman Chowdhury David R. Cheriton School of Computer Science University of Waterloo On Wed, Apr 12, 2017 at 6:41 PM, Wiles, Keith wrote: > > > On Apr 12, 2017, at 4:00 PM, Shihabur Rahman Chowdhury < > shihab.buet@gmail.com> wrote: > > > > Hello, > > > > We are running a simple DPDK application and observing quite low > > throughput. We are currently testing a DPDK application with the > following > > setup > > > > - 2 machines with 2xIntel Xeon E5-2620 CPUs > > - Each machine with a Mellanox single port 10G ConnectX3 card > > - Mellanox DPDK version 16.11 > > - Mellanox OFED 4.0-2.0.0.1 and latest firmware for ConnectX3 > > > > The application is doing almost nothing. It is reading a batch of 64 > > packets from a single rxq, swapping the mac of each packet and writing it > > back to a single txq. The rx and tx is being handled by separate lcores > on > > the same NUMA socket. We are running pktgen on another machine. With 64B > > sized packets we are seeing ~14.8Mpps Tx rate and ~7.3Mpps Rx rate in > > pktgen. We checked the NIC on the machine running the DPDK application > > (with ifconfig) and it looks like there is a large number of packets > being > > dropped by the interface. Our connectx3 card should be theoretically be > > able to handle 10Gbps Rx + 10Gbps Tx throughput (with channel width 4, > the > > theoretical max on PCIe 3.0 should be ~31.2Gbps). Interestingly, when Tx > > rate is reduced in pktgent (to ~9Mpps), the Rx rate increases to ~9Mpps. > > Not sure what is going on here, when you drop the rate to 9Mpps I assume > you stop getting missed frames. > Do you have flow control enabled? > > On the pktgen side are you seeing missed RX packets? > Did you loopback the cable from pktgen machine to the other port on the > pktgen machine and did you get the same Rx/Tx performance in that > configuration? > > > > > We would highly appriciate if we could get some pointers as to what can > be > > possibly causing this mismatch in Rx and Tx. Ideally, we should be able > to > > see ~14Mpps Rx well. Is it because we are using a single port? Or > something > > else? > > > > FYI, we also ran the sample l2fwd application and test-pmd and got > > comparable results in the same setup. > > > > Thanks > > Shihab > > Regards, > Keith > >