From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-vk0-f44.google.com (mail-vk0-f44.google.com [209.85.213.44]) by dpdk.org (Postfix) with ESMTP id 8399737A0 for ; Thu, 22 Jun 2017 20:39:40 +0200 (CEST) Received: by mail-vk0-f44.google.com with SMTP id y70so3637148vky.3 for ; Thu, 22 Jun 2017 11:39:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=eng.ucsd.edu; s=google; h=mime-version:from:date:message-id:subject:to; bh=WFnoM73tw96w3kN4uIerbfCKMNAAlzq8eaAackO/ak8=; b=MB30bHQ3lCsfeH8pBeEUQWzSB/mwgWUr8ZDWBzZtToCTBQIsFjWm3rMy4X5YkdyDqA ECA2PCsI/KFIzO5KHyskaVBh1Ul+cvW4m8rPChHd9XWuSsqX/JRDX/NV/T6mkTPcGwQS hXOonzPUhowcH6CT2kiRLHOnzVOk+xxaBCjBM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=WFnoM73tw96w3kN4uIerbfCKMNAAlzq8eaAackO/ak8=; b=jN2VDgj4H0dou4LBygfbXYz4VeNAZGldA56ptQmhcZVC9TIOs0wldYWkKT6HyQ1hzi 7QoejGdXw7adTsw+2UBhDSaufuJsknXwplyYZF0A62eW7yVfPoC05PfmiL/pFY47H2IY 20DFnzRP9DHRmJRMmEgHWleQUuYteSepaZYzKiZmJvcwltYzNYemdotWtQtsgfzzYrv0 ZNMrgFfNKTCUscf/oFJIAzNa70agi2ujfUXYVX/4KAX1kUhGJ289aD2h9kk/kFopZBUo 3NgLsZm6yLsvuw2A0HvK/Q3poMUgw0bxTrWal/jNxbiNunyqyxvKpqShC8HzNWbZaO1c 9Crg== X-Gm-Message-State: AKS2vOzRlKIivnyC9+MdG0P4bhW4KnsbpBVEnXwszL/9Nj0ligrUy8MT xQg9VWHEvndIW7Cmp2PqLH/RcYGMfISHpTw= X-Received: by 10.31.140.206 with SMTP id o197mr618802vkd.92.1498156779600; Thu, 22 Jun 2017 11:39:39 -0700 (PDT) MIME-Version: 1.0 Received: by 10.103.5.3 with HTTP; Thu, 22 Jun 2017 11:39:39 -0700 (PDT) From: Arjun Roy Date: Thu, 22 Jun 2017 11:39:39 -0700 Message-ID: To: users@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] Mellanox ConnectX-4, DPDK and extreme latency issues X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jun 2017 18:39:40 -0000 Greetings all. I have a weird issue regarding excessive latency using Mellanox ConnectX-4 100Gbe cards, DPDK and packet forwarding. Specifically: running the l3fwd and basicfwd DPDK example programs yields ping latencies of several (5-8) milliseconds. I tried the same test using an Intel X-540 AT2 card on the same systems and the latency was on the order of 4-5 microseconds. Setup: I have three systems, SysA, SysB, and SysC. Each runs Ubuntu 16.04 and kernel 4.4.0-78-generic. Each system is a dual socket numa machine, where each socket is a 12 core (+12 with hyperthreading enabled) Xeon E5-2650. SysA and SysB each have a single Mellanox ConnectX-4 card, connected to numa node 1, showing up as enp129s0f0 and enp129s0f1. SysC has two ConnectX-4 cards, connected to node 0 and node 1. Node 0 has enp4s0f0 and enp4s0f1, while node 1 has enp129s0f0 and enp129s0f1. All machines also have a single dual port Intel X-540 AT2 10Gbe NIC that also supports DPDK. SysC forwards packets between SysA and SysB. SysA is connected to enp129s0f0 on SysC, while SysB is connected to enp4s0f0 on SysC. (Note: I tried a variety of configurations; including connecting SysA and SysB to the same physical cards on SysC, and the same latency issue still persists). No switches involved; all direct connect. If it helps, the driver version is the OFED 4.0-2 and the card firmware is 12.18.2000. Now, with this setup, with normal linux forwarding setup, I can get 0.095 msecs ping on average from SysA to SysB (or vice versa). However, if I run the DPDK forwarding apps, I get about 5-8 msecs. The ping test I'm using is both regular (1 second gaps between pings) and burst mode (flooding ping packets as fast as possible). In either case the latency is 5-8 msecs per ping. I have been running l3fwd with this command line: sudo ./l3fwd -l 2,3 -n4 -w 81:00.0 -w 04:00.0 --socket-mem=1024,1024 -- -p 0x3 -P --config="(1,0,2),(0,0,3)" In this case, I have verified that the cores and numa nodes line up; ie. I'm assigning each port to a core on the local numa node. Regarding my sanity check: I tried the same test with Intel X-540 cards, wired with the same topology (SysA connects to one port on SysC, SysB connects to the other port; note this is the same physical card) and for the same test I get just 4-5 microseconds for ping in flood mode). Any ideas what might be causing multiple milliseconds of latency on the Mellanox cards? Thanks, -Arjun Roy