From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ig0-f176.google.com (mail-ig0-f176.google.com [209.85.213.176]) by dpdk.org (Postfix) with ESMTP id E26EB37A4 for ; Tue, 17 Nov 2015 00:48:35 +0100 (CET) Received: by igcph11 with SMTP id ph11so68473011igc.1 for ; Mon, 16 Nov 2015 15:48:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=infiniteio_com.20150623.gappssmtp.com; s=20150623; h=mime-version:date:message-id:subject:from:to:content-type; bh=rMo60gQvA/UW+18Xpp4V6Hys2YrvH4toKdPLbrV8vGY=; b=CMKAZZc3pNyzAeVB32/IByK0YhSZ9YqMrY+WgmdDyopt46ogsQ4/ntjE2LtN98yRPp 33kRPqCPbQJRb2owNdHs5yho0XSrYkvIq1ZWQTlZRFRSCavJQXp2rEdgjaHOjATWfeLQ /FBZzmFVnNcFUuCS/IUGNQQnIz3lwJNr6UXMIYyL3WncINrKGnGVmHgFh1Hso4jI0ufy 6LCp2DZsIleTl69QVonwu6Qf0RTwoMy1lwrYiuoeZqYJ9uUEqo8KndFACrM6rZUBBDJ4 pt6fLmRHeffbDaiHW5B2i6jNqxIbLn91bL5dc/rl9+4fPh22TCi8SPkE4QtC+UQmPk8v XK3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=rMo60gQvA/UW+18Xpp4V6Hys2YrvH4toKdPLbrV8vGY=; b=GHRRdKAtc0At3hIG93ekxK7470dqkxGeerDAep0PpTE/yijGqF8I9XhfFbGyjn7c17 7SeGmxaHcIzkSrsnoSz3ztyI3LFWt4H+UTfSAedaInYJJvKJnVBkyXU56k8Xj96Qox9I K6B+HRGCUcG1r65oHGkODWTmv4E4R6J4GQFgxwMxOTvKf1HTs0vYvnxVfHwpvgYVpdLV BeTdBgdE54/G8sbvM78mGUC5QPlE5rr9LVE8rEuu+LSpP3YH2VeHaU7ymQ+hnTV5q3Ey uE5RT4OipaK0Kf79bUjMZ7/4LMF9ApH+tsGOYrodKBixrfhmaoHTjWXsml7gM1oocGVz VH8Q== X-Gm-Message-State: ALoCoQkbe+sEtryepkTBalUHZLtM2GmYhySwFwwEuxmw9IX4AWygRLI7UXjij0NZoqNtq6vLe7sq MIME-Version: 1.0 X-Received: by 10.50.225.102 with SMTP id rj6mr19913441igc.95.1447717715216; Mon, 16 Nov 2015 15:48:35 -0800 (PST) Received: by 10.107.140.15 with HTTP; Mon, 16 Nov 2015 15:48:35 -0800 (PST) Date: Mon, 16 Nov 2015 17:48:35 -0600 Message-ID: From: Matt Laswell To: "dev@dpdk.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-dev] How to approach packet TX lockups X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Nov 2015 23:48:36 -0000 Hey Folks, I sent this to the users email list, but I'm not sure how many people are actively reading that list at this point. I'm dealing with a situation in which my application loses the ability to transmit packets out of a port during times of moderate stress. I'd love to hear suggestions for how to approach this problem, as I'm a bit at a loss at the moment. Specifically, I'm using DPDK 1.6r2 running on Ubuntu 14.04LTS on Haswell processors. I'm using the 82599 controller, configured to spread packets across multiple queues. Each queue is accessed by a different lcore in my application; there is therefore concurrent access to the controller, but not to any of the queues. We're binding the ports to the igb_uio driver. The symptoms I see are these: - All transmit out of a particular port stops - rte_eth_tx_burst() indicates that it is sending all of the packets that I give to it - rte_eth_stats_get() gives me stats indicating that no packets are being sent on the affected port. Also, no tx errors, and no pause frames sent or received (opackets = 0, obytes = 0, oerrors = 0, etc.) - All other ports continue to work normally - The affected port continues to receive packets without problems; only TX is affected - Resetting the port via rte_eth_dev_stop() and rte_eth_dev_start() restores things and packets can flow again - The problem is replicable on multiple devices, and doesn't follow one particular port I've tried calling rte_mbuf_sanity_check() on all packets before sending them. I've also instrumented my code to look for packets that have already been sent or freed, as well as cycles in chained packets being sent. I also put a lock around all accesses to rte_eth* calls to synchronize access to the NIC. Given some recent discussion here, I also tried changing the TX RS threshold from 0 to 32, 16, and 1. None of these strategies proved effective. Like I said at the top, I'm a little at a loss at this point. If you were dealing with this set of symptoms, how would you proceed? Thanks in advance. -- Matt Laswell infinite io, inc. laswell@infiniteio.com