From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 561D3A0562 for ; Mon, 30 Mar 2020 09:17:46 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B37541C031; Mon, 30 Mar 2020 09:17:45 +0200 (CEST) Received: from mail-ed1-f66.google.com (mail-ed1-f66.google.com [209.85.208.66]) by dpdk.org (Postfix) with ESMTP id BA7C61C239 for ; Fri, 27 Mar 2020 19:25:35 +0100 (CET) Received: by mail-ed1-f66.google.com with SMTP id a43so12445184edf.6 for ; Fri, 27 Mar 2020 11:25:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=IUYFTEAOvJTIoK406rYnBw/tr7niaSGRvsG0ogo3Ikg=; b=QRa1yBmhhSpvSUT6DBJl0nbexq0hkF3+zxmWJ6UcUIZhp9Qo5s+dTyPmXUwsC6CWhA zzK7i3MGph/IE1CiQ/gu5KR6EdPaHBwvU8XTFaHuxx2WsAlWbiGC2DKgO2/Msip/AlLj Z5hWU+DAG5ntzwB+CHyk5o1gzmdGCjntxZ8M8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=IUYFTEAOvJTIoK406rYnBw/tr7niaSGRvsG0ogo3Ikg=; b=HSQ4DltYtvonqzb+NFoaR37tBOdlu7i0y2H1F9lnIRARBrYwIwYgFn8RZtKEJNYOIm kNI858on3hJJzm7CGZCO0EdWlTxBnlSLJxN18G535I1dNAymBZ3ofsDB+JTkvWsFJJn8 SuewJFViydfZF2+6LOQsmydGt1/V3MSGB68m4brHk4YzSKiWd/3s0VJf1EV0d1HR7cyx fgDtbu1TJkCEk4TeLCRByOP6MeNSp4bCHBbdsn56MfEUmVVmUCOkgC01FKFb1UkxR6yD Q19jhfhDA1sI+JhCvZlYb9I8wI3utdwWtyzzzvW3FEo9oLNMqWzCa8mi1isuXVfxZQEF /epg== X-Gm-Message-State: ANhLgQ2wFWOrsEQkOqPSyKgS8VMgfWmB/Hb0aNt/fHQbSSr7HfSzvH/e kWrXMQk7qd9MpsWsQQHrbMdrCzZ/riY6ZgY1ULh9tA== X-Google-Smtp-Source: ADFU+vtiC1+YZwvmH5CbXHENaONG4fZHQ9QgBEAOb9WQLMalmUfh+jeKLuGzWiG5maqSYp7GpBF4HwgxXCqdcOReHeU= X-Received: by 2002:a50:d4c2:: with SMTP id e2mr459314edj.136.1585333535206; Fri, 27 Mar 2020 11:25:35 -0700 (PDT) MIME-Version: 1.0 References: <2735222.2VHbPRQshP@xps> In-Reply-To: <2735222.2VHbPRQshP@xps> From: Lincoln Lavoie Date: Fri, 27 Mar 2020 14:25:32 -0400 Message-ID: To: Thomas Monjalon Cc: Hrvoje Habjanic , users@dpdk.org, galco@mellanox.com, asafp@mellanox.com, olgas@mellanox.com, ci@dpdk.org X-Mailman-Approved-At: Mon, 30 Mar 2020 09:17:44 +0200 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] [dpdk-ci] DPDK TX problems X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi Thomas, I've captured this as https://bugs.dpdk.org/show_bug.cgi?id=3D429, so we ca= n add this to the list of development items for the testing, etc. Cheers, Lincoln On Thu, Mar 26, 2020 at 4:54 PM Thomas Monjalon wrote= : > Thanks for the interesting feedback. > It seems we should test this performance use case in our labs. > > > 18/02/2020 09:36, Hrvoje Habjanic: > > On 08. 04. 2019. 11:52, Hrvoje Habjani=C4=87 wrote: > > > On 29/03/2019 08:24, Hrvoje Habjani=C4=87 wrote: > > >>> Hi. > > >>> > > >>> I did write an application using dpdk 17.11 (did try also with > 18.11), > > >>> and when doing some performance testing, i'm seeing very odd > behavior. > > >>> To verify that this is not because of my app, i did the same test > with > > >>> l2fwd example app, and i'm still confused by results. > > >>> > > >>> In short, i'm trying to push a lot of L2 packets through dpdk engin= e > - > > >>> packet processing is minimal. When testing, i'm starting with small > > >>> number of packets-per-second, and then gradually increase it to see > > >>> where is the limit. At some point, i do reach this limit - packets > start > > >>> to get dropped. And this is when stuff become weird. > > >>> > > >>> When i reach peek packet rate (at which packets start to get > dropped), i > > >>> would expect that reducing packet rate will remove packet drops. Bu= t, > > >>> this is not the case. For example, let's assume that peek packet > rate is > > >>> 3.5Mpps. At this point everything works ok. Increasing pps to > 4.0Mpps, > > >>> makes a lot of dropped packets. When reducing pps back to 3.5Mpps, > app > > >>> is still broken - packets are still dropped. > > >>> > > >>> At this point, i need to drastically reduce pps (1.4Mpps) to make > > >>> dropped packets go away. Also, app is unable to successfully forwar= d > > >>> anything beyond this 1.4M, despite the fact that in the beginning i= t > did > > >>> forward 3.5M! Only way to recover is to restart the app. > > >>> > > >>> Also, sometimes, the app just stops forwarding any packets - packet= s > are > > >>> received (as seen by counters), but app is unable to send anything > back. > > >>> > > >>> As i did mention, i'm seeing the same behavior with l2fwd example > app. I > > >>> did test dpdk 17.11 and also dpdk 18.11 - the results are the same. > > >>> > > >>> My test environment is HP DL380G8, with 82599ES 10Gig (ixgbe) cards= , > > >>> connected with Cisco nexus 9300 sw. On the other side is ixia test > > >>> appliance. Application is run in virtual machine (VM), using KVM > > >>> (openstack, with sriov enabled, and numa restrictions). I did check > that > > >>> VM is using only cpu's from NUMA node on which network card is > > >>> connected, so there is no cross-numa traffic. Openstack is Queens, > > >>> Ubuntu is Bionic release. Virtual machine is also using ubuntu bion= ic > > >>> as OS. > > >>> > > >>> I do not know how to debug this? Does someone else have the same > > >>> observations? > > >>> > > >>> Regards, > > >>> > > >>> H. > > >> There are additional findings. It seems that when i reach peak pps > > >> rate, application is not fast enough, and i can see rx missed errors > > >> on card statistics on the host. At the same time, tx side starts to > > >> show problems (tx burst starts to show it did not send all packets). > > >> Shortly after that, tx falls apart completely and top pps rate drops= . > > >> > > >> Since i did not disable pause frames, i can see on the switch "RX > > >> pause" frame counter is increasing. On the other hand, if i disable > > >> pause frames (on the nic of server), host driver (ixgbe) reports "TX > > >> unit hang" in dmesg, and issues card reset. Of course, after reset > > >> none of the dpdk apps in VM's on this host does not work. > > >> > > >> Is it possible that at time of congestion DPDK does not release mbuf= s > > >> back to the pool, and tx ring becomes "filled" with zombie packets > > >> (not send by card and also having ref counter as they are in use)? > > >> > > >> Is there a way to check mempool or tx ring for "left-owers"? Is is > > >> possible to somehow "flush" tx ring and/or mempool? > > >> > > >> H. > > > After few more test, things become even weirder - if i do not free > mbufs > > > which are not sent, but resend them again, i can "survive" > over-the-peek > > > event! But, then peek rate starts to drop gradually ... > > > > > > I would ask if someone can try this on their platform and report back= ? > I > > > would really like to know if this is problem with my deployment, or > > > there is something wrong with dpdk? > > > > > > Test should be simple - use l2fwd or l3fwd, and determine max pps. Th= en > > > drive pps 30%over max, and then return back and confirm that you can > > > still get max pps. > > > > > > Thanks in advance. > > > > > > H. > > > > > > > I did receive few mails from users facing this issue, asking how it was > > resolved. > > > > Unfortunately, there is no real fix. It seems that this issue is relate= d > > to card and hardware used. I'm still not sure which is more to blame, > > but the combination i had is definitely problematic. > > > > Anyhow, in the end, i did conclude that card driver have some issues > > when it is saturated with packets. My suspicion is that driver/software > > does not properly free packets, and then DPDK mempool becomes > > fragmented, and this causes performance drops. Restarting software > > releases pools, and restores proper functionality. > > > > After no luck with ixgbe, we migrated to Mellanox (4LX), and now there > > is no more of this permanent performance drop. With mlx, when limit is > > reached, reducing number of packets restores packet forwarding, and thi= s > > limit seems to be stable. > > > > Also, we moved to newer servers - DL380G10, and got significant > > performance increase. Also, we moved to newer switch (also cisco), with > > 25G ports, which reduced latency - almost by factor of 2! > > > > I did not try old ixgbe on newer server, but i did try Intel's XL710, > > and it is not as happy as Mellanox. It gives better PPS, but it is more > > unstable in terms of maximum bw (has similar issues as ixgbe). > > > > Regards, > > > > H. > > > > > --=20 *Lincoln Lavoie* Senior Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 lylavoie@iol.unh.edu https://www.iol.unh.edu +1-603-674-2755 (m)