From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 636E0A0524 for ; Fri, 31 Jan 2020 15:48:50 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 3D1381C0D7; Fri, 31 Jan 2020 15:48:50 +0100 (CET) Received: from mail-lj1-f175.google.com (mail-lj1-f175.google.com [209.85.208.175]) by dpdk.org (Postfix) with ESMTP id 2866E1C0CD for ; Fri, 31 Jan 2020 15:48:49 +0100 (CET) Received: by mail-lj1-f175.google.com with SMTP id w1so7388055ljh.5 for ; Fri, 31 Jan 2020 06:48:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=vNMtlCyqivYLx5GYRa/iNun2poxp80NlsjKsZZkADqA=; b=KkmqFNcsoVlr/PKNDFD8GSp3F2u+KcL/Pmr6MsRD6if/CmfBpRe+PJ1lJctrPBWBUe 53qFZvkPvl2C4L8Lr/qh4sqlG04/9BL7JutVYy3xUfShyMb5EVxTEqVzWYPfLIofLaX2 eyY6yWbFZ7kz/3yfqGeg/kaSNiZRbnznBXO30= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=vNMtlCyqivYLx5GYRa/iNun2poxp80NlsjKsZZkADqA=; b=TY92HnAUm4QPoQhs5lCFdE22KGjKHDai0VHPLIZ4dGhFb5IpUxzcbZTtYfSQUaENSu OT652sBlNIVDf3J6osaPsMtEx9kXQ78GUdxOXvLP8CEvOVQk3GCXMO1HotJAFgRxZydx hhViDBA+hPfHC6Ca5uOUYZlQ6ThbBRrK3GZ6It01xgUNgIrvBZpMn3+dNkXpOXO+r9N8 JdR5H4LEoWnxnw5pc2DveSfGCbdPMwyWnpCD28lYMUEZgEKcxZ+yB4/JlIl9jUjMsYDT fmLnuGKF2QstqWD0x6LjluGeYOtwxPmxMhQ9ac3VFQCvUQb+72kBgzmh6gFli9sTV01g w4Yw== X-Gm-Message-State: APjAAAWTYdbphHhqkIn76YcTPKWcQO49qBpu74f9snaud8CLOVsXrO+o tfmBaA2uNinNeewxoI8LaqE6g28UcpzZvpRiC/Rk2Q== X-Google-Smtp-Source: APXvYqwWn3XpN+uBbxvwUEFbGiK5aHi+dmwPNLaa0NUu8+L+6vYpxCSBTzAR1eFKEC1vgB/yHkbjKD3XIo78CvVgBUE= X-Received: by 2002:a2e:e12:: with SMTP id 18mr6229355ljo.123.1580482128620; Fri, 31 Jan 2020 06:48:48 -0800 (PST) MIME-Version: 1.0 References: <1827408.xz2uEaWSZ7@xps> <9DEEADBC57E43F4DA73B571777FECECA41EAB066@SHSMSX104.ccr.corp.intel.com> In-Reply-To: <9DEEADBC57E43F4DA73B571777FECECA41EAB066@SHSMSX104.ccr.corp.intel.com> From: Jeremy Plsek Date: Fri, 31 Jan 2020 09:48:12 -0500 Message-ID: To: "Chen, Zhaoyan" Cc: Lincoln Lavoie , Thomas Monjalon , "ci@dpdk.org" , "Tu, Lijuan" , "Xu, Qian Q" Content-Type: multipart/alternative; boundary="000000000000d9ebe2059d70abce" Subject: Re: [dpdk-ci] Intel performance test is failing X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" --000000000000d9ebe2059d70abce Content-Type: text/plain; charset="UTF-8" It's possible this is happening more frequently because of enabling overlayfs. We also changed how the baseline is retrieved because of the overlayfs system, but this issue was happening before this change. I think this started happening after a few months after upgrading to Ubuntu 18.04. Here are the recent results (login to view and download the artifacts): https://lab.dpdk.org/results/dashboard/patchsets/9344/#env-19 https://lab.dpdk.org/results/dashboard/patchsets/9343/#env-19 https://lab.dpdk.org/results/dashboard/patchsets/9340/#env-19 Here are some results that were "off" (see the "Run 1" tab): https://lab.dpdk.org/results/dashboard/patchsets/9346/#env-19 https://lab.dpdk.org/results/dashboard/patchsets/9331/#env-19 https://lab.dpdk.org/results/dashboard/patchsets/9321/#env-19 And here are some older tests where this occurred: https://lab.dpdk.org/results/dashboard/patchsets/8813/#env-19 https://lab.dpdk.org/results/dashboard/patchsets/8795/#env-19 I also found some tests where it happened on the 40g platform: https://lab.dpdk.org/results/dashboard/patchsets/9188/#env-18 https://lab.dpdk.org/results/dashboard/patchsets/8870/#env-18 (older, before enabling overlayfs): https://lab.dpdk.org/results/dashboard/patchsets/7665/#env-18 https://lab.dpdk.org/results/dashboard/patchsets/7653/#env-18 On Thu, Jan 30, 2020 at 10:17 PM Chen, Zhaoyan wrote: > Hi, Jeremy and Lincoln, > > > > Did you change anything on the Intel testbed? DTS code or testbed > environment? > > And could you send me the detailed test logs? We will do a quick check. > > > > Thanks > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Lincoln Lavoie > *Sent:* Friday, January 24, 2020 4:29 AM > *To:* Jeremy Plsek > *Cc:* Thomas Monjalon ; ci@dpdk.org; Tu, Lijuan < > lijuan.tu@intel.com>; Chen, Zhaoyan ; Xu, Qian Q < > qian.q.xu@intel.com> > *Subject:* Re: [dpdk-ci] Intel performance test is failing > > > > In the "off cases," is that specific to that hardware, or does it happen > on all hardware variants from time-to-time? > > > > On Thu, Jan 23, 2020 at 11:25 AM Jeremy Plsek wrote: > > On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon > wrote: > > > > Hi, > > > > The test iol-intel-Performance is failing on all patches > > for a couple of days. > > It seems to be due to a perf drop with a 10G device > > > > Can we disable this test until we understand what happens? > > > > We need also to understand how to avoid such failure, > > and how to monitor and disable it faster. > > > > Thanks > > > > > Hi Thomas, > > I've disabled it for now until someone from the Intel team can look > into it (cc'd). > > I've also seem problems where the results are completely off > (-15mpps), and I've been rerunning those when they happen. I'm hoping > those can be resolved too. > > -- > Jeremy Plsek > UNH InterOperability Laboratory > > > > > -- > > *Lincoln Lavoie* > > Senior Engineer, Broadband Technologies > > 21 Madbury Rd., Ste. 100, Durham, NH 03824 > > lylavoie@iol.unh.edu > > https://www.iol.unh.edu > > +1-603-674-2755 (m) > > > -- Jeremy Plsek UNH InterOperability Laboratory --000000000000d9ebe2059d70abce Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
It's possible this is happening more frequently b= ecause of enabling overlayfs.
We also changed how the baseline is= retrieved because of the overlayfs system, but this issue was happening be= fore this change.
I think this started happening after a few mont= hs after upgrading to Ubuntu 18.04.

Here are t= he recent results (login to view and download the artifacts):
Here are some results that were "off" (see the "= Run 1" tab):

And here are some older tests wh= ere this occurred:
https://lab.dpdk.org/results/dashboard/patchset= s/8813/#env-19

I also found some tests where it happ= ened on the 40g platform:
(older, before enabling overlayfs):
On Thu, Jan 30, 2020 at 10:17 PM Chen, Zhaoyan <<= a href=3D"mailto:zhaoyan.chen@intel.com">zhaoyan.chen@intel.com> wro= te:

Hi, Jeremy and Lincoln,

=C2=A0

Did you change anything on the Intel test= bed? DTS code or testbed environment?

And could you send me the detailed test l= ogs? We will do a quick check.

=C2=A0

Thanks

=C2=A0

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Lincoln Lavoie <lylavoie@iol.unh.edu>
Sent: Friday, January 24, 2020 4:29 AM
To: Jeremy Plsek <jplsek@iol.unh.edu>
Cc: Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Chen, Zhaoyan &l= t;zhaoyan.chen@= intel.com>; Xu, Qian Q <qian.q.xu@intel.com>
Subject: Re: [dpdk-ci] Intel performance test is failing

=C2=A0

In the "off case= s," is that specific to that hardware, or does it happen on all hardwa= re variants from time-to-time?

=C2=A0

On Thu, Jan 23, 2020 at 11:25 AM Jeremy Plsek <jplsek@iol.unh.edu> wrote:

On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon <= ;thomas@monjalon.n= et> wrote:
>
> Hi,
>
> The test iol-intel-Performance is failing on all patches
> for a couple of days.
> It seems to be due to a perf drop with a 10G device
>
> Can we disable this test until we understand what happens?
>
> We need also to understand how to avoid such failure,
> and how to monitor and disable it faster.
>
> Thanks
>
>
Hi Thomas,

I've disabled it for now until someone from the Intel team can look
into it (cc'd).

I've also seem problems where the results are completely off
(-15mpps), and I've been rerunning those when they happen. I'm hopi= ng
those can be resolved too.

--
Jeremy Plsek
UNH InterOperability Laboratory


=C2=A0

--

Lincoln Lavoie

Senior Engineer, Broadband Technologies

21 Madbury Rd., Ste. 100, Durham, NH 03824=

+1-603-674-2755 (m)



--
Jeremy Plsek
UNH InterOperability Laboratory
<= /div>
--000000000000d9ebe2059d70abce--