Re: [dpdk-ci] Intel performance test is failing

["Chen, Zhaoyan" <zhaoyan.chen@intel.com>]

[-- Attachment #1: Type: text/plain, Size: 4570 bytes --]

Thanks Jeremy.

I quickly checked per-patches results and logs for the failures. That seems not always failed case.. like fluctuation.

For this case, could you help to check
- if BIOS’s turbo boost is off, C0/C1 is off, C6 is off
- if cores’ are isolated in kernel’s parameter
- if other tasks are scheduled on the testbed when running performance test

If all settings are good, we may consider it’s a regression or unstable performance issue. We will double check the performance by IXIA with latest DPDK master when we back to office (1 week later).

Thanks. Any problem, please let me know.


Regards,
Zhaoyan Chen

From: Jeremy Plsek <jplsek@iol.unh.edu>
Sent: Friday, January 31, 2020 10:48 PM
To: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Cc: Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>
Subject: Re: [dpdk-ci] Intel performance test is failing

It's possible this is happening more frequently because of enabling overlayfs.
We also changed how the baseline is retrieved because of the overlayfs system, but this issue was happening before this change.
I think this started happening after a few months after upgrading to Ubuntu 18.04.

Here are the recent results (login to view and download the artifacts):
https://lab.dpdk.org/results/dashboard/patchsets/9344/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/9343/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/9340/#env-19

Here are some results that were "off" (see the "Run 1" tab):
https://lab.dpdk.org/results/dashboard/patchsets/9346/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/9331/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/9321/#env-19

And here are some older tests where this occurred:
https://lab.dpdk.org/results/dashboard/patchsets/8813/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/8795/#env-19

I also found some tests where it happened on the 40g platform:
https://lab.dpdk.org/results/dashboard/patchsets/9188/#env-18
https://lab.dpdk.org/results/dashboard/patchsets/8870/#env-18
(older, before enabling overlayfs):
https://lab.dpdk.org/results/dashboard/patchsets/7665/#env-18
https://lab.dpdk.org/results/dashboard/patchsets/7653/#env-18


On Thu, Jan 30, 2020 at 10:17 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote:
Hi, Jeremy and Lincoln,

Did you change anything on the Intel testbed? DTS code or testbed environment?
And could you send me the detailed test logs? We will do a quick check.

Thanks


Regards,
Zhaoyan Chen

From: Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>
Sent: Friday, January 24, 2020 4:29 AM
To: Jeremy Plsek <jplsek@iol.unh.edu<mailto:jplsek@iol.unh.edu>>
Cc: Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>
Subject: Re: [dpdk-ci] Intel performance test is failing

In the "off cases," is that specific to that hardware, or does it happen on all hardware variants from time-to-time?

On Thu, Jan 23, 2020 at 11:25 AM Jeremy Plsek <jplsek@iol.unh.edu<mailto:jplsek@iol.unh.edu>> wrote:
On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>> wrote:
>
> Hi,
>
> The test iol-intel-Performance is failing on all patches
> for a couple of days.
> It seems to be due to a perf drop with a 10G device
>
> Can we disable this test until we understand what happens?
>
> We need also to understand how to avoid such failure,
> and how to monitor and disable it faster.
>
> Thanks
>
>
Hi Thomas,

I've disabled it for now until someone from the Intel team can look
into it (cc'd).

I've also seem problems where the results are completely off
(-15mpps), and I've been rerunning those when they happen. I'm hoping
those can be resolved too.

--
Jeremy Plsek
UNH InterOperability Laboratory


--
Lincoln Lavoie
Senior Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>
https://www.iol.unh.edu
+1-603-674-2755 (m)
[http://homeautomation.lavoieholdings.com/_/rsrc/1390068882701/unh-iol-logo.png]<https://www.iol.unh.edu/>


--
Jeremy Plsek
UNH InterOperability Laboratory

[-- Attachment #2: Type: text/html, Size: 17542 bytes --]