DPDK CI discussions
 help / color / Atom feed
* [dpdk-ci] Intel performance test is failing
@ 2020-01-23 15:58 Thomas Monjalon
  2020-01-23 16:25 ` Jeremy Plsek
  0 siblings, 1 reply; 6+ messages in thread
From: Thomas Monjalon @ 2020-01-23 15:58 UTC (permalink / raw)
  To: jplsek; +Cc: ci

Hi,

The test iol-intel-Performance is failing on all patches
for a couple of days.
It seems to be due to a perf drop with a 10G device

Can we disable this test until we understand what happens?

We need also to understand how to avoid such failure,
and how to monitor and disable it faster.

Thanks



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-ci] Intel performance test is failing
  2020-01-23 15:58 [dpdk-ci] Intel performance test is failing Thomas Monjalon
@ 2020-01-23 16:25 ` Jeremy Plsek
  2020-01-23 20:28   ` Lincoln Lavoie
  0 siblings, 1 reply; 6+ messages in thread
From: Jeremy Plsek @ 2020-01-23 16:25 UTC (permalink / raw)
  To: Thomas Monjalon; +Cc: ci, Tu, Lijuan, Chen, Zhaoyan, Xu, Qian Q

On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> Hi,
>
> The test iol-intel-Performance is failing on all patches
> for a couple of days.
> It seems to be due to a perf drop with a 10G device
>
> Can we disable this test until we understand what happens?
>
> We need also to understand how to avoid such failure,
> and how to monitor and disable it faster.
>
> Thanks
>
>
Hi Thomas,

I've disabled it for now until someone from the Intel team can look
into it (cc'd).

I've also seem problems where the results are completely off
(-15mpps), and I've been rerunning those when they happen. I'm hoping
those can be resolved too.

--
Jeremy Plsek
UNH InterOperability Laboratory

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-ci] Intel performance test is failing
  2020-01-23 16:25 ` Jeremy Plsek
@ 2020-01-23 20:28   ` Lincoln Lavoie
  2020-01-31  3:17     ` Chen, Zhaoyan
  0 siblings, 1 reply; 6+ messages in thread
From: Lincoln Lavoie @ 2020-01-23 20:28 UTC (permalink / raw)
  To: Jeremy Plsek; +Cc: Thomas Monjalon, ci, Tu, Lijuan, Chen, Zhaoyan, Xu, Qian Q

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

In the "off cases," is that specific to that hardware, or does it happen on
all hardware variants from time-to-time?

On Thu, Jan 23, 2020 at 11:25 AM Jeremy Plsek <jplsek@iol.unh.edu> wrote:

> On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon <thomas@monjalon.net>
> wrote:
> >
> > Hi,
> >
> > The test iol-intel-Performance is failing on all patches
> > for a couple of days.
> > It seems to be due to a perf drop with a 10G device
> >
> > Can we disable this test until we understand what happens?
> >
> > We need also to understand how to avoid such failure,
> > and how to monitor and disable it faster.
> >
> > Thanks
> >
> >
> Hi Thomas,
>
> I've disabled it for now until someone from the Intel team can look
> into it (cc'd).
>
> I've also seem problems where the results are completely off
> (-15mpps), and I've been rerunning those when they happen. I'm hoping
> those can be resolved too.
>
> --
> Jeremy Plsek
> UNH InterOperability Laboratory
>


-- 
*Lincoln Lavoie*
Senior Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu/>

[-- Attachment #2: Type: text/html, Size: 2304 bytes --]

<div dir="ltr"><div class="gmail_default" style="font-size:small">In the &quot;off cases,&quot; is that specific to that hardware, or does it happen on all hardware variants from time-to-time?</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 23, 2020 at 11:25 AM Jeremy Plsek &lt;<a href="mailto:jplsek@iol.unh.edu">jplsek@iol.unh.edu</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon &lt;<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>&gt; wrote:<br>
&gt;<br>
&gt; Hi,<br>
&gt;<br>
&gt; The test iol-intel-Performance is failing on all patches<br>
&gt; for a couple of days.<br>
&gt; It seems to be due to a perf drop with a 10G device<br>
&gt;<br>
&gt; Can we disable this test until we understand what happens?<br>
&gt;<br>
&gt; We need also to understand how to avoid such failure,<br>
&gt; and how to monitor and disable it faster.<br>
&gt;<br>
&gt; Thanks<br>
&gt;<br>
&gt;<br>
Hi Thomas,<br>
<br>
I&#39;ve disabled it for now until someone from the Intel team can look<br>
into it (cc&#39;d).<br>
<br>
I&#39;ve also seem problems where the results are completely off<br>
(-15mpps), and I&#39;ve been rerunning those when they happen. I&#39;m hoping<br>
those can be resolved too.<br>
<br>
--<br>
Jeremy Plsek<br>
UNH InterOperability Laboratory<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><b>Lincoln Lavoie</b><br></div><div>Senior Engineer, Broadband Technologies</div><div>21 Madbury Rd., Ste. 100, Durham, NH 03824</div><div><a href="mailto:lylavoie@iol.unh.edu" target="_blank">lylavoie@iol.unh.edu</a></div><div><a href="https://www.iol.unh.edu" target="_blank">https://www.iol.unh.edu</a></div><div>+1-603-674-2755 (m)<br></div><div><a href="https://www.iol.unh.edu/" target="_blank"><img src="http://homeautomation.lavoieholdings.com/_/rsrc/1390068882701/unh-iol-logo.png"></a></div></div></div></div></div></div></div></div></div></div></div></div>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-ci] Intel performance test is failing
  2020-01-23 20:28   ` Lincoln Lavoie
@ 2020-01-31  3:17     ` Chen, Zhaoyan
  2020-01-31 14:48       ` Jeremy Plsek
  0 siblings, 1 reply; 6+ messages in thread
From: Chen, Zhaoyan @ 2020-01-31  3:17 UTC (permalink / raw)
  To: Lincoln Lavoie, Jeremy Plsek
  Cc: Thomas Monjalon, ci, Tu, Lijuan, Xu, Qian Q, Chen, Zhaoyan

[-- Attachment #1: Type: text/plain, Size: 1869 bytes --]

Hi, Jeremy and Lincoln,

Did you change anything on the Intel testbed? DTS code or testbed environment?
And could you send me the detailed test logs? We will do a quick check.

Thanks


Regards,
Zhaoyan Chen

From: Lincoln Lavoie <lylavoie@iol.unh.edu>
Sent: Friday, January 24, 2020 4:29 AM
To: Jeremy Plsek <jplsek@iol.unh.edu>
Cc: Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>
Subject: Re: [dpdk-ci] Intel performance test is failing

In the "off cases," is that specific to that hardware, or does it happen on all hardware variants from time-to-time?

On Thu, Jan 23, 2020 at 11:25 AM Jeremy Plsek <jplsek@iol.unh.edu<mailto:jplsek@iol.unh.edu>> wrote:
On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>> wrote:
>
> Hi,
>
> The test iol-intel-Performance is failing on all patches
> for a couple of days.
> It seems to be due to a perf drop with a 10G device
>
> Can we disable this test until we understand what happens?
>
> We need also to understand how to avoid such failure,
> and how to monitor and disable it faster.
>
> Thanks
>
>
Hi Thomas,

I've disabled it for now until someone from the Intel team can look
into it (cc'd).

I've also seem problems where the results are completely off
(-15mpps), and I've been rerunning those when they happen. I'm hoping
those can be resolved too.

--
Jeremy Plsek
UNH InterOperability Laboratory


--
Lincoln Lavoie
Senior Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>
https://www.iol.unh.edu
+1-603-674-2755 (m)
[http://homeautomation.lavoieholdings.com/_/rsrc/1390068882701/unh-iol-logo.png]<https://www.iol.unh.edu/>

[-- Attachment #2: Type: text/html, Size: 7413 bytes --]

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
	{font-family:宋体;
	panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:"Calibri Light";
	panose-1:2 15 3 2 2 2 4 3 2 4;}
@font-face
	{font-family:"\@宋体";
	panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
	{mso-style-name:msonormal;
	mso-margin-top-alt:auto;
	margin-right:0in;
	mso-margin-bottom-alt:auto;
	margin-left:0in;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
span.EmailStyle18
	{mso-style-type:personal-reply;
	font-family:"Calibri Light",sans-serif;
	color:#1F497D;
	font-weight:normal;
	font-style:normal;
	text-decoration:none none;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-family:"Calibri",sans-serif;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.25in 1.0in 1.25in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">Hi, Jeremy and Lincoln,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">Did you change anything on the Intel testbed? DTS code or testbed environment?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">And could you send me the detailed test logs? We will do a quick check.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">Thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D"><o:p>&nbsp;</o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">Regards,<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">Zhaoyan Chen<o:p></o:p></span></i></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Lincoln Lavoie &lt;lylavoie@iol.unh.edu&gt; <br>
<b>Sent:</b> Friday, January 24, 2020 4:29 AM<br>
<b>To:</b> Jeremy Plsek &lt;jplsek@iol.unh.edu&gt;<br>
<b>Cc:</b> Thomas Monjalon &lt;thomas@monjalon.net&gt;; ci@dpdk.org; Tu, Lijuan &lt;lijuan.tu@intel.com&gt;; Chen, Zhaoyan &lt;zhaoyan.chen@intel.com&gt;; Xu, Qian Q &lt;qian.q.xu@intel.com&gt;<br>
<b>Subject:</b> Re: [dpdk-ci] Intel performance test is failing<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">In the &quot;off cases,&quot; is that specific to that hardware, or does it happen on all hardware variants from time-to-time?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Jan 23, 2020 at 11:25 AM Jeremy Plsek &lt;<a href="mailto:jplsek@iol.unh.edu">jplsek@iol.unh.edu</a>&gt; wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon &lt;<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>&gt; wrote:<br>
&gt;<br>
&gt; Hi,<br>
&gt;<br>
&gt; The test iol-intel-Performance is failing on all patches<br>
&gt; for a couple of days.<br>
&gt; It seems to be due to a perf drop with a 10G device<br>
&gt;<br>
&gt; Can we disable this test until we understand what happens?<br>
&gt;<br>
&gt; We need also to understand how to avoid such failure,<br>
&gt; and how to monitor and disable it faster.<br>
&gt;<br>
&gt; Thanks<br>
&gt;<br>
&gt;<br>
Hi Thomas,<br>
<br>
I've disabled it for now until someone from the Intel team can look<br>
into it (cc'd).<br>
<br>
I've also seem problems where the results are completely off<br>
(-15mpps), and I've been rerunning those when they happen. I'm hoping<br>
those can be resolved too.<br>
<br>
--<br>
Jeremy Plsek<br>
UNH InterOperability Laboratory<o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><b>Lincoln Lavoie</b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Senior Engineer, Broadband Technologies<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">21 Madbury Rd., Ste. 100, Durham, NH 03824<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="mailto:lylavoie@iol.unh.edu" target="_blank">lylavoie@iol.unh.edu</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://www.iol.unh.edu" target="_blank">https://www.iol.unh.edu</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">&#43;1-603-674-2755 (m)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://www.iol.unh.edu/" target="_blank"><span style="text-decoration:none"><img border="0" id="_x0000_i1025" src="http://homeautomation.lavoieholdings.com/_/rsrc/1390068882701/unh-iol-logo.png"></span></a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-ci] Intel performance test is failing
  2020-01-31  3:17     ` Chen, Zhaoyan
@ 2020-01-31 14:48       ` Jeremy Plsek
  2020-02-03  3:57         ` Chen, Zhaoyan
  0 siblings, 1 reply; 6+ messages in thread
From: Jeremy Plsek @ 2020-01-31 14:48 UTC (permalink / raw)
  To: Chen, Zhaoyan; +Cc: Lincoln Lavoie, Thomas Monjalon, ci, Tu, Lijuan, Xu, Qian Q

[-- Attachment #1: Type: text/plain, Size: 3285 bytes --]

It's possible this is happening more frequently because of enabling
overlayfs.
We also changed how the baseline is retrieved because of the overlayfs
system, but this issue was happening before this change.
I think this started happening after a few months after upgrading to Ubuntu
18.04.

Here are the recent results (login to view and download the artifacts):
https://lab.dpdk.org/results/dashboard/patchsets/9344/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/9343/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/9340/#env-19

Here are some results that were "off" (see the "Run 1" tab):
https://lab.dpdk.org/results/dashboard/patchsets/9346/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/9331/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/9321/#env-19

And here are some older tests where this occurred:
https://lab.dpdk.org/results/dashboard/patchsets/8813/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/8795/#env-19

I also found some tests where it happened on the 40g platform:
https://lab.dpdk.org/results/dashboard/patchsets/9188/#env-18
https://lab.dpdk.org/results/dashboard/patchsets/8870/#env-18
(older, before enabling overlayfs):
https://lab.dpdk.org/results/dashboard/patchsets/7665/#env-18
https://lab.dpdk.org/results/dashboard/patchsets/7653/#env-18


On Thu, Jan 30, 2020 at 10:17 PM Chen, Zhaoyan <zhaoyan.chen@intel.com>
wrote:

> Hi, Jeremy and Lincoln,
>
>
>
> Did you change anything on the Intel testbed? DTS code or testbed
> environment?
>
> And could you send me the detailed test logs? We will do a quick check.
>
>
>
> Thanks
>
>
>
>
>
> *Regards,*
>
> *Zhaoyan Chen*
>
>
>
> *From:* Lincoln Lavoie <lylavoie@iol.unh.edu>
> *Sent:* Friday, January 24, 2020 4:29 AM
> *To:* Jeremy Plsek <jplsek@iol.unh.edu>
> *Cc:* Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <
> lijuan.tu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com>; Xu, Qian Q <
> qian.q.xu@intel.com>
> *Subject:* Re: [dpdk-ci] Intel performance test is failing
>
>
>
> In the "off cases," is that specific to that hardware, or does it happen
> on all hardware variants from time-to-time?
>
>
>
> On Thu, Jan 23, 2020 at 11:25 AM Jeremy Plsek <jplsek@iol.unh.edu> wrote:
>
> On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon <thomas@monjalon.net>
> wrote:
> >
> > Hi,
> >
> > The test iol-intel-Performance is failing on all patches
> > for a couple of days.
> > It seems to be due to a perf drop with a 10G device
> >
> > Can we disable this test until we understand what happens?
> >
> > We need also to understand how to avoid such failure,
> > and how to monitor and disable it faster.
> >
> > Thanks
> >
> >
> Hi Thomas,
>
> I've disabled it for now until someone from the Intel team can look
> into it (cc'd).
>
> I've also seem problems where the results are completely off
> (-15mpps), and I've been rerunning those when they happen. I'm hoping
> those can be resolved too.
>
> --
> Jeremy Plsek
> UNH InterOperability Laboratory
>
>
>
>
> --
>
> *Lincoln Lavoie*
>
> Senior Engineer, Broadband Technologies
>
> 21 Madbury Rd., Ste. 100, Durham, NH 03824
>
> lylavoie@iol.unh.edu
>
> https://www.iol.unh.edu
>
> +1-603-674-2755 (m)
>
> <https://www.iol.unh.edu/>
>


-- 
Jeremy Plsek
UNH InterOperability Laboratory

[-- Attachment #2: Type: text/html, Size: 9225 bytes --]

<div dir="ltr"><div>It&#39;s possible this is happening more frequently because of enabling overlayfs.</div><div>We also changed how the baseline is retrieved because of the overlayfs system, but this issue was happening before this change.</div><div>I think this started happening after a few months after upgrading to Ubuntu 18.04.<br></div><div><br></div><div>Here are the recent results (login to view and download the artifacts):</div><div><a href="https://lab.dpdk.org/results/dashboard/patchsets/9344/#env-19">https://lab.dpdk.org/results/dashboard/patchsets/9344/#env-19</a></div><div><a href="https://lab.dpdk.org/results/dashboard/patchsets/9343/#env-19">https://lab.dpdk.org/results/dashboard/patchsets/9343/#env-19</a></div><div><a href="https://lab.dpdk.org/results/dashboard/patchsets/9340/#env-19">https://lab.dpdk.org/results/dashboard/patchsets/9340/#env-19</a></div><div><br></div><div>Here are some results that were &quot;off&quot; (see the &quot;Run 1&quot; tab):</div><div><a href="https://lab.dpdk.org/results/dashboard/patchsets/9346/#env-19">https://lab.dpdk.org/results/dashboard/patchsets/9346/#env-19</a></div><div><a href="https://lab.dpdk.org/results/dashboard/patchsets/9331/#env-19">https://lab.dpdk.org/results/dashboard/patchsets/9331/#env-19</a></div><div><a href="https://lab.dpdk.org/results/dashboard/patchsets/9321/#env-19">https://lab.dpdk.org/results/dashboard/patchsets/9321/#env-19</a></div><div><br></div><div>And here are some older tests where this occurred:</div><div><a href="https://lab.dpdk.org/results/dashboard/patchsets/8813/#env-19">https://lab.dpdk.org/results/dashboard/patchsets/8813/#env-19</a><div><a href="https://lab.dpdk.org/results/dashboard/patchsets/8795/#env-19">https://lab.dpdk.org/results/dashboard/patchsets/8795/#env-19</a></div><div><br></div><div>I also found some tests where it happened on the 40g platform:</div><div><a href="https://lab.dpdk.org/results/dashboard/patchsets/9188/#env-18">https://lab.dpdk.org/results/dashboard/patchsets/9188/#env-18</a></div><div><a href="https://lab.dpdk.org/results/dashboard/patchsets/8870/#env-18">https://lab.dpdk.org/results/dashboard/patchsets/8870/#env-18</a></div><div>(older, before enabling overlayfs):</div><div><a href="https://lab.dpdk.org/results/dashboard/patchsets/7665/#env-18">https://lab.dpdk.org/results/dashboard/patchsets/7665/#env-18</a></div><div><a href="https://lab.dpdk.org/results/dashboard/patchsets/7653/#env-18">https://lab.dpdk.org/results/dashboard/patchsets/7653/#env-18</a></div></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 30, 2020 at 10:17 PM Chen, Zhaoyan &lt;<a href="mailto:zhaoyan.chen@intel.com">zhaoyan.chen@intel.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">





<div lang="EN-US">
<div class="gmail-m_-5920278011901207711WordSection1">
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:rgb(31,73,125)">Hi, Jeremy and Lincoln,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:rgb(31,73,125)">Did you change anything on the Intel testbed? DTS code or testbed environment?
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:rgb(31,73,125)">And could you send me the detailed test logs? We will do a quick check.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:rgb(31,73,125)">Thanks<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><i><span style="font-size:10pt;font-family:&quot;Calibri Light&quot;,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></i></p>
<p class="MsoNormal"><i><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:rgb(31,73,125)">Regards,<u></u><u></u></span></i></p>
<p class="MsoNormal"><i><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:rgb(31,73,125)">Zhaoyan Chen<u></u><u></u></span></i></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div style="border-color:currentcolor currentcolor currentcolor blue;border-style:none none none solid;border-width:medium medium medium 1.5pt;padding:0in 0in 0in 4pt">
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> Lincoln Lavoie &lt;<a href="mailto:lylavoie@iol.unh.edu" target="_blank">lylavoie@iol.unh.edu</a>&gt; <br>
<b>Sent:</b> Friday, January 24, 2020 4:29 AM<br>
<b>To:</b> Jeremy Plsek &lt;<a href="mailto:jplsek@iol.unh.edu" target="_blank">jplsek@iol.unh.edu</a>&gt;<br>
<b>Cc:</b> Thomas Monjalon &lt;<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>&gt;; <a href="mailto:ci@dpdk.org" target="_blank">ci@dpdk.org</a>; Tu, Lijuan &lt;<a href="mailto:lijuan.tu@intel.com" target="_blank">lijuan.tu@intel.com</a>&gt;; Chen, Zhaoyan &lt;<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>&gt;; Xu, Qian Q &lt;<a href="mailto:qian.q.xu@intel.com" target="_blank">qian.q.xu@intel.com</a>&gt;<br>
<b>Subject:</b> Re: [dpdk-ci] Intel performance test is failing<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12pt">In the &quot;off cases,&quot; is that specific to that hardware, or does it happen on all hardware variants from time-to-time?<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Thu, Jan 23, 2020 at 11:25 AM Jeremy Plsek &lt;<a href="mailto:jplsek@iol.unh.edu" target="_blank">jplsek@iol.unh.edu</a>&gt; wrote:<u></u><u></u></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon &lt;<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>&gt; wrote:<br>
&gt;<br>
&gt; Hi,<br>
&gt;<br>
&gt; The test iol-intel-Performance is failing on all patches<br>
&gt; for a couple of days.<br>
&gt; It seems to be due to a perf drop with a 10G device<br>
&gt;<br>
&gt; Can we disable this test until we understand what happens?<br>
&gt;<br>
&gt; We need also to understand how to avoid such failure,<br>
&gt; and how to monitor and disable it faster.<br>
&gt;<br>
&gt; Thanks<br>
&gt;<br>
&gt;<br>
Hi Thomas,<br>
<br>
I&#39;ve disabled it for now until someone from the Intel team can look<br>
into it (cc&#39;d).<br>
<br>
I&#39;ve also seem problems where the results are completely off<br>
(-15mpps), and I&#39;ve been rerunning those when they happen. I&#39;m hoping<br>
those can be resolved too.<br>
<br>
--<br>
Jeremy Plsek<br>
UNH InterOperability Laboratory<u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><b>Lincoln Lavoie</b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Senior Engineer, Broadband Technologies<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">21 Madbury Rd., Ste. 100, Durham, NH 03824<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="mailto:lylavoie@iol.unh.edu" target="_blank">lylavoie@iol.unh.edu</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="https://www.iol.unh.edu" target="_blank">https://www.iol.unh.edu</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">+1-603-674-2755 (m)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="https://www.iol.unh.edu/" target="_blank"><span style="text-decoration:none"><img id="gmail-m_-5920278011901207711_x0000_i1025" src="http://homeautomation.lavoieholdings.com/_/rsrc/1390068882701/unh-iol-logo.png" border="0"></span></a><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>

</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div></div>Jeremy Plsek<br></div><div>UNH InterOperability Laboratory<br></div></div></div></div></div></div></div></div></div></div>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-ci] Intel performance test is failing
  2020-01-31 14:48       ` Jeremy Plsek
@ 2020-02-03  3:57         ` Chen, Zhaoyan
  0 siblings, 0 replies; 6+ messages in thread
From: Chen, Zhaoyan @ 2020-02-03  3:57 UTC (permalink / raw)
  To: Jeremy Plsek
  Cc: Lincoln Lavoie, Thomas Monjalon, ci, Tu, Lijuan, Xu, Qian Q,
	Chen, Zhaoyan

[-- Attachment #1: Type: text/plain, Size: 4570 bytes --]

Thanks Jeremy.

I quickly checked per-patches results and logs for the failures. That seems not always failed case.. like fluctuation.

For this case, could you help to check
- if BIOS’s turbo boost is off, C0/C1 is off, C6 is off
- if cores’ are isolated in kernel’s parameter
- if other tasks are scheduled on the testbed when running performance test

If all settings are good, we may consider it’s a regression or unstable performance issue. We will double check the performance by IXIA with latest DPDK master when we back to office (1 week later).

Thanks. Any problem, please let me know.


Regards,
Zhaoyan Chen

From: Jeremy Plsek <jplsek@iol.unh.edu>
Sent: Friday, January 31, 2020 10:48 PM
To: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Cc: Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>
Subject: Re: [dpdk-ci] Intel performance test is failing

It's possible this is happening more frequently because of enabling overlayfs.
We also changed how the baseline is retrieved because of the overlayfs system, but this issue was happening before this change.
I think this started happening after a few months after upgrading to Ubuntu 18.04.

Here are the recent results (login to view and download the artifacts):
https://lab.dpdk.org/results/dashboard/patchsets/9344/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/9343/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/9340/#env-19

Here are some results that were "off" (see the "Run 1" tab):
https://lab.dpdk.org/results/dashboard/patchsets/9346/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/9331/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/9321/#env-19

And here are some older tests where this occurred:
https://lab.dpdk.org/results/dashboard/patchsets/8813/#env-19
https://lab.dpdk.org/results/dashboard/patchsets/8795/#env-19

I also found some tests where it happened on the 40g platform:
https://lab.dpdk.org/results/dashboard/patchsets/9188/#env-18
https://lab.dpdk.org/results/dashboard/patchsets/8870/#env-18
(older, before enabling overlayfs):
https://lab.dpdk.org/results/dashboard/patchsets/7665/#env-18
https://lab.dpdk.org/results/dashboard/patchsets/7653/#env-18


On Thu, Jan 30, 2020 at 10:17 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote:
Hi, Jeremy and Lincoln,

Did you change anything on the Intel testbed? DTS code or testbed environment?
And could you send me the detailed test logs? We will do a quick check.

Thanks


Regards,
Zhaoyan Chen

From: Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>
Sent: Friday, January 24, 2020 4:29 AM
To: Jeremy Plsek <jplsek@iol.unh.edu<mailto:jplsek@iol.unh.edu>>
Cc: Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>
Subject: Re: [dpdk-ci] Intel performance test is failing

In the "off cases," is that specific to that hardware, or does it happen on all hardware variants from time-to-time?

On Thu, Jan 23, 2020 at 11:25 AM Jeremy Plsek <jplsek@iol.unh.edu<mailto:jplsek@iol.unh.edu>> wrote:
On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>> wrote:
>
> Hi,
>
> The test iol-intel-Performance is failing on all patches
> for a couple of days.
> It seems to be due to a perf drop with a 10G device
>
> Can we disable this test until we understand what happens?
>
> We need also to understand how to avoid such failure,
> and how to monitor and disable it faster.
>
> Thanks
>
>
Hi Thomas,

I've disabled it for now until someone from the Intel team can look
into it (cc'd).

I've also seem problems where the results are completely off
(-15mpps), and I've been rerunning those when they happen. I'm hoping
those can be resolved too.

--
Jeremy Plsek
UNH InterOperability Laboratory


--
Lincoln Lavoie
Senior Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>
https://www.iol.unh.edu
+1-603-674-2755 (m)
[http://homeautomation.lavoieholdings.com/_/rsrc/1390068882701/unh-iol-logo.png]<https://www.iol.unh.edu/>


--
Jeremy Plsek
UNH InterOperability Laboratory

[-- Attachment #2: Type: text/html, Size: 17542 bytes --]

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
	{font-family:宋体;
	panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:"Calibri Light";
	panose-1:2 15 3 2 2 2 4 3 2 4;}
@font-face
	{font-family:"\@宋体";
	panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
	{mso-style-name:msonormal;
	mso-margin-top-alt:auto;
	margin-right:0in;
	mso-margin-bottom-alt:auto;
	margin-left:0in;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
span.EmailStyle18
	{mso-style-type:personal-reply;
	font-family:"Calibri Light",sans-serif;
	color:#1F497D;
	font-weight:normal;
	font-style:normal;
	text-decoration:none none;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-family:"Calibri",sans-serif;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.25in 1.0in 1.25in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">Thanks Jeremy.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">I quickly checked per-patches results and logs for the failures. That seems not always failed case.. like fluctuation.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">For this case, could you help to check<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">- if BIOS’s turbo boost is off, C0/C1 is off, C6 is off<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">- if cores’ are isolated in kernel’s parameter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">- if other tasks are scheduled on the testbed when running performance test<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">If all settings are good, we may consider it’s a regression or unstable performance issue. We will double check the performance by IXIA with latest DPDK master when we back
 to office (1 week later). &nbsp;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">Thanks. Any problem, please let me know.<o:p></o:p></span></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D"><o:p>&nbsp;</o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D"><o:p>&nbsp;</o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">Regards,<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">Zhaoyan Chen<o:p></o:p></span></i></p>
<p class="MsoNormal"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Jeremy Plsek &lt;jplsek@iol.unh.edu&gt; <br>
<b>Sent:</b> Friday, January 31, 2020 10:48 PM<br>
<b>To:</b> Chen, Zhaoyan &lt;zhaoyan.chen@intel.com&gt;<br>
<b>Cc:</b> Lincoln Lavoie &lt;lylavoie@iol.unh.edu&gt;; Thomas Monjalon &lt;thomas@monjalon.net&gt;; ci@dpdk.org; Tu, Lijuan &lt;lijuan.tu@intel.com&gt;; Xu, Qian Q &lt;qian.q.xu@intel.com&gt;<br>
<b>Subject:</b> Re: [dpdk-ci] Intel performance test is failing<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<div>
<div>
<p class="MsoNormal">It's possible this is happening more frequently because of enabling overlayfs.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">We also changed how the baseline is retrieved because of the overlayfs system, but this issue was happening before this change.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I think this started happening after a few months after upgrading to Ubuntu 18.04.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">Here are the recent results (login to view and download the artifacts):<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://lab.dpdk.org/results/dashboard/patchsets/9344/#env-19">https://lab.dpdk.org/results/dashboard/patchsets/9344/#env-19</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://lab.dpdk.org/results/dashboard/patchsets/9343/#env-19">https://lab.dpdk.org/results/dashboard/patchsets/9343/#env-19</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://lab.dpdk.org/results/dashboard/patchsets/9340/#env-19">https://lab.dpdk.org/results/dashboard/patchsets/9340/#env-19</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">Here are some results that were &quot;off&quot; (see the &quot;Run 1&quot; tab):<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://lab.dpdk.org/results/dashboard/patchsets/9346/#env-19">https://lab.dpdk.org/results/dashboard/patchsets/9346/#env-19</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://lab.dpdk.org/results/dashboard/patchsets/9331/#env-19">https://lab.dpdk.org/results/dashboard/patchsets/9331/#env-19</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://lab.dpdk.org/results/dashboard/patchsets/9321/#env-19">https://lab.dpdk.org/results/dashboard/patchsets/9321/#env-19</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">And here are some older tests where this occurred:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://lab.dpdk.org/results/dashboard/patchsets/8813/#env-19">https://lab.dpdk.org/results/dashboard/patchsets/8813/#env-19</a><o:p></o:p></p>
<div>
<p class="MsoNormal"><a href="https://lab.dpdk.org/results/dashboard/patchsets/8795/#env-19">https://lab.dpdk.org/results/dashboard/patchsets/8795/#env-19</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">I also found some tests where it happened on the 40g platform:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://lab.dpdk.org/results/dashboard/patchsets/9188/#env-18">https://lab.dpdk.org/results/dashboard/patchsets/9188/#env-18</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://lab.dpdk.org/results/dashboard/patchsets/8870/#env-18">https://lab.dpdk.org/results/dashboard/patchsets/8870/#env-18</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">(older, before enabling overlayfs):<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://lab.dpdk.org/results/dashboard/patchsets/7665/#env-18">https://lab.dpdk.org/results/dashboard/patchsets/7665/#env-18</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://lab.dpdk.org/results/dashboard/patchsets/7653/#env-18">https://lab.dpdk.org/results/dashboard/patchsets/7653/#env-18</a><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Jan 30, 2020 at 10:17 PM Chen, Zhaoyan &lt;<a href="mailto:zhaoyan.chen@intel.com">zhaoyan.chen@intel.com</a>&gt; wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">Hi, Jeremy and Lincoln,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">&nbsp;</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">Did you change anything on the Intel testbed? DTS code or testbed environment?
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">And could you send me the detailed test logs? We will do a quick check.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">&nbsp;</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">Thanks</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">&nbsp;</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-size:10.0pt;font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">&nbsp;</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">Regards,</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">Zhaoyan Chen</span></i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:&quot;Calibri Light&quot;,sans-serif;color:#1F497D">&nbsp;</span><o:p></o:p></p>
<div style="border:none;border-left:solid windowtext 1.5pt;padding:0in 0in 0in 4.0pt;border-color:currentcolor currentcolor currentcolor blue">
<div>
<div style="border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0in 0in 0in;border-color:currentcolor currentcolor">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b> Lincoln Lavoie &lt;<a href="mailto:lylavoie@iol.unh.edu" target="_blank">lylavoie@iol.unh.edu</a>&gt;
<br>
<b>Sent:</b> Friday, January 24, 2020 4:29 AM<br>
<b>To:</b> Jeremy Plsek &lt;<a href="mailto:jplsek@iol.unh.edu" target="_blank">jplsek@iol.unh.edu</a>&gt;<br>
<b>Cc:</b> Thomas Monjalon &lt;<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>&gt;;
<a href="mailto:ci@dpdk.org" target="_blank">ci@dpdk.org</a>; Tu, Lijuan &lt;<a href="mailto:lijuan.tu@intel.com" target="_blank">lijuan.tu@intel.com</a>&gt;; Chen, Zhaoyan &lt;<a href="mailto:zhaoyan.chen@intel.com" target="_blank">zhaoyan.chen@intel.com</a>&gt;; Xu,
 Qian Q &lt;<a href="mailto:qian.q.xu@intel.com" target="_blank">qian.q.xu@intel.com</a>&gt;<br>
<b>Subject:</b> Re: [dpdk-ci] Intel performance test is failing<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:12.0pt">In the &quot;off cases,&quot; is that specific to that hardware, or does it happen on all hardware variants from time-to-time?</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Thu, Jan 23, 2020 at 11:25 AM Jeremy Plsek &lt;<a href="mailto:jplsek@iol.unh.edu" target="_blank">jplsek@iol.unh.edu</a>&gt; wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon &lt;<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>&gt; wrote:<br>
&gt;<br>
&gt; Hi,<br>
&gt;<br>
&gt; The test iol-intel-Performance is failing on all patches<br>
&gt; for a couple of days.<br>
&gt; It seems to be due to a perf drop with a 10G device<br>
&gt;<br>
&gt; Can we disable this test until we understand what happens?<br>
&gt;<br>
&gt; We need also to understand how to avoid such failure,<br>
&gt; and how to monitor and disable it faster.<br>
&gt;<br>
&gt; Thanks<br>
&gt;<br>
&gt;<br>
Hi Thomas,<br>
<br>
I've disabled it for now until someone from the Intel team can look<br>
into it (cc'd).<br>
<br>
I've also seem problems where the results are completely off<br>
(-15mpps), and I've been rerunning those when they happen. I'm hoping<br>
those can be resolved too.<br>
<br>
--<br>
Jeremy Plsek<br>
UNH InterOperability Laboratory<o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>Lincoln Lavoie</b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Senior Engineer, Broadband Technologies<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">21 Madbury Rd., Ste. 100, Durham, NH 03824<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="mailto:lylavoie@iol.unh.edu" target="_blank">lylavoie@iol.unh.edu</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="https://www.iol.unh.edu" target="_blank">https://www.iol.unh.edu</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&#43;1-603-674-2755 (m)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="https://www.iol.unh.edu/" target="_blank"><span style="text-decoration:none"><img border="0" id="gmail-m_-5920278011901207711_x005f_x0000_i1025" src="http://homeautomation.lavoieholdings.com/_/rsrc/1390068882701/unh-iol-logo.png"></span></a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<br>
-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Jeremy Plsek<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">UNH InterOperability Laboratory<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, back to index

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-23 15:58 [dpdk-ci] Intel performance test is failing Thomas Monjalon
2020-01-23 16:25 ` Jeremy Plsek
2020-01-23 20:28   ` Lincoln Lavoie
2020-01-31  3:17     ` Chen, Zhaoyan
2020-01-31 14:48       ` Jeremy Plsek
2020-02-03  3:57         ` Chen, Zhaoyan

DPDK CI discussions

Archives are clonable:
	git clone --mirror http://inbox.dpdk.org/ci/0 ci/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 ci ci/ http://inbox.dpdk.org/ci \
		ci@dpdk.org
	public-inbox-index ci


Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.ci


AGPL code for this site: git clone https://public-inbox.org/ public-inbox