DPDK CI discussions
 help / color / Atom feed
* [dpdk-ci] Minutes of DPDK Lab Meeting, July 17th
@ 2018-07-19 14:54 O'Driscoll, Tim
  2018-08-16  9:33 ` Yigit, Ferruh
  0 siblings, 1 reply; 4+ messages in thread
From: O'Driscoll, Tim @ 2018-07-19 14:54 UTC (permalink / raw)
  To: ci
  Cc: Bob Noseworthy, Mcnamara, John, Shepard Siegel, Thomas Monjalon,
	Erez Scop, Shreyansh Jain, Xu, Qian Q, pmacarth, Matt Spencer,
	George Zhao, Mishra, Shishir, Lixuming, Tkachuk, Georgii,
	Trishan de Lanerolle, Sean Campbell, Ali Alnubani, May Chen,
	Lodha, Nishant, Zhang, Chun, Malla, Malathi, khemendra kumar,
	graeme.gregory, Yigit, Ferruh, Tu, Lijuan

[-- Attachment #1: Type: text/plain, Size: 1029 bytes --]

UNH Policies and Procedures Doc:
Document is available at: https://docs.google.com/document/d/1rtSwpNKltVNyDKNWgeTV5gaYeDoAPlK-sfm8XE7o_5s/edit?usp=sharing
I reviewed it and it looks good to me.
Please add any comments to the google doc, or send them directly to Bob (ren@iol.unh.edu<mailto:ren@iol.unh.edu>).

Dashboard:
Dashboard is in much better shape now and most tests are passing.
We do need to monitor the results and determine if the tolerance is right. If it's too high then all tests will pass and we'll miss any genuine issues. If it's too low then we'll have too many tests failing when there isn't really a problem. Agreed that tuning the tolerance is the responsibility of each vendor.
For Intel, we need to add the specific test config to the results page. We need to confirm the config details to Patrick, and he'll add this to the results pages. Other vendors may want to do something similar.

Hardware:
Shreyansh is almost ready to ship the NXP hardware. It should be in UNH in 2-3 weeks.





[-- Attachment #2: Type: text/html, Size: 5952 bytes --]

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:Wingdings;
	panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:#0563C1;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:#954F72;
	text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
	{mso-style-priority:34;
	margin-top:0in;
	margin-right:0in;
	margin-bottom:0in;
	margin-left:.5in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-family:"Calibri",sans-serif;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
/* List Definitions */
@list l0
	{mso-list-id:724375174;
	mso-list-type:hybrid;
	mso-list-template-ids:2124727560 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
	{mso-level-number-format:bullet;
	mso-level-text:\F0B7;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-.25in;
	font-family:Symbol;}
@list l0:level2
	{mso-level-number-format:bullet;
	mso-level-text:o;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-.25in;
	font-family:"Courier New";}
@list l0:level3
	{mso-level-number-format:bullet;
	mso-level-text:\F0A7;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-.25in;
	font-family:Wingdings;}
@list l0:level4
	{mso-level-number-format:bullet;
	mso-level-text:\F0B7;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-.25in;
	font-family:Symbol;}
@list l0:level5
	{mso-level-number-format:bullet;
	mso-level-text:o;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-.25in;
	font-family:"Courier New";}
@list l0:level6
	{mso-level-number-format:bullet;
	mso-level-text:\F0A7;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-.25in;
	font-family:Wingdings;}
@list l0:level7
	{mso-level-number-format:bullet;
	mso-level-text:\F0B7;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-.25in;
	font-family:Symbol;}
@list l0:level8
	{mso-level-number-format:bullet;
	mso-level-text:o;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-.25in;
	font-family:"Courier New";}
@list l0:level9
	{mso-level-number-format:bullet;
	mso-level-text:\F0A7;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-.25in;
	font-family:Wingdings;}
ol
	{margin-bottom:0in;}
ul
	{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">UNH Policies and Procedures Doc:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Document is available at: <a href="https://docs.google.com/document/d/1rtSwpNKltVNyDKNWgeTV5gaYeDoAPlK-sfm8XE7o_5s/edit?usp=sharing">
https://docs.google.com/document/d/1rtSwpNKltVNyDKNWgeTV5gaYeDoAPlK-sfm8XE7o_5s/edit?usp=sharing</a><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">I reviewed it and it looks good to me.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Please add any comments to the google doc, or send them directly to Bob (<a href="mailto:ren@iol.unh.edu">ren@iol.unh.edu</a>).<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Dashboard:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Dashboard is in much better shape now and most tests are passing.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">We do need to monitor the results and determine if the tolerance is right. If it&#8217;s too high then all tests will pass and we&#8217;ll miss any genuine issues. If it&#8217;s too low then we&#8217;ll have too many tests failing when
 there isn&#8217;t really a problem. Agreed that tuning the tolerance is the responsibility of each vendor.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">For Intel, we need to add the specific test config to the results page. We need to confirm the config details to Patrick, and he&#8217;ll add this to the results pages. Other vendors may want to do something similar.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Hardware:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Shreyansh is almost ready to ship the NXP hardware. It should be in UNH in 2-3 weeks.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
</body>
</html>

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [dpdk-ci] Minutes of DPDK Lab Meeting, July 17th
  2018-07-19 14:54 [dpdk-ci] Minutes of DPDK Lab Meeting, July 17th O'Driscoll, Tim
@ 2018-08-16  9:33 ` Yigit, Ferruh
  2018-08-16 19:58   ` Jeremy Plsek
  0 siblings, 1 reply; 4+ messages in thread
From: Yigit, Ferruh @ 2018-08-16  9:33 UTC (permalink / raw)
  To: ren; +Cc: O'Driscoll, Tim, ci

[-- Attachment #1: Type: text/plain, Size: 4458 bytes --]

Hi Bob,

A few comments/questions on "UNH Policies and Procedures Doc":

- Objective and Scope of Project
                - 2: "DPDK-enabled applications", what are these applications, are they refer to testpmd/l2fwd like test applications, or OVS/VPP like other products using DPDK? And if these are other products, will they run for all vendors or for the vendor requested it?
                - 3: "Demonstrate any new feature performance of DPDK", is this via updating test scripts, if not how these new test will be run and how results will be shared?
                - What is the plan to use Lab for Continuous Integration, adding more sanity checks than performance checks by time, as far as I know this was one of the initial plans for a lab.

- DPDK Branch(s) to test
                - 5.1.1: "master", a little background, in dpdk development there are multiple sub-trees, some specific patches targets specific trees, these sub-trees are merged into main tree before release candidate, and there is a target to do regular integration from these sub-trees.
As a result of this process, for example a patch sent for next-net sub-tree may not apply cleanly to main repo, so it won't be tested. But that patch will be applied to next-net tree and a week later next-net tree will be merged into main tree, this patch can be something affects the performance, but it won't be detected. Later, when a patch arrives that can be applied on main tree, it will reveal the performance issue, but suspect will be the wrong patch and the problematic patch will be already merged. We need a solution for this. There are 5 sub-trees merged into main tree and more than half of the patches are coming to main repo through them.

- Private DPDK-Member only Dashboard Specification
- 5.6.1.2: "The delta-values of the script output, per test performed.", in member-only dashboard, why not show base value too, since it will be updated regularly, via "--update-expected argument", it would be good to see both current baseline and the diff.

- I am for defining a change management system, there are multiple vendor and multiple requests, it would be good to trace, discuss and record the result for all of them systematically.

Thanks,
ferruh

From: O'Driscoll, Tim
Sent: Thursday, July 19, 2018 3:55 PM
To: 'ci@dpdk.org' <ci@dpdk.org>
Cc: 'Bob Noseworthy' <ren@iol.unh.edu>; Mcnamara, John <john.mcnamara@intel.com>; 'Shepard Siegel' <shepard.siegel@atomicrules.com>; 'Thomas Monjalon' <thomas@monjalon.net>; 'Erez Scop' <erezsc@mellanox.com>; 'Shreyansh Jain' <shreyansh.jain@nxp.com>; Xu, Qian Q <qian.q.xu@intel.com>; 'pmacarth@iol.unh.edu' <pmacarth@iol.unh.edu>; 'Matt Spencer' <Matt.Spencer@arm.com>; 'George Zhao' <George.Y.Zhao@huawei.com>; 'Mishra, Shishir' <Shishir.Mishra@spirent.com>; 'Lixuming' <lixuming@huawei.com>; Tkachuk, Georgii <georgii.tkachuk@intel.com>; 'Trishan de Lanerolle' <tdelanerolle@linuxfoundation.org>; 'Sean Campbell' <scampbel@qti.qualcomm.com>; 'Ali Alnubani' <alialnu@mellanox.com>; 'May Chen' <May.Chen@huawei.com>; 'Lodha, Nishant' <Nishant.Lodha@cavium.com>; Zhang, Chun <chun.zhang@intel.com>; 'Malla, Malathi' <Malathi.Malla@spirent.com>; 'khemendra kumar' <khemendra.kumar13@gmail.com>; 'graeme.gregory@linaro.org' <graeme.gregory@linaro.org>; Yigit, Ferruh <ferruh.yigit@intel.com>; Tu, Lijuan <lijuan.tu@intel.com>
Subject: Minutes of DPDK Lab Meeting, July 17th

UNH Policies and Procedures Doc:
Document is available at: https://docs.google.com/document/d/1rtSwpNKltVNyDKNWgeTV5gaYeDoAPlK-sfm8XE7o_5s/edit?usp=sharing
I reviewed it and it looks good to me.
Please add any comments to the google doc, or send them directly to Bob (ren@iol.unh.edu<mailto:ren@iol.unh.edu>).

Dashboard:
Dashboard is in much better shape now and most tests are passing.
We do need to monitor the results and determine if the tolerance is right. If it's too high then all tests will pass and we'll miss any genuine issues. If it's too low then we'll have too many tests failing when there isn't really a problem. Agreed that tuning the tolerance is the responsibility of each vendor.
For Intel, we need to add the specific test config to the results page. We need to confirm the config details to Patrick, and he'll add this to the results pages. Other vendors may want to do something similar.

Hardware:
Shreyansh is almost ready to ship the NXP hardware. It should be in UNH in 2-3 weeks.





[-- Attachment #2: Type: text/html, Size: 10096 bytes --]

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:#0563C1;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:#954F72;
	text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
	{mso-style-priority:34;
	margin-top:0in;
	margin-right:0in;
	margin-bottom:0in;
	margin-left:.5in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
span.EmailStyle18
	{mso-style-type:personal;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
span.EmailStyle19
	{mso-style-type:personal;
	font-family:"Calibri",sans-serif;
	color:#1F497D;}
span.EmailStyle20
	{mso-style-type:personal-compose;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-size:10.0pt;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hi Bob,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">A few comments/questions on &#8220;UNH Policies and Procedures Doc&#8221;:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">- Objective and Scope of Project<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - 2: &#8220;DPDK-enabled applications&#8221;, what are these applications, are they refer to testpmd/l2fwd like test applications, or OVS/VPP like other products using DPDK? And if these are other products,
 will they run for all vendors or for the vendor requested it?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - 3: &#8220;Demonstrate any new feature performance of DPDK&#8221;, is this via updating test scripts, if not how these new test will be run and how results will be shared?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - What is the plan to use Lab for Continuous Integration, adding more sanity checks than performance checks by time, as far as I know this was one of the initial plans for a lab.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">- DPDK Branch(s) to test<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - 5.1.1: &#8220;master&#8221;, a little background, in dpdk development there are multiple sub-trees, some specific patches targets specific trees, these sub-trees are merged into main tree before release
 candidate, and there is a target to do regular integration from these sub-trees.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">As a result of this process, for example a patch sent for next-net sub-tree may not apply cleanly to main repo, so it won&#8217;t be tested. But that patch will be applied to next-net tree and a week later next-net
 tree will be merged into main tree, this patch can be something affects the performance, but it won&#8217;t be detected. Later, when a patch arrives that can be applied on main tree, it will reveal the performance issue, but suspect will be the wrong patch and the
 problematic patch will be already merged. <b>We need a solution for this.</b> There are 5 sub-trees merged into main tree and more than half of the patches are coming to main repo through them.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">- Private DPDK-Member only Dashboard Specification
<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="color:#1F497D">- 5.6.1.2: &#8220;The delta-values of the script output, per test performed.&#8221;, in member-only dashboard, why not show base value too, since it will be updated regularly, via &#8220;--update-expected
 argument&#8221;, it would be good to see both current baseline and the diff.<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="color:#1F497D"><o:p>&nbsp;</o:p></span></a></p>
<p class="MsoNormal"><span style="color:#1F497D">- I am for defining a change management system, there are multiple vendor and multiple requests, it would be good to trace, discuss and record the result for all of them systematically.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">ferruh<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><a name="_____replyseparator"></a><b>From:</b> O'Driscoll, Tim
<br>
<b>Sent:</b> Thursday, July 19, 2018 3:55 PM<br>
<b>To:</b> 'ci@dpdk.org' &lt;ci@dpdk.org&gt;<br>
<b>Cc:</b> 'Bob Noseworthy' &lt;ren@iol.unh.edu&gt;; Mcnamara, John &lt;john.mcnamara@intel.com&gt;; 'Shepard Siegel' &lt;shepard.siegel@atomicrules.com&gt;; 'Thomas Monjalon' &lt;thomas@monjalon.net&gt;; 'Erez Scop' &lt;erezsc@mellanox.com&gt;; 'Shreyansh Jain' &lt;shreyansh.jain@nxp.com&gt;;
 Xu, Qian Q &lt;qian.q.xu@intel.com&gt;; 'pmacarth@iol.unh.edu' &lt;pmacarth@iol.unh.edu&gt;; 'Matt Spencer' &lt;Matt.Spencer@arm.com&gt;; 'George Zhao' &lt;George.Y.Zhao@huawei.com&gt;; 'Mishra, Shishir' &lt;Shishir.Mishra@spirent.com&gt;; 'Lixuming' &lt;lixuming@huawei.com&gt;; Tkachuk, Georgii
 &lt;georgii.tkachuk@intel.com&gt;; 'Trishan de Lanerolle' &lt;tdelanerolle@linuxfoundation.org&gt;; 'Sean Campbell' &lt;scampbel@qti.qualcomm.com&gt;; 'Ali Alnubani' &lt;alialnu@mellanox.com&gt;; 'May Chen' &lt;May.Chen@huawei.com&gt;; 'Lodha, Nishant' &lt;Nishant.Lodha@cavium.com&gt;; Zhang,
 Chun &lt;chun.zhang@intel.com&gt;; 'Malla, Malathi' &lt;Malathi.Malla@spirent.com&gt;; 'khemendra kumar' &lt;khemendra.kumar13@gmail.com&gt;; 'graeme.gregory@linaro.org' &lt;graeme.gregory@linaro.org&gt;; Yigit, Ferruh &lt;ferruh.yigit@intel.com&gt;; Tu, Lijuan &lt;lijuan.tu@intel.com&gt;<br>
<b>Subject:</b> Minutes of DPDK Lab Meeting, July 17th<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">UNH Policies and Procedures Doc:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Document is available at: <a href="https://docs.google.com/document/d/1rtSwpNKltVNyDKNWgeTV5gaYeDoAPlK-sfm8XE7o_5s/edit?usp=sharing">
https://docs.google.com/document/d/1rtSwpNKltVNyDKNWgeTV5gaYeDoAPlK-sfm8XE7o_5s/edit?usp=sharing</a><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">I reviewed it and it looks good to me.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Please add any comments to the google doc, or send them directly to Bob (<a href="mailto:ren@iol.unh.edu">ren@iol.unh.edu</a>).<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Dashboard:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Dashboard is in much better shape now and most tests are passing.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">We do need to monitor the results and determine if the tolerance is right. If it&#8217;s too high then all tests will pass and we&#8217;ll miss any genuine issues. If it&#8217;s too low then we&#8217;ll have too many tests failing when
 there isn&#8217;t really a problem. Agreed that tuning the tolerance is the responsibility of each vendor.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">For Intel, we need to add the specific test config to the results page. We need to confirm the config details to Patrick, and he&#8217;ll add this to the results pages. Other vendors may want to do something similar.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Hardware:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Shreyansh is almost ready to ship the NXP hardware. It should be in UNH in 2-3 weeks.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p>&nbsp;</o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
</div>
</body>
</html>

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [dpdk-ci] Minutes of DPDK Lab Meeting, July 17th
  2018-08-16  9:33 ` Yigit, Ferruh
@ 2018-08-16 19:58   ` Jeremy Plsek
  2018-08-17 16:30     ` Yigit, Ferruh
  0 siblings, 1 reply; 4+ messages in thread
From: Jeremy Plsek @ 2018-08-16 19:58 UTC (permalink / raw)
  To: Yigit, Ferruh; +Cc: Bob Noseworthy, O'Driscoll, Tim, ci

Hi Ferruh,

With regards to "DPDK Branch(s) to test", I agree with testing the
patch against the appropriate branch.

For some history, we originally tried to apply patches to the
appropriate branches (before Patchwork 2.0) based on the heuristics of
the subject line, but we ended up having a lot of apply errors. After
a meeting, the group agreed with testing against master for now.

But with Patchwork 2.0, is there a way to tell what branch a
patch/series is supposed to be applied against? If not, how can we
figure this out in a reliable programmatic way?

On Thu, Aug 16, 2018 at 5:33 AM Yigit, Ferruh <ferruh.yigit@intel.com> wrote:
>
> Hi Bob,
>
>
>
> A few comments/questions on “UNH Policies and Procedures Doc”:
>
>
>
> - Objective and Scope of Project
>
>                 - 2: “DPDK-enabled applications”, what are these applications, are they refer to testpmd/l2fwd like test applications, or OVS/VPP like other products using DPDK? And if these are other products, will they run for all vendors or for the vendor requested it?
>
>                 - 3: “Demonstrate any new feature performance of DPDK”, is this via updating test scripts, if not how these new test will be run and how results will be shared?
>
>                 - What is the plan to use Lab for Continuous Integration, adding more sanity checks than performance checks by time, as far as I know this was one of the initial plans for a lab.
>
>
>
> - DPDK Branch(s) to test
>
>                 - 5.1.1: “master”, a little background, in dpdk development there are multiple sub-trees, some specific patches targets specific trees, these sub-trees are merged into main tree before release candidate, and there is a target to do regular integration from these sub-trees.
>
> As a result of this process, for example a patch sent for next-net sub-tree may not apply cleanly to main repo, so it won’t be tested. But that patch will be applied to next-net tree and a week later next-net tree will be merged into main tree, this patch can be something affects the performance, but it won’t be detected. Later, when a patch arrives that can be applied on main tree, it will reveal the performance issue, but suspect will be the wrong patch and the problematic patch will be already merged. We need a solution for this. There are 5 sub-trees merged into main tree and more than half of the patches are coming to main repo through them.
>
>
>
> - Private DPDK-Member only Dashboard Specification
>
> - 5.6.1.2: “The delta-values of the script output, per test performed.”, in member-only dashboard, why not show base value too, since it will be updated regularly, via “--update-expected argument”, it would be good to see both current baseline and the diff.
>
>
>
> - I am for defining a change management system, there are multiple vendor and multiple requests, it would be good to trace, discuss and record the result for all of them systematically.
>
>
>
> Thanks,
>
> ferruh
>
>
>
> From: O'Driscoll, Tim
> Sent: Thursday, July 19, 2018 3:55 PM
> To: 'ci@dpdk.org' <ci@dpdk.org>
> Cc: 'Bob Noseworthy' <ren@iol.unh.edu>; Mcnamara, John <john.mcnamara@intel.com>; 'Shepard Siegel' <shepard.siegel@atomicrules.com>; 'Thomas Monjalon' <thomas@monjalon.net>; 'Erez Scop' <erezsc@mellanox.com>; 'Shreyansh Jain' <shreyansh.jain@nxp.com>; Xu, Qian Q <qian.q.xu@intel.com>; 'pmacarth@iol.unh.edu' <pmacarth@iol.unh.edu>; 'Matt Spencer' <Matt.Spencer@arm.com>; 'George Zhao' <George.Y.Zhao@huawei.com>; 'Mishra, Shishir' <Shishir.Mishra@spirent.com>; 'Lixuming' <lixuming@huawei.com>; Tkachuk, Georgii <georgii.tkachuk@intel.com>; 'Trishan de Lanerolle' <tdelanerolle@linuxfoundation.org>; 'Sean Campbell' <scampbel@qti.qualcomm.com>; 'Ali Alnubani' <alialnu@mellanox.com>; 'May Chen' <May.Chen@huawei.com>; 'Lodha, Nishant' <Nishant.Lodha@cavium.com>; Zhang, Chun <chun.zhang@intel.com>; 'Malla, Malathi' <Malathi.Malla@spirent.com>; 'khemendra kumar' <khemendra.kumar13@gmail.com>; 'graeme.gregory@linaro.org' <graeme.gregory@linaro.org>; Yigit, Ferruh <ferruh.yigit@intel.com>; Tu, Lijuan <lijuan.tu@intel.com>
> Subject: Minutes of DPDK Lab Meeting, July 17th
>
>
>
> UNH Policies and Procedures Doc:
>
> Document is available at: https://docs.google.com/document/d/1rtSwpNKltVNyDKNWgeTV5gaYeDoAPlK-sfm8XE7o_5s/edit?usp=sharing
>
> I reviewed it and it looks good to me.
>
> Please add any comments to the google doc, or send them directly to Bob (ren@iol.unh.edu).
>
>
>
> Dashboard:
>
> Dashboard is in much better shape now and most tests are passing.
>
> We do need to monitor the results and determine if the tolerance is right. If it’s too high then all tests will pass and we’ll miss any genuine issues. If it’s too low then we’ll have too many tests failing when there isn’t really a problem. Agreed that tuning the tolerance is the responsibility of each vendor.
>
> For Intel, we need to add the specific test config to the results page. We need to confirm the config details to Patrick, and he’ll add this to the results pages. Other vendors may want to do something similar.
>
>
>
> Hardware:
>
> Shreyansh is almost ready to ship the NXP hardware. It should be in UNH in 2-3 weeks.
>
>
>
>
>
>
>
>



--
Jeremy Plsek
UNH InterOperability Laboratory

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [dpdk-ci] Minutes of DPDK Lab Meeting, July 17th
  2018-08-16 19:58   ` Jeremy Plsek
@ 2018-08-17 16:30     ` Yigit, Ferruh
  0 siblings, 0 replies; 4+ messages in thread
From: Yigit, Ferruh @ 2018-08-17 16:30 UTC (permalink / raw)
  To: Jeremy Plsek; +Cc: Bob Noseworthy, O'Driscoll, Tim, ci

Hi Jeremy,

Testing multiple sub-tree will make things more complex, that is why I was reluctant to suggest this, I think it is better to stick to single main repo.
And there is no clear way of knowing what is the target tree for a patch.

What about to trigger the regression test from main repo for given number of commits?
This can be used when a sub-tree merged into main repo, main repo can be tested for new commits.

Still there is a downside of patch being already merged, agree this is bad.
But at list will detect the correct patch that cause the issue and will catch early, at least earlier than RC.

Regards,
ferruh

> -----Original Message-----
> From: Jeremy Plsek [mailto:jplsek@iol.unh.edu]
> Sent: Thursday, August 16, 2018 8:58 PM
> To: Yigit, Ferruh <ferruh.yigit@intel.com>
> Cc: Bob Noseworthy <ren@iol.unh.edu>; O'Driscoll, Tim
> <tim.odriscoll@intel.com>; ci@dpdk.org
> Subject: Re: [dpdk-ci] Minutes of DPDK Lab Meeting, July 17th
> 
> Hi Ferruh,
> 
> With regards to "DPDK Branch(s) to test", I agree with testing the patch
> against the appropriate branch.
> 
> For some history, we originally tried to apply patches to the appropriate
> branches (before Patchwork 2.0) based on the heuristics of the subject line,
> but we ended up having a lot of apply errors. After a meeting, the group
> agreed with testing against master for now.
> 
> But with Patchwork 2.0, is there a way to tell what branch a patch/series is
> supposed to be applied against? If not, how can we figure this out in a
> reliable programmatic way?
> 
> On Thu, Aug 16, 2018 at 5:33 AM Yigit, Ferruh <ferruh.yigit@intel.com>
> wrote:
> >
> > Hi Bob,
> >
> >
> >
> > A few comments/questions on “UNH Policies and Procedures Doc”:
> >
> >
> >
> > - Objective and Scope of Project
> >
> >                 - 2: “DPDK-enabled applications”, what are these applications, are
> they refer to testpmd/l2fwd like test applications, or OVS/VPP like other
> products using DPDK? And if these are other products, will they run for all
> vendors or for the vendor requested it?
> >
> >                 - 3: “Demonstrate any new feature performance of DPDK”, is this
> via updating test scripts, if not how these new test will be run and how
> results will be shared?
> >
> >                 - What is the plan to use Lab for Continuous Integration, adding
> more sanity checks than performance checks by time, as far as I know this
> was one of the initial plans for a lab.
> >
> >
> >
> > - DPDK Branch(s) to test
> >
> >                 - 5.1.1: “master”, a little background, in dpdk development there
> are multiple sub-trees, some specific patches targets specific trees, these
> sub-trees are merged into main tree before release candidate, and there is a
> target to do regular integration from these sub-trees.
> >
> > As a result of this process, for example a patch sent for next-net sub-tree
> may not apply cleanly to main repo, so it won’t be tested. But that patch will
> be applied to next-net tree and a week later next-net tree will be merged
> into main tree, this patch can be something affects the performance, but it
> won’t be detected. Later, when a patch arrives that can be applied on main
> tree, it will reveal the performance issue, but suspect will be the wrong patch
> and the problematic patch will be already merged. We need a solution for
> this. There are 5 sub-trees merged into main tree and more than half of the
> patches are coming to main repo through them.
> >
> >
> >
> > - Private DPDK-Member only Dashboard Specification
> >
> > - 5.6.1.2: “The delta-values of the script output, per test performed.”, in
> member-only dashboard, why not show base value too, since it will be
> updated regularly, via “--update-expected argument”, it would be good to
> see both current baseline and the diff.
> >
> >
> >
> > - I am for defining a change management system, there are multiple
> vendor and multiple requests, it would be good to trace, discuss and record
> the result for all of them systematically.
> >
> >
> >
> > Thanks,
> >
> > ferruh
> >
> >
> >
> > From: O'Driscoll, Tim
> > Sent: Thursday, July 19, 2018 3:55 PM
> > To: 'ci@dpdk.org' <ci@dpdk.org>
> > Cc: 'Bob Noseworthy' <ren@iol.unh.edu>; Mcnamara, John
> > <john.mcnamara@intel.com>; 'Shepard Siegel'
> > <shepard.siegel@atomicrules.com>; 'Thomas Monjalon'
> > <thomas@monjalon.net>; 'Erez Scop' <erezsc@mellanox.com>;
> 'Shreyansh
> > Jain' <shreyansh.jain@nxp.com>; Xu, Qian Q <qian.q.xu@intel.com>;
> > 'pmacarth@iol.unh.edu' <pmacarth@iol.unh.edu>; 'Matt Spencer'
> > <Matt.Spencer@arm.com>; 'George Zhao'
> <George.Y.Zhao@huawei.com>;
> > 'Mishra, Shishir' <Shishir.Mishra@spirent.com>; 'Lixuming'
> > <lixuming@huawei.com>; Tkachuk, Georgii <georgii.tkachuk@intel.com>;
> > 'Trishan de Lanerolle' <tdelanerolle@linuxfoundation.org>; 'Sean
> > Campbell' <scampbel@qti.qualcomm.com>; 'Ali Alnubani'
> > <alialnu@mellanox.com>; 'May Chen' <May.Chen@huawei.com>; 'Lodha,
> > Nishant' <Nishant.Lodha@cavium.com>; Zhang, Chun
> > <chun.zhang@intel.com>; 'Malla, Malathi' <Malathi.Malla@spirent.com>;
> > 'khemendra kumar' <khemendra.kumar13@gmail.com>;
> > 'graeme.gregory@linaro.org' <graeme.gregory@linaro.org>; Yigit, Ferruh
> > <ferruh.yigit@intel.com>; Tu, Lijuan <lijuan.tu@intel.com>
> > Subject: Minutes of DPDK Lab Meeting, July 17th
> >
> >
> >
> > UNH Policies and Procedures Doc:
> >
> > Document is available at:
> >
> https://docs.google.com/document/d/1rtSwpNKltVNyDKNWgeTV5gaYeDoA
> PlK-sf
> > m8XE7o_5s/edit?usp=sharing
> >
> > I reviewed it and it looks good to me.
> >
> > Please add any comments to the google doc, or send them directly to Bob
> (ren@iol.unh.edu).
> >
> >
> >
> > Dashboard:
> >
> > Dashboard is in much better shape now and most tests are passing.
> >
> > We do need to monitor the results and determine if the tolerance is right.
> If it’s too high then all tests will pass and we’ll miss any genuine issues. If it’s
> too low then we’ll have too many tests failing when there isn’t really a
> problem. Agreed that tuning the tolerance is the responsibility of each
> vendor.
> >
> > For Intel, we need to add the specific test config to the results page. We
> need to confirm the config details to Patrick, and he’ll add this to the results
> pages. Other vendors may want to do something similar.
> >
> >
> >
> > Hardware:
> >
> > Shreyansh is almost ready to ship the NXP hardware. It should be in UNH in
> 2-3 weeks.
> >
> >
> >
> >
> >
> >
> >
> >
> 
> 
> 
> --
> Jeremy Plsek
> UNH InterOperability Laboratory

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, back to index

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-07-19 14:54 [dpdk-ci] Minutes of DPDK Lab Meeting, July 17th O'Driscoll, Tim
2018-08-16  9:33 ` Yigit, Ferruh
2018-08-16 19:58   ` Jeremy Plsek
2018-08-17 16:30     ` Yigit, Ferruh

DPDK CI discussions

Archives are clonable:
	git clone --mirror http://inbox.dpdk.org/ci/0 ci/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 ci ci/ http://inbox.dpdk.org/ci \
		ci@dpdk.org
	public-inbox-index ci


Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.ci


AGPL code for this site: git clone https://public-inbox.org/ public-inbox