[dpdk-ci] Could we have some agreements on the CI then discuss the opens

[dpdk-ci] Could we have some agreements on the CI then discuss the opens

[Xu, Qian Q]

[-- Attachment #1: Type: text/plain, Size: 1718 bytes --]

Hi, all
I just think that if we can have some agreements on the DPDK CI tasks firstly, then discuss about the open list.
Possible agreement parts:

1.      Schedule tool: Jenkins.
LF has the Jenkins as the schedule tool. So I wonder if all agree on this schedule tool for scheduling build and regression test.


2.      Create per patch check(patchcheck and build) by using Jenkins to trigger
For each patch check, currently we cover the Patchcheck and build.
I noticed Thomas has a separate mail about checkpatch, so does it mean we can remove patchcheck from the build test?
For build test, Intel can provide Intel IA based per patch build report. If there is common format, we can follow it.



3.      Create daily or weekly functional/build regression test based on git tree, also using Jenkins to trigger
For the functional/build regression tests, Intel has already sent out the daily build and functional regression test. If there is common format, we can also follow it.
Besides Intel, I have also seen the IBM's daily build report. Any others want to publish the daily/weekly functional regression tests?

Opens:

1.      Centralized or distributed performance lab. Is the decision more dependent on budget or the thoughts?

2.      Performance report center. Do you want to publish the performance report and which is the preferred format?

3.      The code review tool is still by mailing list. Is it the final decision?

4.      How about the central bug system? Do we want to have one?

Proposal:  Could we have a CI weekly sync-up meeting for discussion only on CI? If most people on the mailing list from EU and PRC, then we could find a more friendly time for PRC people.

[-- Attachment #2: Type: text/html, Size: 10439 bytes --]

Re: [dpdk-ci] Could we have some agreements on the CI then discuss the opens

[Thomas Monjalon]

Hi,

2016-11-15 09:07, Xu, Qian Q:
> Hi, all
> I just think that if we can have some agreements on the DPDK CI tasks firstly, then discuss about the open list.

Sorry, I don't understand what you mean.

> Possible agreement parts:
> 
> 1.      Schedule tool: Jenkins.
> LF has the Jenkins as the schedule tool. So I wonder if all agree on this schedule tool for scheduling build and regression test.

The schedule tool must be a personal choice for each test instance.
Are you talking about the reference lab?

> 2.      Create per patch check(patchcheck and build) by using Jenkins to trigger
> For each patch check, currently we cover the Patchcheck and build.
> I noticed Thomas has a separate mail about checkpatch, so does it mean we can remove patchcheck from the build test?

Yes
The email checkpatch@dpdk.org is the address of this test instance.
If you send a patch directly to this instance, you will receive a private report.
It could be a good idea to do the same at Intel so we can test the compilation,
especially with ICC, before sending a public patch.
And this feature should be documented somewhere.

> For build test, Intel can provide Intel IA based per patch build report. If there is common format, we can follow it.

Now that the release 16.11 is done, I'll work on sharing some scripts.

> 3.      Create daily or weekly functional/build regression test based on git tree, also using Jenkins to trigger
> For the functional/build regression tests, Intel has already sent out the daily build and functional regression test. If there is common format, we can also follow it.
> Besides Intel, I have also seen the IBM's daily build report. Any others want to publish the daily/weekly functional regression tests?
> 
> Opens:
> 
> 1.      Centralized or distributed performance lab. Is the decision more dependent on budget or the thoughts?

Anyway the per-patch checks will be distributed and aggregated in patchwork.
If we build a reference lab inside Linux Foundation, it will be part of the
distributed CI.
So your question should be:
Are we going to have a budget for a reference lab?
Which tests will be run in this CI instance?

> 2.      Performance report center. Do you want to publish the performance report and which is the preferred format?
> 
> 3.      The code review tool is still by mailing list. Is it the final decision?
> 
> 4.      How about the central bug system? Do we want to have one?

+1, I commit to have a central bug tracking on dpdk.org during December.

> Proposal:  Could we have a CI weekly sync-up meeting for discussion only on CI? If most people on the mailing list from EU and PRC, then we could find a more friendly time for PRC people.

If interested people are Chinese and French, it would be a good idea to have
an IRC meeting.

Re: [dpdk-ci] Could we have some agreements on the CI then discuss the opens

[Xu, Qian Q]

See below, THX. 

-----Original Message-----
From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com] 
Sent: Tuesday, November 15, 2016 5:43 PM
To: Xu, Qian Q <qian.q.xu@intel.com>
Cc: ci@dpdk.org
Subject: Re: [dpdk-ci] Could we have some agreements on the CI then discuss the opens

Hi,

2016-11-15 09:07, Xu, Qian Q:
> Hi, all
> I just think that if we can have some agreements on the DPDK CI tasks firstly, then discuss about the open list.

Sorry, I don't understand what you mean.
---Sorry, I mean we have different thoughts on CI, so we can first have something that all agreed on then we can discuss something that we have different ideas/thoughts. 

> Possible agreement parts:
> 
> 1.      Schedule tool: Jenkins.
> LF has the Jenkins as the schedule tool. So I wonder if all agree on this schedule tool for scheduling build and regression test.

The schedule tool must be a personal choice for each test instance.
Are you talking about the reference lab?

---I mean the common Jenkins tool for everyone to access; Yes, it may be only valid for the reference lab. It's fine if we use the internal Jenkins tool to trigger build or regression test. 

> 2.      Create per patch check(patchcheck and build) by using Jenkins to trigger
> For each patch check, currently we cover the Patchcheck and build.
> I noticed Thomas has a separate mail about checkpatch, so does it mean we can remove patchcheck from the build test?

Yes
The email checkpatch@dpdk.org is the address of this test instance.
If you send a patch directly to this instance, you will receive a private report.
It could be a good idea to do the same at Intel so we can test the compilation, especially with ICC, before sending a public patch.

---We have some discussion on ICC within Intel and we are fine to drop ICC for per patch build, we will keep ICC in our daily test since ICC is not very popular for the community usage. 
However, the idea is cool and meantime, Intel also has the similar idea and implementation, then we may apply it for our internal other checks. 

And this feature should be documented somewhere.

> For build test, Intel can provide Intel IA based per patch build report. If there is common format, we can follow it.

Now that the release 16.11 is done, I'll work on sharing some scripts.

> 3.      Create daily or weekly functional/build regression test based on git tree, also using Jenkins to trigger
> For the functional/build regression tests, Intel has already sent out the daily build and functional regression test. If there is common format, we can also follow it.
> Besides Intel, I have also seen the IBM's daily build report. Any others want to publish the daily/weekly functional regression tests?
> 
> Opens:
> 
> 1.      Centralized or distributed performance lab. Is the decision more dependent on budget or the thoughts?

Anyway the per-patch checks will be distributed and aggregated in patchwork.----Agreed. 
If we build a reference lab inside Linux Foundation, it will be part of the distributed CI.
So your question should be:
Are we going to have a budget for a reference lab?---- Yes.  
Which tests will be run in this CI instance?----Here I prefer only performance test in the reference lab. 

> 2.      Performance report center. Do you want to publish the performance report and which is the preferred format?
> 
> 3.      The code review tool is still by mailing list. Is it the final decision?
> 
> 4.      How about the central bug system? Do we want to have one?

+1, I commit to have a central bug tracking on dpdk.org during December.
----Good to know. 

> Proposal:  Could we have a CI weekly sync-up meeting for discussion only on CI? If most people on the mailing list from EU and PRC, then we could find a more friendly time for PRC people.

If interested people are Chinese and French, it would be a good idea to have an IRC meeting.
-----What does IRC meeting mean? 

Re: [dpdk-ci] Could we have some agreements on the CI then discuss the opens

[Thomas Monjalon]

2016-11-15 10:11, Xu, Qian Q:
> > Which tests will be run in this CI instance?
> Here I prefer only performance test in the reference lab. 

Could you explain why you prefer having only performance test in the
reference lab?
Could it be regular performance tests + per-patch performance tests?

> > > Proposal:  Could we have a CI weekly sync-up meeting for
> > > discussion only on CI? If most people on the mailing list from
> > > EU and PRC, then we could find a more friendly time for PRC people.
> 
> > If interested people are Chinese and French, it would be a good idea
> > to have an IRC meeting.
> 
> What does IRC meeting mean?

I mean not a phone meeting but a chat meeting instead.
I suggest using a dedicated room #dpdk-ci on freenode.

Re: [dpdk-ci] Could we have some agreements on the CI then discuss the opens

[Xu, Qian Q]

-----Original Message-----
From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com] 
Sent: Tuesday, November 15, 2016 6:19 PM
To: Xu, Qian Q <qian.q.xu@intel.com>
Cc: ci@dpdk.org
Subject: Re: [dpdk-ci] Could we have some agreements on the CI then discuss the opens

2016-11-15 10:11, Xu, Qian Q:
> > Which tests will be run in this CI instance?
> Here I prefer only performance test in the reference lab. 

Could you explain why you prefer having only performance test in the reference lab?
Could it be regular performance tests + per-patch performance tests?

--- I guess we may have limited budget(correct me if we are very rich, then I am fine to put more things in it!), then we may need make the maximum use of the money. Performance is the key for dpdk, so we can focus on performance test in the open lab. 
As the previous discussion, besides me, some people also think that performance test reports should be provided in an open lab. 
Just quote Jerome's words here, and I agreed with it. Besides that, I only think we may use the performance lab as the demo or training lab for more audiences. 
The performance can be the regular performance test with the software traffic generator as first step. We may think about per-patch performance test later, maybe per-patchset is more accurate. And we'd better to solve the multiple repos' apply issue. 

"Hi Thomas & Qian,
> IMHO, performance results should be centralized and executed in a
> trusted & controlled environment.
> If official DPDK numbers are coming from private lab's vendors,
> perception might be that they are not 100% neutral. That would probably
> not help DPDK community to be seen open & transparent."


I mean not a phone meeting but a chat meeting instead.
I suggest using a dedicated room #dpdk-ci on freenode.
----OK, I haven't had that meeting before may need your help to know how to dial in from PRC.