From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f41.google.com (mail-wm0-f41.google.com [74.125.82.41]) by dpdk.org (Postfix) with ESMTP id E3C7347D1 for ; Mon, 7 Nov 2016 11:17:36 +0100 (CET) Received: by mail-wm0-f41.google.com with SMTP id a197so172002330wmd.0 for ; Mon, 07 Nov 2016 02:17:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:user-agent:in-reply-to :references:mime-version:content-transfer-encoding; bh=bV50n3XwyE5EwgZdl8tJSzEGgHPcv1x9Dr8xjK3OcAw=; b=x/60wQVcp+RayORDSCGgifR1hsTHWLBRmWi6yj2ck9n7os0XNZOEvaGW9pM8rGY9iw X2NVqNfD3jlhCLH157gPR5gq5yiLQ2mXalAAA8gbXdJHOBxleQpLtngyqwVlwrO7HTiR KsGofyYbBWm+ZDktolCI0mqIdPBdTOAcuqpUxuAWVESGKSwI9hmwkDpLlaSqTVYLoHUH kqpFmB7OHmpPmHbsxE6LJ+z/VuE7U/GwLLzJ1PP+BzwXyTvlzYk2yzz3cjw8Ll/ODkqu dVXzTBg/YIoco9kx9Tx8ZVi9PQhz3EA5+rg5/N2qR8GJLUqCrFnirIoMF/ULrIRsDE46 ol9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:user-agent :in-reply-to:references:mime-version:content-transfer-encoding; bh=bV50n3XwyE5EwgZdl8tJSzEGgHPcv1x9Dr8xjK3OcAw=; b=c7nZKSyrmwECYsu7fk6tLAcK8QlwuWEC207QGp+j9+h6GwWWevi72Eu2YrzDJCr0WA 79iyV3yNkkaZnbxGM5WSRnwHWkgD1VEUYyS1CTQrR/z5VecQNOoeWlOdpulFDG5dKBEM TJ+e8cw3iT8L8HX7RKH1LQQacxVrk1YSF+N13RI9QlnWm86+tIdn1lPsIRVlcdQ+9QcD NGJuD6M876gLaAASJqtPus7sYccbWD6CMeCUXbGbLP3JTzU1yHWfky+VnYCkM7A0uuX+ s7m8X29SLBF0rQLIO5XNpsaPN992mQfwUmWcEvTpqW/w/ZVgrEd8DFuiOSRsZnITY1B3 gjgQ== X-Gm-Message-State: ABUngveBR8G1KSLMssa4VdahZIIq56iofOYkG13n1hEI0+LJP3pswyDPe5d47+inNqfJdSb0 X-Received: by 10.28.9.131 with SMTP id 125mr1521360wmj.22.1478513856470; Mon, 07 Nov 2016 02:17:36 -0800 (PST) Received: from xps13.localnet (184.203.134.77.rev.sfr.net. [77.134.203.184]) by smtp.gmail.com with ESMTPSA id f76sm12598883wmd.15.2016.11.07.02.17.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 07 Nov 2016 02:17:35 -0800 (PST) From: Thomas Monjalon To: "Xu, Qian Q" Cc: moving@dpdk.org, "Liu, Yong" , ci@dpdk.org Date: Mon, 07 Nov 2016 11:17:34 +0100 Message-ID: <1689822.FXvyOjK9nz@xps13> User-Agent: KMail/4.14.10 (Linux/4.5.4-1-ARCH; KDE/4.14.11; x86_64; ; ) In-Reply-To: <82F45D86ADE5454A95A89742C8D1410E3923B784@shsmsx102.ccr.corp.intel.com> References: <86228AFD5BCD8E4EBFD2B90117B5E81E60310FA1@SHSMSX103.ccr.corp.intel.com> <3804736.OkjAMiHs6v@xps13> <82F45D86ADE5454A95A89742C8D1410E3923B784@shsmsx102.ccr.corp.intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Subject: Re: [dpdk-ci] [dpdk-moving] proposal for DPDK CI improvement X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Nov 2016 10:17:37 -0000 Hi Qian, 2016-11-07 07:55, Xu, Qian Q: > I think the discussion about CI is a good start. I agreed on the general ideas: > 1. It's good to have more contributors for CI and it's a community effort. > 2. Building a distributed CI system is good and necessary. > 3. "When and Where" is the very basic and important questions. > > Add my 2 cents here. > 1. Distributed test vs Centralized lab > We can put the build and functional tests on our distributed lab. As to the performance, as we all know, performance is key to DPDK. > So I suggested we can have the centralized lab for the performance testing, and some comments as below: > a). Do we want to publish the performance report on different platforms with different HW/NICs? Anyone against on publishing performance numbers? > b). If the answer to the first question is "Yes", so how to ensure others trust the performance and how to reproduce the performance if we don't have the platforms/HWs? > As Marvin said, transparency and independence is the advantage for open centralized lab. Besides, we can demonstrate to all audience about DPDK performance with the > Lab. Of course, we need the control of the system, not allow others to access it randomly. It's another topic of access control. I even think that if the lab can be used as > the training lab or demo lab when we have the community training or performance demo days(I just named the events). > > 2. Besides "When and Where", then "What" and "How" > When: > - regularly on a git tree ---what tests need to be done here? Propose to have the daily build, daily functional regression, daily performance regression > - after each patch submission -> report available via patchwork----what tests need to be done? Build test as the first one, maybe we can add functional or performance in future. > > How to collect and display the results? > Thanks Thomas for the hard work on patchwork upgrade. And it's good to see the CheckPatch display here. > IMHO, to build the complete distributed system needs very big effort. Thomas, any effort estimation and the schedule for it? It must be a collective effort. I plan to publish a new git repository really soon to help building a test lab. The first version will allow to send some test reports correctly formatted. The next step will be to help applying patches (on right branch with series support). > a). Currently, there is only " S/W/F for Success/Warning/Fail counters" in tests, so does it refer to build test or functional test or performance test? It can be any test, including performance ones. A major performance regression must be seen as a failed test. > If it only referred to build test, then you may need change the title to Build S/W/F. Then how many architecture or platforms for the builds? For example, we support Intel IA build, > ARM build, IBM power build. Then we may need collect build results from INTEL/IBM/ARM and etc to show the total S/W/F. For example, if the build is passed on IA but failed on IBM, then we > Need record it as 1S/0W/1F. I don't know if we need collect the warning information here. The difference between warnings and failures is a matter of severity. The checkpatch errors are reported as warnings. > b). How about performance result display on website? No matter distributed or centralized lab, we need a place to show the performance number or the performance trend to > ensure no performance regression? Do you have any plan to implement it? No I have no plan but I expect it to be solved by ones working on performance tests, maybe you? :) If a private lab can publish some web graphs of performance evolutions, it is great. If we can do it in a centralized lab, it is also great. If we can have a web interface to gather every performance numbers and graphs, it is really really great! > 3. Proposal to have a CI mailing list for people working on CI to have the regular meetings only discussing about CI? Maybe we can have more frequent meetings at first to have an alignment. Then > We can reduce the frequency if the solution is settle down. Current call is covering many other topics. What do you think? The mailing list is now created: ci@dpdk.org. About meetings, I feel we can start working through ci@dpdk.org and see how efficient it is. Though if you need a meeting, feel free to propose.