From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f47.google.com (mail-wm0-f47.google.com [74.125.82.47]) by dpdk.org (Postfix) with ESMTP id D5A3D100F for ; Sat, 5 Nov 2016 20:15:13 +0100 (CET) Received: by mail-wm0-f47.google.com with SMTP id a197so110533512wmd.0 for ; Sat, 05 Nov 2016 12:15:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:user-agent:in-reply-to:references :mime-version:content-transfer-encoding; bh=aWqD5w0W2zWgZhHYRXYWRdyVB2qtYFZ260rZiznC1Rg=; b=lPeccnDTII+PVLd99NG/4Wx1TmL4aHuQTZFGJ6ikXz9jyHVeP+tMhaSqVBOqxCx7ZW 5bkNdZ3osQqHraBytSuuuPMXAgWP4EO7iEAAPypgJmufJjO3raca5+MFhi0Fg+h8BJhb Hc03yrni8CiuNGSESxEpG2rT2A3ZG+gBMhnx8Cp7+U6hZLSWipOZJoRyvY56i5KP1QFn b9LBrs3+iiyk2vHcxShnqE3rdjfh7jRPVNh+IiAfWoV8cCwtbi9uQuyfImkKk+9x/YlL wXw7H2LQczjmcthvRxgIE32UKqla0bMlmhE9LzWkBKWwoN+e2IuAywvsEK1GafRF80K7 I1ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:subject:date:message-id:user-agent :in-reply-to:references:mime-version:content-transfer-encoding; bh=aWqD5w0W2zWgZhHYRXYWRdyVB2qtYFZ260rZiznC1Rg=; b=CApf+Azb6n5Axz8gm59xM0RdBm5o6T8pcRd5srKMlTAWFxGvGqI8B+2YEX/tfZBHx/ 14WOt35kLuSuqGZJoYrrdswxTfoTK0duHPKbT9kz/ldtE+K4eKDMWxiTZBio/45SzhaD j8ue4qiENz1GNcjLA3/Ta88MZGCG/agcQixbIpon/FZK8arRdN4U658brYOZeJXNcyLt Ab0+uODgfH4R1bsOoda9dR8GF+HGlsHCGtGMI2WR4iOThkUzS4hYZxqbHGA/AEQ+ypvF tv2zSJhsOIgIMuDkyFMvwvYJkXJv3yb+/XZsxaQHBf1kTv8UztHiQOa1t5kCEjXNJAiE 98Xg== X-Gm-Message-State: ABUngvdRwOTSX/4DbbeT8CTDMxmj79PiC09n9Q6L+OyiJKVb8s0+QsfocYmEay4/2NBu2JJA X-Received: by 10.194.189.198 with SMTP id gk6mr16313248wjc.167.1478373313254; Sat, 05 Nov 2016 12:15:13 -0700 (PDT) Received: from xps13.localnet (184.203.134.77.rev.sfr.net. [77.134.203.184]) by smtp.gmail.com with ESMTPSA id q134sm3721780wme.3.2016.11.05.12.15.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 05 Nov 2016 12:15:12 -0700 (PDT) From: Thomas Monjalon To: moving@dpdk.org, "Liu, Yong" Date: Sat, 05 Nov 2016 20:15:11 +0100 Message-ID: <3804736.OkjAMiHs6v@xps13> User-Agent: KMail/4.14.10 (Linux/4.5.4-1-ARCH; KDE/4.14.11; x86_64; ; ) In-Reply-To: <86228AFD5BCD8E4EBFD2B90117B5E81E60310FA1@SHSMSX103.ccr.corp.intel.com> References: <86228AFD5BCD8E4EBFD2B90117B5E81E60310FA1@SHSMSX103.ccr.corp.intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Subject: Re: [dpdk-moving] proposal for DPDK CI improvement X-BeenThere: moving@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK community structure changes List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Nov 2016 19:15:14 -0000 2016-11-05 04:47, Liu, Yong: > Currently, DPDK CI has done in a distributed way. Several companies running > their own CI tests internally. Some companies running their own CI tests > internally. Some of them (Intel and IBM) provided their test reports to > mailing list, but others keep their test results for internal use only. I'm confident we'll have more contributors to the distributed CI when it will be well advertised (see below). > There are two possible approaches that we can consider for improving DPDK CI: > > 1. Create a centralized DPDK CI lab and buildup required infrastructure. > 2. Continue with a distributed model but improve reporting and visibility. I think these two approaches are good: 1. The centralized open lab can help as a reference 2. The distributed CI instances will bring more diversity > We think the main advantages of a centralized approach are: > Transparency: Everybody can see and access the test infrastructure, see what > exactly how the servers are configured and what tests have been > run and their result. The community can review and agree > collectively when new tests are required. > Flexibility: Testing can be performed on demand. Instead of a developer > submitting a patch, having it tested by a distributed CI > infrastructure and then getting test results. The developer can > access the CI infrastructure and trigger the tests manually > before submitting the patch, thus speeding up the development > process, make short test cycle. It is possible to offer such flexibility in private CI labs by offering an email address where we can send some patches to be tested. However there can be an issue of hardware bandwith/availability to solve. This is the same issue for open/centralized labs or private labs. A test lab accepting any private request can be abused. That's why I think forcing to send the patches publically to the mailing list is a good policy. > Independence: Instead of each vendor providing their own performance results, > having these generated in a centralized lab run by an > independent body will increase confidence that DPDK users have > in the test results. > > There is one example of how this was done for another project. > (https://wiki.fd.io/view/CSIT). > > In their wiki page, you can get the idea about how to configure the servers > and run test cases on these servers. The test report for all releases can be > found on the page. You can also browse the detail test report for each release > if click the link. If click their Jekin's link, you can see the trend of > project status. I do not see an explanation of how to use the CSIT lab on demand (what you described as "Flexibility"). How does it work? > The main disadvantages of a centralized approach are relocating equipment > from separate vendor labs will require a project budget. We can depend on > budget to decide which infrastructure should be deployed in the public test > lab. > > For distributed model, we essentially continue as what we are at present. > Vendors can independently choose the CI tests that they run, and the reports > that they choose to make public. We can add enhancements to Patchwork to > display test results to make tracking easier, and can also look at other ways > to make test reports more visible. Yes, I'm working on it. One year ago, we discussed a CI integration in patchwork: https://lists.ozlabs.org/pipermail/patchwork/2015-July/001363.html It is now implemented in patchwork and available on dpdk.org: http://dpdk.org/ml/archives/dev/2016-September/046282.html A first basic test (checkpatch) is integrated. See this example: http://dpdk.org/patch/16953 The detailed report in test-report mailing list archives is referenced with an hyperlink (in the "Description" column). The next step (work in progress) is to publish some scripts in a new git repository dpdk-ci to help integrating more test labs in patchwork. It basically requires only to receive and send some emails from the lab. Note that any open or centralized lab can be also integrated in patchwork. Note also that this patchwork integration covers only the tests run when a patch is submitted. The tests run regularly on a git tree won't appear in this interface. > The main advantages of a distributed approach are: > There's no requirement for a project budget. The other major advantage is to have a better test coverage. Many companies need to have some internal DPDK tests for their needs. If they use them to provide some public reports, they can avoid having some regressions with their specific use cases or hardware. That's why the distributed CI approach is a win-win. > The disadvantages of a distributed approach are: > We lost the benefits of transparency, independence and the ability to run > tests on demand that are described under the centralized approach above. CI > testing and the publication of the results remains under the control of > vendors (or others who choose to run CI tests). > > Based on the above, we would like to propose that a centralized CI lab. > Details of the required budget and funding for this will obviously need to be > determined, but for now our proposal will focus purely on the technical scope > and benefits. Thanks for the detailed description of the tests that you expect. I think the CI discussion must be thought with two major questions: When? and Where? When: - regularly on a git tree - after each patch submission -> report available via patchwork Where: - in a private lab - in a foundation lab Both private and foundation labs can be more or less open. My conclusion: every kind of tests have some benefits and are welcome!