From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f45.google.com (mail-wm0-f45.google.com [74.125.82.45]) by dpdk.org (Postfix) with ESMTP id BB3E5F72 for ; Mon, 7 Nov 2016 10:51:37 +0100 (CET) Received: by mail-wm0-f45.google.com with SMTP id p190so170966690wmp.1 for ; Mon, 07 Nov 2016 01:51:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:user-agent:mime-version :content-transfer-encoding; bh=5nWKtepteE6n8dnauOusziwSc541vqf7jndc/J9pV6Y=; b=R2aIxW+2S9PjSqJEGhSp65BjquuYOcu9r3zbE+tyZQ5ySA0sk5Sn3aFGtAmVRnzNMV mKshaRj6E2F6oryQ+U5bhDfC9kmmg+cFW4Lok5K/GRal9dcMaMgdlfzeuAljT9UhP+1L pQuL53IG8U9PiYIFEaIoNThbAdjxRApGs27veTxZIvBAxbNJNuHGxpfeV2UYNYPDlIrL SQMgInsqSYqfYXWWQZnuxZ2CaRftE669qv8HbWgZKpNr3+e0ednSvvnc/NzWHEyDoQoN uW6efhNSfbGQARfux/GSgKKuUpeNQJTcGkI63u+/cV4NtbTEEIWmzVbjwo7ggjoxYgqT vy0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:subject:date:message-id:user-agent :mime-version:content-transfer-encoding; bh=5nWKtepteE6n8dnauOusziwSc541vqf7jndc/J9pV6Y=; b=fJHydPkBUern4kGK55x1vrq03qjR7gln5VQ7cGE0tfkP0xGEMHXvD7xnME2Ka4UKgc 3cBEil65jYiJOZaXPAO7ZBhErtDBaC8r1UtjD7PJggfaCi+V2P5GOgOOW3QUjJeI5a/v nb0dygmylvCmQFAnbiv1tB4k5jma7bEaZ865IpNxbPq6omhJmRINY7NR/Mpcobx7Hw4k QJiXE5a25HwhX2TMlz0py4TpTAg3/VVQenyjqbQkHYdjY+8IXgK8hOWL8P8y0kljHV9e 6ij3X/1KjHEFh8s1TqCiafc/wPB2GUvrVO4P4DxbG1nVZtFGIa5+IUqfxL22wVTj7Wvz SoiQ== X-Gm-Message-State: ABUngvcKADgp3mJ6zy8MiNJ0LtCosJt77+qKSbmu+YFuE3Eu7NIPjIgTHdZSpcNAVbHzyKCc X-Received: by 10.194.29.161 with SMTP id l1mr4614076wjh.97.1478512296886; Mon, 07 Nov 2016 01:51:36 -0800 (PST) Received: from xps13.localnet (184.203.134.77.rev.sfr.net. [77.134.203.184]) by smtp.gmail.com with ESMTPSA id z6sm21015421wjt.24.2016.11.07.01.51.36 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 07 Nov 2016 01:51:36 -0800 (PST) From: Thomas Monjalon To: ci@dpdk.org Date: Mon, 07 Nov 2016 10:51:35 +0100 Message-ID: <1882715.28clKk7TcT@xps13> User-Agent: KMail/4.14.10 (Linux/4.5.4-1-ARCH; KDE/4.14.11; x86_64; ; ) MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Subject: [dpdk-ci] Fwd: Re: [dpdk-moving] proposal for DPDK CI improvement X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Nov 2016 09:51:38 -0000 From: Thomas Monjalon To: moving@dpdk.org, Liu, Yong 2016-11-05 04:47, Liu, Yong: > Currently, DPDK CI has done in a distributed way. Several companies running > their own CI tests internally. Some companies running their own CI tests > internally. Some of them (Intel and IBM) provided their test reports to > mailing list, but others keep their test results for internal use only. I'm confident we'll have more contributors to the distributed CI when it will be well advertised (see below). > There are two possible approaches that we can consider for improving DPDK CI: > > 1. Create a centralized DPDK CI lab and buildup required infrastructure. > 2. Continue with a distributed model but improve reporting and visibility. I think these two approaches are good: 1. The centralized open lab can help as a reference 2. The distributed CI instances will bring more diversity > We think the main advantages of a centralized approach are: > Transparency: Everybody can see and access the test infrastructure, see what > exactly how the servers are configured and what tests have been > run and their result. The community can review and agree > collectively when new tests are required. > Flexibility: Testing can be performed on demand. Instead of a developer > submitting a patch, having it tested by a distributed CI > infrastructure and then getting test results. The developer can > access the CI infrastructure and trigger the tests manually > before submitting the patch, thus speeding up the development > process, make short test cycle. It is possible to offer such flexibility in private CI labs by offering an email address where we can send some patches to be tested. However there can be an issue of hardware bandwith/availability to solve. This is the same issue for open/centralized labs or private labs. A test lab accepting any private request can be abused. That's why I think forcing to send the patches publically to the mailing list is a good policy. > Independence: Instead of each vendor providing their own performance results, > having these generated in a centralized lab run by an > independent body will increase confidence that DPDK users have > in the test results. > > There is one example of how this was done for another project. > (https://wiki.fd.io/view/CSIT). > > In their wiki page, you can get the idea about how to configure the servers > and run test cases on these servers. The test report for all releases can be > found on the page. You can also browse the detail test report for each release > if click the link. If click their Jekin's link, you can see the trend of > project status. I do not see an explanation of how to use the CSIT lab on demand (what you described as "Flexibility"). How does it work? > The main disadvantages of a centralized approach are relocating equipment > from separate vendor labs will require a project budget. We can depend on > budget to decide which infrastructure should be deployed in the public test > lab. > > For distributed model, we essentially continue as what we are at present. > Vendors can independently choose the CI tests that they run, and the reports > that they choose to make public. We can add enhancements to Patchwork to > display test results to make tracking easier, and can also look at other ways > to make test reports more visible. Yes, I'm working on it. One year ago, we discussed a CI integration in patchwork: https://lists.ozlabs.org/pipermail/patchwork/2015-July/001363.html It is now implemented in patchwork and available on dpdk.org: http://dpdk.org/ml/archives/dev/2016-September/046282.html A first basic test (checkpatch) is integrated. See this example: http://dpdk.org/patch/16953 The detailed report in test-report mailing list archives is referenced with an hyperlink (in the "Description" column). The next step (work in progress) is to publish some scripts in a new git repository dpdk-ci to help integrating more test labs in patchwork. It basically requires only to receive and send some emails from the lab. Note that any open or centralized lab can be also integrated in patchwork. Note also that this patchwork integration covers only the tests run when a patch is submitted. The tests run regularly on a git tree won't appear in this interface. > The main advantages of a distributed approach are: > There's no requirement for a project budget. The other major advantage is to have a better test coverage. Many companies need to have some internal DPDK tests for their needs. If they use them to provide some public reports, they can avoid having some regressions with their specific use cases or hardware. That's why the distributed CI approach is a win-win. > The disadvantages of a distributed approach are: > We lost the benefits of transparency, independence and the ability to run > tests on demand that are described under the centralized approach above. CI > testing and the publication of the results remains under the control of > vendors (or others who choose to run CI tests). > > Based on the above, we would like to propose that a centralized CI lab. > Details of the required budget and funding for this will obviously need to be > determined, but for now our proposal will focus purely on the technical scope > and benefits. Thanks for the detailed description of the tests that you expect. I think the CI discussion must be thought with two major questions: When? and Where? When: - regularly on a git tree - after each patch submission -> report available via patchwork Where: - in a private lab - in a foundation lab Both private and foundation labs can be more or less open. My conclusion: every kind of tests have some benefits and are welcome!