From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f180.google.com (mail-wi0-f180.google.com [209.85.212.180]) by dpdk.org (Postfix) with ESMTP id 5493FC3B6 for ; Fri, 24 Jul 2015 10:44:12 +0200 (CEST) Received: by wibxm9 with SMTP id xm9so17975810wib.1 for ; Fri, 24 Jul 2015 01:44:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:organization :user-agent:mime-version:content-transfer-encoding:content-type; bh=1ZrG9XbCPEtWXD45zIlRnWB8vdgqyOgvHUyYC0fSOVs=; b=Ncl3mYUDXftnbPBBX9v9lvH24BHL2fHB8NN75KFKnp1zD8rRQGU6euWKkP1CWnUnUi NFREQx+Wn02aQTdDXa/TN+z4TLwAFCFQyMLkdibNPzVeIJpzFiDFV2xAZQaN4TzTG1yn tvH2b02zfevn9HmWkGmk04pHYOIbJwQuKUtDTsECUQZmUrmrbFxJTkrW07nnsduiJNyP aKfzFJjh5wxIrqO3Q/WbPi7y4mKtIT5XxtIWrwSuu9cQ/Jg2G9N3rFdw/xmq5JzwxRFG 2nnOUb3FhWYjb0w8nv1AtfajNxo6j/mEuQRuwIPno+0jemUI3FfB4o5LbK/uHsO0k3PZ oWHw== X-Gm-Message-State: ALoCoQmQifm7wDo7wh/WVN1P0Who+NI8aIjlV+Ike3EPIhoWhlq7X6wXhqzY+Ku/bWvOX5SCkSlw X-Received: by 10.180.87.199 with SMTP id ba7mr5212772wib.81.1437727452157; Fri, 24 Jul 2015 01:44:12 -0700 (PDT) Received: from xps13.localnet (136-92-190-109.dsl.ovh.fr. [109.190.92.136]) by smtp.gmail.com with ESMTPSA id ed10sm2388987wic.0.2015.07.24.01.44.10 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 Jul 2015 01:44:11 -0700 (PDT) From: Thomas Monjalon To: waterman.cao@intel.com, chaozhu@linux.vnet.ibm.com, zlu@ezchip.com, bruce.richardson@intel.com, olivier.matz@6wind.com, david.marchand@6wind.com, rahul.lakkireddy@chelsio.com, olgas@mellanox.com, ssujith@cisco.com, yongwang@vmware.com, syuu@cloudius-systems.com Date: Fri, 24 Jul 2015 10:42:53 +0200 Message-ID: <2087826.shqeVCGW43@xps13> Organization: 6WIND User-Agent: KMail/4.14.8 (Linux/4.0.4-2-ARCH; KDE/4.14.8; x86_64; ; ) MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Cc: dev@dpdk.org Subject: [dpdk-dev] Fwd: distributed tests of a patch X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Jul 2015 08:44:12 -0000 Hi, With the help of Thomas Herbert and Stephen Finucane, we started a discussion to have a distributed continuous integration system in patchwork: http://thread.gmane.org/gmane.comp.version-control.patchwork/1242 The goal is to allow many contributors to run their own tests when a patch is received and check all the results in patchwork before accepting it. As DPDK is gaining more and more hardware support, it becomes impossible to test every hardware combinations when making a change. Ideally, it would be nice to have tests run by each hardware vendor. It is the same combination issue to test every OS and every build options. Clearly, the test coverage of the DPDK will never be 100%, but a distributed effort can help a lot to avoid dropping the support of some hardware or OS. ----------------------------------- From: thomas.monjalon@6wind.com To: patchwork@lists.ozlabs.org CC: therbert@redhat.com, stephen.finucane@intel.com Hello, Summary: it would be very nice to take into account some distributed tests in patchwork. Longer text below... Patchwork is incredibly useful to list the patches received in the mailing list of an Open Source project and classify them thanks to the associated status. A patch is accepted after doing some review, reading the discussion, having some specific associated tags (A/R/T column) and testing. This last point may be automated but there is no way of gathering test results from many contributors in a central view which is patchwork. It would be really useful to add a new column to count the automated test results Success, Warnings and Errors (S/W/E) as it is done for manual tags Acked-by/Reviewed-by/Tested-by. As it is done for managing the patches, an XML-RPC API could be used to add a new test report associated to a patch ID. A test report could be also received by email on a mailing list dedicated to test reports. An advantage of sending test reports to a mailing list, is to be able to receive them without the need of patchwork, and eventually do some custom processing. It is also possible to offload the storage of the test reports in the mailing list archives. Last advantage, the mailing list administration allows to whitelist the test contributors. With such a distributed test system, many contributors can run different test suites on their own set of hardware / OS environment for each submitted patch, improving the quality of the impacted project. Below is a workflow giving more details. Let's consider a project having 3 mailing lists: - devel@ to receive the patches and comments. - test@ is a mailing list without archive to send patches to test. - reports@ is a mailing list with archives to store the test reports. Each patch from devel@ is sent to test@ with prefix [patchwork id]. Private tests may be requested by sending patches to test@. If an email is received in test@ without [PATCH] keyword, it is filtered out. Each email sent to test@ is forwarded to each registered test platforms (i.e. testers are members of the test@ mailing list). Each test platform apply the patch (and previous patches of the series) on its local git tree. Each test platform must reply 1 or more reports with exactly the same subject to the original sender or reports@ if List-Id was devel@. A test report may be linked to a test suite doing several tests. The first line begins with Test-Label: followed by a test name. The second line begins with Test-Status: followed by SUCCESS, WARNING or ERROR. The WARNING keyword may be used by tests known to have some false positives or not so important. The detailed report must be inline to allow easy reading from mail archives. When the report is received in reports@, the patchwork id entry is updated to be associated to a new test entry including the test label, status and the URL to the report archives. The new test result is visible in 2 patchwork views. The patch list has a column to display the count of test success, warnings and errors. These counters are a link to the patch view with test section unfold. The detailed patch view has a test section (fold by default) like the headers section. When fold, it shows only the counters. When unfold, it shows one test result per line: the test label is followed by the status being a HTML link to the full report. When refreshing one of these views, it is possible to see the test progress, i.e. how many tests are complete. In this scheme, there is no definitive global test status. The idea is to wait to have a common number of success and no error (removing possible false positives) to consider a patch as good. This judgement is done by the maintainers. In case a test need to be re-run, it is possible to send an email to test@, keeping the patch id and the test label. When the new report will be received, the counters and the detailed view show only the last result of this test. That's the end of this long story. Hope you like it and welcome the idea!