From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 45255A04A4 for ; Tue, 26 May 2020 23:10:15 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 16BBC1D716; Tue, 26 May 2020 23:10:15 +0200 (CEST) Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) by dpdk.org (Postfix) with ESMTP id 735181D681 for ; Tue, 26 May 2020 23:10:13 +0200 (CEST) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id 5AACE1485; Tue, 26 May 2020 17:10:12 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Tue, 26 May 2020 17:10:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s=fm1; bh= ijh2CeONLRwpppbIAAKs0fmEeefdmSXEsM+Oop/ecnE=; b=DpDA3qvBZ2FrWXWU LY7+cTsUb0tfvuoj5Q43YWF6iqQ+uyDguUBWenvmHNuUElrRjS6MIQXdzYpjuSGj hWWPv6Uu8+nrPV1j+1Fl6lItbq2SaZzFxi3uDByrAJNox7PPaN/6+tLLpdBTm8Lt qkk4DFIymUOpIyi+F8t3ENZehO8lbqfMytuyNjhSLIaQ42zaYCSuVbzjuejiCSl4 QfGighGji6KA5L6eDfGVf9VD7BUPW+QINdSmu0s+pGJoQ8s1l71M00cgnKi9Kthl AaK5LxRmklM+vbGp5yhQYY8SRLc/9itvl07SzAEFnH2NfADcvNhdegqLo65uVQ4r E6AF2A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=ijh2CeONLRwpppbIAAKs0fmEeefdmSXEsM+Oop/ec nE=; b=1pyKNcYNybBEsYXhS0CdrnNMQVrHydnhXuoTgGgdB1HKO8V05PRe9bW3n GVieQSVKQtictzfBVmV8eChBc3MiYFxqt/0eli+g/KQI1Z/iSjqs68PXU971itZQ 03PTgJbw1pWESzSfPafYFvN4ccMBSyZwm9DICAalPn4r30WuM7g9Zef2iRHrbVGw GV4VrgtWzUk2u2XVNDOQ0tO08zThN8EiYulO++tqOpr+7zm04dMw8+pT73jf1TnT oJb4Qep+27SIrNOTwqoZt1Qnw3Sp75YfYV1qjc5hNy8uvDmv123FJhk4utcUj8+f MOBXJlh6YRtpgYhMCWYoRcUUQyeLg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduhedruddvvddgudehfecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkjghfggfgtgesthfuredttddtvdenucfhrhhomhepvfhhohhm rghsucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenuc ggtffrrghtthgvrhhnpeffvdffjeeuteelfeeileduudeugfetjeelveefkeejfeeigeeh teffvdekfeegudenucffohhmrghinhepughpughkrdhorhhgnecukfhppeejjedrudefge drvddtfedrudekgeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhl fhhrohhmpehthhhomhgrshesmhhonhhjrghlohhnrdhnvght X-ME-Proxy: Received: from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id 3B4F13280059; Tue, 26 May 2020 17:10:11 -0400 (EDT) From: Thomas Monjalon To: Lincoln Lavoie Cc: ci@dpdk.org, James Hendergart Date: Tue, 26 May 2020 23:10:10 +0200 Message-ID: <3163283.oOSAsHYAT8@thomas> In-Reply-To: References: <5232496.1u8oYCttyy@thomas> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Subject: Re: [dpdk-ci] CI reliability X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" 26/05/2020 22:27, Lincoln Lavoie: > On Sun, May 24, 2020 at 5:50 AM Thomas Monjalon wrote: > > > Hi all, > > > > I think we have a CI reliability issue in general. > > Perhaps we lack some alert mechanism warning test platform maintainers > > when too many tests are failing. > > > > Recent example: the community lab compilation test is failing on > > Fedora 31 for at least 2 weeks, and I don't see any action to fix it: > > https://lab.dpdk.org/results/dashboard/patchsets/11040/ > > > > Because of such recurring errors, the whole CI becomes irrelevant. > > This has been fixed as of yesterday. The failure was caused by a commit to > the SPDK repos in how they pull in their dependencies, which was done in a > way that is not compatible with docker. The team created a work around so > that case is fixed, but there is always a risk where other commits for > those type of items could cause a failure in the containers. Thanks for fixing > I asked Brandon to change the scripts that run the testing in the > containers to try and catch failures from docker separately, so they can be > flagged as infrastructure, compared to failures of the build. Yes good idea. When compiling external projects, we can see some errors which are not due to the DPDK patch. I guess we validate any upgrade of the external projects before making them live? > I'm also very surprised, this was not raised during the CI meeting, or by > anyone else. I'm wondering if this is caused by the actual error logs > being a little abstracted from the emails, i.e. they are a link and a zip > file away for the actual email text, so maybe folks are not really looking > into the output as closely as they should be. Is this something we can > make better by including more detail in the email text, so issues are > caught more quickly? I think the table in the report is already quite expressive. As I proposed above, I think we need a better monitoring. If the same test is failing on many DPDK patches, it should raise an alarm.