From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 29F7B4408A; Tue, 21 May 2024 18:08:40 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 01A63402E2; Tue, 21 May 2024 18:08:40 +0200 (CEST) Received: from wfout6-smtp.messagingengine.com (wfout6-smtp.messagingengine.com [64.147.123.149]) by mails.dpdk.org (Postfix) with ESMTP id D3CFA4025C for ; Tue, 21 May 2024 18:08:38 +0200 (CEST) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailfout.west.internal (Postfix) with ESMTP id 86A6E1C00070; Tue, 21 May 2024 12:08:35 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Tue, 21 May 2024 12:08:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm1; t=1716307715; x=1716394115; bh=1rpu/YfTVBUdsrVwqHbth3rkMOZpXHxyLJ/hqtMava4=; b= VyJLCcdupzhkhRSp30busAjtJf/PiaW3T9iaWWoHNOj7mZLqhPIZS9r7sCTZvWzD qH+8X1bAgMhFoSmuCZAxNSxxOcJ70AdDcq2hSG5xLDuWIvDWbl7rUPmysAQUM0I3 MvTUcVVWETkLS++KxlnjAi0O2kWCB5J2zYlwYAIY/R0KJ6ZC4PL0TDGVONlKjkN0 z1Yoms7XaldT33J2MNwh+3oerOb0X5COnHbtamI7G9KfwbiGmjz7Bd7AEeqR03Wo xqjeElz9WkiV4tcWL9xBqDHp9ztO+caDkg/trxbWO+5k2GubKBuymw7hFbiYIavk e06RxPi9DlGD8ZGBtebRCQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1716307715; x= 1716394115; bh=1rpu/YfTVBUdsrVwqHbth3rkMOZpXHxyLJ/hqtMava4=; b=j AhsoeuuJa7RtBmukYZXhf2P7IUAw+2wSP/0DIdcPtr3Y0tDojLRJ0tP2rK5gXd4F df3j7Znykt2vlWwXqbZISj2uHecrO34/QwoZ1463T//NtL6wbfVyNYViEg90Rw4N pe3Sj+pOeLBeYeU5QSNVNmzlaYMH/2KDag+in7JnfOXCKcIBQjRQtk48Fb9UusAO zmE9oX3XgbylHkPK7o8PzqLwsqJvo7o88F6lGgh9UF8uli4OGK9e6SsOXFIlOPfX 5RIXbhB6QXiriIrLJhGm6CuC+n0S6tf6yA4mNzWO3n8Er621HDNSWN6niLwWZKjC ORNjQP3zuYvv9a5q1D3KA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeivddgleehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkjghfggfgtgesthfuredttddtjeenucfhrhhomhepvfhhohhm rghsucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenuc ggtffrrghtthgvrhhnpeejudevheeiveduuddtveffgfdtgeekueevjeffjeegtdeggeek gfdvuefgfeekjeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfh hrohhmpehthhhomhgrshesmhhonhhjrghlohhnrdhnvght X-ME-Proxy: Feedback-ID: i47234305:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 21 May 2024 12:08:33 -0400 (EDT) From: Thomas Monjalon To: Patrick Robb Cc: ci@dpdk.org, ahassick@iol.unh.edu, "aaron.conole@redhat.com" , alialnu@nvidia.com Subject: Re: [PATCH 1/1] tools: check for pending test status when parsing emails Date: Tue, 21 May 2024 18:08:32 +0200 Message-ID: <2000540.8hb0ThOEGa@thomas> In-Reply-To: References: <20240517192222.20555-1-probb@iol.unh.edu> <2219864.1BCLMh4Saa@thomas> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="utf-8" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org 20/05/2024 23:36, Patrick Robb: > 2. UNH Lab triggers some testrun pipelines in our CI system (jenkins). > The first action the pipeline takes is to create in our database a > test result record for each testrun, setting the status to PENDING. It > is important to note that one patchwork context, Like > "iol-compile-amd64-testing," may consist of many individual testruns, > each for different distros, hardware, environment etc. > 3. When each testrun completes, it will send a report to Patchwork > with the new result (pass or fail). When it does this it will update > the context's results table, changing the environment's result from > pending to pass/fail. So, when the first report comes in for, say, > context "iol-compile-amd64-testing," you would see 1 pass/fail, 12 > pending, or similar. Then, as subsequent testruns complete, and report > their results, the updated table comes with the new report. The > overall context result (the _Testing {PASS/FAIL/PENDING}_ at the top > of the test report email) is determined in the manner you might > expect, i.e. if there is at least one testrun fail result, overall > context is fail, else if there is at least one pending result, overall > context is pending, else if all results are passing, overall result is > passing. As an example, when testing is nearly complete, the top of > the report email may look like this: > > _Testing PENDING_ > > Branch: tags/v22.11 > > a409653a123bf105970a25c594711a3cdc44d139 --> testing pass > > Test environment and result as below: > > +------------------------------------+-----------------------------------------------------+ > | Environment | dpdk_meson_compile | > +====================================+====================+ > | Ubuntu 20.04 ARM SVE | PASS | > +------------------------------------+--------------------+ > | Debian 12 with MUSDK | PENDING | > +------------------------------------+--------------------+ > | Fedora 37 (ARM) | PASS | > +------------------------------------+--------------------+ > | Ubuntu 20.04 (ARM) | PASS | > +------------------------------------+--------------------+ > | Fedora 38 (ARM) | PASS | > +------------------------------------+--------------------+ > | Fedora 39 (ARM) | PENDING | > +------------------------------------+--------------------+ > | Debian 12 (arm) | PASS | > +------------------------------------+--------------------+ > | CentOS Stream 9 (ARM) | PASS | > +------------------------------------+--------------------+ > | Debian 11 (Buster) (ARM) | PASS | > +------------------------------------+--------------------+ > | Ubuntu 20.04 ARM GCC Cross Compile | PASS | > +------------------------------------+--------------------+ It is quite strange to receive a new email each time a line of the table is updated. > 4. Eventually, all testruns are complete for a patchwork context, and > the table switches from pending to pass or fail. > > This does not slow the delivery of results, nor does it increase the > number of test report emails sent. We still send only 1 email per > testrun. I had not realised that so many emails are sent. I thought it was 1 patchwork context == 1 email. > This way it is plainly visible to the user when all testing is > complete, and it also flags for the submitter and for CI people if > some infra failure prevents a testrun from completing, or from a > result being properly emailed, etc. The idea is to provide more > complete status updates and check against infra fails better, but > without any adverse effects in user experience or load on the email > server. I understand it gives a new information: test is pending.