From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <ci-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 29F7B4408A;
	Tue, 21 May 2024 18:08:40 +0200 (CEST)
Received: from mails.dpdk.org (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 01A63402E2;
	Tue, 21 May 2024 18:08:40 +0200 (CEST)
Received: from wfout6-smtp.messagingengine.com
 (wfout6-smtp.messagingengine.com [64.147.123.149])
 by mails.dpdk.org (Postfix) with ESMTP id D3CFA4025C
 for <ci@dpdk.org>; Tue, 21 May 2024 18:08:38 +0200 (CEST)
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44])
 by mailfout.west.internal (Postfix) with ESMTP id 86A6E1C00070;
 Tue, 21 May 2024 12:08:35 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute4.internal (MEProxy); Tue, 21 May 2024 12:08:35 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h=
 cc:cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:subject:subject:to:to; s=fm1; t=1716307715;
 x=1716394115; bh=1rpu/YfTVBUdsrVwqHbth3rkMOZpXHxyLJ/hqtMava4=; b=
 VyJLCcdupzhkhRSp30busAjtJf/PiaW3T9iaWWoHNOj7mZLqhPIZS9r7sCTZvWzD
 qH+8X1bAgMhFoSmuCZAxNSxxOcJ70AdDcq2hSG5xLDuWIvDWbl7rUPmysAQUM0I3
 MvTUcVVWETkLS++KxlnjAi0O2kWCB5J2zYlwYAIY/R0KJ6ZC4PL0TDGVONlKjkN0
 z1Yoms7XaldT33J2MNwh+3oerOb0X5COnHbtamI7G9KfwbiGmjz7Bd7AEeqR03Wo
 xqjeElz9WkiV4tcWL9xBqDHp9ztO+caDkg/trxbWO+5k2GubKBuymw7hFbiYIavk
 e06RxPi9DlGD8ZGBtebRCQ==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy
 :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1716307715; x=
 1716394115; bh=1rpu/YfTVBUdsrVwqHbth3rkMOZpXHxyLJ/hqtMava4=; b=j
 AhsoeuuJa7RtBmukYZXhf2P7IUAw+2wSP/0DIdcPtr3Y0tDojLRJ0tP2rK5gXd4F
 df3j7Znykt2vlWwXqbZISj2uHecrO34/QwoZ1463T//NtL6wbfVyNYViEg90Rw4N
 pe3Sj+pOeLBeYeU5QSNVNmzlaYMH/2KDag+in7JnfOXCKcIBQjRQtk48Fb9UusAO
 zmE9oX3XgbylHkPK7o8PzqLwsqJvo7o88F6lGgh9UF8uli4OGK9e6SsOXFIlOPfX
 5RIXbhB6QXiriIrLJhGm6CuC+n0S6tf6yA4mNzWO3n8Er621HDNSWN6niLwWZKjC
 ORNjQP3zuYvv9a5q1D3KA==
X-ME-Sender: <xms:AsdMZhqyz3-81N7IL5QvHTahOe2bvFSzhc0Kagn5jWwnzn_YRuDWUw>
 <xme:AsdMZjpNYAm7Gb323PcnBzv6iNq8UXJVV1exqnBilBJWVI3q-2_2vmR0p6UWHPSDk
 S8zxC0r4LcQOGTi-A>
X-ME-Received: <xmr:AsdMZuNVO_jlvWd79L0FnN6J3FbYPDArwerl8akCm6TsIC612bcRjnRIpt1BZHerKuGM_7q5ShS3lLzUaeM66-aUCA>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeivddgleehucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhephffvvefufffkjghfggfgtgesthfuredttddtjeenucfhrhhomhepvfhhohhm
 rghsucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenuc
 ggtffrrghtthgvrhhnpeejudevheeiveduuddtveffgfdtgeekueevjeffjeegtdeggeek
 gfdvuefgfeekjeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfh
 hrohhmpehthhhomhgrshesmhhonhhjrghlohhnrdhnvght
X-ME-Proxy: <xmx:AsdMZs7vlEybcbvQ3hPohupKrsUr7zaQ781t6nlhVnBo_tywxn-mXw>
 <xmx:AsdMZg7zCF0hIXdsS8BcZJO-70qWEjVDQztJJzST0V1ty1aHIKco2w>
 <xmx:AsdMZkit_FxY0ObXe_gjoXjNp6F8HkRxwieI0wmAxVWOCHL0JpXEMg>
 <xmx:AsdMZi4JQjx7o2DiXJmX-kEtaaODGzv7FsWbXqWMKO91oWCC7vKOTw>
 <xmx:A8dMZqS3ACoGkHSg5m-85nSNklrjEjGVcZ4dnQ-luaxkSctLHBOdf9Br>
Feedback-ID: i47234305:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue,
 21 May 2024 12:08:33 -0400 (EDT)
From: Thomas Monjalon <thomas@monjalon.net>
To: Patrick Robb <probb@iol.unh.edu>
Cc: ci@dpdk.org, ahassick@iol.unh.edu,
 "aaron.conole@redhat.com" <aaron.conole@redhat.com>, alialnu@nvidia.com
Subject: Re: [PATCH 1/1] tools: check for pending test status when parsing
 emails
Date: Tue, 21 May 2024 18:08:32 +0200
Message-ID: <2000540.8hb0ThOEGa@thomas>
In-Reply-To: <CAJvnSUCS53QuadT6zsT4R6SZ_VzhZgjLXm6fzDCT9KdFHKoRJQ@mail.gmail.com>
References: <20240517192222.20555-1-probb@iol.unh.edu>
 <2219864.1BCLMh4Saa@thomas>
 <CAJvnSUCS53QuadT6zsT4R6SZ_VzhZgjLXm6fzDCT9KdFHKoRJQ@mail.gmail.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 7Bit
Content-Type: text/plain; charset="utf-8"
X-BeenThere: ci@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK CI discussions <ci.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/ci>,
 <mailto:ci-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/ci/>
List-Post: <mailto:ci@dpdk.org>
List-Help: <mailto:ci-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/ci>,
 <mailto:ci-request@dpdk.org?subject=subscribe>
Errors-To: ci-bounces@dpdk.org

20/05/2024 23:36, Patrick Robb:
> 2. UNH Lab triggers some testrun pipelines in our CI system (jenkins).
> The first action the pipeline takes is to create in our database a
> test result record for each testrun, setting the status to PENDING. It
> is important to note that one patchwork context, Like
> "iol-compile-amd64-testing," may consist of many individual testruns,
> each for different distros, hardware, environment etc.
> 3. When each testrun completes, it will send a report to Patchwork
> with the new result (pass or fail). When it does this it will update
> the context's results table, changing the environment's result from
> pending to pass/fail. So, when the first report comes in for, say,
> context "iol-compile-amd64-testing," you would see 1 pass/fail, 12
> pending, or similar. Then, as subsequent testruns complete, and report
> their results, the updated table comes with the new report. The
> overall context result (the _Testing {PASS/FAIL/PENDING}_ at the top
> of the test report email) is determined in the manner you might
> expect, i.e. if there is at least one testrun fail result, overall
> context is fail, else if there is at least one pending result, overall
> context is pending, else if all results are passing, overall result is
> passing. As an example, when testing is nearly complete, the top of
> the report email may look like this:
> 
> _Testing PENDING_
> 
> Branch: tags/v22.11
> 
> a409653a123bf105970a25c594711a3cdc44d139 --> testing pass
> 
> Test environment and result as below:
> 
> +------------------------------------+-----------------------------------------------------+
> |            Environment             |       dpdk_meson_compile      |
> +====================================+====================+
> | Ubuntu 20.04 ARM SVE                          | PASS               |
> +------------------------------------+--------------------+
> | Debian 12 with MUSDK                           | PENDING        |
> +------------------------------------+--------------------+
> | Fedora 37 (ARM)                                     | PASS               |
> +------------------------------------+--------------------+
> | Ubuntu 20.04 (ARM)                                | PASS               |
> +------------------------------------+--------------------+
> | Fedora 38 (ARM)                                     | PASS               |
> +------------------------------------+--------------------+
> | Fedora 39 (ARM)                                     | PENDING        |
> +------------------------------------+--------------------+
> | Debian 12 (arm)                                       | PASS               |
> +------------------------------------+--------------------+
> | CentOS Stream 9 (ARM)                         | PASS               |
> +------------------------------------+--------------------+
> | Debian 11 (Buster) (ARM)                        | PASS               |
> +------------------------------------+--------------------+
> | Ubuntu 20.04 ARM GCC Cross Compile | PASS               |
> +------------------------------------+--------------------+

It is quite strange to receive a new email each time a line of the table is updated.

> 4. Eventually, all testruns are complete for a patchwork context, and
> the table switches from pending to pass or fail.
> 
> This does not slow the delivery of results, nor does it increase the
> number of test report emails sent. We still send only 1 email per
> testrun.

I had not realised that so many emails are sent.
I thought it was 1 patchwork context == 1 email.

> This way it is plainly visible to the user when all testing is
> complete, and it also flags for the submitter and for CI people if
> some infra failure prevents a testrun from completing, or from a
> result being properly emailed, etc. The idea is to provide more
> complete status updates and check against infra fails better, but
> without any adverse effects in user experience or load on the email
> server.

I understand it gives a new information: test is pending.