From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3590F4408E; Tue, 21 May 2024 19:23:18 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1072140695; Tue, 21 May 2024 19:23:18 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 0F727402F0 for ; Tue, 21 May 2024 19:23:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1716312196; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Og8cjqJh8PxmpKXtvJ25shcJCObeNeQcM7dDnePL7Gs=; b=gQmItW5XfeNo3lUzC8hiQrl/jlS3nD7pnm5qewGoNN8OdN1jd7pjJG4OvhPUMMmd42nUP6 CgPuM5izDNOCEnDyRB+SXp2Lz03FvJ7dx1pdsVr1oCFGbQtvvxGl04ONcTuL2dD7h4nwwB 4CnOtq2LcjfIUjSV03PHYlRNNs+B1Ek= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-355-Vqi9bpOaMz2NBoljlnhEAQ-1; Tue, 21 May 2024 13:23:12 -0400 X-MC-Unique: Vqi9bpOaMz2NBoljlnhEAQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6DDEC380673B; Tue, 21 May 2024 17:23:12 +0000 (UTC) Received: from RHTRH0061144 (dhcp-17-72.bos.redhat.com [10.18.17.72]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 16045200A78F; Tue, 21 May 2024 17:23:12 +0000 (UTC) From: Aaron Conole To: Patrick Robb Cc: Thomas Monjalon , ci@dpdk.org, ahassick@iol.unh.edu, alialnu@nvidia.com Subject: Re: [PATCH 1/1] tools: check for pending test status when parsing emails In-Reply-To: (Patrick Robb's message of "Mon, 20 May 2024 17:36:33 -0400") References: <20240517192222.20555-1-probb@iol.unh.edu> <20240517192222.20555-2-probb@iol.unh.edu> <2219864.1BCLMh4Saa@thomas> User-Agent: Gnus/5.13 (Gnus v5.13) Date: Tue, 21 May 2024 13:23:11 -0400 Message-ID: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.6 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Patrick Robb writes: > On Mon, May 20, 2024 at 3:03=E2=80=AFPM Thomas Monjalon wrote: >> >> 17/05/2024 21:22, Patrick Robb: >> > Signed-off-by: Patrick Robb >> >> Please could you explain what it is doing? >> Having a workflow understanding would be nice. +1. It would usually be used in the commit message to show the rationale for the change. Here's a suggestion for an even shorter distillation from your message below: Today, the community CI infrastructure only uses post-result reporting, such as "SUCCESS", "FAILED", and "WARNING". These results get reported only after a test finishes. This creates some confusion about whether a test might have been started for the series in question. It isn't easy to tell at-a-glance which tests are currently running for a given patch or series. This patch aims to introduce support for a "PENDING" state in the CI infrastructure. This allows labs to indicate which tests have started and are awaiting results. That means test writers should now send a "PENDING" status for tests as they start, and then update with a post-test result after. With this change, understanding which tests ran at-a-glance is something we can achieve. This change should have no affect on the actual tests being run. Maybe a v2 with something like this as the commit message? WDYT? > Yes good idea. > > For context, pending is already a supported check state in patchwork > server: https://patchwork.readthedocs.io/en/latest/usage/overview/#checks > > 1. DPDK patch is submitted. Patch is acquired by UNH Lab. > 2. UNH Lab triggers some testrun pipelines in our CI system (jenkins). > The first action the pipeline takes is to create in our database a > test result record for each testrun, setting the status to PENDING. It > is important to note that one patchwork context, Like > "iol-compile-amd64-testing," may consist of many individual testruns, > each for different distros, hardware, environment etc. > 3. When each testrun completes, it will send a report to Patchwork > with the new result (pass or fail). When it does this it will update > the context's results table, changing the environment's result from > pending to pass/fail. So, when the first report comes in for, say, > context "iol-compile-amd64-testing," you would see 1 pass/fail, 12 > pending, or similar. Then, as subsequent testruns complete, and report > their results, the updated table comes with the new report. The > overall context result (the _Testing {PASS/FAIL/PENDING}_ at the top > of the test report email) is determined in the manner you might > expect, i.e. if there is at least one testrun fail result, overall > context is fail, else if there is at least one pending result, overall > context is pending, else if all results are passing, overall result is > passing. As an example, when testing is nearly complete, the top of > the report email may look like this: > > _Testing PENDING_ > > Branch: tags/v22.11 > > a409653a123bf105970a25c594711a3cdc44d139 --> testing pass > > Test environment and result as below: > > +------------------------------------+-----------------------------------= ------------------+ > | Environment | dpdk_meson_compile | > +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D+ > | Ubuntu 20.04 ARM SVE | PASS | > +------------------------------------+--------------------+ > | Debian 12 with MUSDK | PENDING | > +------------------------------------+--------------------+ > | Fedora 37 (ARM) | PASS = | > +------------------------------------+--------------------+ > | Ubuntu 20.04 (ARM) | PASS = | > +------------------------------------+--------------------+ > | Fedora 38 (ARM) | PASS = | > +------------------------------------+--------------------+ > | Fedora 39 (ARM) | PENDING | > +------------------------------------+--------------------+ > | Debian 12 (arm) | PASS = | > +------------------------------------+--------------------+ > | CentOS Stream 9 (ARM) | PASS | > +------------------------------------+--------------------+ > | Debian 11 (Buster) (ARM) | PASS | > +------------------------------------+--------------------+ > | Ubuntu 20.04 ARM GCC Cross Compile | PASS | > +------------------------------------+--------------------+ > > > 4. Eventually, all testruns are complete for a patchwork context, and > the table switches from pending to pass or fail. > > This does not slow the delivery of results, nor does it increase the > number of test report emails sent. We still send only 1 email per > testrun. > > This way it is plainly visible to the user when all testing is > complete, and it also flags for the submitter and for CI people if > some infra failure prevents a testrun from completing, or from a > result being properly emailed, etc. The idea is to provide more > complete status updates and check against infra fails better, but > without any adverse effects in user experience or load on the email > server. > >> >> > --- a/tools/update-pw.sh >> > +++ b/tools/update-pw.sh >> > @@ -49,6 +49,7 @@ case $status in >> > 'SUCCESS') pwstatus=3D'success' ;; >> > 'WARNING') pwstatus=3D'warning' ;; >> > 'FAILURE') pwstatus=3D'fail' ;; >> > + 'PENDING') pwstatus=3D'pending' ;; >> >> >>