From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 83A704407C; Mon, 20 May 2024 23:36:46 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7CC14402F2; Mon, 20 May 2024 23:36:46 +0200 (CEST) Received: from mail-oo1-f45.google.com (mail-oo1-f45.google.com [209.85.161.45]) by mails.dpdk.org (Postfix) with ESMTP id 12DC1402E9 for ; Mon, 20 May 2024 23:36:45 +0200 (CEST) Received: by mail-oo1-f45.google.com with SMTP id 006d021491bc7-5b31f2c6e52so1234449eaf.2 for ; Mon, 20 May 2024 14:36:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1716241004; x=1716845804; darn=dpdk.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=T3vhj7zv1/rTjUaXWv1EzW5NqZvbAU9celSJCP+EC6k=; b=a5bzw0YZSUazJR4BBi9W27qWlrRZBu/8PXePfIVyZ5M+VSGCqA+6a6C9XqVCAk2Bmj ZdJaKu+2aDU8lV8VM1H8qRXGTX2THCyu6ymqfLlTYR9c9JUd/bTWZ/RrsFfL8q69zdq3 GJa8+XypeSjhVPwC7b/HdnKoVEBXWTY3kMcHw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716241004; x=1716845804; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T3vhj7zv1/rTjUaXWv1EzW5NqZvbAU9celSJCP+EC6k=; b=F2GjCYdgsJcqsKnkiwhjkbCJ9ZCiNzPhO8Ce75wQVnH2t+zlnxpL4PAG8D1HtxYxUP ck+Ushl3NCqMEQ03xQnest6iACj4NesLuyGrsz/vdFEOChLfTlRZaEnpYeikykv7hetX JrYxHMqlVkXbNJday4A9AM9j6Xi6rfvSLGgeDy+HtE1nPHdcvJl9Tb2inKKkLpNr8XCm OF8rz2zTVJhN6O0T/XNnuYZmPqFIcq1/BFtW7S2dSHgR4nItbvHo5CRSAPXjyhb3f5Zm ENM/kOnwGFw4rNLtfwvBxKqlhxET8DhIxAYCcVFGz3qbtpOCWPs8dzcbendcOuKv0g3k MQZg== X-Gm-Message-State: AOJu0YwnsMKdr3m8q9A1gPcrVsLS6K79e+FRTdSj2dpp5FDPZIXs+Fme QsGLMvxGsYMNCG7YZjz3P6sAKaBetRTlrSm6fmczX5OADG4aUaRiHnJEMartqXW5ndQw1fUbwvD KLqqmFhS7bsbjlCf3J/p9tgeoxOrGRniJaGEaVA== X-Google-Smtp-Source: AGHT+IFYBnD7d+aGSzcdCGX5lonqOWR1GXv9hD+DsTrMVClQi5ptBn7+Wq8JXtBPr81A4Nz5sJHz4Lq0PKAi/MwFzc8= X-Received: by 2002:a05:6820:161f:b0:5b2:2b8:e419 with SMTP id 006d021491bc7-5b365c02154mr6970320eaf.0.1716241004135; Mon, 20 May 2024 14:36:44 -0700 (PDT) MIME-Version: 1.0 References: <20240517192222.20555-1-probb@iol.unh.edu> <20240517192222.20555-2-probb@iol.unh.edu> <2219864.1BCLMh4Saa@thomas> In-Reply-To: <2219864.1BCLMh4Saa@thomas> From: Patrick Robb Date: Mon, 20 May 2024 17:36:33 -0400 Message-ID: Subject: Re: [PATCH 1/1] tools: check for pending test status when parsing emails To: Thomas Monjalon Cc: ci@dpdk.org, ahassick@iol.unh.edu, "aaron.conole@redhat.com" , alialnu@nvidia.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org On Mon, May 20, 2024 at 3:03=E2=80=AFPM Thomas Monjalon wrote: > > 17/05/2024 21:22, Patrick Robb: > > Signed-off-by: Patrick Robb > > Please could you explain what it is doing? > Having a workflow understanding would be nice. Yes good idea. For context, pending is already a supported check state in patchwork server: https://patchwork.readthedocs.io/en/latest/usage/overview/#checks 1. DPDK patch is submitted. Patch is acquired by UNH Lab. 2. UNH Lab triggers some testrun pipelines in our CI system (jenkins). The first action the pipeline takes is to create in our database a test result record for each testrun, setting the status to PENDING. It is important to note that one patchwork context, Like "iol-compile-amd64-testing," may consist of many individual testruns, each for different distros, hardware, environment etc. 3. When each testrun completes, it will send a report to Patchwork with the new result (pass or fail). When it does this it will update the context's results table, changing the environment's result from pending to pass/fail. So, when the first report comes in for, say, context "iol-compile-amd64-testing," you would see 1 pass/fail, 12 pending, or similar. Then, as subsequent testruns complete, and report their results, the updated table comes with the new report. The overall context result (the _Testing {PASS/FAIL/PENDING}_ at the top of the test report email) is determined in the manner you might expect, i.e. if there is at least one testrun fail result, overall context is fail, else if there is at least one pending result, overall context is pending, else if all results are passing, overall result is passing. As an example, when testing is nearly complete, the top of the report email may look like this: _Testing PENDING_ Branch: tags/v22.11 a409653a123bf105970a25c594711a3cdc44d139 --> testing pass Test environment and result as below: +------------------------------------+-------------------------------------= ----------------+ | Environment | dpdk_meson_compile | +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D+ | Ubuntu 20.04 ARM SVE | PASS | +------------------------------------+--------------------+ | Debian 12 with MUSDK | PENDING | +------------------------------------+--------------------+ | Fedora 37 (ARM) | PASS = | +------------------------------------+--------------------+ | Ubuntu 20.04 (ARM) | PASS | +------------------------------------+--------------------+ | Fedora 38 (ARM) | PASS = | +------------------------------------+--------------------+ | Fedora 39 (ARM) | PENDING | +------------------------------------+--------------------+ | Debian 12 (arm) | PASS = | +------------------------------------+--------------------+ | CentOS Stream 9 (ARM) | PASS | +------------------------------------+--------------------+ | Debian 11 (Buster) (ARM) | PASS | +------------------------------------+--------------------+ | Ubuntu 20.04 ARM GCC Cross Compile | PASS | +------------------------------------+--------------------+ 4. Eventually, all testruns are complete for a patchwork context, and the table switches from pending to pass or fail. This does not slow the delivery of results, nor does it increase the number of test report emails sent. We still send only 1 email per testrun. This way it is plainly visible to the user when all testing is complete, and it also flags for the submitter and for CI people if some infra failure prevents a testrun from completing, or from a result being properly emailed, etc. The idea is to provide more complete status updates and check against infra fails better, but without any adverse effects in user experience or load on the email server. > > > --- a/tools/update-pw.sh > > +++ b/tools/update-pw.sh > > @@ -49,6 +49,7 @@ case $status in > > 'SUCCESS') pwstatus=3D'success' ;; > > 'WARNING') pwstatus=3D'warning' ;; > > 'FAILURE') pwstatus=3D'fail' ;; > > + 'PENDING') pwstatus=3D'pending' ;; > > >