From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1BCD0440AC; Thu, 23 May 2024 23:48:12 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id ED371402B3; Thu, 23 May 2024 23:48:11 +0200 (CEST) Received: from mail-oo1-f45.google.com (mail-oo1-f45.google.com [209.85.161.45]) by mails.dpdk.org (Postfix) with ESMTP id 0040A402A8 for ; Thu, 23 May 2024 23:48:10 +0200 (CEST) Received: by mail-oo1-f45.google.com with SMTP id 006d021491bc7-5b27bbcb5f0so3708128eaf.3 for ; Thu, 23 May 2024 14:48:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1716500890; x=1717105690; darn=dpdk.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=OPuHKVsOvY2cvxM67y5XgT1fVmZ5ysL/nHx0vjrmR5w=; b=PGjxSBXKYwVFQo4l3lUbgVbyjaxjTt7Xx51qqe4MMW7aA4vgE+oTHpp9zOiVBl0MhZ Kk9fAskqtstCBtJP4QlDS0cqFoNqenM2m25lXgppOn8FXRkKb9l7xuzfu/dFMoPRhcIu aBbMT3t+S0WUOfbsjkul8/SDvZLAMn46txJJw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716500890; x=1717105690; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OPuHKVsOvY2cvxM67y5XgT1fVmZ5ysL/nHx0vjrmR5w=; b=Ef6+u1CyAmOwtmUK6vgQN/F/77PDauppuabp3igulgxA5hrxRk10DIzYENR3D74wAH K2XuXyYckMJeDqzmMoD/qpHdLFShUxNyUFH5zq3s5Re9GhlfyywZ8LMpC8QN7DZ58bJL ia61RK66jpgilM9XxGqk7biYHtlMwEbOfPGKZdMBevRmMVdeCYtCpMNE0Nmqr5ANV7X/ hW6cA4nBbE6C3jYOfOPIGMjBYNzbQ6TK5v8Xrq1aP1z2vIVkgAhjty0BpPGX4jl6hoZK u7rYvj6BYn2vAEtMjZSqLg0LUsY5mNjps+n52owpZwKURUPMvaRBqM8oBJFK1KT37VBr v3Gw== X-Gm-Message-State: AOJu0YxN9Rm2GQZ8ZJEnukCKoT5xdc5TqWjv6BO/uAB4r5wXs7AboVQE 63ZF/TuFYshzPaqcr8ZBKVKpxj36YAoRStFPBrrHBwfj98gMAt6zPiUjTxRueU0/92rbpva54Ye Xt9EZzxcyHOPuOB8bKTTcWPzLI3Rtjzi1vM0Slw== X-Google-Smtp-Source: AGHT+IHnTy47AR4iI9ogxYbdyrCJ2te0M40a+FsvTK7LjJU54t1mxWsabM8KFad6Q+pytVWyrsQ/Dl1Hj8bU6eAbcWM= X-Received: by 2002:a05:6820:80a:b0:5af:36ef:27d0 with SMTP id 006d021491bc7-5b95fd9c650mr836122eaf.1.1716500890156; Thu, 23 May 2024 14:48:10 -0700 (PDT) MIME-Version: 1.0 References: <20240517192222.20555-1-probb@iol.unh.edu> <2219864.1BCLMh4Saa@thomas> <2000540.8hb0ThOEGa@thomas> In-Reply-To: <2000540.8hb0ThOEGa@thomas> From: Patrick Robb Date: Thu, 23 May 2024 17:47:59 -0400 Message-ID: Subject: Re: [PATCH 1/1] tools: check for pending test status when parsing emails To: Thomas Monjalon Cc: ci@dpdk.org, ahassick@iol.unh.edu, "aaron.conole@redhat.com" , alialnu@nvidia.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org On Tue, May 21, 2024 at 12:08=E2=80=AFPM Thomas Monjalon wrote: > > 20/05/2024 23:36, Patrick Robb: > > 2. UNH Lab triggers some testrun pipelines in our CI system (jenkins). > > The first action the pipeline takes is to create in our database a > > test result record for each testrun, setting the status to PENDING. It > > is important to note that one patchwork context, Like > > "iol-compile-amd64-testing," may consist of many individual testruns, > > each for different distros, hardware, environment etc. > > 3. When each testrun completes, it will send a report to Patchwork > > with the new result (pass or fail). When it does this it will update > > the context's results table, changing the environment's result from > > pending to pass/fail. So, when the first report comes in for, say, > > context "iol-compile-amd64-testing," you would see 1 pass/fail, 12 > > pending, or similar. Then, as subsequent testruns complete, and report > > their results, the updated table comes with the new report. The > > overall context result (the _Testing {PASS/FAIL/PENDING}_ at the top > > of the test report email) is determined in the manner you might > > expect, i.e. if there is at least one testrun fail result, overall > > context is fail, else if there is at least one pending result, overall > > context is pending, else if all results are passing, overall result is > > passing. As an example, when testing is nearly complete, the top of > > the report email may look like this: > > > > _Testing PENDING_ > > > > Branch: tags/v22.11 > > > > a409653a123bf105970a25c594711a3cdc44d139 --> testing pass > > > > Test environment and result as below: > > > > +------------------------------------+---------------------------------= --------------------+ > > | Environment | dpdk_meson_compile | > > +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D+ > > | Ubuntu 20.04 ARM SVE | PASS | > > +------------------------------------+--------------------+ > > | Debian 12 with MUSDK | PENDING | > > +------------------------------------+--------------------+ > > | Fedora 37 (ARM) | PASS = | > > +------------------------------------+--------------------+ > > | Ubuntu 20.04 (ARM) | PASS = | > > +------------------------------------+--------------------+ > > | Fedora 38 (ARM) | PASS = | > > +------------------------------------+--------------------+ > > | Fedora 39 (ARM) | PENDING = | > > +------------------------------------+--------------------+ > > | Debian 12 (arm) | PASS = | > > +------------------------------------+--------------------+ > > | CentOS Stream 9 (ARM) | PASS | > > +------------------------------------+--------------------+ > > | Debian 11 (Buster) (ARM) | PASS = | > > +------------------------------------+--------------------+ > > | Ubuntu 20.04 ARM GCC Cross Compile | PASS | > > +------------------------------------+--------------------+ > > It is quite strange to receive a new email each time a line of the table = is updated. > > > 4. Eventually, all testruns are complete for a patchwork context, and > > the table switches from pending to pass or fail. > > > > This does not slow the delivery of results, nor does it increase the > > number of test report emails sent. We still send only 1 email per > > testrun. > > I had not realised that so many emails are sent. > I thought it was 1 patchwork context =3D=3D 1 email. This is how it worked until last year, but our test results delivery was slow in some cases. So, we implemented "tail reporting" i.e. the ability for an environment to report its own test run when it finishes testing, as opposed to relying on a testrun aggregator which runs later. If a test run fails, we want to share that information as soon as we can, not block it on other testing.