From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BAE7D42C55 for ; Thu, 8 Jun 2023 03:14:13 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 91A44410D3; Thu, 8 Jun 2023 03:14:13 +0200 (CEST) Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) by mails.dpdk.org (Postfix) with ESMTP id 7384640042 for ; Thu, 8 Jun 2023 03:14:12 +0200 (CEST) Received: by mail-oi1-f177.google.com with SMTP id 5614622812f47-39c77cf32deso383477b6e.0 for ; Wed, 07 Jun 2023 18:14:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1686186851; x=1688778851; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=+IWw4frmGh7J9gy6Y7AyNuxDNggNxn+hf/0Z3T5Bffk=; b=fAMfx6ijDStfU1k7HUHPzLVvsm5R1/IGNlUwn2tVSvZtWScJl4Dzh4MFfPP2FsbFmQ cX6hc14O22QpTXo9cvq2HMjY/7PHDsNnySqMOAUwZAu6gLyAd7CM0/iZ+7804MWCwuOc k7Omdchg5zgJ/uoNdL7OsVu+N4DzNsxeGXBVA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686186851; x=1688778851; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+IWw4frmGh7J9gy6Y7AyNuxDNggNxn+hf/0Z3T5Bffk=; b=Q9Kxt+BtxRhGL6UiEN9vWRM/oj4q29+kokQIxQi9XJZ+YVLqvEOflX2737wsyfvl9j QvvpVXoiZRdMhPSQq1xXftg+jPcv1pg8Ah/gFO5WZSaXlPHt3ooSeCAj0QupS/IUa7Fq DkvofdHGQtMC/WIh11jMtHcgv0Jsq/BRql0oeCFLWrMWRXhK4/LhwF9SllrG4xZUdz1Y ogOTOTYtOLY1soiAd/ozDBkpA51w7djvT6bzgIeU3sdTyohjmL12TNzu9u/U23GX17g1 rqmdR/XGK4zFgHRYDcg0ZPVtqIBDxpMpxZKUmPdKSj/OpTbl090lKcyfhKBj26cgcVo4 Gjgw== X-Gm-Message-State: AC+VfDzbKpqGVEneNRcDbUh5KQFqSCVfBmWtbX75oPdGYsj+IMCzHt6h Us1N68oAmgZiVymFfv1W0AHtQfg5chJlWRSE5uDHGg== X-Google-Smtp-Source: ACHHUZ7ARqMDZaEYvjyKd/6ypXjh/lGiCCiiq3roOwc+u2KNTFDUOVtEdi7EuojNlxw67nbRHjYJdMCPRg9rmcnCsgM= X-Received: by 2002:aca:bc43:0:b0:39c:6024:430a with SMTP id m64-20020acabc43000000b0039c6024430amr376916oif.8.1686186851553; Wed, 07 Jun 2023 18:14:11 -0700 (PDT) MIME-Version: 1.0 References: <3fa6546b-8152-e317-30f0-30d5118b9fc4@amd.com> In-Reply-To: From: Patrick Robb Date: Wed, 7 Jun 2023 21:14:00 -0400 Message-ID: Subject: Re: Email Based Re-Testing Framework To: Aaron Conole Cc: Ferruh Yigit , ci@dpdk.org, "Tu, Lijuan" , zhoumin , Michael Santana , Lincoln Lavoie Content-Type: multipart/alternative; boundary="0000000000004fe98b05fd93f87c" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --0000000000004fe98b05fd93f87c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable > > What would be the purpose of [RETEST UNH-IOL]? > Agreed, this is redundant, provided we will be using the labels/contexts used on patchwork. That seems to be the idea behind Aaron's proposed format and I think we should move to use his format since it previously reached some consensus, and appears to be easy to parse. > We need to specify the patchwork identifier of the patch. > > We could make a script similar to the checkpatch run on dpdk.org: > https://git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh > The easiest way to run it is to make the script as the receiver of the > mail. > If the lab can receive the mails from the mailing list, > then just need to filter the retest requests for its own lab. Yes, I think this is reasonable. I don't think this process is likely to change much, and if we can provide a script to live on the dpdk-ci repo which checks for retest requests, we can reasonably expect the labs to separately set up an environment to handle running that script and triggering their re-tests. Thanks Thomas. On Wed, Jun 7, 2023 at 8:53=E2=80=AFAM Aaron Conole wr= ote: > Patrick Robb writes: > > > Also it can be useful to run daily sub-tree testing by request, if > possible. > > > > That wouldn't be too difficult. I'll make a ticket for this. Although, > for testing on the daily sub-trees, since that's > > UNH-IOL specific, that wouldn't necessarily have to be done via an emai= l > based testing request framework. We > > could also just add a button to our dashboard which triggers a sub-tree > ci run. That would help keep narrow > > the scope of what the email based retesting framework is for. But, both > email or a dashboard button would > > both work. > > We had discussed this long ago - including agreeing on a format, IIRC. > > See the thread starting here: > https://mails.dpdk.org/archives/ci/2021-May/001189.html > > The idea was to have a line like: > > Recheck-request: > > where was the tests in the check labels. In fact, what > started the discussion was a patch for the pw-ci scripts that > implemented part of it. > > I don't see how to make your proposal as easily parsed. > > WDYT? Can you re-read that thread and come up with comments? > > > On Tue, Jun 6, 2023 at 1:53=E2=80=AFPM Ferruh Yigit > wrote: > > > > On 6/6/2023 5:56 PM, Patrick Robb wrote: > > > Hello all, > > > > > > I'd like to revive the conversation about a request from the communi= ty > > > for an email based re-testing framework. The idea is that using one > > > standardized format, dpdk developers could email the test-report > mailing > > > list, requesting a rerun on their patch series for "X" set of tests = at > > > "Y" lab. I think that since patchwork testing labels (ie. > > > iol-broadcom-Performance, github-robot: build, loongarch-compilation= ) > > > are already visible on patch pages on patchwork, those labels are th= e > > > most reasonable ones to expect developers to use when requesting a > > > re-test. We probably wouldn't want to get any more general than that= , > > > like, say, rerunning all CI testing for a specific patch series at a > > > specific lab, since it would result in a significant amount of > "wasted" > > > testing capacity. > > > > > > The standard email format those of us at the Community Lab are > thinking > > > of is like below. Developers would request retests by emailing the > > > test-report mailing list with email bodies like: > > > > > > [RETEST UNH-IOL] > > > iol-abi-testing > > > iol-broadcom-Performance > > > > > > [RETEST Intel] > > > intel-Functional > > > > > > [RETEST Loongson] > > > loongarch-compilation > > > > > > [RETEST GHA] > > > github-robot: build > > > > > > From there, it would be up to the various labs to poll the test-repo= rt > > > mailing list archive (or use a similar method) to check for such > > > requests, and trigger a CI testing rerun based on the labels provide= d > in > > > the re-test email. If there is interest from other labs, UNH might > also > > > be able to host the entire set of re-test requests, allowing other > labs > > > to poll a curated list hosted by UNH. One simple approach would be f= or > > > labs to download all emails sent to test-report and parse with regex > to > > > determine the re-test list for their specific lab. But, if anyone ha= s > > > any better ideas for aggregating the emails to be parsed, suggestion= s > > > are welcome! If this approach sounds reasonable to everyone, we coul= d > > > determine a timeline by which labs would implement the functionality > > > needed to trigger re-tests. Or, we can just add re-testing for vario= us > > > labs if/when they add this functionality - whatever is better. Happy > to > > > discuss at the CI meeting on Thursday. > > > > > > > +1 to re-testing framework. > > > > Also it can be useful to run daily sub-tree testing by request, if > possible. > > --=20 Patrick Robb Technical Service Manager UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 www.iol.unh.edu --0000000000004fe98b05fd93f87c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
What wou= ld be the purpose of [RETEST UNH-IOL]?
Agreed, this is= redundant, provided we will be using the labels/contexts used on patchwork= . That seems to be the idea behind Aaron's proposed format and I think = we should move to use his format since it previously reached some consensus= , and appears to be easy to parse.
=C2=A0
We need to specify the patchwork identifie= r of the patch.

We could make a script similar to the checkpatch run= on=C2=A0= dpdk.org:
https://git.dpdk.org/too= ls/dpdk-ci/tree/tests/checkpatch.sh
The easiest way to run it is to = make the script as the receiver of the mail.
If the lab can receive the = mails from the mailing list,
then just need to filter the retest request= s for its own lab.
Yes, I think this is reasonable. I don&= #39;t think this process is likely to change much, and if we can provide a = script to live on the dpdk-ci repo which checks for retest requests, we can= reasonably expect the labs to separately=C2=A0set up an environment to han= dle running that script and triggering their re-tests. Thanks Thomas.


On Wed, Jun 7, 2023 at 8:53=E2=80=AFAM Aaron Conole <aconole@redhat.com> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex">Patrick Robb <probb@iol.unh.edu> w= rites:

>=C2=A0 Also it can be useful to run daily sub-tree testing by request, = if possible.
>
> That wouldn't be too difficult. I'll make a ticket for this. A= lthough, for testing on the daily sub-trees, since that's
> UNH-IOL specific, that wouldn't necessarily have to be done via an= email based testing request framework. We
> could also just add a button to our dashboard which triggers a sub-tre= e ci run. That would help keep narrow
> the scope of what the email based retesting framework is for. But, bot= h email or a dashboard button would
> both work.

We had discussed this long ago - including agreeing on a format, IIRC.

See the thread starting here:
=C2=A0 https://mails.dpdk.org/archives/ci/202= 1-May/001189.html

The idea was to have a line like:

Recheck-request: <test names>

where <test names> was the tests in the check labels.=C2=A0 In fact, = what
started the discussion was a patch for the pw-ci scripts that
implemented part of it.

I don't see how to make your proposal as easily parsed.

WDYT?=C2=A0 Can you re-read that thread and come up with comments?

> On Tue, Jun 6, 2023 at 1:53=E2=80=AFPM Ferruh Yigit <ferruh.yigit@amd.com> wr= ote:
>
>=C2=A0 On 6/6/2023 5:56 PM, Patrick Robb wrote:
>=C2=A0 > Hello all,
>=C2=A0 >
>=C2=A0 > I'd like to revive the conversation about a request fro= m the community
>=C2=A0 > for an email based re-testing framework. The idea is that u= sing one
>=C2=A0 > standardized format, dpdk developers could email the test-r= eport mailing
>=C2=A0 > list, requesting a rerun on their patch series for "X&= quot; set of tests at
>=C2=A0 > "Y" lab. I think that since patchwork testing lab= els (ie.
>=C2=A0 > iol-broadcom-Performance, github-robot: build, loongarch-co= mpilation)
>=C2=A0 > are already visible on patch pages on patchwork, those labe= ls are the
>=C2=A0 > most reasonable ones to expect developers to use when reque= sting a
>=C2=A0 > re-test. We probably wouldn't want to get any more gene= ral than that,
>=C2=A0 > like, say, rerunning all CI testing for a specific patch se= ries at a
>=C2=A0 > specific lab, since it would result in a significant amount= of "wasted"
>=C2=A0 > testing capacity.
>=C2=A0 >
>=C2=A0 > The standard email format those of us at the Community Lab = are thinking
>=C2=A0 > of is like below. Developers would request retests by email= ing the
>=C2=A0 > test-report mailing list with email bodies like:
>=C2=A0 >
>=C2=A0 > [RETEST UNH-IOL]
>=C2=A0 > iol-abi-testing
>=C2=A0 > iol-broadcom-Performance
>=C2=A0 >
>=C2=A0 > [RETEST Intel]
>=C2=A0 > intel-Functional
>=C2=A0 >
>=C2=A0 > [RETEST Loongson]
>=C2=A0 > loongarch-compilation
>=C2=A0 >
>=C2=A0 > [RETEST GHA]
>=C2=A0 > github-robot: build
>=C2=A0 >
>=C2=A0 > From there, it would be up to the various labs to poll the = test-report
>=C2=A0 > mailing list archive (or use a similar method) to check for= such
>=C2=A0 > requests, and trigger a CI testing rerun based on the label= s provided in
>=C2=A0 > the re-test email. If there is interest from other labs, UN= H might also
>=C2=A0 > be able to host the entire set of re-test requests, allowin= g other labs
>=C2=A0 > to poll a curated list hosted by UNH. One simple approach w= ould be for
>=C2=A0 > labs to download all emails sent to test-report and parse w= ith regex to
>=C2=A0 > determine the re-test list for their specific lab. But, if = anyone has
>=C2=A0 > any better ideas for aggregating the emails to be parsed, s= uggestions
>=C2=A0 > are welcome! If this approach sounds reasonable to everyone= , we could
>=C2=A0 > determine a timeline by which labs would implement the func= tionality
>=C2=A0 > needed to trigger re-tests. Or, we can just add re-testing = for various
>=C2=A0 > labs if/when they add this functionality - whatever is bett= er. Happy to
>=C2=A0 > discuss at the CI meeting on Thursday.
>=C2=A0 >
>
>=C2=A0 +1 to re-testing framework.
>
>=C2=A0 Also it can be useful to run daily sub-tree testing by request, = if possible.



--

Patrick Robb

<= span style=3D"font-size:10pt;font-family:Arial;color:rgb(0,0,0);background-= color:transparent;vertical-align:baseline;white-space:pre-wrap">Technical S= ervice Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu


--0000000000004fe98b05fd93f87c--