From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <ci-bounces@dpdk.org> Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BAE7D42C55 for <public@inbox.dpdk.org>; Thu, 8 Jun 2023 03:14:13 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 91A44410D3; Thu, 8 Jun 2023 03:14:13 +0200 (CEST) Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) by mails.dpdk.org (Postfix) with ESMTP id 7384640042 for <ci@dpdk.org>; Thu, 8 Jun 2023 03:14:12 +0200 (CEST) Received: by mail-oi1-f177.google.com with SMTP id 5614622812f47-39c77cf32deso383477b6e.0 for <ci@dpdk.org>; Wed, 07 Jun 2023 18:14:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1686186851; x=1688778851; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=+IWw4frmGh7J9gy6Y7AyNuxDNggNxn+hf/0Z3T5Bffk=; b=fAMfx6ijDStfU1k7HUHPzLVvsm5R1/IGNlUwn2tVSvZtWScJl4Dzh4MFfPP2FsbFmQ cX6hc14O22QpTXo9cvq2HMjY/7PHDsNnySqMOAUwZAu6gLyAd7CM0/iZ+7804MWCwuOc k7Omdchg5zgJ/uoNdL7OsVu+N4DzNsxeGXBVA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686186851; x=1688778851; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+IWw4frmGh7J9gy6Y7AyNuxDNggNxn+hf/0Z3T5Bffk=; b=Q9Kxt+BtxRhGL6UiEN9vWRM/oj4q29+kokQIxQi9XJZ+YVLqvEOflX2737wsyfvl9j QvvpVXoiZRdMhPSQq1xXftg+jPcv1pg8Ah/gFO5WZSaXlPHt3ooSeCAj0QupS/IUa7Fq DkvofdHGQtMC/WIh11jMtHcgv0Jsq/BRql0oeCFLWrMWRXhK4/LhwF9SllrG4xZUdz1Y ogOTOTYtOLY1soiAd/ozDBkpA51w7djvT6bzgIeU3sdTyohjmL12TNzu9u/U23GX17g1 rqmdR/XGK4zFgHRYDcg0ZPVtqIBDxpMpxZKUmPdKSj/OpTbl090lKcyfhKBj26cgcVo4 Gjgw== X-Gm-Message-State: AC+VfDzbKpqGVEneNRcDbUh5KQFqSCVfBmWtbX75oPdGYsj+IMCzHt6h Us1N68oAmgZiVymFfv1W0AHtQfg5chJlWRSE5uDHGg== X-Google-Smtp-Source: ACHHUZ7ARqMDZaEYvjyKd/6ypXjh/lGiCCiiq3roOwc+u2KNTFDUOVtEdi7EuojNlxw67nbRHjYJdMCPRg9rmcnCsgM= X-Received: by 2002:aca:bc43:0:b0:39c:6024:430a with SMTP id m64-20020acabc43000000b0039c6024430amr376916oif.8.1686186851553; Wed, 07 Jun 2023 18:14:11 -0700 (PDT) MIME-Version: 1.0 References: <CAJvnSUBc79Y+yA1gQRsifjH7BPYQKGxNaXr43x-HmFnPrQcOag@mail.gmail.com> <3fa6546b-8152-e317-30f0-30d5118b9fc4@amd.com> <CAJvnSUAbhwRqSw5jHRuP4Dwpa5eC5wCKMP+kFfCJE5NqePGVCw@mail.gmail.com> <f7tbkhr4enp.fsf@redhat.com> In-Reply-To: <f7tbkhr4enp.fsf@redhat.com> From: Patrick Robb <probb@iol.unh.edu> Date: Wed, 7 Jun 2023 21:14:00 -0400 Message-ID: <CAJvnSUA95NVd+o13dP5BAxy2r0-RztVGA3murWthMYXpSTW0oQ@mail.gmail.com> Subject: Re: Email Based Re-Testing Framework To: Aaron Conole <aconole@redhat.com> Cc: Ferruh Yigit <ferruh.yigit@amd.com>, ci@dpdk.org, "Tu, Lijuan" <lijuan.tu@intel.com>, zhoumin <zhoumin@loongson.cn>, Michael Santana <maicolgabriel@hotmail.com>, Lincoln Lavoie <lylavoie@iol.unh.edu> Content-Type: multipart/alternative; boundary="0000000000004fe98b05fd93f87c" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions <ci.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/ci>, <mailto:ci-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/ci/> List-Post: <mailto:ci@dpdk.org> List-Help: <mailto:ci-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/ci>, <mailto:ci-request@dpdk.org?subject=subscribe> Errors-To: ci-bounces@dpdk.org --0000000000004fe98b05fd93f87c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable > > What would be the purpose of [RETEST UNH-IOL]? > Agreed, this is redundant, provided we will be using the labels/contexts used on patchwork. That seems to be the idea behind Aaron's proposed format and I think we should move to use his format since it previously reached some consensus, and appears to be easy to parse. > We need to specify the patchwork identifier of the patch. > > We could make a script similar to the checkpatch run on dpdk.org: > https://git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh > The easiest way to run it is to make the script as the receiver of the > mail. > If the lab can receive the mails from the mailing list, > then just need to filter the retest requests for its own lab. Yes, I think this is reasonable. I don't think this process is likely to change much, and if we can provide a script to live on the dpdk-ci repo which checks for retest requests, we can reasonably expect the labs to separately set up an environment to handle running that script and triggering their re-tests. Thanks Thomas. On Wed, Jun 7, 2023 at 8:53=E2=80=AFAM Aaron Conole <aconole@redhat.com> wr= ote: > Patrick Robb <probb@iol.unh.edu> writes: > > > Also it can be useful to run daily sub-tree testing by request, if > possible. > > > > That wouldn't be too difficult. I'll make a ticket for this. Although, > for testing on the daily sub-trees, since that's > > UNH-IOL specific, that wouldn't necessarily have to be done via an emai= l > based testing request framework. We > > could also just add a button to our dashboard which triggers a sub-tree > ci run. That would help keep narrow > > the scope of what the email based retesting framework is for. But, both > email or a dashboard button would > > both work. > > We had discussed this long ago - including agreeing on a format, IIRC. > > See the thread starting here: > https://mails.dpdk.org/archives/ci/2021-May/001189.html > > The idea was to have a line like: > > Recheck-request: <test names> > > where <test names> was the tests in the check labels. In fact, what > started the discussion was a patch for the pw-ci scripts that > implemented part of it. > > I don't see how to make your proposal as easily parsed. > > WDYT? Can you re-read that thread and come up with comments? > > > On Tue, Jun 6, 2023 at 1:53=E2=80=AFPM Ferruh Yigit <ferruh.yigit@amd.c= om> > wrote: > > > > On 6/6/2023 5:56 PM, Patrick Robb wrote: > > > Hello all, > > > > > > I'd like to revive the conversation about a request from the communi= ty > > > for an email based re-testing framework. The idea is that using one > > > standardized format, dpdk developers could email the test-report > mailing > > > list, requesting a rerun on their patch series for "X" set of tests = at > > > "Y" lab. I think that since patchwork testing labels (ie. > > > iol-broadcom-Performance, github-robot: build, loongarch-compilation= ) > > > are already visible on patch pages on patchwork, those labels are th= e > > > most reasonable ones to expect developers to use when requesting a > > > re-test. We probably wouldn't want to get any more general than that= , > > > like, say, rerunning all CI testing for a specific patch series at a > > > specific lab, since it would result in a significant amount of > "wasted" > > > testing capacity. > > > > > > The standard email format those of us at the Community Lab are > thinking > > > of is like below. Developers would request retests by emailing the > > > test-report mailing list with email bodies like: > > > > > > [RETEST UNH-IOL] > > > iol-abi-testing > > > iol-broadcom-Performance > > > > > > [RETEST Intel] > > > intel-Functional > > > > > > [RETEST Loongson] > > > loongarch-compilation > > > > > > [RETEST GHA] > > > github-robot: build > > > > > > From there, it would be up to the various labs to poll the test-repo= rt > > > mailing list archive (or use a similar method) to check for such > > > requests, and trigger a CI testing rerun based on the labels provide= d > in > > > the re-test email. If there is interest from other labs, UNH might > also > > > be able to host the entire set of re-test requests, allowing other > labs > > > to poll a curated list hosted by UNH. One simple approach would be f= or > > > labs to download all emails sent to test-report and parse with regex > to > > > determine the re-test list for their specific lab. But, if anyone ha= s > > > any better ideas for aggregating the emails to be parsed, suggestion= s > > > are welcome! If this approach sounds reasonable to everyone, we coul= d > > > determine a timeline by which labs would implement the functionality > > > needed to trigger re-tests. Or, we can just add re-testing for vario= us > > > labs if/when they add this functionality - whatever is better. Happy > to > > > discuss at the CI meeting on Thursday. > > > > > > > +1 to re-testing framework. > > > > Also it can be useful to run daily sub-tree testing by request, if > possible. > > --=20 Patrick Robb Technical Service Manager UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 www.iol.unh.edu --0000000000004fe98b05fd93f87c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px = 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">What wou= ld be the purpose of [RETEST UNH-IOL]?<br></blockquote><div>Agreed, this is= redundant, provided we will be using the labels/contexts used on patchwork= . That seems to be the idea behind Aaron's proposed format and I think = we should move to use his format since it previously reached some consensus= , and appears to be easy to parse.</div><div>=C2=A0</div><blockquote class= =3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg= b(204,204,204);padding-left:1ex">We need to specify the patchwork identifie= r of the patch.<br><br>We could make a script similar to the checkpatch run= on=C2=A0<a href=3D"http://dpdk.org/" rel=3D"noreferrer" target=3D"_blank">= dpdk.org</a>:<br><a href=3D"https://git.dpdk.org/tools/dpdk-ci/tree/tests/c= heckpatch.sh" rel=3D"noreferrer" target=3D"_blank">https://git.dpdk.org/too= ls/dpdk-ci/tree/tests/checkpatch.sh</a><br>The easiest way to run it is to = make the script as the receiver of the mail.<br>If the lab can receive the = mails from the mailing list,<br>then just need to filter the retest request= s for its own lab.</blockquote><div>Yes, I think this is reasonable. I don&= #39;t think this process is likely to change much, and if we can provide a = script to live on the dpdk-ci repo which checks for retest requests, we can= reasonably expect the labs to separately=C2=A0set up an environment to han= dle running that script and triggering their re-tests. Thanks Thomas.<br><b= r><br></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"= gmail_attr">On Wed, Jun 7, 2023 at 8:53=E2=80=AFAM Aaron Conole <<a href= =3D"mailto:aconole@redhat.com">aconole@redhat.com</a>> wrote:<br></div><= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex">Patrick Robb <<a href= =3D"mailto:probb@iol.unh.edu" target=3D"_blank">probb@iol.unh.edu</a>> w= rites:<br> <br> >=C2=A0 Also it can be useful to run daily sub-tree testing by request, = if possible.<br> ><br> > That wouldn't be too difficult. I'll make a ticket for this. A= lthough, for testing on the daily sub-trees, since that's<br> > UNH-IOL specific, that wouldn't necessarily have to be done via an= email based testing request framework. We<br> > could also just add a button to our dashboard which triggers a sub-tre= e ci run. That would help keep narrow<br> > the scope of what the email based retesting framework is for. But, bot= h email or a dashboard button would<br> > both work. <br> <br> We had discussed this long ago - including agreeing on a format, IIRC.<br> <br> See the thread starting here:<br> =C2=A0 <a href=3D"https://mails.dpdk.org/archives/ci/2021-May/001189.html" = rel=3D"noreferrer" target=3D"_blank">https://mails.dpdk.org/archives/ci/202= 1-May/001189.html</a><br> <br> The idea was to have a line like:<br> <br> Recheck-request: <test names><br> <br> where <test names> was the tests in the check labels.=C2=A0 In fact, = what<br> started the discussion was a patch for the pw-ci scripts that<br> implemented part of it.<br> <br> I don't see how to make your proposal as easily parsed.<br> <br> WDYT?=C2=A0 Can you re-read that thread and come up with comments?<br> <br> > On Tue, Jun 6, 2023 at 1:53=E2=80=AFPM Ferruh Yigit <<a href=3D"mai= lto:ferruh.yigit@amd.com" target=3D"_blank">ferruh.yigit@amd.com</a>> wr= ote:<br> ><br> >=C2=A0 On 6/6/2023 5:56 PM, Patrick Robb wrote:<br> >=C2=A0 > Hello all,<br> >=C2=A0 > <br> >=C2=A0 > I'd like to revive the conversation about a request fro= m the community<br> >=C2=A0 > for an email based re-testing framework. The idea is that u= sing one<br> >=C2=A0 > standardized format, dpdk developers could email the test-r= eport mailing<br> >=C2=A0 > list, requesting a rerun on their patch series for "X&= quot; set of tests at<br> >=C2=A0 > "Y" lab. I think that since patchwork testing lab= els (ie.<br> >=C2=A0 > iol-broadcom-Performance, github-robot: build, loongarch-co= mpilation)<br> >=C2=A0 > are already visible on patch pages on patchwork, those labe= ls are the<br> >=C2=A0 > most reasonable ones to expect developers to use when reque= sting a<br> >=C2=A0 > re-test. We probably wouldn't want to get any more gene= ral than that,<br> >=C2=A0 > like, say, rerunning all CI testing for a specific patch se= ries at a<br> >=C2=A0 > specific lab, since it would result in a significant amount= of "wasted"<br> >=C2=A0 > testing capacity.<br> >=C2=A0 > <br> >=C2=A0 > The standard email format those of us at the Community Lab = are thinking<br> >=C2=A0 > of is like below. Developers would request retests by email= ing the<br> >=C2=A0 > test-report mailing list with email bodies like:<br> >=C2=A0 > <br> >=C2=A0 > [RETEST UNH-IOL]<br> >=C2=A0 > iol-abi-testing<br> >=C2=A0 > iol-broadcom-Performance<br> >=C2=A0 > <br> >=C2=A0 > [RETEST Intel]<br> >=C2=A0 > intel-Functional<br> >=C2=A0 > <br> >=C2=A0 > [RETEST Loongson]<br> >=C2=A0 > loongarch-compilation<br> >=C2=A0 > <br> >=C2=A0 > [RETEST GHA]<br> >=C2=A0 > github-robot: build<br> >=C2=A0 > <br> >=C2=A0 > From there, it would be up to the various labs to poll the = test-report<br> >=C2=A0 > mailing list archive (or use a similar method) to check for= such<br> >=C2=A0 > requests, and trigger a CI testing rerun based on the label= s provided in<br> >=C2=A0 > the re-test email. If there is interest from other labs, UN= H might also<br> >=C2=A0 > be able to host the entire set of re-test requests, allowin= g other labs<br> >=C2=A0 > to poll a curated list hosted by UNH. One simple approach w= ould be for<br> >=C2=A0 > labs to download all emails sent to test-report and parse w= ith regex to<br> >=C2=A0 > determine the re-test list for their specific lab. But, if = anyone has<br> >=C2=A0 > any better ideas for aggregating the emails to be parsed, s= uggestions<br> >=C2=A0 > are welcome! If this approach sounds reasonable to everyone= , we could<br> >=C2=A0 > determine a timeline by which labs would implement the func= tionality<br> >=C2=A0 > needed to trigger re-tests. Or, we can just add re-testing = for various<br> >=C2=A0 > labs if/when they add this functionality - whatever is bett= er. Happy to<br> >=C2=A0 > discuss at the CI meeting on Thursday.<br> >=C2=A0 > <br> ><br> >=C2=A0 +1 to re-testing framework.<br> ><br> >=C2=A0 Also it can be useful to run daily sub-tree testing by request, = if possible.<br> <br> </blockquote></div><br clear=3D"all"><div><br></div><span class=3D"gmail_si= gnature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature"><d= iv dir=3D"ltr"><p dir=3D"ltr" style=3D"line-height:1.2;margin-top:0pt;margi= n-bottom:0pt"><font color=3D"#000000" face=3D"Arial"><span style=3D"font-si= ze:13.3333px;white-space:pre-wrap">Patrick Robb</span></font></p><p style= =3D"color:rgb(34,34,34);line-height:1.2;margin-top:0pt;margin-bottom:0pt"><= span style=3D"font-size:10pt;font-family:Arial;color:rgb(0,0,0);background-= color:transparent;vertical-align:baseline;white-space:pre-wrap">Technical S= ervice Manager</span></p><p dir=3D"ltr" style=3D"color:rgb(34,34,34);line-h= eight:1.2;margin-top:0pt;margin-bottom:0pt"><span style=3D"font-size:10pt;f= ont-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-ali= gn:baseline;white-space:pre-wrap">UNH InterOperability Laboratory</span></p= ><p dir=3D"ltr" style=3D"color:rgb(34,34,34);line-height:1.2;margin-top:0pt= ;margin-bottom:0pt"><span style=3D"font-size:10pt;font-family:Arial;color:r= gb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:= pre-wrap">21 Madbury Rd, Suite 100, Durham, NH 03824</span></p><p dir=3D"lt= r" style=3D"color:rgb(34,34,34);line-height:1.2;margin-top:0pt;margin-botto= m:0pt"><span style=3D"font-size:10pt;font-family:Arial;color:rgb(17,85,204)= ;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"= ><a href=3D"http://www.iol.unh.edu/" style=3D"color:rgb(17,85,204)" target= =3D"_blank">www.iol.unh.edu</a></span></p><p dir=3D"ltr" style=3D"color:rgb= (34,34,34);line-height:1.2;margin-top:0pt;margin-bottom:0pt"><br></p><p dir= =3D"ltr" style=3D"color:rgb(34,34,34);line-height:1.2;margin-top:0pt;margin= -bottom:0pt"><span style=3D"font-size:10pt;font-family:Arial;color:rgb(51,5= 1,51);background-color:transparent;vertical-align:baseline;white-space:pre-= wrap"><img src=3D"https://lh4.googleusercontent.com/7sTY8VswXadak_YT0J13osh= 5ockNVRX2BuYaRsKoTTpkpilBokA0WlocYHLB4q7XUgXNHka6-ns47S8R_am0sOt7MYQQ1ILQ3S= -P5aezsrjp3-IsJMmMrErHWmTARNgZhpAx06n2" width=3D"150" height=3D"37" style= =3D"border: none;"></span></p></div></div> --0000000000004fe98b05fd93f87c--