From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 13B57430C6; Mon, 21 Aug 2023 22:36:45 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DB41640DF8; Mon, 21 Aug 2023 22:36:44 +0200 (CEST) Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) by mails.dpdk.org (Postfix) with ESMTP id 49D7C40DF5 for ; Mon, 21 Aug 2023 22:36:43 +0200 (CEST) Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-58c4f61ca12so41915597b3.3 for ; Mon, 21 Aug 2023 13:36:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1692650202; x=1693255002; h=cc:to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=w6v+Pf5fvxBSCyUDrgHgyXHkuCeG1k0CK6Rn4wou05w=; b=jwUonnMd9C+t0C9Ii3Detg8H4sTIzyXpnSwyJCEXA+3U8qNICgLpeLPjV9cp3h+7kl 3wB3Czci46dmzSXTlDTJq4V8zOUQaR25qO8uCv8GDCzNn6T/sZkFqKR3KLY7v+T0v9cq Ux1nkH5RooA8n1gGVDb3bfLoEAkuOnmbw7EVM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692650202; x=1693255002; h=cc:to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=w6v+Pf5fvxBSCyUDrgHgyXHkuCeG1k0CK6Rn4wou05w=; b=LeoD9ki0K0bAkl7Y3JoXFB9ifPxJOKLtdNzw+K52NA+2ARUuQP5saiLpPAM44MvMeY hY00Z4GUOCK4nz9ulzNPsuF93GaaR+wChRVyXP+LWCmXDkUGBGaqoZGVBaoR2u1zX+2X LAZiS+EmhZ3wII54NSJ5tJDzaP9zcebOFap1s6yjakmrIa9cICdKpEEPBW1bzD5pNfuy VW76a1f7sXMIRQfQKvYBdPYdZUidApm2d9REpM9Q5WVu6bkvanUTWYXm5u5GE+pvjXt6 uWbZcQLGL3XiHRWpsPcjAoWWDJfIQjn5oSzj707hX/5nElUNRx1xdlyw8CDuDXF7FsTq 7YnA== X-Gm-Message-State: AOJu0YwWSRoY0Rk0l/16HhD5tQd5m1FnHWpY619GTaYqK5toP44RYr/N LBUJDUB8iHjGYFvy9mJNhmoCYP6x+2m9jhVANYFtgAb1XtmmcS6dk418Zg== X-Google-Smtp-Source: AGHT+IEqxm8+/2cOqaj6Q6DQ2y7Dh5fvKV1aqmDacvjzLHVk9wSgQNsWkZvS8T5fC6lTPqhg3g2Yqzr2KXy5NCrw7Mw= X-Received: by 2002:a0d:f183:0:b0:58f:bda3:8dd with SMTP id a125-20020a0df183000000b0058fbda308ddmr8027546ywf.32.1692650202404; Mon, 21 Aug 2023 13:36:42 -0700 (PDT) MIME-Version: 1.0 From: Adam Hassick Date: Mon, 21 Aug 2023 16:37:08 -0400 Message-ID: Subject: Retesting Framework Announcement Draft To: ci@dpdk.org Cc: Patrick Robb Content-Type: multipart/alternative; boundary="0000000000000b397e060374d612" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --0000000000000b397e060374d612 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi All, Below is our final draft of our announcement to the DPDK developers informing them of our new recheck request system. We would like your feedback and suggestions before we send it out. Thanks, Adam -- Hello DPDK Developers, Currently, various testing labs perform CI testing on new patch series sent to dev@dpdk.org and report their results to https://patchwork.dpdk.org/project/dpdk/list/. On each series on the patch list, the results appear in the test category contexts for IOL (community lab), GitHub, LoongSon, and Intel. If a reported failure on a series seems suspect to the patch submitter or maintainer, then there is an interest in requesting a retest on the series for the failing label(s) in order to verify the failure is not spurious or false positive. This retest demonstrates to the submitter or maintainer that the failure can be reliably reproduced. Unfortunately, at present, the best way to accomplish this is to reach out to lab maintainers via email or slack. This is not ideal for developers in need of quick test results. Going forward, CI testing labs will be implementing the option to request retest for their respective test labels on patchwork via emails sent to the dev mailing list. This feature is ready today for labels reported by the UNH-IOL Community Lab, and will soon also be an option for the Github Robot at least. In order to request a retest on your patch series, send an email reply to one of your series=E2=80=99s patch or cover letter emails with email conten= t of the format used below: Recheck-request: The valid delimiter is a comma optionally followed by a space or a newline character: =E2=80=9C,=E2=80=9D =E2=80=9C, =E2=80=9C =E2=80=9C,\n=E2=80=9D Valid examples: Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance, iol-compile-arm64-testing, Recheck-request: iol-compile-amd64-testing,iol-broadcom-Performance, iol-compile-arm64-testing, Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance, iol-compile-arm64-testing Invalid examples: Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance Recheck-request: iol-compile-amd64-testing iol-broadcom-Performance,iol-compile-arm64-testing, Some important notes: 1. At present, there is only support for retesting the series as it existed when the lab received it. As in, if the lab applied the series on DPDK mainline when the head was commit X, and a retest is requested, then retests will be run using those same sources applied on top of commit X. This is important to note because this means retest requests will not provide a solution to your patch being submitted when the tree is in a = =E2=80=9Cbad state.=E2=80=9D 2. For any series submitted earlier than August 2023, you must submit a retest request in reply to a patch email, NOT in reply to a cover letter email. 3. The initial policy is to accept no more than one retest request per patch per lab. 4. Your patch should begin to retest within 30 minutes of your request, but wait time is subject to the testing queue just like any other series. As= a result, retesting will be slower during peak submission time. Improvements we are considering for v2 of the email retesting framework: 1. Add in an option to re-apply on the latest commit on DPDK mainline. So, if your patch was originally applied on commit X, and you want to retest= , but have it be applied to commit Y (latest), you could specify that. Und= er these circumstances, we would have to do a retest of all labels, since i= t would be inappropriate to mix reports for results from different commits= . 2. Add a policy for vetting retest requesters - so maybe only maintainers, or maybe only maintainers and the submitter, or another set of people. 3. Add in an option to request a retest for next-* branches and/or LTS branches. --0000000000000b397e060374d612 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi All,

Below is our final draft of our announcement to th= e DPDK developers informing them of our new recheck request system. We woul= d like your feedback and suggestions before we send it out.

<= /font>
Thanks,
Adam

--

Hello DPDK = Developers,


Currently, various testing labs perform CI testing on new patch se= ries sent to dev@dpdk.org<= /span> and report thei= r results to https://patchwork.dpdk.org/project/dpdk/list/. On each series on the patch li= st, the results appear in the test category contexts for IOL (community lab= ), GitHub, LoongSon, and Intel.


If a reported failure on a series seems suspec= t to the patch submitter or maintainer, then there is an interest in reques= ting a retest on the series for the failing label(s) in order to verify the= failure is not spurious or false positive. This retest demonstrates to the= submitter or maintainer that the failure can be reliably reproduced. Unfor= tunately, at present, the best way to accomplish this is to reach out to la= b maintainers via email or slack. This is not ideal for developers in need = of quick test results.


Going forward, CI testing labs will be implementing the= option to request retest for their respective test labels on patchwork via= emails sent to the dev mailing list. This feature is ready today for label= s reported by the UNH-IOL Community Lab, and will soon also be an option fo= r the Github Robot at least.


In order to request a retest on your patch series= , send an email reply to one of your series=E2=80=99s patch or cover letter= emails with email content of the format used below:=C2=A0

Recheck-reques= t: <test names>


The valid delimiter is a comma optionally followed = by a space or a newline character: =E2=80=9C,=E2=80=9D =E2=80=9C, =E2=80=9C= =E2=80=9C,\n=E2=80=9D


Valid examples:

Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance,= iol-compile-arm64-testing,

Recheck= -request: iol-compile-amd64-testing,iol-broadcom-Performance, iol-compile-a= rm64-testing,

= Recheck-request: iol-= compile-amd64-testing, iol-broadcom-Performance, iol-compile-arm64-testing<= /span>


Invalid examples:

Recheck-reques= t: iol-compile-amd64-testing,=C2=A0 iol-broadcom-Performance<= /p>

Recheck-request: iol-compile-amd64-testing iol-br= oadcom-Performance,iol-compile-arm64-testing,


Some important notes:

  1. At present, there is only support for retesting the series as it existed w= hen the lab received it. As in, if the lab applied the series on DPDK mainl= ine when the head was commit X, and a retest is requested, then retests wil= l be run using those same sources applied on top of commit X. This is impor= tant to note because this means retest requests will not provide a solution= to your patch being submitted when the tree is in a =E2=80=9Cbad state.=E2= =80=9D

  2. For any series submitted earlier= than August 2023, you must submit a retest request in reply to a patch ema= il, NOT in reply to a cover letter email.

  3. The initial policy is to accept no more than one retest request per patc= h per lab.

  4. Your patch should begin to r= etest within 30 minutes of your request, but wait time is subject to the te= sting queue just like any other series. As a result, retesting will be slow= er during peak submission time.


Improvements we are considering= for v2 of the email retesting framework:

  1. =

    Add in an option to re-apply o= n the latest commit on DPDK mainline. So, if your patch was originally appl= ied on commit X, and you want to retest, but have it be applied to commit Y= (latest), you could specify that. Under these circumstances, we would have= to do a retest of all labels, since it would be inappropriate to mix repor= ts for results from different commits.

  2. Add a policy for vetting retest requesters - so maybe only maintainers, or= maybe only maintainers and the submitter, or another set of people.=C2=A0<= /span>

  3. Add in an option to request a retest fo= r next-* branches and/or LTS branches.



--0000000000000b397e060374d612--