From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 27FF143102; Fri, 25 Aug 2023 16:51:30 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8F24642BC9; Fri, 25 Aug 2023 16:51:28 +0200 (CEST) Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) by mails.dpdk.org (Postfix) with ESMTP id 6BC4840695 for ; Fri, 25 Aug 2023 16:51:27 +0200 (CEST) Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-58c92a2c52dso12191917b3.2 for ; Fri, 25 Aug 2023 07:51:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1692975086; x=1693579886; h=cc:to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=qTIn2qiG8QMXETru18pNrQoejOrAjhLZpAGR6KsZrQ8=; b=hV7+/StZAiDZjbdZ7TgsSI6dclgLG/AW3Kz8aNHDRi0WME0YJ8IMzjGJ2VL79cmH2d 9ftKCRphLnlwLh4/E8OKvNoBAafTmey/eSrtQbpthzbuB0FGaBpJ3LwhgCYowkv2BkFz e1xe5ZyxeBRo5GReZ9CnZYG9IX+5re2ARYVqI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692975086; x=1693579886; h=cc:to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=qTIn2qiG8QMXETru18pNrQoejOrAjhLZpAGR6KsZrQ8=; b=Re+FRimjEexmrmYpbJpSdzjfAJo3jDsOFWeT1RxE7HRCsZrSM8bsQ2OElBeb7y/6Pt 0g3L3reRhZ+kLoR0zvhrELTClLyFbh88XPTOIuN/bo7wkrEqI9lJGpC1Js85W/++nwfA sUQXYxI5Onj8wIjcguCgzUypKZgKy+t11tdyIpUlqwsizJKjHDEG9ImS729fOgXQEn7P 1xQUpTJLSOOsMDoDxXLfgw6gFAyKEBXfS2Nt/VeC761NayBqKpbdRf45+Kg33Bzqw+J2 hM0xpcKekvCDlQ+euCsqGSNyV5QLcyvIAJTP355rX7K0j6BlBMaFhI1Xjn52sVz1UptS 2mIg== X-Gm-Message-State: AOJu0Yzmb65rvx6OB6+ZTxQUwki3xp7pXz0y10ca8yIFAu2Rc7Nhi7mz bjpFtsZJYPbeSvyr/JcWdtW40yyfk15XEsO/l6PX8Pl0UrOS3EowVR8e7A== X-Google-Smtp-Source: AGHT+IFZmYmP0Rsb4wztWMFEHOlTLWuNyCkW8jeyvWpFuZpBmjRNYFAGaLjK/6TncXwVEg0r6rWK6vegsVljlI2987Y= X-Received: by 2002:a81:dd0f:0:b0:586:9ccc:28f5 with SMTP id e15-20020a81dd0f000000b005869ccc28f5mr16451156ywn.51.1692975086567; Fri, 25 Aug 2023 07:51:26 -0700 (PDT) MIME-Version: 1.0 From: Adam Hassick Date: Fri, 25 Aug 2023 10:51:58 -0400 Message-ID: Subject: CI Testing Automated Recheck Request Framework To: dev@dpdk.org Cc: ci@dpdk.org, dts@dpdk.org Content-Type: multipart/alternative; boundary="000000000000a61e370603c07a60" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --000000000000a61e370603c07a60 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello DPDK developers, Currently, various testing labs perform CI testing on new patch series sent to dev@dpdk.org and report their results to https://patchwork.dpdk.org/project/dpdk/list/. On each series on the patch list, the results appear in the test category contexts for IOL (community lab), GitHub, and LoongSon. If a reported failure on a series seems suspicious to the patch submitter or maintainer, then there may be an interest in requesting a retest on the series for the failing label(s) in order to verify the failure is not spurious or a false positive. This retest demonstrates to the submitter or maintainer that the failure can be reliably reproduced. Unfortunately, at present, the best way to accomplish this is to reach out to lab maintainers via email or Slack. This is not ideal for developers in need of quick test results. Going forward, CI testing labs will be implementing the option to request retest for their respective test labels on patchwork via emails sent to the dev mailing list. This feature is ready today for labels reported by the UNH-IOL Community Lab, and will soon also be an option for the Github Robot at least. In order to request a retest on your patch series, send an email reply to one of your series=E2=80=99s patch or cover letter emails with email conten= t of the format used below: Recheck-request: The valid delimiter is a comma optionally followed by a space: =E2=80=9C,= =E2=80=9D =E2=80=9C, =E2=80=9C Valid examples: Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance, iol-compile-arm64-testing, Recheck-request: iol-compile-amd64-testing,iol-broadcom-Performance, iol-compile-arm64-testing, Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance, iol-compile-arm64-testing Invalid examples: Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance Recheck-request: iol-compile-amd64-testing iol-broadcom-Performance,iol-compile-arm64-testing, Some important notes: 1. At present, there is only support for retesting the series as it existed when the lab received it. As in, if the lab applied the series on DPDK mainline when the head was commit X, and a retest is requested, then retests will be run using those same sources applied on top of commit X. This is important to note because this means retest requests will not provide a solution to your patch being submitted when the tree is in a = =E2=80=9Cbad state.=E2=80=9D Getting test results with your patch applied on the curr= ent DPDK mainline could be achieved by re-submitting the patch to the mailing lis= t as a workaround. 2. For any series submitted earlier than August 2023, you must submit a retest request in reply to a patch email, NOT in reply to a cover letter email. 3. The initial policy is to accept no more than one retest request per patch series version per lab. 4. Your patch should begin to retest within 15 minutes of your request, but wait time is subject to the testing queue just like any other series. As= a result, retesting will be slower during peak submission time. Improvements we are considering for v2 of the email retesting framework: 1. Add in an option to re-apply on the latest commit on DPDK mainline. So, if your patch was originally applied on commit X, and you want to retest= , but have it be applied to commit Y (latest), you could specify that. Und= er these circumstances, we would have to do a retest of all labels, since i= t would be inappropriate to mix reports for results from different commits= . 2. Add a policy for vetting retest requesters - so maybe only maintainers, or maybe only maintainers and the submitter, or another set of people. 3. Add in an option to request a retest for next-* branches and/or LTS branches. --000000000000a61e370603c07a60 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Hello DPDK developers,


Currently, various testin= g labs perform CI testing on new patch series sent to dev@dpdk.org and report their results to https://patchwork.dpdk.org/project/d= pdk/list/. On each series on the patch list, the results appear in the = test category contexts for IOL (community lab), GitHub, and LoongSon.


If a reported failure on a series seems suspicious to the patch s= ubmitter or maintainer, then there may be an interest in requesting a retes= t on the series for the failing label(s) in order to verify the failure is = not spurious or a false positive. This retest demonstrates to the submitter= or maintainer that the failure can be reliably reproduced. Unfortunately, = at present, the best way to accomplish this is to reach out to lab maintain= ers via email or Slack. This is not ideal for developers in need of quick t= est results.


Going forward, CI testing labs will be implemen= ting the option to request retest for their respective test labels on patch= work via emails sent to the dev mailing list. This feature is ready today f= or labels reported by the UNH-IOL Community Lab, and will soon also be an o= ption for the Github Robot at least.


In order to request a r= etest on your patch series, send an email reply to one of your series=E2=80= =99s patch or cover letter emails with email content of the format used bel= ow:=C2=A0


Recheck-request: <test names>

The valid delimiter is a comma optionally followed by a space: = =E2=80=9C,=E2=80=9D =E2=80=9C, =E2=80=9C


Valid examples= :

Recheck-request: iol-compile-amd64-testing, iol-broadcom-= Performance, iol-compile-arm64-testing,

Recheck-request: io= l-compile-amd64-testing,iol-broadcom-Performance, iol-compile-arm64-testing= ,

Recheck-request: iol-compile-amd64-testing, iol-broadcom-= Performance, iol-compile-arm64-testing


Invalid exampl= es:

Recheck-request: iol-compile-amd64-testing,=C2=A0 iol-b= roadcom-Performance

Recheck-request: iol-compile-amd64-test= ing iol-broadcom-Performance,iol-compile-arm64-testing,


Some important notes:

  1. At present, = there is only support for retesting the series as it existed when the lab r= eceived it. As in, if the lab applied the series on DPDK mainline when the = head was commit X, and a retest is requested, then retests will be run usin= g those same sources applied on top of commit X. This is important to note = because this means retest requests will not provide a solution to your patc= h being submitted when the tree is in a =E2=80=9Cbad state.=E2=80=9D Gettin= g test results with your patch applied on the current DPDK mainline could b= e achieved by re-submitting the patch to the mailing list as a workaround.<= /span>

  2. Fo= r any series submitted earlier than August 2023, you must submit a retest r= equest in reply to a patch email, NOT in reply to a cover letter email.

  3. The i= nitial policy is to accept no more than one retest request per patch series= version per lab.

  4. Your patch should begin to retest within 15 minutes of your= request, but wait time is subject to the testing queue just like any other= series. As a result, retesting will be slower during peak submission time.=


Improvements we are considering for v2 of= the email retesting framework:

  1. Add in an option to re-apply on the latest commit on DPDK mainline. So, = if your patch was originally applied on commit X, and you want to retest, b= ut have it be applied to commit Y (latest), you could specify that. Under t= hese circumstances, we would have to do a retest of all labels, since it wo= uld be inappropriate to mix reports for results from different commits.

  2. Add a policy for vetting retest requesters - so maybe only maintainers, or= maybe only maintainers and the submitter, or another set of people.=C2=A0<= /span>

  3. Add in an option to request a retest for next-* branches and/or LTS bra= nches.

--000000000000a61e370603c07a60--