From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <ci-bounces@dpdk.org> Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7593441FCE; Wed, 30 Aug 2023 14:18:25 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 75B7740289; Wed, 30 Aug 2023 14:18:25 +0200 (CEST) Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by mails.dpdk.org (Postfix) with ESMTP id 53CFF40277; Wed, 30 Aug 2023 14:18:23 +0200 (CEST) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id BC9CE320039A; Wed, 30 Aug 2023 08:18:21 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Wed, 30 Aug 2023 08:18:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm1; t= 1693397901; x=1693484301; bh=gyGQ64rIfF6OzcSp8DWl/B/zsZg/1WE33pT pBLjQ7l0=; b=DHmapPr226xo2QLsmxMy5PnGt0mDa55j4GmksaUgDvfJe9I5YdS vZbykWJse82XuxGlGzKyc6we0QWJy104TWFMkIGc1bqJ6mNQ8nKsAdwi6C7nhvWI gTNiIfFEbLMjwBHTOqmTxyTMajx9+JRDWqL7nILEpyIIvnp86jJnmDPreqZ6uo4w Yp62ErUEalJbix7W+UGUxISiRo45o6wq1hp9zJAZGI9l2aA366U1RYiMFoZh+Xps 3P2b33gRsT7V9PybtvMJv822ytQbpnHyqD6T9W/001pKePx5ZVZmxUQ8pvMJu25S bCH2xeKp0mPeadt5ADRf1S8tPOREgZdUvTg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1693397901; x=1693484301; bh=gyGQ64rIfF6OzcSp8DWl/B/zsZg/1WE33pT pBLjQ7l0=; b=rsuzUGJCv9m6csxPdq7lTm+0UgnaUEnyu9Ned2O1i1ttHnnaffT L/9rge1yfzKMZN/WYgqJoRTXzkqw0hU3iw8MmedUzQpKlxpigdRfg3dKE14g63Za SOFT74eMxM7RpDbV44Tqve6PXAIPCUUgO8VuSmG2hmboc3tULeBT4TcX40jt/cRn XfnlqrMOGyFANDZIhbubd+BPxzpm0Qo/QBMEQIaiaa09YfaX1yAAis7mu729s1mu iXVRYD457TgFoWswhPEXhkuRhaYK7be18jSmY/ScFbeRu9+oIcezQhwZ85ZSf71v aEygcI8B2REC2HzmQywMBiCcXp6HBTslFyw== X-ME-Sender: <xms:jDPvZLzfkiFELMgcJWJkbR6YsByB81J5NecMPruhG1rSE4W73wASag> <xme:jDPvZDRH0tHuP4gHycNC2T7A_3YFHuUT0V3VzrS64VeLnXnH-4dzUq3UdUgaDCa8d PNmxdWYJLqnhbL7cw> X-ME-Received: <xmr:jDPvZFVh0fVd7c6iAcvvaSOEjFwnz7mZn-u4UcvdRcv3PZddUuJ-0-bykolguIZQ6hq7crf2x2tGQQRB3NTniC9Ysg> X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudefkedgheduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne goufhushhpvggtthffohhmrghinhculdegledmnecujfgurhephffvvefufffkjghfggfg tgesthhqredttddtjeenucfhrhhomhepvfhhohhmrghsucfoohhnjhgrlhhonhcuoehthh homhgrshesmhhonhhjrghlohhnrdhnvghtqeenucggtffrrghtthgvrhhnpeehleegleff ieejueethefhvdelgfehteelhedvueegudeuhffhhefgffejheetudenucffohhmrghinh eplhhothdrhhhofidpughpughkrdhorhhgnecuvehluhhsthgvrhfuihiivgeptdenucfr rghrrghmpehmrghilhhfrhhomhepthhhohhmrghssehmohhnjhgrlhhonhdrnhgvth X-ME-Proxy: <xmx:jTPvZFg72HyE5xTpQ0oqn7Y0ftMQEITkBiVcI34h7X-cYILso6w7vQ> <xmx:jTPvZNAYxq6U57qYmPlO9yRBy62XNFyp8oGw_Rjewf6jAB0uuq5mxg> <xmx:jTPvZOIMSiUQejdhoe-ZkAWcG5nU20b7qZZlPOBFUIVNiLiOrPGFzQ> <xmx:jTPvZK5Zyrvl5_wQjtqMgqrhtUH-U9M4zEvBl1K2kJBbK64Ds27hcg> Feedback-ID: i47234305:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 30 Aug 2023 08:18:20 -0400 (EDT) From: Thomas Monjalon <thomas@monjalon.net> To: Adam Hassick <ahassick@iol.unh.edu> Cc: dev@dpdk.org, ci@dpdk.org Subject: Re: CI Testing Automated Recheck Request Framework Date: Wed, 30 Aug 2023 14:18:18 +0200 Message-ID: <3756491.kQq0lBPeGt@thomas> In-Reply-To: <CAC-YWqiXqBYyzPsc4UD7LbUHKha_Vb3=Aot+dQomuRLojy2hvA@mail.gmail.com> References: <CAC-YWqiXqBYyzPsc4UD7LbUHKha_Vb3=Aot+dQomuRLojy2hvA@mail.gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions <ci.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/ci>, <mailto:ci-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/ci/> List-Post: <mailto:ci@dpdk.org> List-Help: <mailto:ci-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/ci>, <mailto:ci-request@dpdk.org?subject=subscribe> Errors-To: ci-bounces@dpdk.org Thank you, it will help a lot. How/Where are we going to document this? Maybe in the "Testing" web page? https://core.dpdk.org/testing/ 25/08/2023 16:51, Adam Hassick: > Hello DPDK developers, >=20 > Currently, various testing labs perform CI testing on new patch series se= nt > to dev@dpdk.org and report their results to > https://patchwork.dpdk.org/project/dpdk/list/. On each series on the patch > list, the results appear in the test category contexts for IOL (community > lab), GitHub, and LoongSon. >=20 > If a reported failure on a series seems suspicious to the patch submitter > or maintainer, then there may be an interest in requesting a retest on the > series for the failing label(s) in order to verify the failure is not > spurious or a false positive. This retest demonstrates to the submitter or > maintainer that the failure can be reliably reproduced. Unfortunately, at > present, the best way to accomplish this is to reach out to lab maintaine= rs > via email or Slack. This is not ideal for developers in need of quick test > results. >=20 > Going forward, CI testing labs will be implementing the option to request > retest for their respective test labels on patchwork via emails sent to t= he > dev mailing list. This feature is ready today for labels reported by the > UNH-IOL Community Lab, and will soon also be an option for the Github Rob= ot > at least. >=20 > In order to request a retest on your patch series, send an email reply to > one of your series=E2=80=99s patch or cover letter emails with email cont= ent of the > format used below: >=20 > Recheck-request: <test names> >=20 > The valid delimiter is a comma optionally followed by a space: =E2=80=9C,= =E2=80=9D =E2=80=9C, =E2=80=9C >=20 > Valid examples: >=20 > Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance, > iol-compile-arm64-testing, >=20 > Recheck-request: iol-compile-amd64-testing,iol-broadcom-Performance, > iol-compile-arm64-testing, >=20 > Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance, > iol-compile-arm64-testing >=20 > Invalid examples: >=20 > Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance >=20 > Recheck-request: iol-compile-amd64-testing > iol-broadcom-Performance,iol-compile-arm64-testing, >=20 > Some important notes: >=20 > 1. >=20 > At present, there is only support for retesting the series as it exist= ed > when the lab received it. As in, if the lab applied the series on DPDK > mainline when the head was commit X, and a retest is requested, then > retests will be run using those same sources applied on top of commit = X. > This is important to note because this means retest requests will not > provide a solution to your patch being submitted when the tree is in a= =E2=80=9Cbad > state.=E2=80=9D Getting test results with your patch applied on the cu= rrent DPDK > mainline could be achieved by re-submitting the patch to the mailing l= ist > as a workaround. > 2. >=20 > For any series submitted earlier than August 2023, you must submit a > retest request in reply to a patch email, NOT in reply to a cover lett= er > email. > 3. >=20 > The initial policy is to accept no more than one retest request per > patch series version per lab. > 4. >=20 > Your patch should begin to retest within 15 minutes of your request, b= ut > wait time is subject to the testing queue just like any other series. = As a > result, retesting will be slower during peak submission time. >=20 >=20 > Improvements we are considering for v2 of the email retesting framework: >=20 > 1. >=20 > Add in an option to re-apply on the latest commit on DPDK mainline. So, > if your patch was originally applied on commit X, and you want to rete= st, > but have it be applied to commit Y (latest), you could specify that. U= nder > these circumstances, we would have to do a retest of all labels, since= it > would be inappropriate to mix reports for results from different commi= ts. > 2. >=20 > Add a policy for vetting retest requesters - so maybe only maintainers, > or maybe only maintainers and the submitter, or another set of people. > 3. >=20 > Add in an option to request a retest for next-* branches and/or LTS > branches. >=20