From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 651E643359; Fri, 17 Nov 2023 23:20:52 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5B5A1410FB; Fri, 17 Nov 2023 23:20:52 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 41F6F4028B for ; Fri, 17 Nov 2023 23:20:50 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700259649; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NpTEUrV6wjpVi+Ijj9YPwqAJmkp9Cxbyr0osvmJu5zQ=; b=L3yySZQLT9EX2ihg4MltArWJyHatWwwVUt7bLQ1cj1E7tfonRRDPUkdXN3XbPvoB00V4te wkE1Awb52RoKjliVoaLIgpLnBAR4FY7U980Dk7e+ZxzKTIh7haukI23KTGsophG4GsWGty ychMV0wNrac0tioEPukNhgOTKmTrNBY= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-531-VQ7qKBIUPYeenUfCX1ApXQ-1; Fri, 17 Nov 2023 17:20:47 -0500 X-MC-Unique: VQ7qKBIUPYeenUfCX1ApXQ-1 Received: by mail-ed1-f72.google.com with SMTP id 4fb4d7f45d1cf-543d2bc7d9dso1886912a12.1 for ; Fri, 17 Nov 2023 14:20:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700259646; x=1700864446; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NpTEUrV6wjpVi+Ijj9YPwqAJmkp9Cxbyr0osvmJu5zQ=; b=rJOEb1aVjXmHWjRFMmV843a2zgQxS2aJ2EdJWSy3BRRWrCeEmfdli4vf0nSMbDISjp X0PmNcvDveDKcBNK8pOU24EeNBxvXngLQAxHV7B3GgKbWbGDwjCdXGbuqG9oplrktfsO Jm1QNVD2pZdXeAqqiG9TNBabEiIuk9gjqMvSYC30FD5NUkX61RhlH8QODHkihpREob8S zMBg1Vd62FUbZ03eQDccivKorjGqaZfrM9qY7hFsEnDMQjIu1B4qurV2LuqETgdnVtx3 lqV64+IRuIebTbsLX7UyznaHWJxPS0CO8Ds/rcqgcEdVWUr3PwTWLbHBOjlnYgE/2wSL hYkg== X-Gm-Message-State: AOJu0Yx7zvUfqMhQrFwkrA9Mclg9Lxrti/13V1zYRjilYHk4ghfzNrF3 PE23xgB+V43zxbCBRg1/qvzjr8N66p4+kcVTo61wJrJypx2V/qYlD9lHplNo4Xu02vbEZQzHVtP tWm0YsG0jl+9LkqWaDA== X-Received: by 2002:a05:6402:2d2:b0:543:bd27:f4a9 with SMTP id b18-20020a05640202d200b00543bd27f4a9mr286575edx.12.1700259645943; Fri, 17 Nov 2023 14:20:45 -0800 (PST) X-Google-Smtp-Source: AGHT+IH5INIIHXnnH4oB3ni+NLbMlPFe4clZF67VqRC/1YuhxzHkJGMnrx5IskAysCfsN7Rz2Qrbasq4xHavpx7rtMc= X-Received: by 2002:a05:6402:2d2:b0:543:bd27:f4a9 with SMTP id b18-20020a05640202d200b00543bd27f4a9mr286563edx.12.1700259645629; Fri, 17 Nov 2023 14:20:45 -0800 (PST) MIME-Version: 1.0 References: <20231107203158.1261199-1-aconole@redhat.com> <20231107203158.1261199-4-aconole@redhat.com> In-Reply-To: From: Michael Santana Date: Fri, 17 Nov 2023 17:20:32 -0500 Message-ID: Subject: Re: [RFC v2 3/3] github: add a tool for restarting checks To: Aaron Conole Cc: ci@dpdk.org, David Marchand , Thomas Monjalon , Patrick Robb , Dumitru Ceara X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org On Fri, Nov 17, 2023 at 2:31=E2=80=AFPM Aaron Conole w= rote: > > Michael Santana writes: > > > On Tue, Nov 7, 2023 at 3:32=E2=80=AFPM Aaron Conole wrote: > >> > >> The recheck framework can track specific recheck labels and track a > >> state for them. Add a tool that will restart github workflow runs > >> and mark the git build table to check for a repoll. This will allow > >> existing monitor tools to re-poll the workflow and report rerun > >> results. > >> > >> Signed-off-by: Aaron Conole > > Acked-by: Michael Santana > > Thanks. > > >> --- > >> github_restart | 141 ++++++++++++++++++++++++++++++++++++++++++++++= + > >> series_db_lib.sh | 16 ++++++ > >> 2 files changed, 157 insertions(+) > >> create mode 100755 github_restart > >> > >> diff --git a/github_restart b/github_restart > >> new file mode 100755 > >> index 0000000..5ba9677 > >> --- /dev/null > >> +++ b/github_restart > >> @@ -0,0 +1,141 @@ > >> +#!/bin/bash > >> +# SPDX-Identifier: gpl-2.0-or-later > >> +# Copyright (C) 2023, Red Hat, Inc. > >> +# > >> +# Restarts a github job run. This can be used in conjunction with > >> +# the recheck requests to provide the ability for a user to restart > >> +# a test - in case the workflow is suspected of having a spurious run= . > >> +# > >> +# Licensed under the terms of the GNU General Public License as publi= shed > >> +# by the Free Software Foundation; either version 2 of the License, o= r > >> +# (at your option) any later version. You may obtain a copy of the > >> +# license at > >> +# > >> +# https://www.gnu.org/licenses/old-licenses/gpl-2.0.html > >> +# > >> +# Unless required by applicable law or agreed to in writing, software > >> +# distributed under the License is distributed on an "AS IS" BASIS, W= ITHOUT > >> +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. Se= e the > >> +# License for the specific language governing permissions and limitat= ions > >> +# under the License. > >> + > >> +wait_start=3D"no" > >> + > >> +while [ "$1" !=3D "" ]; do > >> + if echo "$1" | grep -q -s -E ^--pw-project=3D ; then > >> + pw_project=3D$(echo "$1" | sed s/^--pw-project=3D//) > >> + shift > >> + elif echo "$1" | grep -q -s -E ^--pw-instance=3D ; then > >> + pw_instance=3D$(echo "$1" | sed s/^--pw-instance=3D//) > >> + shift > >> + elif echo "$1" | grep -q -s -E ^--pw-credentials=3D ; then > >> + pw_credential=3D$(echo "$1" | sed s/^--pw-credentials=3D//) > >> + shift > >> + elif echo "$1" | grep -q -s -E ^--series-id=3D ; then > >> + series_id=3D$(echo "$1" | sed s/^--series-id=3D//) > >> + shift > >> + elif echo "$1" | grep -q -s -E ^--wait-start ; then > >> + wait_start=3D"yes" > >> + if echo "$1" | grep -q -s -E ^--wait-start=3D ; then > >> + wait_start=3D$(echo "$1" | sed s/^--wait-start=3D//) > >> + fi > >> + shift > >> + elif echo "$1" | grep -q -s -E ^--github-token=3D ; then > >> + github_token=3D$(echo "$1" | sed s/^--github-token=3D//) > >> + shift > >> + elif echo "$1" | grep -q -s -E ^--repository=3D ; then > >> + reponame=3D$(echo "$1" | sed s/^--repository=3D//) > >> + shift > >> + elif echo "$1" | grep -q -s -E ^--run-id=3D ; then > >> + runid=3D$(echo "$1" | sed s/^--repository=3D//) > >> + shift > >> + elif echo "$1" | grep -q -s -E ^--sha=3D ; then > >> + sha=3D$(echo "$1" | sed s/^--sha=3D//) > >> + shift > >> + elif echo "$1" | grep -q -s -E ^--workflow=3D ; then > >> + workflow=3D$(echo "$1" | sed s/^--workflow=3D//) > >> + shift > >> + elif echo "$1" | grep -q -s -E ^--help ; then > >> + echo "github restarting script" > >> + echo "$0: args" > >> + echo " --pw-project=3D Project name" > >> + echo " --pw-instance=3D URL for pw" > >> + echo " --series-id=3Did Series ID for reference" > >> + echo " --github-token=3Dtoken Token for github" > >> + echo " --wait-start[=3Dyes|no] Wait for the remote = to start" > >> + echo " --repository=3Drepo Repository name (ex = foo/bar)" > >> + echo " --run-id=3Did run ID" > >> + echo " --workflow=3Dworkflow Workflow name" > >> + echo " --sha=3Dcommit Commit sha" > >> + echo "" > >> + exit 0 > >> + else > >> + echo "Unknown option: '$1'" >&2 > >> + echo "Rerun with --help for details" >&2 > >> + exit 1 > >> + fi > >> +done > > This entire block makes me feel like maybe we should switch to python := ) > > Agreed. We are on shell scripts for the time being, but a good effort > would be to re-do things in python and use better frameworks for > everything (for example, passing data between scripts with json rather > than the sqlite output strings directly). I wonder how useful ChatGPT and its siblings would be at converting all of our shell scripts into python :P > > >> + > >> +source $(dirname $0)/series_db_lib.sh > >> + > >> +if [ "X$wait_start" !=3D "Xno" -a "X$wait_start" !=3D "Xyes" ]; then > >> + echo "Unrecognized '$wait_start' argument to --wait-start=3D" >&2 > >> + echo "valid values are 'yes' or 'no'." >&2 > >> + exit 1 > >> +fi > >> + > >> +if [ "X$github_token" =3D=3D "X" ]; then > >> + echo "Please set a github API token." >&2 > >> + echo "Use --help for more info." >&2 > >> + exit 1 > >> +fi > >> + > >> +if [ "X$reponame" =3D=3D "X" ]; then > >> + echo "Please set a repository (ie: '--repository=3Downer/repo')."= >&2 > >> + echo "Use --help for more info." >&2 > >> + exit 1 > >> +fi > >> + > >> +AUTH=3D"Authorization: token ${github_token}" > >> +APP=3D"Accept: application/vnd.github.v3+json" > >> + > >> +if [ "X$runid" =3D=3D "X" ]; then > >> + > >> + # lookup the runs based on the shasum > >> + > >> + if [ "X$sha" =3D=3D "X" ]; then > >> + echo "Need a runid or shasum to key off." >&2 > >> + echo "See --help for more details." >&2 > >> + exit 1 > >> + fi > >> + > >> + comma=3D"" > >> + for job in $(curl -s -S -H "${AUTH}" -H "${APP}" \ > >> + > >> "https://api.github.com/repos/${reponame}/actions/runs?head_sha=3D${sh= a}" > >> | \ > >> + jq -rc '.workflow_runs[].id') > >> + do > >> + runid=3D"${comma}${job}" > >> + comma=3D"," > >> + done > >> +fi > >> + > >> +echo -n "{\"results\":[" > >> +comma=3D"" > >> +for job in $(echo "$runid" | sed 's/,/ /'); do > >> + result=3D$(curl -s -X POST -L -S -H "${AUTH}" -H "${APP}" \ > >> + "https://api.github.com/repos/${reponame}/actions/r= uns/$job/rerun") > > Okay! > >> + msg=3D$(echo "$result" | jq -rc '.message') > >> + > >> + echo -n "$comma{\"run\":$job,\"result\":" > >> + if [ "X$msg" =3D=3D "Xnull" ]; then > >> + echo -n "\"sent\"" > >> + if [ "X$series_id" !=3D "X" ]; then > >> + echo -n ",\"gap_sync\":\"reset\"" > >> + set_unsynced_for_series "$series_id" "$pw_instance" "gap_= sync" > >> + fi > >> + else > >> + echo -n "\"err\",\"error\":\"$msg\"" > >> + fi > >> + echo -n "}" > >> +done > >> +echo "]}" > >> diff --git a/series_db_lib.sh b/series_db_lib.sh > >> index ca33c1f..3f052ad 100644 > >> --- a/series_db_lib.sh > >> +++ b/series_db_lib.sh > >> @@ -382,6 +382,14 @@ function set_synced_for_series() { > >> echo "update git_builds set gap_sync=3D1, obs_sync=3D1 where patc= hwork_instance=3D\"$instance\" and series_id=3D$series_id;" | series_db_exe= cute > >> } > >> > >> +function set_unsynced_for_series() { > >> + local series_id=3D"$1" > >> + local instance=3D"$2" > >> + local ci_instance=3D"$3" > >> + > >> + echo "update git_builds set $ci_instance=3D0 where patchwork_inst= ance=3D\"$instance\" and series_id=3D$series_id;" | series_db_execute > >> +} > >> + > >> function insert_commit() { > >> local series_id=3D"$1" > >> local patch_id=3D"$2" > >> @@ -407,6 +415,14 @@ function get_patch_id_by_series_id_and_sha() { > >> echo "select patch_id from git_builds where patchwork_instance=3D= \"$instance\" and series_id=3D$series_id and sha=3D\"$sha\";" | series_db_e= xecute > >> } > >> > >> +function get_sha_for_series_id_and_patch() { > > Is this function being used? > > Yes - but it should have appeared in 2/3 rather than here in 3/3. I'll > fix it before applying things and I plan to send a PATCH series out with > those changes as well. > > >> + local series_id=3D"$1" > >> + local patch_id=3D"$2" > >> + local instance=3D"$3" > >> + > >> + echo "select sha from git_builds where patchwork_instance=3D\"$in= stance\" and series_id=3D\"$series_id\" and patch_id=3D\"$patch_id\"" | ser= ies_db_execute > >> +} > >> + > >> function get_recheck_requests_by_project() { > >> local recheck_instance=3D"$1" > >> local recheck_project=3D"$2" > >> -- > >> 2.41.0 > >> >