From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6846D43355; Fri, 17 Nov 2023 17:36:07 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 60D47402EA; Fri, 17 Nov 2023 17:36:07 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 1F26640285 for ; Fri, 17 Nov 2023 17:36:06 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700238965; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HmDVk1P8kNiY7W1PvXVfersXAz9sK4N7/SetHmQjZXY=; b=V8q5aKxdoyVlmwiEeeZhhuA8G6E5Dy7AApYw1/dI/sp34SGLC32nacuFDdkvOdMg174D+9 TuHZGgvUqk0Lzovsz4g7jiojbgQNvJPMbWwFPItFDI2zlDQTkTM7VUEqA8tdy40zn49Wkq 19UugPfERnO3nHhBW+YQiAA0PfG+8fI= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-30-dx7thw_VPZWx06_DMljkoA-1; Fri, 17 Nov 2023 11:36:04 -0500 X-MC-Unique: dx7thw_VPZWx06_DMljkoA-1 Received: by mail-ed1-f69.google.com with SMTP id 4fb4d7f45d1cf-53e26bbbc63so1687726a12.3 for ; Fri, 17 Nov 2023 08:36:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700238963; x=1700843763; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HmDVk1P8kNiY7W1PvXVfersXAz9sK4N7/SetHmQjZXY=; b=CkgkP5+RMwnLPGNfNjgAddG1hM1x1v4DJbfKssAf6eq0vopgpa8oNOY1OrRe8j5RUT x14by4UEY+Du3m7q0Sp/Z2T8JWdJI18QPz59LgpFzb/geHyLR2JbP0GGTw2u5sAQFiKw I04xXWhmPNfYxAcv94GzKqO33nQ1g/51/dAvBa/2o9m9f7FQHCjzz1lIhbVcvOp9FGAN zuP6cbRqBWC5s2BiQirf0wIEYdoEDU9b0U/iYNaLsLPy+kRsqFnmn03I+PeK8dGKxzp5 4ARWljxoIStN3zPJbSxeR3/j41EBh0JKJ/3z7G3yGpvMUeEFZBCdkUwQG7ElK2SA+PgL PQxg== X-Gm-Message-State: AOJu0YzPSMIr7hfSaFhQbvmORQQ9AQttjOAwaV6UOp+Yu/kbp7DLdkgx hb+1et1jz0bCB8Z5eli2cpAE4bSqF7b/by0oJLp/VIOYi8t/dsFBXAsDwlMwyJjeW3XdpDWIvYl Bi343c82ppHo9WsCrQA== X-Received: by 2002:aa7:cc8b:0:b0:548:5272:db61 with SMTP id p11-20020aa7cc8b000000b005485272db61mr988145edt.15.1700238963058; Fri, 17 Nov 2023 08:36:03 -0800 (PST) X-Google-Smtp-Source: AGHT+IGzdlPZKDBAHH7UoCznLKqh7U+8SnoNCdrVaxzRYYWU/xBaACE72f7Ne7I5R0s+IB9u2dt407LVmPMrQ/hx/h8= X-Received: by 2002:aa7:cc8b:0:b0:548:5272:db61 with SMTP id p11-20020aa7cc8b000000b005485272db61mr988125edt.15.1700238962684; Fri, 17 Nov 2023 08:36:02 -0800 (PST) MIME-Version: 1.0 References: <20231107203158.1261199-1-aconole@redhat.com> <20231107203158.1261199-3-aconole@redhat.com> In-Reply-To: <20231107203158.1261199-3-aconole@redhat.com> From: Michael Santana Date: Fri, 17 Nov 2023 11:35:50 -0500 Message-ID: Subject: Re: [RFC v2 2/3] recheck: Add a recheck parser for patchwork comments To: Aaron Conole Cc: ci@dpdk.org, David Marchand , Thomas Monjalon , Patrick Robb , Dumitru Ceara X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org On Tue, Nov 7, 2023 at 3:32=E2=80=AFPM Aaron Conole wr= ote: > > Add a recheck parsing tool that will allow for labs to build a > recheck workflow based on specific recheck labels and projects, > with an associated state machine and querying interface. > > The output of the recheck parsing tool is json and can be fed to > jq or other json parsing utilities for better field support. > > Signed-off-by: Aaron Conole Acked-by: Michael Santana > --- > pw_mon | 58 ++++++++++++++++++++++++++- > recheck_tool | 100 +++++++++++++++++++++++++++++++++++++++++++++++ > series_db_lib.sh | 64 +++++++++++++++++++++++++++++- > 3 files changed, 220 insertions(+), 2 deletions(-) > create mode 100755 recheck_tool > > diff --git a/pw_mon b/pw_mon > index da4b9a9..29a9cb4 100755 > --- a/pw_mon > +++ b/pw_mon > @@ -1,6 +1,6 @@ > #!/bin/sh > # SPDX-Identifier: gpl-2.0-or-later > -# Copyright (C) 2018, Red Hat, Inc. > +# Copyright (C) 2018-2023 Red Hat, Inc. > # > # Monitors a project on a patchwork instance for new series submissions > # Records the submissions in the series database (and emits them on the > @@ -44,6 +44,7 @@ if [ "$1" !=3D "" ]; then > fi > fi > > +recheck_filter=3D"" > > while [ "$1" !=3D "" ]; do > if echo "$1" | grep -q -s -E ^--pw-project=3D ; then > @@ -67,6 +68,10 @@ while [ "$1" !=3D "" ]; do > echo " --add-filter-recheck=3Dfilter Adds a filter to flag t= hat a recheck needs to be done" > echo "" > exit 0 > + elif echo "$1" | grep -q -s -E ^--add-filter-recheck=3D; then > + filter_str=3D$(echo "$1" | sed s/--add-filter-recheck=3D//) > + recheck_filter=3D"$filter_str $recheck_filter" > + shift > else > echo "Unknown option: '$1'" > echo "Rerun with --help for details" > @@ -191,7 +196,58 @@ function check_superseded_series() { > done > } > > +function run_recheck() { > + local recheck_list=3D$(echo "$7" | sed -e 's/^Recheck-request: // ' = -e 's/,/ /g') > + > + for filter in $recheck_filter; do > + for check in $recheck_list; do > + if [ "$filter" =3D=3D "$check" ]; then > + insert_recheck_request_if_needed "$1" "$3" "$8" "$check"= "$2" "$9" > + fi > + done > + done > +} > + > +function check_patch_for_retest_request() { > + local patch_url=3D"$1" > + > + local patch_json=3D$(curl -s "$userpw" "$patch_url") > + local patch_comments_url=3D$(echo "$patch_json" | jq -rc '.comments'= ) > + local patch_id=3D$(echo "$patch_json" | jq -rc '.id') > + if [ "Xnull" !=3D "X$patch_comments_url" ]; then > + local comments_json=3D$(curl -s "$userpw" "$patch_comments_url") > + > + local seq_end=3D$(echo "$comments_json" | jq -rc 'length') > + if [ "$seq_end" -a $seq_end -gt 0 ]; then > + seq_end=3D$((seq_end-1)) > + for comment_id in $(seq 0 $seq_end); do > + local recheck_requested=3D$(echo "$comments_json" | jq -= rc ".[$comment_id].content" | grep "^Recheck-request: ") > + if [ "X$recheck_requested" !=3D "X" ]; then > + local msgid=3D$(echo "$comments_json" | jq -rc ".[$c= omment_id].msgid") > + run_recheck "$pw_instance" "$series_id" "$project" "= $url" "$repo" "$branchname" "$recheck_requested" "$msgid" "$patch_id" > + fi > + done > + fi > + fi > +} > + > +function check_series_needs_retest() { > + local pw_instance=3D"$1" > + > + series_get_active_branches "$pw_instance" | while IFS=3D\| read -r s= eries_id project url repo branchname; do > + local patch_comments_url=3D$(curl -s "$userpw" "$url" | jq -rc '= .patches[] | .url') > + > + for patch in $patch_comments_url; do > + check_patch_for_retest_request $patch > + done > + done > +} > + > check_undownloaded_series "$pw_instance" "$pw_project" > check_completed_series "$pw_instance" "$pw_project" > check_new_series "$pw_instance" "$pw_project" > check_superseded_series "$pw_instance" > + > +# check for retest requests after a series is still passing all the > +# checks above > +check_series_needs_retest "$pw_instance" > diff --git a/recheck_tool b/recheck_tool > new file mode 100755 > index 0000000..98031a9 > --- /dev/null > +++ b/recheck_tool > @@ -0,0 +1,100 @@ > +#!/bin/sh > +# SPDX-Identifier: gpl-2.0-or-later > +# Copyright (C) 2023 Red Hat, Inc. > +# > +# Licensed under the terms of the GNU General Public License as publishe= d > +# by the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. You may obtain a copy of the > +# license at > +# > +# https://www.gnu.org/licenses/old-licenses/gpl-2.0.html > +# > +# Unless required by applicable law or agreed to in writing, software > +# distributed under the License is distributed on an "AS IS" BASIS, WITH= OUT > +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See t= he > +# License for the specific language governing permissions and limitation= s > +# under the License. > + > +mode=3D"select" > + > +while [ "$1" !=3D "" ]; do > + if echo "$1" | grep -q -s -E ^--help ; then > + echo "recheck / retest state machine script" > + echo "" > + echo "$0:" > + echo " --pw-project=3D: Patchwork project." > + echo " --pw-instance=3D: Patchwork instance." > + echo " --filter=3D: Job / request for recheck." > + echo " --state=3D<0..>: Resync state ID." > + echo " --msgid=3D: Message ID to select." > + echo " --update: Set tool in update mode" > + echo " --new-state=3D<0..>: New state ID to set" > + echo " --series-id=3D<..>: Series ID" > + echo "" > + echo "Will spit out a parsable json for each db line when select= ing" LGTM. But to Dumitru's point, maybe we should add documentation this this in a follow up patch > + exit 0 > + elif echo "$1" | grep -q -s -E ^--pw-project=3D ; then > + pw_project=3D$(echo "$1" | sed s/^--pw-project=3D//) > + elif echo "$1" | grep -q -s -E ^--pw-instance=3D ; then > + pw_instance=3D$(echo "$1" | sed s/^--pw-instance=3D//) > + elif echo "$1" | grep -q -s -E ^--filter=3D ; then > + filter=3D$(echo "$1" | sed s/^--filter=3D//) > + elif echo "$1" | grep -q -s -E ^--state=3D ; then > + recheck_state=3D$(echo "$1" | sed s/^--state=3D//) > + elif echo "$1" | grep -q -s -E ^--msgid=3D ; then > + message_id=3D$(echo "$1" | sed s/^--msgid=3D//) > + elif echo "$1" | grep -q -s -E ^--update ; then > + mode=3D"update" > + elif echo "$1" | grep -q -s -E ^--new-state=3D ; then > + new_recheck_state=3D$(echo "$1" | sed s/^--new-state=3D//) > + elif echo "$1" | grep -q -s -E ^--series-id=3D ; then > + series_id=3D$(echo "$1" | sed s/^--series-id=3D//) > + else > + echo "Unknown option: '$1'" > + echo "Rerun with --help for details" > + exit 1 > + fi > + shift > +done > + > +source $(dirname $0)/series_db_lib.sh > + > +if [ "$mode" =3D=3D "select" ]; then > + echo -n "{\"rechecks\":[" > + printed=3D"" > + for request in $(get_recheck_requests_by_project "$pw_instance" \ > + "$pw_project" \ > + "$recheck_state" \ > + "$filter"); do > + message_id=3D$(echo $request | cut -d\| -f1) > + series_id=3D$(echo $request | cut -d\| -f2) > + patch_id=3D$(echo $request | cut -d\| -f3) > + > + sha=3D$(get_sha_for_series_id_and_patch "$series_id" "$patch_id"= \ > + "$pw_instance") > + echo -n "$printed{\"pw_instance\":\"$pw_instance\",\"series_id\"= :$series_id,\"patch_id\":$patch_id,\"sha\":\"$sha\",\"msg_id\":\"$message_i= d\",\"state\":\"$recheck_state\",\"requested\":\"$filter\"}" okay, smart use of the -n for the echo command. I see you are also taking care of the coma which is good! > + printed=3D"," > + done > + echo "]}" > +elif [ "$mode" =3D=3D "update" ]; then > + if [ "X$new_recheck_state" =3D=3D "X" -o "X$series_id" =3D=3D "X" ];= then > + echo "Need to set a series-id and a new recheck state when updat= ing." > + exit 1 > + fi > + > + request=3D$(get_recheck_request "$pw_instance" "$pw_project" "$messa= ge_id" \ > + "$filter" "$series_id" "$recheck_state= ") > + if [ "X$request" =3D=3D "X" ]; then > + echo "{\"result\":\"notfound\"}" > + exit 0 > + fi > + > + set_recheck_request_state "$pw_instance" "$pw_project" "$message_id"= \ > + "$filter" "$series_id" "$new_recheck_state= " > + > + echo "{\"result\":\"executed\",\"recheck\":{\"pw_instance\": \"$pw_i= nstance\", \"series_id\":$series_id, \"msg_id\":\"$message_id\", \"state\":= \"$new_recheck_state\", \"requested\": \"$filter\"}}" okay > +else > + echo "Unknown state: $mode" > + exit 1 > +fi > + > diff --git a/series_db_lib.sh b/series_db_lib.sh > index 6c2d98e..ca33c1f 100644 > --- a/series_db_lib.sh > +++ b/series_db_lib.sh > @@ -1,6 +1,6 @@ > #!/bin/sh > # SPDX-Identifier: gpl-2.0-or-later > -# Copyright (C) 2018,2019 Red Hat, Inc. > +# Copyright (C) 2018-2023 Red Hat, Inc. > # > # Licensed under the terms of the GNU General Public License as publishe= d > # by the Free Software Foundation; either version 2 of the License, or > @@ -114,6 +114,22 @@ EOF > run_db_command "INSERT INTO series_schema_version(id) values (7)= ;" > fi > > + run_db_command "select * from series_schema_version;" | egrep '^8$' = > /dev/null 2>&1 > + if [ $? -eq 1 ]; then > + sqlite3 ${HOME}/.series-db < +CREATE TABLE recheck_requests ( > +recheck_id INTEGER, > +recheck_message_id STRING, > +recheck_requested_by STRING, > +recheck_series STRING, > +recheck_patch INTEGER, > +patchwork_instance STRING, > +patchwork_project STRING, > +recheck_sync INTEGER > +); > +EOF > + run_db_command "INSERT INTO series_schema_version(id) values (8)= ;" > + fi > } > > function series_db_exists() { > @@ -390,3 +406,49 @@ function get_patch_id_by_series_id_and_sha() { > > echo "select patch_id from git_builds where patchwork_instance=3D\"$= instance\" and series_id=3D$series_id and sha=3D\"$sha\";" | series_db_exec= ute > } > + > +function get_recheck_requests_by_project() { > + local recheck_instance=3D"$1" > + local recheck_project=3D"$2" > + local recheck_state=3D"$3" > + local recheck_requested_by=3D"$4" > + > + series_db_exists > + > + echo "select recheck_message_id,recheck_series,recheck_patch from re= check_requests where patchwork_instance=3D\"$recheck_instance\" and patchwo= rk_project=3D\"$recheck_project\" and recheck_sync=3D$recheck_state and rec= heck_requested_by=3D\"$recheck_requested_by\";" | series_db_execute > +} > + > +function insert_recheck_request_if_needed() { > + local recheck_instance=3D"$1" > + local recheck_project=3D"$2" > + local recheck_msgid=3D"$3" > + local recheck_requested_by=3D"$4" > + local recheck_series=3D"$5" > + local recheck_patch=3D"$6" > + > + if ! echo "select * from recheck_requests where recheck_message_id= =3D\"$recheck_msgid\";" | series_db_execute | grep $recheck_msgid >/dev/nul= l 2>&1; then > + echo "INSERT INTO recheck_requests (recheck_message_id, recheck_= requested_by, recheck_series, recheck_patch, patchwork_instance, patchwork_= project, recheck_sync) values (\"$recheck_msgid\", \"$recheck_requested_by\= ", \"$recheck_series\", $recheck_patch, \"$recheck_instance\", \"$recheck_p= roject\", 0);" | series_db_execute > + fi > +} > + > +function get_recheck_request() { > + local recheck_instance=3D"$1" > + local recheck_project=3D"$2" > + local recheck_msgid=3D"$3" > + local recheck_requested_by=3D"$4" > + local recheck_series=3D"$5" > + local recheck_state=3D"$6" > + > + echo "select * from recheck_requests where patchwork_instance=3D\"$r= echeck_instance\" and patchwork_project=3D\"$recheck_project\" and recheck_= requested_by=3D\"$recheck_requested_by\" and recheck_series=3D\"$recheck_se= ries\" and recheck_message_id=3D\"$recheck_msgid\" and recheck_sync=3D$rech= eck_state;" | series_db_execute > +} > + > +function set_recheck_request_state() { > + local recheck_instance=3D"$1" > + local recheck_project=3D"$2" > + local recheck_msgid=3D"$3" > + local recheck_requested_by=3D"$4" > + local recheck_series=3D"$5" > + local recheck_state=3D"$6" > + > + echo "UPDATE recheck_requests set recheck_sync=3D$recheck_state wher= e patchwork_instance=3D\"$recheck_instance\" and patchwork_project=3D\"$rec= heck_project\" and recheck_requested_by=3D\"$recheck_requested_by\" and rec= heck_series=3D\"$recheck_series\";" | series_db_execute > +} > -- > 2.41.0 >