From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3094D43358; Fri, 17 Nov 2023 20:28:54 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2AC6F40285; Fri, 17 Nov 2023 20:28:54 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id BCA864021E for ; Fri, 17 Nov 2023 20:28:52 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700249332; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rylF2oFM+HHgOeCJbqr0kxKpWmgueGSB3TK8fyl7JQY=; b=FfJdquDsM0qUIvISxeJjK/wHE6JF4v66L5UeoaoCtib9xtH1cnrfHQNjzft3ojxVi8mtr7 STNIWdkCTJOMFtaVl5EHa7/a3J0/0gkHepE5OLZdCjDwUGsA2WA9QkVCJ/NmhiDPLIJmVr 9WpijzCe+CuXmvl/7i060/l1eYNUjsQ= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-209-gT_8niVGPSKJVH_r_7OpKg-1; Fri, 17 Nov 2023 14:28:46 -0500 X-MC-Unique: gT_8niVGPSKJVH_r_7OpKg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EDAE83C0BE47; Fri, 17 Nov 2023 19:28:45 +0000 (UTC) Received: from RHTPC1VM0NT (unknown [10.22.34.37]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 84BBEC15881; Fri, 17 Nov 2023 19:28:45 +0000 (UTC) From: Aaron Conole To: Michael Santana Cc: ci@dpdk.org, David Marchand , Thomas Monjalon , Patrick Robb , Dumitru Ceara Subject: Re: [RFC v2 2/3] recheck: Add a recheck parser for patchwork comments References: <20231107203158.1261199-1-aconole@redhat.com> <20231107203158.1261199-3-aconole@redhat.com> Date: Fri, 17 Nov 2023 14:28:45 -0500 In-Reply-To: (Michael Santana's message of "Fri, 17 Nov 2023 11:35:50 -0500") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.3 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Michael Santana writes: > On Tue, Nov 7, 2023 at 3:32=E2=80=AFPM Aaron Conole = wrote: >> >> Add a recheck parsing tool that will allow for labs to build a >> recheck workflow based on specific recheck labels and projects, >> with an associated state machine and querying interface. >> >> The output of the recheck parsing tool is json and can be fed to >> jq or other json parsing utilities for better field support. >> >> Signed-off-by: Aaron Conole > Acked-by: Michael Santana Thanks >> --- >> pw_mon | 58 ++++++++++++++++++++++++++- >> recheck_tool | 100 +++++++++++++++++++++++++++++++++++++++++++++++ >> series_db_lib.sh | 64 +++++++++++++++++++++++++++++- >> 3 files changed, 220 insertions(+), 2 deletions(-) >> create mode 100755 recheck_tool >> >> diff --git a/pw_mon b/pw_mon >> index da4b9a9..29a9cb4 100755 >> --- a/pw_mon >> +++ b/pw_mon >> @@ -1,6 +1,6 @@ >> #!/bin/sh >> # SPDX-Identifier: gpl-2.0-or-later >> -# Copyright (C) 2018, Red Hat, Inc. >> +# Copyright (C) 2018-2023 Red Hat, Inc. >> # >> # Monitors a project on a patchwork instance for new series submissions >> # Records the submissions in the series database (and emits them on the >> @@ -44,6 +44,7 @@ if [ "$1" !=3D "" ]; then >> fi >> fi >> >> +recheck_filter=3D"" >> >> while [ "$1" !=3D "" ]; do >> if echo "$1" | grep -q -s -E ^--pw-project=3D ; then >> @@ -67,6 +68,10 @@ while [ "$1" !=3D "" ]; do >> echo " --add-filter-recheck=3Dfilter Adds a filter to flag = that a recheck needs to be done" >> echo "" >> exit 0 >> + elif echo "$1" | grep -q -s -E ^--add-filter-recheck=3D; then >> + filter_str=3D$(echo "$1" | sed s/--add-filter-recheck=3D//) >> + recheck_filter=3D"$filter_str $recheck_filter" >> + shift >> else >> echo "Unknown option: '$1'" >> echo "Rerun with --help for details" >> @@ -191,7 +196,58 @@ function check_superseded_series() { >> done >> } >> >> +function run_recheck() { >> + local recheck_list=3D$(echo "$7" | sed -e 's/^Recheck-request: // '= -e 's/,/ /g') >> + >> + for filter in $recheck_filter; do >> + for check in $recheck_list; do >> + if [ "$filter" =3D=3D "$check" ]; then >> + insert_recheck_request_if_needed "$1" "$3" "$8" "$check= " "$2" "$9" >> + fi >> + done >> + done >> +} >> + >> +function check_patch_for_retest_request() { >> + local patch_url=3D"$1" >> + >> + local patch_json=3D$(curl -s "$userpw" "$patch_url") >> + local patch_comments_url=3D$(echo "$patch_json" | jq -rc '.comments= ') >> + local patch_id=3D$(echo "$patch_json" | jq -rc '.id') >> + if [ "Xnull" !=3D "X$patch_comments_url" ]; then >> + local comments_json=3D$(curl -s "$userpw" "$patch_comments_url"= ) >> + >> + local seq_end=3D$(echo "$comments_json" | jq -rc 'length') >> + if [ "$seq_end" -a $seq_end -gt 0 ]; then >> + seq_end=3D$((seq_end-1)) >> + for comment_id in $(seq 0 $seq_end); do >> + local recheck_requested=3D$(echo "$comments_json" | jq = -rc ".[$comment_id].content" | grep "^Recheck-request: ") >> + if [ "X$recheck_requested" !=3D "X" ]; then >> + local msgid=3D$(echo "$comments_json" | jq -rc ".[$= comment_id].msgid") >> + run_recheck "$pw_instance" "$series_id" "$project" = "$url" "$repo" "$branchname" "$recheck_requested" "$msgid" "$patch_id" >> + fi >> + done >> + fi >> + fi >> +} >> + >> +function check_series_needs_retest() { >> + local pw_instance=3D"$1" >> + >> + series_get_active_branches "$pw_instance" | while IFS=3D\| read -r = series_id project url repo branchname; do >> + local patch_comments_url=3D$(curl -s "$userpw" "$url" | jq -rc = '.patches[] | .url') >> + >> + for patch in $patch_comments_url; do >> + check_patch_for_retest_request $patch >> + done >> + done >> +} >> + >> check_undownloaded_series "$pw_instance" "$pw_project" >> check_completed_series "$pw_instance" "$pw_project" >> check_new_series "$pw_instance" "$pw_project" >> check_superseded_series "$pw_instance" >> + >> +# check for retest requests after a series is still passing all the >> +# checks above >> +check_series_needs_retest "$pw_instance" >> diff --git a/recheck_tool b/recheck_tool >> new file mode 100755 >> index 0000000..98031a9 >> --- /dev/null >> +++ b/recheck_tool >> @@ -0,0 +1,100 @@ >> +#!/bin/sh >> +# SPDX-Identifier: gpl-2.0-or-later >> +# Copyright (C) 2023 Red Hat, Inc. >> +# >> +# Licensed under the terms of the GNU General Public License as publish= ed >> +# by the Free Software Foundation; either version 2 of the License, or >> +# (at your option) any later version. You may obtain a copy of the >> +# license at >> +# >> +# https://www.gnu.org/licenses/old-licenses/gpl-2.0.html >> +# >> +# Unless required by applicable law or agreed to in writing, software >> +# distributed under the License is distributed on an "AS IS" BASIS, WIT= HOUT >> +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See = the >> +# License for the specific language governing permissions and limitatio= ns >> +# under the License. >> + >> +mode=3D"select" >> + >> +while [ "$1" !=3D "" ]; do >> + if echo "$1" | grep -q -s -E ^--help ; then >> + echo "recheck / retest state machine script" >> + echo "" >> + echo "$0:" >> + echo " --pw-project=3D: Patchwork project." >> + echo " --pw-instance=3D: Patchwork instance." >> + echo " --filter=3D: Job / request for recheck." >> + echo " --state=3D<0..>: Resync state ID." >> + echo " --msgid=3D: Message ID to select." >> + echo " --update: Set tool in update mode" >> + echo " --new-state=3D<0..>: New state ID to set" >> + echo " --series-id=3D<..>: Series ID" >> + echo "" >> + echo "Will spit out a parsable json for each db line when selec= ting" > LGTM. But to Dumitru's point, maybe we should add documentation this > this in a follow up patch >> + exit 0 >> + elif echo "$1" | grep -q -s -E ^--pw-project=3D ; then >> + pw_project=3D$(echo "$1" | sed s/^--pw-project=3D//) >> + elif echo "$1" | grep -q -s -E ^--pw-instance=3D ; then >> + pw_instance=3D$(echo "$1" | sed s/^--pw-instance=3D//) >> + elif echo "$1" | grep -q -s -E ^--filter=3D ; then >> + filter=3D$(echo "$1" | sed s/^--filter=3D//) >> + elif echo "$1" | grep -q -s -E ^--state=3D ; then >> + recheck_state=3D$(echo "$1" | sed s/^--state=3D//) >> + elif echo "$1" | grep -q -s -E ^--msgid=3D ; then >> + message_id=3D$(echo "$1" | sed s/^--msgid=3D//) >> + elif echo "$1" | grep -q -s -E ^--update ; then >> + mode=3D"update" >> + elif echo "$1" | grep -q -s -E ^--new-state=3D ; then >> + new_recheck_state=3D$(echo "$1" | sed s/^--new-state=3D//) >> + elif echo "$1" | grep -q -s -E ^--series-id=3D ; then >> + series_id=3D$(echo "$1" | sed s/^--series-id=3D//) >> + else >> + echo "Unknown option: '$1'" >> + echo "Rerun with --help for details" >> + exit 1 >> + fi >> + shift >> +done >> + >> +source $(dirname $0)/series_db_lib.sh >> + >> +if [ "$mode" =3D=3D "select" ]; then >> + echo -n "{\"rechecks\":[" >> + printed=3D"" >> + for request in $(get_recheck_requests_by_project "$pw_instance" \ >> + "$pw_project" \ >> + "$recheck_state" \ >> + "$filter"); do >> + message_id=3D$(echo $request | cut -d\| -f1) >> + series_id=3D$(echo $request | cut -d\| -f2) >> + patch_id=3D$(echo $request | cut -d\| -f3) >> + >> + sha=3D$(get_sha_for_series_id_and_patch "$series_id" "$patch_id= " \ >> + "$pw_instance") >> + echo -n >> "$printed{\"pw_instance\":\"$pw_instance\",\"series_id\":$series_id,\"pa= tch_id\":$patch_id,\"sha\":\"$sha\",\"msg_id\":\"$message_id\",\"state\":\"= $recheck_state\",\"requested\":\"$filter\"}" > okay, smart use of the -n for the echo command. I see you are also > taking care of the coma which is good! >> + printed=3D"," >> + done >> + echo "]}" >> +elif [ "$mode" =3D=3D "update" ]; then >> + if [ "X$new_recheck_state" =3D=3D "X" -o "X$series_id" =3D=3D "X" ]= ; then >> + echo "Need to set a series-id and a new recheck state when upda= ting." >> + exit 1 >> + fi >> + >> + request=3D$(get_recheck_request "$pw_instance" "$pw_project" "$mess= age_id" \ >> + "$filter" "$series_id" "$recheck_stat= e") >> + if [ "X$request" =3D=3D "X" ]; then >> + echo "{\"result\":\"notfound\"}" >> + exit 0 >> + fi >> + >> + set_recheck_request_state "$pw_instance" "$pw_project" "$message_id= " \ >> + "$filter" "$series_id" "$new_recheck_stat= e" >> + >> + echo "{\"result\":\"executed\",\"recheck\":{\"pw_instance\": >> \"$pw_instance\", \"series_id\":$series_id, >> \"msg_id\":\"$message_id\", \"state\":\"$new_recheck_state\", >> \"requested\": \"$filter\"}}" > okay >> +else >> + echo "Unknown state: $mode" >> + exit 1 >> +fi >> + >> diff --git a/series_db_lib.sh b/series_db_lib.sh >> index 6c2d98e..ca33c1f 100644 >> --- a/series_db_lib.sh >> +++ b/series_db_lib.sh >> @@ -1,6 +1,6 @@ >> #!/bin/sh >> # SPDX-Identifier: gpl-2.0-or-later >> -# Copyright (C) 2018,2019 Red Hat, Inc. >> +# Copyright (C) 2018-2023 Red Hat, Inc. >> # >> # Licensed under the terms of the GNU General Public License as publish= ed >> # by the Free Software Foundation; either version 2 of the License, or >> @@ -114,6 +114,22 @@ EOF >> run_db_command "INSERT INTO series_schema_version(id) values (7= );" >> fi >> >> + run_db_command "select * from series_schema_version;" | egrep '^8$'= > /dev/null 2>&1 >> + if [ $? -eq 1 ]; then >> + sqlite3 ${HOME}/.series-db <> +CREATE TABLE recheck_requests ( >> +recheck_id INTEGER, >> +recheck_message_id STRING, >> +recheck_requested_by STRING, >> +recheck_series STRING, >> +recheck_patch INTEGER, >> +patchwork_instance STRING, >> +patchwork_project STRING, >> +recheck_sync INTEGER >> +); >> +EOF >> + run_db_command "INSERT INTO series_schema_version(id) values (8= );" >> + fi >> } >> >> function series_db_exists() { >> @@ -390,3 +406,49 @@ function get_patch_id_by_series_id_and_sha() { >> >> echo "select patch_id from git_builds where patchwork_instance=3D\"= $instance\" and series_id=3D$series_id and sha=3D\"$sha\";" | series_db_exe= cute >> } >> + >> +function get_recheck_requests_by_project() { >> + local recheck_instance=3D"$1" >> + local recheck_project=3D"$2" >> + local recheck_state=3D"$3" >> + local recheck_requested_by=3D"$4" >> + >> + series_db_exists >> + >> + echo "select recheck_message_id,recheck_series,recheck_patch from r= echeck_requests where patchwork_instance=3D\"$recheck_instance\" and patchw= ork_project=3D\"$recheck_project\" and recheck_sync=3D$recheck_state and re= check_requested_by=3D\"$recheck_requested_by\";" | series_db_execute >> +} >> + >> +function insert_recheck_request_if_needed() { >> + local recheck_instance=3D"$1" >> + local recheck_project=3D"$2" >> + local recheck_msgid=3D"$3" >> + local recheck_requested_by=3D"$4" >> + local recheck_series=3D"$5" >> + local recheck_patch=3D"$6" >> + >> + if ! echo "select * from recheck_requests where recheck_message_id= =3D\"$recheck_msgid\";" | series_db_execute | grep $recheck_msgid >/dev/nul= l 2>&1; then >> + echo "INSERT INTO recheck_requests (recheck_message_id, recheck= _requested_by, recheck_series, recheck_patch, patchwork_instance, patchwork= _project, recheck_sync) values (\"$recheck_msgid\", \"$recheck_requested_by= \", \"$recheck_series\", $recheck_patch, \"$recheck_instance\", \"$recheck_= project\", 0);" | series_db_execute >> + fi >> +} >> + >> +function get_recheck_request() { >> + local recheck_instance=3D"$1" >> + local recheck_project=3D"$2" >> + local recheck_msgid=3D"$3" >> + local recheck_requested_by=3D"$4" >> + local recheck_series=3D"$5" >> + local recheck_state=3D"$6" >> + >> + echo "select * from recheck_requests where patchwork_instance=3D\"$= recheck_instance\" and patchwork_project=3D\"$recheck_project\" and recheck= _requested_by=3D\"$recheck_requested_by\" and recheck_series=3D\"$recheck_s= eries\" and recheck_message_id=3D\"$recheck_msgid\" and recheck_sync=3D$rec= heck_state;" | series_db_execute >> +} >> + >> +function set_recheck_request_state() { >> + local recheck_instance=3D"$1" >> + local recheck_project=3D"$2" >> + local recheck_msgid=3D"$3" >> + local recheck_requested_by=3D"$4" >> + local recheck_series=3D"$5" >> + local recheck_state=3D"$6" >> + >> + echo "UPDATE recheck_requests set recheck_sync=3D$recheck_state whe= re patchwork_instance=3D\"$recheck_instance\" and patchwork_project=3D\"$re= check_project\" and recheck_requested_by=3D\"$recheck_requested_by\" and re= check_series=3D\"$recheck_series\";" | series_db_execute >> +} >> -- >> 2.41.0 >>