From: Dumitru Ceara <dceara@redhat.com>
To: Aaron Conole <aconole@redhat.com>, ci@dpdk.org
Cc: Michael Santana <msantana@redhat.com>,
David Marchand <dmarchan@redhat.com>,
Thomas Monjalon <thomas@monjalon.net>,
Patrick Robb <probb@iol.unh.edu>
Subject: Re: [RFC v2 2/3] recheck: Add a recheck parser for patchwork comments
Date: Thu, 9 Nov 2023 15:45:09 -0500 [thread overview]
Message-ID: <eb9c7d7c-60f9-4872-b61f-e61ca665a970@redhat.com> (raw)
In-Reply-To: <20231107203158.1261199-3-aconole@redhat.com>
On 11/7/23 21:31, Aaron Conole wrote:
> Add a recheck parsing tool that will allow for labs to build a
> recheck workflow based on specific recheck labels and projects,
> with an associated state machine and querying interface.
>
> The output of the recheck parsing tool is json and can be fed to
> jq or other json parsing utilities for better field support.
>
> Signed-off-by: Aaron Conole <aconole@redhat.com>
> ---
Awesome, OVN really needs this kind of recheck capability in patchwork.
One comment inline.
> pw_mon | 58 ++++++++++++++++++++++++++-
> recheck_tool | 100 +++++++++++++++++++++++++++++++++++++++++++++++
> series_db_lib.sh | 64 +++++++++++++++++++++++++++++-
> 3 files changed, 220 insertions(+), 2 deletions(-)
> create mode 100755 recheck_tool
>
> diff --git a/pw_mon b/pw_mon
> index da4b9a9..29a9cb4 100755
> --- a/pw_mon
> +++ b/pw_mon
> @@ -1,6 +1,6 @@
> #!/bin/sh
> # SPDX-Identifier: gpl-2.0-or-later
> -# Copyright (C) 2018, Red Hat, Inc.
> +# Copyright (C) 2018-2023 Red Hat, Inc.
> #
> # Monitors a project on a patchwork instance for new series submissions
> # Records the submissions in the series database (and emits them on the
> @@ -44,6 +44,7 @@ if [ "$1" != "" ]; then
> fi
> fi
>
> +recheck_filter=""
>
> while [ "$1" != "" ]; do
> if echo "$1" | grep -q -s -E ^--pw-project= ; then
> @@ -67,6 +68,10 @@ while [ "$1" != "" ]; do
> echo " --add-filter-recheck=filter Adds a filter to flag that a recheck needs to be done"
> echo ""
> exit 0
> + elif echo "$1" | grep -q -s -E ^--add-filter-recheck=; then
> + filter_str=$(echo "$1" | sed s/--add-filter-recheck=//)
> + recheck_filter="$filter_str $recheck_filter"
> + shift
> else
> echo "Unknown option: '$1'"
> echo "Rerun with --help for details"
> @@ -191,7 +196,58 @@ function check_superseded_series() {
> done
> }
>
> +function run_recheck() {
> + local recheck_list=$(echo "$7" | sed -e 's/^Recheck-request: // ' -e 's/,/ /g')
> +
> + for filter in $recheck_filter; do
> + for check in $recheck_list; do
> + if [ "$filter" == "$check" ]; then
> + insert_recheck_request_if_needed "$1" "$3" "$8" "$check" "$2" "$9"
> + fi
> + done
> + done
> +}
> +
> +function check_patch_for_retest_request() {
> + local patch_url="$1"
> +
> + local patch_json=$(curl -s "$userpw" "$patch_url")
> + local patch_comments_url=$(echo "$patch_json" | jq -rc '.comments')
> + local patch_id=$(echo "$patch_json" | jq -rc '.id')
> + if [ "Xnull" != "X$patch_comments_url" ]; then
> + local comments_json=$(curl -s "$userpw" "$patch_comments_url")
> +
> + local seq_end=$(echo "$comments_json" | jq -rc 'length')
> + if [ "$seq_end" -a $seq_end -gt 0 ]; then
> + seq_end=$((seq_end-1))
> + for comment_id in $(seq 0 $seq_end); do
> + local recheck_requested=$(echo "$comments_json" | jq -rc ".[$comment_id].content" | grep "^Recheck-request: ")
This is "user facing", does the expected format of the comment need to
be documented somewhere, e.g., in the README.rst file?
On that note, can we add a reference to some of the documentation in the
email the bot generates as reply to the patch in the series?
Thanks,
Dumitru
> + if [ "X$recheck_requested" != "X" ]; then
> + local msgid=$(echo "$comments_json" | jq -rc ".[$comment_id].msgid")
> + run_recheck "$pw_instance" "$series_id" "$project" "$url" "$repo" "$branchname" "$recheck_requested" "$msgid" "$patch_id"
> + fi
> + done
> + fi
> + fi
> +}
> +
> +function check_series_needs_retest() {
> + local pw_instance="$1"
> +
> + series_get_active_branches "$pw_instance" | while IFS=\| read -r series_id project url repo branchname; do
> + local patch_comments_url=$(curl -s "$userpw" "$url" | jq -rc '.patches[] | .url')
> +
> + for patch in $patch_comments_url; do
> + check_patch_for_retest_request $patch
> + done
> + done
> +}
> +
> check_undownloaded_series "$pw_instance" "$pw_project"
> check_completed_series "$pw_instance" "$pw_project"
> check_new_series "$pw_instance" "$pw_project"
> check_superseded_series "$pw_instance"
> +
> +# check for retest requests after a series is still passing all the
> +# checks above
> +check_series_needs_retest "$pw_instance"
> diff --git a/recheck_tool b/recheck_tool
> new file mode 100755
> index 0000000..98031a9
> --- /dev/null
> +++ b/recheck_tool
> @@ -0,0 +1,100 @@
> +#!/bin/sh
> +# SPDX-Identifier: gpl-2.0-or-later
> +# Copyright (C) 2023 Red Hat, Inc.
> +#
> +# Licensed under the terms of the GNU General Public License as published
> +# by the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version. You may obtain a copy of the
> +# license at
> +#
> +# https://www.gnu.org/licenses/old-licenses/gpl-2.0.html
> +#
> +# Unless required by applicable law or agreed to in writing, software
> +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
> +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
> +# License for the specific language governing permissions and limitations
> +# under the License.
> +
> +mode="select"
> +
> +while [ "$1" != "" ]; do
> + if echo "$1" | grep -q -s -E ^--help ; then
> + echo "recheck / retest state machine script"
> + echo ""
> + echo "$0:"
> + echo " --pw-project=<proj>: Patchwork project."
> + echo " --pw-instance=<inst>: Patchwork instance."
> + echo " --filter=<str>: Job / request for recheck."
> + echo " --state=<0..>: Resync state ID."
> + echo " --msgid=<msgid>: Message ID to select."
> + echo " --update: Set tool in update mode"
> + echo " --new-state=<0..>: New state ID to set"
> + echo " --series-id=<..>: Series ID"
> + echo ""
> + echo "Will spit out a parsable json for each db line when selecting"
> + exit 0
> + elif echo "$1" | grep -q -s -E ^--pw-project= ; then
> + pw_project=$(echo "$1" | sed s/^--pw-project=//)
> + elif echo "$1" | grep -q -s -E ^--pw-instance= ; then
> + pw_instance=$(echo "$1" | sed s/^--pw-instance=//)
> + elif echo "$1" | grep -q -s -E ^--filter= ; then
> + filter=$(echo "$1" | sed s/^--filter=//)
> + elif echo "$1" | grep -q -s -E ^--state= ; then
> + recheck_state=$(echo "$1" | sed s/^--state=//)
> + elif echo "$1" | grep -q -s -E ^--msgid= ; then
> + message_id=$(echo "$1" | sed s/^--msgid=//)
> + elif echo "$1" | grep -q -s -E ^--update ; then
> + mode="update"
> + elif echo "$1" | grep -q -s -E ^--new-state= ; then
> + new_recheck_state=$(echo "$1" | sed s/^--new-state=//)
> + elif echo "$1" | grep -q -s -E ^--series-id= ; then
> + series_id=$(echo "$1" | sed s/^--series-id=//)
> + else
> + echo "Unknown option: '$1'"
> + echo "Rerun with --help for details"
> + exit 1
> + fi
> + shift
> +done
> +
> +source $(dirname $0)/series_db_lib.sh
> +
> +if [ "$mode" == "select" ]; then
> + echo -n "{\"rechecks\":["
> + printed=""
> + for request in $(get_recheck_requests_by_project "$pw_instance" \
> + "$pw_project" \
> + "$recheck_state" \
> + "$filter"); do
> + message_id=$(echo $request | cut -d\| -f1)
> + series_id=$(echo $request | cut -d\| -f2)
> + patch_id=$(echo $request | cut -d\| -f3)
> +
> + sha=$(get_sha_for_series_id_and_patch "$series_id" "$patch_id" \
> + "$pw_instance")
> + echo -n "$printed{\"pw_instance\":\"$pw_instance\",\"series_id\":$series_id,\"patch_id\":$patch_id,\"sha\":\"$sha\",\"msg_id\":\"$message_id\",\"state\":\"$recheck_state\",\"requested\":\"$filter\"}"
> + printed=","
> + done
> + echo "]}"
> +elif [ "$mode" == "update" ]; then
> + if [ "X$new_recheck_state" == "X" -o "X$series_id" == "X" ]; then
> + echo "Need to set a series-id and a new recheck state when updating."
> + exit 1
> + fi
> +
> + request=$(get_recheck_request "$pw_instance" "$pw_project" "$message_id" \
> + "$filter" "$series_id" "$recheck_state")
> + if [ "X$request" == "X" ]; then
> + echo "{\"result\":\"notfound\"}"
> + exit 0
> + fi
> +
> + set_recheck_request_state "$pw_instance" "$pw_project" "$message_id" \
> + "$filter" "$series_id" "$new_recheck_state"
> +
> + echo "{\"result\":\"executed\",\"recheck\":{\"pw_instance\": \"$pw_instance\", \"series_id\":$series_id, \"msg_id\":\"$message_id\", \"state\":\"$new_recheck_state\", \"requested\": \"$filter\"}}"
> +else
> + echo "Unknown state: $mode"
> + exit 1
> +fi
> +
> diff --git a/series_db_lib.sh b/series_db_lib.sh
> index 6c2d98e..ca33c1f 100644
> --- a/series_db_lib.sh
> +++ b/series_db_lib.sh
> @@ -1,6 +1,6 @@
> #!/bin/sh
> # SPDX-Identifier: gpl-2.0-or-later
> -# Copyright (C) 2018,2019 Red Hat, Inc.
> +# Copyright (C) 2018-2023 Red Hat, Inc.
> #
> # Licensed under the terms of the GNU General Public License as published
> # by the Free Software Foundation; either version 2 of the License, or
> @@ -114,6 +114,22 @@ EOF
> run_db_command "INSERT INTO series_schema_version(id) values (7);"
> fi
>
> + run_db_command "select * from series_schema_version;" | egrep '^8$' > /dev/null 2>&1
> + if [ $? -eq 1 ]; then
> + sqlite3 ${HOME}/.series-db <<EOF
> +CREATE TABLE recheck_requests (
> +recheck_id INTEGER,
> +recheck_message_id STRING,
> +recheck_requested_by STRING,
> +recheck_series STRING,
> +recheck_patch INTEGER,
> +patchwork_instance STRING,
> +patchwork_project STRING,
> +recheck_sync INTEGER
> +);
> +EOF
> + run_db_command "INSERT INTO series_schema_version(id) values (8);"
> + fi
> }
>
> function series_db_exists() {
> @@ -390,3 +406,49 @@ function get_patch_id_by_series_id_and_sha() {
>
> echo "select patch_id from git_builds where patchwork_instance=\"$instance\" and series_id=$series_id and sha=\"$sha\";" | series_db_execute
> }
> +
> +function get_recheck_requests_by_project() {
> + local recheck_instance="$1"
> + local recheck_project="$2"
> + local recheck_state="$3"
> + local recheck_requested_by="$4"
> +
> + series_db_exists
> +
> + echo "select recheck_message_id,recheck_series,recheck_patch from recheck_requests where patchwork_instance=\"$recheck_instance\" and patchwork_project=\"$recheck_project\" and recheck_sync=$recheck_state and recheck_requested_by=\"$recheck_requested_by\";" | series_db_execute
> +}
> +
> +function insert_recheck_request_if_needed() {
> + local recheck_instance="$1"
> + local recheck_project="$2"
> + local recheck_msgid="$3"
> + local recheck_requested_by="$4"
> + local recheck_series="$5"
> + local recheck_patch="$6"
> +
> + if ! echo "select * from recheck_requests where recheck_message_id=\"$recheck_msgid\";" | series_db_execute | grep $recheck_msgid >/dev/null 2>&1; then
> + echo "INSERT INTO recheck_requests (recheck_message_id, recheck_requested_by, recheck_series, recheck_patch, patchwork_instance, patchwork_project, recheck_sync) values (\"$recheck_msgid\", \"$recheck_requested_by\", \"$recheck_series\", $recheck_patch, \"$recheck_instance\", \"$recheck_project\", 0);" | series_db_execute
> + fi
> +}
> +
> +function get_recheck_request() {
> + local recheck_instance="$1"
> + local recheck_project="$2"
> + local recheck_msgid="$3"
> + local recheck_requested_by="$4"
> + local recheck_series="$5"
> + local recheck_state="$6"
> +
> + echo "select * from recheck_requests where patchwork_instance=\"$recheck_instance\" and patchwork_project=\"$recheck_project\" and recheck_requested_by=\"$recheck_requested_by\" and recheck_series=\"$recheck_series\" and recheck_message_id=\"$recheck_msgid\" and recheck_sync=$recheck_state;" | series_db_execute
> +}
> +
> +function set_recheck_request_state() {
> + local recheck_instance="$1"
> + local recheck_project="$2"
> + local recheck_msgid="$3"
> + local recheck_requested_by="$4"
> + local recheck_series="$5"
> + local recheck_state="$6"
> +
> + echo "UPDATE recheck_requests set recheck_sync=$recheck_state where patchwork_instance=\"$recheck_instance\" and patchwork_project=\"$recheck_project\" and recheck_requested_by=\"$recheck_requested_by\" and recheck_series=\"$recheck_series\";" | series_db_execute
> +}
next prev parent reply other threads:[~2023-11-09 20:45 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-11-07 20:31 [RFC v2 0/3] Add a recheck framework to pw-ci Aaron Conole
2023-11-07 20:31 ` [RFC v2 1/3] pw_mon: improve command line options Aaron Conole
2023-11-17 16:26 ` Michael Santana
2023-11-17 19:28 ` Aaron Conole
2023-11-07 20:31 ` [RFC v2 2/3] recheck: Add a recheck parser for patchwork comments Aaron Conole
2023-11-09 20:45 ` Dumitru Ceara [this message]
2023-11-10 14:05 ` Aaron Conole
2023-11-17 16:35 ` Michael Santana
2023-11-17 19:28 ` Aaron Conole
2023-11-07 20:31 ` [RFC v2 3/3] github: add a tool for restarting checks Aaron Conole
2023-11-17 16:47 ` Michael Santana
2023-11-17 19:31 ` Aaron Conole
2023-11-17 22:20 ` Michael Santana
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=eb9c7d7c-60f9-4872-b61f-e61ca665a970@redhat.com \
--to=dceara@redhat.com \
--cc=aconole@redhat.com \
--cc=ci@dpdk.org \
--cc=dmarchan@redhat.com \
--cc=msantana@redhat.com \
--cc=probb@iol.unh.edu \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).