From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2BA96432EA; Thu, 9 Nov 2023 21:45:19 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 25287402EA; Thu, 9 Nov 2023 21:45:19 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id A0CA6402E7 for ; Thu, 9 Nov 2023 21:45:17 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1699562717; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JMnIHtp+GKql6pLDws9sTeV188L3zu1gNJnQdNfTS0o=; b=S9kklIH4wG6SnXji6HK3wIAY+UlupgUm+6SNzvuYZ/AG4FmEsN8DNA0p8Sx0QEmM3Y3fGZ hDF6XzpbD2BV2+IZQJs9E/UDure/+J2WrE8w0QIupC7zs/AnOZhV7MvjnvTDORo1ziosgU We91IJdCD2qW045KuuOsM/zul/5mkYI= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-645-PTvGJNNIMUqvOtmlVCXtiA-1; Thu, 09 Nov 2023 15:45:16 -0500 X-MC-Unique: PTvGJNNIMUqvOtmlVCXtiA-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-66d87503d24so15383336d6.3 for ; Thu, 09 Nov 2023 12:45:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699562714; x=1700167514; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JMnIHtp+GKql6pLDws9sTeV188L3zu1gNJnQdNfTS0o=; b=dfqJnyovtEx4KB0o0e2Qm6izmOGV6vHDqsq2aEawgt+PSVRJ1TR+s9XHV4A+9S/KeN nYU7PljfeKqXOESLUjB+eNvPnn2ox99OaYpY1q1zV5JFolqVeXoMtqEKu6S5s6XMJd4y bcYOlFQ/KzbN4RvVPI1hcmOYljVRZ+7nCvZEwJ72K2w2obSS16XsElA0bHs/kuEB4yih Lp6SIacys7pGeOOcFSGwQWydU/Wkm1q8Rd4mjEilmTbJGZSGv9oJAsUXIzkftxg2Sj1/ utRQ7NxTpGGmzqqysmKTLUnlP3OKl8gSYNP949KP2w6JQRMSwhBr8njRqzxKNnEh2gL9 y35Q== X-Gm-Message-State: AOJu0YyaO7q5P2CESkBHE24dtcWepQELpBIq9Dg2BYV71HGEDdy0MbtT Y2fezBf4eQQ04zwR+fb9PL9yVIbNZCmhb+bc269tXMtgFDL14PCzt++p9tnIaBBfQhPxHyKHM/o 6uCJ+KgCHBX5jIGQ= X-Received: by 2002:a05:6214:2266:b0:675:b8ee:e3a1 with SMTP id gs6-20020a056214226600b00675b8eee3a1mr6957152qvb.7.1699562714487; Thu, 09 Nov 2023 12:45:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IEgzkBd5MTBCRjTVPloYDuR8Gip+Uu+2Wty1JOzdgeWnSzMe4rmbObtQe8cIBqusBwam+IJnQ== X-Received: by 2002:a05:6214:2266:b0:675:b8ee:e3a1 with SMTP id gs6-20020a056214226600b00675b8eee3a1mr6957137qvb.7.1699562714185; Thu, 09 Nov 2023 12:45:14 -0800 (PST) Received: from [10.193.20.185] ([66.187.232.65]) by smtp.gmail.com with ESMTPSA id i2-20020a056214020200b0067101efa98asm2355213qvt.69.2023.11.09.12.45.12 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 09 Nov 2023 12:45:13 -0800 (PST) Message-ID: Date: Thu, 9 Nov 2023 15:45:09 -0500 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC v2 2/3] recheck: Add a recheck parser for patchwork comments To: Aaron Conole , ci@dpdk.org Cc: Michael Santana , David Marchand , Thomas Monjalon , Patrick Robb References: <20231107203158.1261199-1-aconole@redhat.com> <20231107203158.1261199-3-aconole@redhat.com> From: Dumitru Ceara In-Reply-To: <20231107203158.1261199-3-aconole@redhat.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org On 11/7/23 21:31, Aaron Conole wrote: > Add a recheck parsing tool that will allow for labs to build a > recheck workflow based on specific recheck labels and projects, > with an associated state machine and querying interface. > > The output of the recheck parsing tool is json and can be fed to > jq or other json parsing utilities for better field support. > > Signed-off-by: Aaron Conole > --- Awesome, OVN really needs this kind of recheck capability in patchwork. One comment inline. > pw_mon | 58 ++++++++++++++++++++++++++- > recheck_tool | 100 +++++++++++++++++++++++++++++++++++++++++++++++ > series_db_lib.sh | 64 +++++++++++++++++++++++++++++- > 3 files changed, 220 insertions(+), 2 deletions(-) > create mode 100755 recheck_tool > > diff --git a/pw_mon b/pw_mon > index da4b9a9..29a9cb4 100755 > --- a/pw_mon > +++ b/pw_mon > @@ -1,6 +1,6 @@ > #!/bin/sh > # SPDX-Identifier: gpl-2.0-or-later > -# Copyright (C) 2018, Red Hat, Inc. > +# Copyright (C) 2018-2023 Red Hat, Inc. > # > # Monitors a project on a patchwork instance for new series submissions > # Records the submissions in the series database (and emits them on the > @@ -44,6 +44,7 @@ if [ "$1" != "" ]; then > fi > fi > > +recheck_filter="" > > while [ "$1" != "" ]; do > if echo "$1" | grep -q -s -E ^--pw-project= ; then > @@ -67,6 +68,10 @@ while [ "$1" != "" ]; do > echo " --add-filter-recheck=filter Adds a filter to flag that a recheck needs to be done" > echo "" > exit 0 > + elif echo "$1" | grep -q -s -E ^--add-filter-recheck=; then > + filter_str=$(echo "$1" | sed s/--add-filter-recheck=//) > + recheck_filter="$filter_str $recheck_filter" > + shift > else > echo "Unknown option: '$1'" > echo "Rerun with --help for details" > @@ -191,7 +196,58 @@ function check_superseded_series() { > done > } > > +function run_recheck() { > + local recheck_list=$(echo "$7" | sed -e 's/^Recheck-request: // ' -e 's/,/ /g') > + > + for filter in $recheck_filter; do > + for check in $recheck_list; do > + if [ "$filter" == "$check" ]; then > + insert_recheck_request_if_needed "$1" "$3" "$8" "$check" "$2" "$9" > + fi > + done > + done > +} > + > +function check_patch_for_retest_request() { > + local patch_url="$1" > + > + local patch_json=$(curl -s "$userpw" "$patch_url") > + local patch_comments_url=$(echo "$patch_json" | jq -rc '.comments') > + local patch_id=$(echo "$patch_json" | jq -rc '.id') > + if [ "Xnull" != "X$patch_comments_url" ]; then > + local comments_json=$(curl -s "$userpw" "$patch_comments_url") > + > + local seq_end=$(echo "$comments_json" | jq -rc 'length') > + if [ "$seq_end" -a $seq_end -gt 0 ]; then > + seq_end=$((seq_end-1)) > + for comment_id in $(seq 0 $seq_end); do > + local recheck_requested=$(echo "$comments_json" | jq -rc ".[$comment_id].content" | grep "^Recheck-request: ") This is "user facing", does the expected format of the comment need to be documented somewhere, e.g., in the README.rst file? On that note, can we add a reference to some of the documentation in the email the bot generates as reply to the patch in the series? Thanks, Dumitru > + if [ "X$recheck_requested" != "X" ]; then > + local msgid=$(echo "$comments_json" | jq -rc ".[$comment_id].msgid") > + run_recheck "$pw_instance" "$series_id" "$project" "$url" "$repo" "$branchname" "$recheck_requested" "$msgid" "$patch_id" > + fi > + done > + fi > + fi > +} > + > +function check_series_needs_retest() { > + local pw_instance="$1" > + > + series_get_active_branches "$pw_instance" | while IFS=\| read -r series_id project url repo branchname; do > + local patch_comments_url=$(curl -s "$userpw" "$url" | jq -rc '.patches[] | .url') > + > + for patch in $patch_comments_url; do > + check_patch_for_retest_request $patch > + done > + done > +} > + > check_undownloaded_series "$pw_instance" "$pw_project" > check_completed_series "$pw_instance" "$pw_project" > check_new_series "$pw_instance" "$pw_project" > check_superseded_series "$pw_instance" > + > +# check for retest requests after a series is still passing all the > +# checks above > +check_series_needs_retest "$pw_instance" > diff --git a/recheck_tool b/recheck_tool > new file mode 100755 > index 0000000..98031a9 > --- /dev/null > +++ b/recheck_tool > @@ -0,0 +1,100 @@ > +#!/bin/sh > +# SPDX-Identifier: gpl-2.0-or-later > +# Copyright (C) 2023 Red Hat, Inc. > +# > +# Licensed under the terms of the GNU General Public License as published > +# by the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. You may obtain a copy of the > +# license at > +# > +# https://www.gnu.org/licenses/old-licenses/gpl-2.0.html > +# > +# Unless required by applicable law or agreed to in writing, software > +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT > +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the > +# License for the specific language governing permissions and limitations > +# under the License. > + > +mode="select" > + > +while [ "$1" != "" ]; do > + if echo "$1" | grep -q -s -E ^--help ; then > + echo "recheck / retest state machine script" > + echo "" > + echo "$0:" > + echo " --pw-project=: Patchwork project." > + echo " --pw-instance=: Patchwork instance." > + echo " --filter=: Job / request for recheck." > + echo " --state=<0..>: Resync state ID." > + echo " --msgid=: Message ID to select." > + echo " --update: Set tool in update mode" > + echo " --new-state=<0..>: New state ID to set" > + echo " --series-id=<..>: Series ID" > + echo "" > + echo "Will spit out a parsable json for each db line when selecting" > + exit 0 > + elif echo "$1" | grep -q -s -E ^--pw-project= ; then > + pw_project=$(echo "$1" | sed s/^--pw-project=//) > + elif echo "$1" | grep -q -s -E ^--pw-instance= ; then > + pw_instance=$(echo "$1" | sed s/^--pw-instance=//) > + elif echo "$1" | grep -q -s -E ^--filter= ; then > + filter=$(echo "$1" | sed s/^--filter=//) > + elif echo "$1" | grep -q -s -E ^--state= ; then > + recheck_state=$(echo "$1" | sed s/^--state=//) > + elif echo "$1" | grep -q -s -E ^--msgid= ; then > + message_id=$(echo "$1" | sed s/^--msgid=//) > + elif echo "$1" | grep -q -s -E ^--update ; then > + mode="update" > + elif echo "$1" | grep -q -s -E ^--new-state= ; then > + new_recheck_state=$(echo "$1" | sed s/^--new-state=//) > + elif echo "$1" | grep -q -s -E ^--series-id= ; then > + series_id=$(echo "$1" | sed s/^--series-id=//) > + else > + echo "Unknown option: '$1'" > + echo "Rerun with --help for details" > + exit 1 > + fi > + shift > +done > + > +source $(dirname $0)/series_db_lib.sh > + > +if [ "$mode" == "select" ]; then > + echo -n "{\"rechecks\":[" > + printed="" > + for request in $(get_recheck_requests_by_project "$pw_instance" \ > + "$pw_project" \ > + "$recheck_state" \ > + "$filter"); do > + message_id=$(echo $request | cut -d\| -f1) > + series_id=$(echo $request | cut -d\| -f2) > + patch_id=$(echo $request | cut -d\| -f3) > + > + sha=$(get_sha_for_series_id_and_patch "$series_id" "$patch_id" \ > + "$pw_instance") > + echo -n "$printed{\"pw_instance\":\"$pw_instance\",\"series_id\":$series_id,\"patch_id\":$patch_id,\"sha\":\"$sha\",\"msg_id\":\"$message_id\",\"state\":\"$recheck_state\",\"requested\":\"$filter\"}" > + printed="," > + done > + echo "]}" > +elif [ "$mode" == "update" ]; then > + if [ "X$new_recheck_state" == "X" -o "X$series_id" == "X" ]; then > + echo "Need to set a series-id and a new recheck state when updating." > + exit 1 > + fi > + > + request=$(get_recheck_request "$pw_instance" "$pw_project" "$message_id" \ > + "$filter" "$series_id" "$recheck_state") > + if [ "X$request" == "X" ]; then > + echo "{\"result\":\"notfound\"}" > + exit 0 > + fi > + > + set_recheck_request_state "$pw_instance" "$pw_project" "$message_id" \ > + "$filter" "$series_id" "$new_recheck_state" > + > + echo "{\"result\":\"executed\",\"recheck\":{\"pw_instance\": \"$pw_instance\", \"series_id\":$series_id, \"msg_id\":\"$message_id\", \"state\":\"$new_recheck_state\", \"requested\": \"$filter\"}}" > +else > + echo "Unknown state: $mode" > + exit 1 > +fi > + > diff --git a/series_db_lib.sh b/series_db_lib.sh > index 6c2d98e..ca33c1f 100644 > --- a/series_db_lib.sh > +++ b/series_db_lib.sh > @@ -1,6 +1,6 @@ > #!/bin/sh > # SPDX-Identifier: gpl-2.0-or-later > -# Copyright (C) 2018,2019 Red Hat, Inc. > +# Copyright (C) 2018-2023 Red Hat, Inc. > # > # Licensed under the terms of the GNU General Public License as published > # by the Free Software Foundation; either version 2 of the License, or > @@ -114,6 +114,22 @@ EOF > run_db_command "INSERT INTO series_schema_version(id) values (7);" > fi > > + run_db_command "select * from series_schema_version;" | egrep '^8$' > /dev/null 2>&1 > + if [ $? -eq 1 ]; then > + sqlite3 ${HOME}/.series-db < +CREATE TABLE recheck_requests ( > +recheck_id INTEGER, > +recheck_message_id STRING, > +recheck_requested_by STRING, > +recheck_series STRING, > +recheck_patch INTEGER, > +patchwork_instance STRING, > +patchwork_project STRING, > +recheck_sync INTEGER > +); > +EOF > + run_db_command "INSERT INTO series_schema_version(id) values (8);" > + fi > } > > function series_db_exists() { > @@ -390,3 +406,49 @@ function get_patch_id_by_series_id_and_sha() { > > echo "select patch_id from git_builds where patchwork_instance=\"$instance\" and series_id=$series_id and sha=\"$sha\";" | series_db_execute > } > + > +function get_recheck_requests_by_project() { > + local recheck_instance="$1" > + local recheck_project="$2" > + local recheck_state="$3" > + local recheck_requested_by="$4" > + > + series_db_exists > + > + echo "select recheck_message_id,recheck_series,recheck_patch from recheck_requests where patchwork_instance=\"$recheck_instance\" and patchwork_project=\"$recheck_project\" and recheck_sync=$recheck_state and recheck_requested_by=\"$recheck_requested_by\";" | series_db_execute > +} > + > +function insert_recheck_request_if_needed() { > + local recheck_instance="$1" > + local recheck_project="$2" > + local recheck_msgid="$3" > + local recheck_requested_by="$4" > + local recheck_series="$5" > + local recheck_patch="$6" > + > + if ! echo "select * from recheck_requests where recheck_message_id=\"$recheck_msgid\";" | series_db_execute | grep $recheck_msgid >/dev/null 2>&1; then > + echo "INSERT INTO recheck_requests (recheck_message_id, recheck_requested_by, recheck_series, recheck_patch, patchwork_instance, patchwork_project, recheck_sync) values (\"$recheck_msgid\", \"$recheck_requested_by\", \"$recheck_series\", $recheck_patch, \"$recheck_instance\", \"$recheck_project\", 0);" | series_db_execute > + fi > +} > + > +function get_recheck_request() { > + local recheck_instance="$1" > + local recheck_project="$2" > + local recheck_msgid="$3" > + local recheck_requested_by="$4" > + local recheck_series="$5" > + local recheck_state="$6" > + > + echo "select * from recheck_requests where patchwork_instance=\"$recheck_instance\" and patchwork_project=\"$recheck_project\" and recheck_requested_by=\"$recheck_requested_by\" and recheck_series=\"$recheck_series\" and recheck_message_id=\"$recheck_msgid\" and recheck_sync=$recheck_state;" | series_db_execute > +} > + > +function set_recheck_request_state() { > + local recheck_instance="$1" > + local recheck_project="$2" > + local recheck_msgid="$3" > + local recheck_requested_by="$4" > + local recheck_series="$5" > + local recheck_state="$6" > + > + echo "UPDATE recheck_requests set recheck_sync=$recheck_state where patchwork_instance=\"$recheck_instance\" and patchwork_project=\"$recheck_project\" and recheck_requested_by=\"$recheck_requested_by\" and recheck_series=\"$recheck_series\";" | series_db_execute > +}