From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B598B43896; Thu, 11 Jan 2024 20:01:29 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9994040269; Thu, 11 Jan 2024 20:01:29 +0100 (CET) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mails.dpdk.org (Postfix) with ESMTP id D311840266 for ; Thu, 11 Jan 2024 20:01:27 +0100 (CET) Received: by mail.gandi.net (Postfix) with ESMTPSA id E321360007; Thu, 11 Jan 2024 19:01:26 +0000 (UTC) Message-ID: <31d55c32-aa6b-4000-bf3a-0b91d67aa673@ovn.org> Date: Thu, 11 Jan 2024 20:01:28 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Cc: i.maximets@ovn.org, Eelco Chaudron , Dumitru Ceara , Michael Santana Subject: Re: [PATCH] cirrus: add an initial polling mechanism Content-Language: en-US To: Aaron Conole , ci@dpdk.org References: <20231212160813.371433-1-aconole@redhat.com> From: Ilya Maximets Autocrypt: addr=i.maximets@ovn.org; keydata= xsFNBF77bOMBEADVZQ4iajIECGfH3hpQMQjhIQlyKX4hIB3OccKl5XvB/JqVPJWuZQRuqNQG /B70MP6km95KnWLZ4H1/5YOJK2l7VN7nO+tyF+I+srcKq8Ai6S3vyiP9zPCrZkYvhqChNOCF pNqdWBEmTvLZeVPmfdrjmzCLXVLi5De9HpIZQFg/Ztgj1AZENNQjYjtDdObMHuJQNJ6ubPIW cvOOn4WBr8NsP4a2OuHSTdVyAJwcDhu+WrS/Bj3KlQXIdPv3Zm5x9u/56NmCn1tSkLrEgi0i /nJNeH5QhPdYGtNzPixKgPmCKz54/LDxU61AmBvyRve+U80ukS+5vWk8zvnCGvL0ms7kx5sA tETpbKEV3d7CB3sQEym8B8gl0Ux9KzGp5lbhxxO995KWzZWWokVUcevGBKsAx4a/C0wTVOpP FbQsq6xEpTKBZwlCpxyJi3/PbZQJ95T8Uw6tlJkPmNx8CasiqNy2872gD1nN/WOP8m+cIQNu o6NOiz6VzNcowhEihE8Nkw9V+zfCxC8SzSBuYCiVX6FpgKzY/Tx+v2uO4f/8FoZj2trzXdLk BaIiyqnE0mtmTQE8jRa29qdh+s5DNArYAchJdeKuLQYnxy+9U1SMMzJoNUX5uRy6/3KrMoC/ 7zhn44x77gSoe7XVM6mr/mK+ViVB7v9JfqlZuiHDkJnS3yxKPwARAQABzSJJbHlhIE1heGlt ZXRzIDxpLm1heGltZXRzQG92bi5vcmc+wsGUBBMBCAA+AhsDBQsJCAcCBhUKCQgLAgQWAgMB Ah4BAheAFiEEh+ma1RKWrHCY821auffsd8gpv5YFAmP+Y/MFCQjFXhAACgkQuffsd8gpv5Yg OA//eEakvE7xTHNIMdLW5r3XnWSEY44dFDEWTLnS7FbZLLHxPNFXN0GSAA8ZsJ3fE26O5Pxe EEFTf7R/W6hHcSXNK4c6S8wR4CkTJC3XOFJchXCdgSc7xS040fLZwGBuO55WT2ZhQvZj1PzT 8Fco8QKvUXr07saHUaYk2Lv2mRhEPP9zsyy7C2T9zUzG04a3SGdP55tB5Adi0r/Ea+6VJoLI ctN8OaF6BwXpag8s76WAyDx8uCCNBF3cnNkQrCsfKrSE2jrvrJBmvlR3/lJ0OYv6bbzfkKvo 0W383EdxevzAO6OBaI2w+wxBK92SMKQB3R0ZI8/gqCokrAFKI7gtnyPGEKz6jtvLgS3PeOtf 5D7PTz+76F/X6rJGTOxR3bup+w1bP/TPHEPa2s7RyJISC07XDe24n9ZUlpG5ijRvfjbCCHb6 pOEijIj2evcIsniTKER2pL+nkYtx0bp7dZEK1trbcfglzte31ZSOsfme74u5HDxq8/rUHT01 51k/vvUAZ1KOdkPrVEl56AYUEsFLlwF1/j9mkd7rUyY3ZV6oyqxV1NKQw4qnO83XiaiVjQus K96X5Ea+XoNEjV4RdxTxOXdDcXqXtDJBC6fmNPzj4QcxxyzxQUVHJv67kJOkF4E+tJza+dNs 8SF0LHnPfHaSPBFrc7yQI9vpk1XBxQWhw6oJgy3OwU0EXvts4wEQANCXyDOic0j2QKeyj/ga OD1oKl44JQfOgcyLVDZGYyEnyl6b/tV1mNb57y/YQYr33fwMS1hMj9eqY6tlMTNz+ciGZZWV YkPNHA+aFuPTzCLrapLiz829M5LctB2448bsgxFq0TPrr5KYx6AkuWzOVq/X5wYEM6djbWLc VWgJ3o0QBOI4/uB89xTf7mgcIcbwEf6yb/86Cs+jaHcUtJcLsVuzW5RVMVf9F+Sf/b98Lzrr 2/mIB7clOXZJSgtV79Alxym4H0cEZabwiXnigjjsLsp4ojhGgakgCwftLkhAnQT3oBLH/6ix 87ahawG3qlyIB8ZZKHsvTxbWte6c6xE5dmmLIDN44SajAdmjt1i7SbAwFIFjuFJGpsnfdQv1 OiIVzJ44kdRJG8kQWPPua/k+AtwJt/gjCxv5p8sKVXTNtIP/sd3EMs2xwbF8McebLE9JCDQ1 RXVHceAmPWVCq3WrFuX9dSlgf3RWTqNiWZC0a8Hn6fNDp26TzLbdo9mnxbU4I/3BbcAJZI9p 9ELaE9rw3LU8esKqRIfaZqPtrdm1C+e5gZa2gkmEzG+WEsS0MKtJyOFnuglGl1ZBxR1uFvbU VXhewCNoviXxkkPk/DanIgYB1nUtkPC+BHkJJYCyf9Kfl33s/bai34aaxkGXqpKv+CInARg3 fCikcHzYYWKaXS6HABEBAAHCwXwEGAEIACYCGwwWIQSH6ZrVEpascJjzbVq59+x3yCm/lgUC Y/5kJAUJCMVeQQAKCRC59+x3yCm/lpF7D/9Lolx00uxqXz2vt/u9flvQvLsOWa+UBmWPGX9u oWhQ26GjtbVvIf6SECcnNWlu/y+MHhmYkz+h2VLhWYVGJ0q03XkktFCNwUvHp3bTXG3IcPIC eDJUVMMIHXFp7TcuRJhrGqnlzqKverlY6+2CqtCpGMEmPVahMDGunwqFfG65QubZySCHVYvX T9SNga0Ay/L71+eVwcuGChGyxEWhVkpMVK5cSWVzZe7C+gb6N1aTNrhu2dhpgcwe1Xsg4dYv dYzTNu19FRpfc+nVRdVnOto8won1SHGgYSVJA+QPv1x8lMYqKESOHAFE/DJJKU8MRkCeSfqs izFVqTxTk3VXOCMUR4t2cbZ9E7Qb/ZZigmmSgilSrOPgDO5TtT811SzheAN0PvgT+L1Gsztc Q3BvfofFv3OLF778JyVfpXRHsn9rFqxG/QYWMqJWi+vdPJ5RhDl1QUEFyH7ok/ZY60/85FW3 o9OQwoMf2+pKNG3J+EMuU4g4ZHGzxI0isyww7PpEHx6sxFEvMhsOp7qnjPsQUcnGIIiqKlTj H7i86580VndsKrRK99zJrm4s9Tg/7OFP1SpVvNvSM4TRXSzVF25WVfLgeloN1yHC5Wsqk33X XNtNovqA0TLFjhfyyetBsIOgpGakgBNieC9GnY7tC3AG+BqG5jnVuGqSTO+iM/d+lsoa+w== In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-GND-Sasl: i.maximets@ovn.org X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org On 1/11/24 19:37, Aaron Conole wrote: > Aaron Conole writes: > >> Cirrus-CI is used to provide a testing environment for FreeBSD, among >> others. As of now, the ci monitoring framework doesn't support checking >> for cirrus-ci status, but that can be changed. We add an initial polling >> similar to the original GitHub Actions or Travis-CI polling mechanisms >> that merely provides pass/fail/warn with a URL. This can be improved >> later to provide logs on a per-task (similar to GHA Jobs) basis. >> >> Signed-off-by: Aaron Conole >> --- > > Any comments? I hope to apply this soon. Hi. I'm not familiar with this project enough, so not a full review, just a few comments. See below. > >> ci_mon | 6 ++- >> cirrus_mon | 100 +++++++++++++++++++++++++++++++++++++++++++++++ >> series_db_lib.sh | 13 +++++- >> 3 files changed, 116 insertions(+), 3 deletions(-) >> create mode 100755 cirrus_mon >> >> diff --git a/ci_mon b/ci_mon >> index 2e30da4..4b35d38 100755 >> --- a/ci_mon >> +++ b/ci_mon >> @@ -140,13 +140,16 @@ if [ "X" = "X$pw_instance" ]; then >> exit 1 >> fi >> >> -for I in travis github obs dummy; do >> +for I in travis github obs cirrus dummy; do >> >> token=${I}_token >> disable=disable_${I} >> >> if [ "X${!disable}" = "Xyes" ]; then >> + echo "Skiping ${I}" >> continue >> + else >> + echo "Scanning ${I}" >> fi >> >> ./${I}_mon $pw_instance ${!token} "$pw_project" | grep "^pw|" | while IFS="|" \ >> @@ -163,6 +166,7 @@ for I in travis github obs dummy; do >> >> patch_id="$(get_patch_id_by_series_id_and_sha "$series_id" "$shasum" "$pw_chk_instance")" >> if [ "X$patch_id" == "X" ]; then >> + echo "No patchid..." patch_id ? >> patch_id=$(echo $PATCHDATA | jq -rc '.patches[-1].id') >> fi >> PATCHDATA="$(echo "$PATCHDATA" | jq ".patches[] | select(.id==$patch_id)")" >> diff --git a/cirrus_mon b/cirrus_mon >> new file mode 100755 >> index 0000000..f307961 >> --- /dev/null >> +++ b/cirrus_mon >> @@ -0,0 +1,100 @@ >> +#!/bin/bash >> +# SPDX-Identifier: gpl-2.0-or-later >> +# Copyright (C) 2023, Red Hat, Inc. >> +# >> +# Monitors cirrus build history for builds in a series. >> +# Records the builds in the series database (and emits them on the >> +# stdout line for processing) >> +# >> +# Licensed under the terms of the GNU General Public License as published >> +# by the Free Software Foundation; either version 2 of the License, or >> +# (at your option) any later version. You may obtain a copy of the >> +# license at >> +# >> +# https://www.gnu.org/licenses/old-licenses/gpl-2.0.html >> +# >> +# Unless required by applicable law or agreed to in writing, software >> +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT >> +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the >> +# License for the specific language governing permissions and limitations >> +# under the License. >> + >> +[ -f "$(dirname $0)/series_db_lib.sh" ] && source "$(dirname $0)/series_db_lib.sh" >> + >> +while [ "$1" != "" ]; do >> + if echo "$1" | grep -q -s -E ^--pw-instance= ; then >> + pw_instance=$(echo "$1" | sed s/^--pw-instance=//) >> + shift >> + elif echo "$1" | grep -q -s -E ^--cirrus-token= ; then >> + cirrus_token=$(echo "$1" | sed s/^--cirrus-token=//) >> + shift >> + elif echo "$1" | grep -q -s -E ^--help ; then >> + echo "Cirrus CI monitor script" >> + echo "$0: args" >> + echo "Required if not set in ~/.pwmon-rc file:" >> + echo " --pw-instance= URL for pw" >> + echo "" >> + echo "Options:" >> + echo " --cirrus-token= Sets tokenfor web requests" Missing space. And maybe misaligned? >> + echo "" >> + exit 0 >> + elif ! echo "$1" | grep -q -s -E ^-- ; then >> + break >> + fi >> +done >> + >> +echo "scanning started " >> /tmp/cirrus.log >> + >> +if [ "X" = "X$pw_instance" ]; then >> + pw_instance="$1" >> + shift >> +fi >> + >> +if [ "X" = "X$github_token" ]; then >> + github_token="$1" >> + shift >> +fi >> + >> +if [ "X" = "X$pw_project" -a "X$1" != "X" ]; then >> + pw_project="$1" >> + shift >> +fi >> + >> +if [ "X$cirrus_token" != "X" ]; then >> + AUTH="-H \"Authorization: Bearer $cirrus_token\"" >> +fi >> + >> +ci_instance="cirrus_sync" >> + >> +get_unsynced_series "$pw_instance" "$ci_instance" | \ Just curious, if the existing system is upgraded to a version with this patch, will it request statuses for every single patch in a patchwork project on a first run or is there something that prevents that? >> + while IFS="|" read -r series_id patch_id patch_url patch_name sha patchwork_instance patchwork_project repo_name gap_sync obs_sync cirrus_sync; do >> + repo_owner=$(echo "$repo_name" | cut -d/ -f1) >> + repo_real=$(echo "$repo_name" | cut -d/ -f2) >> + >> + graph_string="{ \"query\": \"query BuildBySHAQuery(\$owner: String!, \$name: String!, \$SHA: String){ searchBuilds(repositoryOwner: \$owner, repositoryName: \$name, SHA: \$SHA) { id, status } }\", \"variables\": { \"owner\": \"$repo_owner\", \"name\": \"$repo_real\", \"SHA\": \"$sha\" } }" >> + build_details=$(curl -s "$AUTH" -X POST --data "$graph_string" https://api.cirrus-ci.com/graphql) >> + >> + id=$(echo "$build_details" | jq -rc '.data.searchBuilds[-1].id') >> + status=$(echo "$build_details" | jq '.data.searchBuilds[-1].status') >> + build_url="https://cirrus-ci.com/build/$id" >> + >> + result="in-progress" >> + if [ "$status" == "\"COMPLETED\"" ]; then >> + result="passed" >> + elif [ "$status" == "\"FAILED\"" ]; then >> + result="failed" >> + elif [ "$status" == "\"ABORTED\"" ]; then >> + set_synced_patch "$patch_id" "$patchwork_instance" "$ci_instance" >> + echo "CIRRUS patch_id=$patch_id belonging to series=$series_id on $patchwork_instance was aborted" 1>$2 $2 typo? So, aborted jobs will not be reported? But they will be re-queried every time? >> + continue >> + elif [ "$status" == "\"ERRORED\"" ]; then >> + result="warn" >> + fi >> + >> + if [ "$result" == "in-progress" ]; then >> + echo "CIRRUS patch_id=$patch_id belonging to series=$series_id is not completed[$status]. Skipping" 1>&2 >> + continue >> + fi >> + >> + echo "pw|$pw_instance|build|$series_id|SHA|$sha|$result|$build_url|$patch_name|$repo_name|$test_name" >> +done >> diff --git a/series_db_lib.sh b/series_db_lib.sh >> index c5f42e0..e415870 100644 >> --- a/series_db_lib.sh >> +++ b/series_db_lib.sh >> @@ -130,6 +130,15 @@ recheck_sync INTEGER >> EOF >> run_db_command "INSERT INTO series_schema_version(id) values (8);" >> fi >> + >> + # 0009 - cirrus CI data >> + run_db_command "select * from series_schema_version;" | egrep '^9$' >/dev/null 2>&1 >> + if [ $? -eq 1 ]; then >> + sqlite3 ${HOME}/.series-db <> +ALTER TABLE git_builds ADD COLUMN cirrus_sync INTEGER; >> +EOF >> + run_db_command "INSERT INTO series_schema_version(id) values (9);" >> + fi >> } >> >> function series_db_exists() { >> @@ -379,7 +388,7 @@ function set_synced_for_series() { >> >> series_db_exists >> >> - echo "update git_builds set gap_sync=1, obs_sync=1 where patchwork_instance=\"$instance\" and series_id=$series_id;" | series_db_execute >> + echo "update git_builds set gap_sync=1, cirrus_sync=1, obs_sync=1 where patchwork_instance=\"$instance\" and series_id=$series_id;" | series_db_execute >> } >> >> function set_unsynced_for_series() { >> @@ -402,7 +411,7 @@ function insert_commit() { >> >> series_db_exists >> >> - echo "INSERT INTO git_builds (series_id, patch_id, patch_url, patch_name, sha, patchwork_instance, patchwork_project, repo_name, gap_sync, obs_sync) VALUES($series_id, $patch_id, \"$patch_url\", \"$patch_name\", \"$sha\", \"$instance\", \"$project\", \"$repo_name\", 0, 0);" | series_db_execute >> + echo "INSERT INTO git_builds (series_id, patch_id, patch_url, patch_name, sha, patchwork_instance, patchwork_project, repo_name, gap_sync, obs_sync, cirrus_sync) VALUES($series_id, $patch_id, \"$patch_url\", \"$patch_name\", \"$sha\", \"$instance\", \"$project\", \"$repo_name\", 0, 0, 0);" | series_db_execute >> } >> >> function get_patch_id_by_series_id_and_sha() { >