Re: [PATCH] cirrus: add an initial polling mechanism

[Michael Santana <msantana@redhat.com>]

On Tue, Dec 12, 2023 at 11:08 AM Aaron Conole <aconole@redhat.com> wrote:
>
> Cirrus-CI is used to provide a testing environment for FreeBSD, among
> others.  As of now, the ci monitoring framework doesn't support checking
> for cirrus-ci status, but that can be changed.  We add an initial polling
> similar to the original GitHub Actions or Travis-CI polling mechanisms
> that merely provides pass/fail/warn with a URL.  This can be improved
> later to provide logs on a per-task (similar to GHA Jobs) basis.
>
> Signed-off-by: Aaron Conole <aconole@redhat.com>
For the most part it looks good to me. See inline comments below. Thanks Aaron!
> ---
>  ci_mon           |   6 ++-
>  cirrus_mon       | 100 +++++++++++++++++++++++++++++++++++++++++++++++
>  series_db_lib.sh |  13 +++++-
>  3 files changed, 116 insertions(+), 3 deletions(-)
>  create mode 100755 cirrus_mon
>
> diff --git a/ci_mon b/ci_mon
> index 2e30da4..4b35d38 100755
> --- a/ci_mon
> +++ b/ci_mon
> @@ -140,13 +140,16 @@ if [ "X" = "X$pw_instance" ]; then
>     exit 1
>  fi
>
> -for I in travis github obs dummy; do
> +for I in travis github obs cirrus dummy; do
ok
>
>      token=${I}_token
>      disable=disable_${I}
>
>      if [ "X${!disable}" = "Xyes" ]; then
> +        echo "Skiping ${I}"
>          continue
> +    else
> +        echo "Scanning ${I}"
>      fi
>
>      ./${I}_mon $pw_instance ${!token} "$pw_project" | grep "^pw|" | while IFS="|" \
> @@ -163,6 +166,7 @@ for I in travis github obs dummy; do
>
>          patch_id="$(get_patch_id_by_series_id_and_sha "$series_id" "$shasum" "$pw_chk_instance")"
>          if [ "X$patch_id" == "X" ]; then
> +            echo "No patchid..."
>              patch_id=$(echo $PATCHDATA | jq -rc '.patches[-1].id')
>          fi
>          PATCHDATA="$(echo "$PATCHDATA" | jq ".patches[] | select(.id==$patch_id)")"
> diff --git a/cirrus_mon b/cirrus_mon
> new file mode 100755
> index 0000000..f307961
> --- /dev/null
> +++ b/cirrus_mon

I agree with your commit message. This is very much like the github
and travis pulling. A lot of stuff that's similar to those scripts.

So most of my questions are based on the cirrus specific workflow. It
has been a while since I have worked with cirrus

> @@ -0,0 +1,100 @@
> +#!/bin/bash
> +# SPDX-Identifier: gpl-2.0-or-later
> +# Copyright (C) 2023, Red Hat, Inc.
> +#
> +# Monitors cirrus build history for builds in a series.
> +# Records the builds in the series database (and emits them on the
> +# stdout line for processing)
> +#
> +# Licensed under the terms of the GNU General Public License as published
> +# by the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.  You may obtain a copy of the
> +# license at
> +#
> +#    https://www.gnu.org/licenses/old-licenses/gpl-2.0.html
> +#
> +# Unless required by applicable law or agreed to in writing, software
> +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
> +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
> +# License for the specific language governing permissions and limitations
> +# under the License.
> +
> +[ -f "$(dirname $0)/series_db_lib.sh" ] && source "$(dirname $0)/series_db_lib.sh"
> +
> +while [ "$1" != "" ]; do
> +    if echo "$1" | grep -q -s -E ^--pw-instance= ; then
> +        pw_instance=$(echo "$1" | sed s/^--pw-instance=//)
> +        shift
> +    elif echo "$1" | grep -q -s -E ^--cirrus-token= ; then
> +        cirrus_token=$(echo "$1" | sed s/^--cirrus-token=//)
> +        shift
> +    elif echo "$1" | grep -q -s -E ^--help ; then
> +        echo "Cirrus CI monitor script"
> +        echo "$0: args"
> +        echo "Required if not set in ~/.pwmon-rc file:"
> +        echo "  --pw-instance=<inst url>       URL for pw"
> +        echo ""
> +        echo "Options:"
> +        echo "  --cirrus-token=<token>         Sets tokenfor web requests"
s/tokenfor/token for/
> +        echo ""
> +        exit 0
> +    elif ! echo "$1" | grep -q -s -E ^-- ; then
> +        break
> +    fi
> +done
> +
> +echo "scanning started " >> /tmp/cirrus.log
> +
> +if [ "X" = "X$pw_instance" ]; then
> +    pw_instance="$1"
> +    shift
> +fi
> +
> +if [ "X" = "X$github_token" ]; then
How come cirrus needs a github token? Sorry, it's been a while since I
used cirrus. Are cirrus builds based on github commits?
> +    github_token="$1"
> +    shift
> +fi
> +
> +if [ "X" = "X$pw_project" -a "X$1" != "X" ]; then
> +    pw_project="$1"
> +    shift
> +fi
> +
> +if [ "X$cirrus_token" != "X" ]; then
> +    AUTH="-H \"Authorization: Bearer $cirrus_token\""
> +fi
> +
> +ci_instance="cirrus_sync"
> +
> +get_unsynced_series "$pw_instance" "$ci_instance" | \
> +    while IFS="|" read -r series_id patch_id patch_url patch_name sha patchwork_instance patchwork_project repo_name gap_sync obs_sync cirrus_sync; do
> +        repo_owner=$(echo "$repo_name" | cut -d/ -f1)
> +        repo_real=$(echo "$repo_name" | cut -d/ -f2)
> +
> +        graph_string="{ \"query\": \"query BuildBySHAQuery(\$owner: String!, \$name: String!, \$SHA: String){ searchBuilds(repositoryOwner: \$owner, repositoryName: \$name, SHA: \$SHA) { id, status } }\", \"variables\": { \"owner\": \"$repo_owner\", \"name\": \"$repo_real\", \"SHA\": \"$sha\" } }"
> +        build_details=$(curl -s "$AUTH" -X POST --data "$graph_string" https://api.cirrus-ci.com/graphql)
My other question regarding cirrus workflow was this. Normally with
github we would have a direct URL to the project, e.g.
github.com/ovsrobot/ovs, but I dont see anything like
"cirrus.com/ovsrobot/ovs". Maybe that inmbeded in one of the
variables? or is this why we have the query string using the variables
repo_owner and repo_real? If that's the case then it's a shame cirrus
doesnt have a direct API call into the project itself

> +
> +        id=$(echo "$build_details" | jq -rc '.data.searchBuilds[-1].id')
> +        status=$(echo "$build_details" | jq '.data.searchBuilds[-1].status')
> +        build_url="https://cirrus-ci.com/build/$id"
> +
> +        result="in-progress"
> +        if [ "$status" == "\"COMPLETED\"" ]; then
> +            result="passed"
> +        elif [ "$status" == "\"FAILED\"" ]; then
> +            result="failed"
> +        elif [ "$status" == "\"ABORTED\"" ]; then
> +            set_synced_patch "$patch_id" "$patchwork_instance" "$ci_instance"
> +            echo "CIRRUS patch_id=$patch_id belonging to series=$series_id on $patchwork_instance was aborted" 1>$2
> +            continue
> +        elif [ "$status" == "\"ERRORED\"" ]; then
> +            result="warn"
> +        fi
> +
> +        if [ "$result" == "in-progress" ]; then
> +            echo "CIRRUS patch_id=$patch_id belonging to series=$series_id is not completed[$status]. Skipping" 1>&2
> +            continue
> +        fi
> +
> +        echo "pw|$pw_instance|build|$series_id|SHA|$sha|$result|$build_url|$patch_name|$repo_name|$test_name"
ok
> +done
> diff --git a/series_db_lib.sh b/series_db_lib.sh
> index c5f42e0..e415870 100644
> --- a/series_db_lib.sh
> +++ b/series_db_lib.sh
> @@ -130,6 +130,15 @@ recheck_sync INTEGER
>  EOF
>          run_db_command "INSERT INTO series_schema_version(id) values (8);"
>      fi
> +
> +    # 0009 - cirrus CI data
> +    run_db_command "select * from series_schema_version;" | egrep '^9$' >/dev/null 2>&1
> +    if [ $? -eq 1 ]; then
> +        sqlite3 ${HOME}/.series-db <<EOF
> +ALTER TABLE git_builds ADD COLUMN cirrus_sync INTEGER;
> +EOF
> +        run_db_command "INSERT INTO series_schema_version(id) values (9);"
ok
> +    fi
>  }
>
>  function series_db_exists() {
> @@ -379,7 +388,7 @@ function set_synced_for_series() {
>
>      series_db_exists
>
> -    echo "update git_builds set gap_sync=1, obs_sync=1 where patchwork_instance=\"$instance\" and series_id=$series_id;" | series_db_execute
> +    echo "update git_builds set gap_sync=1, cirrus_sync=1, obs_sync=1 where patchwork_instance=\"$instance\" and series_id=$series_id;" | series_db_execute
ok
>  }
>
>  function set_unsynced_for_series() {
> @@ -402,7 +411,7 @@ function insert_commit() {
>
>      series_db_exists
>
> -    echo "INSERT INTO git_builds (series_id, patch_id, patch_url, patch_name, sha, patchwork_instance, patchwork_project, repo_name, gap_sync, obs_sync) VALUES($series_id, $patch_id, \"$patch_url\", \"$patch_name\", \"$sha\", \"$instance\", \"$project\", \"$repo_name\", 0, 0);" | series_db_execute
> +    echo "INSERT INTO git_builds (series_id, patch_id, patch_url, patch_name, sha, patchwork_instance, patchwork_project, repo_name, gap_sync, obs_sync, cirrus_sync) VALUES($series_id, $patch_id, \"$patch_url\", \"$patch_name\", \"$sha\", \"$instance\", \"$project\", \"$repo_name\", 0, 0, 0);" | series_db_execute
ok
>  }
>
>  function get_patch_id_by_series_id_and_sha() {

> --
> 2.41.0
>