From: Michael Santana <msantana@redhat.com>
To: Aaron Conole <aconole@redhat.com>
Cc: ci@dpdk.org, Eelco Chaudron <echaudro@redhat.com>,
Ilya Maximets <i.maximets@ovn.org>,
Dumitru Ceara <dceara@redhat.com>
Subject: Re: [PATCH] cirrus: add an initial polling mechanism
Date: Thu, 11 Jan 2024 14:36:29 -0500 [thread overview]
Message-ID: <CABVNPRqzE-pj2x=pojt86XVOMT482RCMStEpYTQCthEp6cQ-9w@mail.gmail.com> (raw)
In-Reply-To: <f7t4jfjemwv.fsf@redhat.com>
Sorry got buried in my email queue from break. Let me take a look
On Thu, Jan 11, 2024 at 1:37 PM Aaron Conole <aconole@redhat.com> wrote:
>
> Aaron Conole <aconole@redhat.com> writes:
>
> > Cirrus-CI is used to provide a testing environment for FreeBSD, among
> > others. As of now, the ci monitoring framework doesn't support checking
> > for cirrus-ci status, but that can be changed. We add an initial polling
> > similar to the original GitHub Actions or Travis-CI polling mechanisms
> > that merely provides pass/fail/warn with a URL. This can be improved
> > later to provide logs on a per-task (similar to GHA Jobs) basis.
> >
> > Signed-off-by: Aaron Conole <aconole@redhat.com>
> > ---
>
> Any comments? I hope to apply this soon.
>
> > ci_mon | 6 ++-
> > cirrus_mon | 100 +++++++++++++++++++++++++++++++++++++++++++++++
> > series_db_lib.sh | 13 +++++-
> > 3 files changed, 116 insertions(+), 3 deletions(-)
> > create mode 100755 cirrus_mon
> >
> > diff --git a/ci_mon b/ci_mon
> > index 2e30da4..4b35d38 100755
> > --- a/ci_mon
> > +++ b/ci_mon
> > @@ -140,13 +140,16 @@ if [ "X" = "X$pw_instance" ]; then
> > exit 1
> > fi
> >
> > -for I in travis github obs dummy; do
> > +for I in travis github obs cirrus dummy; do
> >
> > token=${I}_token
> > disable=disable_${I}
> >
> > if [ "X${!disable}" = "Xyes" ]; then
> > + echo "Skiping ${I}"
> > continue
> > + else
> > + echo "Scanning ${I}"
> > fi
> >
> > ./${I}_mon $pw_instance ${!token} "$pw_project" | grep "^pw|" | while IFS="|" \
> > @@ -163,6 +166,7 @@ for I in travis github obs dummy; do
> >
> > patch_id="$(get_patch_id_by_series_id_and_sha "$series_id" "$shasum" "$pw_chk_instance")"
> > if [ "X$patch_id" == "X" ]; then
> > + echo "No patchid..."
> > patch_id=$(echo $PATCHDATA | jq -rc '.patches[-1].id')
> > fi
> > PATCHDATA="$(echo "$PATCHDATA" | jq ".patches[] | select(.id==$patch_id)")"
> > diff --git a/cirrus_mon b/cirrus_mon
> > new file mode 100755
> > index 0000000..f307961
> > --- /dev/null
> > +++ b/cirrus_mon
> > @@ -0,0 +1,100 @@
> > +#!/bin/bash
> > +# SPDX-Identifier: gpl-2.0-or-later
> > +# Copyright (C) 2023, Red Hat, Inc.
> > +#
> > +# Monitors cirrus build history for builds in a series.
> > +# Records the builds in the series database (and emits them on the
> > +# stdout line for processing)
> > +#
> > +# Licensed under the terms of the GNU General Public License as published
> > +# by the Free Software Foundation; either version 2 of the License, or
> > +# (at your option) any later version. You may obtain a copy of the
> > +# license at
> > +#
> > +# https://www.gnu.org/licenses/old-licenses/gpl-2.0.html
> > +#
> > +# Unless required by applicable law or agreed to in writing, software
> > +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
> > +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
> > +# License for the specific language governing permissions and limitations
> > +# under the License.
> > +
> > +[ -f "$(dirname $0)/series_db_lib.sh" ] && source "$(dirname $0)/series_db_lib.sh"
> > +
> > +while [ "$1" != "" ]; do
> > + if echo "$1" | grep -q -s -E ^--pw-instance= ; then
> > + pw_instance=$(echo "$1" | sed s/^--pw-instance=//)
> > + shift
> > + elif echo "$1" | grep -q -s -E ^--cirrus-token= ; then
> > + cirrus_token=$(echo "$1" | sed s/^--cirrus-token=//)
> > + shift
> > + elif echo "$1" | grep -q -s -E ^--help ; then
> > + echo "Cirrus CI monitor script"
> > + echo "$0: args"
> > + echo "Required if not set in ~/.pwmon-rc file:"
> > + echo " --pw-instance=<inst url> URL for pw"
> > + echo ""
> > + echo "Options:"
> > + echo " --cirrus-token=<token> Sets tokenfor web requests"
> > + echo ""
> > + exit 0
> > + elif ! echo "$1" | grep -q -s -E ^-- ; then
> > + break
> > + fi
> > +done
> > +
> > +echo "scanning started " >> /tmp/cirrus.log
> > +
> > +if [ "X" = "X$pw_instance" ]; then
> > + pw_instance="$1"
> > + shift
> > +fi
> > +
> > +if [ "X" = "X$github_token" ]; then
> > + github_token="$1"
> > + shift
> > +fi
> > +
> > +if [ "X" = "X$pw_project" -a "X$1" != "X" ]; then
> > + pw_project="$1"
> > + shift
> > +fi
> > +
> > +if [ "X$cirrus_token" != "X" ]; then
> > + AUTH="-H \"Authorization: Bearer $cirrus_token\""
> > +fi
> > +
> > +ci_instance="cirrus_sync"
> > +
> > +get_unsynced_series "$pw_instance" "$ci_instance" | \
> > + while IFS="|" read -r series_id patch_id patch_url patch_name sha patchwork_instance patchwork_project repo_name gap_sync obs_sync cirrus_sync; do
> > + repo_owner=$(echo "$repo_name" | cut -d/ -f1)
> > + repo_real=$(echo "$repo_name" | cut -d/ -f2)
> > +
> > + graph_string="{ \"query\": \"query BuildBySHAQuery(\$owner: String!, \$name: String!, \$SHA: String){ searchBuilds(repositoryOwner: \$owner, repositoryName: \$name, SHA: \$SHA) { id, status } }\", \"variables\": { \"owner\": \"$repo_owner\", \"name\": \"$repo_real\", \"SHA\": \"$sha\" } }"
> > + build_details=$(curl -s "$AUTH" -X POST --data "$graph_string" https://api.cirrus-ci.com/graphql)
> > +
> > + id=$(echo "$build_details" | jq -rc '.data.searchBuilds[-1].id')
> > + status=$(echo "$build_details" | jq '.data.searchBuilds[-1].status')
> > + build_url="https://cirrus-ci.com/build/$id"
> > +
> > + result="in-progress"
> > + if [ "$status" == "\"COMPLETED\"" ]; then
> > + result="passed"
> > + elif [ "$status" == "\"FAILED\"" ]; then
> > + result="failed"
> > + elif [ "$status" == "\"ABORTED\"" ]; then
> > + set_synced_patch "$patch_id" "$patchwork_instance" "$ci_instance"
> > + echo "CIRRUS patch_id=$patch_id belonging to series=$series_id on $patchwork_instance was aborted" 1>$2
> > + continue
> > + elif [ "$status" == "\"ERRORED\"" ]; then
> > + result="warn"
> > + fi
> > +
> > + if [ "$result" == "in-progress" ]; then
> > + echo "CIRRUS patch_id=$patch_id belonging to series=$series_id is not completed[$status]. Skipping" 1>&2
> > + continue
> > + fi
> > +
> > + echo "pw|$pw_instance|build|$series_id|SHA|$sha|$result|$build_url|$patch_name|$repo_name|$test_name"
> > +done
> > diff --git a/series_db_lib.sh b/series_db_lib.sh
> > index c5f42e0..e415870 100644
> > --- a/series_db_lib.sh
> > +++ b/series_db_lib.sh
> > @@ -130,6 +130,15 @@ recheck_sync INTEGER
> > EOF
> > run_db_command "INSERT INTO series_schema_version(id) values (8);"
> > fi
> > +
> > + # 0009 - cirrus CI data
> > + run_db_command "select * from series_schema_version;" | egrep '^9$' >/dev/null 2>&1
> > + if [ $? -eq 1 ]; then
> > + sqlite3 ${HOME}/.series-db <<EOF
> > +ALTER TABLE git_builds ADD COLUMN cirrus_sync INTEGER;
> > +EOF
> > + run_db_command "INSERT INTO series_schema_version(id) values (9);"
> > + fi
> > }
> >
> > function series_db_exists() {
> > @@ -379,7 +388,7 @@ function set_synced_for_series() {
> >
> > series_db_exists
> >
> > - echo "update git_builds set gap_sync=1, obs_sync=1 where patchwork_instance=\"$instance\" and series_id=$series_id;" | series_db_execute
> > + echo "update git_builds set gap_sync=1, cirrus_sync=1, obs_sync=1 where patchwork_instance=\"$instance\" and series_id=$series_id;" | series_db_execute
> > }
> >
> > function set_unsynced_for_series() {
> > @@ -402,7 +411,7 @@ function insert_commit() {
> >
> > series_db_exists
> >
> > - echo "INSERT INTO git_builds (series_id, patch_id, patch_url, patch_name, sha, patchwork_instance, patchwork_project, repo_name, gap_sync, obs_sync) VALUES($series_id, $patch_id, \"$patch_url\", \"$patch_name\", \"$sha\", \"$instance\", \"$project\", \"$repo_name\", 0, 0);" | series_db_execute
> > + echo "INSERT INTO git_builds (series_id, patch_id, patch_url, patch_name, sha, patchwork_instance, patchwork_project, repo_name, gap_sync, obs_sync, cirrus_sync) VALUES($series_id, $patch_id, \"$patch_url\", \"$patch_name\", \"$sha\", \"$instance\", \"$project\", \"$repo_name\", 0, 0, 0);" | series_db_execute
> > }
> >
> > function get_patch_id_by_series_id_and_sha() {
>
next prev parent reply other threads:[~2024-01-11 19:36 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-12-12 16:08 Aaron Conole
2024-01-11 18:37 ` Aaron Conole
2024-01-11 19:01 ` Ilya Maximets
2024-01-12 17:53 ` Aaron Conole
2024-01-11 19:36 ` Michael Santana [this message]
2024-01-15 20:00 ` Michael Santana
2024-01-19 14:51 ` Aaron Conole
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CABVNPRqzE-pj2x=pojt86XVOMT482RCMStEpYTQCthEp6cQ-9w@mail.gmail.com' \
--to=msantana@redhat.com \
--cc=aconole@redhat.com \
--cc=ci@dpdk.org \
--cc=dceara@redhat.com \
--cc=echaudro@redhat.com \
--cc=i.maximets@ovn.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).