DPDK CI discussions
 help / color / mirror / Atom feed
From: Aaron Conole <aconole@redhat.com>
To: ci@dpdk.org
Cc: Michael Santana <msantana@redhat.com>,
	Ilya Maximets <i.maximets@ovn.org>,
	Patrick Robb <probb@iol.unh.edu>
Subject: [PATCH v2] cirrus: Add an initial polling mechanism
Date: Tue,  5 Mar 2024 16:56:24 -0500	[thread overview]
Message-ID: <20240305215624.2517037-1-aconole@redhat.com> (raw)

Cirrus-CI is used to provide a testing environment for FreeBSD, among
others.  As of now, the ci monitoring framework doesn't support checking
for cirrus-ci status, but that can be changed.  We add an initial polling
similar to the original GitHub Actions or Travis-CI polling mechanisms
that merely provides pass/fail/warn with a URL.  This can be improved
later to provide logs on a per-task (similar to GHA Jobs) basis.

Signed-off-by: Aaron Conole <aconole@redhat.com>
---
v1->v2:
  Addressed comments from Ilya and Michael
  Added a User Agent to the curl command
  Tested with a few different patches
  Rebased on top of latest tree

 ci_mon           |   5 ++-
 cirrus_mon       | 103 +++++++++++++++++++++++++++++++++++++++++++++++
 series_db_lib.sh |  13 +++++-
 3 files changed, 118 insertions(+), 3 deletions(-)
 create mode 100755 cirrus_mon

diff --git a/ci_mon b/ci_mon
index 94ec1a3..81c80c3 100755
--- a/ci_mon
+++ b/ci_mon
@@ -140,13 +140,16 @@ if [ "X" = "X$pw_instance" ]; then
    exit 1
 fi
 
-for I in travis github obs dummy; do
+for I in travis github obs cirrus dummy; do
 
     token=${I}_token
     disable=disable_${I}
 
     if [ "X${!disable}" = "Xyes" ]; then
+        echo "Skiping ${I}"
         continue
+    else
+        echo "Scanning ${I}"
     fi
 
     ./${I}_mon $pw_instance ${!token} "$pw_project" | grep "^pw|" | while IFS="|" \
diff --git a/cirrus_mon b/cirrus_mon
new file mode 100755
index 0000000..0c60f67
--- /dev/null
+++ b/cirrus_mon
@@ -0,0 +1,103 @@
+#!/bin/bash
+# SPDX-Identifier: gpl-2.0-or-later
+# Copyright (C) 2023, Red Hat, Inc.
+#
+# Monitors cirrus build history for builds in a series.
+# Records the builds in the series database (and emits them on the
+# stdout line for processing)
+#
+# Licensed under the terms of the GNU General Public License as published
+# by the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.  You may obtain a copy of the
+# license at
+#
+#    https://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+[ -f "$(dirname $0)/series_db_lib.sh" ] && source "$(dirname $0)/series_db_lib.sh"
+
+while [ "$1" != "" ]; do
+    if echo "$1" | grep -q -s -E ^--pw-instance= ; then
+        pw_instance=$(echo "$1" | sed s/^--pw-instance=//)
+        shift
+    elif echo "$1" | grep -q -s -E ^--cirrus-token= ; then
+        cirrus_token=$(echo "$1" | sed s/^--cirrus-token=//)
+        shift
+    elif echo "$1" | grep -q -s -E ^--help ; then
+        echo "Cirrus CI monitor script"
+        echo "$0: args"
+        echo "Required if not set in ~/.pwmon-rc file:"
+        echo "  --pw-instance=<inst url>        URL for pw"
+        echo ""
+        echo "Options:"
+        echo "  --cirrus-token=<token>          Sets token for web requests"
+        echo ""
+        echo "Script invocation form (used by ci_mon):"
+        echo "$0 <pw_instance> <cirrus_token> <pw_project>"
+        echo ""
+        echo "  NOTE: For the script invocation from, pw_project is currently"
+        echo "        ignored"
+        exit 0
+    elif ! echo "$1" | grep -q -s -E ^-- ; then
+        break
+    fi
+done
+
+if [ "X" = "X$pw_instance" -a "X$1" != "X" ]; then
+    pw_instance="$1"
+    shift
+fi
+
+if [ "X" = "X$cirrus_token" -a "X$1" != "X" ]; then
+    cirrus_token="$1"
+    shift
+fi
+
+if [ "X" = "X$pw_project" -a "X$1" != "X" ]; then
+    pw_project="$1"
+    shift
+fi
+
+if [ "X$cirrus_token" != "X" ]; then
+    AUTH="-H \"Authorization: Bearer $cirrus_token\""
+fi
+
+ci_instance="cirrus_sync"
+
+get_unsynced_series "$pw_instance" "$ci_instance" | \
+    while IFS="|" read -r series_id patch_id patch_url patch_name sha patchwork_instance patchwork_project repo_name gap_sync obs_sync cirrus_sync; do
+        repo_owner=$(echo "$repo_name" | cut -d/ -f1)
+        repo_real=$(echo "$repo_name" | cut -d/ -f2)
+
+        graph_string="{ \"query\": \"query BuildBySHAQuery(\$owner: String!, \$name: String!, \$SHA: String){ searchBuilds(repositoryOwner: \$owner, repositoryName: \$name, SHA: \$SHA) { id, status } }\", \"variables\": { \"owner\": \"$repo_owner\", \"name\": \"$repo_real\", \"SHA\": \"$sha\" } }"
+        build_details=$(curl -s -A "(pw-ci) cirrus_mon" $AUTH -X POST --data "$graph_string" https://api.cirrus-ci.com/graphql)
+
+        id=$(echo "$build_details" | jq -rc '.data.searchBuilds[-1].id')
+        status=$(echo "$build_details" | jq '.data.searchBuilds[-1].status')
+        build_url="https://cirrus-ci.com/build/$id"
+
+        result="in-progress"
+        if [ "$status" == "\"COMPLETED\"" ]; then
+            result="passed"
+        elif [ "$status" == "\"FAILED\"" ]; then
+            result="failed"
+        elif [ "$status" == "\"ABORTED\"" ]; then
+            set_synced_patch "$patch_id" "$patchwork_instance" "$ci_instance"
+            echo "CIRRUS patch_id=$patch_id belonging to series=$series_id on $patchwork_instance was aborted" 1>&2
+            continue
+        elif [ "$status" == "\"ERRORED\"" ]; then
+            result="warn"
+        fi
+
+        if [ "$result" == "in-progress" ]; then
+            echo "CIRRUS patch_id=$patch_id belonging to series=$series_id is not completed[$status]. Skipping" 1>&2
+            continue
+        fi
+
+        echo "pw|$pw_instance|build|$series_id|SHA|$sha|$result|$build_url|$patch_name|$repo_name|$test_name"
+done
diff --git a/series_db_lib.sh b/series_db_lib.sh
index 0635469..4741d14 100644
--- a/series_db_lib.sh
+++ b/series_db_lib.sh
@@ -141,6 +141,15 @@ check_url STRING
 EOF
         run_db_command "INSERT INTO series_schema_version(id) values (9);"
     fi
+
+    # 0010 - cirrus CI data
+    run_db_command "select * from series_schema_version;" | egrep '^10$' >/dev/null 2>&1
+    if [ $? -eq 1 ]; then
+        sqlite3 ${HOME}/.series-db <<EOF
+ALTER TABLE git_builds ADD COLUMN cirrus_sync INTEGER;
+EOF
+        run_db_command "INSERT INTO series_schema_version(id) values (10);"
+    fi
 }
 
 function series_db_exists() {
@@ -390,7 +399,7 @@ function set_synced_for_series() {
 
     series_db_exists
 
-    echo "update git_builds set gap_sync=1, obs_sync=1 where patchwork_instance=\"$instance\" and series_id=$series_id;" | series_db_execute
+    echo "update git_builds set gap_sync=1, obs_sync=1, cirrus_sync=1 where patchwork_instance=\"$instance\" and series_id=$series_id;" | series_db_execute
 }
 
 function set_unsynced_for_series() {
@@ -413,7 +422,7 @@ function insert_commit() {
 
     series_db_exists
 
-    echo "INSERT INTO git_builds (series_id, patch_id, patch_url, patch_name, sha, patchwork_instance, patchwork_project, repo_name, gap_sync, obs_sync) VALUES($series_id, $patch_id, \"$patch_url\", \"$patch_name\", \"$sha\", \"$instance\", \"$project\", \"$repo_name\", 0, 0);" | series_db_execute
+    echo "INSERT INTO git_builds (series_id, patch_id, patch_url, patch_name, sha, patchwork_instance, patchwork_project, repo_name, gap_sync, obs_sync, cirrus_sync) VALUES($series_id, $patch_id, \"$patch_url\", \"$patch_name\", \"$sha\", \"$instance\", \"$project\", \"$repo_name\", 0, 0, 0);" | series_db_execute
 }
 
 function get_patch_id_by_series_id_and_sha() {
-- 
2.41.0


                 reply	other threads:[~2024-03-05 21:56 UTC|newest]

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20240305215624.2517037-1-aconole@redhat.com \
    --to=aconole@redhat.com \
    --cc=ci@dpdk.org \
    --cc=i.maximets@ovn.org \
    --cc=msantana@redhat.com \
    --cc=probb@iol.unh.edu \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).