From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AB2B543895; Thu, 11 Jan 2024 19:37:59 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 88475402AD; Thu, 11 Jan 2024 19:37:59 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 4284240269 for ; Thu, 11 Jan 2024 19:37:58 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704998277; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=X798kJ3tDwi/ENmJ/3AhwaCtLQGuJ63jy51kPygI6tw=; b=SaAZQc9Qmlijk4BXTynO/B1TLn4g8BdNgvKI6C8wxMUV2TfZ902rf4Bl4YXK8iu6CI2/bp keH5OwBThIImMT6e5G0peM+wuzgynjAUTWNrMC70Dpb1kteOLvQJdLCODNj8KYW4sIZBVy egGMYGY8wxtT3ZZL8wY+mLv0CUeKN3s= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-596-sjRy74LpOHyvoY3iLlC11w-1; Thu, 11 Jan 2024 13:37:54 -0500 X-MC-Unique: sjRy74LpOHyvoY3iLlC11w-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D245E3C1E9C7; Thu, 11 Jan 2024 18:37:53 +0000 (UTC) Received: from RHTPC1VM0NT (unknown [10.22.34.170]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4B7CE2166B31; Thu, 11 Jan 2024 18:37:53 +0000 (UTC) From: Aaron Conole To: ci@dpdk.org Cc: Eelco Chaudron , Ilya Maximets , Dumitru Ceara , Michael Santana Subject: Re: [PATCH] cirrus: add an initial polling mechanism References: <20231212160813.371433-1-aconole@redhat.com> Date: Thu, 11 Jan 2024 13:37:52 -0500 In-Reply-To: <20231212160813.371433-1-aconole@redhat.com> (Aaron Conole's message of "Tue, 12 Dec 2023 11:08:13 -0500") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.3 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.6 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Aaron Conole writes: > Cirrus-CI is used to provide a testing environment for FreeBSD, among > others. As of now, the ci monitoring framework doesn't support checking > for cirrus-ci status, but that can be changed. We add an initial polling > similar to the original GitHub Actions or Travis-CI polling mechanisms > that merely provides pass/fail/warn with a URL. This can be improved > later to provide logs on a per-task (similar to GHA Jobs) basis. > > Signed-off-by: Aaron Conole > --- Any comments? I hope to apply this soon. > ci_mon | 6 ++- > cirrus_mon | 100 +++++++++++++++++++++++++++++++++++++++++++++++ > series_db_lib.sh | 13 +++++- > 3 files changed, 116 insertions(+), 3 deletions(-) > create mode 100755 cirrus_mon > > diff --git a/ci_mon b/ci_mon > index 2e30da4..4b35d38 100755 > --- a/ci_mon > +++ b/ci_mon > @@ -140,13 +140,16 @@ if [ "X" = "X$pw_instance" ]; then > exit 1 > fi > > -for I in travis github obs dummy; do > +for I in travis github obs cirrus dummy; do > > token=${I}_token > disable=disable_${I} > > if [ "X${!disable}" = "Xyes" ]; then > + echo "Skiping ${I}" > continue > + else > + echo "Scanning ${I}" > fi > > ./${I}_mon $pw_instance ${!token} "$pw_project" | grep "^pw|" | while IFS="|" \ > @@ -163,6 +166,7 @@ for I in travis github obs dummy; do > > patch_id="$(get_patch_id_by_series_id_and_sha "$series_id" "$shasum" "$pw_chk_instance")" > if [ "X$patch_id" == "X" ]; then > + echo "No patchid..." > patch_id=$(echo $PATCHDATA | jq -rc '.patches[-1].id') > fi > PATCHDATA="$(echo "$PATCHDATA" | jq ".patches[] | select(.id==$patch_id)")" > diff --git a/cirrus_mon b/cirrus_mon > new file mode 100755 > index 0000000..f307961 > --- /dev/null > +++ b/cirrus_mon > @@ -0,0 +1,100 @@ > +#!/bin/bash > +# SPDX-Identifier: gpl-2.0-or-later > +# Copyright (C) 2023, Red Hat, Inc. > +# > +# Monitors cirrus build history for builds in a series. > +# Records the builds in the series database (and emits them on the > +# stdout line for processing) > +# > +# Licensed under the terms of the GNU General Public License as published > +# by the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. You may obtain a copy of the > +# license at > +# > +# https://www.gnu.org/licenses/old-licenses/gpl-2.0.html > +# > +# Unless required by applicable law or agreed to in writing, software > +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT > +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the > +# License for the specific language governing permissions and limitations > +# under the License. > + > +[ -f "$(dirname $0)/series_db_lib.sh" ] && source "$(dirname $0)/series_db_lib.sh" > + > +while [ "$1" != "" ]; do > + if echo "$1" | grep -q -s -E ^--pw-instance= ; then > + pw_instance=$(echo "$1" | sed s/^--pw-instance=//) > + shift > + elif echo "$1" | grep -q -s -E ^--cirrus-token= ; then > + cirrus_token=$(echo "$1" | sed s/^--cirrus-token=//) > + shift > + elif echo "$1" | grep -q -s -E ^--help ; then > + echo "Cirrus CI monitor script" > + echo "$0: args" > + echo "Required if not set in ~/.pwmon-rc file:" > + echo " --pw-instance= URL for pw" > + echo "" > + echo "Options:" > + echo " --cirrus-token= Sets tokenfor web requests" > + echo "" > + exit 0 > + elif ! echo "$1" | grep -q -s -E ^-- ; then > + break > + fi > +done > + > +echo "scanning started " >> /tmp/cirrus.log > + > +if [ "X" = "X$pw_instance" ]; then > + pw_instance="$1" > + shift > +fi > + > +if [ "X" = "X$github_token" ]; then > + github_token="$1" > + shift > +fi > + > +if [ "X" = "X$pw_project" -a "X$1" != "X" ]; then > + pw_project="$1" > + shift > +fi > + > +if [ "X$cirrus_token" != "X" ]; then > + AUTH="-H \"Authorization: Bearer $cirrus_token\"" > +fi > + > +ci_instance="cirrus_sync" > + > +get_unsynced_series "$pw_instance" "$ci_instance" | \ > + while IFS="|" read -r series_id patch_id patch_url patch_name sha patchwork_instance patchwork_project repo_name gap_sync obs_sync cirrus_sync; do > + repo_owner=$(echo "$repo_name" | cut -d/ -f1) > + repo_real=$(echo "$repo_name" | cut -d/ -f2) > + > + graph_string="{ \"query\": \"query BuildBySHAQuery(\$owner: String!, \$name: String!, \$SHA: String){ searchBuilds(repositoryOwner: \$owner, repositoryName: \$name, SHA: \$SHA) { id, status } }\", \"variables\": { \"owner\": \"$repo_owner\", \"name\": \"$repo_real\", \"SHA\": \"$sha\" } }" > + build_details=$(curl -s "$AUTH" -X POST --data "$graph_string" https://api.cirrus-ci.com/graphql) > + > + id=$(echo "$build_details" | jq -rc '.data.searchBuilds[-1].id') > + status=$(echo "$build_details" | jq '.data.searchBuilds[-1].status') > + build_url="https://cirrus-ci.com/build/$id" > + > + result="in-progress" > + if [ "$status" == "\"COMPLETED\"" ]; then > + result="passed" > + elif [ "$status" == "\"FAILED\"" ]; then > + result="failed" > + elif [ "$status" == "\"ABORTED\"" ]; then > + set_synced_patch "$patch_id" "$patchwork_instance" "$ci_instance" > + echo "CIRRUS patch_id=$patch_id belonging to series=$series_id on $patchwork_instance was aborted" 1>$2 > + continue > + elif [ "$status" == "\"ERRORED\"" ]; then > + result="warn" > + fi > + > + if [ "$result" == "in-progress" ]; then > + echo "CIRRUS patch_id=$patch_id belonging to series=$series_id is not completed[$status]. Skipping" 1>&2 > + continue > + fi > + > + echo "pw|$pw_instance|build|$series_id|SHA|$sha|$result|$build_url|$patch_name|$repo_name|$test_name" > +done > diff --git a/series_db_lib.sh b/series_db_lib.sh > index c5f42e0..e415870 100644 > --- a/series_db_lib.sh > +++ b/series_db_lib.sh > @@ -130,6 +130,15 @@ recheck_sync INTEGER > EOF > run_db_command "INSERT INTO series_schema_version(id) values (8);" > fi > + > + # 0009 - cirrus CI data > + run_db_command "select * from series_schema_version;" | egrep '^9$' >/dev/null 2>&1 > + if [ $? -eq 1 ]; then > + sqlite3 ${HOME}/.series-db < +ALTER TABLE git_builds ADD COLUMN cirrus_sync INTEGER; > +EOF > + run_db_command "INSERT INTO series_schema_version(id) values (9);" > + fi > } > > function series_db_exists() { > @@ -379,7 +388,7 @@ function set_synced_for_series() { > > series_db_exists > > - echo "update git_builds set gap_sync=1, obs_sync=1 where patchwork_instance=\"$instance\" and series_id=$series_id;" | series_db_execute > + echo "update git_builds set gap_sync=1, cirrus_sync=1, obs_sync=1 where patchwork_instance=\"$instance\" and series_id=$series_id;" | series_db_execute > } > > function set_unsynced_for_series() { > @@ -402,7 +411,7 @@ function insert_commit() { > > series_db_exists > > - echo "INSERT INTO git_builds (series_id, patch_id, patch_url, patch_name, sha, patchwork_instance, patchwork_project, repo_name, gap_sync, obs_sync) VALUES($series_id, $patch_id, \"$patch_url\", \"$patch_name\", \"$sha\", \"$instance\", \"$project\", \"$repo_name\", 0, 0);" | series_db_execute > + echo "INSERT INTO git_builds (series_id, patch_id, patch_url, patch_name, sha, patchwork_instance, patchwork_project, repo_name, gap_sync, obs_sync, cirrus_sync) VALUES($series_id, $patch_id, \"$patch_url\", \"$patch_name\", \"$sha\", \"$instance\", \"$project\", \"$repo_name\", 0, 0, 0);" | series_db_execute > } > > function get_patch_id_by_series_id_and_sha() {