From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5E36643896; Thu, 11 Jan 2024 20:36:48 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3AFB540269; Thu, 11 Jan 2024 20:36:48 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 03D1240266 for ; Thu, 11 Jan 2024 20:36:46 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1705001806; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VjB8qmmps27I+vCDYzg28Nln75xG4pJ2NiE2rywxSh4=; b=UDeBKIwZQuYEKVVWyaNOi1+uOjRVXeEWsx+rgiHxKK1ca21T6NUGz64WbZ2W5ELxNoIvUM DgOhSpq9EF67MNPeiWVd9Y3QoA1TSnALUzOgqeteI8BQKiEK7fV+u+s1Vgi2ctDQboUhe3 z7Esx0lENS27ozJtvHye6c5jPfpQ8oY= Received: from mail-lf1-f72.google.com (mail-lf1-f72.google.com [209.85.167.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-576-DaK9tKKAN6-0S9H0PgFOjg-1; Thu, 11 Jan 2024 14:36:43 -0500 X-MC-Unique: DaK9tKKAN6-0S9H0PgFOjg-1 Received: by mail-lf1-f72.google.com with SMTP id 2adb3069b0e04-50e5aa11579so4405343e87.1 for ; Thu, 11 Jan 2024 11:36:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705001801; x=1705606601; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VjB8qmmps27I+vCDYzg28Nln75xG4pJ2NiE2rywxSh4=; b=qLGEFuZa/B0Nw7rO7iobl3asJLLXOq/S3Wqx9BKrhFLqplDG3jke3a4qOq4rRba3HD CtatJS+R9NWDsBzEpoLQR0t/92mUJhFhVouXocSiTbKr3mOgW5Vt94AhiTAuUW9/S65B YN3hpryXagEwdRyAXyKhfVXdj++phKjVS6YxbX15dtqXLKf3zvg7u2ySg1/7rm8TMIH9 KgEA3sGN5Uu5mbyp+wK7ygirkZhq3pWoQcwzxUwFHz/+YFDesO5laq/NK6ykFg8vCwkX 3qzNgF2H2f7JCKAQ7H6Wv9XcN/MMH49lKbP2e18ClrQvIUacR9omFf3gtC5dXsvTgBuW G8jw== X-Gm-Message-State: AOJu0YyhZFXIZvICqZp5aC3z8JxTlXiYr5U/K3EixK6BKYc7yg5O/a/q 0kkAmA4VC8edC+wV+8plKdykb8psyHojNkK5MMpMZHQDv/V0eEEOm2f1gSlU8OmuOvQsFLvM3yP HQpNQwy0JQ8G9V32qonmnxjwLvA== X-Received: by 2002:a05:6512:1284:b0:50e:74f0:c077 with SMTP id u4-20020a056512128400b0050e74f0c077mr125571lfs.8.1705001801679; Thu, 11 Jan 2024 11:36:41 -0800 (PST) X-Google-Smtp-Source: AGHT+IGutwCq+kUObU1BwcHU6e0g0hc96LlB629HE2UjpNWM3W1QG1ElwmQD+ib90Pa5LjqkLkcBDU4yd2MlarsPYlc= X-Received: by 2002:a05:6512:1284:b0:50e:74f0:c077 with SMTP id u4-20020a056512128400b0050e74f0c077mr125561lfs.8.1705001801299; Thu, 11 Jan 2024 11:36:41 -0800 (PST) MIME-Version: 1.0 References: <20231212160813.371433-1-aconole@redhat.com> In-Reply-To: From: Michael Santana Date: Thu, 11 Jan 2024 14:36:29 -0500 Message-ID: Subject: Re: [PATCH] cirrus: add an initial polling mechanism To: Aaron Conole Cc: ci@dpdk.org, Eelco Chaudron , Ilya Maximets , Dumitru Ceara X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sorry got buried in my email queue from break. Let me take a look On Thu, Jan 11, 2024 at 1:37=E2=80=AFPM Aaron Conole w= rote: > > Aaron Conole writes: > > > Cirrus-CI is used to provide a testing environment for FreeBSD, among > > others. As of now, the ci monitoring framework doesn't support checkin= g > > for cirrus-ci status, but that can be changed. We add an initial polli= ng > > similar to the original GitHub Actions or Travis-CI polling mechanisms > > that merely provides pass/fail/warn with a URL. This can be improved > > later to provide logs on a per-task (similar to GHA Jobs) basis. > > > > Signed-off-by: Aaron Conole > > --- > > Any comments? I hope to apply this soon. > > > ci_mon | 6 ++- > > cirrus_mon | 100 +++++++++++++++++++++++++++++++++++++++++++++++ > > series_db_lib.sh | 13 +++++- > > 3 files changed, 116 insertions(+), 3 deletions(-) > > create mode 100755 cirrus_mon > > > > diff --git a/ci_mon b/ci_mon > > index 2e30da4..4b35d38 100755 > > --- a/ci_mon > > +++ b/ci_mon > > @@ -140,13 +140,16 @@ if [ "X" =3D "X$pw_instance" ]; then > > exit 1 > > fi > > > > -for I in travis github obs dummy; do > > +for I in travis github obs cirrus dummy; do > > > > token=3D${I}_token > > disable=3Ddisable_${I} > > > > if [ "X${!disable}" =3D "Xyes" ]; then > > + echo "Skiping ${I}" > > continue > > + else > > + echo "Scanning ${I}" > > fi > > > > ./${I}_mon $pw_instance ${!token} "$pw_project" | grep "^pw|" | wh= ile IFS=3D"|" \ > > @@ -163,6 +166,7 @@ for I in travis github obs dummy; do > > > > patch_id=3D"$(get_patch_id_by_series_id_and_sha "$series_id" "= $shasum" "$pw_chk_instance")" > > if [ "X$patch_id" =3D=3D "X" ]; then > > + echo "No patchid..." > > patch_id=3D$(echo $PATCHDATA | jq -rc '.patches[-1].id') > > fi > > PATCHDATA=3D"$(echo "$PATCHDATA" | jq ".patches[] | select(.id= =3D=3D$patch_id)")" > > diff --git a/cirrus_mon b/cirrus_mon > > new file mode 100755 > > index 0000000..f307961 > > --- /dev/null > > +++ b/cirrus_mon > > @@ -0,0 +1,100 @@ > > +#!/bin/bash > > +# SPDX-Identifier: gpl-2.0-or-later > > +# Copyright (C) 2023, Red Hat, Inc. > > +# > > +# Monitors cirrus build history for builds in a series. > > +# Records the builds in the series database (and emits them on the > > +# stdout line for processing) > > +# > > +# Licensed under the terms of the GNU General Public License as publis= hed > > +# by the Free Software Foundation; either version 2 of the License, or > > +# (at your option) any later version. You may obtain a copy of the > > +# license at > > +# > > +# https://www.gnu.org/licenses/old-licenses/gpl-2.0.html > > +# > > +# Unless required by applicable law or agreed to in writing, software > > +# distributed under the License is distributed on an "AS IS" BASIS, WI= THOUT > > +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See= the > > +# License for the specific language governing permissions and limitati= ons > > +# under the License. > > + > > +[ -f "$(dirname $0)/series_db_lib.sh" ] && source "$(dirname $0)/serie= s_db_lib.sh" > > + > > +while [ "$1" !=3D "" ]; do > > + if echo "$1" | grep -q -s -E ^--pw-instance=3D ; then > > + pw_instance=3D$(echo "$1" | sed s/^--pw-instance=3D//) > > + shift > > + elif echo "$1" | grep -q -s -E ^--cirrus-token=3D ; then > > + cirrus_token=3D$(echo "$1" | sed s/^--cirrus-token=3D//) > > + shift > > + elif echo "$1" | grep -q -s -E ^--help ; then > > + echo "Cirrus CI monitor script" > > + echo "$0: args" > > + echo "Required if not set in ~/.pwmon-rc file:" > > + echo " --pw-instance=3D URL for pw" > > + echo "" > > + echo "Options:" > > + echo " --cirrus-token=3D Sets tokenfor w= eb requests" > > + echo "" > > + exit 0 > > + elif ! echo "$1" | grep -q -s -E ^-- ; then > > + break > > + fi > > +done > > + > > +echo "scanning started " >> /tmp/cirrus.log > > + > > +if [ "X" =3D "X$pw_instance" ]; then > > + pw_instance=3D"$1" > > + shift > > +fi > > + > > +if [ "X" =3D "X$github_token" ]; then > > + github_token=3D"$1" > > + shift > > +fi > > + > > +if [ "X" =3D "X$pw_project" -a "X$1" !=3D "X" ]; then > > + pw_project=3D"$1" > > + shift > > +fi > > + > > +if [ "X$cirrus_token" !=3D "X" ]; then > > + AUTH=3D"-H \"Authorization: Bearer $cirrus_token\"" > > +fi > > + > > +ci_instance=3D"cirrus_sync" > > + > > +get_unsynced_series "$pw_instance" "$ci_instance" | \ > > + while IFS=3D"|" read -r series_id patch_id patch_url patch_name sh= a patchwork_instance patchwork_project repo_name gap_sync obs_sync cirrus_s= ync; do > > + repo_owner=3D$(echo "$repo_name" | cut -d/ -f1) > > + repo_real=3D$(echo "$repo_name" | cut -d/ -f2) > > + > > + graph_string=3D"{ \"query\": \"query BuildBySHAQuery(\$owner: = String!, \$name: String!, \$SHA: String){ searchBuilds(repositoryOwner: \$o= wner, repositoryName: \$name, SHA: \$SHA) { id, status } }\", \"variables\"= : { \"owner\": \"$repo_owner\", \"name\": \"$repo_real\", \"SHA\": \"$sha\"= } }" > > + build_details=3D$(curl -s "$AUTH" -X POST --data "$graph_strin= g" https://api.cirrus-ci.com/graphql) > > + > > + id=3D$(echo "$build_details" | jq -rc '.data.searchBuilds[-1].= id') > > + status=3D$(echo "$build_details" | jq '.data.searchBuilds[-1].= status') > > + build_url=3D"https://cirrus-ci.com/build/$id" > > + > > + result=3D"in-progress" > > + if [ "$status" =3D=3D "\"COMPLETED\"" ]; then > > + result=3D"passed" > > + elif [ "$status" =3D=3D "\"FAILED\"" ]; then > > + result=3D"failed" > > + elif [ "$status" =3D=3D "\"ABORTED\"" ]; then > > + set_synced_patch "$patch_id" "$patchwork_instance" "$ci_in= stance" > > + echo "CIRRUS patch_id=3D$patch_id belonging to series=3D$s= eries_id on $patchwork_instance was aborted" 1>$2 > > + continue > > + elif [ "$status" =3D=3D "\"ERRORED\"" ]; then > > + result=3D"warn" > > + fi > > + > > + if [ "$result" =3D=3D "in-progress" ]; then > > + echo "CIRRUS patch_id=3D$patch_id belonging to series=3D$s= eries_id is not completed[$status]. Skipping" 1>&2 > > + continue > > + fi > > + > > + echo "pw|$pw_instance|build|$series_id|SHA|$sha|$result|$build= _url|$patch_name|$repo_name|$test_name" > > +done > > diff --git a/series_db_lib.sh b/series_db_lib.sh > > index c5f42e0..e415870 100644 > > --- a/series_db_lib.sh > > +++ b/series_db_lib.sh > > @@ -130,6 +130,15 @@ recheck_sync INTEGER > > EOF > > run_db_command "INSERT INTO series_schema_version(id) values (= 8);" > > fi > > + > > + # 0009 - cirrus CI data > > + run_db_command "select * from series_schema_version;" | egrep '^9$= ' >/dev/null 2>&1 > > + if [ $? -eq 1 ]; then > > + sqlite3 ${HOME}/.series-db < > +ALTER TABLE git_builds ADD COLUMN cirrus_sync INTEGER; > > +EOF > > + run_db_command "INSERT INTO series_schema_version(id) values (= 9);" > > + fi > > } > > > > function series_db_exists() { > > @@ -379,7 +388,7 @@ function set_synced_for_series() { > > > > series_db_exists > > > > - echo "update git_builds set gap_sync=3D1, obs_sync=3D1 where patch= work_instance=3D\"$instance\" and series_id=3D$series_id;" | series_db_exec= ute > > + echo "update git_builds set gap_sync=3D1, cirrus_sync=3D1, obs_syn= c=3D1 where patchwork_instance=3D\"$instance\" and series_id=3D$series_id;"= | series_db_execute > > } > > > > function set_unsynced_for_series() { > > @@ -402,7 +411,7 @@ function insert_commit() { > > > > series_db_exists > > > > - echo "INSERT INTO git_builds (series_id, patch_id, patch_url, patc= h_name, sha, patchwork_instance, patchwork_project, repo_name, gap_sync, ob= s_sync) VALUES($series_id, $patch_id, \"$patch_url\", \"$patch_name\", \"$s= ha\", \"$instance\", \"$project\", \"$repo_name\", 0, 0);" | series_db_exec= ute > > + echo "INSERT INTO git_builds (series_id, patch_id, patch_url, patc= h_name, sha, patchwork_instance, patchwork_project, repo_name, gap_sync, ob= s_sync, cirrus_sync) VALUES($series_id, $patch_id, \"$patch_url\", \"$patch= _name\", \"$sha\", \"$instance\", \"$project\", \"$repo_name\", 0, 0, 0);" = | series_db_execute > > } > > > > function get_patch_id_by_series_id_and_sha() { >