From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9B533438FF; Fri, 19 Jan 2024 15:51:38 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 94A4742D45; Fri, 19 Jan 2024 15:51:38 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 3410042D28 for ; Fri, 19 Jan 2024 15:51:37 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1705675896; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TEXH+rUASzYCqgIpwmZBJd2/Nopsdaulnnu21T4rxS4=; b=ZnKjef+XBVO1l/s8mybmcVe/bpOLLHf3mFfMmmWwVmkp8D4HhnOTrl8dFvs/fjq2F8CrxK CW4MkSh8jjd+sLlxDXz5Cmm50BnBYMlh6mZ0TIRxnE9azYNfh4lbM7G1gvoxTc+WcNdAjN z8sIzn+cXZT89aMWZ7dWMf4b6dCkNNY= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-682-KuzKP2t9PjCvzBofozQhnQ-1; Fri, 19 Jan 2024 09:51:34 -0500 X-MC-Unique: KuzKP2t9PjCvzBofozQhnQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5853238425BD; Fri, 19 Jan 2024 14:51:34 +0000 (UTC) Received: from RHTPC1VM0NT (unknown [10.22.33.141]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0CE36111E3F3; Fri, 19 Jan 2024 14:51:34 +0000 (UTC) From: Aaron Conole To: Michael Santana Cc: ci@dpdk.org, Eelco Chaudron , Ilya Maximets , Dumitru Ceara Subject: Re: [PATCH] cirrus: add an initial polling mechanism References: <20231212160813.371433-1-aconole@redhat.com> Date: Fri, 19 Jan 2024 09:51:33 -0500 In-Reply-To: (Michael Santana's message of "Mon, 15 Jan 2024 15:00:48 -0500") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.3 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Michael Santana writes: > On Tue, Dec 12, 2023 at 11:08=E2=80=AFAM Aaron Conole wrote: >> >> Cirrus-CI is used to provide a testing environment for FreeBSD, among >> others. As of now, the ci monitoring framework doesn't support checking >> for cirrus-ci status, but that can be changed. We add an initial pollin= g >> similar to the original GitHub Actions or Travis-CI polling mechanisms >> that merely provides pass/fail/warn with a URL. This can be improved >> later to provide logs on a per-task (similar to GHA Jobs) basis. >> >> Signed-off-by: Aaron Conole > For the most part it looks good to me. See inline comments below. Thanks = Aaron! >> --- >> ci_mon | 6 ++- >> cirrus_mon | 100 +++++++++++++++++++++++++++++++++++++++++++++++ >> series_db_lib.sh | 13 +++++- >> 3 files changed, 116 insertions(+), 3 deletions(-) >> create mode 100755 cirrus_mon >> >> diff --git a/ci_mon b/ci_mon >> index 2e30da4..4b35d38 100755 >> --- a/ci_mon >> +++ b/ci_mon >> @@ -140,13 +140,16 @@ if [ "X" =3D "X$pw_instance" ]; then >> exit 1 >> fi >> >> -for I in travis github obs dummy; do >> +for I in travis github obs cirrus dummy; do > ok >> >> token=3D${I}_token >> disable=3Ddisable_${I} >> >> if [ "X${!disable}" =3D "Xyes" ]; then >> + echo "Skiping ${I}" >> continue >> + else >> + echo "Scanning ${I}" >> fi >> >> ./${I}_mon $pw_instance ${!token} "$pw_project" | grep "^pw|" | whi= le IFS=3D"|" \ >> @@ -163,6 +166,7 @@ for I in travis github obs dummy; do >> >> patch_id=3D"$(get_patch_id_by_series_id_and_sha "$series_id" "$= shasum" "$pw_chk_instance")" >> if [ "X$patch_id" =3D=3D "X" ]; then >> + echo "No patchid..." >> patch_id=3D$(echo $PATCHDATA | jq -rc '.patches[-1].id') >> fi >> PATCHDATA=3D"$(echo "$PATCHDATA" | jq ".patches[] | select(.id= =3D=3D$patch_id)")" >> diff --git a/cirrus_mon b/cirrus_mon >> new file mode 100755 >> index 0000000..f307961 >> --- /dev/null >> +++ b/cirrus_mon > > I agree with your commit message. This is very much like the github > and travis pulling. A lot of stuff that's similar to those scripts. > > So most of my questions are based on the cirrus specific workflow. It > has been a while since I have worked with cirrus Okay >> @@ -0,0 +1,100 @@ >> +#!/bin/bash >> +# SPDX-Identifier: gpl-2.0-or-later >> +# Copyright (C) 2023, Red Hat, Inc. >> +# >> +# Monitors cirrus build history for builds in a series. >> +# Records the builds in the series database (and emits them on the >> +# stdout line for processing) >> +# >> +# Licensed under the terms of the GNU General Public License as publish= ed >> +# by the Free Software Foundation; either version 2 of the License, or >> +# (at your option) any later version. You may obtain a copy of the >> +# license at >> +# >> +# https://www.gnu.org/licenses/old-licenses/gpl-2.0.html >> +# >> +# Unless required by applicable law or agreed to in writing, software >> +# distributed under the License is distributed on an "AS IS" BASIS, WIT= HOUT >> +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See = the >> +# License for the specific language governing permissions and limitatio= ns >> +# under the License. >> + >> +[ -f "$(dirname $0)/series_db_lib.sh" ] && source "$(dirname $0)/series= _db_lib.sh" >> + >> +while [ "$1" !=3D "" ]; do >> + if echo "$1" | grep -q -s -E ^--pw-instance=3D ; then >> + pw_instance=3D$(echo "$1" | sed s/^--pw-instance=3D//) >> + shift >> + elif echo "$1" | grep -q -s -E ^--cirrus-token=3D ; then >> + cirrus_token=3D$(echo "$1" | sed s/^--cirrus-token=3D//) >> + shift >> + elif echo "$1" | grep -q -s -E ^--help ; then >> + echo "Cirrus CI monitor script" >> + echo "$0: args" >> + echo "Required if not set in ~/.pwmon-rc file:" >> + echo " --pw-instance=3D URL for pw" >> + echo "" >> + echo "Options:" >> + echo " --cirrus-token=3D Sets tokenfor web requ= ests" > s/tokenfor/token for/ >> + echo "" >> + exit 0 >> + elif ! echo "$1" | grep -q -s -E ^-- ; then >> + break >> + fi >> +done >> + >> +echo "scanning started " >> /tmp/cirrus.log >> + >> +if [ "X" =3D "X$pw_instance" ]; then >> + pw_instance=3D"$1" >> + shift >> +fi >> + >> +if [ "X" =3D "X$github_token" ]; then > How come cirrus needs a github token? Sorry, it's been a while since I > used cirrus. Are cirrus builds based on github commits? I'll double check - IIRC, cirrus uses the github token as a log in, but it might be a bad variable name. >> + github_token=3D"$1" >> + shift >> +fi >> + >> +if [ "X" =3D "X$pw_project" -a "X$1" !=3D "X" ]; then >> + pw_project=3D"$1" >> + shift >> +fi >> + >> +if [ "X$cirrus_token" !=3D "X" ]; then >> + AUTH=3D"-H \"Authorization: Bearer $cirrus_token\"" >> +fi >> + >> +ci_instance=3D"cirrus_sync" >> + >> +get_unsynced_series "$pw_instance" "$ci_instance" | \ >> + while IFS=3D"|" read -r series_id patch_id patch_url patch_name sha >> patchwork_instance patchwork_project repo_name gap_sync obs_sync >> cirrus_sync; do >> + repo_owner=3D$(echo "$repo_name" | cut -d/ -f1) >> + repo_real=3D$(echo "$repo_name" | cut -d/ -f2) >> + >> + graph_string=3D"{ \"query\": \"query BuildBySHAQuery(\$owner: >> String!, \$name: String!, \$SHA: String){ >> searchBuilds(repositoryOwner: \$owner, repositoryName: \$name, SHA: >> \$SHA) { id, status } }\", \"variables\": { \"owner\": >> \"$repo_owner\", \"name\": \"$repo_real\", \"SHA\": \"$sha\" } }" >> + build_details=3D$(curl -s "$AUTH" -X POST --data "$graph_string" >> https://api.cirrus-ci.com/graphql) > My other question regarding cirrus workflow was this. Normally with > github we would have a direct URL to the project, e.g. > github.com/ovsrobot/ovs, but I dont see anything like > "cirrus.com/ovsrobot/ovs". Maybe that inmbeded in one of the > variables? or is this why we have the query string using the variables > repo_owner and repo_real? If that's the case then it's a shame cirrus > doesnt have a direct API call into the project itself Cirrus doesn't do API calls like that. Instead, we need to use GraphQL to make requests. The whole API is documented, but there are parts of it that surely need more details than others. Actually, the reason that this first pass doesn't break out all the jobs is because the GraphQL queries have to get a bit complex to build that detail up. It makes the API very powerful, but also very difficult to work with for simple stuff. >> + >> + id=3D$(echo "$build_details" | jq -rc '.data.searchBuilds[-1].i= d') >> + status=3D$(echo "$build_details" | jq '.data.searchBuilds[-1].s= tatus') >> + build_url=3D"https://cirrus-ci.com/build/$id" >> + >> + result=3D"in-progress" >> + if [ "$status" =3D=3D "\"COMPLETED\"" ]; then >> + result=3D"passed" >> + elif [ "$status" =3D=3D "\"FAILED\"" ]; then >> + result=3D"failed" >> + elif [ "$status" =3D=3D "\"ABORTED\"" ]; then >> + set_synced_patch "$patch_id" "$patchwork_instance" "$ci_ins= tance" >> + echo "CIRRUS patch_id=3D$patch_id belonging to series=3D$series_id on >> $patchwork_instance was aborted" 1>$2 >> + continue >> + elif [ "$status" =3D=3D "\"ERRORED\"" ]; then >> + result=3D"warn" >> + fi >> + >> + if [ "$result" =3D=3D "in-progress" ]; then >> + echo "CIRRUS patch_id=3D$patch_id belonging to series=3D$series_id is >> not completed[$status]. Skipping" 1>&2 >> + continue >> + fi >> + >> + echo >> "pw|$pw_instance|build|$series_id|SHA|$sha|$result|$build_url|$patch_nam= e|$repo_name|$test_name" > ok >> +done >> diff --git a/series_db_lib.sh b/series_db_lib.sh >> index c5f42e0..e415870 100644 >> --- a/series_db_lib.sh >> +++ b/series_db_lib.sh >> @@ -130,6 +130,15 @@ recheck_sync INTEGER >> EOF >> run_db_command "INSERT INTO series_schema_version(id) values (8= );" >> fi >> + >> + # 0009 - cirrus CI data >> + run_db_command "select * from series_schema_version;" | egrep '^9$'= >/dev/null 2>&1 >> + if [ $? -eq 1 ]; then >> + sqlite3 ${HOME}/.series-db <> +ALTER TABLE git_builds ADD COLUMN cirrus_sync INTEGER; >> +EOF >> + run_db_command "INSERT INTO series_schema_version(id) values (9= );" > ok >> + fi >> } >> >> function series_db_exists() { >> @@ -379,7 +388,7 @@ function set_synced_for_series() { >> >> series_db_exists >> >> - echo "update git_builds set gap_sync=3D1, obs_sync=3D1 where patchw= ork_instance=3D\"$instance\" and series_id=3D$series_id;" | series_db_execu= te >> + echo "update git_builds set gap_sync=3D1, cirrus_sync=3D1, obs_sync= =3D1 where patchwork_instance=3D\"$instance\" and series_id=3D$series_id;" = | series_db_execute > ok >> } >> >> function set_unsynced_for_series() { >> @@ -402,7 +411,7 @@ function insert_commit() { >> >> series_db_exists >> >> - echo "INSERT INTO git_builds (series_id, patch_id, patch_url, patch= _name, sha, patchwork_instance, patchwork_project, repo_name, gap_sync, obs= _sync) VALUES($series_id, $patch_id, \"$patch_url\", \"$patch_name\", \"$sh= a\", \"$instance\", \"$project\", \"$repo_name\", 0, 0);" | series_db_execu= te >> + echo "INSERT INTO git_builds (series_id, patch_id, patch_url, patch= _name, sha, patchwork_instance, patchwork_project, repo_name, gap_sync, obs= _sync, cirrus_sync) VALUES($series_id, $patch_id, \"$patch_url\", \"$patch_= name\", \"$sha\", \"$instance\", \"$project\", \"$repo_name\", 0, 0, 0);" |= series_db_execute > ok >> } >> >> function get_patch_id_by_series_id_and_sha() { > >> -- >> 2.41.0 >>