From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 099504399C; Mon, 22 Jan 2024 19:17:23 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0467D402C3; Mon, 22 Jan 2024 19:17:23 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 3BA5C40298 for ; Mon, 22 Jan 2024 19:17:21 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1705947440; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FH6quqEqeM9RlqrVzSXv4+1dt9G/44re8SedbrGU+lA=; b=T4csau23aqFt9Mau8YqhkaGEAqPUedtAdNH8z1NvIHcsCGZ7NfXJezEvYLL48VC2IFuSKX BsjFVN/Kfg8NyqcpdZW8rNPUdxJmYUp9IApHuQ66a2TZAWMhij6Pq2fHD5dTcE3ukDLt2d 0RyJbSb/F3af26wCZFb+DwegLvamD8o= Received: from mail-lf1-f70.google.com (mail-lf1-f70.google.com [209.85.167.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-382-RphjMxeCMV69NfjNAzQBfA-1; Mon, 22 Jan 2024 13:17:16 -0500 X-MC-Unique: RphjMxeCMV69NfjNAzQBfA-1 Received: by mail-lf1-f70.google.com with SMTP id 2adb3069b0e04-50ed86962f6so2401612e87.1 for ; Mon, 22 Jan 2024 10:17:15 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705947435; x=1706552235; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FH6quqEqeM9RlqrVzSXv4+1dt9G/44re8SedbrGU+lA=; b=dsUfK+W0kP3EVZljodEUgv+kRyTvHcSzRcPh241PmYHgYiqTQp0j9ziKl4v3tj7YxZ XYNsFvm90TZwsKDEUzSQTSs/fFFE0LhL0PiHRhWQaKibCxDC1pgZRPAbYWWXQJ3Vh0yK 9bYWZ/cAkMwrvIyMn7nr63d5npgJzcN13hmuy6+IpTXxxgZ2guhx4AQivO/74Fx0Znra nQC960Iiz1bg2fQ8+Fxp4oerS7OThAzuFKU+I3moPNBTMzTCv7I9obf47zFj9TjubdpJ Q4D/UtMWlEimibNi/b6eksJvsRRKM29PwizPKT0IRNfFkndLP5pfmr2QkY+YQawhzpQg TLZw== X-Gm-Message-State: AOJu0YxWiCXC1fLnACxq2zWfMwQQA4uw88UIQZViDaXs5Cx+DUZNDstH 3bTvZwZ/i0F5qepQP7jF2LkTxV33/EAQrtbjISr898U9GK9k6oSzg2Oajeb/zYS4sYi0Qr8EWty Faev8eB5mba5DWwIyb5V9rRvli6L0lCO+sslVxLJSaCRi9csotRe4EwhgqLJ6tBT/wmOznquz5f YvzkBomK2bJSeRwA== X-Received: by 2002:ac2:442d:0:b0:50f:13bc:cb1 with SMTP id w13-20020ac2442d000000b0050f13bc0cb1mr770454lfl.206.1705947434649; Mon, 22 Jan 2024 10:17:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IHoQMgg+yRX+un9PshNOJVH7BGm1MwzIr/x2yEjM87rG+jNi8lKXl4Pxe21J/hTaSv//ai/3mhxh43J3EBlb6A= X-Received: by 2002:ac2:442d:0:b0:50f:13bc:cb1 with SMTP id w13-20020ac2442d000000b0050f13bc0cb1mr770450lfl.206.1705947434202; Mon, 22 Jan 2024 10:17:14 -0800 (PST) MIME-Version: 1.0 References: <20240122172635.3641078-1-aconole@redhat.com> <20240122172635.3641078-2-aconole@redhat.com> In-Reply-To: <20240122172635.3641078-2-aconole@redhat.com> From: Michael Santana Date: Mon, 22 Jan 2024 13:17:02 -0500 Message-ID: Subject: Re: [PATCH 1/2] treewide: Add a User Agent for CURL requests To: Aaron Conole Cc: ci@dpdk.org, Ilya Maximets , Jeremy Kerr X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org On Mon, Jan 22, 2024 at 12:26=E2=80=AFPM Aaron Conole = wrote: > > This will help identifying robot usage across PW instances. Thanks Aaron for the quick patch! My only suggestion is to prepend '0daybot' to the user agent (or anything close that you like). That way we know the requests are coming from us. It might be easier for Jeremy to identify than just "pw-post" alone for example Again thanks for the quick turn around! Acked-by: Michael Santana > > Signed-off-by: Aaron Conole > --- > ci_mon | 2 +- > github_get_logs.sh | 8 ++++---- > github_mon | 2 +- > github_restart | 6 ++++-- > jenkins_lib.sh | 5 +++-- > post_pw.sh | 8 ++++---- > pw_mon | 14 +++++++------- > 7 files changed, 24 insertions(+), 21 deletions(-) > > diff --git a/ci_mon b/ci_mon > index 2e30da4..31de0ab 100755 > --- a/ci_mon > +++ b/ci_mon > @@ -159,7 +159,7 @@ for I in travis github obs dummy; do > author=3D$(echo $SERIES_LINE | cut -d\| -f4) > email=3D$(echo $SERIES_LINE | cut -d\| -f5) > > - PATCHDATA=3D$(curl -s $url) > + PATCHDATA=3D$(curl -A "ci-mon-${PROJECT}" -s $url) > > patch_id=3D"$(get_patch_id_by_series_id_and_sha "$series_id" "$s= hasum" "$pw_chk_instance")" > if [ "X$patch_id" =3D=3D "X" ]; then > diff --git a/github_get_logs.sh b/github_get_logs.sh > index 9f44840..19c2278 100755 > --- a/github_get_logs.sh > +++ b/github_get_logs.sh > @@ -74,14 +74,14 @@ print_errored_logs_for_commit () { > run_id=3D"$(echo "$run_meta" | jq -r ".id")" > > # Get the real logs url, download logs, and unzip logs > - logs_url=3D"$(curl -s -S -H "${AUTH}" -H "${APP}" "${redirect_url}" = -I | \ > + logs_url=3D"$(curl -A "github-logs-crawler" -s -S -H "${AUTH}" -H "$= {APP}" "${redirect_url}" -I | \ > grep -i 'Location: '| sed 's/Location: //i' | tr -d '\r')" > - curl -s -S "$logs_url" -o "build_logs_series_${series_id}.zip" > + curl -A "github-logs-crawler" -s -S "$logs_url" -o "build_logs_serie= s_${series_id}.zip" > unzip -o -q "build_logs_series_${series_id}.zip" -d "build_logs_seri= es_${series_id}" > > # Get the names of the failed jobs and the steps that failed > tmp_url=3D"$GITHUB_API/repos/$repo_name/actions/runs/${run_id}/jobs" > - jobs_results=3D"$(curl -s -S -H "${AUTH}" -H "${APP}" "${tmp_url}")" > + jobs_results=3D"$(curl -A "github-logs-crawler" -s -S -H "${AUTH}" -= H "${APP}" "${tmp_url}")" > jobs_results=3D"$(echo "$jobs_results" | jq "[.jobs[] | \ > select(.conclusion=3D=3D\"failure\") | {name, failed_step: .step= s[] | \ > select(.conclusion=3D=3D\"failure\") | {name, conclusion, number= }}]")" > @@ -135,7 +135,7 @@ repo_name=3D$(echo "$repo_name" | sed -e 's@%2F@/@g' = -e 's,%40,@,g') > > # Get GHA runs > tmp_url=3D"$GITHUB_API/repos/$repo_name/actions/runs?per_page=3D9000" > -all_runs=3D"$(curl -s -S -H "${AUTH}" -H "${APP}" "${tmp_url}")" > +all_runs=3D"$(curl -A "github-logs-crawler" -s -S -H "${AUTH}" -H "${APP= }" "${tmp_url}")" > runs=3D$(echo $all_runs | jq -rc ".workflow_runs[] | select(.head_branch= =3D=3D \"series_$series_id\")") > not_found=3D"$(echo "$runs" | jq -rc ".message")" > if [ "$not_found" =3D=3D "Not Found" ]; then > diff --git a/github_mon b/github_mon > index 8670ce8..311c7bd 100755 > --- a/github_mon > +++ b/github_mon > @@ -117,7 +117,7 @@ while IFS=3D"|" read -r series_id patch_id patch_url = patch_name sha patchwork_inst > tmp_url=3D"$GITHUB_API/repos/$repo_name/actions/runs?per_page=3D= 9000" > if [ "$tmp_url" !=3D "$prev_url" ]; then > prev_url=3D"$tmp_url" > - all_runs=3D"$(curl -s -S -H "${AUTH}" -H "${APP}" "${tmp_url= }")" > + all_runs=3D"$(curl -A "github-mon-${pw_project}" -s -S -H "$= {AUTH}" -H "${APP}" "${tmp_url}")" > fi > > runs=3D$(echo "$all_runs" | jq -rc ".workflow_runs[] | select(.h= ead_branch =3D=3D \"series_$series_id\")") > diff --git a/github_restart b/github_restart > index d5d63b9..18af978 100755 > --- a/github_restart > +++ b/github_restart > @@ -114,7 +114,8 @@ if [ "X$runid" =3D=3D "X" ]; then > fi > > comma=3D"" > - for job in $(curl -s -S -H "${AUTH}" -H "${APP}" \ > + for job in $(curl -A "github-restart-${pw_project}" -s -S -H "${AUTH= }" \ > + -H "${APP}" \ > "https://api.github.com/repos/${reponame}/actions/= runs?head_sha=3D${sha}" | \ > jq -rc ".workflow_runs[] ${workflow_select} .id") > do > @@ -126,7 +127,8 @@ fi > echo -n "{\"results\":[" > comma=3D"" > for job in $(echo "$runid" | sed 's/,/ /'); do > - result=3D$(curl -s -X POST -L -S -H "${AUTH}" -H "${APP}" \ > + result=3D$(curl -A "github-restart-${pw_project}" -s -X POST -L -S \ > + -H "${AUTH}" -H "${APP}" \ > "https://api.github.com/repos/${reponame}/actions/runs= /$job/rerun") > msg=3D$(echo "$result" | jq -rc '.message') > > diff --git a/jenkins_lib.sh b/jenkins_lib.sh > index fc7cb1d..09c176c 100644 > --- a/jenkins_lib.sh > +++ b/jenkins_lib.sh > @@ -41,12 +41,12 @@ if [ "X$jenkins_token" !=3D "X" ]; then > jenkins_credentials=3D"$jenkins_user:$jenkins_token" > fi > > -jenkins_crumb_value=3D$(curl -s "http://${jenkins_credentials}@${jenkins= _url}/crumbIssuer/api/xml?xpath=3Dconcat(//crumbRequestField,\":\",//crumb)= ") > +jenkins_crumb_value=3D$(curl -A "jenkins-sub-${PROJECT}" -s "http://${je= nkins_credentials}@${jenkins_url}/crumbIssuer/api/xml?xpath=3Dconcat(//crum= bRequestField,\":\",//crumb)") > > function jenkins_check_for_job() { > local jenkins_job=3Djenkins_${pw_project}_job > > - curl -s -f -X GET \ > + curl -A "jenkins-sub-${PROJECT}" -s -f -X GET \ > "http://${jenkins_credentials}@${jenkins_url}/job/${!jenkins_jo= b}/config.xml" >/dev/null > } > > @@ -86,6 +86,7 @@ function jenkins_submit_series() { > }' > > curl -s -f -X POST \ > + -A "jenkins-sub-${PROJECT}" \ > -H "${jenkins_crumb_value}" \ > --data token=3D"${!jenkins_job_token}" \ > --data-urlencode json=3D"$json_data" \ > diff --git a/post_pw.sh b/post_pw.sh > index e98e8ab..fe2f41c 100755 > --- a/post_pw.sh > +++ b/post_pw.sh > @@ -53,7 +53,7 @@ send_post() { > return 0 > fi > > - report=3D"$(curl -sSf "${link}")" > + report=3D"$(curl -A "pw-post" -sSf "${link}")" > if [ $? -ne 0 ]; then > echo "Failed to get proper server response on link ${link}" 1>&2 > return 0 > @@ -79,7 +79,7 @@ send_post() { > fi > > # Get reports from patch > - checks=3D"$(curl -sSf -X GET \ > + checks=3D"$(curl -A "pw-post" -sSf -X GET \ > --header "Content-Type: application/json" \ > "$api_url")" > if [ $? -ne 0 ]; then > @@ -105,7 +105,7 @@ send_post() { > \"description\": \"$description\"\ > }" > > - curl -sSf -X POST \ > + curl -A "pw-post" -sSf -X POST \ > -H "Authorization: Token ${token}" \ > --header "Content-Type: application/json" \ > --data "$data" \ > @@ -120,7 +120,7 @@ send_post() { > } > > year_month=3D"$(date +"%Y-%B")" > -reports=3D"$(curl -sSf "${mail_archive}${year_month}/thread.html" | \ > +reports=3D"$(curl -A "pw-post" -sSf "${mail_archive}${year_month}/thread= .html" | \ > grep -i 'HREF=3D' | sed -e 's@[0-9]*
  • if [ $? -ne 0 ]; then > echo "Failed to get proper server response on link ${reports}" 1>&2 > diff --git a/pw_mon b/pw_mon > index ee77256..7ff57cf 100755 > --- a/pw_mon > +++ b/pw_mon > @@ -146,7 +146,7 @@ function check_new_series() { > > GET_URL=3D"http://${INSTANCE}/api/series/?project=3D${PROJECT}&since= =3D${SINCE}" > > - response=3D$(curl -s "$userpw" "$GET_URL") > + response=3D$(curl -A "pw-mon-${PROJECT}" -s "$userpw" "$GET_URL") > > series_info=3D$(echo "$response" | jq -rc '.[] | (.id|tostring) + ";= " + .url + ";" + .submitter.name + ";" + .submitter.email + ";" + (.receive= d_all|tostring)') > > @@ -162,7 +162,7 @@ function check_new_series() { > function check_completed_series() { > get_uncompleted_jobs_as_line "$pw_instance" "$pw_project" | while IF= S=3D\| read -r id url submitter_name submitter_email; do > echo "Checking on series: $id" > - local RESPONSE=3D$(curl -s "$userpw" "$url" | jq -rc '.received_= all') > + local RESPONSE=3D$(curl -A "pw-mon-${PROJECT}" -s "$userpw" "$ur= l" | jq -rc '.received_all') > if [ "$RESPONSE" =3D "true" ]; then > echo "Setting series $id to completed" > series_id_set_complete "$pw_instance" "$id" > @@ -183,8 +183,8 @@ function check_superseded_series() { > local pw_instance=3D"$1" > series_get_active_branches "$pw_instance" | while IFS=3D\| read -r s= eries_id project url repo branchname; do > # first query the patch states > - local last_patch_url=3D$(curl -s "$userpw" "$url" | jq -rc '.pat= ches[-1].url') > - local patch_state=3D$(curl -s "$userpw" "$last_patch_url" | jq -= rc '.state') > + local last_patch_url=3D$(curl -A "pw-mon-${PROJECT}" -s "$userpw= " "$url" | jq -rc '.patches[-1].url') > + local patch_state=3D$(curl -A "pw-mon-${PROJECT}" -s "$userpw" "= $last_patch_url" | jq -rc '.state') > > # now check to see if the patch should even be reported... > if [ "$patch_state" =3D "superseded" -o "$patch_state" =3D "reje= cted" -o "$patch_state" =3D "accepted" \ > @@ -214,7 +214,7 @@ function check_patch_for_retest_request() { > local pw_project=3D"$2" > local patch_url=3D"$3" > > - local patch_json=3D$(curl -s "$userpw" "$patch_url") > + local patch_json=3D$(curl -A "pw-mon-${PROJECT}" -s "$userpw" "$patc= h_url") > local patch_comments_url=3D$(echo "$patch_json" | jq -rc '.comments'= ) > local patch_id=3D$(echo "$patch_json" | jq -rc '.id') > local series_id=3D$(echo "$patch_json" | jq -rc '.series[].id') > @@ -231,7 +231,7 @@ function check_patch_for_retest_request() { > fi > > if [ "Xnull" !=3D "X$patch_comments_url" ]; then > - local comments_json=3D$(curl -s "$userpw" "$patch_comments_url") > + local comments_json=3D$(curl -A "pw-mon-${PROJECT}" -s "$userpw"= "$patch_comments_url") > local seq_end=3D$(echo "$comments_json" | jq -rc 'length') > if [ "$seq_end" -a $seq_end -gt 0 ]; then > seq_end=3D$((seq_end-1)) > @@ -250,7 +250,7 @@ function check_series_needs_retest() { > local pw_instance=3D"$1" > local pw_project=3D"$2" > > - local series_list=3D$(curl -s "$userpw" "http://${pw_instance}/api/s= eries/?project=3D${pw_project}&state=3Dnew&state=3Drfc&state=3Dunder-review= &archived=3Dfalse&order=3D-id") > + local series_list=3D$(curl -A "pw-mon-${PROJECT}" -s "$userpw" "http= ://${pw_instance}/api/series/?project=3D${pw_project}&state=3Dnew&state=3Dr= fc&state=3Dunder-review&archived=3Dfalse&order=3D-id") > local n=3D$(echo "$series_list" | jq -rc 'length') > > if [ "Xnull" =3D=3D "X$n" -o "X" =3D=3D "X$n" ]; then > -- > 2.41.0 >