Re: Question about pw-ci behavior

Re: Question about pw-ci behavior

[Aaron Conole]

Ali Alnubani <alialnu@nvidia.com> writes:

> Hello Aaron,
>
> I wanted to share some more statistics,
>
> I see that requests with the pw-ci user agent are making 3,200–4,200
> requests per hour (approximately 50–70 requests per minute).
> This volume appears excessive and places strain on the Patchwork server.
>
> Would you be able to check if polling frequency can be adjusted?

I'm adding a few things.  NOTE that pw-ci marks the series as done when
the states are 'finished' in the status details.  But for example the
listed series is still in a 'valid' state for polling.

I'm going to set it to mark for superceding the patches that are expired
by 30 days.  That should reduce the polling here.  Just had to deal with
a different issue on redirects with a different patchwork server (so I
added some new code).

> On 12/8/25 4:36 PM, Ali Alnubani wrote:
>> Hello Aaron, I hope you're well,
>>
>> I noticed that pw-ci (patches.dpdk.org) is making repeated Patchwork
>> API requests to very old and inactive series IDs. Example:
>>
>> """
>> 66.187.232.140 - - [08/Dec/2025:00:16:48 +0100] "GET /api/series/25093/ HTTP/1.1" 200 1351 "-" "(pw-ci) pw-mon-"
>> 66.187.232.140 - - [08/Dec/2025:00:36:07 +0100] "GET /api/series/25093/ HTTP/1.1" 200 1351 "-" "(pw-ci) pw-mon-"
>> [..]
>> 66.187.232.140 - - [08/Dec/2025:14:16:48 +0100] "GET /api/series/25093/ HTTP/1.1" 200 1351 "-" "(pw-ci) pw-mon-"
>> 66.187.232.140 - - [08/Dec/2025:14:37:25 +0100] "GET /api/series/25093/ HTTP/1.1" 200 1351 "-" "(pw-ci) pw-mon-"
>> 66.187.232.140 - - [08/Dec/2025:15:01:48 +0100] "GET /api/series/25093/ HTTP/1.1" 200 1351 "-" "(pw-ci) pw-mon-"
>> 66.187.232.140 - - [08/Dec/2025:15:17:13 +0100] "GET /api/series/25093/ HTTP/1.1" 200 1351 "-" "(pw-ci) pw-mon-"
>> """
>>
>> Is this expected/necessary?

Yes, because these series have weird state - they are missing patches,
so patchwork is holding them in a waiting state.  I'll add the
auto-clear code and hopefully that will reduce the amount of outstanding
patches we poll.

>> Thanks,
>> Ali

Re: Question about pw-ci behavior

[Aaron Conole]

Aaron Conole <aconole@redhat.com> writes:

> Ali Alnubani <alialnu@nvidia.com> writes:
>
>> Hello Aaron,
>>
>> I wanted to share some more statistics,
>>
>> I see that requests with the pw-ci user agent are making 3,200–4,200
>> requests per hour (approximately 50–70 requests per minute).
>> This volume appears excessive and places strain on the Patchwork server.
>>
>> Would you be able to check if polling frequency can be adjusted?
>
> I'm adding a few things.  NOTE that pw-ci marks the series as done when
> the states are 'finished' in the status details.  But for example the
> listed series is still in a 'valid' state for polling.
>
> I'm going to set it to mark for superceding the patches that are expired
> by 30 days.  That should reduce the polling here.  Just had to deal with
> a different issue on redirects with a different patchwork server (so I
> added some new code).
>

BTW, following is the patch I'm testing out right now just for the 'old'
ones:

diff --git a/pw_mon b/pw_mon
index 06457ee..d3154bc 100755
--- a/pw_mon
+++ b/pw_mon
@@ -173,11 +173,27 @@ function check_new_series() {
 function check_completed_series() {
     get_uncompleted_jobs_as_line "$pw_instance" "$pw_project" | while IFS=\| read -r id url submitter_name submitter_email; do
         echo "Checking on series: $id"
-        local RESPONSE=$(run_curl -A "(pw-ci) pw-mon-${PROJECT}" -s "$url" | jq -rc '.received_all')
+        local JSON=$(run_curl -A "(pw-ci) pw-mon-cc-${pw_project}" -s "$url")
+        local RESPONSE=$(echo "$JSON" | jq -rc '.received_all')
         if [ "$RESPONSE" = "true" ]; then
             echo "Setting series $id to completed"
             series_id_set_complete "$pw_instance" "$id"
         fi
+
+        # Should it be superseded based on a patch detail:
+        local last_patch_url=$(echo "$JSON" | jq -rc '.patches[-1].url')
+        local patch_state=$(run_curl -A "(pw-ci) pw-mon-csp-${PROJECT}" -s "$last_patch_url" | jq -rc '.state')
+
+        # now check to see if the patch should even be reported...
+        # This is for series that should be expired this way.
+        if [ "$patch_state" = "superseded" -o "$patch_state" = "rejected" -o "$patch_state" = "accepted" \
+             -o "$patch_state" = "changes-requested" -o "$patch_state" = "not-applicable" ]; then
+            # Clear the job
+            series_id_set_complete "$pw_instance" "$id"
+            series_clear_branch "$pw_instance" "$id"
+            set_synced_for_series "$id" "$pw_instance"
+        fi
+
     done
     return 0
 }
---

Re: Question about pw-ci behavior

[Aaron Conole]

Aaron Conole <aconole@redhat.com> writes:

> Aaron Conole <aconole@redhat.com> writes:
>
>> Ali Alnubani <alialnu@nvidia.com> writes:
>>
>>> Hello Aaron,
>>>
>>> I wanted to share some more statistics,
>>>
>>> I see that requests with the pw-ci user agent are making 3,200–4,200
>>> requests per hour (approximately 50–70 requests per minute).
>>> This volume appears excessive and places strain on the Patchwork server.
>>>
>>> Would you be able to check if polling frequency can be adjusted?
>>
>> I'm adding a few things.  NOTE that pw-ci marks the series as done when
>> the states are 'finished' in the status details.  But for example the
>> listed series is still in a 'valid' state for polling.
>>
>> I'm going to set it to mark for superceding the patches that are expired
>> by 30 days.  That should reduce the polling here.  Just had to deal with
>> a different issue on redirects with a different patchwork server (so I
>> added some new code).
>>
>
> BTW, following is the patch I'm testing out right now just for the 'old'
> ones:
>

I've added a date check to the patch as well, and am locally testing it.
When it looks right, I'll push and there will be one last "large" query
and that should check the dates and expire the outstanding series that
we poll against.

Re: Question about pw-ci behavior

[Ali Alnubani]

On 12/9/25 6:15 PM, Aaron Conole wrote:

> Aaron Conole <aconole@redhat.com> writes:
>
>> Aaron Conole <aconole@redhat.com> writes:
>>
>>> Ali Alnubani <alialnu@nvidia.com> writes:
>>>
>>>> Hello Aaron,
>>>>
>>>> I wanted to share some more statistics,
>>>>
>>>> I see that requests with the pw-ci user agent are making 3,200–4,200
>>>> requests per hour (approximately 50–70 requests per minute).
>>>> This volume appears excessive and places strain on the Patchwork server.
>>>>
>>>> Would you be able to check if polling frequency can be adjusted?
>>> I'm adding a few things.  NOTE that pw-ci marks the series as done when
>>> the states are 'finished' in the status details.  But for example the
>>> listed series is still in a 'valid' state for polling.
>>>
>>> I'm going to set it to mark for superceding the patches that are expired
>>> by 30 days.  That should reduce the polling here.  Just had to deal with
>>> a different issue on redirects with a different patchwork server (so I
>>> added some new code).
>>>
>> BTW, following is the patch I'm testing out right now just for the 'old'
>> ones:
>>
> I've added a date check to the patch as well, and am locally testing it.
> When it looks right, I'll push and there will be one last "large" query
> and that should check the dates and expire the outstanding series that
> we poll against.
>
Thank you for helping with this and for the updates,

Regards,
Ali

Re: Question about pw-ci behavior

[Aaron Conole]

Ali Alnubani <alialnu@nvidia.com> writes:

> On 12/9/25 6:15 PM, Aaron Conole wrote:
>
>> Aaron Conole <aconole@redhat.com> writes:
>>
>>> Aaron Conole <aconole@redhat.com> writes:
>>>
>>>> Ali Alnubani <alialnu@nvidia.com> writes:
>>>>
>>>>> Hello Aaron,
>>>>>
>>>>> I wanted to share some more statistics,
>>>>>
>>>>> I see that requests with the pw-ci user agent are making 3,200–4,200
>>>>> requests per hour (approximately 50–70 requests per minute).
>>>>> This volume appears excessive and places strain on the Patchwork server.
>>>>>
>>>>> Would you be able to check if polling frequency can be adjusted?
>>>> I'm adding a few things.  NOTE that pw-ci marks the series as done when
>>>> the states are 'finished' in the status details.  But for example the
>>>> listed series is still in a 'valid' state for polling.
>>>>
>>>> I'm going to set it to mark for superceding the patches that are expired
>>>> by 30 days.  That should reduce the polling here.  Just had to deal with
>>>> a different issue on redirects with a different patchwork server (so I
>>>> added some new code).
>>>>
>>> BTW, following is the patch I'm testing out right now just for the 'old'
>>> ones:
>>>
>> I've added a date check to the patch as well, and am locally testing it.
>> When it looks right, I'll push and there will be one last "large" query
>> and that should check the dates and expire the outstanding series that
>> we poll against.
>>
> Thank you for helping with this and for the updates,

I pushed it live.  I hope this cools off the requests.

> Regards,
> Ali

Re: Question about pw-ci behavior

[Ali Alnubani]

On 12/9/25 9:47 PM, Aaron Conole wrote:

> Ali Alnubani <alialnu@nvidia.com> writes:
>
>> On 12/9/25 6:15 PM, Aaron Conole wrote:
>>
>>> Aaron Conole <aconole@redhat.com> writes:
>>>
>>>> Aaron Conole <aconole@redhat.com> writes:
>>>>
>>>>> Ali Alnubani <alialnu@nvidia.com> writes:
>>>>>
>>>>>> Hello Aaron,
>>>>>>
>>>>>> I wanted to share some more statistics,
>>>>>>
>>>>>> I see that requests with the pw-ci user agent are making 3,200–4,200
>>>>>> requests per hour (approximately 50–70 requests per minute).
>>>>>> This volume appears excessive and places strain on the Patchwork server.
>>>>>>
>>>>>> Would you be able to check if polling frequency can be adjusted?
>>>>> I'm adding a few things.  NOTE that pw-ci marks the series as done when
>>>>> the states are 'finished' in the status details.  But for example the
>>>>> listed series is still in a 'valid' state for polling.
>>>>>
>>>>> I'm going to set it to mark for superceding the patches that are expired
>>>>> by 30 days.  That should reduce the polling here.  Just had to deal with
>>>>> a different issue on redirects with a different patchwork server (so I
>>>>> added some new code).
>>>>>
>>>> BTW, following is the patch I'm testing out right now just for the 'old'
>>>> ones:
>>>>
>>> I've added a date check to the patch as well, and am locally testing it.
>>> When it looks right, I'll push and there will be one last "large" query
>>> and that should check the dates and expire the outstanding series that
>>> we poll against.
>>>
>> Thank you for helping with this and for the updates,
> I pushed it live.  I hope this cools off the requests.

I’m seeing a 60–70% reduction in requests per hour — thank you again for 
your help with this.

Regards,
Ali

Re: Question about pw-ci behavior

[Thomas Monjalon]

09/12/2025 17:15, Aaron Conole:
> I've added a date check to the patch as well, and am locally testing it.
> When it looks right, I'll push and there will be one last "large" query
> and that should check the dates and expire the outstanding series that
> we poll against.

What do you mean by "expire"?
You are not changing any state in patchwork, right?

Re: Question about pw-ci behavior

[Aaron Conole]

Thomas Monjalon <thomas@monjalon.net> writes:

> 09/12/2025 17:15, Aaron Conole:
>> I've added a date check to the patch as well, and am locally testing it.
>> When it looks right, I'll push and there will be one last "large" query
>> and that should check the dates and expire the outstanding series that
>> we poll against.
>
> What do you mean by "expire"?
> You are not changing any state in patchwork, right?

No.  Expiring it from pw-ci polling.  pw-ci keeps a cache of all the
patches it thinks it should monitor.  The issue is that we had all sorts
of partial series, or series that are very old and don't have a proper
state (for instance, maybe they got archived with state still as 'new')
we were monitoring them.