DPDK clone/fetch URL

DPDK clone/fetch URL

[Thomas Monjalon]

Hello,

The server git.dpdk.org is not strong enough to hold all CI requests.
When a script is doing a git fetch or clone, it is consuming workload on the server.
Please could you favor the mirror on GitHub in any CI automation?
	https://github.com/DPDK/dpdk.git
This mirror is updated as frequently as git.dpdk.org:
We push to git.dpdk.org where a git hook is replicating the same on GitHub.

Hope we can reduce the workload on our main server.
Thank you everyone for collaborating.

Re: DPDK clone/fetch URL

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 1309 bytes --]

Hello,

Thanks for the heads up Thomas. For the Community Lab, as far as I can
tell, every git clone/fetch for DPDK main and DPDK LTS should be able to be
moved to the GitHub repo.

On the other hand, I don't think next-* branches are mirrored on GitHub (at
least for now), so unless I'm mistaken, we will still have to pull from
git.dpdk.org. But, between DPDK main and LTS, it should still be a sizable
reduction on the load put on your server. I'll discuss with the team today
and we will update here.

Best,
Patrick


On Mon, Nov 27, 2023 at 6:21 AM Thomas Monjalon <thomas@monjalon.net> wrote:

> Hello,
>
> The server git.dpdk.org is not strong enough to hold all CI requests.
> When a script is doing a git fetch or clone, it is consuming workload on
> the server.
> Please could you favor the mirror on GitHub in any CI automation?
>         https://github.com/DPDK/dpdk.git
> This mirror is updated as frequently as git.dpdk.org:
> We push to git.dpdk.org where a git hook is replicating the same on
> GitHub.
>
> Hope we can reduce the workload on our main server.
> Thank you everyone for collaborating.
>
>
>

-- 

Patrick Robb

Technical Service Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu

[-- Attachment #2: Type: text/html, Size: 3931 bytes --]

Re: DPDK clone/fetch URL

[Thomas Monjalon]

27/11/2023 17:09, Patrick Robb:
> Hello,
> 
> Thanks for the heads up Thomas. For the Community Lab, as far as I can
> tell, every git clone/fetch for DPDK main and DPDK LTS should be able to be
> moved to the GitHub repo.
> 
> On the other hand, I don't think next-* branches are mirrored on GitHub (at
> least for now), so unless I'm mistaken, we will still have to pull from
> git.dpdk.org. But, between DPDK main and LTS, it should still be a sizable
> reduction on the load put on your server. I'll discuss with the team today
> and we will update here.

We should mirror next-* trees as well I think, good point.


> On Mon, Nov 27, 2023 at 6:21 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> 
> > Hello,
> >
> > The server git.dpdk.org is not strong enough to hold all CI requests.
> > When a script is doing a git fetch or clone, it is consuming workload on
> > the server.
> > Please could you favor the mirror on GitHub in any CI automation?
> >         https://github.com/DPDK/dpdk.git
> > This mirror is updated as frequently as git.dpdk.org:
> > We push to git.dpdk.org where a git hook is replicating the same on
> > GitHub.
> >
> > Hope we can reduce the workload on our main server.
> > Thank you everyone for collaborating.

Re: DPDK clone/fetch URL

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 2286 bytes --]

Hi Thomas,

Yes, mirroring the next-* branches would help to reduce the complexity of
creating patch series artifacts.
As Patrick wrote, moving the clones for the periodic testing on DPDK main
and DPDK LTS should be a trivial change.

However, most of the load we put on that server originates from our testing
on new patch series.
I am working on a new script for creating the tarball artifacts from
incoming patch series. It only pulls down either 1 or 2 repositories rather
than main and all next-* branches for every single patch.
Once deployed, I expect this script to reduce the load on git.dpdk.org
significantly.
Also, I anticipate that I can modify this new script to pull main down from
GitHub. Then, only incoming patch series for next-* branches will produce
any load on git.dpdk.org.

Thanks,
Adam Hassick

On Mon, Nov 27, 2023 at 3:58 PM Thomas Monjalon <thomas@monjalon.net> wrote:

> 27/11/2023 17:09, Patrick Robb:
> > Hello,
> >
> > Thanks for the heads up Thomas. For the Community Lab, as far as I can
> > tell, every git clone/fetch for DPDK main and DPDK LTS should be able to
> be
> > moved to the GitHub repo.
> >
> > On the other hand, I don't think next-* branches are mirrored on GitHub
> (at
> > least for now), so unless I'm mistaken, we will still have to pull from
> > git.dpdk.org. But, between DPDK main and LTS, it should still be a
> sizable
> > reduction on the load put on your server. I'll discuss with the team
> today
> > and we will update here.
>
> We should mirror next-* trees as well I think, good point.
>
>
> > On Mon, Nov 27, 2023 at 6:21 AM Thomas Monjalon <thomas@monjalon.net>
> wrote:
> >
> > > Hello,
> > >
> > > The server git.dpdk.org is not strong enough to hold all CI requests.
> > > When a script is doing a git fetch or clone, it is consuming workload
> on
> > > the server.
> > > Please could you favor the mirror on GitHub in any CI automation?
> > >         https://github.com/DPDK/dpdk.git
> > > This mirror is updated as frequently as git.dpdk.org:
> > > We push to git.dpdk.org where a git hook is replicating the same on
> > > GitHub.
> > >
> > > Hope we can reduce the workload on our main server.
> > > Thank you everyone for collaborating.
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 3324 bytes --]

Re: DPDK clone/fetch URL

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 2980 bytes --]

Hi Thomas,

Adam and I discussed and I agree with his points. His "patchseries apply"
script rewrite will already alleviate this issue to a significant degree,
but it will be even better if we can also use next-* mirrors on GitHub. Do
you know who I can contact about this?

Also, as of today our LTS testing is now moved over to GitHub, which should
help a little.

On Mon, Nov 27, 2023 at 4:30 PM Adam Hassick <ahassick@iol.unh.edu> wrote:

> Hi Thomas,
>
> Yes, mirroring the next-* branches would help to reduce the complexity of
> creating patch series artifacts.
> As Patrick wrote, moving the clones for the periodic testing on DPDK main
> and DPDK LTS should be a trivial change.
>
> However, most of the load we put on that server originates from our
> testing on new patch series.
> I am working on a new script for creating the tarball artifacts from
> incoming patch series. It only pulls down either 1 or 2 repositories rather
> than main and all next-* branches for every single patch.
> Once deployed, I expect this script to reduce the load on git.dpdk.org
> significantly.
> Also, I anticipate that I can modify this new script to pull main down
> from GitHub. Then, only incoming patch series for next-* branches will
> produce any load on git.dpdk.org.
>
> Thanks,
> Adam Hassick
>
> On Mon, Nov 27, 2023 at 3:58 PM Thomas Monjalon <thomas@monjalon.net>
> wrote:
>
>> 27/11/2023 17:09, Patrick Robb:
>> > Hello,
>> >
>> > Thanks for the heads up Thomas. For the Community Lab, as far as I can
>> > tell, every git clone/fetch for DPDK main and DPDK LTS should be able
>> to be
>> > moved to the GitHub repo.
>> >
>> > On the other hand, I don't think next-* branches are mirrored on GitHub
>> (at
>> > least for now), so unless I'm mistaken, we will still have to pull from
>> > git.dpdk.org. But, between DPDK main and LTS, it should still be a
>> sizable
>> > reduction on the load put on your server. I'll discuss with the team
>> today
>> > and we will update here.
>>
>> We should mirror next-* trees as well I think, good point.
>>
>>
>> > On Mon, Nov 27, 2023 at 6:21 AM Thomas Monjalon <thomas@monjalon.net>
>> wrote:
>> >
>> > > Hello,
>> > >
>> > > The server git.dpdk.org is not strong enough to hold all CI requests.
>> > > When a script is doing a git fetch or clone, it is consuming workload
>> on
>> > > the server.
>> > > Please could you favor the mirror on GitHub in any CI automation?
>> > >         https://github.com/DPDK/dpdk.git
>> > > This mirror is updated as frequently as git.dpdk.org:
>> > > We push to git.dpdk.org where a git hook is replicating the same on
>> > > GitHub.
>> > >
>> > > Hope we can reduce the workload on our main server.
>> > > Thank you everyone for collaborating.
>>
>>
>>
>>

-- 

Patrick Robb

Technical Service Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu

[-- Attachment #2: Type: text/html, Size: 6210 bytes --]

Re: DPDK clone/fetch URL

[David Marchand]

Hello guys,

On Mon, Nov 27, 2023 at 10:30 PM Adam Hassick <ahassick@iol.unh.edu> wrote:
>
> Hi Thomas,
>
> Yes, mirroring the next-* branches would help to reduce the complexity of creating patch series artifacts.
> As Patrick wrote, moving the clones for the periodic testing on DPDK main and DPDK LTS should be a trivial change.
>
> However, most of the load we put on that server originates from our testing on new patch series.
> I am working on a new script for creating the tarball artifacts from incoming patch series. It only pulls down either 1 or 2 repositories rather than main and all next-* branches for every single patch.
> Once deployed, I expect this script to reduce the load on git.dpdk.org significantly.
> Also, I anticipate that I can modify this new script to pull main down from GitHub. Then, only incoming patch series for next-* branches will produce any load on git.dpdk.org.

As mentionned in
https://inbox.dpdk.org/dev/CAJFAV8xSVqkTjrqSL0gWkg9Y+EyMg4=jxwOOzhtaXp6HDeG_cQ@mail.gmail.com/T/#t,
the mirroring script is ready on dpdk.org.
The branches are not up yet, I'll put some update in this thread when they are.


-- 
David Marchand

Re: DPDK clone/fetch URL

[Aaron Conole]

Thomas Monjalon <thomas@monjalon.net> writes:

> Hello,
>
> The server git.dpdk.org is not strong enough to hold all CI requests.
> When a script is doing a git fetch or clone, it is consuming workload on the server.
> Please could you favor the mirror on GitHub in any CI automation?
> 	https://github.com/DPDK/dpdk.git
> This mirror is updated as frequently as git.dpdk.org:
> We push to git.dpdk.org where a git hook is replicating the same on GitHub.
>
> Hope we can reduce the workload on our main server.
> Thank you everyone for collaborating.

Just fixed the github-robot to use the github DPDK mirror.

Re: DPDK clone/fetch URL

[Thomas Monjalon]

30/11/2023 10:27, David Marchand:
> On Mon, Nov 27, 2023 at 10:30 PM Adam Hassick <ahassick@iol.unh.edu> wrote:
> > Yes, mirroring the next-* branches would help to reduce the complexity of creating patch series artifacts.
> > As Patrick wrote, moving the clones for the periodic testing on DPDK main and DPDK LTS should be a trivial change.
> >
> > However, most of the load we put on that server originates from our testing on new patch series.
> > I am working on a new script for creating the tarball artifacts from incoming patch series. It only pulls down either 1 or 2 repositories rather than main and all next-* branches for every single patch.
> > Once deployed, I expect this script to reduce the load on git.dpdk.org significantly.
> > Also, I anticipate that I can modify this new script to pull main down from GitHub. Then, only incoming patch series for next-* branches will produce any load on git.dpdk.org.
> 
> As mentionned in
> https://inbox.dpdk.org/dev/CAJFAV8xSVqkTjrqSL0gWkg9Y+EyMg4=jxwOOzhtaXp6HDeG_cQ@mail.gmail.com/T/#t,
> the mirroring script is ready on dpdk.org.
> The branches are not up yet, I'll put some update in this thread when they are.

The branches are automatically created on GitHub
when there is a push to the corresponding branch on dpdk.org.
Some branches are already created on GitHub.

When testing a patch, you should get a repository name from
the script pw_maintainers_cli.py:
https://git.dpdk.org/tools/dpdk-ci/tree/tools/pw_maintainers_cli.py

We could work on this script to get the GitHub branch name.
For example, instead of git://dpdk.org/next/dpdk-next-net
it should be the GitHub branch name next-net-for-main-repo.

The branch name is the dpdk.org repository name without "dpdk-"
followed by the default branch of the repository on dpdk.org.

This is the list of branches to use on GitHub for patch testing:
	dpdk/main
	dpdk/next-baseband-for-main
	dpdk/next-crypto-for-main
	dpdk/next-eventdev-for-main
	dpdk/next-net-for-main-repo
	dpdk/next-net-brcm-for-next-net
	dpdk/next-net-intel-for-next-net
	dpdk/next-net-mlx-for-next-net
	dpdk/next-net-mrvl-for-main
	dpdk/next-virtio-for-next-net

In addition, we will have some "useless" branches:
	dpdk/next-net-main
	dpdk/next-net-brcm-main
	dpdk/next-net-brcm-pre-submit
	dpdk/next-net-intel-main
	dpdk/next-net-mlx-main
	dpdk/next-virtio-staging
	dpdk/next-baseband-staging
No automatic test should be run on these branches.

And these are the branches that we should test on each push:
	dpdk-stable/21.11-staging
	dpdk-stable/22.11-staging
	dpdk-stable/23.11-staging (to be created)

Note: we could discuss later about merging dpdk and dpdk-stable
in the same GitHub repository.