Addressing git.dpdk.org downtime

Addressing git.dpdk.org downtime

[Ali Alnubani]

Hello everyone,

We’re aware that https://git.dpdk.org has been experiencing availability 
issues lately, largely due to abusive bots and crawlers overloading the servers and ignoring traditional controls like robots.txt.

To address this, we're preparing to deploy Anubis, a traffic-challenging tool, to help ensure reliable and continued access for the community, starting with our cgit instance. This approach has been adopted by several other open-source projects recently to reduce disruptive automated traffic.

Please reach out if you have questions or encounter issues.

See: https://github.com/TecharoHQ/anubis

Thanks,
Ali

Re: Addressing git.dpdk.org downtime

[Bruce Richardson]

On Thu, May 22, 2025 at 08:36:28AM +0000, Ali Alnubani wrote:
> Hello everyone,
> 
> We’re aware that https://git.dpdk.org has been experiencing availability 
> issues lately, largely due to abusive bots and crawlers overloading the servers and ignoring traditional controls like robots.txt.
> 
> To address this, we're preparing to deploy Anubis, a traffic-challenging tool, to help ensure reliable and continued access for the community, starting with our cgit instance. This approach has been adopted by several other open-source projects recently to reduce disruptive automated traffic.
> 
> Please reach out if you have questions or encounter issues.
> 
> See: https://github.com/TecharoHQ/anubis
> 
> Thanks,
> Ali

Thanks Ali, (and others involved) for taking action on this.

To everyone else in the community, just a reminder that, rather than
accessing dpdk.org site directly, you can use
"https://github.com/DPDK/dpdk" as a source for getting and browsing the
DPDK source code. The github mirror is always in sync with the main tree,
and, as an added convenience, also has the various "next" trees mirrored to
it too. This means that you don't need to clone multiple trees to get e.g.
next-net tree as well as main tree.

For any companies looking to have internal mirrors of the DPDK repository
for internal use, it's recommended that those mirrors be based on github
rather than on dpdk.org directly. This all helps reduce load on the server.

Thanks,
/Bruce

Re: Addressing git.dpdk.org downtime

[David Marchand]

On Thu, May 22, 2025 at 11:06 AM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Thu, May 22, 2025 at 08:36:28AM +0000, Ali Alnubani wrote:
> > Hello everyone,
> >
> > We’re aware that https://git.dpdk.org has been experiencing availability
> > issues lately, largely due to abusive bots and crawlers overloading the servers and ignoring traditional controls like robots.txt.
> >
> > To address this, we're preparing to deploy Anubis, a traffic-challenging tool, to help ensure reliable and continued access for the community, starting with our cgit instance. This approach has been adopted by several other open-source projects recently to reduce disruptive automated traffic.
> >
> > Please reach out if you have questions or encounter issues.
> >
> > See: https://github.com/TecharoHQ/anubis
> >
> > Thanks,
> > Ali
>
> Thanks Ali, (and others involved) for taking action on this.
>
> To everyone else in the community, just a reminder that, rather than
> accessing dpdk.org site directly, you can use
> "https://github.com/DPDK/dpdk" as a source for getting and browsing the
> DPDK source code. The github mirror is always in sync with the main tree,
> and, as an added convenience, also has the various "next" trees mirrored to
> it too. This means that you don't need to clone multiple trees to get e.g.
> next-net tree as well as main tree.

To add on top of this, we discussed some additional mirroring during
the techboard in Prague to ease life for (lazy?) maintainers and users
who are not confortable with manipulating different git repositories.
One request was to add mirroring of the stable branches and tags in a
single repository with the main DPDK branch, and the DPDK/dpdk github
repository seemed like the best location.

I have just set up this mirroring down to 20.11 branch.
The DPDK/dpdk github repository now contains most (if not all) of the
active branches and tags.


-- 
David Marchand

Re: Addressing git.dpdk.org downtime

[Ali Alnubani]

On 5/22/25 11:36 AM, Ali Alnubani wrote:

> Hello everyone,
>
> We’re aware that https://git.dpdk.org has been experiencing availability
> issues lately, largely due to abusive bots and crawlers overloading the servers and ignoring traditional controls like robots.txt.
>
> To address this, we're preparing to deploy Anubis, a traffic-challenging tool, to help ensure reliable and continued access for the community, starting with our cgit instance. This approach has been adopted by several other open-source projects recently to reduce disruptive automated traffic.
>
> Please reach out if you have questions or encounter issues.
>
> See: https://github.com/TecharoHQ/anubis
>
> Thanks,
> Ali

The new service is now live on git.dpdk.org. Regular browsing and git 
operations should work as usual.
HTTP traffic is now forcefully redirected to HTTPS as part of this change.

Thanks for your understanding and support,

Regards,
Ali