From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8820F433E7; Mon, 27 Nov 2023 22:30:30 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 62F7E40A6B; Mon, 27 Nov 2023 22:30:30 +0100 (CET) Received: from mail-yb1-f175.google.com (mail-yb1-f175.google.com [209.85.219.175]) by mails.dpdk.org (Postfix) with ESMTP id AA21640156 for ; Mon, 27 Nov 2023 22:30:28 +0100 (CET) Received: by mail-yb1-f175.google.com with SMTP id 3f1490d57ef6-db35caa1749so4253836276.2 for ; Mon, 27 Nov 2023 13:30:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1701120628; x=1701725428; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=dKZUy94Ya04LaZOnyogZxowT5rgozruxTXOr9J0Wmus=; b=EkW9iRtVg0rQOgCLLGxgOJKm5Fk1KmL99g7z5yePTO700WVsPJG2O1RSA/Rl44ohnG +vGWVUW0Aay8Z65puAYpQRy7b5G2rUMRadfBgy6ruqcv1htdmVVL0oBXRoT8TO4aW1qk YKCy1iH3X+Nvj7dsxupGXeiBjmzdExMiW4SGg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701120628; x=1701725428; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=dKZUy94Ya04LaZOnyogZxowT5rgozruxTXOr9J0Wmus=; b=B5+t/6Nn+h9n/BKNka1JJQLZgqTGEYQmKE0Fyb++GpyRMcA8dBmZMDnKqcg+hVUQzu d5HFW2+x958uG9InjJHHOzalzmAoEJiio4erl7L91nq+mt150i2X5LLwgXneN7bakpkc la7u7fnMyqXop5zQs7yjqc0NQLACG1Sz4n7FmzMzQyWsK1y+ybn29UG0yUN7sLAycA2N agssxa7AGJ7YJ7J72pCgO8cJk7W2EaSeoHx30xzrhIsCMBu8K9YuD0vnATo5nH8nW1Wd rGBueA1rof3N6AKaWMlUtUf0dhUV71ONt3HSV9XQbMBSa2PNYWdXER/Bib0QwliUtSll J8Uw== X-Gm-Message-State: AOJu0YycvFIJbcIhtBY0lCZDG1SgLORk5xlzEbU+jKSZoXE5vq9a7f0R cI65H3y5DkDtjAeDxSYmD2ZI6TZrMIUpoJ9+E3Jb+g== X-Google-Smtp-Source: AGHT+IHfKFpr/eUkO1o9RKaGQYbZim8fGiCw6Cw837NzjTUNWCtEeJYC279a2/4i5ErvqD7KjocXZ5EACY2gTyOqFyQ= X-Received: by 2002:a5b:749:0:b0:da0:66ed:ea1e with SMTP id s9-20020a5b0749000000b00da066edea1emr12779414ybq.11.1701120627866; Mon, 27 Nov 2023 13:30:27 -0800 (PST) MIME-Version: 1.0 References: <5933180.BEx9A2HvPv@thomas> <5770736.7s5MMGUR32@thomas> In-Reply-To: <5770736.7s5MMGUR32@thomas> From: Adam Hassick Date: Mon, 27 Nov 2023 16:32:54 -0500 Message-ID: Subject: Re: DPDK clone/fetch URL To: Thomas Monjalon Cc: Patrick Robb , ci@dpdk.org, alialnu@nvidia.com Content-Type: multipart/alternative; boundary="000000000000be9742060b290239" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --000000000000be9742060b290239 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Thomas, Yes, mirroring the next-* branches would help to reduce the complexity of creating patch series artifacts. As Patrick wrote, moving the clones for the periodic testing on DPDK main and DPDK LTS should be a trivial change. However, most of the load we put on that server originates from our testing on new patch series. I am working on a new script for creating the tarball artifacts from incoming patch series. It only pulls down either 1 or 2 repositories rather than main and all next-* branches for every single patch. Once deployed, I expect this script to reduce the load on git.dpdk.org significantly. Also, I anticipate that I can modify this new script to pull main down from GitHub. Then, only incoming patch series for next-* branches will produce any load on git.dpdk.org. Thanks, Adam Hassick On Mon, Nov 27, 2023 at 3:58=E2=80=AFPM Thomas Monjalon wrote: > 27/11/2023 17:09, Patrick Robb: > > Hello, > > > > Thanks for the heads up Thomas. For the Community Lab, as far as I can > > tell, every git clone/fetch for DPDK main and DPDK LTS should be able t= o > be > > moved to the GitHub repo. > > > > On the other hand, I don't think next-* branches are mirrored on GitHub > (at > > least for now), so unless I'm mistaken, we will still have to pull from > > git.dpdk.org. But, between DPDK main and LTS, it should still be a > sizable > > reduction on the load put on your server. I'll discuss with the team > today > > and we will update here. > > We should mirror next-* trees as well I think, good point. > > > > On Mon, Nov 27, 2023 at 6:21=E2=80=AFAM Thomas Monjalon > wrote: > > > > > Hello, > > > > > > The server git.dpdk.org is not strong enough to hold all CI requests. > > > When a script is doing a git fetch or clone, it is consuming workload > on > > > the server. > > > Please could you favor the mirror on GitHub in any CI automation? > > > https://github.com/DPDK/dpdk.git > > > This mirror is updated as frequently as git.dpdk.org: > > > We push to git.dpdk.org where a git hook is replicating the same on > > > GitHub. > > > > > > Hope we can reduce the workload on our main server. > > > Thank you everyone for collaborating. > > > > --000000000000be9742060b290239 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Thomas,

Yes, mirroring the next-* branches would help to reduce the complexity o= f creating patch series artifacts.
As Patrick wrote, moving the clones f= or the periodic testing on DPDK main and DPDK LTS should be a trivial chang= e.

However, most of the load we put on that server originates = from our testing on new patch series.
I am working on a new script for c= reating the tarball artifacts from incoming patch series. It only pulls dow= n either 1 or 2 repositories rather than main and all next-* branches for e= very single patch.
Once deployed, I expect this script to reduce the loa= d on git.dpdk.org significantly.
Als= o, I anticipate that I can modify this new script to pull main down from Gi= tHub. Then, only incoming patch series for next-* branches will produce any= load on git.dpdk.org.
Thanks,
Adam Hassick

On Mon, Nov 27, 2023 at 3:58=E2=80= =AFPM Thomas Monjalon <thomas@mon= jalon.net> wrote:
27/11/2023 17:09, Patrick Robb:
> Hello,
>
> Thanks for the heads up Thomas. For the Community Lab, as far as I can=
> tell, every git clone/fetch for DPDK main and DPDK LTS should be able = to be
> moved to the GitHub repo.
>
> On the other hand, I don't think next-* branches are mirrored on G= itHub (at
> least for now), so unless I'm mistaken, we will still have to pull= from
> g= it.dpdk.org. But, between DPDK main and LTS, it should still be a sizab= le
> reduction on the load put on your server. I'll discuss with the te= am today
> and we will update here.

We should mirror next-* trees as well I think, good point.


> On Mon, Nov 27, 2023 at 6:21=E2=80=AFAM Thomas Monjalon <thomas@monjalon.net> = wrote:
>
> > Hello,
> >
> > The server git.dpdk.org is not strong enough to hold all CI request= s.
> > When a script is doing a git fetch or clone, it is consuming work= load on
> > the server.
> > Please could you favor the mirror on GitHub in any CI automation?=
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https://github.com/DPDK/d= pdk.git
> > This mirror is updated as frequently as git.dpdk.org:
> > We push to git.dpdk.org where a git hook is replicating the same on=
> > GitHub.
> >
> > Hope we can reduce the workload on our main server.
> > Thank you everyone for collaborating.



--000000000000be9742060b290239--