From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A6FDB433F8; Tue, 28 Nov 2023 22:16:19 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 839AB42DDD; Tue, 28 Nov 2023 22:16:19 +0100 (CET) Received: from mail-oa1-f48.google.com (mail-oa1-f48.google.com [209.85.160.48]) by mails.dpdk.org (Postfix) with ESMTP id 4FA64410FD for ; Tue, 28 Nov 2023 22:16:18 +0100 (CET) Received: by mail-oa1-f48.google.com with SMTP id 586e51a60fabf-1fa486a0e10so1786371fac.3 for ; Tue, 28 Nov 2023 13:16:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1701206177; x=1701810977; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=jaH/cZdkjF49cGBO0/TAFlWXP4NWa1JHtDIl1nvlots=; b=ObNkSEBHN9+gRbV8tYX2qRTpL5rMaE6us85SgK+6v+0OpH/i07tO7JJ5tAnLJh4C34 uW2+jmdqQ/EeruDB8HADsQA0E1RpIwUjrJMUeIqXqvlJs0+atAb1vdkUWQhD1uvYHBnq nZD1qodCByrjz2hq+BxXcgfSlpHvTCg7iE5h0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701206177; x=1701810977; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=jaH/cZdkjF49cGBO0/TAFlWXP4NWa1JHtDIl1nvlots=; b=rc7JQIcf228qbPgbK77KNC/JXvVyuNnEhiCn2Nvonzyxfxcg+34WvpOEJHz+cSujex Y0gHvxmp1RHDRqME6c08fOD7Ib7JBxe3pZgWmV+rFfzrnwcp3WK7BRcPVcxsJ/hnEh0v H2Ay/aJ6osc6kfPgX18uBFFZ+rw904VvYtUpYYX/5C8335pOOH2El8Jxz1mTeGiWmWNB dyMGdO8OgnvYD9pt0AKMS2ezTjHxPp9cjwbgYmy0ocmGWT0fPIwOIYXV2m3enNjdfa7B gPAFdssC53sloWNpAsRAn15xZb9078NckyTHjXgyEjG3Ubf02OHkKcUnNPK3ebl8kMO5 eYRg== X-Gm-Message-State: AOJu0YxCqc5bV6LakZhSigLHnxDLqxzrFWAr1PE3Urp6JdzLWhqJu1BP JITM4+A0V+seimrWpAVIm6pAJf+OpEMgAdVAlkzc5Q== X-Google-Smtp-Source: AGHT+IHx2PJrSiAl7DBMoi++/3H1i85iJ3FhcLB9+VD5NAsO7tsAc62cFuAnpnup9Qnra2m77YA/6CS0RwLRs+LxdRM= X-Received: by 2002:a05:6870:8dc3:b0:1fa:1d14:2a43 with SMTP id lq3-20020a0568708dc300b001fa1d142a43mr17541254oab.24.1701206177200; Tue, 28 Nov 2023 13:16:17 -0800 (PST) MIME-Version: 1.0 References: <5933180.BEx9A2HvPv@thomas> <5770736.7s5MMGUR32@thomas> In-Reply-To: From: Patrick Robb Date: Tue, 28 Nov 2023 16:16:06 -0500 Message-ID: Subject: Re: DPDK clone/fetch URL To: Thomas Monjalon Cc: ci@dpdk.org, alialnu@nvidia.com, Adam Hassick Content-Type: multipart/alternative; boundary="000000000000e1ce34060b3cedf5" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --000000000000e1ce34060b3cedf5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Thomas, Adam and I discussed and I agree with his points. His "patchseries apply" script rewrite will already alleviate this issue to a significant degree, but it will be even better if we can also use next-* mirrors on GitHub. Do you know who I can contact about this? Also, as of today our LTS testing is now moved over to GitHub, which should help a little. On Mon, Nov 27, 2023 at 4:30=E2=80=AFPM Adam Hassick = wrote: > Hi Thomas, > > Yes, mirroring the next-* branches would help to reduce the complexity of > creating patch series artifacts. > As Patrick wrote, moving the clones for the periodic testing on DPDK main > and DPDK LTS should be a trivial change. > > However, most of the load we put on that server originates from our > testing on new patch series. > I am working on a new script for creating the tarball artifacts from > incoming patch series. It only pulls down either 1 or 2 repositories rath= er > than main and all next-* branches for every single patch. > Once deployed, I expect this script to reduce the load on git.dpdk.org > significantly. > Also, I anticipate that I can modify this new script to pull main down > from GitHub. Then, only incoming patch series for next-* branches will > produce any load on git.dpdk.org. > > Thanks, > Adam Hassick > > On Mon, Nov 27, 2023 at 3:58=E2=80=AFPM Thomas Monjalon > wrote: > >> 27/11/2023 17:09, Patrick Robb: >> > Hello, >> > >> > Thanks for the heads up Thomas. For the Community Lab, as far as I can >> > tell, every git clone/fetch for DPDK main and DPDK LTS should be able >> to be >> > moved to the GitHub repo. >> > >> > On the other hand, I don't think next-* branches are mirrored on GitHu= b >> (at >> > least for now), so unless I'm mistaken, we will still have to pull fro= m >> > git.dpdk.org. But, between DPDK main and LTS, it should still be a >> sizable >> > reduction on the load put on your server. I'll discuss with the team >> today >> > and we will update here. >> >> We should mirror next-* trees as well I think, good point. >> >> >> > On Mon, Nov 27, 2023 at 6:21=E2=80=AFAM Thomas Monjalon >> wrote: >> > >> > > Hello, >> > > >> > > The server git.dpdk.org is not strong enough to hold all CI requests= . >> > > When a script is doing a git fetch or clone, it is consuming workloa= d >> on >> > > the server. >> > > Please could you favor the mirror on GitHub in any CI automation? >> > > https://github.com/DPDK/dpdk.git >> > > This mirror is updated as frequently as git.dpdk.org: >> > > We push to git.dpdk.org where a git hook is replicating the same on >> > > GitHub. >> > > >> > > Hope we can reduce the workload on our main server. >> > > Thank you everyone for collaborating. >> >> >> >> --=20 Patrick Robb Technical Service Manager UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 www.iol.unh.edu --000000000000e1ce34060b3cedf5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Thomas,

Adam and I discussed and I a= gree with his points. His "patchseries apply" script rewrite will= already alleviate=C2=A0this issue to a significant degree, but it will be = even better if we can also use next-* mirrors on GitHub. Do you know who I = can contact about this?

Also, as of today our LTS = testing is now moved over to GitHub, which should help a little.=C2=A0

On Mon, Nov 27, 2023 at 4:30=E2=80=AFPM Adam Hassick <ahassick@iol.unh.edu> wrote:
Hi Thomas,

Yes, mirroring the next-* bra= nches would help to reduce the complexity of creating patch series artifact= s.
As Patrick wrote, moving the clones for the periodic testing on DPDK = main and DPDK LTS should be a trivial change.

However, most of= the load we put on that server originates from our testing on new patch se= ries.
I am working on a new script for creating the tarball artifacts fr= om incoming patch series. It only pulls down either 1 or 2 repositories rat= her than main and all next-* branches for every single patch.
Once deplo= yed, I expect this script to reduce the load on git.dpdk.org significantly.
Also, I anticipat= e that I can modify this new script to pull main down from GitHub. Then, on= ly incoming patch series for next-* branches will produce any load on git.dpdk.org.

Thanks,
Adam Hassick

On Mon, Nov 27, 2023 at 3:58= =E2=80=AFPM Thomas Monjalon <thomas@monjalon.net> wrote:
27/11/2023 17:09, Patrick Robb:
> Hello,
>
> Thanks for the heads up Thomas. For the Community Lab, as far as I can=
> tell, every git clone/fetch for DPDK main and DPDK LTS should be able = to be
> moved to the GitHub repo.
>
> On the other hand, I don't think next-* branches are mirrored on G= itHub (at
> least for now), so unless I'm mistaken, we will still have to pull= from
> g= it.dpdk.org. But, between DPDK main and LTS, it should still be a sizab= le
> reduction on the load put on your server. I'll discuss with the te= am today
> and we will update here.

We should mirror next-* trees as well I think, good point.


> On Mon, Nov 27, 2023 at 6:21=E2=80=AFAM Thomas Monjalon <thomas@monjalon.net> = wrote:
>
> > Hello,
> >
> > The server git.dpdk.org is not strong enough to hold all CI request= s.
> > When a script is doing a git fetch or clone, it is consuming work= load on
> > the server.
> > Please could you favor the mirror on GitHub in any CI automation?=
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https://github.com/DPDK/d= pdk.git
> > This mirror is updated as frequently as git.dpdk.org:
> > We push to git.dpdk.org where a git hook is replicating the same on=
> > GitHub.
> >
> > Hope we can reduce the workload on our main server.
> > Thank you everyone for collaborating.





--

Patrick Robb

<= span style=3D"font-size:10pt;font-family:Arial;color:rgb(0,0,0);background-= color:transparent;vertical-align:baseline;white-space:pre-wrap">Technical S= ervice Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu


--000000000000e1ce34060b3cedf5--