* [PATCH] doc: ensure sphinx output is reproducible
@ 2023-06-29 12:58 christian.ehrhardt
2023-06-29 13:02 ` Christian Ehrhardt
2023-07-03 15:29 ` Thomas Monjalon
0 siblings, 2 replies; 6+ messages in thread
From: christian.ehrhardt @ 2023-06-29 12:58 UTC (permalink / raw)
To: dev; +Cc: Luca Boccassi, Christian Ehrhardt
From: Christian Ehrhardt <christian.ehrhardt@canonical.com>
By adding -j we build in parallel, to make building on multiprocessor
machines more effective. While that works it does also break
reproducible builds as the order of the sphinx generated searchindex.js
is depending on execution speed of the individual processes.
Signed-off-by: Christian Ehrhardt <christian.ehrhardt@canonical.com>
---
buildtools/call-sphinx-build.py | 5 -----
1 file changed, 5 deletions(-)
diff --git a/buildtools/call-sphinx-build.py b/buildtools/call-sphinx-build.py
index 39a60d09fa..d8879306de 100755
--- a/buildtools/call-sphinx-build.py
+++ b/buildtools/call-sphinx-build.py
@@ -15,12 +15,7 @@
# set the version in environment for sphinx to pick up
os.environ['DPDK_VERSION'] = version
-# for sphinx version >= 1.7 add parallelism using "-j auto"
-ver = run([sphinx, '--version'], stdout=PIPE,
- stderr=STDOUT).stdout.decode().split()[-1]
sphinx_cmd = [sphinx] + extra_args
-if Version(ver) >= Version('1.7'):
- sphinx_cmd += ['-j', 'auto']
# find all the files sphinx will process so we can write them as dependencies
srcfiles = []
--
2.41.0
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] doc: ensure sphinx output is reproducible
2023-06-29 12:58 [PATCH] doc: ensure sphinx output is reproducible christian.ehrhardt
@ 2023-06-29 13:02 ` Christian Ehrhardt
2023-07-03 15:29 ` Thomas Monjalon
1 sibling, 0 replies; 6+ messages in thread
From: Christian Ehrhardt @ 2023-06-29 13:02 UTC (permalink / raw)
To: dev; +Cc: Luca Boccassi
On Thu, Jun 29, 2023 at 2:58 PM <christian.ehrhardt@canonical.com> wrote:
>
> From: Christian Ehrhardt <christian.ehrhardt@canonical.com>
>
> By adding -j we build in parallel, to make building on multiprocessor
> machines more effective. While that works it does also break
> reproducible builds as the order of the sphinx generated searchindex.js
> is depending on execution speed of the individual processes.
Just FYI (this didn't fit fit well in the commit message) an example
of such a fail can be seen at
https://salsa.debian.org/paelzer-guest/dpdk/-/jobs/4372883
If you download the artifact, extract dpdk-doc, apply js-beautify for
readability and then diff it you'll find it
same-content-different-order.
Examples of two builds:
- https://paste.ubuntu.com/p/VhWYNRv7kN/
- https://paste.ubuntu.com/p/KcQk4Km9xM/
> Signed-off-by: Christian Ehrhardt <christian.ehrhardt@canonical.com>
> ---
> buildtools/call-sphinx-build.py | 5 -----
> 1 file changed, 5 deletions(-)
>
> diff --git a/buildtools/call-sphinx-build.py b/buildtools/call-sphinx-build.py
> index 39a60d09fa..d8879306de 100755
> --- a/buildtools/call-sphinx-build.py
> +++ b/buildtools/call-sphinx-build.py
> @@ -15,12 +15,7 @@
> # set the version in environment for sphinx to pick up
> os.environ['DPDK_VERSION'] = version
>
> -# for sphinx version >= 1.7 add parallelism using "-j auto"
> -ver = run([sphinx, '--version'], stdout=PIPE,
> - stderr=STDOUT).stdout.decode().split()[-1]
> sphinx_cmd = [sphinx] + extra_args
> -if Version(ver) >= Version('1.7'):
> - sphinx_cmd += ['-j', 'auto']
>
> # find all the files sphinx will process so we can write them as dependencies
> srcfiles = []
> --
> 2.41.0
>
--
Christian Ehrhardt
Senior Staff Engineer and acting Director, Ubuntu Server
Canonical Ltd
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] doc: ensure sphinx output is reproducible
2023-06-29 12:58 [PATCH] doc: ensure sphinx output is reproducible christian.ehrhardt
2023-06-29 13:02 ` Christian Ehrhardt
@ 2023-07-03 15:29 ` Thomas Monjalon
2023-07-06 12:49 ` Christian Ehrhardt
1 sibling, 1 reply; 6+ messages in thread
From: Thomas Monjalon @ 2023-07-03 15:29 UTC (permalink / raw)
To: Christian Ehrhardt; +Cc: dev, Luca Boccassi, david.marchand
29/06/2023 14:58, christian.ehrhardt@canonical.com:
> From: Christian Ehrhardt <christian.ehrhardt@canonical.com>
>
> By adding -j we build in parallel, to make building on multiprocessor
> machines more effective. While that works it does also break
> reproducible builds as the order of the sphinx generated searchindex.js
> is depending on execution speed of the individual processes.
[...]
> -if Version(ver) >= Version('1.7'):
> - sphinx_cmd += ['-j', 'auto']
What is the impact on build speed on an average machine?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] doc: ensure sphinx output is reproducible
2023-07-03 15:29 ` Thomas Monjalon
@ 2023-07-06 12:49 ` Christian Ehrhardt
2023-11-27 16:45 ` Thomas Monjalon
0 siblings, 1 reply; 6+ messages in thread
From: Christian Ehrhardt @ 2023-07-06 12:49 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: dev, Luca Boccassi, david.marchand
On Mon, Jul 3, 2023 at 5:29 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 29/06/2023 14:58, christian.ehrhardt@canonical.com:
> > From: Christian Ehrhardt <christian.ehrhardt@canonical.com>
> >
> > By adding -j we build in parallel, to make building on multiprocessor
> > machines more effective. While that works it does also break
> > reproducible builds as the order of the sphinx generated searchindex.js
> > is depending on execution speed of the individual processes.
> [...]
> > -if Version(ver) >= Version('1.7'):
> > - sphinx_cmd += ['-j', 'auto']
>
> What is the impact on build speed on an average machine?
Hi,
I haven't tested this in isolation as it was just a mandatory change
on the Debian/Ubuntu side.
And the time for exactly and only the doc build is hidden inside the
concurrency of meson.
But I can compare a full build [1] and a full build with the change [2].
That is an average build machine and it is 35 seconds slower with the
change to no more do doc builds in parallel.
[1]: https://launchpadlibrarian.net/673520160/buildlog_ubuntu-mantic-amd64.dpdk_22.11.2-2_BUILDING.txt.gz
[2]: https://launchpadlibrarian.net/674783718/buildlog_ubuntu-mantic-amd64.dpdk_22.11.2-3_BUILDING.txt.gz
--
Christian Ehrhardt
Senior Staff Engineer and acting Director, Ubuntu Server
Canonical Ltd
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] doc: ensure sphinx output is reproducible
2023-07-06 12:49 ` Christian Ehrhardt
@ 2023-11-27 16:45 ` Thomas Monjalon
2023-11-27 17:00 ` Bruce Richardson
0 siblings, 1 reply; 6+ messages in thread
From: Thomas Monjalon @ 2023-11-27 16:45 UTC (permalink / raw)
To: Luca Boccassi, Christian Ehrhardt; +Cc: dev, david.marchand
06/07/2023 14:49, Christian Ehrhardt:
> On Mon, Jul 3, 2023 at 5:29 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > 29/06/2023 14:58, christian.ehrhardt@canonical.com:
> > > From: Christian Ehrhardt <christian.ehrhardt@canonical.com>
> > >
> > > By adding -j we build in parallel, to make building on multiprocessor
> > > machines more effective. While that works it does also break
> > > reproducible builds as the order of the sphinx generated searchindex.js
> > > is depending on execution speed of the individual processes.
> > [...]
> > > -if Version(ver) >= Version('1.7'):
> > > - sphinx_cmd += ['-j', 'auto']
> >
> > What is the impact on build speed on an average machine?
>
> Hi,
> I haven't tested this in isolation as it was just a mandatory change
> on the Debian/Ubuntu side.
> And the time for exactly and only the doc build is hidden inside the
> concurrency of meson.
> But I can compare a full build [1] and a full build with the change [2].
>
> That is an average build machine and it is 35 seconds slower with the
> change to no more do doc builds in parallel.
I would prefer adding an option for reproducible build
(which is not a common requirement).
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] doc: ensure sphinx output is reproducible
2023-11-27 16:45 ` Thomas Monjalon
@ 2023-11-27 17:00 ` Bruce Richardson
0 siblings, 0 replies; 6+ messages in thread
From: Bruce Richardson @ 2023-11-27 17:00 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: Luca Boccassi, Christian Ehrhardt, dev, david.marchand
On Mon, Nov 27, 2023 at 05:45:52PM +0100, Thomas Monjalon wrote:
> 06/07/2023 14:49, Christian Ehrhardt:
> > On Mon, Jul 3, 2023 at 5:29 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > >
> > > 29/06/2023 14:58, christian.ehrhardt@canonical.com:
> > > > From: Christian Ehrhardt <christian.ehrhardt@canonical.com>
> > > >
> > > > By adding -j we build in parallel, to make building on multiprocessor
> > > > machines more effective. While that works it does also break
> > > > reproducible builds as the order of the sphinx generated searchindex.js
> > > > is depending on execution speed of the individual processes.
> > > [...]
> > > > -if Version(ver) >= Version('1.7'):
> > > > - sphinx_cmd += ['-j', 'auto']
> > >
> > > What is the impact on build speed on an average machine?
> >
> > Hi,
> > I haven't tested this in isolation as it was just a mandatory change
> > on the Debian/Ubuntu side.
> > And the time for exactly and only the doc build is hidden inside the
> > concurrency of meson.
> > But I can compare a full build [1] and a full build with the change [2].
> >
> > That is an average build machine and it is 35 seconds slower with the
> > change to no more do doc builds in parallel.
>
> I would prefer adding an option for reproducible build
> (which is not a common requirement).
>
Taking a slightly different tack, is it possible to sort the searchindex.js
file post-build, so that even reproducible builds get the benefits of
parallelism?
/Bruce
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-11-27 17:01 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-06-29 12:58 [PATCH] doc: ensure sphinx output is reproducible christian.ehrhardt
2023-06-29 13:02 ` Christian Ehrhardt
2023-07-03 15:29 ` Thomas Monjalon
2023-07-06 12:49 ` Christian Ehrhardt
2023-11-27 16:45 ` Thomas Monjalon
2023-11-27 17:00 ` Bruce Richardson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).