From: "Richardson, Bruce" <bruce.richardson@intel.com>
To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
Cc: Olivier Matz <olivier.matz@6wind.com>,
"dev@dpdk.org" <dev@dpdk.org>,
"Yigit, Ferruh" <ferruh.yigit@intel.com>,
"thomas.monjalon@6wind.com" <thomas.monjalon@6wind.com>
Subject: Re: [dpdk-dev] [PATCH] mk: optimize directory dependencies
Date: Tue, 24 Jan 2017 13:26:41 +0000 [thread overview]
Message-ID: <59AF69C657FD0841A61C55336867B5B035B8DFDE@IRSMSX103.ger.corp.intel.com> (raw)
In-Reply-To: <20170124125652.GA7561@localhost.localdomain>
> -----Original Message-----
> From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> Sent: Tuesday, January 24, 2017 12:57 PM
> To: Richardson, Bruce <bruce.richardson@intel.com>
> Cc: Olivier Matz <olivier.matz@6wind.com>; dev@dpdk.org; Yigit, Ferruh
> <ferruh.yigit@intel.com>; thomas.monjalon@6wind.com
> Subject: Re: [dpdk-dev] [PATCH] mk: optimize directory dependencies
>
> On Tue, Jan 24, 2017 at 12:15:06PM +0000, Bruce Richardson wrote:
> > On Tue, Jan 24, 2017 at 05:10:15PM +0530, Jerin Jacob wrote:
> > > On Mon, Jan 23, 2017 at 06:19:13PM +0100, Olivier Matz wrote:
> > > > Before this patch, the management of dependencies between
> > > > directories had several issues:
> > > >
> > > > - the generation of .depdirs, done at configuration is slow: it can
> take
> > > > more than one minute on some slow targets (usually ~10s on a
> standard
> > > > PC).
> > > >
> > > > - for instance, it is possible to expressed a dependency like:
> > > > - app/foo depends on lib/librte_foo
> > > > - and lib/librte_foo depends on app/bar
> > > > But this won't work because the directories are traversed with a
> > > > depth-first algorithm, so we have to choose between doing 'app'
> before
> > > > or after 'lib'.
> > > >
> > > > - the script depdirs-rule.sh is too complex.
> > > >
> > > > - we cannot use "make -d" for debug, because the output of make is
> used for
> > > > the generation of .depdirs.
> > > >
> > > > This patch moves the DEPDIRS-* variables in the upper Makefile,
> > > > making the dependencies much easier to calculate. A DEPDIRS
> > > > variable is still used to process library dependencies in LDLIBS.
> > > >
> > > > After this commit, "make config" is almost immediate.
> > > >
> > > > Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
> > >
> > > Tested both approach on ThunderX. This patch looks better
> > >
> > > Tested-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > >
> > > ➜ [master]GB-2S [dpdk-master] $ time make config
> > > T=arm64-thunderx-linuxapp-gcc Configuration done
> > >
> > > real 0m18.112s
> > > user 0m2.810s
> > > sys 0m0.660s
> > >
> > > ➜ [master]GB-2S [dpdk-master] $ pwclient git-am 19859 Applying patch
> > > #19859 using 'git am'
> > > Description: [dpdk-dev] mk: parallelize make config
> > > Applying: mk: parallelize make config
> > >
> > > ➜ [master]GB-2S [dpdk-master] $ rm -rf build ➜ [master]GB-2S
> > > [dpdk-master] $ time make config T=arm64-thunderx-linuxapp-gcc -j 8
> > > Configuration done
> > >
> > > real 0m2.812s
> > > user 0m3.020s
> > > sys 0m0.870s
> > >
> > > ➜ [master]GB-2S [dpdk-master] $ rm -rf build ➜ [master]GB-2S
> > > [dpdk-master] $ time make config T=arm64-thunderx-linuxapp-gcc -j 16
> > > Configuration done
> > >
> > > real 0m1.748s
> > > user 0m3.040s
> > > sys 0m1.020s
> > >
> > > ➜ [master]GB-2S [dpdk-master] $ rm -rf build ➜ [master]GB-2S
> > > [dpdk-master] $ time make config T=arm64-thunderx-linuxapp-gcc -j 32
> > > Configuration done
> > >
> > > real 0m1.422s
> > > user 0m3.380s
> > > sys 0m1.080s
> > >
> > > ➜ [master]GB-2S [dpdk-master] $ pwclient git-am 19918 Applying patch
> > > #19918 using 'git am'
> > > Description: [dpdk-dev] mk: optimize directory dependencies
> > > Applying: mk: optimize directory dependencies
> > >
> > > ➜ [master]GB-2S [dpdk-master] $ rm -rf build ➜ [master]GB-2S
> > > [dpdk-master] $ time make config T=arm64-thunderx-linuxapp-gcc
> > > Configuration done
> > >
> > > real 0m0.064s
> > > user 0m0.000s
> > > sys 0m0.000s
> > > ➜ [master]GB-2S [dpdk-master] $ rm -rf build ➜ [master]GB-2S
> > > [dpdk-master] $ time make config T=arm64-thunderx-linuxapp-gcc -j 8
> > > Configuration done
> > >
> > > real 0m0.055s
> > > user 0m0.000s
> > > sys 0m0.000s
> > >
> >
> > I agree that Olivier's patch is faster. However, I think I prefer
> > having the library dependencies in the makefiles for the libs
> > themselves rather than up a level. Given that we are only looking at
> > ~2second of a difference here in your tests - assuming -j flag - what
> > is the actual build time differences? My suspicion is that after
> > Ferruh's simpler
>
> Without patch - 18sec
> With patch -j1 - 18 sec
> With patch -j2 - 9.2 sec
> With patch -j4 - 4.9 sec
> With patch -j8 - 2.8 sec
> With patch -j16 - 1.7 sec
> With patch -j32 - 1.4 sec
>
> > patch is applied, the config time becomes such a small part of the
> > build, that the extra benefits from Oliviers work is not worth the
> > extra complexity.
>
> The low-end embedded SoCs (SoC with 2 to 4 cores) will be get benefited
> out of bring this extra complexity.My take is, if it is manageable
> complexity then take the most optimized one.
Ok. Point taken for the lower-core count parts.
Thomas - can at least one of these patches be merged into 17.02, since it will definitely help us developers? [If Olivier's is too big a change at this point in the cycle, can Ferruh's be taken as an intermediate fix for this release before merging Olivier's in 17.05?]
Regards,
/Bruce
next prev parent reply other threads:[~2017-01-24 13:26 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-01-22 1:50 [dpdk-dev] [PATCH] mk: parallelize make config Ferruh Yigit
2017-01-23 17:18 ` Olivier Matz
2017-01-23 17:19 ` [dpdk-dev] [PATCH] mk: optimize directory dependencies Olivier Matz
2017-01-24 11:19 ` Robin Jarry
2017-01-24 11:26 ` Bruce Richardson
2017-01-24 12:31 ` Robin Jarry
2017-01-24 11:40 ` Jerin Jacob
2017-01-24 12:15 ` Bruce Richardson
2017-01-24 12:56 ` Jerin Jacob
2017-01-24 13:26 ` Richardson, Bruce [this message]
2017-01-24 14:50 ` Olivier MATZ
2017-01-24 14:55 ` Wiles, Keith
2017-03-01 11:25 ` Thomas Monjalon
2017-03-01 12:10 ` Bruce Richardson
2017-03-01 12:30 ` Olivier Matz
2017-01-24 13:05 ` Ferruh Yigit
2017-03-17 17:13 ` Olivier Matz
2017-03-17 17:47 ` Robin Jarry
2017-03-20 8:31 ` Olivier Matz
2017-03-24 13:21 ` [dpdk-dev] [PATCH v2] " Olivier Matz
2017-03-27 21:33 ` Thomas Monjalon
2017-03-28 10:34 ` Ferruh Yigit
2017-03-30 8:51 ` Olivier Matz
2017-03-30 9:27 ` Ferruh Yigit
2017-03-30 12:11 ` Olivier Matz
2017-03-30 12:32 ` [dpdk-dev] [PATCH] mk: fix dependencies to optional configs Olivier Matz
2017-03-30 12:37 ` Ferruh Yigit
2017-03-30 13:37 ` Thomas Monjalon
2017-01-23 17:50 ` [dpdk-dev] [PATCH] mk: parallelize make config Wiles, Keith
2017-01-24 8:42 ` Olivier MATZ
2017-01-24 10:02 ` Bruce Richardson
2017-01-23 19:03 ` Michał Mirosław
2017-01-30 9:41 ` Ferruh Yigit
2017-01-24 10:52 ` Bruce Richardson
2017-01-29 15:29 ` Thomas Monjalon
2017-01-30 9:46 ` Ferruh Yigit
2017-01-30 10:21 ` [dpdk-dev] [PATCH v2] " Ferruh Yigit
2017-01-30 18:13 ` Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=59AF69C657FD0841A61C55336867B5B035B8DFDE@IRSMSX103.ger.corp.intel.com \
--to=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=ferruh.yigit@intel.com \
--cc=jerin.jacob@caviumnetworks.com \
--cc=olivier.matz@6wind.com \
--cc=thomas.monjalon@6wind.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).