From: Neil Horman <nhorman@tuxdriver.com>
To: "Wiles, Keith" <keith.wiles@intel.com>
Cc: Neil Horman <nhorman@redhat.com>,
Thomas Monjalon <thomas.monjalon@6wind.com>,
"dev@dpdk.org" <dev@dpdk.org>,
"Mcnamara, John" <john.mcnamara@intel.com>
Subject: Re: [dpdk-dev] [PATCH] validate_abi: build faster by augmenting make with job count
Date: Wed, 20 Jul 2016 16:16:17 -0400 [thread overview]
Message-ID: <20160720201617.GA23515@hmsreliant.think-freely.org> (raw)
In-Reply-To: <CDB4778C-5EF7-4728-A77D-229F76A13225@intel.com>
On Wed, Jul 20, 2016 at 07:47:32PM +0000, Wiles, Keith wrote:
>
> > On Jul 20, 2016, at 12:48 PM, Neil Horman <nhorman@redhat.com> wrote:
> >
> > On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote:
> >> 2016-07-20 13:09, Neil Horman:
> >>> From: Neil Horman <nhorman@redhat.com>
> >>>
> >>> John Mcnamara and I were discussing enhacing the validate_abi script to build
> >>> the dpdk tree faster with multiple jobs. Theres no reason not to do it, so this
> >>> implements that requirement. It uses a MAKE_JOBS variable that can be set by
> >>> the user to limit the job count. By default the job count is set to the number
> >>> of online cpus.
> >>
> >> Please could you use the variable name DPDK_MAKE_JOBS?
> >> This name is already used in scripts/test-build.sh.
> >>
> > Sure
> >
> >>> +if [ -z "$MAKE_JOBS" ]
> >>> +then
> >>> + # This counts the number of cpus on the system
> >>> + MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
> >>> +fi
> >>
> >> Is lscpu common enough?
> >>
> > I'm not sure how to answer that. lscpu is part of the util-linux package, which
> > is part of any base install. Theres a variant for BSD, but I'm not sure how
> > common it is there.
> > Neil
> >
> >> Another acceptable default would be just "-j" without any number.
> >> It would make the number of jobs unlimited.
>
> I think the best is just use -j as it tries to use the correct number of jobs based on the number of cores, right?
>
-j with no argument (or -j 0), is sort of, maybe what you want. With either of
those options, make will just issue jobs as fast as it processes dependencies.
Dependent on how parallel the build is, that can lead to tons of waiting process
(i.e. more than your number of online cpus), which can actually hurt your build
time.
While its fine in los of cases, its not always fine, and with this
implementation you can still opt in to that behavior by setting DPDK_MAKE_JOBS=0
Neil
>
next prev parent reply other threads:[~2016-07-20 20:16 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-07-20 17:09 Neil Horman
2016-07-20 17:40 ` Thomas Monjalon
2016-07-20 17:48 ` Neil Horman
2016-07-20 19:47 ` Wiles, Keith
2016-07-20 20:15 ` Thomas Monjalon
2016-07-20 20:16 ` Neil Horman [this message]
2016-07-20 22:32 ` Wiles, Keith
2016-07-21 13:54 ` Neil Horman
2016-07-21 14:09 ` Wiles, Keith
2016-07-21 15:06 ` Neil Horman
2016-07-21 15:22 ` Wiles, Keith
2016-07-21 18:34 ` Neil Horman
2016-07-24 18:08 ` Wiles, Keith
2016-08-01 11:49 ` Neil Horman
2016-08-01 16:16 ` Wiles, Keith
2016-08-01 18:08 ` Neil Horman
2016-07-20 19:02 ` [dpdk-dev] [PATCH v2] " Neil Horman
2016-07-22 10:46 ` Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160720201617.GA23515@hmsreliant.think-freely.org \
--to=nhorman@tuxdriver.com \
--cc=dev@dpdk.org \
--cc=john.mcnamara@intel.com \
--cc=keith.wiles@intel.com \
--cc=nhorman@redhat.com \
--cc=thomas.monjalon@6wind.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).