From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <nhorman@redhat.com>
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28])
 by dpdk.org (Postfix) with ESMTP id 76C3E5908
 for <dev@dpdk.org>; Thu, 21 Jul 2016 20:34:05 +0200 (CEST)
Received: from int-mx11.intmail.prod.int.phx2.redhat.com
 (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (No client certificate requested)
 by mx1.redhat.com (Postfix) with ESMTPS id 6B0533E2CC;
 Thu, 21 Jul 2016 18:34:04 +0000 (UTC)
Received: from hmsreliant.think-freely.org (unused [10.10.51.23] (may be
 forged))
 by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id
 u6LIY1ft001421
 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
 Thu, 21 Jul 2016 14:34:03 -0400
Date: Thu, 21 Jul 2016 14:34:01 -0400
From: Neil Horman <nhorman@redhat.com>
To: "Wiles, Keith" <keith.wiles@intel.com>
Cc: Neil Horman <nhorman@tuxdriver.com>,
 Thomas Monjalon <thomas.monjalon@6wind.com>, "dev@dpdk.org" <dev@dpdk.org>,
 "Mcnamara, John" <john.mcnamara@intel.com>
Message-ID: <20160721183401.GE10032@hmsreliant.think-freely.org>
References: <1469034588-1847-1-git-send-email-nhorman@tuxdriver.com>
 <6330653.K6A9CLrOZy@xps13>
 <20160720174849.GE28844@hmsreliant.think-freely.org>
 <CDB4778C-5EF7-4728-A77D-229F76A13225@intel.com>
 <20160720201617.GA23515@hmsreliant.think-freely.org>
 <9309C1B3-D86C-4265-97A4-58E3D7958944@intel.com>
 <20160721135433.GA9628@hmsreliant.think-freely.org>
 <9B6D664A-6C02-4982-BBA1-628C94514164@intel.com>
 <20160721150604.GC10032@hmsreliant.think-freely.org>
 <64F83F10-008B-4DB0-9662-989A8F57BE80@intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <64F83F10-008B-4DB0-9662-989A8F57BE80@intel.com>
User-Agent: Mutt/1.6.2 (2016-07-01)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16
 (mx1.redhat.com [10.5.110.30]); Thu, 21 Jul 2016 18:34:04 +0000 (UTC)
Subject: Re: [dpdk-dev] [PATCH] validate_abi: build faster by augmenting
 make with job count
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Thu, 21 Jul 2016 18:34:05 -0000

On Thu, Jul 21, 2016 at 03:22:45PM +0000, Wiles, Keith wrote:
> 
> > On Jul 21, 2016, at 10:06 AM, Neil Horman <nhorman@redhat.com> wrote:
> > 
> > On Thu, Jul 21, 2016 at 02:09:19PM +0000, Wiles, Keith wrote:
> >> 
> >>> On Jul 21, 2016, at 8:54 AM, Neil Horman <nhorman@tuxdriver.com> wrote:
> >>> 
> >>> On Wed, Jul 20, 2016 at 10:32:28PM +0000, Wiles, Keith wrote:
> >>>> 
> >>>>> On Jul 20, 2016, at 3:16 PM, Neil Horman <nhorman@tuxdriver.com> wrote:
> >>>>> 
> >>>>> On Wed, Jul 20, 2016 at 07:47:32PM +0000, Wiles, Keith wrote:
> >>>>>> 
> >>>>>>> On Jul 20, 2016, at 12:48 PM, Neil Horman <nhorman@redhat.com> wrote:
> >>>>>>> 
> >>>>>>> On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote:
> >>>>>>>> 2016-07-20 13:09, Neil Horman:
> >>>>>>>>> From: Neil Horman <nhorman@redhat.com>
> >>>>>>>>> 
> >>>>>>>>> John Mcnamara and I were discussing enhacing the validate_abi script to build
> >>>>>>>>> the dpdk tree faster with multiple jobs.  Theres no reason not to do it, so this
> >>>>>>>>> implements that requirement.  It uses a MAKE_JOBS variable that can be set by
> >>>>>>>>> the user to limit the job count.  By default the job count is set to the number
> >>>>>>>>> of online cpus.
> >>>>>>>> 
> >>>>>>>> Please could you use the variable name DPDK_MAKE_JOBS?
> >>>>>>>> This name is already used in scripts/test-build.sh.
> >>>>>>>> 
> >>>>>>> Sure
> >>>>>>> 
> >>>>>>>>> +if [ -z "$MAKE_JOBS" ]
> >>>>>>>>> +then
> >>>>>>>>> +	# This counts the number of cpus on the system
> >>>>>>>>> +	MAKE_JOBS=`lscpu -p=cpu | grep -v "#" | wc -l`
> >>>>>>>>> +fi
> >>>>>>>> 
> >>>>>>>> Is lscpu common enough?
> >>>>>>>> 
> >>>>>>> I'm not sure how to answer that.  lscpu is part of the util-linux package, which
> >>>>>>> is part of any base install.  Theres a variant for BSD, but I'm not sure how
> >>>>>>> common it is there.
> >>>>>>> Neil
> >>>>>>> 
> >>>>>>>> Another acceptable default would be just "-j" without any number.
> >>>>>>>> It would make the number of jobs unlimited.
> >>>>>> 
> >>>>>> I think the best is just use -j as it tries to use the correct number of jobs based on the number of cores, right?
> >>>>>> 
> >>>>> -j with no argument (or -j 0), is sort of, maybe what you want.  With either of
> >>>>> those options, make will just issue jobs as fast as it processes dependencies.
> >>>>> Dependent on how parallel the build is, that can lead to tons of waiting process
> >>>>> (i.e. more than your number of online cpus), which can actually hurt your build
> >>>>> time.
> >>>> 
> >>>> I read the manual and looked at the code, which supports your statement. (I think I had some statement on stack overflow and the last time I believe anything on the internet :-) I have not seen a lot of differences in compile times with -j on my system. Mostly I suspect it is the number of paths in the dependency, cores and memory on the system.
> >>>> 
> >>>> I have 72 lcores or 2 sockets, 18 cores per socket. Xeon 2.3Ghz cores.
> >>>> 
> >>>> $ export RTE_TARGET=x86_64-native-linuxapp-gcc 
> >>>> 
> >>>> $ time make install T=${RTE_TARGET}
> >>>> real	0m59.445s user	0m27.344s sys	0m7.040s
> >>>> 
> >>>> $ time make install T=${RTE_TARGET} -j
> >>>> real	0m26.584s user	0m14.380s sys	0m5.120s
> >>>> 
> >>>> # Remove the x86_64-native-linuxapp-gcc
> >>>> 
> >>>> $ time make install T=${RTE_TARGET} -j 72
> >>>> real	0m23.454s user	0m10.832s sys	0m4.664s
> >>>> 
> >>>> $ time make install T=${RTE_TARGET} -j 8
> >>>> real	0m23.812s user	0m10.672s sys	0m4.276s
> >>>> 
> >>>> cd x86_64-native-linuxapp-gcc
> >>>> $ make clean
> >>>> $ time make
> >>>> real	0m28.539s user	0m9.820s sys	0m3.620s
> >>>> 
> >>>> # Do a make clean between each build.
> >>>> 
> >>>> $ time make -j
> >>>> real	0m7.217s user	0m6.532s sys	0m2.332s
> >>>> 
> >>>> $ time make -j 8
> >>>> real	0m8.256s user	0m6.472s sys	0m2.456s
> >>>> 
> >>>> $ time make -j 72
> >>>> real	0m6.866s user	0m6.184s sys	0m2.216s
> >>>> 
> >>>> Just the real time numbers in the following table.
> >>>> 
> >>>> processes     real Time   depdirs
> >>>>    no -j             59.4s        Yes
> >>>>      -j 8             23.8s        Yes
> >>>>     -j 72            23.5s        Yes
> >>>>       -j               26.5s        Yes
> >>>> 
> >>>>    no -j             28.5s         No
> >>>>      -j 8               8.2s         No
> >>>>     -j 72              6.8s         No
> >>>>       -j                 7.2s         No
> >>>> 
> >>>> Looks like the depdirs build time on my system:
> >>>> $ make clean -j
> >>>> $ rm .depdirs
> >>>> $ time make -j
> >>>> real	0m23.734s user	0m11.228s sys	0m4.844s
> >>>> 
> >>>> About 16 seconds, which is not a lot of savings. Now the difference from no -j to -j is a lot, but the difference between -j and -j <cpu_count> is not a huge saving. This leads me back to over engineering the problem when ‘-j’ would work just as well here.
> >>>> 
> >>>> Even on my MacBook Pro i7 system the difference is not that much 1m8s without depdirs build for -j in a VirtualBox with all 4 cores 8G RAM. Compared to 1m13s with -j 4 option.
> >>>> 
> >>>> I just wonder if it makes a lot of sense to use cpuinfo in this given case if it turns out to be -j works with the 80% rule?
> >>>> 
> >>> It may, but that seems to be reason to me to just set DPDK_MAKE_JOBS=0, and
> >>> you'll get that behavior
> >> 
> >> Just to be sure, ‘make -j 0’ is not a valid argument to the -j option. It looks like you have to do ‘-j’ or ‘-j N’ or no option where N != 0
> >> 
> >> I think we just use -j which gets us the 80% rule and the best performance without counting cores.
> >> 
> > Thats odd, specifying 0 works for me.  If it doesn't for you, specify $MAX_INT
> > or some other huge number would be comparable
> 
> rkwiles@supermicro (master):~/.../dpdk/x86_64-native-linuxapp-gcc$ make --version
> GNU Make 4.1
> Built for x86_64-pc-linux-gnu
> Copyright (C) 1988-2014 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> 
> rkwiles@supermicro (master):~/.../dpdk/x86_64-native-linuxapp-gcc$ make -j 0
> make: the '-j' option requires a positive integer argument
> 
> rkwiles@supermicro (master):~/.../dpdk/x86_64-native-linuxapp-gcc$ lsb_release -a
> No LSB modules are available.
> Distributor ID:	Ubuntu
> Description:	Ubuntu 16.04.1 LTS
> Release:	16.04
> Codename:	xenial
> 
I'm not saying your variant doesn't work, only that my copy of make does, but
its possible that I have some alternately patched version (I used to fix make
bugs way back when, so I may have an impure copy).  Regardless, my comment is
still valid, if you want to have unlimited jobs, you can just export
DPDK_MAKE_JOBS=<some very large number>

Neil

> > 
> > Neil
> > 
> >>> 
> >>> Neil
> >>> 
> >>>> On some other project with a lot more files like the FreeBSD or Linux distro, yes it would make a fair amount of real time difference.
> >>>> 
> >>>> Keith
> >>>> 
> >>>>> 
> >>>>> While its fine in los of cases, its not always fine, and with this
> >>>>> implementation you can still opt in to that behavior by setting DPDK_MAKE_JOBS=0
> >>>>> 
> >>>>> Neil
>