From: Ferruh Yigit <ferruh.yigit@intel.com>
To: Thomas Monjalon <thomas.monjalon@6wind.com>
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] [PATCH] mk: parallelize make config
Date: Mon, 30 Jan 2017 09:46:53 +0000 [thread overview]
Message-ID: <dda8f4b4-e7a2-5b45-3d9a-1c0d2d40eddd@intel.com> (raw)
In-Reply-To: <1800797.YCEvINBUOG@xps13>
On 1/29/2017 3:29 PM, Thomas Monjalon wrote:
> 2017-01-22 01:50, Ferruh Yigit:
>> make config dependency resolving was always running serial,
>> parallelize it for better performance.
>
> It could be interesting to explain why it was not parallelized,
> and how you made it possible.
I will try to explain in next version.
>
> The test script should be updated as below:
>
> --- a/devtools/test-build.sh
> +++ b/devtools/test-build.sh
> - make T=$2 O=$1 config
> + make -j$J T=$2 O=$1 config
I will update.
>
>
>> --- a/mk/internal/rte.depdirs-post.mk
>> +++ b/mk/internal/rte.depdirs-post.mk
>> @@ -29,11 +29,12 @@
>> # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>> # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>>
>> -.PHONY: depdirs
>> -depdirs:
>> - @for d in $(DEPDIRS-y); do \
>> - $(RTE_SDK)/buildtools/depdirs-rule.sh $(S) $$d ; \
>> - done
>> +.PHONY: depdirs $(DEPDIRS-y)
>> +depdirs: $(DEPDIRS-y)
>> + @echo ""
>
> Why this echo "" ?
If there is no dependency (DEPDIRS-y) set and there is nothing to do
here, make prints a line like "nothing to do", since we redirect the
output to the files, that msg goes into .depdir files. To prevent that
msg, added a dummy action here.
>
>> +
>> +$(DEPDIRS-y):
>> + @$(RTE_SDK)/buildtools/depdirs-rule.sh $(S) $@
>
>
>> --- a/mk/rte.sdkdepdirs.mk
>> +++ b/mk/rte.sdkdepdirs.mk
>> @@ -36,19 +36,22 @@ ifeq (,$(wildcard $(RTE_OUTPUT)/Makefile))
>> $(error "need a make config first")
>> endif
>>
>> +DEPDIRS = $(addsuffix /.depdirs, $(addprefix $(BUILDDIR)/,$(ROOTDIRS-y)))
>
> These DEPDIRS are files, although DEPDIRS in other contexts are directories.
> I think it should be renamed. DEPDIR_FILES?
Make sense, I will update.
>
>> # use a "for" in a shell to process dependencies: we don't want this
>> # task to be run in parallel.
>
> You forgot to remove this obsolete comment.
Will update on next version.
>
>> .PHONY: depdirs
>> depdirs: $(RTE_OUTPUT)/.depdirs
>> -$(RTE_OUTPUT)/.depdirs: $(RTE_OUTPUT)/.config
>> - @rm -f $(RTE_OUTPUT)/.depdirs ; \
>> - for d in $(ROOTDIRS-y); do \
>> - if [ -f $(RTE_SRCDIR)/$$d/Makefile ]; then \
>> - [ -d $(BUILDDIR)/$$d ] || mkdir -p $(BUILDDIR)/$$d ; \
>> - $(MAKE) S=$$d -f $(RTE_SRCDIR)/$$d/Makefile depdirs \
>> - >> $(RTE_OUTPUT)/.depdirs ; \
>> - fi ; \
>> - done
>> +$(RTE_OUTPUT)/.depdirs: $(DEPDIRS)
>> + @rm -f $@
>> + @for f in $(DEPDIRS); do cat $$f >> $@; done
>> + @sort -u -o $@ $@
>> +
>> +$(DEPDIRS): $(RTE_OUTPUT)/.config
>> + @f=$(lastword $(subst /, ,$(dir $@))); \
>
> Could you use $(notdir $(@D)) ?
Can be, I will try.
>
>> + [ -d $(BUILDDIR)/$$f ] || mkdir -p $(BUILDDIR)/$$f; \
>> + rm -f $@; \
>
> Why this removal?
To be sure we are not using old files, but this can be done below line
with using ">" I guess.
>
>> + $(MAKE) S=$$f -f $(RTE_SRCDIR)/$$f/Makefile depdirs >> $@
>
> This part is a bit complicated.
> Could it be simplified by better naming $f?
I will try.
>
>
>> --- a/mk/rte.subdir.mk
>> +++ b/mk/rte.subdir.mk
>> @@ -76,7 +76,7 @@ clean: _postclean
>> # include .depdirs and define rules to order priorities between build
>> # of directories.
>> #
>> -include $(RTE_OUTPUT)/.depdirs
>> +-include $(RTE_OUTPUT)/.depdirs
>>
>> define depdirs_rule
>> $(1): $(sort $(patsubst $(S)/%,%,$(LOCAL_DEPDIRS-$(S)/$(1))))
>> @@ -84,16 +84,15 @@ endef
>>
>> $(foreach d,$(DIRS-y),$(eval $(call depdirs_rule,$(d))))
>>
>> +DEPDIRS = $(wildcard $(addprefix $(S)/,$(DIRS-y)))
>>
>> # use a "for" in a shell to process dependencies: we don't want this
>> # task to be run in parallel.
>
> You forgot to remove this obsolete comment.
Right, will update.
>
>> -.PHONY: depdirs
>> -depdirs:
>> - @for d in $(DIRS-y); do \
>> - if [ -f $(SRCDIR)/$$d/Makefile ]; then \
>> - $(MAKE) S=$S/$$d -f $(SRCDIR)/$$d/Makefile depdirs ; \
>> - fi ; \
>> - done
>> +.PHONY: depdirs $(DEPDIRS)
>> +depdirs: $(DEPDIRS)
>> +
>> +$(DEPDIRS):
>> + @$(MAKE) S=$@ -f $(RTE_SRCDIR)/$@/Makefile depdirs
>
next prev parent reply other threads:[~2017-01-30 9:46 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-01-22 1:50 Ferruh Yigit
2017-01-23 17:18 ` Olivier Matz
2017-01-23 17:19 ` [dpdk-dev] [PATCH] mk: optimize directory dependencies Olivier Matz
2017-01-24 11:19 ` Robin Jarry
2017-01-24 11:26 ` Bruce Richardson
2017-01-24 12:31 ` Robin Jarry
2017-01-24 11:40 ` Jerin Jacob
2017-01-24 12:15 ` Bruce Richardson
2017-01-24 12:56 ` Jerin Jacob
2017-01-24 13:26 ` Richardson, Bruce
2017-01-24 14:50 ` Olivier MATZ
2017-01-24 14:55 ` Wiles, Keith
2017-03-01 11:25 ` Thomas Monjalon
2017-03-01 12:10 ` Bruce Richardson
2017-03-01 12:30 ` Olivier Matz
2017-01-24 13:05 ` Ferruh Yigit
2017-03-17 17:13 ` Olivier Matz
2017-03-17 17:47 ` Robin Jarry
2017-03-20 8:31 ` Olivier Matz
2017-03-24 13:21 ` [dpdk-dev] [PATCH v2] " Olivier Matz
2017-03-27 21:33 ` Thomas Monjalon
2017-03-28 10:34 ` Ferruh Yigit
2017-03-30 8:51 ` Olivier Matz
2017-03-30 9:27 ` Ferruh Yigit
2017-03-30 12:11 ` Olivier Matz
2017-03-30 12:32 ` [dpdk-dev] [PATCH] mk: fix dependencies to optional configs Olivier Matz
2017-03-30 12:37 ` Ferruh Yigit
2017-03-30 13:37 ` Thomas Monjalon
2017-01-23 17:50 ` [dpdk-dev] [PATCH] mk: parallelize make config Wiles, Keith
2017-01-24 8:42 ` Olivier MATZ
2017-01-24 10:02 ` Bruce Richardson
2017-01-23 19:03 ` Michał Mirosław
2017-01-30 9:41 ` Ferruh Yigit
2017-01-24 10:52 ` Bruce Richardson
2017-01-29 15:29 ` Thomas Monjalon
2017-01-30 9:46 ` Ferruh Yigit [this message]
2017-01-30 10:21 ` [dpdk-dev] [PATCH v2] " Ferruh Yigit
2017-01-30 18:13 ` Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=dda8f4b4-e7a2-5b45-3d9a-1c0d2d40eddd@intel.com \
--to=ferruh.yigit@intel.com \
--cc=dev@dpdk.org \
--cc=thomas.monjalon@6wind.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).