[dpdk-dev] [PATCH RFC] Update/Improve build system

[dpdk-dev] [PATCH RFC] Update/Improve build system

[Gonzalez Monroy, Sergio]

Hi,



I would like to propose an update on the way the libraries are being built on DPDK.



Motivation/Issues:

- No agreement on libraries to build (separated, combined, different grouping).

- People having issues building their applications cannot rely on the information given by ldd when using shared libraries because currently we are not building against dependent libraries.

- External library dependencies are introduce in multiple places (ie. when building librte_vhost, -lfuse is added into the lib CFLAGS, LDFLAGS and also need to update rte.app.mk).

- Do we want to keep support for building shared libraries using LD? Currently unused as both Linux and BSD enable linking using CC.



Proposal:

- Define external lib dependencies per module/lib/dir. ie. EXT_DEP_LIBS = -lfuse -lm

- Define internal lib dependencies per module/lib/dir. ie. DPDK_DEP_LIBS = -lrte_eal

- Remove compile config option CONFIG_RTE_BUILD_COMBINE_LIBS and always build all wanted libraries (see below about different libraries to build).

- Shared linking: we want to link apps and libs against dependant libraries. Basically add EXT_DEP_LIBS and DPDK_DEP_LIBS when linking libs and apps.

- Static linking: we want to link apps against dependant libraries.



>From my point of view, we can classify DPDK libraries into three categories (note that not all libraries are included and the following is just for illustration purposes):

CORE_LIBS: librte_kvargs, librte_mbuf, librte_malloc, librte_mempool, librte_ring, librte_eal, libethdev?

NON_CORE_LIBS: librte_vhost, librte_cfgfile, librte_cmdline, librte_sched, librte_lpm, librte_acl, ...

PMD_LIBS: librte_pmd_i40e, librte_pmd_e1000, librte_ixgbe, ...



IMHO CORE_LIBS are a set of libraries that have inter-dependencies and are always required to build an application. Such set of core libraries should be provided as a single library.

I can think of a few different ways to go about this:

A) Build just one combined library for static or shared libraries.

     Pros: it would reduce and simplify makefile rules, as well as make it easier for users to build their own apps against DPDK.

     Cons: limited flexibility when building custom apps with selected libraries.     Link apps with something like:

                --whole-archive -ldpdk --no-whole-archive --as-needed $(EXT_DEP_LIBS) --no-as-needed

B) Build CORE_LIBS (as a single lib, librte_core), NON_CORE_LIBS and PMD_LIBS:

     Pros: flexibility.

     Cons: more complex build process, as we would add dependencies for each NON_CORE_LIB and PMD_LIB.     Link apps with something like:

                --whole-archive $(PMD_LIBS) --no-whole-archive $(NON_CORE_LIBS) -lrte_core --as-needed $(EXT_DEP_LIBS) --no-as-needed

C) Similar to B but building a single library containing both CORE_LIBS and NON_CORE_LIBS.

     Link apps with something like:

                --whole-archive $(PMD_LIBS) --no-whole-archive -lrte_xxxx --as-needed $(EXT_DEP_LIBS) --no-as-needed

D) Build all libs in both A) and B) but linking DPDK provided apps only against combined lib for slightly simpler makefile rules.



I would say that D) is a good balance, although not being the simplest.



Thoughts?



Thanks,

Sergio

Re: [dpdk-dev] [PATCH RFC] Update/Improve build system

[Gonzalez Monroy, Sergio]

Hi,

Apologies for the previous email format, I seems like I misconfigured something on my email-client.
In case people find it annoying to read with extra line spaces, here is the RFC with fixed formatting:


I would like to propose an update on the way the libraries are being built on DPDK.

Motivation/Issues:
- No agreement on libraries to build (separated, combined, different grouping).
- People having issues building their applications cannot rely on the information given by ldd when using shared libraries because currently we are not building against dependent libraries.
- External library dependencies are introduce in multiple places (ie. when building librte_vhost, -lfuse is added into the lib CFLAGS, LDFLAGS and also need to update rte.app.mk).
- Do we want to keep support for building shared libraries using LD? Currently unused as both Linux and BSD enable linking using CC.

Proposal:
- Define external lib dependencies per module/lib/dir. ie. EXT_DEP_LIBS = -lfuse -lm
- Define internal lib dependencies per module/lib/dir. ie. DPDK_DEP_LIBS = -lrte_eal
- Remove compile config option CONFIG_RTE_BUILD_COMBINE_LIBS and always build all wanted libraries (see below about different libraries to build).
- Shared linking: we want to link apps and libs against dependant libraries. Basically add EXT_DEP_LIBS and DPDK_DEP_LIBS when linking libs and apps.
- Static linking: we want to link apps against dependant libraries.

>From my point of view, we can classify DPDK libraries into three categories (note that not all libraries are included and the following is just for illustration purposes):
CORE_LIBS: librte_kvargs, librte_mbuf, librte_malloc, librte_mempool, librte_ring, librte_eal, libethdev?
NON_CORE_LIBS: librte_vhost, librte_cfgfile, librte_cmdline, librte_sched, librte_lpm, librte_acl, ...
PMD_LIBS: librte_pmd_i40e, librte_pmd_e1000, librte_ixgbe, ...

IMHO CORE_LIBS are a set of libraries that have inter-dependencies and are always required to build an application. Such set of core libraries should be provided as a single library.
I can think of a few different ways to go about this:
A) Build just one combined library for static or shared libraries.
     Pros: it would reduce and simplify makefile rules, as well as make it easier for users to build their own apps against DPDK.
     Cons: limited flexibility when building custom apps with selected libraries.     Link apps with something like:
                --whole-archive -ldpdk --no-whole-archive --as-needed $(EXT_DEP_LIBS) --no-as-needed
B) Build CORE_LIBS (as a single lib, librte_core), NON_CORE_LIBS and PMD_LIBS:
     Pros: flexibility.
     Cons: more complex build process, as we would add dependencies for each NON_CORE_LIB and PMD_LIB.     Link apps with something like:
                --whole-archive $(PMD_LIBS) --no-whole-archive $(NON_CORE_LIBS) -lrte_core --as-needed $(EXT_DEP_LIBS) --no-as-needed
C) Similar to B but building a single library containing both CORE_LIBS and NON_CORE_LIBS.
     Link apps with something like:
                --whole-archive $(PMD_LIBS) --no-whole-archive -lrte_xxxx --as-needed $(EXT_DEP_LIBS) --no-as-needed
D) Build all libs in both A) and B) but linking DPDK provided apps only against combined lib for slightly simpler makefile rules.

I would say that D) is a good balance, although not being the simplest.

Thoughts?

Thanks,
Sergio

Re: [dpdk-dev] [PATCH RFC] Update/Improve build system

[Matthew Hall]

On Thu, Oct 30, 2014 at 09:18:23AM +0000, Gonzalez Monroy, Sergio wrote:
> I would say that D) is a good balance, although not being the simplest.

A, or D. Depending on things such as, "If you run the DPDK on Random Platform 
X," where X could be something like Power CPUs or other weird stuff, will all 
of the things needed for the Combined Lib 1) be compilable, 2) be able to load 
w/o errors.

For example, I could see probe ctor functions from various PMD's blowing up on 
unsupported hardware. Like how the rte_pmd_virtio had issues when I tried it 
on my VM system.

If we think we can make sure no platform specific stuff breaks when it ends up 
in Combined Lib A then A is probably the easiest for all.

Matthew.

Re: [dpdk-dev] [PATCH RFC] Update/Improve build system

[Gonzalez Monroy, Sergio]

> From: Matthew Hall [mailto:mhall@mhcomputing.net]
> Sent: Thursday, October 30, 2014 8:50 PM
> 
> On Thu, Oct 30, 2014 at 09:18:23AM +0000, Gonzalez Monroy, Sergio wrote:
> > I would say that D) is a good balance, although not being the simplest.
> 
> A, or D. Depending on things such as, "If you run the DPDK on Random
> Platform X," where X could be something like Power CPUs or other weird
> stuff, will all of the things needed for the Combined Lib 1) be compilable, 2)
> be able to load w/o errors.
> 
> For example, I could see probe ctor functions from various PMD's blowing up
> on unsupported hardware. Like how the rte_pmd_virtio had issues when I
> tried it on my VM system.
> 
IMHO the underlying issue was that virtio was unsupported for that platform.
What I am trying to say is that any app built against a combined shared lib or combined/separated 
static lib will have that same issue if the feature/PMD is unsupported for the platform.
In the virtio case, building against combined shared or combined/separated static DPDK libs
would have the same result because the virtio PMD would be in the app.

> If we think we can make sure no platform specific stuff breaks when it ends
> up in Combined Lib A then A is probably the easiest for all.
> 
I agree, a combined lib would simplify all app/lib linking, making it easier for the user and less error prone.

One of the downsides that comes to mind is a flow work (I think you mentioned it) where you
have multiple apps building against a single DPDK copy, and each app uses different features/PMDs.
In that scenario, having separated libs would make your life easier as you would not need to  have
multiple DPDK copies customized for each app, giving you the flexibility to hand pick each lib you
want to include into your app.
That flow work still presents some issues as they may be features that are incompatible between
each other and would need to be in different DPDK copies.

Regards,
Sergio

> Matthew.

Re: [dpdk-dev] [PATCH RFC] Update/Improve build system

[Matthew Hall]

On Fri, Oct 31, 2014 at 10:45:07AM +0000, Gonzalez Monroy, Sergio wrote:
> That flow work still presents some issues as they may be features that are 
> incompatible between each other and would need to be in different DPDK 
> copies.
> 
> Regards,
> Sergio

So I think the two questions are:

1) Will there ever be cases where something will compile on some platform but 
break at runtime and thus need to be excluded from a library?

2) Are there incompatible features where the library could break if both are 
enabled at once?

If the answer to both questions is 'No' then the simple single-shared, 
single-static, is definitely simplest and most reliable.

Matthew.

Re: [dpdk-dev] [PATCH RFC] Update/Improve build system

[Neil Horman]

On Fri, Oct 31, 2014 at 10:45:07AM +0000, Gonzalez Monroy, Sergio wrote:
> > From: Matthew Hall [mailto:mhall@mhcomputing.net]
> > Sent: Thursday, October 30, 2014 8:50 PM
> > 
> > On Thu, Oct 30, 2014 at 09:18:23AM +0000, Gonzalez Monroy, Sergio wrote:
> > > I would say that D) is a good balance, although not being the simplest.
> > 
> > A, or D. Depending on things such as, "If you run the DPDK on Random
> > Platform X," where X could be something like Power CPUs or other weird
> > stuff, will all of the things needed for the Combined Lib 1) be compilable, 2)
> > be able to load w/o errors.
> > 
> > For example, I could see probe ctor functions from various PMD's blowing up
> > on unsupported hardware. Like how the rte_pmd_virtio had issues when I
> > tried it on my VM system.
> > 
> IMHO the underlying issue was that virtio was unsupported for that platform.
> What I am trying to say is that any app built against a combined shared lib or combined/separated 
> static lib will have that same issue if the feature/PMD is unsupported for the platform.
> In the virtio case, building against combined shared or combined/separated static DPDK libs
> would have the same result because the virtio PMD would be in the app.
> 
> > If we think we can make sure no platform specific stuff breaks when it ends
> > up in Combined Lib A then A is probably the easiest for all.
> > 
> I agree, a combined lib would simplify all app/lib linking, making it easier for the user and less error prone.
> 
> One of the downsides that comes to mind is a flow work (I think you mentioned it) where you
> have multiple apps building against a single DPDK copy, and each app uses different features/PMDs.
> In that scenario, having separated libs would make your life easier as you would not need to  have
> multiple DPDK copies customized for each app, giving you the flexibility to hand pick each lib you
> want to include into your app.
> That flow work still presents some issues as they may be features that are incompatible between
> each other and would need to be in different DPDK copies.
> 
It might be reasonable here to consider pmd's separately from the rest of the
system, given that, they are supposed be built as isolated libraries that don't
export any symbols.  Some pmd's break that assumption, but they should really be
fixed anyway.  If you did that, you can build the dpdk as a single library, and
build the pmds as separate libraries (shared or static).

Neil

> Regards,
> Sergio
> 
> > Matthew.
>