[dpdk-stable] [PATCH] pci: fix missing pci bus with shared library build

[dpdk-stable] [PATCH] pci: fix missing pci bus with shared library build

[Stephen Hemminger]

If DPDK is built as a shared library, then any application linked
with rte.app.mk will not find any PCI devices. When the application
is started no ethernet devices are found.

This is because the link order of libraries on the command line matters.
And PCI is before EAL. That causes there to be no dependency on PCI
so linker ignores linking the library. 
Swapping the order fixes this.

Fixes: c752998b5e2e ("pci: introduce library and driver")
Cc: stable@dpdk.org
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
 mk/rte.app.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index a277c808ed8e..470b92e4d73e 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
 _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
-_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
 _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched
-- 
2.20.1

Re: [dpdk-stable] [PATCH] pci: fix missing pci bus with shared library build

[Stephen Hemminger]

On Mon, 15 Jul 2019 16:41:36 -0700
Stephen Hemminger <stephen@networkplumber.org> wrote:

> If DPDK is built as a shared library, then any application linked
> with rte.app.mk will not find any PCI devices. When the application
> is started no ethernet devices are found.
> 
> This is because the link order of libraries on the command line matters.
> And PCI is before EAL. That causes there to be no dependency on PCI
> so linker ignores linking the library. 
> Swapping the order fixes this.
> 
> Fixes: c752998b5e2e ("pci: introduce library and driver")
> Cc: stable@dpdk.org
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
>  mk/rte.app.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> index a277c808ed8e..470b92e4d73e 100644
> --- a/mk/rte.app.mk
> +++ b/mk/rte.app.mk
> @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
>  _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
>  _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
>  _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
> -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
>  _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
> +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
>  _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
>  _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
>  _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched

This only seems to happen on 18.11 and if configuration is reduced.

Re: [dpdk-stable] [PATCH] pci: fix missing pci bus with shared library build

[Stephen Hemminger]

On Mon, 15 Jul 2019 16:41:36 -0700
Stephen Hemminger <stephen@networkplumber.org> wrote:

> If DPDK is built as a shared library, then any application linked
> with rte.app.mk will not find any PCI devices. When the application
> is started no ethernet devices are found.
> 
> This is because the link order of libraries on the command line matters.
> And PCI is before EAL. That causes there to be no dependency on PCI
> so linker ignores linking the library. 
> Swapping the order fixes this.
> 
> Fixes: c752998b5e2e ("pci: introduce library and driver")
> Cc: stable@dpdk.org
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
>  mk/rte.app.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> index a277c808ed8e..470b92e4d73e 100644
> --- a/mk/rte.app.mk
> +++ b/mk/rte.app.mk
> @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
>  _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
>  _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
>  _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
> -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
>  _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
> +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
>  _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
>  _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
>  _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched

It still happens with 19.08. Testpmd works but only because it is
linked with so many things. But l3fwd fails...

# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: failed to parse device "02:00.0"
EAL: Unable to parse device '02:00.0'
EAL: Error - exiting with code: 1
  Cause: Invalid EAL parameters

Re: [dpdk-stable] [dpdk-dev] [PATCH] pci: fix missing pci bus with shared library build

[Bruce Richardson]

On Mon, Jul 15, 2019 at 05:19:12PM -0700, Stephen Hemminger wrote:
> On Mon, 15 Jul 2019 16:41:36 -0700
> Stephen Hemminger <stephen@networkplumber.org> wrote:
> 
> > If DPDK is built as a shared library, then any application linked
> > with rte.app.mk will not find any PCI devices. When the application
> > is started no ethernet devices are found.
> > 
> > This is because the link order of libraries on the command line matters.
> > And PCI is before EAL. That causes there to be no dependency on PCI
> > so linker ignores linking the library. 
> > Swapping the order fixes this.
> > 
> > Fixes: c752998b5e2e ("pci: introduce library and driver")
> > Cc: stable@dpdk.org
> > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> > ---
> >  mk/rte.app.mk | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> > index a277c808ed8e..470b92e4d73e 100644
> > --- a/mk/rte.app.mk
> > +++ b/mk/rte.app.mk
> > @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
> >  _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
> >  _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
> >  _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
> > -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
> >  _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
> > +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
> >  _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
> >  _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
> >  _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched
> 
> It still happens with 19.08. Testpmd works but only because it is
> linked with so many things. But l3fwd fails...
> 
> # ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
> EAL: Detected 8 lcore(s)
> EAL: Detected 1 NUMA nodes
> EAL: failed to parse device "02:00.0"
> EAL: Unable to parse device '02:00.0'
> EAL: Error - exiting with code: 1
>   Cause: Invalid EAL parameters

I don't think the position of these is going to be the cause here, the more
likely cause is that the pci bus driver - and all other drivers - are not
linked into apps for shared library builds. You always need to pass "-d"
parameter to load drivers at init time (or have them installed in the
correct driver path). For example, for me with a shared library build the
following gives a no ports error:

	sudo ./build/l2fwd -c F00000 -- -p 3

while this succeeds and runs fine

	sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3

Re: [dpdk-stable] [dpdk-dev] [PATCH] pci: fix missing pci bus with shared library build

[Stephen Hemminger]

On Tue, 16 Jul 2019 09:46:04 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:

> On Mon, Jul 15, 2019 at 05:19:12PM -0700, Stephen Hemminger wrote:
> > On Mon, 15 Jul 2019 16:41:36 -0700
> > Stephen Hemminger <stephen@networkplumber.org> wrote:
> >   
> > > If DPDK is built as a shared library, then any application linked
> > > with rte.app.mk will not find any PCI devices. When the application
> > > is started no ethernet devices are found.
> > > 
> > > This is because the link order of libraries on the command line matters.
> > > And PCI is before EAL. That causes there to be no dependency on PCI
> > > so linker ignores linking the library. 
> > > Swapping the order fixes this.
> > > 
> > > Fixes: c752998b5e2e ("pci: introduce library and driver")
> > > Cc: stable@dpdk.org
> > > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> > > ---
> > >  mk/rte.app.mk | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> > > index a277c808ed8e..470b92e4d73e 100644
> > > --- a/mk/rte.app.mk
> > > +++ b/mk/rte.app.mk
> > > @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
> > >  _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
> > > -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
> > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched  
> > 
> > It still happens with 19.08. Testpmd works but only because it is
> > linked with so many things. But l3fwd fails...
> > 
> > # ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
> > EAL: Detected 8 lcore(s)
> > EAL: Detected 1 NUMA nodes
> > EAL: failed to parse device "02:00.0"
> > EAL: Unable to parse device '02:00.0'
> > EAL: Error - exiting with code: 1
> >   Cause: Invalid EAL parameters  
> 
> I don't think the position of these is going to be the cause here, the more
> likely cause is that the pci bus driver - and all other drivers - are not
> linked into apps for shared library builds. You always need to pass "-d"
> parameter to load drivers at init time (or have them installed in the
> correct driver path). For example, for me with a shared library build the
> following gives a no ports error:
> 
> 	sudo ./build/l2fwd -c F00000 -- -p 3
> 
> while this succeeds and runs fine
> 
> 	sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3

It is broken with original link order.
With the patch it works fine.

Re: [dpdk-stable] [dpdk-dev] [PATCH] pci: fix missing pci bus with shared library build

[Stephen Hemminger]

On Tue, 16 Jul 2019 09:46:04 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:

> I don't think the position of these is going to be the cause here, the more
> likely cause is that the pci bus driver - and all other drivers - are not
> linked into apps for shared library builds. You always need to pass "-d"
> parameter to load drivers at init time (or have them installed in the
> correct driver path). For example, for me with a shared library build the
> following gives a no ports error:
> 
> 	sudo ./build/l2fwd -c F00000 -- -p 3
> 
> while this succeeds and runs fine
> 
> 	sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3


Application should never have to use -d to force driver load!
The purpose of all the pmdinfo, and probing logic is to not do this.

Re: [dpdk-stable] [dpdk-dev] [PATCH] pci: fix missing pci bus with shared library build

[Stephen Hemminger]

On Tue, 16 Jul 2019 09:46:04 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:

> On Mon, Jul 15, 2019 at 05:19:12PM -0700, Stephen Hemminger wrote:
> > On Mon, 15 Jul 2019 16:41:36 -0700
> > Stephen Hemminger <stephen@networkplumber.org> wrote:
> >   
> > > If DPDK is built as a shared library, then any application linked
> > > with rte.app.mk will not find any PCI devices. When the application
> > > is started no ethernet devices are found.
> > > 
> > > This is because the link order of libraries on the command line matters.
> > > And PCI is before EAL. That causes there to be no dependency on PCI
> > > so linker ignores linking the library. 
> > > Swapping the order fixes this.
> > > 
> > > Fixes: c752998b5e2e ("pci: introduce library and driver")
> > > Cc: stable@dpdk.org
> > > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> > > ---
> > >  mk/rte.app.mk | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> > > index a277c808ed8e..470b92e4d73e 100644
> > > --- a/mk/rte.app.mk
> > > +++ b/mk/rte.app.mk
> > > @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
> > >  _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
> > > -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
> > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched  
> > 
> > It still happens with 19.08. Testpmd works but only because it is
> > linked with so many things. But l3fwd fails...
> > 
> > # ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
> > EAL: Detected 8 lcore(s)
> > EAL: Detected 1 NUMA nodes
> > EAL: failed to parse device "02:00.0"
> > EAL: Unable to parse device '02:00.0'
> > EAL: Error - exiting with code: 1
> >   Cause: Invalid EAL parameters  
> 
> I don't think the position of these is going to be the cause here, the more
> likely cause is that the pci bus driver - and all other drivers - are not
> linked into apps for shared library builds. You always need to pass "-d"
> parameter to load drivers at init time (or have them installed in the
> correct driver path). For example, for me with a shared library build the
> following gives a no ports error:
> 
> 	sudo ./build/l2fwd -c F00000 -- -p 3
> 
> while this succeeds and runs fine
> 
> 	sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3

There is something else going on.
This patch isn't fixing the problem consistently.

The suggestion of using -d is still wrong. We need to figure out how to get applications
to run the same whether using shared library or statically linked.

Re: [dpdk-stable] [dpdk-dev] [PATCH] pci: fix missing pci bus with shared library build

[Stephen Hemminger]

On Tue, 16 Jul 2019 09:46:04 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:

> On Mon, Jul 15, 2019 at 05:19:12PM -0700, Stephen Hemminger wrote:
> > On Mon, 15 Jul 2019 16:41:36 -0700
> > Stephen Hemminger <stephen@networkplumber.org> wrote:
> >   
> > > If DPDK is built as a shared library, then any application linked
> > > with rte.app.mk will not find any PCI devices. When the application
> > > is started no ethernet devices are found.
> > > 
> > > This is because the link order of libraries on the command line matters.
> > > And PCI is before EAL. That causes there to be no dependency on PCI
> > > so linker ignores linking the library. 
> > > Swapping the order fixes this.
> > > 
> > > Fixes: c752998b5e2e ("pci: introduce library and driver")
> > > Cc: stable@dpdk.org
> > > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> > > ---
> > >  mk/rte.app.mk | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> > > index a277c808ed8e..470b92e4d73e 100644
> > > --- a/mk/rte.app.mk
> > > +++ b/mk/rte.app.mk
> > > @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
> > >  _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
> > > -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
> > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
> > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched  
> > 
> > It still happens with 19.08. Testpmd works but only because it is
> > linked with so many things. But l3fwd fails...
> > 
> > # ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
> > EAL: Detected 8 lcore(s)
> > EAL: Detected 1 NUMA nodes
> > EAL: failed to parse device "02:00.0"
> > EAL: Unable to parse device '02:00.0'
> > EAL: Error - exiting with code: 1
> >   Cause: Invalid EAL parameters  
> 
> I don't think the position of these is going to be the cause here, the more
> likely cause is that the pci bus driver - and all other drivers - are not
> linked into apps for shared library builds. You always need to pass "-d"
> parameter to load drivers at init time (or have them installed in the
> correct driver path). For example, for me with a shared library build the
> following gives a no ports error:
> 
> 	sudo ./build/l2fwd -c F00000 -- -p 3
> 
> while this succeeds and runs fine
> 
> 	sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3

The root cause is that recent gcc won't run constructor on unused libraries.
Testing a patch to take --as-needed off of PCI library.

See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library

Re: [dpdk-stable] [dpdk-dev] [PATCH] pci: fix missing pci bus with shared library build

[Thomas Monjalon]

19/07/2019 22:55, Stephen Hemminger:
> On Tue, 16 Jul 2019 09:46:04 +0100
> Bruce Richardson <bruce.richardson@intel.com> wrote:
> 
> > On Mon, Jul 15, 2019 at 05:19:12PM -0700, Stephen Hemminger wrote:
> > > On Mon, 15 Jul 2019 16:41:36 -0700
> > > Stephen Hemminger <stephen@networkplumber.org> wrote:
> > >   
> > > > If DPDK is built as a shared library, then any application linked
> > > > with rte.app.mk will not find any PCI devices. When the application
> > > > is started no ethernet devices are found.
> > > > 
> > > > This is because the link order of libraries on the command line matters.
> > > > And PCI is before EAL. That causes there to be no dependency on PCI
> > > > so linker ignores linking the library. 
> > > > Swapping the order fixes this.
> > > > 
> > > > Fixes: c752998b5e2e ("pci: introduce library and driver")
> > > > Cc: stable@dpdk.org
> > > > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> > > > ---
> > > >  mk/rte.app.mk | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > 
> > > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> > > > index a277c808ed8e..470b92e4d73e 100644
> > > > --- a/mk/rte.app.mk
> > > > +++ b/mk/rte.app.mk
> > > > @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
> > > >  _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
> > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
> > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
> > > > -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
> > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
> > > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
> > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
> > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
> > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched  
> > > 
> > > It still happens with 19.08. Testpmd works but only because it is
> > > linked with so many things. But l3fwd fails...
> > > 
> > > # ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
> > > EAL: Detected 8 lcore(s)
> > > EAL: Detected 1 NUMA nodes
> > > EAL: failed to parse device "02:00.0"
> > > EAL: Unable to parse device '02:00.0'
> > > EAL: Error - exiting with code: 1
> > >   Cause: Invalid EAL parameters  
> > 
> > I don't think the position of these is going to be the cause here, the more
> > likely cause is that the pci bus driver - and all other drivers - are not
> > linked into apps for shared library builds. You always need to pass "-d"
> > parameter to load drivers at init time (or have them installed in the
> > correct driver path). For example, for me with a shared library build the
> > following gives a no ports error:
> > 
> > 	sudo ./build/l2fwd -c F00000 -- -p 3
> > 
> > while this succeeds and runs fine
> > 
> > 	sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3
> 
> The root cause is that recent gcc won't run constructor on unused libraries.
> Testing a patch to take --as-needed off of PCI library.
> 
> See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library

The constructor is run when calling dlopen, right?

Note: dlopen with -d is a feature.
The original idea was to be able to specify which driver we want to use.
If we want an automatic dlopen, like modprobe, then we need more scripts.
But I understand you are against the whole dlopen idea.

Re: [dpdk-stable] [dpdk-dev] [PATCH] pci: fix missing pci bus with shared library build

[Bruce Richardson]

On Mon, Jul 22, 2019 at 09:38:27AM +0200, Thomas Monjalon wrote:
> 19/07/2019 22:55, Stephen Hemminger:
> > On Tue, 16 Jul 2019 09:46:04 +0100
> > Bruce Richardson <bruce.richardson@intel.com> wrote:
> > 
> > > On Mon, Jul 15, 2019 at 05:19:12PM -0700, Stephen Hemminger wrote:
> > > > On Mon, 15 Jul 2019 16:41:36 -0700
> > > > Stephen Hemminger <stephen@networkplumber.org> wrote:
> > > >   
> > > > > If DPDK is built as a shared library, then any application linked
> > > > > with rte.app.mk will not find any PCI devices. When the application
> > > > > is started no ethernet devices are found.
> > > > > 
> > > > > This is because the link order of libraries on the command line matters.
> > > > > And PCI is before EAL. That causes there to be no dependency on PCI
> > > > > so linker ignores linking the library. 
> > > > > Swapping the order fixes this.
> > > > > 
> > > > > Fixes: c752998b5e2e ("pci: introduce library and driver")
> > > > > Cc: stable@dpdk.org
> > > > > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> > > > > ---
> > > > >  mk/rte.app.mk | 2 +-
> > > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > > 
> > > > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> > > > > index a277c808ed8e..470b92e4d73e 100644
> > > > > --- a/mk/rte.app.mk
> > > > > +++ b/mk/rte.app.mk
> > > > > @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
> > > > >  _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
> > > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
> > > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
> > > > > -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
> > > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
> > > > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
> > > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
> > > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
> > > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched  
> > > > 
> > > > It still happens with 19.08. Testpmd works but only because it is
> > > > linked with so many things. But l3fwd fails...
> > > > 
> > > > # ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
> > > > EAL: Detected 8 lcore(s)
> > > > EAL: Detected 1 NUMA nodes
> > > > EAL: failed to parse device "02:00.0"
> > > > EAL: Unable to parse device '02:00.0'
> > > > EAL: Error - exiting with code: 1
> > > >   Cause: Invalid EAL parameters  
> > > 
> > > I don't think the position of these is going to be the cause here, the more
> > > likely cause is that the pci bus driver - and all other drivers - are not
> > > linked into apps for shared library builds. You always need to pass "-d"
> > > parameter to load drivers at init time (or have them installed in the
> > > correct driver path). For example, for me with a shared library build the
> > > following gives a no ports error:
> > > 
> > > 	sudo ./build/l2fwd -c F00000 -- -p 3
> > > 
> > > while this succeeds and runs fine
> > > 
> > > 	sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3
> > 
> > The root cause is that recent gcc won't run constructor on unused libraries.
> > Testing a patch to take --as-needed off of PCI library.
> > 
> > See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library
> 
> The constructor is run when calling dlopen, right?
> 
> Note: dlopen with -d is a feature.
> The original idea was to be able to specify which driver we want to use.
> If we want an automatic dlopen, like modprobe, then we need more scripts.
> But I understand you are against the whole dlopen idea.
> 

This issue is more of a problem for development systems where we EAL path
is not really usable for finding the drivers. For a properly deployed
system where we use DPDK installed to /usr/local or /usr, the EAL PMD path
will be correctly configured and properly probe all drivers.

Re: [dpdk-stable] [dpdk-dev] [PATCH] pci: fix missing pci bus with shared library build

[Stephen Hemminger]

On Mon, 22 Jul 2019 10:06:11 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:

> On Mon, Jul 22, 2019 at 09:38:27AM +0200, Thomas Monjalon wrote:
> > 19/07/2019 22:55, Stephen Hemminger:  
> > > On Tue, 16 Jul 2019 09:46:04 +0100
> > > Bruce Richardson <bruce.richardson@intel.com> wrote:
> > >   
> > > > On Mon, Jul 15, 2019 at 05:19:12PM -0700, Stephen Hemminger wrote:  
> > > > > On Mon, 15 Jul 2019 16:41:36 -0700
> > > > > Stephen Hemminger <stephen@networkplumber.org> wrote:
> > > > >     
> > > > > > If DPDK is built as a shared library, then any application linked
> > > > > > with rte.app.mk will not find any PCI devices. When the application
> > > > > > is started no ethernet devices are found.
> > > > > > 
> > > > > > This is because the link order of libraries on the command line matters.
> > > > > > And PCI is before EAL. That causes there to be no dependency on PCI
> > > > > > so linker ignores linking the library. 
> > > > > > Swapping the order fixes this.
> > > > > > 
> > > > > > Fixes: c752998b5e2e ("pci: introduce library and driver")
> > > > > > Cc: stable@dpdk.org
> > > > > > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> > > > > > ---
> > > > > >  mk/rte.app.mk | 2 +-
> > > > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > > > 
> > > > > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> > > > > > index a277c808ed8e..470b92e4d73e 100644
> > > > > > --- a/mk/rte.app.mk
> > > > > > +++ b/mk/rte.app.mk
> > > > > > @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
> > > > > >  _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
> > > > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
> > > > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
> > > > > > -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
> > > > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
> > > > > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
> > > > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
> > > > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
> > > > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched    
> > > > > 
> > > > > It still happens with 19.08. Testpmd works but only because it is
> > > > > linked with so many things. But l3fwd fails...
> > > > > 
> > > > > # ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
> > > > > EAL: Detected 8 lcore(s)
> > > > > EAL: Detected 1 NUMA nodes
> > > > > EAL: failed to parse device "02:00.0"
> > > > > EAL: Unable to parse device '02:00.0'
> > > > > EAL: Error - exiting with code: 1
> > > > >   Cause: Invalid EAL parameters    
> > > > 
> > > > I don't think the position of these is going to be the cause here, the more
> > > > likely cause is that the pci bus driver - and all other drivers - are not
> > > > linked into apps for shared library builds. You always need to pass "-d"
> > > > parameter to load drivers at init time (or have them installed in the
> > > > correct driver path). For example, for me with a shared library build the
> > > > following gives a no ports error:
> > > > 
> > > > 	sudo ./build/l2fwd -c F00000 -- -p 3
> > > > 
> > > > while this succeeds and runs fine
> > > > 
> > > > 	sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3  
> > > 
> > > The root cause is that recent gcc won't run constructor on unused libraries.
> > > Testing a patch to take --as-needed off of PCI library.
> > > 
> > > See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library  
> > 
> > The constructor is run when calling dlopen, right?
> > 
> > Note: dlopen with -d is a feature.
> > The original idea was to be able to specify which driver we want to use.
> > If we want an automatic dlopen, like modprobe, then we need more scripts.
> > But I understand you are against the whole dlopen idea.
> >   
> 
> This issue is more of a problem for development systems where we EAL path
> is not really usable for finding the drivers. For a properly deployed
> system where we use DPDK installed to /usr/local or /usr, the EAL PMD path
> will be correctly configured and properly probe all drivers.

The problem is that bus drivers register themselves in constructors and
these construtors are not run with as-needed.

One part of fixing this is:
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index df917f946497..46bdff8ec5e8 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -130,6 +130,9 @@ ifeq ($(CONFIG_RTE_LIBRTE_FSLMC_BUS),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_COMMON_FSLMC)   += -lrte_common_fslmc
 endif
 
+# Bus devices register in constructor so always link
+_LDLIBS-y      += --no-as-needed
+
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PCI_BUS)        += -lrte_bus_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_VDEV_BUS)       += -lrte_bus_vdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_DPAA_BUS)       += -lrte_bus_dpaa
@@ -137,6 +140,8 @@ ifeq ($(CONFIG_RTE_EAL_VFIO),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS)      += -lrte_bus_fslmc
 endif
 
+_LDLIBS-y      += --as-needed
+
 ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),n)
 # plugins (link only if static libraries)
 

Re: [dpdk-stable] [dpdk-dev] [PATCH] pci: fix missing pci bus with shared library build

[Thomas Monjalon]

22/07/2019 18:43, Stephen Hemminger:
> On Mon, 22 Jul 2019 10:06:11 +0100
> Bruce Richardson <bruce.richardson@intel.com> wrote:
> > On Mon, Jul 22, 2019 at 09:38:27AM +0200, Thomas Monjalon wrote:
> > > 19/07/2019 22:55, Stephen Hemminger:  
> > > > The root cause is that recent gcc won't run constructor on unused libraries.
> > > > Testing a patch to take --as-needed off of PCI library.
> > > > 
> > > > See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library  
> > > 
> > > The constructor is run when calling dlopen, right?
> > > 
> > > Note: dlopen with -d is a feature.
> > > The original idea was to be able to specify which driver we want to use.
> > > If we want an automatic dlopen, like modprobe, then we need more scripts.
> > > But I understand you are against the whole dlopen idea.
> > 
> > This issue is more of a problem for development systems where we EAL path
> > is not really usable for finding the drivers. For a properly deployed
> > system where we use DPDK installed to /usr/local or /usr, the EAL PMD path
> > will be correctly configured and properly probe all drivers.
> 
> The problem is that bus drivers register themselves in constructors and
> these construtors are not run with as-needed.

Are the constructors run on dlopen of the bus driver?

Re: [dpdk-stable] [dpdk-dev] [PATCH] pci: fix missing pci bus with shared library build

[Stephen Hemminger]

On Mon, 22 Jul 2019 19:04:55 +0200
Thomas Monjalon <thomas@monjalon.net> wrote:

> 22/07/2019 18:43, Stephen Hemminger:
> > On Mon, 22 Jul 2019 10:06:11 +0100
> > Bruce Richardson <bruce.richardson@intel.com> wrote:  
> > > On Mon, Jul 22, 2019 at 09:38:27AM +0200, Thomas Monjalon wrote:  
> > > > 19/07/2019 22:55, Stephen Hemminger:    
> > > > > The root cause is that recent gcc won't run constructor on unused libraries.
> > > > > Testing a patch to take --as-needed off of PCI library.
> > > > > 
> > > > > See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library    
> > > > 
> > > > The constructor is run when calling dlopen, right?
> > > > 
> > > > Note: dlopen with -d is a feature.
> > > > The original idea was to be able to specify which driver we want to use.
> > > > If we want an automatic dlopen, like modprobe, then we need more scripts.
> > > > But I understand you are against the whole dlopen idea.  
> > > 
> > > This issue is more of a problem for development systems where we EAL path
> > > is not really usable for finding the drivers. For a properly deployed
> > > system where we use DPDK installed to /usr/local or /usr, the EAL PMD path
> > > will be correctly configured and properly probe all drivers.  
> > 
> > The problem is that bus drivers register themselves in constructors and
> > these construtors are not run with as-needed.  
> 
> Are the constructors run on dlopen of the bus driver?
> 
> 
> 

Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.

The core principle is that dynamic build of DPDK should act the same as old
statically linked DPDK. Otherwise, the user experience is even worse, and all
the example documentation is wrong.