Re: [dpdk-dev] [PATCH] pci: fix missing pci bus with shared library build

[Bruce Richardson <bruce.richardson@intel.com>]

On Tue, Jul 23, 2019 at 11:11:14AM -0700, Stephen Hemminger wrote:
> On Tue, 23 Jul 2019 13:30:33 +0100
> Bruce Richardson <bruce.richardson@intel.com> wrote:
> 
> > On Mon, Jul 22, 2019 at 11:53:26AM -0700, Stephen Hemminger wrote:
> > > On Mon, 22 Jul 2019 19:31:08 +0200
> > > Thomas Monjalon <thomas@monjalon.net> wrote:
> > >   
> > > > 22/07/2019 19:13, Stephen Hemminger:  
> > > > > Thomas Monjalon <thomas@monjalon.net> wrote:    
> > > > > > Are the constructors run on dlopen of the bus driver?    
> > > > > 
> > > > > Yes, constructors are run on dlopen.
> > > > > But application should not have to ask DPDK to dlopen the bus devices.
> > > > > 
> > > > > The core principle is that dynamic build of DPDK should act the same as old
> > > > > statically linked DPDK. Otherwise, the user experience is even worse, and all
> > > > > the example documentation is wrong.    
> > > > 
> > > > OK, this is where I wanted to bring the discussion.
> > > > You are arguing against a design which is in DPDK from some early days.
> > > > So this is an interesting discussion to have.
> > > > Do we want to change the "plugin model" we have?
> > > > Or do we want to simply drop this model (dlopen calls)
> > > > and replace it with strong dynamic linking?
> > > > 
> > > >   
> > > 
> > > What I think should happen (and isn't is):
> > > 
> > > 1. The PCI bus library is linked with --whole-archive, and --no-as-needed.
> > >    This causes constructor to be called and register the bus.
> > >   
> > 
> > This should be applied to the whole of the bus drivers, not just the PCI
> > bus.
> > 
> > > 2. As part of the build process all the PCI drivers pmdinfo would get
> > >    constructed into a table of vendor/device to PMD shared library name.
> > > 
> > > 3. PMD's are linked as --whole-archive, and --as-needed.
> > >   
> > 
> > I'm not sure I agree with this change to always link in all the PMDs. It
> > prevents an app from being used with just a subset of the drivers needed.
> > 
> > > 4. New code in PCI probe which looks for existing entries (static or -d)
> > >    for devices. If device is still not found it refers to the table of PMD's
> > >    (from #2) and calls dlopen for that device (and adds it to static table).
> > > 
> > > This would allow examples and customer applications to Just Work without
> > > having to know the PMD that is present. It would also solve the problem
> > > that currently if applications is linked with -ldpdk linker script then
> > > all PMD's get pulled into the application address space.
> > >   
> > 
> > In all this you seem to be assuming that the drivers are not picked up at
> > runtime from the RTE_EAL_PMD_PATH. In real world cases where a user is
> > building an app, and not developing DPDK itself, the DPDK libraries should
> > be installed in /usr(/local)/lib64 and the drivers in
> > .../lib64/dpdk/dpdk-19.08. In that case, the bus drivers and the PMD
> > drivers are all loaded at runtime for each app, without having any
> > dependency on having a specific one be present, allowing a user to remove
> > any drivers unnecessary for the current hardware.
> > 
> > Did you try installing DPDK using "ninja install" or "make install" before
> > running any apps using it?
> > 
> > /Bruce
> 
> I was using "make install-runtime" into a local chroot directory like
> a distribution package builder does.
> 
> 
> There are multiple use cases:
> 1. Developer build DPDK and application together on one machine (and running on another).
>    Or software appliance being built on one machine and run in many environments.
>    Also cross builds are like this.
> 
Sure, but the expectation here is that the resultant build is installed in
system locations, or a known location. As part of the build EAL_PMD_PATH
needs to be set to the final driver location and then the driver loading
issue requiring "-d" flag goes away.

> 2. Distribution building a package (and installing in standard library locations /lib etc).
> 
No issue here, since package building is not going to be a problem and when
installed the drivers are all picked up from the EAL_PMD_PATH.

> 3. Demo machine where build is local and native.
> 
> DPDK seems to always focus on #3 which is least interesting for real production.

You are missing our primary target use-case for the future -
* developer builds application using distro-supplied DPDK.

Individual developers building their own DPDK alongside their app should be
a thing of the past. DPDK needs to be come like other libraries e.g. zlib,
openssl, where app developers just pick it up from their linux distro and
not feel the need to roll their own.

Yes, there will be cases where things are more memory constrained etc. and
this model won't work, but despite this, it should be our primary target.
To get there, though, there is still work to be done, e.g. in increasing
ABI compatibility to allow easier adoption by distros, etc.

/Bruce