Re: [dpdk-dev] [PATCH 2/2] devtools: add path to additional shared object files

["Richardson, Bruce" <bruce.richardson@intel.com>]

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Wednesday, December 18, 2019 3:32 PM
> To: Laatz, Kevin <kevin.laatz@intel.com>
> Cc: Ruifeng Wang <ruifeng.wang@arm.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Aaron Conole <aconole@redhat.com>; Michael
> Santana <maicolgabriel@hotmail.com>; Thomas Monjalon
> <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com>; Andrew
> Rybchenko <arybchenko@solarflare.com>; dev <dev@dpdk.org>; Gavin Hu
> <gavin.hu@arm.com>; Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>;
> nd <nd@arm.com>
> Subject: Re: [dpdk-dev] [PATCH 2/2] devtools: add path to additional
> shared object files
> 
> On Wed, Dec 18, 2019 at 2:43 PM Laatz, Kevin <kevin.laatz@intel.com>
> wrote:
> >
> > On 18/12/2019 08:23, David Marchand wrote:
> > > On Wed, Dec 18, 2019 at 6:39 AM Ruifeng Wang <ruifeng.wang@arm.com>
> wrote:
> > >> librte_mempool_ring.so and librte_pmd_null.so are in 'drivers'
> folder.
> > >> Add 'drivers' into LD_LIBRARY_PATH so that testpmd can find and
> > >> make use of these shared libraries.
> > >>
> > >> Signed-off-by: Ruifeng Wang <ruifeng.wang@arm.com>
> > >> Reviewed-by: Gavin Hu <gavin.hu@arm.com>
> > >> ---
> > >>   devtools/test-null.sh | 2 +-
> > >>   1 file changed, 1 insertion(+), 1 deletion(-)
> > >>
> > >> diff --git a/devtools/test-null.sh b/devtools/test-null.sh index
> > >> f39af2c06..548de8113 100755
> > >> --- a/devtools/test-null.sh
> > >> +++ b/devtools/test-null.sh
> > >> @@ -20,7 +20,7 @@ if [ ! -f "$testpmd" ] ; then
> > >>   fi
> > >>
> > >>   if ldd $testpmd | grep -q librte_ ; then
> > >> -       export LD_LIBRARY_PATH=$build/lib:$LD_LIBRARY_PATH
> > >> +       export
> > >> + LD_LIBRARY_PATH=$build/drivers:$build/lib:$LD_LIBRARY_PATH
> > >>          libs='-d librte_mempool_ring.so -d librte_pmd_null.so'
> > >>   else
> > >>          libs=
> > >> --
> > >> 2.17.1
> > >>
> > > I'm surprised to see this.
> > > So far, the CI ran fine without it, so something is different in the
> > > environment.
> > >
> > > I can see that the RPATH entry disappeared from the testpmd binary.
> > > Xenial:
> > > # readelf -d build/app/dpdk-testpmd |grep RPATH
> > >   0x000000000000000f (RPATH)              Library rpath:
> > > [$ORIGIN/../lib:$ORIGIN/../drivers:XXXXXXXXXXXXX]
> > >
> > > (not sure what XXXX purpose is, but different topic)
> > >
> > > Bionic:
> > > # readelf -d build/app/dpdk-testpmd |grep RPATH
> >
> > It looks like RPATH just changed to be named RUNPATH in Bionic:
> >
> > # readelf -d build/app/dpdk-testpmd | grep R.*PATH
> >   0x000000000000001d (RUNPATH)            Library runpath:
> > [$ORIGIN/../lib:$ORIGIN/../drivers:XXXXXXXXXXXXX]
> 
> Did some experiment with some test program and .so of mine.
> TL;DR, if I understand correctly, RPATH on the binary applies to all
> lookups, even in a subsequent .so code.
> But RUNPATH only applies to the current ELF, meaning that the dlopen() in
> my intermediate .so does not get it.
> 
> dlopen() is called from librte_eal.so, and RUNPATH on testpmd is not
> enough.
> 
> 
> Details:
> [dmarchan@dmarchan plop]$ cat main.c
> extern void loader(void);
> 
> int main(int argc, char *argv[])
> {
>     loader();
>     return 0;
> }
> [dmarchan@dmarchan plop]$ cat loader/loader.c #include <dlfcn.h> #include
> <stdio.h>
> 
> void loader(void)
> {
>     if (dlopen("lib.so", RTLD_NOW) == NULL)
>         fprintf(stderr, "%s\n", dlerror()); }
> 
> [dmarchan@dmarchan plop]$ cat lib/lib.c
> #include <stdio.h>
> 
> __attribute__((constructor))
> static void plop(void)
> {
>     fprintf(stdout, "plop\n");
> }
> 
> 
> # no rpath/runpath
> [dmarchan@dmarchan plop]$ gcc -o lib/lib.so -Wall -Werror -shared -fPIC
> lib/lib.c [dmarchan@dmarchan plop]$ gcc -o loader/loader.so -Wall -Werror
> -shared -fPIC loader/loader.c -ldl [dmarchan@dmarchan plop]$ gcc -o main -
> Wall -Werror main.c loader/loader.so [dmarchan@dmarchan plop]$ readelf -d
> main |grep R.*PATH [dmarchan@dmarchan plop]$ ./main
> lib.so: cannot open shared object file: No such file or directory
> 
> # using rpath on final binary
> [dmarchan@dmarchan plop]$ gcc -o main -Wall -Werror main.c
> loader/loader.so -Wl,-rpath,loader:lib [dmarchan@dmarchan plop]$ readelf -
> d main |grep R.*PATH
>  0x000000000000000f (RPATH)              Library rpath: [loader:lib]
> [dmarchan@dmarchan plop]$ ./main
> plop
> 
> # using runpath on final binary
> [dmarchan@dmarchan plop]$ gcc -o main -Wall -Werror main.c
> loader/loader.so -Wl,-enable-new-dtag,-rpath,loader:lib
> [dmarchan@dmarchan plop]$ readelf -d main |grep R.*PATH
>  0x000000000000001d (RUNPATH)            Library runpath: [loader:lib]
> [dmarchan@dmarchan plop]$ ./main
> lib.so: cannot open shared object file: No such file or directory
> 
> # using runpath on loader
> [dmarchan@dmarchan plop]$ gcc -o loader/loader.so -Wall -Werror -shared -
> fPIC loader/loader.c -ldl -Wl,-enable-new-dtag,-rpath,lib
> [dmarchan@dmarchan plop]$ readelf -d loader/loader.so |grep R.*PATH
>  0x000000000000001d (RUNPATH)            Library runpath: [lib]
> [dmarchan@dmarchan plop]$ gcc -o main -Wall -Werror main.c
> loader/loader.so -Wl,-enable-new-dtag,-rpath,loader
> [dmarchan@dmarchan plop]$ readelf -d main |grep R.*PATH
>  0x000000000000001d (RUNPATH)            Library runpath: [loader]
> [dmarchan@dmarchan plop]$ ./main
> plop
> 
> 
> > > Adding Bruce and Kevin, as I think this is the same issue than:
> > > http://mails.dpdk.org/archives/dev/2019-December/153627.html
> > > Could it be a change in meson?
> >
> > Yes, looks like the same error to me.
> >
> > I'm not sure this is solely a meson issue, I often need to pass -d
> > librte_mempool_ring.so when using make builds too. Maybe I'm missing
> > something here?
> 
> In a make build directory, all libraries ends up in the same lib/
> directory and the test-null.sh script work with the current
> LD_LIBRARY_PATH.
> 
> 
> Ruifeng patch seems valid to me, but I would love some explanations in the
> commitlog.
> 
> --
> David Marchand

Really, if we are running a shared-library build, all sorts of problems are
to be expected unless we do an "install" to place the .so files in their correct
locations on the filesystem.

For example, on install, the drivers are copied to a drivers directory off the
lib folder, but then symlinked to lib too, so that e.g. the bus drivers can be
found as dependencies of the other drivers.

For running binaries within the build directory, I always recommend using a 
static build instead, to avoid all these issues - this is why static linking
is still the DPDK default.

In terms of this patch, I have no problem with it if it fixed the issue.

/Bruce