Re: [dpdk-dev] [PATCH] mk: fix application compilation with lmnl and mlx5

[Shahaf Shuler <shahafs@mellanox.com>]

Tuesday, July 24, 2018 6:14 PM, Adrien Mazarguil:
> Subject: Re: [PATCH] mk: fix application compilation with lmnl and mlx5
> On Tue, Jul 24, 2018 at 01:03:28PM +0000, Shahaf Shuler wrote:
> > Tuesday, July 24, 2018 3:56 PM, Adrien Mazarguil:
> > > Subject: Re: [PATCH] mk: fix application compilation with lmnl and
> > > mlx5

[...]

> > > > Can we consider different options:
> > > > 1. always statically link libmnl
> > > > 2. dlopen libmnl just like we do for verbs/mlx5
> > >
> > > Regarding 2, unlike rdma-core/MLNX_OFED, libmnl should be available
> > > pretty much everywhere iproute2 can be found. The minimal version
> > > supported
> > > (1.0.3) was released in 2012.
> > >
> > > Using the glue approach for such a small library seems overkill;
> > > should we choose this path, we must also consider to get rid of it
> > > entirely since doing so would require more glue code than what mlx5
> needs from this library.
> > >
> > > So with the current approach, either the application or the PMD
> > > inherits a dependency to libmnl, depending on whether
> > > CONFIG_RTE_BUILD_SHARED_LIB is respectively disabled or enabled.
> > >
> > > If disabled, applications that want static linkage can specify
> > > -static as part of their compilation flags to let the compiler
> > > automatically look for libmnl.a as needed.
> >
> > Can you confirm static compilation w/ libmnl is working w/o any issues?
> 
> Challenge accepted.

😊

 Using -static is not something DPDK applications usually
> do and needs a few tweaks to compile successfully though (unless you
> meant something else by "static compilation"?)

Yes this is what I meant. However I was under the impression we can statically link to libmnl while keeping the rdma-core w/ dynamic linkage. 

> 
> First you need to force rdma-core to generate static versions of
> libmlx4/libmlx5 and libiberbs as it's not the default:
> 
>  $ cmake -DENABLE_STATIC=1 .
> 
> Next, patch DPDK to remove references to -lgcc_s, -fPIC and -export-
> dynamic:
> 
>  $ sed -i 's/[[:space:]]*-lgcc_s//' $(git grep -l -- -lgcc_s)  $ sed -i 's/[[:space:]]*-
> fPIC//' $(git grep -l -- -fPIC)  $ sed -i 's/[[:space:]]*-export-dynamic//' $(git
> grep -l -- -export-dynamic)

It seems the fPIC and export-dynamic are defined only when compiling the DPDK as shared library. 
The gcc_s is from eal. Do you think it is correct to put it only when compiling as shared library? It seems this approach was taken in the rte.vars.mk files. 

> 
> Then append rdma-core dependencies to libmnl users (mlx5) in order to
> avoid undefined references to libnl/libm stuff normally pulled by libibverbs:
> 
>  $ sed -i 's/-lmnl/-lmnl -lnl-route-3 -lnl-3 -lm/' $(git grep -l -- -lmnl)
> 
> Configure DPDK with mlx5 enabled:
> 
>  $ make config T=x86_64-native-linuxapp-gcc  $ sed -i
> 's/^\(CONFIG_RTE_LIBRTE_MLX5_PMD\)=.*/\1=y/' build/.config
> 
> Finally add -static to $CC (the easiest method I could find) while compiling
> DPDK:
> 
>  $ make CC='gcc -static'
> 
> You can ignore warnings about statically linking "getaddrinfo" and friends.
> What matters is we now have a testpmd binary with no dependencies:
> 
>  $ readelf -d build/app/testpmd | grep NEEDED  $
> 
>  $ ldd build/app/testpmd
>          not a dynamic executable
>  $
> 
> Now start testpmd with a mlx5 adapter and a couple of representors for fun:
> 
>  # ./build/app/testpmd -w 06:00.0,representor=[0-1] -n 4 -c 0x80 -- -i --rxq=1
> --txq=1
>  EAL: Detected 32 lcore(s)
>  EAL: Detected 2 NUMA nodes
>  EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>  EAL: No free hugepages reported in hugepages-1048576kB
>  EAL: Probing VFIO support...
>  EAL: PCI device 0000:06:00.0 on NUMA socket 0
>  EAL:   probe driver: 15b3:1017 net_mlx5
>  [...]
>  Configuring Port 0 (socket 0)
>  Port 0: 24:8A:07:8D:AE:F6
>  Configuring Port 1 (socket 0)
>  Port 1: A6:41:D9:89:DB:12
>  Configuring Port 2 (socket 0)
>  Port 2: FE:A8:15:80:DE:C0
>  Checking link statuses...
>  Done
>  testpmd>
> 
> Phew!

Nicely done. 

So if indeed it is possible w/ some fixes on the DPDK to fully support static linkage of both rdma-core and libmnl, maybe we should consider such compilation flag for mlx drivers. This can be very good alternative to the current DLOPEN approach. 

> 
> > To put this in perspective, this also applies to all other
> > dependencies
> > > it will collect while compiling DPDK (libz, libdl, libpcap, libnuma
> > > to name a few).
> > >
> > > In my opinion, the purpose of *_DLOPEN_DEPS is to deal with large,
> > > nonstandard libraries where versioning issues are commonplace. This
> > > doesn't apply to libmnl, which shouldn't be a maintenance nightmare
> > > to package maintainers. I suggest to leave things as is.
> 
> --
> Adrien Mazarguil
> 6WIND