From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by dpdk.org (Postfix) with ESMTP id F3D4B25D9 for ; Wed, 25 Jul 2018 11:07:43 +0200 (CEST) Received: by mail-wm0-f65.google.com with SMTP id o18-v6so5228103wmc.0 for ; Wed, 25 Jul 2018 02:07:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=zyRG/NozNLHBgBZNgacgmzOuNzqjE8Q+v5y0M4FRFt4=; b=RXMAUBU7N7ZXBvfL4aSHDglOQm9wlakwdi+Qw5mezCSL+ozVsks6qwXKZt1fyNltjo KqhmAIsVhTlfsnfqt1snHvRXBBUF/etMbruqlwF03Vcfm92rbDfLz6FQB2HKT/ayp/ec C2oNYPsX3j6tlqiKJXWEY1DEGorsj0qK23IgBBbrAFaCNBme7koeQwkN9fqGWIn8Wjel onaA6HiiwY5XmKFLZH2UFoaiPjahLkm0ufxhbLp5akZXO4cLwNXixYay+JKLtHdGmYUT ZDuBffJHFE1F7EpFxkEJrLT4q1Nj8yeq+O+hoao78eVHY133PoiTERK1LXpfAFrV4aKB PF8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=zyRG/NozNLHBgBZNgacgmzOuNzqjE8Q+v5y0M4FRFt4=; b=BWA32tFSn95Cei/FXA2SXHzc7lE0NTLmZdl9XxM+VRPktQs2lrs3Xds6BVemqZ5zRt z05KFyNXOy3OJz5CYibixb72PLbekd7fXYYw8Z2mX8xlUT2xqFpDFTpeHJ4fVF+hd/n+ L8UvxzcbU3vxzHXTT6cjSLUbLK6TuZ9pFt/R6dd/Q8Mn0G91vQElM7gUTx1fcEBaDzFI vZgovgZ4mNqU/G8qbQeTnkMklOn0tnAcbY3Ljyf7rjMZyjdWucd6tZ7qUyj9JoSCCuDO 3v2VG9wfSklYGoulu2MPUeIf5v69dVAjqgWvQAU2RoT5cec2GIgkaIVdj5V3c9jJuEa7 +ZAw== X-Gm-Message-State: AOUpUlHkHUn/9A4/DXkgNYu7GaV9xMhfPu+iHkO6Hd5NIC6Sia+2Pk0R BY1Zyrbn2niHkayp6DltzK2aYw== X-Google-Smtp-Source: AAOMgpd7afGxNHcxaIl5X+3K8/dBtOAaQ5cby5YtLUFp5hurhAUyaEVK+mWtG98rv7pwa0SDKOQO7g== X-Received: by 2002:a1c:d892:: with SMTP id p140-v6mr4386805wmg.76.1532509663675; Wed, 25 Jul 2018 02:07:43 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id b22-v6sm3960103wme.48.2018.07.25.02.07.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jul 2018 02:07:42 -0700 (PDT) Date: Wed, 25 Jul 2018 11:07:26 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: Thomas Monjalon , =?utf-8?B?TsOpbGlv?= Laranjeiro , "dev@dpdk.org" , Yongseok Koh Message-ID: <20180725090726.GV5211@6wind.com> References: <14690e825609ee181e3cd522302d4788ef436f35.1532424524.git.nelio.laranjeiro@6wind.com> <20180724125556.GS5211@6wind.com> <20180724151349.GT5211@6wind.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Subject: Re: [dpdk-dev] [PATCH] mk: fix application compilation with lmnl and mlx5 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jul 2018 09:07:44 -0000 On Wed, Jul 25, 2018 at 06:39:32AM +0000, Shahaf Shuler wrote: > Tuesday, July 24, 2018 6:14 PM, Adrien Mazarguil: > > Subject: Re: [PATCH] mk: fix application compilation with lmnl and mlx5 > > On Tue, Jul 24, 2018 at 01:03:28PM +0000, Shahaf Shuler wrote: > > > Tuesday, July 24, 2018 3:56 PM, Adrien Mazarguil: > > > > Subject: Re: [PATCH] mk: fix application compilation with lmnl and > > > > mlx5 > > [...] > > > > > > Can we consider different options: > > > > > 1. always statically link libmnl > > > > > 2. dlopen libmnl just like we do for verbs/mlx5 > > > > > > > > Regarding 2, unlike rdma-core/MLNX_OFED, libmnl should be available > > > > pretty much everywhere iproute2 can be found. The minimal version > > > > supported > > > > (1.0.3) was released in 2012. > > > > > > > > Using the glue approach for such a small library seems overkill; > > > > should we choose this path, we must also consider to get rid of it > > > > entirely since doing so would require more glue code than what mlx5 > > needs from this library. > > > > > > > > So with the current approach, either the application or the PMD > > > > inherits a dependency to libmnl, depending on whether > > > > CONFIG_RTE_BUILD_SHARED_LIB is respectively disabled or enabled. > > > > > > > > If disabled, applications that want static linkage can specify > > > > -static as part of their compilation flags to let the compiler > > > > automatically look for libmnl.a as needed. > > > > > > Can you confirm static compilation w/ libmnl is working w/o any issues? > > > > Challenge accepted. > > 😊 > > Using -static is not something DPDK applications usually > > do and needs a few tweaks to compile successfully though (unless you > > meant something else by "static compilation"?) > > Yes this is what I meant. However I was under the impression we can statically link to libmnl while keeping the rdma-core w/ dynamic linkage. Hmm, a much simpler approach if you don't actually want to make the whole binary static but only get rid of the libmnl.so run-time dependency is to replace "-lmnl" with e.g. "-l:libmnl.a". Doing so makes it part of the resulting binaries like any other DPDK libraries (librte_whatever.a). This can also be achieved through a linker script or by simply removing the .so version from the library search path. However while users are free to compile DPDK however they want, we shouldn't hard-code it this way since libmnl will eventually have multiple users in DPDK, this will lead to inefficient symbol duplication and possible link-time issues. > > First you need to force rdma-core to generate static versions of > > libmlx4/libmlx5 and libiberbs as it's not the default: > > > > $ cmake -DENABLE_STATIC=1 . > > > > Next, patch DPDK to remove references to -lgcc_s, -fPIC and -export- > > dynamic: > > > > $ sed -i 's/[[:space:]]*-lgcc_s//' $(git grep -l -- -lgcc_s) $ sed -i 's/[[:space:]]*- > > fPIC//' $(git grep -l -- -fPIC) $ sed -i 's/[[:space:]]*-export-dynamic//' $(git > > grep -l -- -export-dynamic) > > It seems the fPIC and export-dynamic are defined only when compiling the DPDK as shared library. > The gcc_s is from eal. Do you think it is correct to put it only when compiling as shared library? It seems this approach was taken in the rte.vars.mk files. Removing these flags is necessary even with CONFIG_RTE_BUILD_SHARED_LIB disabled in order to use -static. They appear to be present regardless. Besides -fPIC, they usually do not need to be specified when generating shared libraries. I'm almost 100% sure we could get rid of -lgcc_s without any impact. A bit less sure about -export-dynamic, however we now use map files to export public symbols. Someone should try. -fPIC can be kept when generating .a libraries by the way. It's a bit less efficient when the resulting file is linked with a program but shouldn't hurt much. It remains mandatory if the resulting library is to be made part of a shared library. We should keep it just in case. > > Then append rdma-core dependencies to libmnl users (mlx5) in order to > > avoid undefined references to libnl/libm stuff normally pulled by libibverbs: > > > > $ sed -i 's/-lmnl/-lmnl -lnl-route-3 -lnl-3 -lm/' $(git grep -l -- -lmnl) > > > > Configure DPDK with mlx5 enabled: > > > > $ make config T=x86_64-native-linuxapp-gcc $ sed -i > > 's/^\(CONFIG_RTE_LIBRTE_MLX5_PMD\)=.*/\1=y/' build/.config > > > > Finally add -static to $CC (the easiest method I could find) while compiling > > DPDK: > > > > $ make CC='gcc -static' > > > > You can ignore warnings about statically linking "getaddrinfo" and friends. > > What matters is we now have a testpmd binary with no dependencies: > > > > $ readelf -d build/app/testpmd | grep NEEDED $ > > > > $ ldd build/app/testpmd > > not a dynamic executable > > $ > > > > Now start testpmd with a mlx5 adapter and a couple of representors for fun: > > > > # ./build/app/testpmd -w 06:00.0,representor=[0-1] -n 4 -c 0x80 -- -i --rxq=1 > > --txq=1 > > EAL: Detected 32 lcore(s) > > EAL: Detected 2 NUMA nodes > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > > EAL: No free hugepages reported in hugepages-1048576kB > > EAL: Probing VFIO support... > > EAL: PCI device 0000:06:00.0 on NUMA socket 0 > > EAL: probe driver: 15b3:1017 net_mlx5 > > [...] > > Configuring Port 0 (socket 0) > > Port 0: 24:8A:07:8D:AE:F6 > > Configuring Port 1 (socket 0) > > Port 1: A6:41:D9:89:DB:12 > > Configuring Port 2 (socket 0) > > Port 2: FE:A8:15:80:DE:C0 > > Checking link statuses... > > Done > > testpmd> > > > > Phew! > > Nicely done. > > So if indeed it is possible w/ some fixes on the DPDK to fully support static linkage of both rdma-core and libmnl, maybe we should consider such compilation flag for mlx drivers. This can be very good alternative to the current DLOPEN approach. Although this is one way to generate statically linked DPDK applications, currently DPDK obviously doesn't support static linkage. I wouldn't recommend it. The approach of linking .a files instead of .so and still generate a dynamic program also has drawbacks: duplication and link time issues with multiple users. For instance, this may cause mlx4 and mlx5 to conflict on libibverbs and libmnl symbols, leaving one unable to include both PMDs into their program or prevent it from linking with these libraries for its own needs. I don't think it's worth the trouble. > > > To put this in perspective, this also applies to all other > > > dependencies > > > > it will collect while compiling DPDK (libz, libdl, libpcap, libnuma > > > > to name a few). > > > > > > > > In my opinion, the purpose of *_DLOPEN_DEPS is to deal with large, > > > > nonstandard libraries where versioning issues are commonplace. This > > > > doesn't apply to libmnl, which shouldn't be a maintenance nightmare > > > > to package maintainers. I suggest to leave things as is. -- Adrien Mazarguil 6WIND