[dpdk-users] dpdk-20.11 and shared libraries

[dpdk-users] dpdk-20.11 and shared libraries

[Templin (US), Fred L]

Hi, I have put a lot of time into learning about dpdk-20.11 but I have not yet found
documentation on how to use it with shared libraries.  In dpdk-20.11, the examples
are all built as "static" by default meaning that the entire kitchen sink of all DPDK
libraries are linked in during the "ld" phase and the resulting binary is huge. When
built as "static", the examples all appear to work fine as everything is loaded by
default at runtime.

When I build the examples as "shared", however, the story is completely different.
Examples that run fine as "static" do not work at all when built as "shared". It is
clear that somehow the build procedure is not giving adequate instructions to the
loader so that all necessary libraries will be ready to go at runtime.

I believe what needs to happen is that initialization code needs to be added to the
example "main.c" modules to initialize functions that will be needed by lower levels
but that are not being picked up on by the loader. But, I can find no documentation
for how to do this. In my next message, I will send a code example of what I have
tried so far.

Stephen, can you provide guidance on how to work with dpdk-20.11 in shared
library environments? I would be happy to be pointed to any documentation
I may be missing.

Thanks - Fred

Re: [dpdk-users] dpdk-20.11 and shared libraries

[Templin (US), Fred L]

Hi, as an example of where I am having trouble with shared libraries, I am
trying to modify the "helloworld" example program so that I can invoke it
as: "helloworld --vdev net_null0". Working by looking at the low-level
dpdk-20.11 driver code, I tried adding a call to "rte_vdev_init()" to the
helloworld main.c as shown in the code diffs below. When I tried to
build (as shared), I got an error from the loader saying the reference
to 'rte_vdev_init' is undefined.

Am I even barking up the right tree?

Fred

--- main.c	2021-04-20 08:17:37.895698070 -0700
+++ main.c.net_null	2021-04-20 08:24:51.958198682 -0700
@@ -14,6 +14,9 @@
 #include <rte_per_lcore.h>
 #include <rte_lcore.h>
 #include <rte_debug.h>
+#if 1
+#include <rte_bus_vdev.h>
+#endif
 
 static int
 lcore_hello(__rte_unused void *arg)
@@ -30,6 +33,12 @@
 	int ret;
 	unsigned lcore_id;
 
+#if 1
+	ret = rte_vdev_init("net_null", NULL);
+	if (ret < 0)
+		rte_panic("Cannot init net_null\n");
+#endif
+
 	ret = rte_eal_init(argc, argv);
 	if (ret < 0)
 		rte_panic("Cannot init EAL\n");

cc -O3 -include rte_config.h -march=native -I/usr/local/include -DALLOW_EXPERIMENTAL_API main.c -o build/helloworld-shared  -L/usr/local/lib/x86_64-linux-gnu -Wl,--as-needed -lrte_node -lrte_graph -lrte_bpf -lrte_flow_classify -lrte_pipeline -lrte_table -lrte_port -lrte_fib -lrte_ipsec -lrte_vhost -lrte_stack -lrte_security -lrte_sched -lrte_reorder -lrte_rib -lrte_regexdev -lrte_rawdev -lrte_pdump -lrte_power -lrte_member -lrte_lpm -lrte_latencystats -lrte_kni -lrte_jobstats -lrte_ip_frag -lrte_gso -lrte_gro -lrte_eventdev -lrte_efd -lrte_distributor -lrte_cryptodev -lrte_compressdev -lrte_cfgfile -lrte_bitratestats -lrte_bbdev -lrte_acl -lrte_timer -lrte_hash -lrte_metrics -lrte_cmdline -lrte_pci -lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool -lrte_rcu -lrte_ring -lrte_eal -lrte_telemetry -lrte_kvargs
/tmp/ccgU3Rpi.o: In function `main':
main.c:(.text.startup+0x15): undefined reference to `rte_vdev_init'
collect2: error: ld returned 1 exit status
Makefile:32: recipe for target 'build/helloworld-shared' failed
make: *** [build/helloworld-shared] Error 1

> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Templin (US), Fred L
> Sent: Tuesday, April 20, 2021 9:25 AM
> To: users@dpdk.org; Stephen Hemminger <stephen@networkplumber.org>
> Subject: [dpdk-users] dpdk-20.11 and shared libraries
> 
> Hi, I have put a lot of time into learning about dpdk-20.11 but I have not yet found
> documentation on how to use it with shared libraries.  In dpdk-20.11, the examples
> are all built as "static" by default meaning that the entire kitchen sink of all DPDK
> libraries are linked in during the "ld" phase and the resulting binary is huge. When
> built as "static", the examples all appear to work fine as everything is loaded by
> default at runtime.
> 
> When I build the examples as "shared", however, the story is completely different.
> Examples that run fine as "static" do not work at all when built as "shared". It is
> clear that somehow the build procedure is not giving adequate instructions to the
> loader so that all necessary libraries will be ready to go at runtime.
> 
> I believe what needs to happen is that initialization code needs to be added to the
> example "main.c" modules to initialize functions that will be needed by lower levels
> but that are not being picked up on by the loader. But, I can find no documentation
> for how to do this. In my next message, I will send a code example of what I have
> tried so far.
> 
> Stephen, can you provide guidance on how to work with dpdk-20.11 in shared
> library environments? I would be happy to be pointed to any documentation
> I may be missing.
> 
> Thanks - Fred
> 

Re: [dpdk-users] dpdk-20.11 and shared libraries

[David Marchand]

On Tue, Apr 20, 2021 at 6:25 PM Templin (US), Fred L
<Fred.L.Templin@boeing.com> wrote:
>
> Hi, I have put a lot of time into learning about dpdk-20.11 but I have not yet found
> documentation on how to use it with shared libraries.  In dpdk-20.11, the examples
> are all built as "static" by default meaning that the entire kitchen sink of all DPDK
> libraries are linked in during the "ld" phase and the resulting binary is huge. When
> built as "static", the examples all appear to work fine as everything is loaded by
> default at runtime.
>
> When I build the examples as "shared", however, the story is completely different.
> Examples that run fine as "static" do not work at all when built as "shared". It is
> clear that somehow the build procedure is not giving adequate instructions to the
> loader so that all necessary libraries will be ready to go at runtime.
>
> I believe what needs to happen is that initialization code needs to be added to the
> example "main.c" modules to initialize functions that will be needed by lower levels
> but that are not being picked up on by the loader. But, I can find no documentation
> for how to do this. In my next message, I will send a code example of what I have
> tried so far.

I suppose you are testing with a local build (i.e. a dpdk not
installed system wide).

Example with my build env:
$ $HOME/builds/build-gcc-shared/examples/dpdk-helloworld --no-huge -m
512 --vdev net_null0
EAL: Detected 28 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Detected shared linkage of DPDK
EAL: failed to parse device "net_null0"
EAL: Unable to parse device 'net_null0'
PANIC in main():
Cannot init EAL
5: [/home/dmarchan/builds/build-gcc-shared/examples/dpdk-helloworld()
[0x40116e]]
4: [/usr/lib64/libc.so.6(__libc_start_main+0xf2) [0x7f7ece854082]]
3: [/home/dmarchan/builds/build-gcc-shared/examples/dpdk-helloworld()
[0x4010b6]]
2: [/home/dmarchan/builds/build-gcc-shared/examples/../lib/librte_eal.so.21(__rte_panic+0xbe)
[0x7f7eceb828cc]]
1: [/home/dmarchan/builds/build-gcc-shared/examples/../lib/librte_eal.so.21(rte_dump_stack+0x1b)
[0x7f7eceba26cb]]
Aborted (core dumped)

You have several solutions.
You can simply install dpdk system wide.

Or, try to pass -d <your_driver.so> in EAL cmdline options.
$ LD_LIBRARY_PATH=$HOME/builds/build-gcc-shared/install/usr/local/lib64
$HOME/builds/build-gcc-shared/examples/dpdk-helloworld --no-huge -m
512 -d $HOME/builds/build-gcc-shared/install/usr/local/lib64/dpdk/pmds-21.2/librte_net_null.so
--vdev net_null0
EAL: Detected 28 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /run/user/1001/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
hello from core 1
hello from core 2
hello from core 3
hello from core 4
hello from core 5
hello from core 6
hello from core 7
hello from core 8
hello from core 9
hello from core 10
hello from core 11
hello from core 12
hello from core 13
hello from core 14
hello from core 15
hello from core 16
hello from core 17
hello from core 18
hello from core 19
hello from core 20
hello from core 21
hello from core 22
hello from core 23
hello from core 24
hello from core 25
hello from core 26
hello from core 27
hello from core 0


Or you can also pass a directory to this option and EAL will try to
load any driver in this directory.


-- 
David Marchand