[dpdk-users] DPDK 16.04 link changes cause PMD drivers to not be loaded

[dpdk-users] DPDK 16.04 link changes cause PMD drivers to not be loaded

[Aurojit Panda]

Hi
I was trying to run testpmd from DPDK 16.04 on Linux with kernel version 4.4.0-1 (ld version 2.26). My machine has two 
XL710QDA2 NICs, and I built DPDK as a shared, combined library (i.e., CONFIG_RTE_BUILD_SHARED_LIB=y and 
CONFIG_RTE_BUILD_COMBINE_LIBS=y in config/common_linuxapp). I found that the issue is due to ld not linking against all 
libraries with the new linker script (introduced in 948fd64befc3726) but am not sure how to fix this. As evidence for 
this being caused by this change:

$ LD_TRACE_LOADED_OBJECTS=y ./testpmd -c 0x1c00 -n 4 -w 82:00.0 -w 82:00.1 --file-prefix "send" -- -i
         linux-vdso.so.1 (0x00007ffe11b3e000)
         librte_distributor.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_distributor.so.1.1 
(0x00007fa55eb85000)
         librte_reorder.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_reorder.so.1.1 (0x00007fa55e982000)
         librte_pipeline.so.3.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_pipeline.so.3.1 (0x00007fa55e77d000)
         librte_table.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_table.so.2.1 (0x00007fa55e55e000)
         librte_port.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_port.so.2.1 (0x00007fa55e34e000)
         librte_timer.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_timer.so.1.1 (0x00007fa55e145000)
         librte_hash.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_hash.so.2.1 (0x00007fa55df37000)
         librte_jobstats.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_jobstats.so.1.1 (0x00007fa55dd35000)
         librte_lpm.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_lpm.so.2.1 (0x00007fa55db2e000)
         librte_power.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_power.so.1.1 (0x00007fa55d91f000)
         librte_acl.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_acl.so.2.1 (0x00007fa55d705000)
         librte_meter.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_meter.so.1.1 (0x00007fa55d504000)
         librte_sched.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_sched.so.1.1 (0x00007fa55d2fd000)
         librte_vhost.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_vhost.so.2.1 (0x00007fa55d0e6000)
         librte_kvargs.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_kvargs.so.1.1 (0x00007fa55cee4000)
         librte_mbuf.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_mbuf.so.2.1 (0x00007fa55cce2000)
         librte_ip_frag.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_ip_frag.so.1.1 (0x00007fa55cada000)
         libethdev.so.3.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/libethdev.so.3.1 (0x00007fa55c84f000)
         librte_cryptodev.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_cryptodev.so.1.1 (0x00007fa55c647000)
         librte_mempool.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_mempool.so.1.1 (0x00007fa55c444000)
         librte_ring.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_ring.so.1.1 (0x00007fa55c242000)
         librte_eal.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_eal.so.2.1 (0x00007fa55bfe1000)
         librte_cmdline.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_cmdline.so.2.1 (0x00007fa55bdd8000)
         librte_cfgfile.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_cfgfile.so.2.1 (0x00007fa55bbd6000)
         librte_pmd_bond.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_pmd_bond.so.1.1 (0x00007fa55b9c4000)
         libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fa55b7ae000)
         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa55b5aa000)
         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa55b38d000)
         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa55afe9000)
         /lib64/ld-linux-x86-64.so.2 (0x00007fa55ed87000)
         libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa55aceb000)
         librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fa55aae3000)


As you can see librte_pmd_i40e.so is missing from the list of files. Furthermore, using

$ LD_PRELOAD="../lib/librte_pmd_i40e.so" ./testpmd -c 0x1c00 -n 4 -w 82:00.0 -w 82:00.1 --file-prefix "send" -- -i 
correctly enumerates the NICs.

I unfortunately do not how to fix the linker script as checked in, but wanted to report this bug in case someone has a fix.

Thanks

Panda

Re: [dpdk-users] DPDK 16.04 link changes cause PMD drivers to not be loaded

[Aurojit Panda]

[Cross-posting to dev]

Panu Matilainen wrote:
> On 04/20/2016 05:26 PM, Aurojit Panda wrote:
>> I am sorry that is a bit unintuitive considering:
>>
>> (a) This behavior differs between static and shared builds of DPDK.
>> - In fact this behavior was identical in 2.2, and even in mainline
>> before 948fd64befc3726 went in.
>> (b) You already know the EAL_PMD_PATH at build time, and it makes it
>> quite hard to ship scripts or anything to build DPDK, since now the
>> configuration file becomes dependent on location.
>
> The expectation is that shared libraries are installed to a shared, known location. Otherwise there's not that much
> point in using shared libraries.
>
> Note that you can also use the EAL option -d to load either single PMDs or entire directories at runtime (regardless of
> EAL_PMD_PATH) so you dont have to know the path at build time if that's what bothers you most.
>

I am actually bothered by needing to supply the path at all. You seem to assume that knowing an install path is 
essential for shared libraries, while you can in reality embed this in the rpath. Shared libraries are used for many 
reasons, including interop with some runtime, in which case they are not installed at a known place. I would prefer for 
those PMDs that are included with DPDK that either they be loaded from the "known" build path for them.

>>
>> I wonder if you would consider changing this, as it stands just building
>> DPDK after setting CONFIG_RTE_SHARED_LIB=y results in a testpmd that
>> cannot run.
>
> In shared library configuration, testpmd is not directly runnable regardless of the drivers since its missing all the
> other libraries too:
>
> [pmatilai@sopuli dpdk]$ build/app/testpmd
> build/app/testpmd: error while loading shared libraries: librte_distributor.so.1.1: cannot open shared object file: No
> such file or directory
> [pmatilai@sopuli dpdk]$
>
> You'll need to get those libraries into linkers path anyway, either by installing them to a common location or by
> extending LD_LIBRARY_PATH. In either case, you need to know where the libraries are anyway.
>
> All of which is not to say there might not be room for improvement, but the linking behavior is not going to change.
> Again, PMDs are *plugins* that are *meant* to be loaded at runtime. That allows for all sorts of flexibility especially
> for packaging and shipping, at some extra cost in setup complexity.

I am all for a plugin architecture, I was merely suggesting that you embed some path infromation at the beginning. Also 
please note:
(a) This behavior changed recently.
(b) This change is entirely undocumented, which is why I was reporting it in the first place.
(c) It is actually quite unintutive, because previously ensuring LD_LIBRARY_PATH was correct was all that was required 
to get any DPDK application to interact with ports.

>
> For your own purposes, you can of course tweak the linking settings as much as you like. Look for "plugins" in
> mk/rte.app.mk and change the shared lib condition on the line above to "y" and there you have it. But that's not the way
> plugins are meant to be used.

That is not a reasonable solution given that it makes it very hard to track future changes to DPDK without merges. My 
alternatives neither break people's abilities to use plugins, nor do they impact behavior.

>
> Oh and BTW, please don't top-post.
>
> - Panu -
>

Re: [dpdk-users] DPDK 16.04 link changes cause PMD drivers to not be loaded

[Panu Matilainen]

On 04/20/2016 05:26 PM, Aurojit Panda wrote:
> I am sorry that is a bit unintuitive considering:
>
> (a) This behavior differs between static and shared builds of DPDK.
>      - In fact this behavior was identical in 2.2, and even in mainline
> before 948fd64befc3726 went in.
> (b) You already know the EAL_PMD_PATH at build time, and it makes it
> quite hard to ship scripts or anything to build DPDK, since now the
> configuration file becomes dependent on location.

The expectation is that shared libraries are installed to a shared, 
known location. Otherwise there's not that much point in using shared 
libraries.

Note that you can also use the EAL option -d to load either single PMDs 
or entire directories at runtime (regardless of EAL_PMD_PATH) so you 
dont have to know the path at build time if that's what bothers you most.

>
> I wonder if you would consider changing this, as it stands just building
> DPDK after setting CONFIG_RTE_SHARED_LIB=y results in a testpmd that
> cannot run.

In shared library configuration, testpmd is not directly runnable 
regardless of the drivers since its missing all the other libraries too:

[pmatilai@sopuli dpdk]$ build/app/testpmd
build/app/testpmd: error while loading shared libraries: 
librte_distributor.so.1.1: cannot open shared object file: No such file 
or directory
[pmatilai@sopuli dpdk]$

You'll need to get those libraries into linkers path anyway, either by 
installing them to a common location or by extending LD_LIBRARY_PATH. In 
either case, you need to know where the libraries are anyway.

All of which is not to say there might not be room for improvement, but 
the linking behavior is not going to change. Again, PMDs are *plugins* 
that are *meant* to be loaded at runtime. That allows for all sorts of 
flexibility especially for packaging and shipping, at some extra cost in 
setup complexity.

For your own purposes, you can of course tweak the linking settings as 
much as you like. Look for "plugins" in mk/rte.app.mk and change the 
shared lib condition on the line above to "y" and there you have it. But 
that's not the way plugins are meant to be used.

Oh and BTW, please don't top-post.

	- Panu -

Re: [dpdk-users] DPDK 16.04 link changes cause PMD drivers to not be loaded

[Aurojit Panda]

I am sorry that is a bit unintuitive considering:

(a) This behavior differs between static and shared builds of DPDK.
     - In fact this behavior was identical in 2.2, and even in mainline before 948fd64befc3726 went in.
(b) You already know the EAL_PMD_PATH at build time, and it makes it quite hard to ship scripts or anything to build 
DPDK, since now the configuration file becomes dependent on location.

I wonder if you would consider changing this, as it stands just building DPDK after setting CONFIG_RTE_SHARED_LIB=y 
results in a testpmd that cannot run.

Thanks

Panda

Panu Matilainen wrote:
> On 04/19/2016 04:42 AM, Aurojit Panda wrote:
>> Hi
>> I was trying to run testpmd from DPDK 16.04 on Linux with kernel version
>> 4.4.0-1 (ld version 2.26). My machine has two
>> XL710QDA2 NICs, and I built DPDK as a shared, combined library (i.e.,
>> CONFIG_RTE_BUILD_SHARED_LIB=y and
>> CONFIG_RTE_BUILD_COMBINE_LIBS=y in config/common_linuxapp). I found that
>> the issue is due to ld not linking against all
>> libraries with the new linker script (introduced in 948fd64befc3726) but
>> am not sure how to fix this. As evidence for
>> this being caused by this change:
>>
> [...]
>> As you can see librte_pmd_i40e.so is missing from the list of files.
>> Furthermore, using
>>
>> $ LD_PRELOAD="../lib/librte_pmd_i40e.so" ./testpmd -c 0x1c00 -n 4 -w
>> 82:00.0 -w 82:00.1 --file-prefix "send" -- -i
>> correctly enumerates the NICs.
>>
>> I unfortunately do not how to fix the linker script as checked in, but
>> wanted to report this bug in case someone has a fix.
>
> The PMDs are plugins which are not really supposed to be linked in but loaded dynamically. Set CONFIG_RTE_EAL_PMD_PATH
> to the location of your drivers and they'll get loaded on DPDK initialization.
>
> - Panu -
>

Re: [dpdk-users] DPDK 16.04 link changes cause PMD drivers to not be loaded

[Panu Matilainen]

On 04/19/2016 04:42 AM, Aurojit Panda wrote:
> Hi
> I was trying to run testpmd from DPDK 16.04 on Linux with kernel version
> 4.4.0-1 (ld version 2.26). My machine has two
> XL710QDA2 NICs, and I built DPDK as a shared, combined library (i.e.,
> CONFIG_RTE_BUILD_SHARED_LIB=y and
> CONFIG_RTE_BUILD_COMBINE_LIBS=y in config/common_linuxapp). I found that
> the issue is due to ld not linking against all
> libraries with the new linker script (introduced in 948fd64befc3726) but
> am not sure how to fix this. As evidence for
> this being caused by this change:
>
[...]
> As you can see librte_pmd_i40e.so is missing from the list of files.
> Furthermore, using
>
> $ LD_PRELOAD="../lib/librte_pmd_i40e.so" ./testpmd -c 0x1c00 -n 4 -w
> 82:00.0 -w 82:00.1 --file-prefix "send" -- -i
> correctly enumerates the NICs.
>
> I unfortunately do not how to fix the linker script as checked in, but
> wanted to report this bug in case someone has a fix.

The PMDs are plugins which are not really supposed to be linked in but 
loaded dynamically. Set CONFIG_RTE_EAL_PMD_PATH to the location of your 
drivers and they'll get loaded on DPDK initialization.

	- Panu -

[dpdk-users] DPDK 16.04 link changes cause PMD drivers to not be loaded

[Aurojit Panda]

Hi
I was trying to run testpmd from DPDK 16.04 on Linux with kernel version 4.4.0-1 (ld version 2.26). My machine has two
XL710QDA2 NICs, and I built DPDK as a shared, combined library (i.e., CONFIG_RTE_BUILD_SHARED_LIB=y and
CONFIG_RTE_BUILD_COMBINE_LIBS=y in config/common_linuxapp). I found that the issue is due to ld not linking against all
libraries with the new linker script (introduced in 948fd64befc3726) but am not sure how to fix this. As evidence for
this being caused by this change:

$ LD_TRACE_LOADED_OBJECTS=y ./testpmd -c 0x1c00 -n 4 -w 82:00.0 -w 82:00.1 --file-prefix "send" -- -i
           linux-vdso.so.1 (0x00007ffe11b3e000)
           librte_distributor.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_distributor.so.1.1
(0x00007fa55eb85000)
           librte_reorder.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_reorder.so.1.1 (0x00007fa55e982000)
           librte_pipeline.so.3.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_pipeline.so.3.1 (0x00007fa55e77d000)
           librte_table.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_table.so.2.1 (0x00007fa55e55e000)
           librte_port.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_port.so.2.1 (0x00007fa55e34e000)
           librte_timer.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_timer.so.1.1 (0x00007fa55e145000)
           librte_hash.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_hash.so.2.1 (0x00007fa55df37000)
           librte_jobstats.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_jobstats.so.1.1 (0x00007fa55dd35000)
           librte_lpm.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_lpm.so.2.1 (0x00007fa55db2e000)
           librte_power.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_power.so.1.1 (0x00007fa55d91f000)
           librte_acl.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_acl.so.2.1 (0x00007fa55d705000)
           librte_meter.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_meter.so.1.1 (0x00007fa55d504000)
           librte_sched.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_sched.so.1.1 (0x00007fa55d2fd000)
           librte_vhost.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_vhost.so.2.1 (0x00007fa55d0e6000)
           librte_kvargs.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_kvargs.so.1.1 (0x00007fa55cee4000)
           librte_mbuf.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_mbuf.so.2.1 (0x00007fa55cce2000)
           librte_ip_frag.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_ip_frag.so.1.1 (0x00007fa55cada000)
           libethdev.so.3.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/libethdev.so.3.1 (0x00007fa55c84f000)
           librte_cryptodev.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_cryptodev.so.1.1 (0x00007fa55c647000)
           librte_mempool.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_mempool.so.1.1 (0x00007fa55c444000)
           librte_ring.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_ring.so.1.1 (0x00007fa55c242000)
           librte_eal.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_eal.so.2.1 (0x00007fa55bfe1000)
           librte_cmdline.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_cmdline.so.2.1 (0x00007fa55bdd8000)
           librte_cfgfile.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_cfgfile.so.2.1 (0x00007fa55bbd6000)
           librte_pmd_bond.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_pmd_bond.so.1.1 (0x00007fa55b9c4000)
           libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fa55b7ae000)
           libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa55b5aa000)
           libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa55b38d000)
           libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa55afe9000)
           /lib64/ld-linux-x86-64.so.2 (0x00007fa55ed87000)
           libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa55aceb000)
           librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fa55aae3000)


As you can see librte_pmd_i40e.so is missing from the list of files. Furthermore, using

$ LD_PRELOAD="../lib/librte_pmd_i40e.so" ./testpmd -c 0x1c00 -n 4 -w 82:00.0 -w 82:00.1 --file-prefix "send" -- -i
correctly enumerates the NICs.

I unfortunately do not how to fix the linker script as checked in, but wanted to report this bug in case someone has a fix.

Thanks

Panda