[dpdk-dev] Did we reduce unnecessary linkage too well?

[dpdk-dev] Did we reduce unnecessary linkage too well?

[Christian Ehrhardt]

Hi,
I was finally getting to more deeply re-validate Openvswitch 2.6 together
with DPDK 16.07. And I think I found a whiplash of our effort to reduce
unnecessary hard linkage.

Trying to avoid cross-posting, picking DPDK list and the main involved
people on TO/CC.

TL;DR:
- pmd drivers are no more "auto"-loaded
- adding -d ...so to all consuming applications feels obnoxious
- do we really have to intentionally overlink some?
- I hope I just overlook something trivial to fix this.


Detail:

Essentially in the past openvswitch was linked to "the one" combined
library being libdpdk.so.
Due to all the great work on separating the libraries for versioning and
reducing overlinking I now have an openvswitch that only depends on the
core dpdk libs.

$ldd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk | grep rte
       librte_pdump.so.1 => /usr/lib/x86_64-linux-gnu/librte_pdump.so.1
       librte_pmd_ring.so.2 =>
/usr/lib/x86_64-linux-gnu/librte_pmd_ring.so.2
       librte_mbuf.so.2 => /usr/lib/x86_64-linux-gnu/librte_mbuf.so.2
       librte_vhost.so.3 => /usr/lib/x86_64-linux-gnu/librte_vhost.so.3
       librte_mempool.so.2 => /usr/lib/x86_64-linux-gnu/librte_mempool.so.2
       librte_meter.so.1 => /usr/lib/x86_64-linux-gnu/librte_meter.so.1
       librte_ring.so.1 => /usr/lib/x86_64-linux-gnu/librte_ring.so.1
       librte_eal.so.2 => /usr/lib/x86_64-linux-gnu/librte_eal.so.2
       librte_kvargs.so.1 => /usr/lib/x86_64-linux-gnu/librte_kvargs.so.1

As we all know that is due to the build of openvswitch now utilizing the
linker script which is
$cat /usr/lib/x86_64-linux-gnu/libdpdk.so
GROUP ( librte_power.so librte_pmd_null_crypto.so librte_ip_frag.so
librte_pmd_e1000.so librte_acl.so librte_port.so librte_mbuf.so
librte_pmd_virtio.so librte_kvargs.so li
brte_cryptodev.so librte_reorder.so librte_pmd_ring.so librte_eal.so
librte_distributor.so librte_pmd_pcap.so librte_pmd_af_packet.so
librte_pipeline.so librte_pdump.so lib
rte_pmd_vhost.so librte_jobstats.so librte_pmd_enic.so librte_mempool.so
librte_pmd_ixgbe.so librte_cmdline.so librte_meter.so librte_timer.so
librte_kni.so librte_pmd_vmxn
et3_uio.so librte_ring.so librte_pmd_fm10k.so librte_cfgfile.so
librte_pmd_xenvirt.so librte_pmd_i40e.so librte_pmd_null.so librte_lpm.so
librte_pmd_ena.so librte_vhost.so
libethdev.so librte_pmd_bnxt.so librte_sched.so librte_table.so
librte_pmd_bond.so librte_hash.so librte_pmd_cxgbe.so )

You see it lists the PMDs, but since this is more a plugin structure there
are no references by openvswitch to them. So they don't make it into the
eventual binary.
Which is kind of what we wanted - avoid over-linking right?

But this leads to IMHO unexpected behaviour. For example I have a system
properly prepared with two ixgbe cards and start OVS-DPDK in "the right
way" I get the issue of not having a valid PMD driver.
"could not open network device dpdk0"
=> https://launchpadlibrarian.net/287083118/ovs-vswitchd-bad.log

Now this can work if one tells EAL to load the proper driver, so after
$ovs-vsctl set Open_vSwitch . "other_config:dpdk-extra=[...] -d
/usr/lib/x86_64-linux-gnu/librte_pmd_ixgbe.so"
Everything works fine now and as-expected.
=> https://launchpadlibrarian.net/287083132/ovs-vswitchd-good.log
One can see ixgbe pmd driver loading and taking over the devices.

I think this puts too much pressure to know the right drivers to the admin
- so we can't really make this the way it should be right?
It feels obnoxious having to add a full path to a .so file to a ovsdb
parameter.

It might be worth to note that I can reproduce the same behavior with
l2fwd, without the -d to the lib on the same systemit quits with:
EAL: Error - exiting with code: 1
  Cause: No Ethernet ports - bye
I considered this a test/developer tool where it is kind of ok.
But now I think it is time to discuss it in the scope of more
customer-centric dpdk consuming applications like openvswitch+dpdk.

The build logs of how the tested openvswitch and dpdk were build are here -
just in case one of you want/can spot an issue causing all this in there:

   - openvswitch (all of it, not just openvswitch-dpdk):
   https://launchpadlibrarian.net/284431032/buildlog_ubuntu-yakkety-amd64.openvswitch_2.6.0~git20160912.dc61b4e-0ubuntu4_BUILDING.txt.gz
   - dpdk-16.07:
   https://launchpadlibrarian.net/285321110/buildlog_ubuntu-yakkety-amd64.dpdk_16.07-0ubuntu3_BUILDING.txt.gz


I haven't seen nor found a discussion on this - nor people testing my
packages complainning so far -, but find it so weird that I start to think
it is just me or my broken setup :-)
So I guess there is hope that there is something obvious I overlook -
better that than needing major reworks.
Like a default search dir for the libs that I mismatch in my setup or
anything like it?
If not, should we do something - I hate to say that - to intentionally
overlink dpdk libraries to get all pre-provided PMDs autoloaded?

Kind Regards,

-- 
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

Re: [dpdk-dev] Did we reduce unnecessary linkage too well?

[Panu Matilainen]

On 09/29/2016 09:58 AM, Christian Ehrhardt wrote:
> Hi,
> I was finally getting to more deeply re-validate Openvswitch 2.6 together
> with DPDK 16.07. And I think I found a whiplash of our effort to reduce
> unnecessary hard linkage.
>
> Trying to avoid cross-posting, picking DPDK list and the main involved
> people on TO/CC.
>
> TL;DR:
> - pmd drivers are no more "auto"-loaded
> - adding -d ...so to all consuming applications feels obnoxious
> - do we really have to intentionally overlink some?
> - I hope I just overlook something trivial to fix this.

Yup. Set CONFIG_RTE_EAL_PMD_PATH to the path where your PMDs are 
installed. Note that since the plugin autoloader in DPDK doesn't make 
assumptions about names, it'll try to load *everything* in that path, so 
you don't want it pointing to eg /usr/lib directly.

What we have on Fedora and RHEL is a /usr/lib(64)/dpdk-pmds/ directory 
with symlinks to the actual pmds which reside in /usr/lib(64) because 
some of them provide actual API.

	- Panu -

Re: [dpdk-dev] Did we reduce unnecessary linkage too well?

[Christian Ehrhardt]

On Thu, Sep 29, 2016 at 9:20 AM, Panu Matilainen <pmatilai@redhat.com>
wrote:

>
> Yup. Set CONFIG_RTE_EAL_PMD_PATH to the path where your PMDs are
> installed. Note that since the plugin autoloader in DPDK doesn't make
> assumptions about names, it'll try to load *everything* in that path, so
> you don't want it pointing to eg /usr/lib directly.
>
> What we have on Fedora and RHEL is a /usr/lib(64)/dpdk-pmds/ directory
> with symlinks to the actual pmds which reside in /usr/lib(64) because some
> of them provide actual API.


Great, thanks Panu!
Will create something around that now.
The symlink is about what came to my mind before reading your second
sentence - so good with that.


-- 
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

Re: [dpdk-dev] Did we reduce unnecessary linkage too well?

[Bruce Richardson]

On Thu, Sep 29, 2016 at 09:26:48AM +0200, Christian Ehrhardt wrote:
> On Thu, Sep 29, 2016 at 9:20 AM, Panu Matilainen <pmatilai@redhat.com>
> wrote:
> 
> >
> > Yup. Set CONFIG_RTE_EAL_PMD_PATH to the path where your PMDs are
> > installed. Note that since the plugin autoloader in DPDK doesn't make
> > assumptions about names, it'll try to load *everything* in that path, so
> > you don't want it pointing to eg /usr/lib directly.

Is this something we should look to change? To me having some sort of
naming convention might not be a bad thing, so that we can point it at generic
folders.

/Bruce

Re: [dpdk-dev] Did we reduce unnecessary linkage too well?

[Christian Ehrhardt]

On Fri, Sep 30, 2016 at 12:15 PM, Bruce Richardson <
bruce.richardson@intel.com> wrote:

> To me having some sort of
> naming convention might not be a bad thing, so that we can point it at
> generic
> folders.
>

Distributions don't really agree on the prefix path for the libraries
/usr/lib<changesstarthere>.
So I'd expect that it is hard for someone out there whatever you pick :-/
That said I'd not pick any default by dpdk.org.

In my case I thin I just "left" the linkage discussions too early and by
that forgot to implement RTE_EAL_PMD_PATH which I now did.

If there are no PMDs at all or no matter how many PMDs where there it can't
probe something the error messages might point a bit more towards
the RTE_EAL_PMD_PATH.
E.g. no drivers found, please make sure you have PMDs for your network
devices in <where RTE_EAL_PMD_PATH points to>
And if RTE_EAL_PMD_PATH is "" (=disabled) let them know that as well.
And finally one might suggest trying with EAL option -d.
To then give up and die - just with more words before :-)
That might help.


-- 
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

Re: [dpdk-dev] Did we reduce unnecessary linkage too well?

[Panu Matilainen]

On 09/30/2016 01:15 PM, Bruce Richardson wrote:
> On Thu, Sep 29, 2016 at 09:26:48AM +0200, Christian Ehrhardt wrote:
>> On Thu, Sep 29, 2016 at 9:20 AM, Panu Matilainen <pmatilai@redhat.com>
>> wrote:
>>
>>>
>>> Yup. Set CONFIG_RTE_EAL_PMD_PATH to the path where your PMDs are
>>> installed. Note that since the plugin autoloader in DPDK doesn't make
>>> assumptions about names, it'll try to load *everything* in that path, so
>>> you don't want it pointing to eg /usr/lib directly.
>
> Is this something we should look to change? To me having some sort of
> naming convention might not be a bad thing, so that we can point it at generic
> folders.

Plugins for program/library X are nearly always in a sub-directory of 
their own, outside the linker path because ... well, they're plugins and 
not something you should link to, and having them in separate 
directories makes it possible to have multiple versions co-exist on the 
system by simply placing the plugins into a versioned directory.
That's why the current plugin autoloader is the way it is - it's the 
de-facto standard for dealing with plugins.

The DPDK case is a bit convoluted since some of the alleged plugins also 
provide library APIs, so at least those DSOs need to be present in 
linker paths. Plugins usually also lack soname version, because that 
doesn't make much sense for plugins, libraries with APIs are different 
there too.

Anyway, naming conventions are flimsy and fall apart in situations that 
the directory approach easily handles. So I dont see a point in changing 
that. What *would* be good is creating and populating that directory 
from DPDK "make install" step automatically, at least when 
RTE_EAL_PMD_PATH is set.

	- Panu -






>
> /Bruce
>