[dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud Images

[dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud Images

[Matthew Hall]

Hello,

I was working to get my open source project running in a VirtualBox Vagrant VM 
powered by an Ubuntu Cloud image the last few days to make my project and DPDK 
more developer friendly with a prebuilt environment. During this I was fixing 
the ugly hardcodings I'd used to hack it together on a desktop.

In the process I found a few things I was confused about.

1) Ubuntu doesn't include the UIO module by default in the Cloud image... only 
the virtio-net. I was wondering if we had anybody in good contact with their 
kernel group to lobby for inclusion of the UIO modules in the stock Cloud 
kernel, without having to grab linux-generic and linux-headers-generic first.

2) The directions for activating virtio-net based interfaces seem out-of-date. 
They refer to the PMD as rte_virtio_net_pmd, when the PMD calls itself 
rte_virtio_pmd in my copy of DPDK 1.7.1 w/ my clang compilation patches added.

I am getting some odd errors when I'm trying to load my app:

EAL: PCI device 0000:00:08.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
EAL:   0000:00:08.0 not managed by UIO driver, skipping

3) If I try to bind the device to the igb_uio driver even though it seems like 
the wrong thing to do, just for testing, this is what happens (NOTE: unbound 
the 00:08.0 device from the kernel to show this):

$ sudo tools/dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver
============================================
<none>

Network devices using kernel driver
===================================
0000:00:03.0 'Virtio network device' if= drv=virtio-pci unused=igb_uio

Other network devices
=====================
0000:00:08.0 'Virtio network device' unused=igb_uio

$ sudo tools/dpdk_nic_bind.py -b igb_uio 00:08.0
Error: bind failed for 0000:00:08.0 - Cannot bind to driver igb_uio

vagrant@vagrant-ubuntu-trusty-64:/vagrant/external/dpdk$ dmesg | tail
[ 1766.445609] igb_uio: Use MSIX interrupt by default
[ 1824.602075] igb_uio: probe of 0000:00:08.0 failed with error -2
[ 1824.602742] igb_uio: probe of 0000:00:08.0 failed with error -2

4) I found some old email threads asking about this:

http://comments.gmane.org/gmane.comp.networking.dpdk.devel/1357
(there are some others as well but this seemed closest)

But the only thing present in that thread seemed to be irritated replies which 
didn't really explain the different virtio PMD's out there and how they work, 
and didn't explain which ones were in-tree and out-of-tree either.

So let me ask this again, when somebody wrote "virtio-net or virtio-net + uio 
(QEMU, VirtualBox)" into the supported page (http://dpdk.org/doc/nics), who 
tested this to prove it worked? How did they get it to work on VirtualBox? The 
last reply stating "you have the source code" didn't really explain how they 
proved this stuff ever worked in the first place.

Thanks,
Matthew.

Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud Images

[Matthew Hall]

Another problem regarding virtio-net-pmd. When I tried using virtio-net-pmd, 
it compiles fine but then hits a weird error also during EAL init process:

EAL: open shared lib /vagrant/external/virtio-net-pmd/librte_pmd_virtio.so
EAL: /vagrant/external/virtio-net-pmd/librte_pmd_virtio.so: undefined symbol: per_lcore__lcore_id

It doesn't seem to have a link dependency against any DPDK library that might 
contain such a symbol, either:

$ ldd librte_pmd_virtio.so
    linux-vdso.so.1 =>  (0x00007fffd61fc000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa2d971f000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa2d9d00000)

$ nm librte_pmd_virtio.so | fgrep -i lcore
                 U per_lcore__lcore_id
00000000000014ee t rte_lcore_id

man nm says this means: "U" The symbol is undefined.

At present I am using the common setup, static DPDK w/ COMBINE_LIBS. The 
directions don't state that a shared lib DPDK is required, and if it is 
required, that increases complexity and reduces performance so it'd be better 
not to be forced to require this unless there's a good reason.

Instead I think it would be good if this driver didn't have to be external so 
it would just work right by default as part of the normal DPDK... right now it 
seems like it's a lot harder to set it up than it really needs to be.

Another weird issue... when I tried to compile a DPDK shared lib using clang I 
got this really, really weird error:

/usr/bin/ld: test: hidden symbol `mknod' in /usr/lib/x86_64-linux-gnu/libc_nonshared.a(mknod.oS) is referenced by DSO
/usr/bin/ld: final link failed: Bad value

Did anybody ever see something this before?

Thanks,
Matthew.

On Mon, Oct 13, 2014 at 10:45:23PM -0700, Matthew Hall wrote:
> Hello,
> 
> I was working to get my open source project running in a VirtualBox Vagrant VM 
> powered by an Ubuntu Cloud image the last few days to make my project and DPDK 
> more developer friendly with a prebuilt environment. During this I was fixing 
> the ugly hardcodings I'd used to hack it together on a desktop.
> 
> In the process I found a few things I was confused about.
> 
> 1) Ubuntu doesn't include the UIO module by default in the Cloud image... only 
> the virtio-net. I was wondering if we had anybody in good contact with their 
> kernel group to lobby for inclusion of the UIO modules in the stock Cloud 
> kernel, without having to grab linux-generic and linux-headers-generic first.
> 
> 2) The directions for activating virtio-net based interfaces seem out-of-date. 
> They refer to the PMD as rte_virtio_net_pmd, when the PMD calls itself 
> rte_virtio_pmd in my copy of DPDK 1.7.1 w/ my clang compilation patches added.
> 
> I am getting some odd errors when I'm trying to load my app:
> 
> EAL: PCI device 0000:00:08.0 on NUMA socket -1
> EAL:   probe driver: 1af4:1000 rte_virtio_pmd
> EAL:   0000:00:08.0 not managed by UIO driver, skipping
> 
> 3) If I try to bind the device to the igb_uio driver even though it seems like 
> the wrong thing to do, just for testing, this is what happens (NOTE: unbound 
> the 00:08.0 device from the kernel to show this):
> 
> $ sudo tools/dpdk_nic_bind.py --status
> 
> Network devices using DPDK-compatible driver
> ============================================
> <none>
> 
> Network devices using kernel driver
> ===================================
> 0000:00:03.0 'Virtio network device' if= drv=virtio-pci unused=igb_uio
> 
> Other network devices
> =====================
> 0000:00:08.0 'Virtio network device' unused=igb_uio
> 
> $ sudo tools/dpdk_nic_bind.py -b igb_uio 00:08.0
> Error: bind failed for 0000:00:08.0 - Cannot bind to driver igb_uio
> 
> vagrant@vagrant-ubuntu-trusty-64:/vagrant/external/dpdk$ dmesg | tail
> [ 1766.445609] igb_uio: Use MSIX interrupt by default
> [ 1824.602075] igb_uio: probe of 0000:00:08.0 failed with error -2
> [ 1824.602742] igb_uio: probe of 0000:00:08.0 failed with error -2
> 
> 4) I found some old email threads asking about this:
> 
> http://comments.gmane.org/gmane.comp.networking.dpdk.devel/1357
> (there are some others as well but this seemed closest)
> 
> But the only thing present in that thread seemed to be irritated replies which 
> didn't really explain the different virtio PMD's out there and how they work, 
> and didn't explain which ones were in-tree and out-of-tree either.
> 
> So let me ask this again, when somebody wrote "virtio-net or virtio-net + uio 
> (QEMU, VirtualBox)" into the supported page (http://dpdk.org/doc/nics), who 
> tested this to prove it worked? How did they get it to work on VirtualBox? The 
> last reply stating "you have the source code" didn't really explain how they 
> proved this stuff ever worked in the first place.
> 
> Thanks,
> Matthew.

Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud Images

[Matthew Hall]

On Mon, Oct 13, 2014 at 11:03:53PM -0700, Matthew Hall wrote:
> Another weird issue... when I tried to compile a DPDK shared lib using clang I 
> got this really, really weird error:
> 
> /usr/bin/ld: test: hidden symbol `mknod' in /usr/lib/x86_64-linux-gnu/libc_nonshared.a(mknod.oS) is referenced by DSO
> /usr/bin/ld: final link failed: Bad value

Note: this specific error seems to be a bug in the behavior of DPDK 
compilation when the following two options are enabled simultaneously:

CONFIG_RTE_BUILD_SHARED_LIB=y
CONFIG_RTE_BUILD_COMBINE_LIBS=y

I think this is a pretty serious problem for anybody that's packaging or 
distributing a complete DPDK because compiling both the static and dynamic 
DPDK's at the same time as one another is going to fail with this weird error.

Matthew.

Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud Images

[Matthew Hall]

On Mon, Oct 13, 2014 at 11:03:53PM -0700, Matthew Hall wrote:
> Another problem regarding virtio-net-pmd. When I tried using virtio-net-pmd, 
> it compiles fine but then hits a weird error also during EAL init process:
> 
> EAL: open shared lib /vagrant/external/virtio-net-pmd/librte_pmd_virtio.so
> EAL: /vagrant/external/virtio-net-pmd/librte_pmd_virtio.so: undefined symbol: per_lcore__lcore_id
> 
> It doesn't seem to have a link dependency against any DPDK library that might 
> contain such a symbol, either:
> 
> $ ldd librte_pmd_virtio.so
>     linux-vdso.so.1 =>  (0x00007fffd61fc000)
>     libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa2d971f000)
>     /lib64/ld-linux-x86-64.so.2 (0x00007fa2d9d00000)
> 
> $ nm librte_pmd_virtio.so | fgrep -i lcore
>                  U per_lcore__lcore_id
> 00000000000014ee t rte_lcore_id
> 
> man nm says this means: "U" The symbol is undefined.
> 
> At present I am using the common setup, static DPDK w/ COMBINE_LIBS. The 
> directions don't state that a shared lib DPDK is required, and if it is 
> required, that increases complexity and reduces performance so it'd be better 
> not to be forced to require this unless there's a good reason.

The PMD seems to load if DPDK is build w/ shared libraries:

EAL: open shared lib /vagrant/external/virtio-net-pmd/librte_pmd_virtio.so
librte_pmd_virtio version 1.2
    Copyright 2013-2014 6WIND S.A. provided without warranty.

However the documentation doesn't state that this is required... and requiring 
it is very irritating for developers. So I really hope we could just 
incorporate this PMD into the DPDK itself or fix whatever bugs make it not 
work with a static DPDK, or perhaps allow it to be build as a static lib and 
linked into one's app during compile so there's no need to load it as a 
plugin.

At minimum the documentation should clearly state the precise steps needed to 
get this to work so it doesn't just blow up on the user with very weird series 
of errors as seen above.

Matthew.

Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud Images

[Gonzalez Monroy, Sergio]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Matthew Hall
> Sent: Tuesday, October 14, 2014 7:34 AM
> To: dev@dpdk.org
> Subject: Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud
> Images
> 
> 
> 
> On Mon, Oct 13, 2014 at 11:03:53PM -0700, Matthew Hall wrote:
> > Another weird issue... when I tried to compile a DPDK shared lib using
> > clang I got this really, really weird error:
> >
> > /usr/bin/ld: test: hidden symbol `mknod' in
> > /usr/lib/x86_64-linux-gnu/libc_nonshared.a(mknod.oS) is referenced by
> > DSO
> > /usr/bin/ld: final link failed: Bad value
> 
> Note: this specific error seems to be a bug in the behavior of DPDK
> compilation when the following two options are enabled simultaneously:
> 
> CONFIG_RTE_BUILD_SHARED_LIB=y
> CONFIG_RTE_BUILD_COMBINE_LIBS=y
> 

Hi Matthew,

The problem as you point out is that the combined library is not being linked properly, 
in this case we are not linking against libc when building the shared library.
One way of fixing this issue is to use CC instead of LD to do the linking.

I have submitted a patch that updates the lib building process which main purpose was
to fix this issue but it seems that the removing of CONFIG_RTE_BUILD_COMBINE_LIBS
took over its real purpose.
Another fix that the patch fixes is linking against other external required libs
(ie. libm, librt, etc) when needed, so they would show on ldd if they are required.

Please have a look at http://dpdk.org/ml/archives/dev/2014-October/006453.html.
I would expect many of your issues to be fixed with that patch and would really
appreciate any feedback on it.

Thanks,
Sergio


> I think this is a pretty serious problem for anybody that's packaging or
> distributing a complete DPDK because compiling both the static and dynamic
> DPDK's at the same time as one another is going to fail with this weird error.
> 
> Matthew.

Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud Images

[Matthew Hall]

Yes, I suspected some of your patches were related to this besides just the COMBINE_LIBS options. However none of these will fix the issue that the virtio non-UIO PMD doesn't work right with a statically linked DPDK, so, despite all your fixes I've still got problems getting it all to work right.

Matthew.
-- 
Sent from my mobile device.

On October 14, 2014 1:22:56 AM PDT, "Gonzalez Monroy, Sergio" <sergio.gonzalez.monroy@intel.com> wrote:
>
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Matthew Hall
>> Sent: Tuesday, October 14, 2014 7:34 AM
>> To: dev@dpdk.org
>> Subject: Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu
>Cloud
>> Images
>> 
>> 
>> 
>> On Mon, Oct 13, 2014 at 11:03:53PM -0700, Matthew Hall wrote:
>> > Another weird issue... when I tried to compile a DPDK shared lib
>using
>> > clang I got this really, really weird error:
>> >
>> > /usr/bin/ld: test: hidden symbol `mknod' in
>> > /usr/lib/x86_64-linux-gnu/libc_nonshared.a(mknod.oS) is referenced
>by
>> > DSO
>> > /usr/bin/ld: final link failed: Bad value
>> 
>> Note: this specific error seems to be a bug in the behavior of DPDK
>> compilation when the following two options are enabled
>simultaneously:
>> 
>> CONFIG_RTE_BUILD_SHARED_LIB=y
>> CONFIG_RTE_BUILD_COMBINE_LIBS=y
>> 
>
>Hi Matthew,
>
>The problem as you point out is that the combined library is not being
>linked properly, 
>in this case we are not linking against libc when building the shared
>library.
>One way of fixing this issue is to use CC instead of LD to do the
>linking.
>
>I have submitted a patch that updates the lib building process which
>main purpose was
>to fix this issue but it seems that the removing of
>CONFIG_RTE_BUILD_COMBINE_LIBS
>took over its real purpose.
>Another fix that the patch fixes is linking against other external
>required libs
>(ie. libm, librt, etc) when needed, so they would show on ldd if they
>are required.
>
>Please have a look at
>http://dpdk.org/ml/archives/dev/2014-October/006453.html.
>I would expect many of your issues to be fixed with that patch and
>would really
>appreciate any feedback on it.
>
>Thanks,
>Sergio
>
>
>> I think this is a pretty serious problem for anybody that's packaging
>or
>> distributing a complete DPDK because compiling both the static and
>dynamic
>> DPDK's at the same time as one another is going to fail with this
>weird error.
>> 
>> Matthew.

Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud Images

[Gonzalez Monroy, Sergio]

Hi Matthew,

Could you provide steps to reproduce your issue with virtio non-UIO PMD (virtio-net-pmd) using static combined DPDK lib?

Regarding one of your comments:
> It doesn't seem to have a link dependency against any DPDK library that might contain such a symbol, either:
> 
> $ ldd librte_pmd_virtio.so
>     linux-vdso.so.1 =>  (0x00007fffd61fc000)
>     libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa2d971f000)
>     /lib64/ld-linux-x86-64.so.2 (0x00007fa2d9d00000)
>
librte_pmd_virtio.so does depend on DPDK, but ldd won't show it because it was not build against DPDK libs which it makes
sense if you are building static lib DPDK.

Thanks,
Sergio

> -----Original Message-----
> From: Matthew Hall [mailto:mhall@mhcomputing.net]
> Sent: Tuesday, October 14, 2014 9:35 AM
> To: Gonzalez Monroy, Sergio; dev@dpdk.org
> Subject: RE: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud
> Images
> 
> Yes, I suspected some of your patches were related to this besides just the
> COMBINE_LIBS options. However none of these will fix the issue that the
> virtio non-UIO PMD doesn't work right with a statically linked DPDK, so,
> despite all your fixes I've still got problems getting it all to work right.
> 
> Matthew.
> --
> Sent from my mobile device.
> 
> On October 14, 2014 1:22:56 AM PDT, "Gonzalez Monroy, Sergio"
> <sergio.gonzalez.monroy@intel.com> wrote:
> >
> >
> >> -----Original Message-----
> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Matthew Hall
> >> Sent: Tuesday, October 14, 2014 7:34 AM
> >> To: dev@dpdk.org
> >> Subject: Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu
> >Cloud
> >> Images
> >>
> >>
> >>
> >> On Mon, Oct 13, 2014 at 11:03:53PM -0700, Matthew Hall wrote:
> >> > Another weird issue... when I tried to compile a DPDK shared lib
> >using
> >> > clang I got this really, really weird error:
> >> >
> >> > /usr/bin/ld: test: hidden symbol `mknod' in
> >> > /usr/lib/x86_64-linux-gnu/libc_nonshared.a(mknod.oS) is referenced
> >by
> >> > DSO
> >> > /usr/bin/ld: final link failed: Bad value
> >>
> >> Note: this specific error seems to be a bug in the behavior of DPDK
> >> compilation when the following two options are enabled
> >simultaneously:
> >>
> >> CONFIG_RTE_BUILD_SHARED_LIB=y
> >> CONFIG_RTE_BUILD_COMBINE_LIBS=y
> >>
> >
> >Hi Matthew,
> >
> >The problem as you point out is that the combined library is not being
> >linked properly, in this case we are not linking against libc when
> >building the shared library.
> >One way of fixing this issue is to use CC instead of LD to do the
> >linking.
> >
> >I have submitted a patch that updates the lib building process which
> >main purpose was to fix this issue but it seems that the removing of
> >CONFIG_RTE_BUILD_COMBINE_LIBS took over its real purpose.
> >Another fix that the patch fixes is linking against other external
> >required libs (ie. libm, librt, etc) when needed, so they would show on
> >ldd if they are required.
> >
> >Please have a look at
> >http://dpdk.org/ml/archives/dev/2014-October/006453.html.
> >I would expect many of your issues to be fixed with that patch and
> >would really appreciate any feedback on it.
> >
> >Thanks,
> >Sergio
> >
> >
> >> I think this is a pretty serious problem for anybody that's packaging
> >or
> >> distributing a complete DPDK because compiling both the static and
> >dynamic
> >> DPDK's at the same time as one another is going to fail with this
> >weird error.
> >>
> >> Matthew.

Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud Images

[Matthew Hall]

Hi Sergio,

Sorry for the delay, this project is part-time.

On Tue, Oct 14, 2014 at 12:16:07PM +0000, Gonzalez Monroy, Sergio wrote:
> Could you provide steps to reproduce your issue with virtio non-UIO PMD (virtio-net-pmd) using static combined DPDK lib?

I am getting two different kinds of problems. See below.

> librte_pmd_virtio.so does depend on DPDK, but ldd won't show it because it 
> was not build against DPDK libs which it makes sense if you are building 
> static lib DPDK.

The virtio non UIO PMD doesn't depend on DPDK even if everything's built as a 
.so . To me this seems buggy but maybe I missed something:

$ ldd librte_pmd_virtio.so
    linux-vdso.so.1 =>  (0x00007fff7adfe000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa0d810c000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa0d86ed000)

Now, about the problems...

1) This first problem happens using it with DPDK built as shared 
libraries. It gets stuck looping forever in this code if I build it as a 
shared library, and DPDK won't initialize:

(gdb)
850     while (vq->last_used == vq->vr.used->idx) {
(gdb)
851         usleep(100);
(gdb) bt
#0  virtio_send_command (vq=0x7fffed0ffdc8, ctrl=0x7fffffffe2b0, dlen=0x7fffffffe2a0, pkt_num=1) at virtio_user.c:851
#1  0x00007ffff193f73c in virtio_dev_promiscuous_disable (dev=0x7ffff7dd9120 <rte_eth_devices>) at virtio_user.c:914
#2  0x00007ffff7bd2f37 in rte_eth_promiscuous_disable (port_id=0 '\000') at /vagrant/external/dpdk/lib/librte_ether/rte_ethdev.c:1055
#3  0x00007ffff7bd271b in rte_eth_dev_config_restore (port_id=0 '\000') at /vagrant/external/dpdk/lib/librte_ether/rte_ethdev.c:788
#4  0x00007ffff7bd25b1 in rte_eth_dev_start (port_id=0 '\000') at /vagrant/external/dpdk/lib/librte_ether/rte_ethdev.c:828
#5  0x000000000042f2ea in main (argc=3, argv=0x7fffffffe668) at sdn_sensor.c:399
(gdb)

It appears to do this whether or not I use VBoxManage modifyvm <vm_name> 
--nicpromiscN allow-all or not on the VM. So it can't be caused just by VBox 
messing up the promisc settings.

2) The second problem concerns compiling against DPDK 1.7.1 (plug my minor 
clang compilation fixes) with these settings:

# Compile to share library
CONFIG_RTE_BUILD_SHARED_LIB=n
# Combine to one single library
CONFIG_RTE_BUILD_COMBINE_LIBS=y
CONFIG_RTE_LIBNAME="intel_dpdk"

Then virtio-net-pmd is compiled using make RTE_INCLUDE=../dpdk/build/include . 
Just as printed above, no runtime dependency on DPDK is there (as expected in 
the static case, but not what I believe should be expected in the dynamic 
case).

>From there, the app is launched. Then the following error appears, and no 
Ethernet ports can be detected since the builtin UIO PMD driver can't see the 
ports either, so the app exits.

EAL: open shared lib /vagrant/external/virtio-net-pmd/librte_pmd_virtio.so
EAL: /vagrant/external/virtio-net-pmd/librte_pmd_virtio.so: undefined symbol: per_lcore__lcore_id

Running nm and nm -D shows this:

$ nm librte_pmd_virtio.so | fgrep -i per_lcore__lcore_id
U per_lcore__lcore_id

According to the nm manpage, "U" means "The symbol is undefined".

Since it didn't get added from a static lib, and our DPDK wasn't linked and 
loaded in dynamic mode where the linker could hook up the symbol, loading 
librte_pmd_virtio.so fails and no ports can be found.

Given all of this, I must re-ask the question I previously asked: who provided 
QA before claiming on the DPDK website that this driver works with VirtualBox, 
and precisely and reproducibly, how did they get it to run right? Because it's 
impossible to get anything to work on my machine using all the directions I've 
been able to find thus far.

Thanks,
Matthew.

Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud Images

[Gonzalez Monroy, Sergio]

Hi Matthew,

> -----Original Message-----
> From: Matthew Hall [mailto:mhall@mhcomputing.net]
> Sent: Friday, October 17, 2014 9:57 AM
> To: Gonzalez Monroy, Sergio
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud
> Images
> 
[...]
> The virtio non UIO PMD doesn't depend on DPDK even if everything's built as
> a .so . To me this seems buggy but maybe I missed something:
> 
> $ ldd librte_pmd_virtio.so
>     linux-vdso.so.1 =>  (0x00007fff7adfe000)
>     libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa0d810c000)
>     /lib64/ld-linux-x86-64.so.2 (0x00007fa0d86ed000)
> 
You are right about the dependency (probably I did not explain the issue properly).

As you point out below, when building static DPDK we should not expect ldd to report any DPDK dependency.
When building shared DPDK libs, we should expect such dependency expect for the fact that we are not linking against DPDK
libraries when building librte_pmd_virtio.so, which as you mention is buggy.
The problem is not exclusive of librte_pmd_virtio.so module as you can easily check by doing ldd on DPDK shared libraries.
One way to fix the buggy behavior with librte_pmd_virtio would be to build against dpdk libs when dpdk is built as shared libs.

The issue is that we have two different ways of building DPDK, static or shared libs, and the building process differs.
Let's use librte_pmd_virtio shared object building process as an example:
- we do not want to build against static DPDK libraries as this would result in duplicated code in librte_pmd_virtio and other apps (ie. testpmd)
- we want to link against shared DPDK libs to add dependencies and provide reliable information (ie. ldd)

Currently librte_pmd_virtio is built without linking against PDDK libs, therefore no dependencies are included even when DPDK is built as shared libs.
With this way of building the shared object, the module will be loaded successfully on apps built with static/shared DPDK libs as long as all undefined
symbols in librte_pmd_virtio are defined in either the app (for static) or any of the loaded shared libs (for shared).

> Now, about the problems...
> 
I have not been able to reproduce these problems.
My setup was QEMU, dpdk-1.7.1, virtio-net-pmd, fedora 20 (both host and guest), GCC/CLANG.
I have successfully loaded the module running testpmd with static/shared DPDK libs using GCC and CLANG.

[...]
> 2) The second problem concerns compiling against DPDK 1.7.1 (plug my minor
> clang compilation fixes) with these settings:
> 
> # Compile to share library
> CONFIG_RTE_BUILD_SHARED_LIB=n
> # Combine to one single library
> CONFIG_RTE_BUILD_COMBINE_LIBS=y
> CONFIG_RTE_LIBNAME="intel_dpdk"
> 
> Then virtio-net-pmd is compiled using make
> RTE_INCLUDE=../dpdk/build/include .
> Just as printed above, no runtime dependency on DPDK is there (as
> expected in the static case, but not what I believe should be expected in the
> dynamic case).
> 
> From there, the app is launched. Then the following error appears, and no
> Ethernet ports can be detected since the builtin UIO PMD driver can't see the
> ports either, so the app exits.
> 
> EAL: open shared lib /vagrant/external/virtio-net-pmd/librte_pmd_virtio.so
> EAL: /vagrant/external/virtio-net-pmd/librte_pmd_virtio.so: undefined
> symbol: per_lcore__lcore_id
> 
Are we talking about a DPDK or custom app?
Do you only see the issue when CONFIG_RTE_BUILD_COMBINE_LIBS=y?

> Running nm and nm -D shows this:
> 
> $ nm librte_pmd_virtio.so | fgrep -i per_lcore__lcore_id U
> per_lcore__lcore_id
> 
This is expected behavior as the symbol is defined in librte_eal.
The dynamic linker will resolve the undefined reference when loading the module in run-time.

Thanks,
Sergio

Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud Images

[Matthew Hall]

On Tue, Oct 21, 2014 at 01:22:27PM +0000, Gonzalez Monroy, Sergio wrote:
> As you point out below, when building static DPDK we should not expect ldd 
> to report any DPDK dependency. When building shared DPDK libs, we should 
> expect such dependency expect for the fact that we are not linking against 
> DPDK libraries when building librte_pmd_virtio.so, which as you mention is 
> buggy.

Yes. I agree. Can we see about fixing this bug?

> - we do not want to build against static DPDK libraries as this would result 
> in duplicated code in librte_pmd_virtio and other apps (ie. testpmd)
>
> - we want to link against shared DPDK libs to add dependencies and provide 
> reliable information (ie. ldd)

OK... but now it's impossible to use librte_pmd_virtio w/o mandatory share 
library performance loss. I strongly dislike being force to do this.

> > Now, about the problems...
>
> I have not been able to reproduce these problems. My setup was QEMU, 
> dpdk-1.7.1, virtio-net-pmd, fedora 20 (both host and guest), GCC/CLANG. I 
> have successfully loaded the module running testpmd with static/shared DPDK 
> libs using GCC and CLANG.

OK... let me try to clarify this point again. In this official DPDK support 
device document, http://www.dpdk.org/doc/nics , it says:

    Paravirtualization
    virtio-net or virtio-net + uio (QEMU, VirtualBox)

As I've stated, when testing this on VirtualBox it does not work for me and 
gets into an infinite initialization loop which I documented in my last mail. 
But the same code works fine if it's using the VBox Intel 82545EM VNIC and 
appropriate driver. Also the VBox virtio-net device works completely fine 
using the kernel virtio-net driver. This making the virtio PMD's the most 
likely suspect, especially since the UIO based one can't init itself, and the 
non UIO one gets stuck in the loop.

So my question is very simple.

1) Who tested this setup using *****VirtualBox NOT QEMU*****? QEMU doesn't 
help at all for my app because I'm trying to prepackage it as a Vagrant VM and 
Vagrant uses VirtualBox. It also doesn't help repro my bug because the 
virtio-net device is not 100% same between QEMU and VBOX so you can't compare 
1-1.

2) Who made the instructions to configure this with VirtualBox? I could not 
find any such thing.

3) Who ever got this to work right in the first place?

It's been multiple weeks of emailing and I still have no answer who placed 
this inaccurate text on the website. Nobody answered the last guy who asked it 
in 2013 either. So now it's IMPOSSIBLE for me to know if it worked and I 
configured it wrong or it never worked in the first place.

> > EAL: open shared lib /vagrant/external/virtio-net-pmd/librte_pmd_virtio.so
> > EAL: /vagrant/external/virtio-net-pmd/librte_pmd_virtio.so: undefined
> > symbol: per_lcore__lcore_id
> > 
> Are we talking about a DPDK or custom app?
> Do you only see the issue when CONFIG_RTE_BUILD_COMBINE_LIBS=y?

Issue happens in my DPDK based app.

Can happen anytime you use static linked DPDK app w/ the librte_pmd_virtio. 
Because the link process of librte_pmd_virtio is broken.

> > Running nm and nm -D shows this:
> > 
> > $ nm librte_pmd_virtio.so | fgrep -i per_lcore__lcore_id U
> > per_lcore__lcore_id
> > 
> This is expected behavior as the symbol is defined in librte_eal.
> The dynamic linker will resolve the undefined reference when loading the module in run-time.

I am aware it's "expected behavior". But have the undefined symbol, and no 
dependency upon the DPDK .so and no link against the DPDK .a is NOT "expected 
behavior". It will break anytime you try to make a static app with this PMD 
available.

Matthew.

Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud Images

[Gonzalez Monroy, Sergio]

> -----Original Message-----
> From: Matthew Hall [mailto:mhall@mhcomputing.net]
> Subject: Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud
> Images
> 
> > - we do not want to build against static DPDK libraries as this would
> > result in duplicated code in librte_pmd_virtio and other apps (ie.
> > testpmd)
> >
> > - we want to link against shared DPDK libs to add dependencies and
> > provide reliable information (ie. ldd)
> 
> OK... but now it's impossible to use librte_pmd_virtio w/o mandatory share
> library performance loss. I strongly dislike being force to do this.
> 
You are not forced to use shared libraries. This module loads successfully with an app (testpmd) built against static DPDK libs.

> OK... let me try to clarify this point again. In this official DPDK support device
> document, http://www.dpdk.org/doc/nics , it says:
> 
>     Paravirtualization
>     virtio-net or virtio-net + uio (QEMU, VirtualBox)
> 
> As I've stated, when testing this on VirtualBox it does not work for me and
> gets into an infinite initialization loop which I documented in my last mail.
> But the same code works fine if it's using the VBox Intel 82545EM VNIC and
> appropriate driver. Also the VBox virtio-net device works completely fine
> using the kernel virtio-net driver. This making the virtio PMD's the most likely
> suspect, especially since the UIO based one can't init itself, and the non UIO
> one gets stuck in the loop.
> 
I have reproduced this issue in VirtualBox:
- For UIO Virtio PMD, there is an issue with igb_uio module and virtio vbox backend device, I fail to bind igb_uio driver to the virtio device.
- For non-UIO Virtio PMD, the module fails to initialize properly as you have indicated in your previous post (stuck in a loop).

I get this behavior with testpmd regardless of DPDK being built as static or shared.

> > > EAL: open shared lib
> > > /vagrant/external/virtio-net-pmd/librte_pmd_virtio.so
> > > EAL: /vagrant/external/virtio-net-pmd/librte_pmd_virtio.so:
> > > undefined
> > > symbol: per_lcore__lcore_id
> > >
> > Are we talking about a DPDK or custom app?
> > Do you only see the issue when CONFIG_RTE_BUILD_COMBINE_LIBS=y?
> 
> Issue happens in my DPDK based app.
> 
> Can happen anytime you use static linked DPDK app w/ the
> librte_pmd_virtio.
> Because the link process of librte_pmd_virtio is broken.
> 
The linking is not broken if we are assuming apps built against static DPDK libs.
I can't think of other way of linking this module to be used in apps with static DPDK libs.

> > > Running nm and nm -D shows this:
> > >
> > > $ nm librte_pmd_virtio.so | fgrep -i per_lcore__lcore_id U
> > > per_lcore__lcore_id
> > >
> > This is expected behavior as the symbol is defined in librte_eal.
> > The dynamic linker will resolve the undefined reference when loading the
> module in run-time.
> 
> I am aware it's "expected behavior". But have the undefined symbol, and no
> dependency upon the DPDK .so and no link against the DPDK .a is NOT
> "expected behavior". It will break anytime you try to make a static app with
> this PMD available.
> 
Have you built static DPDK libs and run testpmd?

Your undefined symbol error is most likely because the symbol is not in the dynamic symbol table of you app.
You need to pass -rdynamic to GCC or -export-dynamic to LD when building your app.

Thanks,
Sergio

> Matthew.

Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud Images

[Matthew Hall]

On Wed, Oct 22, 2014 at 03:20:40PM +0000, Gonzalez Monroy, Sergio wrote:
> You are not forced to use shared libraries. This module loads successfully 
> with an app (testpmd) built against static DPDK libs.

It sounds like it just requires additional options as mentioned later in your 
mail. We should document the recommended set of link flags for static and 
dynamic link w/ DPDK for apps, as it seems to cause issues for everybody 
making non-example DPDK apps. I've seen several threads about it since joining 
the mailing list a few months back. Maybe it is documented already but I 
didn't see it. I'm willing to write it up but somebody will have to help 
verify I got it right.

> I have reproduced this issue in VirtualBox:
> 
> - For UIO Virtio PMD, there is an issue with igb_uio module and virtio vbox 
> backend device, I fail to bind igb_uio driver to the virtio device.
> 
> - For non-UIO Virtio PMD, the module fails to initialize properly as you 
> have indicated in your previous post (stuck in a loop).
> 
> I get this behavior with testpmd regardless of DPDK being built as static or 
> shared.

THANK YOU!!! I am so glad to hear I'm not crazy and it really does not work.

So... back to the Supported NICs page. Right not it claims VirtualBox is 
supported but 2 people have seen that it doesn't work at all with either 
driver. Is it intended to be supported configuration or not?

If it is intended to be supported can we find someone who can help us fix the 
bugs? It's not code I know much about. If it is not intended to work on VBox 
can we delete it from the documentation so nobody besides me and you and the 
other guy in 2013 waste more time trying to use it if it wasn't supposed to 
work in the first place?

> Your undefined symbol error is most likely because the symbol is not in the 
> dynamic symbol table of you app. You need to pass -rdynamic to GCC or 
> -export-dynamic to LD when building your app.

Good advice. I'll try it and see. Of course it won't fix the infinite loop 
though so the driver still won't help much even with the change present.

> Thanks,
> Sergio

Thanks to you for help verifying / reproducing / identifying the issues.

Matthew.

Re: [dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud Images

[Gonzalez Monroy, Sergio]

> -----Original Message-----
> From: Matthew Hall [mailto:mhall@mhcomputing.net]
> Sent: Wednesday, October 22, 2014 7:42 PM
> On Wed, Oct 22, 2014 at 03:20:40PM +0000, Gonzalez Monroy, Sergio wrote:
>
> > You are not forced to use shared libraries. This module loads
> > successfully with an app (testpmd) built against static DPDK libs.
> 
> It sounds like it just requires additional options as mentioned later in your
> mail. We should document the recommended set of link flags for static and
> dynamic link w/ DPDK for apps, as it seems to cause issues for everybody
> making non-example DPDK apps. I've seen several threads about it since
> joining the mailing list a few months back. Maybe it is documented already
> but I didn't see it. I'm willing to write it up but somebody will have to help
> verify I got it right.
>
In this regard, I would be shortly posting an RFC on ways I think we can improve
the current build system, which would also impact how apps are built.

> So... back to the Supported NICs page. Right not it claims VirtualBox is
> supported but 2 people have seen that it doesn't work at all with either
> driver. Is it intended to be supported configuration or not?
> 
> If it is intended to be supported can we find someone who can help us fix the
> bugs? It's not code I know much about. If it is not intended to work on VBox
> can we delete it from the documentation so nobody besides me and you and
> the other guy in 2013 waste more time trying to use it if it wasn't supposed to
> work in the first place?
> 
I am aware of QEMU/KVM, VMware and Xen being supported/tested, so
probably the documentation should be updated accordingly.

> > Your undefined symbol error is most likely because the symbol is not
> > in the dynamic symbol table of you app. You need to pass -rdynamic to
> > GCC or -export-dynamic to LD when building your app.
> 
> Good advice. I'll try it and see. Of course it won't fix the infinite loop though
> so the driver still won't help much even with the change present.
> 
It certainly won't fix the loop problem :)

Thanks,
Sergio