[dpdk-dev] Debugging EAL PCI / Driver Init

[dpdk-dev] Debugging EAL PCI / Driver Init

[Matthew Hall]

Hello,

I am running into a problem where Eth driver init works fine in a sample app 
and finds my NICs, and the NICs appear in rte_eal_pci_dump(stdout) but they 
don't show up in rte_eth_dev_count() even after rte_eal_pci_probe() is called 
the same as the sample apps, so my app won't boot.

I have a lot of experience using the older versions of the DPDK where you had 
to call the PMD init functions manually but no experience with the later 
versions where the DPDK is supposed to init the PMDs itself automatically.

What do I have to do to dump the most possible debug output on why the driver 
list for my PCI devices always seems empty? Any places I should look to see 
the issue? Maybe I didn't link it together with the right DPDK libs? I used 
the combined DPDK static lib libintel_dpdk.a to make things simpler as I had 
seen recommended in various places.

Thanks,
Matthew.

Re: [dpdk-dev] Debugging EAL PCI / Driver Init

[Matthew Hall]

I did a bit more experimentation and found the following. If I unmark the 
rte_igb_pmd_init function as static, and call it directly from my code, the 
driver will load, and the port count increments to 2:

EAL: PCI device 0000:01:00.0 on NUMA socket -1
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL:   PCI memory mapped at 0x7f09d45f2000
EAL:   PCI memory mapped at 0x7f09d4730000
PMD: eth_igb_dev_init(): port_id 0 vendorID=0x8086 deviceID=0x1521

EAL: PCI device 0000:01:00.1 on NUMA socket -1
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL:   PCI memory mapped at 0x7f09d0f00000
EAL:   PCI memory mapped at 0x7f09d472c000
PMD: eth_igb_dev_init(): port_id 1 vendorID=0x8086 deviceID=0x1521

So it seems like when you enable these options:

# Combine to one single library
CONFIG_RTE_BUILD_COMBINE_LIBS=y
CONFIG_RTE_LIBNAME="intel_dpdk"

You don't really get a working DPDK inside of "-lintel_dpdk". Someone 
suggested linking with "-Xlinker -lintel_dpdk" but that didn't seem to help.

Is there a secret to getting a single integrated static library, where all of 
the PMD's end up in the PCI driver list so rte_eal_pci_probe can find them? Or 
some secret to linking against the combined library which works properly?

Matthew.

On Fri, Aug 01, 2014 at 10:51:38AM -0700, Matthew Hall wrote:
> Hello,
> 
> I am running into a problem where Eth driver init works fine in a sample app 
> and finds my NICs, and the NICs appear in rte_eal_pci_dump(stdout) but they 
> don't show up in rte_eth_dev_count() even after rte_eal_pci_probe() is called 
> the same as the sample apps, so my app won't boot.
> 
> I have a lot of experience using the older versions of the DPDK where you had 
> to call the PMD init functions manually but no experience with the later 
> versions where the DPDK is supposed to init the PMDs itself automatically.
> 
> What do I have to do to dump the most possible debug output on why the driver 
> list for my PCI devices always seems empty? Any places I should look to see 
> the issue? Maybe I didn't link it together with the right DPDK libs? I used 
> the combined DPDK static lib libintel_dpdk.a to make things simpler as I had 
> seen recommended in various places.
> 
> Thanks,
> Matthew.

Re: [dpdk-dev] Debugging EAL PCI / Driver Init

[Matthew Hall]

Also, when using the separate libraries the problem still happens:

-lethdev -lrte_cfgfile -lrte_cmdline -lrte_distributor -lrte_hash 
-lrte_ip_frag -lrte_lpm -lrte_malloc -lrte_mbuf -lrte_mempool -lrte_pmd_e1000 
-lrte_pmd_pcap -lrte_pmd_virtio_uio -lrte_pmd_vmxnet3_uio -lrte_port 
-lrte_ring -lrte_table -lrte_timer -lrte_eal -lbsd -ldl -lpcap -lpthread

So it seems there is a special order or link technique which must be used, or 
rte_eal's PCI code won't be able to load the PMD's during PCI probing. Is it 
documented anywhere how to get that to work?

Thanks,
Matthew.

On Sat, Aug 02, 2014 at 08:29:04AM -0700, Matthew Hall wrote:
> I did a bit more experimentation and found the following. If I unmark the 
> rte_igb_pmd_init function as static, and call it directly from my code, the 
> driver will load, and the port count increments to 2:
> 
> EAL: PCI device 0000:01:00.0 on NUMA socket -1
> EAL:   probe driver: 8086:1521 rte_igb_pmd
> EAL:   PCI memory mapped at 0x7f09d45f2000
> EAL:   PCI memory mapped at 0x7f09d4730000
> PMD: eth_igb_dev_init(): port_id 0 vendorID=0x8086 deviceID=0x1521
> 
> EAL: PCI device 0000:01:00.1 on NUMA socket -1
> EAL:   probe driver: 8086:1521 rte_igb_pmd
> EAL:   PCI memory mapped at 0x7f09d0f00000
> EAL:   PCI memory mapped at 0x7f09d472c000
> PMD: eth_igb_dev_init(): port_id 1 vendorID=0x8086 deviceID=0x1521
> 
> So it seems like when you enable these options:
> 
> # Combine to one single library
> CONFIG_RTE_BUILD_COMBINE_LIBS=y
> CONFIG_RTE_LIBNAME="intel_dpdk"
> 
> You don't really get a working DPDK inside of "-lintel_dpdk". Someone 
> suggested linking with "-Xlinker -lintel_dpdk" but that didn't seem to help.
> 
> Is there a secret to getting a single integrated static library, where all of 
> the PMD's end up in the PCI driver list so rte_eal_pci_probe can find them? Or 
> some secret to linking against the combined library which works properly?
> 
> Matthew.
> 
> On Fri, Aug 01, 2014 at 10:51:38AM -0700, Matthew Hall wrote:
> > Hello,
> > 
> > I am running into a problem where Eth driver init works fine in a sample app 
> > and finds my NICs, and the NICs appear in rte_eal_pci_dump(stdout) but they 
> > don't show up in rte_eth_dev_count() even after rte_eal_pci_probe() is called 
> > the same as the sample apps, so my app won't boot.
> > 
> > I have a lot of experience using the older versions of the DPDK where you had 
> > to call the PMD init functions manually but no experience with the later 
> > versions where the DPDK is supposed to init the PMDs itself automatically.
> > 
> > What do I have to do to dump the most possible debug output on why the driver 
> > list for my PCI devices always seems empty? Any places I should look to see 
> > the issue? Maybe I didn't link it together with the right DPDK libs? I used 
> > the combined DPDK static lib libintel_dpdk.a to make things simpler as I had 
> > seen recommended in various places.
> > 
> > Thanks,
> > Matthew.

Re: [dpdk-dev] Debugging EAL PCI / Driver Init

[Matthew Hall]

On Sun, Aug 03, 2014 at 01:37:06AM +0900, Masaru Oki wrote:
> cc links library funtion from archive only if call from other object.
> but new dpdk pmd library has constractor section and not call directly.
> ld always links library funtion with constractor section.
> use -Xlinker, or use ld instead of cc.

Hello Oki-san,

The trick to fix it was this, I finally found it in the example Makefiles with 
V=1 flag.

-Wl,--whole-archive -Wl,--start-group -lintel_dpdk -Wl,--end-group -Wl,--no-whole-archive

Thank you for the advice you provided, I couldn't have fixed it without your 
suggestions... it got me to look more closely at the linking. Importantly, 
"-Wl,--whole-archive" includes the entire archive whether or not it's called 
from other objects, so we don't lose the constructors, just like you said.

Matthew.

Re: [dpdk-dev] Debugging EAL PCI / Driver Init

[Alex Markuze]

Hi Matt, Dev
I'm Trying to compile ann app linking to dpdk and dpdk based libs.
And I'm seeing the same issue you've reported.
The probe function doesn't seem to find any ixgbevf(SRIOV VM) ports.
Same code compiled as a dpdk app works fine.

In your solution to this issue you are referring to  -lintel_dpdk? I
couldn't find any reference to it.

Thanks
Alex.


On Sat, Aug 2, 2014 at 7:46 PM, Matthew Hall <mhall@mhcomputing.net> wrote:
> On Sun, Aug 03, 2014 at 01:37:06AM +0900, Masaru Oki wrote:
>> cc links library funtion from archive only if call from other object.
>> but new dpdk pmd library has constractor section and not call directly.
>> ld always links library funtion with constractor section.
>> use -Xlinker, or use ld instead of cc.
>
> Hello Oki-san,
>
> The trick to fix it was this, I finally found it in the example Makefiles with
> V=1 flag.
>
> -Wl,--whole-archive -Wl,--start-group -lintel_dpdk -Wl,--end-group -Wl,--no-whole-archive
>
> Thank you for the advice you provided, I couldn't have fixed it without your
> suggestions... it got me to look more closely at the linking. Importantly,
> "-Wl,--whole-archive" includes the entire archive whether or not it's called
> from other objects, so we don't lose the constructors, just like you said.
>
> Matthew.

Re: [dpdk-dev] Debugging EAL PCI / Driver Init

[Alex Markuze]

Resolved just as Matt has described.
To remove any ambiguity (for future reference).

This line in the gcc command resolves the issue in my case (a
different nic may need a different lib):
--whole-archive -Wl,--start-group -lrte_pmd_ixgbe -Wl,--end-group -Wl,

The problem is that the probe command polls over all pci devices and
tries to find a matching driver, these drivers register
With these macros which will only be called when the --whole-archive
option is provided and its actually the reason for this flags
existence. Without this flag the driver list is empty.

PMD_REGISTER_DRIVER(rte_ixgbe_driver);
PMD_REGISTER_DRIVER(rte_ixgbevf_driver);


On Sun, Aug 3, 2014 at 1:38 PM, Alex Markuze <alex@weka.io> wrote:
> Hi Matt, Dev
> I'm Trying to compile ann app linking to dpdk and dpdk based libs.
> And I'm seeing the same issue you've reported.
> The probe function doesn't seem to find any ixgbevf(SRIOV VM) ports.
> Same code compiled as a dpdk app works fine.
>
> In your solution to this issue you are referring to  -lintel_dpdk? I
> couldn't find any reference to it.
>
> Thanks
> Alex.
>
>
> On Sat, Aug 2, 2014 at 7:46 PM, Matthew Hall <mhall@mhcomputing.net> wrote:
>> On Sun, Aug 03, 2014 at 01:37:06AM +0900, Masaru Oki wrote:
>>> cc links library funtion from archive only if call from other object.
>>> but new dpdk pmd library has constractor section and not call directly.
>>> ld always links library funtion with constractor section.
>>> use -Xlinker, or use ld instead of cc.
>>
>> Hello Oki-san,
>>
>> The trick to fix it was this, I finally found it in the example Makefiles with
>> V=1 flag.
>>
>> -Wl,--whole-archive -Wl,--start-group -lintel_dpdk -Wl,--end-group -Wl,--no-whole-archive
>>
>> Thank you for the advice you provided, I couldn't have fixed it without your
>> suggestions... it got me to look more closely at the linking. Importantly,
>> "-Wl,--whole-archive" includes the entire archive whether or not it's called
>> from other objects, so we don't lose the constructors, just like you said.
>>
>> Matthew.

Re: [dpdk-dev] Debugging EAL PCI / Driver Init

[Matthew Hall]

There is an option in the DPDK build config which compiles every DPDK lib into a single static lib. I don't have the name of the option in front of me but it had COMBINE in its name. When this option is used you can get every function in the whole DPDK with a single library. After that I had a lot fewer linking issues.

Matthew.
-- 
Sent from my mobile device.

On August 3, 2014 4:41:51 AM PDT, Alex Markuze <alex@weka.io> wrote:
>Resolved just as Matt has described.
>To remove any ambiguity (for future reference).
>
>This line in the gcc command resolves the issue in my case (a
>different nic may need a different lib):
>--whole-archive -Wl,--start-group -lrte_pmd_ixgbe -Wl,--end-group -Wl,
>
>The problem is that the probe command polls over all pci devices and
>tries to find a matching driver, these drivers register
>With these macros which will only be called when the --whole-archive
>option is provided and its actually the reason for this flags
>existence. Without this flag the driver list is empty.
>
>PMD_REGISTER_DRIVER(rte_ixgbe_driver);
>PMD_REGISTER_DRIVER(rte_ixgbevf_driver);
>
>
>On Sun, Aug 3, 2014 at 1:38 PM, Alex Markuze <alex@weka.io> wrote:
>> Hi Matt, Dev
>> I'm Trying to compile ann app linking to dpdk and dpdk based libs.
>> And I'm seeing the same issue you've reported.
>> The probe function doesn't seem to find any ixgbevf(SRIOV VM) ports.
>> Same code compiled as a dpdk app works fine.
>>
>> In your solution to this issue you are referring to  -lintel_dpdk? I
>> couldn't find any reference to it.
>>
>> Thanks
>> Alex.
>>
>>
>> On Sat, Aug 2, 2014 at 7:46 PM, Matthew Hall <mhall@mhcomputing.net>
>wrote:
>>> On Sun, Aug 03, 2014 at 01:37:06AM +0900, Masaru Oki wrote:
>>>> cc links library funtion from archive only if call from other
>object.
>>>> but new dpdk pmd library has constractor section and not call
>directly.
>>>> ld always links library funtion with constractor section.
>>>> use -Xlinker, or use ld instead of cc.
>>>
>>> Hello Oki-san,
>>>
>>> The trick to fix it was this, I finally found it in the example
>Makefiles with
>>> V=1 flag.
>>>
>>> -Wl,--whole-archive -Wl,--start-group -lintel_dpdk -Wl,--end-group
>-Wl,--no-whole-archive
>>>
>>> Thank you for the advice you provided, I couldn't have fixed it
>without your
>>> suggestions... it got me to look more closely at the linking.
>Importantly,
>>> "-Wl,--whole-archive" includes the entire archive whether or not
>it's called
>>> from other objects, so we don't lose the constructors, just like you
>said.
>>>
>>> Matthew.

Re: [dpdk-dev] Debugging EAL PCI / Driver Init

[Jayakumar, Muthurajan]

Are you referring to CONFIG_RTE_BUILD_COMBINE_LIBS ? 

(ps: referenced here http://dpdk.org/ml/archives/dev/2013-October/000639.html)

Thanks,
M Jay


-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Matthew Hall
Sent: Sunday, August 03, 2014 10:23 AM
To: Alex Markuze
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] Debugging EAL PCI / Driver Init

There is an option in the DPDK build config which compiles every DPDK lib into a single static lib. I don't have the name of the option in front of me but it had COMBINE in its name. When this option is used you can get every function in the whole DPDK with a single library. After that I had a lot fewer linking issues.

Matthew.
--
Sent from my mobile device.

On August 3, 2014 4:41:51 AM PDT, Alex Markuze <alex@weka.io> wrote:
>Resolved just as Matt has described.
>To remove any ambiguity (for future reference).
>
>This line in the gcc command resolves the issue in my case (a different 
>nic may need a different lib):
>--whole-archive -Wl,--start-group -lrte_pmd_ixgbe -Wl,--end-group -Wl,
>
>The problem is that the probe command polls over all pci devices and 
>tries to find a matching driver, these drivers register With these 
>macros which will only be called when the --whole-archive option is 
>provided and its actually the reason for this flags existence. Without 
>this flag the driver list is empty.
>
>PMD_REGISTER_DRIVER(rte_ixgbe_driver);
>PMD_REGISTER_DRIVER(rte_ixgbevf_driver);
>
>
>On Sun, Aug 3, 2014 at 1:38 PM, Alex Markuze <alex@weka.io> wrote:
>> Hi Matt, Dev
>> I'm Trying to compile ann app linking to dpdk and dpdk based libs.
>> And I'm seeing the same issue you've reported.
>> The probe function doesn't seem to find any ixgbevf(SRIOV VM) ports.
>> Same code compiled as a dpdk app works fine.
>>
>> In your solution to this issue you are referring to  -lintel_dpdk? I 
>> couldn't find any reference to it.
>>
>> Thanks
>> Alex.
>>
>>
>> On Sat, Aug 2, 2014 at 7:46 PM, Matthew Hall <mhall@mhcomputing.net>
>wrote:
>>> On Sun, Aug 03, 2014 at 01:37:06AM +0900, Masaru Oki wrote:
>>>> cc links library funtion from archive only if call from other
>object.
>>>> but new dpdk pmd library has constractor section and not call
>directly.
>>>> ld always links library funtion with constractor section.
>>>> use -Xlinker, or use ld instead of cc.
>>>
>>> Hello Oki-san,
>>>
>>> The trick to fix it was this, I finally found it in the example
>Makefiles with
>>> V=1 flag.
>>>
>>> -Wl,--whole-archive -Wl,--start-group -lintel_dpdk -Wl,--end-group
>-Wl,--no-whole-archive
>>>
>>> Thank you for the advice you provided, I couldn't have fixed it
>without your
>>> suggestions... it got me to look more closely at the linking.
>Importantly,
>>> "-Wl,--whole-archive" includes the entire archive whether or not
>it's called
>>> from other objects, so we don't lose the constructors, just like you
>said.
>>>
>>> Matthew.

Re: [dpdk-dev] Debugging EAL PCI / Driver Init

[Matthew Hall]

Yes, that's the one! It made a lot of my linking problems go away when I used that along with the whole-archive flags.

Matthew
-- 
Sent from my mobile device.

On August 3, 2014 11:13:55 AM PDT, "Jayakumar, Muthurajan" <muthurajan.jayakumar@intel.com> wrote:
>
>Are you referring to CONFIG_RTE_BUILD_COMBINE_LIBS ? 
>
>(ps: referenced here
>http://dpdk.org/ml/archives/dev/2013-October/000639.html)
>
>Thanks,
>M Jay
>
>
>-----Original Message-----
>From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Matthew Hall
>Sent: Sunday, August 03, 2014 10:23 AM
>To: Alex Markuze
>Cc: dev@dpdk.org
>Subject: Re: [dpdk-dev] Debugging EAL PCI / Driver Init
>
>There is an option in the DPDK build config which compiles every DPDK
>lib into a single static lib. I don't have the name of the option in
>front of me but it had COMBINE in its name. When this option is used
>you can get every function in the whole DPDK with a single library.
>After that I had a lot fewer linking issues.
>
>Matthew.
>--
>Sent from my mobile device.
>
>On August 3, 2014 4:41:51 AM PDT, Alex Markuze <alex@weka.io> wrote:
>>Resolved just as Matt has described.
>>To remove any ambiguity (for future reference).
>>
>>This line in the gcc command resolves the issue in my case (a
>different 
>>nic may need a different lib):
>>--whole-archive -Wl,--start-group -lrte_pmd_ixgbe -Wl,--end-group -Wl,
>>
>>The problem is that the probe command polls over all pci devices and 
>>tries to find a matching driver, these drivers register With these 
>>macros which will only be called when the --whole-archive option is 
>>provided and its actually the reason for this flags existence. Without
>
>>this flag the driver list is empty.
>>
>>PMD_REGISTER_DRIVER(rte_ixgbe_driver);
>>PMD_REGISTER_DRIVER(rte_ixgbevf_driver);
>>
>>
>>On Sun, Aug 3, 2014 at 1:38 PM, Alex Markuze <alex@weka.io> wrote:
>>> Hi Matt, Dev
>>> I'm Trying to compile ann app linking to dpdk and dpdk based libs.
>>> And I'm seeing the same issue you've reported.
>>> The probe function doesn't seem to find any ixgbevf(SRIOV VM) ports.
>>> Same code compiled as a dpdk app works fine.
>>>
>>> In your solution to this issue you are referring to  -lintel_dpdk? I
>
>>> couldn't find any reference to it.
>>>
>>> Thanks
>>> Alex.
>>>
>>>
>>> On Sat, Aug 2, 2014 at 7:46 PM, Matthew Hall <mhall@mhcomputing.net>
>>wrote:
>>>> On Sun, Aug 03, 2014 at 01:37:06AM +0900, Masaru Oki wrote:
>>>>> cc links library funtion from archive only if call from other
>>object.
>>>>> but new dpdk pmd library has constractor section and not call
>>directly.
>>>>> ld always links library funtion with constractor section.
>>>>> use -Xlinker, or use ld instead of cc.
>>>>
>>>> Hello Oki-san,
>>>>
>>>> The trick to fix it was this, I finally found it in the example
>>Makefiles with
>>>> V=1 flag.
>>>>
>>>> -Wl,--whole-archive -Wl,--start-group -lintel_dpdk -Wl,--end-group
>>-Wl,--no-whole-archive
>>>>
>>>> Thank you for the advice you provided, I couldn't have fixed it
>>without your
>>>> suggestions... it got me to look more closely at the linking.
>>Importantly,
>>>> "-Wl,--whole-archive" includes the entire archive whether or not
>>it's called
>>>> from other objects, so we don't lose the constructors, just like
>you
>>said.
>>>>
>>>> Matthew.