Troubleshooting DPDK in Intel Ethernet NIC

Troubleshooting DPDK in Intel Ethernet NIC

[Antón Rey Villaverde]

[-- Attachment #1: Type: text/plain, Size: 3788 bytes --]

Hi,
I have a problem while trying to manage my physical Ethernet interface from
DPDK (latest version compiled from source).
I have a:
NIC: 0000:00:1f.6 Ethernet controller: Intel Corporation Ethernet
Connection (13) I219-LM (rev 20)

which belongs to the list of supported NICs in DPDK.

Also, I have bound my NICto vfio-pci, not the kernel:

dpdk-23.07$ usertools/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================ 0000:00:1f.6 'Ethernet
Connection (13) I219-LM 15fb' drv=vfio-pci unused=e1000e

I also enabled hugepages (2G).

when I enter "sudo build/app/dpdk-testpmd -c7 -- -i" I get:
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
testpmd: No probed ethernet devices
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=163456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Done
testpmd>

When I enter "show port info all" in the testpmd CLI I get nothing (no
ports).

When I start testpmd with "-a 0000:00:1f.6" (the address of the ethernet
NIC I want to manage from DPDK), The same happens (no ports).

If I try to start testpmd with "-d build/drivers/librte_net_e1000.so" (the
shared library of the driver of that device), I get a crash:

dpdk-23.07$ sudo build/app/dpdk-testpmd -c7 -a 0000:00:1f.6 -d
build/drivers/librte_net_e1000.so -- -i
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: UIO_RESOURCE_LIST tailq is already registered PANIC in
tailqinitfn_rte_uio_tailq(): Cannot initialize tailq: UIO_RESOURCE_LIST
0: build/app/dpdk-testpmd (rte_dump_stack+0x32) [55be1839afe2]
1: build/app/dpdk-testpmd (__rte_panic+0xf1) [55be18369e68]
2: /home/anton/tdr/dpdk-23.07/build/drivers/librte_bus_pci.so.23
(7f52153f6000+0x5d86) [7f52153fbd86]
3: /lib64/ld-linux-x86-64.so.2 (7f5215f90000+0x11b9a) [7f5215fa1b9a]
4: /lib64/ld-linux-x86-64.so.2 (7f5215f90000+0x11ca1) [7f5215fa1ca1]
5: /lib/x86_64-linux-gnu/libc.so.6 (_dl_catch_exception+0xe5)
[7f5215970985]
6: /lib64/ld-linux-x86-64.so.2 (7f5215f90000+0x160cf) [7f5215fa60cf]
7: /lib/x86_64-linux-gnu/libc.so.6 (_dl_catch_exception+0x88)
[7f5215970928]
8: /lib64/ld-linux-x86-64.so.2 (7f5215f90000+0x1560a) [7f5215fa560a]
9: /lib/x86_64-linux-gnu/libdl.so.2 (7f5215e15000+0x134c) [7f5215e1634c]
10: /lib/x86_64-linux-gnu/libc.so.6 (_dl_catch_exception+0x88)
[7f5215970928]
11: /lib/x86_64-linux-gnu/libc.so.6 (_dl_catch_error+0x33) [7f52159709f3]
12: /lib/x86_64-linux-gnu/libdl.so.2 (7f5215e15000+0x1b59) [7f5215e16b59]
13: /lib/x86_64-linux-gnu/libdl.so.2 (dlopen+0x4a) [7f5215e163da]
14: build/app/dpdk-testpmd (55be17b79000+0x8013ee) [55be1837a3ee]
15: build/app/dpdk-testpmd (eal_plugins_init+0x14f) [55be1837a6d8]
16: build/app/dpdk-testpmd (rte_eal_init+0x1af) [55be1839ec21]
17: build/app/dpdk-testpmd (main+0xde) [55be17da108b]
18: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xf3) [7f5215834083]
19: build/app/dpdk-testpmd (_start+0x2e) [55be17cef1fe]
Aborted

I have also checked that:
I have IOMMU enabled:
cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.15.0-83-generic
root=UUID=3c080ff1-1c00-4b5c-aa00-2084f9e68794 ro i915.enable_psr=1 quiet
splash intel_iommu=on vt.handoff=7
and other virtualization flags enabled (VT-x, VT-d), and the Secure Boot is
disabled.
Also the NIC works fine in normal circumstances (managed by the OS kernel).

I am running Ubuntu 20.04.6 on a Dell Precision 3560 laptop (1 year old).

I don't know whether this is a bug or I missed something, so I would
appreciate any guidance. Thanks in advance

Anton

[-- Attachment #2: Type: text/html, Size: 4524 bytes --]

Re: Troubleshooting DPDK in Intel Ethernet NIC

[David Marchand]

Hello,

On Fri, Sep 8, 2023 at 8:42 AM Antón Rey Villaverde
<anton.rey@cactussoft.biz> wrote:
>
> Hi,
> I have a problem while trying to manage my physical Ethernet interface from DPDK (latest version compiled from source).
> I have a:
> NIC: 0000:00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (13) I219-LM (rev 20)
>
> which belongs to the list of supported NICs in DPDK.
>
> Also, I have bound my NICto vfio-pci, not the kernel:
>
> dpdk-23.07$ usertools/dpdk-devbind.py --status
> Network devices using DPDK-compatible driver ============================================ 0000:00:1f.6 'Ethernet Connection (13) I219-LM 15fb' drv=vfio-pci unused=e1000e
>
> I also enabled hugepages (2G).
>
> when I enter "sudo build/app/dpdk-testpmd -c7 -- -i" I get:
> EAL: Detected CPU lcores: 8
> EAL: Detected NUMA nodes: 1
> EAL: Detected static linkage of DPDK
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> EAL: VFIO support initialized
> TELEMETRY: No legacy callbacks, legacy socket not created
> testpmd: No probed ethernet devices
> Interactive-mode selected
> testpmd: create a new mbuf pool <mb_pool_0>: n=163456, size=2176, socket=0
> testpmd: preferred mempool ops selected: ring_mp_mc
> Done
> testpmd>
>
> When I enter "show port info all" in the testpmd CLI I get nothing (no ports).
>
> When I start testpmd with "-a 0000:00:1f.6" (the address of the ethernet NIC I want to manage from DPDK), The same happens (no ports).

Afaics, this nic is not supported by DPDK drivers.
You will have to find some other nic, or run on another server with
supported nics.
Some details below.


For a statically linked DPDK application (which is the default for
testpmd), you can check with dpdk-pmd-info.py which PCI devices are
supported.
Example:
$ ./usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | grep -i 15fb


>
> If I try to start testpmd with "-d build/drivers/librte_net_e1000.so" (the shared library of the driver of that device), I get a crash:
>
> dpdk-23.07$ sudo build/app/dpdk-testpmd -c7 -a 0000:00:1f.6 -d build/drivers/librte_net_e1000.so -- -i
> EAL: Detected CPU lcores: 8
> EAL: Detected NUMA nodes: 1
> EAL: Detected static linkage of DPDK

Your testpmd binary has been statically linked (as I said above, this
is the default) and you can see the log message confirming it.


> EAL: UIO_RESOURCE_LIST tailq is already registered PANIC in tailqinitfn_rte_uio_tailq(): Cannot initialize tailq: UIO_RESOURCE_LIST
> 0: build/app/dpdk-testpmd (rte_dump_stack+0x32) [55be1839afe2]
> 1: build/app/dpdk-testpmd (__rte_panic+0xf1) [55be18369e68]
> 2: /home/anton/tdr/dpdk-23.07/build/drivers/librte_bus_pci.so.23 (7f52153f6000+0x5d86) [7f52153fbd86]
> 3: /lib64/ld-linux-x86-64.so.2 (7f5215f90000+0x11b9a) [7f5215fa1b9a]
> 4: /lib64/ld-linux-x86-64.so.2 (7f5215f90000+0x11ca1) [7f5215fa1ca1]
> 5: /lib/x86_64-linux-gnu/libc.so.6 (_dl_catch_exception+0xe5) [7f5215970985]
> 6: /lib64/ld-linux-x86-64.so.2 (7f5215f90000+0x160cf) [7f5215fa60cf]
> 7: /lib/x86_64-linux-gnu/libc.so.6 (_dl_catch_exception+0x88) [7f5215970928]
> 8: /lib64/ld-linux-x86-64.so.2 (7f5215f90000+0x1560a) [7f5215fa560a]
> 9: /lib/x86_64-linux-gnu/libdl.so.2 (7f5215e15000+0x134c) [7f5215e1634c]
> 10: /lib/x86_64-linux-gnu/libc.so.6 (_dl_catch_exception+0x88) [7f5215970928]
> 11: /lib/x86_64-linux-gnu/libc.so.6 (_dl_catch_error+0x33) [7f52159709f3]
> 12: /lib/x86_64-linux-gnu/libdl.so.2 (7f5215e15000+0x1b59) [7f5215e16b59]
> 13: /lib/x86_64-linux-gnu/libdl.so.2 (dlopen+0x4a) [7f5215e163da]
> 14: build/app/dpdk-testpmd (55be17b79000+0x8013ee) [55be1837a3ee]
> 15: build/app/dpdk-testpmd (eal_plugins_init+0x14f) [55be1837a6d8]
> 16: build/app/dpdk-testpmd (rte_eal_init+0x1af) [55be1839ec21]
> 17: build/app/dpdk-testpmd (main+0xde) [55be17da108b]
> 18: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xf3) [7f5215834083]
> 19: build/app/dpdk-testpmd (_start+0x2e) [55be17cef1fe]
> Aborted

This is not a crash, EAL willingly aborts because initialising UIO
shared tailq has been requested twice and this is not something
supported.
This happened as you requested to load a driver that is embedded in
the testpmd binary and has already been initialised.


-- 
David Marchand

Re: Troubleshooting DPDK in Intel Ethernet NIC

[Antón Rey Villaverde]

[-- Attachment #1: Type: text/plain, Size: 5149 bytes --]

Hi, thanks for the quick response and the feedback.

*> Afaics, this nic is not supported by DPDK drivers.*

Indeed there is nothing about "15fb" (the PCI ID of my NIC) in the output
of ./usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd , so I guess it is
not supported.

On the other hand, I do see that the* I219* family of my NIC (I219-LM) is
present in the list https://core.dpdk.org/supported/nics/intel/ under
e1000e category.

Is this a contradiction or am I missing something?

Regards

On Fri, Sep 8, 2023 at 9:11 AM David Marchand <david.marchand@redhat.com>
wrote:

> Hello,
>
> On Fri, Sep 8, 2023 at 8:42 AM Antón Rey Villaverde
> <anton.rey@cactussoft.biz> wrote:
> >
> > Hi,
> > I have a problem while trying to manage my physical Ethernet interface
> from DPDK (latest version compiled from source).
> > I have a:
> > NIC: 0000:00:1f.6 Ethernet controller: Intel Corporation Ethernet
> Connection (13) I219-LM (rev 20)
> >
> > which belongs to the list of supported NICs in DPDK.
> >
> > Also, I have bound my NICto vfio-pci, not the kernel:
> >
> > dpdk-23.07$ usertools/dpdk-devbind.py --status
> > Network devices using DPDK-compatible driver
> ============================================ 0000:00:1f.6 'Ethernet
> Connection (13) I219-LM 15fb' drv=vfio-pci unused=e1000e
> >
> > I also enabled hugepages (2G).
> >
> > when I enter "sudo build/app/dpdk-testpmd -c7 -- -i" I get:
> > EAL: Detected CPU lcores: 8
> > EAL: Detected NUMA nodes: 1
> > EAL: Detected static linkage of DPDK
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: Selected IOVA mode 'VA'
> > EAL: VFIO support initialized
> > TELEMETRY: No legacy callbacks, legacy socket not created
> > testpmd: No probed ethernet devices
> > Interactive-mode selected
> > testpmd: create a new mbuf pool <mb_pool_0>: n=163456, size=2176,
> socket=0
> > testpmd: preferred mempool ops selected: ring_mp_mc
> > Done
> > testpmd>
> >
> > When I enter "show port info all" in the testpmd CLI I get nothing (no
> ports).
> >
> > When I start testpmd with "-a 0000:00:1f.6" (the address of the ethernet
> NIC I want to manage from DPDK), The same happens (no ports).
>
> Afaics, this nic is not supported by DPDK drivers.
> You will have to find some other nic, or run on another server with
> supported nics.
> Some details below.
>
>
> For a statically linked DPDK application (which is the default for
> testpmd), you can check with dpdk-pmd-info.py which PCI devices are
> supported.
> Example:
> $ ./usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | grep -i 15fb
>
>
> >
> > If I try to start testpmd with "-d build/drivers/librte_net_e1000.so"
> (the shared library of the driver of that device), I get a crash:
> >
> > dpdk-23.07$ sudo build/app/dpdk-testpmd -c7 -a 0000:00:1f.6 -d
> build/drivers/librte_net_e1000.so -- -i
> > EAL: Detected CPU lcores: 8
> > EAL: Detected NUMA nodes: 1
> > EAL: Detected static linkage of DPDK
>
> Your testpmd binary has been statically linked (as I said above, this
> is the default) and you can see the log message confirming it.
>
>
> > EAL: UIO_RESOURCE_LIST tailq is already registered PANIC in
> tailqinitfn_rte_uio_tailq(): Cannot initialize tailq: UIO_RESOURCE_LIST
> > 0: build/app/dpdk-testpmd (rte_dump_stack+0x32) [55be1839afe2]
> > 1: build/app/dpdk-testpmd (__rte_panic+0xf1) [55be18369e68]
> > 2: /home/anton/tdr/dpdk-23.07/build/drivers/librte_bus_pci.so.23
> (7f52153f6000+0x5d86) [7f52153fbd86]
> > 3: /lib64/ld-linux-x86-64.so.2 (7f5215f90000+0x11b9a) [7f5215fa1b9a]
> > 4: /lib64/ld-linux-x86-64.so.2 (7f5215f90000+0x11ca1) [7f5215fa1ca1]
> > 5: /lib/x86_64-linux-gnu/libc.so.6 (_dl_catch_exception+0xe5)
> [7f5215970985]
> > 6: /lib64/ld-linux-x86-64.so.2 (7f5215f90000+0x160cf) [7f5215fa60cf]
> > 7: /lib/x86_64-linux-gnu/libc.so.6 (_dl_catch_exception+0x88)
> [7f5215970928]
> > 8: /lib64/ld-linux-x86-64.so.2 (7f5215f90000+0x1560a) [7f5215fa560a]
> > 9: /lib/x86_64-linux-gnu/libdl.so.2 (7f5215e15000+0x134c) [7f5215e1634c]
> > 10: /lib/x86_64-linux-gnu/libc.so.6 (_dl_catch_exception+0x88)
> [7f5215970928]
> > 11: /lib/x86_64-linux-gnu/libc.so.6 (_dl_catch_error+0x33) [7f52159709f3]
> > 12: /lib/x86_64-linux-gnu/libdl.so.2 (7f5215e15000+0x1b59) [7f5215e16b59]
> > 13: /lib/x86_64-linux-gnu/libdl.so.2 (dlopen+0x4a) [7f5215e163da]
> > 14: build/app/dpdk-testpmd (55be17b79000+0x8013ee) [55be1837a3ee]
> > 15: build/app/dpdk-testpmd (eal_plugins_init+0x14f) [55be1837a6d8]
> > 16: build/app/dpdk-testpmd (rte_eal_init+0x1af) [55be1839ec21]
> > 17: build/app/dpdk-testpmd (main+0xde) [55be17da108b]
> > 18: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xf3)
> [7f5215834083]
> > 19: build/app/dpdk-testpmd (_start+0x2e) [55be17cef1fe]
> > Aborted
>
> This is not a crash, EAL willingly aborts because initialising UIO
> shared tailq has been requested twice and this is not something
> supported.
> This happened as you requested to load a driver that is embedded in
> the testpmd binary and has already been initialised.
>
>
> --
> David Marchand
>
>

[-- Attachment #2: Type: text/html, Size: 6117 bytes --]

Re: Troubleshooting DPDK in Intel Ethernet NIC

[David Marchand]

On Fri, Sep 8, 2023 at 11:05 AM Antón Rey Villaverde
<anton.rey@cactussoft.biz> wrote:
> > Afaics, this nic is not supported by DPDK drivers.
>
> Indeed there is nothing about "15fb" (the PCI ID of my NIC) in the output of ./usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd , so I guess it is not supported.
>
> On the other hand, I do see that the I219 family of my NIC (I219-LM) is present in the list https://core.dpdk.org/supported/nics/intel/ under e1000e category.
>
> Is this a contradiction or am I missing something?

I reached the point where I can't help :-).

Some I219 nics support has been added ealier this year (commit
a33e1a5bcd3f ("net/e1000: support more I219 devices")).
Maybe adding support for your nic is a simple matter, but passing the
ball to Intel maintainers who are the right people to answer.


-- 
David Marchand