[dpdk-users] DPDK 17.05 does not find X510 card

[dpdk-users] DPDK 17.05 does not find X510 card

[Martin Drašar]

Hello,

I have a CentOS 7 system with two 2-port X710 cards:

> [root@dhcp13-55 pain]# lspci -nn | grep X710
> 04:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)
> 04:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)
> 05:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)
> 05:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)
> [root@dhcp13-55 pain]# lib/dpdk/dpdk-17.05.1/usertools/dpdk-devbind.py --status
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:04:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused=
> 0000:04:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused=
> 0000:05:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused=
> 0000:05:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused=

According to tth 17.05 release notes, the cards should work. However,
when I run the dpdk_init(), cards are not identified, and
rte_eth_dev_count() returns 0.

Do you have any idea, what may be the cause or how should I proceed to
debug it?

Thank you,
Martin

Re: [dpdk-users] DPDK 17.05 does not find X710 card

[Martin Drašar]

Sorry, there was a typo in the subject. X710, not X510.

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Rosen, Rami]

Hi Martin, 
> dpdk_init()

I assume you run your own application.
Can you post the log you get when running testpmd with --log-level=8 and -w I40E0_PCI_BUS_ID ....?

Also can you post "lspci -n"?
Regards,
Rami Rosen

 

-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Martin Drašar
Sent: Monday, June 18, 2018 11:56
To: users@dpdk.org
Subject: [dpdk-users] DPDK 17.05 does not find X510 card

Hello,

I have a CentOS 7 system with two 2-port X710 cards:

> [root@dhcp13-55 pain]# lspci -nn | grep X710
> 04:00.0 Ethernet controller [0200]: Intel Corporation Ethernet 
> Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)
> 04:00.1 Ethernet controller [0200]: Intel Corporation Ethernet 
> Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)
> 05:00.0 Ethernet controller [0200]: Intel Corporation Ethernet 
> Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)
> 05:00.1 Ethernet controller [0200]: Intel Corporation Ethernet 
> Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)
> [root@dhcp13-55 pain]# lib/dpdk/dpdk-17.05.1/usertools/dpdk-devbind.py 
> --status
> 
> Network devices using DPDK-compatible driver 
> ============================================
> 0000:04:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' 
> drv=igb_uio unused=
> 0000:04:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' 
> drv=igb_uio unused=
> 0000:05:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' 
> drv=igb_uio unused=
> 0000:05:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' 
> drv=igb_uio unused=

According to tth 17.05 release notes, the cards should work. However, when I run the dpdk_init(), cards are not identified, and
rte_eth_dev_count() returns 0.

Do you have any idea, what may be the cause or how should I proceed to debug it?

Thank you,
Martin

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Martin Drašar]

Dne 18.6.2018 v 11:54 Rosen, Rami napsal(a):
> Hi Martin, 
>> dpdk_init()
> 
> I assume you run your own application.
> Can you post the log you get when running testpmd with --log-level=8 and -w I40E0_PCI_BUS_ID ....?
> 
> Also can you post "lspci -n"?
> Regards,
> Rami Rosen

Hello,

thank you for getting back to me so quick. I did some debugging and it
turned out that i40e drivers were not linked when building the dpdk. I
fixed this, but I have encountered another problem...

Although the cards are discovered, I get this error:

> EAL: pci_map_resource(): cannot mmap(102, 0x7fffc0000000, 0x1000000, 0x0): Operation not permitted (0xffffffffffffffff)

I've seen some information about his mmap-related behavior and some
hints about setting iomem=relaxed in grub. As making changes to grub
means completely reinstalling my machine, I would like to ask, if you
think that this could be the way to go.

Thanks,
Martin

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Rosen, Rami]

Hi, Martin,
How do you bind the device ? is it with igb_uio/vfio  ? In case this is what you do - can you please try with uio_pci_generic and tell if you get the same error ? 
The uio_pci_generic should be included in your distro as one of the kernel modules, you need just to do modprobe uio_pci_generic before binding to DPDK.

Regards,
Rami Rosen


-----Original Message-----
From: Martin Drašar [mailto:drasar@ics.muni.cz] 
Sent: Monday, June 18, 2018 15:41
To: Rosen, Rami <rami.rosen@intel.com>; users@dpdk.org
Subject: Re: [dpdk-users] DPDK 17.05 does not find X510 card

Dne 18.6.2018 v 11:54 Rosen, Rami napsal(a):
> Hi Martin,
>> dpdk_init()
> 
> I assume you run your own application.
> Can you post the log you get when running testpmd with --log-level=8 and -w I40E0_PCI_BUS_ID ....?
> 
> Also can you post "lspci -n"?
> Regards,
> Rami Rosen

Hello,

thank you for getting back to me so quick. I did some debugging and it turned out that i40e drivers were not linked when building the dpdk. I fixed this, but I have encountered another problem...

Although the cards are discovered, I get this error:

> EAL: pci_map_resource(): cannot mmap(102, 0x7fffc0000000, 0x1000000, 
> 0x0): Operation not permitted (0xffffffffffffffff)

I've seen some information about his mmap-related behavior and some hints about setting iomem=relaxed in grub. As making changes to grub means completely reinstalling my machine, I would like to ask, if you think that this could be the way to go.

Thanks,
Martin

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Martin Drašar]

Dne 18.6.2018 v 15:09 Rosen, Rami napsal(a):
> Hi, Martin,
> How do you bind the device ? is it with igb_uio/vfio  ? In case this is what you do - can you please try with uio_pci_generic and tell if you get the same error ? 
> The uio_pci_generic should be included in your distro as one of the kernel modules, you need just to do modprobe uio_pci_generic before binding to DPDK.
> 
> Regards,
> Rami Rosen

Hello Rami,

yes the devices are bound using the igb_uio. I cannot use
uio_pci_generic, as the X710 is not supported in these drivers.

> [root@dhcp13-55 pain]# lsmod | grep uio
> uio_pci_generic        12588  0
> igb_uio                13224  0
> uio                    19259  2 uio_pci_generic,igb_uio

> [root@dhcp13-55 pain]# lib/dpdk/dpdk-17.05.1/usertools/dpdk-devbind.py -b uio_pci_generic 0000:04:00.0
> Error: bind failed for 0000:04:00.0 - Cannot bind to driver uio_pci_generic

This is listed in known issues:

https://doc.dpdk.org/guides/rel_notes/known_issues.html#uio-pci-generic-module-bind-failed-in-x710-xl710-xxv710

Regards,
Martin

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Rosen, Rami]

Hi Martin,

Yes, you are right.

Which kernel are you running ? (uname -r)

Regards,
Rami Rosen

-----Original Message-----
From: Martin Drašar [mailto:drasar@ics.muni.cz] 
Sent: Monday, June 18, 2018 16:23
To: Rosen, Rami <rami.rosen@intel.com>; users@dpdk.org
Subject: Re: [dpdk-users] DPDK 17.05 does not find X510 card

Dne 18.6.2018 v 15:09 Rosen, Rami napsal(a):
> Hi, Martin,
> How do you bind the device ? is it with igb_uio/vfio  ? In case this is what you do - can you please try with uio_pci_generic and tell if you get the same error ? 
> The uio_pci_generic should be included in your distro as one of the kernel modules, you need just to do modprobe uio_pci_generic before binding to DPDK.
> 
> Regards,
> Rami Rosen

Hello Rami,

yes the devices are bound using the igb_uio. I cannot use uio_pci_generic, as the X710 is not supported in these drivers.

> [root@dhcp13-55 pain]# lsmod | grep uio
> uio_pci_generic        12588  0
> igb_uio                13224  0
> uio                    19259  2 uio_pci_generic,igb_uio

> [root@dhcp13-55 pain]# lib/dpdk/dpdk-17.05.1/usertools/dpdk-devbind.py 
> -b uio_pci_generic 0000:04:00.0
> Error: bind failed for 0000:04:00.0 - Cannot bind to driver 
> uio_pci_generic

This is listed in known issues:

https://doc.dpdk.org/guides/rel_notes/known_issues.html#uio-pci-generic-module-bind-failed-in-x710-xl710-xxv710

Regards,
Martin

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Martin Drašar]

Dne 18.6.2018 v 15:34 Rosen, Rami napsal(a):
> Hi Martin,
> 
> Yes, you are right.
> 
> Which kernel are you running ? (uname -r)
> 
> Regards,
> Rami Rosen

I am running 3.10.0-693.el7.x86_64.

Martin

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Martin Drašar]

Using iomem=relaxed did not change anything. Once I manage to update
UEFI drivers on my X520, I will try it to see if it is a driver issue.

Martin

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Rosen, Rami]

Hi Martin,
I assume  that you get the mmap() error that you posted earlier when 
running your own DPDK application. And again, if this is the case, can
you please post here 
the full log you get when running testpmd with --log-level=8 and -w I40E0_PCI_BUS_ID .... (without running any other DPDK application). This may give a direction to solving the issue. Also please post 
here the exact command you use to launch testpmd.

Regards,
Rami Rosen





-----Original Message-----
From: Martin Drašar [mailto:drasar@ics.muni.cz] 
Sent: Tuesday, June 19, 2018 00:36
To: Rosen, Rami <rami.rosen@intel.com>; users@dpdk.org
Subject: Re: [dpdk-users] DPDK 17.05 does not find X510 card

Using iomem=relaxed did not change anything. Once I manage to update UEFI drivers on my X520, I will try it to see if it is a driver issue.

Martin

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Martin Drašar]

Dne 19.6.2018 v 2:21 Rosen, Rami napsal(a):
> Hi Martin,
> I assume  that you get the mmap() error that you posted earlier when 
> running your own DPDK application. And again, if this is the case, can
> you please post here 
> the full log you get when running testpmd with --log-level=8 and -w I40E0_PCI_BUS_ID .... (without running any other DPDK application). This may give a direction to solving the issue. Also please post 
> here the exact command you use to launch testpmd.
> 
> Regards,
> Rami Rosen

Hi,

I am atttaching the testpmd output. There is not much else than what I
posted earlier. I am launching the testpmd as you instructed:

sudo ./testpmd --log-level=8 -w 0000:04:00.0 -w 0000:04:00.1 -w
0000:05:00.0 -w 0000:05:00.1

I have also swapped the 05:00 card for X520. So, now the system has one
X710 and one X520 using different drivers, but the situation is the same.

This system has the secure boot on, so I guess it may be the culprit,
because the same application is working with the same cards on an
unrestricted system. However, the only advice I found so far regarding
the 'iomem=relaxed' did not help, so I have currently no idea how to
proceed.

Thank you for any advice.

Martin
-------------- next part --------------
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 1
EAL: Detected lcore 2 as core 1 on socket 0
EAL: Detected lcore 3 as core 1 on socket 1
EAL: Detected lcore 4 as core 2 on socket 0
EAL: Detected lcore 5 as core 2 on socket 1
EAL: Detected lcore 6 as core 3 on socket 0
EAL: Detected lcore 7 as core 3 on socket 1
EAL: Detected lcore 8 as core 4 on socket 0
EAL: Detected lcore 9 as core 4 on socket 1
EAL: Detected lcore 10 as core 5 on socket 0
EAL: Detected lcore 11 as core 5 on socket 1
EAL: Detected lcore 12 as core 6 on socket 0
EAL: Detected lcore 13 as core 6 on socket 1
EAL: Detected lcore 14 as core 8 on socket 0
EAL: Detected lcore 15 as core 8 on socket 1
EAL: Detected lcore 16 as core 9 on socket 0
EAL: Detected lcore 17 as core 9 on socket 1
EAL: Detected lcore 18 as core 10 on socket 0
EAL: Detected lcore 19 as core 10 on socket 1
EAL: Detected lcore 20 as core 11 on socket 0
EAL: Detected lcore 21 as core 11 on socket 1
EAL: Detected lcore 22 as core 12 on socket 0
EAL: Detected lcore 23 as core 12 on socket 1
EAL: Detected lcore 24 as core 13 on socket 0
EAL: Detected lcore 25 as core 13 on socket 1
EAL: Detected lcore 26 as core 14 on socket 0
EAL: Detected lcore 27 as core 14 on socket 1
EAL: Detected lcore 28 as core 0 on socket 0
EAL: Detected lcore 29 as core 0 on socket 1
EAL: Detected lcore 30 as core 1 on socket 0
EAL: Detected lcore 31 as core 1 on socket 1
EAL: Detected lcore 32 as core 2 on socket 0
EAL: Detected lcore 33 as core 2 on socket 1
EAL: Detected lcore 34 as core 3 on socket 0
EAL: Detected lcore 35 as core 3 on socket 1
EAL: Detected lcore 36 as core 4 on socket 0
EAL: Detected lcore 37 as core 4 on socket 1
EAL: Detected lcore 38 as core 5 on socket 0
EAL: Detected lcore 39 as core 5 on socket 1
EAL: Detected lcore 40 as core 6 on socket 0
EAL: Detected lcore 41 as core 6 on socket 1
EAL: Detected lcore 42 as core 8 on socket 0
EAL: Detected lcore 43 as core 8 on socket 1
EAL: Detected lcore 44 as core 9 on socket 0
EAL: Detected lcore 45 as core 9 on socket 1
EAL: Detected lcore 46 as core 10 on socket 0
EAL: Detected lcore 47 as core 10 on socket 1
EAL: Detected lcore 48 as core 11 on socket 0
EAL: Detected lcore 49 as core 11 on socket 1
EAL: Detected lcore 50 as core 12 on socket 0
EAL: Detected lcore 51 as core 12 on socket 1
EAL: Detected lcore 52 as core 13 on socket 0
EAL: Detected lcore 53 as core 13 on socket 1
EAL: Detected lcore 54 as core 14 on socket 0
EAL: Detected lcore 55 as core 14 on socket 1
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 56 lcore(s)
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory)
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0xe00000000 bytes
EAL: Virtual area found at 0x7fb000000000 (size = 0xe00000000)
EAL: Ask a virtual area of 0xe00000000 bytes
EAL: Virtual area found at 0x7fa1c0000000 (size = 0xe00000000)
EAL: Requesting 56 pages of size 1024MB from socket 0
EAL: Requesting 56 pages of size 1024MB from socket 1
EAL: TSC frequency is ~1997675 KHz
EAL: Master lcore 0 is ready (tid=c335940;cpuset=[0])
EAL: lcore 8 is ready (tid=715f700;cpuset=[8])
EAL: lcore 12 is ready (tid=515b700;cpuset=[12])
EAL: lcore 22 is ready (tid=fffff700;cpuset=[22])
EAL: lcore 38 is ready (tid=f7fef700;cpuset=[38])
EAL: lcore 34 is ready (tid=f9ff3700;cpuset=[34])
EAL: lcore 10 is ready (tid=615d700;cpuset=[10])
EAL: lcore 6 is ready (tid=8161700;cpuset=[6])
EAL: lcore 30 is ready (tid=fbff7700;cpuset=[30])
EAL: lcore 54 is ready (tid=effdf700;cpuset=[54])
EAL: lcore 18 is ready (tid=2155700;cpuset=[18])
EAL: lcore 28 is ready (tid=fcff9700;cpuset=[28])
EAL: lcore 50 is ready (tid=f1fe3700;cpuset=[50])
EAL: lcore 26 is ready (tid=fdffb700;cpuset=[26])
EAL: lcore 46 is ready (tid=f3fe7700;cpuset=[46])
EAL: lcore 16 is ready (tid=3157700;cpuset=[16])
EAL: lcore 52 is ready (tid=f0fe1700;cpuset=[52])
EAL: lcore 14 is ready (tid=4159700;cpuset=[14])
EAL: lcore 7 is ready (tid=7960700;cpuset=[7])
EAL: lcore 49 is ready (tid=f27e4700;cpuset=[49])
EAL: lcore 31 is ready (tid=fb7f6700;cpuset=[31])
EAL: lcore 3 is ready (tid=9964700;cpuset=[3])
EAL: lcore 32 is ready (tid=faff5700;cpuset=[32])
EAL: lcore 11 is ready (tid=595c700;cpuset=[11])
EAL: lcore 40 is ready (tid=f6fed700;cpuset=[40])
EAL: lcore 4 is ready (tid=9163700;cpuset=[4])
EAL: lcore 9 is ready (tid=695e700;cpuset=[9])
EAL: lcore 24 is ready (tid=feffd700;cpuset=[24])
EAL: lcore 37 is ready (tid=f87f0700;cpuset=[37])
EAL: lcore 1 is ready (tid=a966700;cpuset=[1])
EAL: lcore 47 is ready (tid=f37e6700;cpuset=[47])
EAL: lcore 45 is ready (tid=f47e8700;cpuset=[45])
EAL: lcore 21 is ready (tid=952700;cpuset=[21])
EAL: lcore 29 is ready (tid=fc7f8700;cpuset=[29])
EAL: lcore 35 is ready (tid=f97f2700;cpuset=[35])
EAL: lcore 5 is ready (tid=8962700;cpuset=[5])
EAL: lcore 53 is ready (tid=f07e0700;cpuset=[53])
EAL: lcore 51 is ready (tid=f17e2700;cpuset=[51])
EAL: lcore 2 is ready (tid=a165700;cpuset=[2])
EAL: lcore 44 is ready (tid=f4fe9700;cpuset=[44])
EAL: lcore 20 is ready (tid=1153700;cpuset=[20])
EAL: lcore 33 is ready (tid=fa7f4700;cpuset=[33])
EAL: lcore 13 is ready (tid=495a700;cpuset=[13])
EAL: lcore 39 is ready (tid=f77ee700;cpuset=[39])
EAL: lcore 43 is ready (tid=f57ea700;cpuset=[43])
EAL: lcore 42 is ready (tid=f5feb700;cpuset=[42])
EAL: lcore 55 is ready (tid=ef7de700;cpuset=[55])
EAL: lcore 48 is ready (tid=f2fe5700;cpuset=[48])
EAL: lcore 27 is ready (tid=fd7fa700;cpuset=[27])
EAL: lcore 36 is ready (tid=f8ff1700;cpuset=[36])
EAL: lcore 25 is ready (tid=fe7fc700;cpuset=[25])
EAL: lcore 23 is ready (tid=ff7fe700;cpuset=[23])
EAL: lcore 15 is ready (tid=3958700;cpuset=[15])
EAL: lcore 41 is ready (tid=f67ec700;cpuset=[41])
EAL: lcore 17 is ready (tid=2956700;cpuset=[17])
EAL: lcore 19 is ready (tid=1954700;cpuset=[19])
EAL: Driver (net_ark) doesn't match the device
EAL: Driver (net_avp) doesn't match the device
EAL: Driver (net_bnxt) doesn't match the device
EAL: Driver (net_cxgbe) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: Driver (net_e1000_em) doesn't match the device
EAL: Driver (net_ena) doesn't match the device
EAL: Driver (net_enic) doesn't match the device
EAL: Driver (net_fm10k) doesn't match the device
EAL: PCI device 0000:04:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1572 net_i40e
EAL: pci_map_resource(): cannot mmap(231, 0x7fbe00000000, 0x1000000, 0x0): Operation not permitted (0xffffffffffffffff)
EAL: Requested device 0000:04:00.0 cannot be used
EAL: Driver (net_ark) doesn't match the device
EAL: Driver (net_avp) doesn't match the device
EAL: Driver (net_bnxt) doesn't match the device
EAL: Driver (net_cxgbe) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: Driver (net_e1000_em) doesn't match the device
EAL: Driver (net_ena) doesn't match the device
EAL: Driver (net_enic) doesn't match the device
EAL: Driver (net_fm10k) doesn't match the device
EAL: PCI device 0000:04:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1572 net_i40e
EAL: pci_map_resource(): cannot mmap(231, 0x7fbe00000000, 0x1000000, 0x0): Operation not permitted (0xffffffffffffffff)
EAL: Requested device 0000:04:00.1 cannot be used
EAL: Driver (net_ark) doesn't match the device
EAL: Driver (net_avp) doesn't match the device
EAL: Driver (net_bnxt) doesn't match the device
EAL: Driver (net_cxgbe) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: Driver (net_e1000_em) doesn't match the device
EAL: Driver (net_ena) doesn't match the device
EAL: Driver (net_enic) doesn't match the device
EAL: Driver (net_fm10k) doesn't match the device
EAL: Driver (net_i40e) doesn't match the device
EAL: Driver (net_i40e_vf) doesn't match the device
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL:   probe driver: 8086:154d net_ixgbe
EAL: pci_map_resource(): cannot mmap(231, 0x7fbe00000000, 0x80000, 0x0): Operation not permitted (0xffffffffffffffff)
EAL: Requested device 0000:05:00.0 cannot be used
EAL: Driver (net_ark) doesn't match the device
EAL: Driver (net_avp) doesn't match the device
EAL: Driver (net_bnxt) doesn't match the device
EAL: Driver (net_cxgbe) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: Driver (net_e1000_em) doesn't match the device
EAL: Driver (net_ena) doesn't match the device
EAL: Driver (net_enic) doesn't match the device
EAL: Driver (net_fm10k) doesn't match the device
EAL: Driver (net_i40e) doesn't match the device
EAL: Driver (net_i40e_vf) doesn't match the device
EAL: PCI device 0000:05:00.1 on NUMA socket 0
EAL:   probe driver: 8086:154d net_ixgbe
EAL: pci_map_resource(): cannot mmap(231, 0x7fbe00000000, 0x80000, 0x0): Operation not permitted (0xffffffffffffffff)
EAL: Requested device 0000:05:00.1 cannot be used
EAL: No probed ethernet devices
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=587456, size=2176, socket=0
USER1: create a new mbuf pool <mbuf_pool_socket_1>: n=587456, size=2176, socket=1
Done
No commandline core given, start packet forwarding
io packet forwarding - ports=0 - cores=0 - streams=0 - NUMA support enabled, MP over anonymous pages disabled

  io packet forwarding - CRC stripping enabled - packets/burst=32
  nb forwarding cores=1 - nb forwarding ports=0
  RX queues=1 - RX desc=128 - RX free threshold=0
  RX threshold registers: pthresh=0 hthresh=0 wthresh=0
  TX queues=1 - TX desc=512 - TX free threshold=0
  TX threshold registers: pthresh=0 hthresh=0 wthresh=0
  TX RS bit threshold=0 - TXQ flags=0x0
Press enter to exit
Telling cores to stop...
Waiting for lcores to finish...

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.

Bye...

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Martin Drašar]

Just some additional details (strace around the pic probe):

> write(1, "EAL: PCI device 0000:05:00.0 on "..., 46EAL: PCI device 0000:05:00.0 on NUMA socket 0) = 46
> sendto(3, "<30>Jun 19 16:05:21 framework[42"..., 84, MSG_NOSIGNAL, NULL, 0) = 84
> write(1, "EAL:   probe driver: 8086:154d n"..., 41EAL:   probe driver: 8086:154d net_ixgbe) = 41
> sendto(3, "<30>Jun 19 16:05:21 framework[42"..., 79, MSG_NOSIGNAL, NULL, 0) = 79
> openat(AT_FDCWD, "/sys/bus/pci/devices/0000:05:00.0/uio", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 94
> getdents(94, /* 3 entries */, 32768)    = 72
> close(94)                               = 0
> open("/dev/uio2", O_RDWR)               = 94
> open("/sys/class/uio/uio2/device/config", O_RDWR) = 95
> open("/sys/bus/pci/devices/0000:05:00.0/resource0", O_RDWR) = 96
> mmap(0x7fa400000000, 524288, PROT_READ|PROT_WRITE, MAP_SHARED, 96, 0) = -1 EPERM (Operation not permitted)
> write(1, "EAL: pci_map_resource(): cannot "..., 117EAL: pci_map_resource(): cannot mmap(96, 0x7fa400000000, 0x80000, 0x0): Operation not permitted (0xffffffffffffffff)) = 117
> sendto(3, "<27>Jun 19 16:05:21 framework[42"..., 155, MSG_NOSIGNAL, NULL, 0) = 155
> close(96)                               = 0
> close(95)                               = 0
> close(94)                               = 0
> write(1, "EAL: Requested device 0000:05:00"..., 50EAL: Requested device 0000:05:00.0 cannot be used) = 50

Martin

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Avi Cohen (A)]

Look at this 
https://stackoverflow.com/questions/8213671/mmap-operation-not-permitted


> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Martin Dra?ar
> Sent: Tuesday, 19 June, 2018 5:13 PM
> To: Rosen, Rami; users@dpdk.org
> Subject: Re: [dpdk-users] DPDK 17.05 does not find X510 card
> 
> Just some additional details (strace around the pic probe):
> 
> > write(1, "EAL: PCI device 0000:05:00.0 on "..., 46EAL: PCI device
> > 0000:05:00.0 on NUMA socket 0) = 46 sendto(3, "<30>Jun 19 16:05:21
> framework[42"..., 84, MSG_NOSIGNAL, NULL, 0) = 84
> > write(1, "EAL:   probe driver: 8086:154d n"..., 41EAL:   probe driver:
> 8086:154d net_ixgbe) = 41
> > sendto(3, "<30>Jun 19 16:05:21 framework[42"..., 79, MSG_NOSIGNAL,
> > NULL, 0) = 79 openat(AT_FDCWD, "/sys/bus/pci/devices/0000:05:00.0/uio",
> O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 94
> > getdents(94, /* 3 entries */, 32768)    = 72
> > close(94)                               = 0
> > open("/dev/uio2", O_RDWR)               = 94
> > open("/sys/class/uio/uio2/device/config", O_RDWR) = 95
> > open("/sys/bus/pci/devices/0000:05:00.0/resource0", O_RDWR) = 96
> > mmap(0x7fa400000000, 524288, PROT_READ|PROT_WRITE, MAP_SHARED,
> 96, 0)
> > = -1 EPERM (Operation not permitted) write(1, "EAL:
> > pci_map_resource(): cannot "..., 117EAL: pci_map_resource(): cannot
> mmap(96, 0x7fa400000000, 0x80000, 0x0): Operation not permitted
> (0xffffffffffffffff)) = 117 sendto(3, "<27>Jun 19 16:05:21 framework[42"..., 155,
> MSG_NOSIGNAL, NULL, 0) = 155
> > close(96)                               = 0
> > close(95)                               = 0
> > close(94)                               = 0
> > write(1, "EAL: Requested device 0000:05:00"..., 50EAL: Requested
> > device 0000:05:00.0 cannot be used) = 50
> 
> Martin

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Martin Drašar]

Dne 21.6.2018 v 7:58 Avi Cohen (A) napsal(a):
> Look at this 
> https://stackoverflow.com/questions/8213671/mmap-operation-not-permitted

Hello Avi,

thank you for chiming in. Neither iomem runtime parameter or allowing
mmap on 0 page helped me. As for the latter, DPDK correctly maps to PCI
region, so that is not the problem.

Looking at the mmap source, there are basically only two points that
result in EPERM error. First one is related to incorrect flag
combination (not a DPDK case) the second is related to the file
descriptor of the resource being locked (should not be, but it is the
only possibility).

That lock may have to do something with Secure boot being on, as almost
the same system without secure boot is working ok. I built a custom
kernel with CONFIG_STRICT_DEVMEM turned off, but it's going to take me
some time to hammer it into the system, mainly because I am leaving for
a vacation today.

But it would be great if anyone, who has experience running DPDK on
secure booted systems, could share their experience.

Cheers,
Martin

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Stephen Hemminger]

On Thu, 21 Jun 2018 13:02:34 +0200
Martin Drašar <drasar@ics.muni.cz> wrote:

> Dne 21.6.2018 v 7:58 Avi Cohen (A) napsal(a):
> > Look at this 
> > https://stackoverflow.com/questions/8213671/mmap-operation-not-permitted  
> 
> Hello Avi,
> 
> thank you for chiming in. Neither iomem runtime parameter or allowing
> mmap on 0 page helped me. As for the latter, DPDK correctly maps to PCI
> region, so that is not the problem.
> 
> Looking at the mmap source, there are basically only two points that
> result in EPERM error. First one is related to incorrect flag
> combination (not a DPDK case) the second is related to the file
> descriptor of the resource being locked (should not be, but it is the
> only possibility).
> 
> That lock may have to do something with Secure boot being on, as almost
> the same system without secure boot is working ok. I built a custom
> kernel with CONFIG_STRICT_DEVMEM turned off, but it's going to take me
> some time to hammer it into the system, mainly because I am leaving for
> a vacation today.
> 
> But it would be great if anyone, who has experience running DPDK on
> secure booted systems, could share their experience.

Secure boot disables direct access to memory which is required for DPDK
via uio. This is by design since UIO would allow userspace (as root)
to read or write any location in memory.

Secure boot should work with VFIO. As long as you have an IOMMU,
VFIO is a much better solution.

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Rosen, Rami]

Hi all,

>Secure boot should work with VFIO. As long as you have an IOMMU, 
>VFIO is a much better solution.

Stephen is right; moreover, DPDK documentation says explicitly that
...
"If UEFI secure boot is enabled, the Linux kernel may disallow the use of UIO on the system. Therefore, devices for use by DPDK should be bound to the vfio-pci kernel module rather than igb_uio or uio_pci_generic. For more details see Binding and Unbinding Network Ports to/from the Kernel Modules below."
...
See section 4.1, "UIO", in:
https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html

Regards,
Rami Rosen

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Martin Drašar]

Hi,

Stephen, thanks four input. I have already read that part of documentation Rami posted, but that ‘may disallow’ left me hoping there are some situations when it is not disallowed. If it is indeed not possible using the igb uio, I will have to do it with the vfio. However, and that is the reason why I was doing the dances with the kernel, I was not able to find a guide to bind the cards to vfio in non-virtual environent, other than a notice that all ports must be bound to vfio at once. 

Can you point me to some resources regarding the iommu and vfio?

Thanks, 
Martin

23. 6. 2018 v 15:00, Rosen, Rami <rami.rosen@intel.com>:

> Hi all,
> 
>> Secure boot should work with VFIO. As long as you have an IOMMU, 
>> VFIO is a much better solution.
> 
> Stephen is right; moreover, DPDK documentation says explicitly that
> ...
> "If UEFI secure boot is enabled, the Linux kernel may disallow the use of UIO on the system. Therefore, devices for use by DPDK should be bound to the vfio-pci kernel module rather than igb_uio or uio_pci_generic. For more details see Binding and Unbinding Network Ports to/from the Kernel Modules below."
> ...
> See section 4.1, "UIO", in:
> https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html
> 
> Regards,
> Rami Rosen
> 
> 

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Rosen, Rami]

Hi Martin,

There is info about VFIO and IOMMU for DPDK in the 
Linux Getting Started Guide:
https://doc.dpdk.org/guides/linux_gsg/index.html


Generally, you need to follow section 8.1.3 for IOMMU:
...
8.1.3 Linux boot command line

If it wants to use VFIO, use the following additional grub parameters:
iommu=pt intel_iommu=on
...
And follow section 4.2 (VFIO)
...
For VFIO you need modprobe vfio-pci
...

And use it with --bind  when running the dpdk-devnbind.py script.

Regards,
Rami Rosen


-----Original Message-----
From: Martin Drašar [mailto:drasar@ics.muni.cz] 
Sent: Monday, June 25, 2018 11:24
To: Rosen, Rami <rami.rosen@intel.com>
Cc: Stephen Hemminger <stephen@networkplumber.org>; Avi Cohen (A) <avi.cohen@huawei.com>; users@dpdk.org
Subject: Re: [dpdk-users] DPDK 17.05 does not find X510 card

Hi,

Stephen, thanks four input. I have already read that part of documentation Rami posted, but that ‘may disallow’ left me hoping there are some situations when it is not disallowed. If it is indeed not possible using the igb uio, I will have to do it with the vfio. However, and that is the reason why I was doing the dances with the kernel, I was not able to find a guide to bind the cards to vfio in non-virtual environent, other than a notice that all ports must be bound to vfio at once. 

Can you point me to some resources regarding the iommu and vfio?

Thanks,
Martin

23. 6. 2018 v 15:00, Rosen, Rami <rami.rosen@intel.com>:

> Hi all,
> 
>> Secure boot should work with VFIO. As long as you have an IOMMU, VFIO 
>> is a much better solution.
> 
> Stephen is right; moreover, DPDK documentation says explicitly that 
> ...
> "If UEFI secure boot is enabled, the Linux kernel may disallow the use of UIO on the system. Therefore, devices for use by DPDK should be bound to the vfio-pci kernel module rather than igb_uio or uio_pci_generic. For more details see Binding and Unbinding Network Ports to/from the Kernel Modules below."
> ...
> See section 4.1, "UIO", in:
> https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html
> 
> Regards,
> Rami Rosen
> 
> 

Re: [dpdk-users] DPDK 17.05 does not find X510 card

[Martin Drašar]

Dne 25.6.2018 v 11:35 Rosen, Rami napsal(a):
> Hi Martin,
> 
> There is info about VFIO and IOMMU for DPDK in the 
> Linux Getting Started Guide:
> https://doc.dpdk.org/guides/linux_gsg/index.html
> 
> 
> Generally, you need to follow section 8.1.3 for IOMMU:
> ...
> 8.1.3 Linux boot command line
> 
> If it wants to use VFIO, use the following additional grub parameters:
> iommu=pt intel_iommu=on
> ...
> And follow section 4.2 (VFIO)
> ...
> For VFIO you need modprobe vfio-pci
> ...
> 
> And use it with --bind  when running the dpdk-devnbind.py script.
> 
> Regards,
> Rami Rosen

Thank you Rami and everyone who participated!

I finally managed to run if with the VFIO drivers. I completely missed
the part about IOMMU enabling in Getting started guide.

Cheers,
Martin