* [dpdk-users] Hard Crash with X710 and Pktgen
@ 2017-11-13 19:35 James Bensley
2017-11-13 21:49 ` Wiles, Keith
0 siblings, 1 reply; 10+ messages in thread
From: James Bensley @ 2017-11-13 19:35 UTC (permalink / raw)
To: users
Hi All,
I have some HPE DL boxes that use "HP" NICs (OEM Intel X710s). I have
compiled DPDK & Pktgen but when I try to run Pktgen the box hard
crashes (it locks up and I have bounce it using IPMI).
The main output is below although a full transcript of my attempt is
here: https://null.53bits.co.uk/uploads/linux/applications/dpdk/CentOS-DL120-DPDK-NIC-Bind.txt
Essentially I can bind the NICs to DPDK using the igb_uio driver,
which leads to the hard crash, or I can bind the NICs to DPDK using
the vfio-pci driver but then Pktgen reports that there are no NICs
available to use and exits.
Does anyone have any ideas whats going on here? What is "failed to
select IOMMU type" telling me?
CentOS 7 / Kernel 4.13 / DPDK 17.08 / Pktgen 3.4.2.
Cheers,
James.
$ sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 2-6 -n 1 -w 09:00.0
-w 09:00.1 -v -- -P -m [3-4].0 [5-6].1
Copyright (c) <2010-2017>, Intel Corporation. All rights reserved.
Powered by DPDK
EAL: Detected 16 lcore(s)
EAL: RTE Version: 'DPDK 17.08.0'
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:09:00.0 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
<< HARD LOCK-UP >>
$ sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 2-6 -n 1 -w 09:00.0
-w 09:00.1 -v -- -P -m [3-4].0 [5-6].1
Copyright (c) <2010-2017>, Intel Corporation. All rights reserved.
Powered by DPDK
EAL: Detected 16 lcore(s)
EAL: RTE Version: 'DPDK 17.08.0'
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:09:00.0 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
EAL: 0000:09:00.0 failed to select IOMMU type
EAL: Requested device 0000:09:00.0 cannot be used
EAL: PCI device 0000:09:00.1 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
EAL: 0000:09:00.1 failed to select IOMMU type
EAL: Requested device 0000:09:00.1 cannot be used
Copyright (c) <2010-2017>, Intel Corporation. All rights reserved.
Pktgen created by: Keith Wiles -- >>> Powered by DPDK <<<
>>> Packet Burst 64, RX Desc 1024, TX Desc 2048, mbufs/port 16384, mbuf cache 2048
!PANIC!: *** Did not find any ports to use ***
PANIC in pktgen_config_ports():
*** Did not find any ports to use ***6:
[./app/x86_64-native-linuxapp-gcc/pktgen() [0x44ba76]]
5: [/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f16b7238c05]]
4: [./app/x86_64-native-linuxapp-gcc/pktgen(main+0x57c) [0x448edc]]
3: [./app/x86_64-native-linuxapp-gcc/pktgen(pktgen_config_ports+0x15b7)
[0x470d37]]
2: [./app/x86_64-native-linuxapp-gcc/pktgen(__rte_panic+0xb8) [0x4432bc]]
1: [./app/x86_64-native-linuxapp-gcc/pktgen(rte_dump_stack+0x1a) [0x4b245a]]
Aborted
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [dpdk-users] Hard Crash with X710 and Pktgen 2017-11-13 19:35 [dpdk-users] Hard Crash with X710 and Pktgen James Bensley @ 2017-11-13 21:49 ` Wiles, Keith 2017-11-14 4:49 ` Muhammad Zain-ul-Abideen 2017-11-14 6:29 ` Rosen, Rami 0 siblings, 2 replies; 10+ messages in thread From: Wiles, Keith @ 2017-11-13 21:49 UTC (permalink / raw) To: James Bensley; +Cc: users > On Nov 13, 2017, at 11:35 AM, James Bensley <jwbensley@gmail.com> wrote: > > Hi All, > > I have some HPE DL boxes that use "HP" NICs (OEM Intel X710s). I have > compiled DPDK & Pktgen but when I try to run Pktgen the box hard > crashes (it locks up and I have bounce it using IPMI). > > The main output is below although a full transcript of my attempt is > here: https://null.53bits.co.uk/uploads/linux/applications/dpdk/CentOS-DL120-DPDK-NIC-Bind.txt > > Essentially I can bind the NICs to DPDK using the igb_uio driver, > which leads to the hard crash, or I can bind the NICs to DPDK using > the vfio-pci driver but then Pktgen reports that there are no NICs > available to use and exits. In Pktgen I just use the normal rte_eth_device_count() (whatever it is called) to find the ports in the system. This means something in that configuration is not setup and DPDK is not finding the ports in the system. Has nothing to do with Pktgen that I know of, but I have not tried vfio-pci driver and can not as I am out of town. > > Does anyone have any ideas whats going on here? What is "failed to > select IOMMU type" telling me? For the hard lockup problem try using testpmd application and see if that the same problem if not then it will be next week before I can look at. Let me know if testpmd works or not. > > CentOS 7 / Kernel 4.13 / DPDK 17.08 / Pktgen 3.4.2. > > Cheers, > James. > > > $ sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 2-6 -n 1 -w 09:00.0 > -w 09:00.1 -v -- -P -m [3-4].0 [5-6].1 > > Copyright (c) <2010-2017>, Intel Corporation. All rights reserved. > Powered by DPDK > EAL: Detected 16 lcore(s) > EAL: RTE Version: 'DPDK 17.08.0' > EAL: No free hugepages reported in hugepages-1048576kB > EAL: Probing VFIO support... > EAL: PCI device 0000:09:00.0 on NUMA socket 0 > EAL: probe driver: 8086:1572 net_i40e > > << HARD LOCK-UP >> > > > $ sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 2-6 -n 1 -w 09:00.0 > -w 09:00.1 -v -- -P -m [3-4].0 [5-6].1 > > Copyright (c) <2010-2017>, Intel Corporation. All rights reserved. > Powered by DPDK > EAL: Detected 16 lcore(s) > EAL: RTE Version: 'DPDK 17.08.0' > EAL: No free hugepages reported in hugepages-1048576kB > EAL: Probing VFIO support... > EAL: VFIO support initialized > EAL: PCI device 0000:09:00.0 on NUMA socket 0 > EAL: probe driver: 8086:1572 net_i40e > EAL: 0000:09:00.0 failed to select IOMMU type > EAL: Requested device 0000:09:00.0 cannot be used > EAL: PCI device 0000:09:00.1 on NUMA socket 0 > EAL: probe driver: 8086:1572 net_i40e > EAL: 0000:09:00.1 failed to select IOMMU type > EAL: Requested device 0000:09:00.1 cannot be used > > Copyright (c) <2010-2017>, Intel Corporation. All rights reserved. > Pktgen created by: Keith Wiles -- >>> Powered by DPDK <<< > >>>> Packet Burst 64, RX Desc 1024, TX Desc 2048, mbufs/port 16384, mbuf cache 2048 > !PANIC!: *** Did not find any ports to use *** > PANIC in pktgen_config_ports(): > *** Did not find any ports to use ***6: > [./app/x86_64-native-linuxapp-gcc/pktgen() [0x44ba76]] > 5: [/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f16b7238c05]] > 4: [./app/x86_64-native-linuxapp-gcc/pktgen(main+0x57c) [0x448edc]] > 3: [./app/x86_64-native-linuxapp-gcc/pktgen(pktgen_config_ports+0x15b7) > [0x470d37]] > 2: [./app/x86_64-native-linuxapp-gcc/pktgen(__rte_panic+0xb8) [0x4432bc]] > 1: [./app/x86_64-native-linuxapp-gcc/pktgen(rte_dump_stack+0x1a) [0x4b245a]] > Aborted Regards, Keith ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-users] Hard Crash with X710 and Pktgen 2017-11-13 21:49 ` Wiles, Keith @ 2017-11-14 4:49 ` Muhammad Zain-ul-Abideen 2017-11-14 6:29 ` Rosen, Rami 1 sibling, 0 replies; 10+ messages in thread From: Muhammad Zain-ul-Abideen @ 2017-11-14 4:49 UTC (permalink / raw) To: Wiles, Keith; +Cc: James Bensley, users where is -p argument ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-users] Hard Crash with X710 and Pktgen 2017-11-13 21:49 ` Wiles, Keith 2017-11-14 4:49 ` Muhammad Zain-ul-Abideen @ 2017-11-14 6:29 ` Rosen, Rami 2017-11-14 8:37 ` James Bensley 1 sibling, 1 reply; 10+ messages in thread From: Rosen, Rami @ 2017-11-14 6:29 UTC (permalink / raw) To: Wiles, Keith, James Bensley; +Cc: users Hi, James, First I agree with Keith about that this means something in that configuration is not setup correctly and DPDK is not finding the ports in the system, and has nothing to do with Pktgen. It seems to me that if you will try to run any application (like testpmd) on this setup you will get the same problem. Can you please check and confirm ? Second, I think the root cause for not finding the ports is around this message ("failed to select IOMMU type") in: ... > EAL: probe driver: 8086:1572 net_i40e > EAL: 0000:09:00.0 failed to select IOMMU type > EAL: Requested device 0000:09:00.0 cannot be used > EAL: PCI device 0000:09:00.1 on NUMA socket 0 > EAL: probe driver: 8086:1572 net_i40e > EAL: 0000:09:00.1 failed to select IOMMU type > EAL: Requested device 0000:09:00.1 cannot be used ... What does " find /sys/kernel/iommu_groups/ -type l" give ? Could it be that 0000:09:00.0 and 0000:09:00.1 belongs to an IOMMU group in which there are other devices ? Regards, Rami Rosen -----Original Message----- From: users [mailto:users-bounces@dpdk.org] On Behalf Of Wiles, Keith Sent: Monday, November 13, 2017 23:50 To: James Bensley <jwbensley@gmail.com> Cc: users@dpdk.org Subject: Re: [dpdk-users] Hard Crash with X710 and Pktgen > On Nov 13, 2017, at 11:35 AM, James Bensley <jwbensley@gmail.com> wrote: > > Hi All, > > I have some HPE DL boxes that use "HP" NICs (OEM Intel X710s). I have > compiled DPDK & Pktgen but when I try to run Pktgen the box hard > crashes (it locks up and I have bounce it using IPMI). > > The main output is below although a full transcript of my attempt is > here: > https://null.53bits.co.uk/uploads/linux/applications/dpdk/CentOS-DL120 > -DPDK-NIC-Bind.txt > > Essentially I can bind the NICs to DPDK using the igb_uio driver, > which leads to the hard crash, or I can bind the NICs to DPDK using > the vfio-pci driver but then Pktgen reports that there are no NICs > available to use and exits. In Pktgen I just use the normal rte_eth_device_count() (whatever it is called) to find the ports in the system. This means something in that configuration is not setup and DPDK is not finding the ports in the system. Has nothing to do with Pktgen that I know of, but I have not tried vfio-pci driver and can not as I am out of town. > > Does anyone have any ideas whats going on here? What is "failed to > select IOMMU type" telling me? For the hard lockup problem try using testpmd application and see if that the same problem if not then it will be next week before I can look at. Let me know if testpmd works or not. > > CentOS 7 / Kernel 4.13 / DPDK 17.08 / Pktgen 3.4.2. > > Cheers, > James. > > > $ sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 2-6 -n 1 -w 09:00.0 > -w 09:00.1 -v -- -P -m [3-4].0 [5-6].1 > > Copyright (c) <2010-2017>, Intel Corporation. All rights reserved. > Powered by DPDK > EAL: Detected 16 lcore(s) > EAL: RTE Version: 'DPDK 17.08.0' > EAL: No free hugepages reported in hugepages-1048576kB > EAL: Probing VFIO support... > EAL: PCI device 0000:09:00.0 on NUMA socket 0 > EAL: probe driver: 8086:1572 net_i40e > > << HARD LOCK-UP >> > > > $ sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 2-6 -n 1 -w 09:00.0 > -w 09:00.1 -v -- -P -m [3-4].0 [5-6].1 > > Copyright (c) <2010-2017>, Intel Corporation. All rights reserved. > Powered by DPDK > EAL: Detected 16 lcore(s) > EAL: RTE Version: 'DPDK 17.08.0' > EAL: No free hugepages reported in hugepages-1048576kB > EAL: Probing VFIO support... > EAL: VFIO support initialized > EAL: PCI device 0000:09:00.0 on NUMA socket 0 > EAL: probe driver: 8086:1572 net_i40e > EAL: 0000:09:00.0 failed to select IOMMU type > EAL: Requested device 0000:09:00.0 cannot be used > EAL: PCI device 0000:09:00.1 on NUMA socket 0 > EAL: probe driver: 8086:1572 net_i40e > EAL: 0000:09:00.1 failed to select IOMMU type > EAL: Requested device 0000:09:00.1 cannot be used > > Copyright (c) <2010-2017>, Intel Corporation. All rights reserved. > Pktgen created by: Keith Wiles -- >>> Powered by DPDK <<< > >>>> Packet Burst 64, RX Desc 1024, TX Desc 2048, mbufs/port 16384, mbuf >>>> cache 2048 > !PANIC!: *** Did not find any ports to use *** PANIC in > pktgen_config_ports(): > *** Did not find any ports to use ***6: > [./app/x86_64-native-linuxapp-gcc/pktgen() [0x44ba76]] > 5: [/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f16b7238c05]] > 4: [./app/x86_64-native-linuxapp-gcc/pktgen(main+0x57c) [0x448edc]] > 3: > [./app/x86_64-native-linuxapp-gcc/pktgen(pktgen_config_ports+0x15b7) > [0x470d37]] > 2: [./app/x86_64-native-linuxapp-gcc/pktgen(__rte_panic+0xb8) > [0x4432bc]] > 1: [./app/x86_64-native-linuxapp-gcc/pktgen(rte_dump_stack+0x1a) > [0x4b245a]] Aborted Regards, Keith ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-users] Hard Crash with X710 and Pktgen 2017-11-14 6:29 ` Rosen, Rami @ 2017-11-14 8:37 ` James Bensley 2017-11-14 10:59 ` Johnson, Brian 0 siblings, 1 reply; 10+ messages in thread From: James Bensley @ 2017-11-14 8:37 UTC (permalink / raw) To: Rosen, Rami, Wiles, Keith, users Hi guys, Thank you all for your responses! I initially thought this was unlikely to be a Pktgen specific issue however I thought it worth mentioning I am using Pktgen just in case. On 14 November 2017 at 04:49, Muhammad Zain-ul-Abideen <zain2294@gmail.com> wrote: > where is -p argument Do you mean upper case "-P"? Lower case “p” isn’t an option in “Pktgen –-help” output. On 13 November 2017 at 19:35, James Bensley <jwbensley@gmail.com> wrote: > $ sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 2-6 -n 1 -w 09:00.0 -w 09:00.1 -v -- -P -m [3-4].0 [5-6].1 On 13 November 2017 at 21:49, Wiles, Keith <keith.wiles@intel.com> wrote: > For the hard lockup problem try using testpmd application and see if that the same problem if not then it will be next week before I can look at. > > Let me know if testpmd works or not. Thanks for taking the time to reply whilst traveling Keith! Sadly I’m getting the same behaviour with testpmd: $ ./dpdk-devbind.py --status-dev net Network devices using DPDK-compatible driver ============================================ 0000:09:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused= 0000:09:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused= Network devices using kernel driver =================================== 0000:02:00.0 'I350 Gigabit Network Connection 1521' if=eno1 drv=igb unused=igb_uio *Active* 0000:02:00.1 'I350 Gigabit Network Connection 1521' if=eno2 drv=igb unused=igb_uio 0000:06:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f0 drv=tg3 unused=igb_uio 0000:06:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f1 drv=tg3 unused=igb_uio 0000:06:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f2 drv=tg3 unused=igb_uio *Active* 0000:06:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f3 drv=tg3 unused=igb_uio $ sudo ./testpmd -l 0-3 -n 4 -- -i --portmask=0x1 --nb-cores=2 EAL: Detected 16 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: PCI device 0000:02:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:02:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:09:00.0 on NUMA socket 0 EAL: probe driver: 8086:1572 net_i40e << HARD LOCK-UP >> $ ./dpdk-devbind.py --status-dev net Network devices using DPDK-compatible driver ============================================ 0000:09:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=igb_uio 0000:09:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=igb_uio Network devices using kernel driver =================================== 0000:02:00.0 'I350 Gigabit Network Connection 1521' if=eno1 drv=igb unused=igb_uio,vfio-pci *Active* 0000:02:00.1 'I350 Gigabit Network Connection 1521' if=eno2 drv=igb unused=igb_uio,vfio-pci 0000:06:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f0 drv=tg3 unused=igb_uio,vfio-pci 0000:06:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f1 drv=tg3 unused=igb_uio,vfio-pci 0000:06:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f2 drv=tg3 unused=igb_uio,vfio-pci *Active* 0000:06:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f3 drv=tg3 unused=igb_uio,vfio-pci $ sudo ./testpmd -l 0-3 -n 4 -- -i --portmask=0x1 --nb-cores=2 EAL: Detected 16 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:02:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:02:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:09:00.0 on NUMA socket 0 EAL: probe driver: 8086:1572 net_i40e EAL: 0000:09:00.0 failed to select IOMMU type EAL: Requested device 0000:09:00.0 cannot be used EAL: PCI device 0000:09:00.1 on NUMA socket 0 EAL: probe driver: 8086:1572 net_i40e EAL: 0000:09:00.1 failed to select IOMMU type EAL: Requested device 0000:09:00.1 cannot be used EAL: No probed ethernet devices Interactive-mode selected USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0 Done On 14 November 2017 at 06:29, Rosen, Rami <rami.rosen@intel.com> wrote: > Second, I think the root cause for not finding the ports is around this message ("failed to select IOMMU type") in: > ... >> EAL: probe driver: 8086:1572 net_i40e >> EAL: 0000:09:00.0 failed to select IOMMU type >> EAL: Requested device 0000:09:00.0 cannot be used >> EAL: PCI device 0000:09:00.1 on NUMA socket 0 >> EAL: probe driver: 8086:1572 net_i40e >> EAL: 0000:09:00.1 failed to select IOMMU type >> EAL: Requested device 0000:09:00.1 cannot be used > ... > > What does " find /sys/kernel/iommu_groups/ -type l" > give ? > > Could it be that 0000:09:00.0 and 0000:09:00.1 belongs to an IOMMU group in which there are other devices ? To me (a layman) this looks correct, do you agree? $ find /sys/kernel/iommu_groups/ -type l | grep 09 /sys/kernel/iommu_groups/36/devices/0000:09:00.0 /sys/kernel/iommu_groups/37/devices/0000:09:00.1 $ find /sys/kernel/iommu_groups/ -type l | grep -E "36|37" /sys/kernel/iommu_groups/36/devices/0000:09:00.0 /sys/kernel/iommu_groups/37/devices/0000:09:00.1 Cheers, James. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-users] Hard Crash with X710 and Pktgen 2017-11-14 8:37 ` James Bensley @ 2017-11-14 10:59 ` Johnson, Brian 2017-11-14 20:20 ` Wiles, Keith 2017-11-15 17:22 ` James Bensley 0 siblings, 2 replies; 10+ messages in thread From: Johnson, Brian @ 2017-11-14 10:59 UTC (permalink / raw) To: James Bensley; +Cc: Rosen, Rami, Wiles, Keith, users Do you have Intel VT-d enabled in the BIOS and intel_iommu=on iommu=pt in grub? I have had some issues with enabling this where other devices such as RAID controllers cause the OS to no boot after applying these settings. Removing the grub line and upgrading the RAID firmware fixed the issue. IOMMU<https://dpdk-guide.gitlab.io/dpdk-guide/setup/iommu.html#iommu> An input-output memory management unit (IOMMU) is required for safely driving DMA-capable hardware from userspace and because of that it is a prerequisite for using VFIO. Not all systems have one though, so you’ll need to check that the hardware supports it and that it is enabled in the BIOS settings (VT-d or Virtualization Technology for Directed I/Oon Intel systems) Finally, IOMMU needs to be excplitly enabled in the kernel as well. To do so, pass either intel_iommu=on (for Intel systems) or amd_iommu=on (for AMD systems) added to the kernel command line. In addition it is recommended to use iommu=pt option which improves IO performance for devices in the host. Once the system boots up, check the contents of /sys/kernel/iommu_groups/ directory. If it is non-empty, you have successfully set up IOMMU. To permanently add this to the kernel commandline, append it to GRUB_CMDLINE_LINUX in /etc/default/grub and then execute: # grub2-mkconfig -o /boot/grub2/grub.cfg Sent from my iPhone On Nov 14, 2017, at 12:38 AM, James Bensley <jwbensley@gmail.com<mailto:jwbensley@gmail.com>> wrote: Hi guys, Thank you all for your responses! I initially thought this was unlikely to be a Pktgen specific issue however I thought it worth mentioning I am using Pktgen just in case. On 14 November 2017 at 04:49, Muhammad Zain-ul-Abideen <zain2294@gmail.com<mailto:zain2294@gmail.com>> wrote: where is -p argument Do you mean upper case "-P"? Lower case “p” isn’t an option in “Pktgen –-help” output. On 13 November 2017 at 19:35, James Bensley <jwbensley@gmail.com<mailto:jwbensley@gmail.com>> wrote: $ sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 2-6 -n 1 -w 09:00.0 -w 09:00.1 -v -- -P -m [3-4].0 [5-6].1 On 13 November 2017 at 21:49, Wiles, Keith <keith.wiles@intel.com<mailto:keith.wiles@intel.com>> wrote: For the hard lockup problem try using testpmd application and see if that the same problem if not then it will be next week before I can look at. Let me know if testpmd works or not. Thanks for taking the time to reply whilst traveling Keith! Sadly I’m getting the same behaviour with testpmd: $ ./dpdk-devbind.py --status-dev net Network devices using DPDK-compatible driver ============================================ 0000:09:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused= 0000:09:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused= Network devices using kernel driver =================================== 0000:02:00.0 'I350 Gigabit Network Connection 1521' if=eno1 drv=igb unused=igb_uio *Active* 0000:02:00.1 'I350 Gigabit Network Connection 1521' if=eno2 drv=igb unused=igb_uio 0000:06:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f0 drv=tg3 unused=igb_uio 0000:06:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f1 drv=tg3 unused=igb_uio 0000:06:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f2 drv=tg3 unused=igb_uio *Active* 0000:06:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f3 drv=tg3 unused=igb_uio $ sudo ./testpmd -l 0-3 -n 4 -- -i --portmask=0x1 --nb-cores=2 EAL: Detected 16 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: PCI device 0000:02:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:02:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:09:00.0 on NUMA socket 0 EAL: probe driver: 8086:1572 net_i40e << HARD LOCK-UP >> $ ./dpdk-devbind.py --status-dev net Network devices using DPDK-compatible driver ============================================ 0000:09:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=igb_uio 0000:09:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=igb_uio Network devices using kernel driver =================================== 0000:02:00.0 'I350 Gigabit Network Connection 1521' if=eno1 drv=igb unused=igb_uio,vfio-pci *Active* 0000:02:00.1 'I350 Gigabit Network Connection 1521' if=eno2 drv=igb unused=igb_uio,vfio-pci 0000:06:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f0 drv=tg3 unused=igb_uio,vfio-pci 0000:06:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f1 drv=tg3 unused=igb_uio,vfio-pci 0000:06:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f2 drv=tg3 unused=igb_uio,vfio-pci *Active* 0000:06:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f3 drv=tg3 unused=igb_uio,vfio-pci $ sudo ./testpmd -l 0-3 -n 4 -- -i --portmask=0x1 --nb-cores=2 EAL: Detected 16 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:02:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:02:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:09:00.0 on NUMA socket 0 EAL: probe driver: 8086:1572 net_i40e EAL: 0000:09:00.0 failed to select IOMMU type EAL: Requested device 0000:09:00.0 cannot be used EAL: PCI device 0000:09:00.1 on NUMA socket 0 EAL: probe driver: 8086:1572 net_i40e EAL: 0000:09:00.1 failed to select IOMMU type EAL: Requested device 0000:09:00.1 cannot be used EAL: No probed ethernet devices Interactive-mode selected USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0 Done On 14 November 2017 at 06:29, Rosen, Rami <rami.rosen@intel.com<mailto:rami.rosen@intel.com>> wrote: Second, I think the root cause for not finding the ports is around this message ("failed to select IOMMU type") in: ... EAL: probe driver: 8086:1572 net_i40e EAL: 0000:09:00.0 failed to select IOMMU type EAL: Requested device 0000:09:00.0 cannot be used EAL: PCI device 0000:09:00.1 on NUMA socket 0 EAL: probe driver: 8086:1572 net_i40e EAL: 0000:09:00.1 failed to select IOMMU type EAL: Requested device 0000:09:00.1 cannot be used ... What does " find /sys/kernel/iommu_groups/ -type l" give ? Could it be that 0000:09:00.0 and 0000:09:00.1 belongs to an IOMMU group in which there are other devices ? To me (a layman) this looks correct, do you agree? $ find /sys/kernel/iommu_groups/ -type l | grep 09 /sys/kernel/iommu_groups/36/devices/0000:09:00.0 /sys/kernel/iommu_groups/37/devices/0000:09:00.1 $ find /sys/kernel/iommu_groups/ -type l | grep -E "36|37" /sys/kernel/iommu_groups/36/devices/0000:09:00.0 /sys/kernel/iommu_groups/37/devices/0000:09:00.1 Cheers, James. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-users] Hard Crash with X710 and Pktgen 2017-11-14 10:59 ` Johnson, Brian @ 2017-11-14 20:20 ` Wiles, Keith 2017-11-15 17:24 ` James Bensley 2017-11-15 17:22 ` James Bensley 1 sibling, 1 reply; 10+ messages in thread From: Wiles, Keith @ 2017-11-14 20:20 UTC (permalink / raw) To: Johnson, Brian; +Cc: James Bensley, Rosen, Rami, users Sent from my iPhone On Nov 14, 2017, at 2:59 AM, Johnson, Brian <brian.johnson@intel.com<mailto:brian.johnson@intel.com>> wrote: Do you have Intel VT-d enabled in the BIOS and intel_iommu=on iommu=pt in grub? I have had some issues with enabling this where other devices such as RAID controllers cause the OS to no boot after applying these settings. Removing the grub line and upgrading the RAID firmware fixed the issue. IOMMU<https://dpdk-guide.gitlab.io/dpdk-guide/setup/iommu.html#iommu> An input-output memory management unit (IOMMU) is required for safely driving DMA-capable hardware from userspace and because of that it is a prerequisite for using VFIO. Not all systems have one though, so you’ll need to check that the hardware supports it and that it is enabled in the BIOS settings (VT-d or Virtualization Technology for Directed I/Oon Intel systems) Finally, IOMMU needs to be excplitly enabled in the kernel as well. To do so, pass either intel_iommu=on (for Intel systems) or amd_iommu=on (for AMD systems) added to the kernel command line. In addition it is recommended to use iommu=pt option which improves IO performance for devices in the host. Once the system boots up, check the contents of /sys/kernel/iommu_groups/ directory. If it is non-empty, you have successfully set up IOMMU. To permanently add this to the kernel commandline, append it to GRUB_CMDLINE_LINUX in /etc/default/grub and then execute: # grub2-mkconfig -o /boot/grub2/grub.cfg Sent from my iPhone On Nov 14, 2017, at 12:38 AM, James Bensley <jwbensley@gmail.com<mailto:jwbensley@gmail.com>> wrote: Hi guys, Thank you all for your responses! I initially thought this was unlikely to be a Pktgen specific issue however I thought it worth mentioning I am using Pktgen just in case. On 14 November 2017 at 04:49, Muhammad Zain-ul-Abideen <zain2294@gmail.com<mailto:zain2294@gmail.com>> wrote: where is -p argument Do you mean upper case "-P"? Lower case “p” isn’t an option in “Pktgen –-help” output Pktgen once used -p Portman value and is not used anymore I still see that option being used in some command lines. On 13 November 2017 at 19:35, James Bensley <jwbensley@gmail.com<mailto:jwbensley@gmail.com>> wrote: $ sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 2-6 -n 1 -w 09:00.0 -w 09:00.1 -v -- -P -m [3-4].0 [5-6].1 On 13 November 2017 at 21:49, Wiles, Keith <keith.wiles@intel.com<mailto:keith.wiles@intel.com>> wrote: For the hard lockup problem try using testpmd application and see if that the same problem if not then it will be next week before I can look at. Let me know if testpmd works or not. Thanks for taking the time to reply whilst traveling Keith! Sadly I’m getting the same behaviour with testpmd: $ ./dpdk-devbind.py --status-dev net Network devices using DPDK-compatible driver ============================================ 0000:09:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused= 0000:09:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused= Network devices using kernel driver =================================== 0000:02:00.0 'I350 Gigabit Network Connection 1521' if=eno1 drv=igb unused=igb_uio *Active* 0000:02:00.1 'I350 Gigabit Network Connection 1521' if=eno2 drv=igb unused=igb_uio 0000:06:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f0 drv=tg3 unused=igb_uio 0000:06:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f1 drv=tg3 unused=igb_uio 0000:06:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f2 drv=tg3 unused=igb_uio *Active* 0000:06:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f3 drv=tg3 unused=igb_uio $ sudo ./testpmd -l 0-3 -n 4 -- -i --portmask=0x1 --nb-cores=2 EAL: Detected 16 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: PCI device 0000:02:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:02:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:09:00.0 on NUMA socket 0 EAL: probe driver: 8086:1572 net_i40e << HARD LOCK-UP >> $ ./dpdk-devbind.py --status-dev net Network devices using DPDK-compatible driver ============================================ 0000:09:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=igb_uio 0000:09:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=igb_uio Network devices using kernel driver =================================== 0000:02:00.0 'I350 Gigabit Network Connection 1521' if=eno1 drv=igb unused=igb_uio,vfio-pci *Active* 0000:02:00.1 'I350 Gigabit Network Connection 1521' if=eno2 drv=igb unused=igb_uio,vfio-pci 0000:06:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f0 drv=tg3 unused=igb_uio,vfio-pci 0000:06:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f1 drv=tg3 unused=igb_uio,vfio-pci 0000:06:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f2 drv=tg3 unused=igb_uio,vfio-pci *Active* 0000:06:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f3 drv=tg3 unused=igb_uio,vfio-pci $ sudo ./testpmd -l 0-3 -n 4 -- -i --portmask=0x1 --nb-cores=2 EAL: Detected 16 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:02:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:02:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:09:00.0 on NUMA socket 0 EAL: probe driver: 8086:1572 net_i40e EAL: 0000:09:00.0 failed to select IOMMU type EAL: Requested device 0000:09:00.0 cannot be used EAL: PCI device 0000:09:00.1 on NUMA socket 0 EAL: probe driver: 8086:1572 net_i40e EAL: 0000:09:00.1 failed to select IOMMU type EAL: Requested device 0000:09:00.1 cannot be used EAL: No probed ethernet devices Interactive-mode selected USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0 Done On 14 November 2017 at 06:29, Rosen, Rami <rami.rosen@intel.com<mailto:rami.rosen@intel.com>> wrote: Second, I think the root cause for not finding the ports is around this message ("failed to select IOMMU type") in: ... EAL: probe driver: 8086:1572 net_i40e EAL: 0000:09:00.0 failed to select IOMMU type EAL: Requested device 0000:09:00.0 cannot be used EAL: PCI device 0000:09:00.1 on NUMA socket 0 EAL: probe driver: 8086:1572 net_i40e EAL: 0000:09:00.1 failed to select IOMMU type EAL: Requested device 0000:09:00.1 cannot be used ... What does " find /sys/kernel/iommu_groups/ -type l" give ? Could it be that 0000:09:00.0 and 0000:09:00.1 belongs to an IOMMU group in which there are other devices ? To me (a layman) this looks correct, do you agree? $ find /sys/kernel/iommu_groups/ -type l | grep 09 /sys/kernel/iommu_groups/36/devices/0000:09:00.0 /sys/kernel/iommu_groups/37/devices/0000:09:00.1 $ find /sys/kernel/iommu_groups/ -type l | grep -E "36|37" /sys/kernel/iommu_groups/36/devices/0000:09:00.0 /sys/kernel/iommu_groups/37/devices/0000:09:00.1 Cheers, James. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-users] Hard Crash with X710 and Pktgen 2017-11-14 20:20 ` Wiles, Keith @ 2017-11-15 17:24 ` James Bensley 2017-11-15 18:04 ` James Bensley 0 siblings, 1 reply; 10+ messages in thread From: James Bensley @ 2017-11-15 17:24 UTC (permalink / raw) To: Wiles, Keith, users On 14 November 2017 at 20:20, Wiles, Keith <keith.wiles@intel.com> wrote: > >> On Nov 14, 2017, at 12:38 AM, James Bensley <jwbensley@gmail.com> wrote: >> >>> On 14 November 2017 at 04:49, Muhammad Zain-ul-Abideen >>> <zain2294@gmail.com> wrote: >>> >>> where is -p argument >> >> Do you mean upper case "-P"? Lower case “p” isn’t an option in “Pktgen >> –-help” output > > > Pktgen once used -p Portman value and is not used anymore I still see that > option being used in some command lines. Thanks for confirming Keith! Cheers, James. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-users] Hard Crash with X710 and Pktgen 2017-11-15 17:24 ` James Bensley @ 2017-11-15 18:04 ` James Bensley 0 siblings, 0 replies; 10+ messages in thread From: James Bensley @ 2017-11-15 18:04 UTC (permalink / raw) To: users Hi All, I have Pktgen working on this box now. So for the sake of this list archives: In addition to the 2x10G Intel X710 NIC there is a 2x1G Intel I350 NIC in this box. Trying with the igb_uio or vfio-pci driver for the 1G NIC had the same results as with the 10G NIC; hard lock-up when using igb_uio and DPDK is reporting no free ports available when using vfio-pci. However with vfio-pci the following syslog message was logged: "Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor." After some searching this seems to be an issue with Kernels newer than 3.16 on HPE boxes (e.g. https://forums.lime-technology.com/topic/53049-solved-device-is-ineligible-for-iommu-domain-attach-due-to-platform-rmrr/). Looks like a BIOS setting can solve this issue (https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04781229&sp4ts.oid=5249566). I have multiple Kernels compiled on this box. I was using 4.13 for some other testing, so I booted into 3.10, recompiled DPDK+Pktgen and both the 10G and 1G NICs are working using the igb_uio driver. I'd like to be a 4.10 kernel as I was testing some features in there separate to DPDK so I will try disabling that BIOS setting and see if this works with the newer Kernel version. The box is remote so I can't do that right away! Many thanks all for your time and help it is greatly appreciated. Cheers, James. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-users] Hard Crash with X710 and Pktgen 2017-11-14 10:59 ` Johnson, Brian 2017-11-14 20:20 ` Wiles, Keith @ 2017-11-15 17:22 ` James Bensley 1 sibling, 0 replies; 10+ messages in thread From: James Bensley @ 2017-11-15 17:22 UTC (permalink / raw) To: Johnson, Brian, users On 14 November 2017 at 10:59, Johnson, Brian <brian.johnson@intel.com> wrote: > Do you have Intel VT-d enabled in the BIOS and intel_iommu=on iommu=pt in > grub? ... > Once the system boots up, check the contents of /sys/kernel/iommu_groups/ > directory. If it is non-empty, you have successfully set up IOMMU. Hi Brian, Many thanks for your response! $ ls /sys/kernel/iommu_groups/ 0 1 10 11 12 13 14 15 16 17 18 19 2 20 21 22 23 24 25 26 27 28 29 3 30 31 32 33 34 35 36 37 38 39 4 40 41 5 6 7 8 9 There is "stuff" in there so, looks good to me. $ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-4.13.12-1.el7.elrepo.x86_64 root=UUID=e5b08dc0-3ea6-4259-82ab-820dee91fa52 ro intel_pstate=disable iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=4 crashkernel=auto rhgb quiet $ dmesg | grep -i IOMMU [ 0.000000] DMAR: IOMMU enabled VT-x is enabled: $ grep -m 1 -o "vmx" /proc/cpuinfo vmx VT-d is enabled: $ dmesg | grep -i DMAR [ 0.000000] DMAR: IOMMU enabled Cheers, James. ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2017-11-15 18:05 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-11-13 19:35 [dpdk-users] Hard Crash with X710 and Pktgen James Bensley 2017-11-13 21:49 ` Wiles, Keith 2017-11-14 4:49 ` Muhammad Zain-ul-Abideen 2017-11-14 6:29 ` Rosen, Rami 2017-11-14 8:37 ` James Bensley 2017-11-14 10:59 ` Johnson, Brian 2017-11-14 20:20 ` Wiles, Keith 2017-11-15 17:24 ` James Bensley 2017-11-15 18:04 ` James Bensley 2017-11-15 17:22 ` James Bensley
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).