* [dpdk-users] HELP: MLX4 problem
@ 2016-01-12 9:43 Masaru Kawashima
2016-01-12 15:47 ` Adrien Mazarguil
0 siblings, 1 reply; 4+ messages in thread
From: Masaru Kawashima @ 2016-01-12 9:43 UTC (permalink / raw)
To: users
Hi, all
I'm trying to use dpdk with Mellanox ConnectX-3.
But, mlnx4 pmd cannot work fine.
Following is my environment:
OS: CentOS 6.5 (x86_64)
Mellanox OFED: MLNX_OFED_LINUX-3.1-1.1.0.1-rhel6.5-x86_64
DPDK: MLNX_DPDK-2.1_1.1
I'm trying to run testpmd, but librte_pmd_mlx4 reports
"ibv_exp_query_device() failed",
and EAL reports "No probed ethernet devices".
What can I do next to work mlx4 pmd fine?
Following is the testpmd's log:
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 5 on socket 0
EAL: Detected lcore 6 as core 8 on socket 0
EAL: Detected lcore 7 as core 9 on socket 0
EAL: Detected lcore 8 as core 10 on socket 0
EAL: Detected lcore 9 as core 11 on socket 0
EAL: Detected lcore 10 as core 12 on socket 0
EAL: Detected lcore 11 as core 13 on socket 0
EAL: Detected lcore 12 as core 0 on socket 1
EAL: Detected lcore 13 as core 1 on socket 1
EAL: Detected lcore 14 as core 2 on socket 1
EAL: Detected lcore 15 as core 3 on socket 1
EAL: Detected lcore 16 as core 4 on socket 1
EAL: Detected lcore 17 as core 5 on socket 1
EAL: Detected lcore 18 as core 8 on socket 1
EAL: Detected lcore 19 as core 9 on socket 1
EAL: Detected lcore 20 as core 10 on socket 1
EAL: Detected lcore 21 as core 11 on socket 1
EAL: Detected lcore 22 as core 12 on socket 1
EAL: Detected lcore 23 as core 13 on socket 1
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 24 lcore(s)
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x80000000 bytes
EAL: Virtual area found at 0x7f2940000000 (size = 0x80000000)
EAL: Ask a virtual area of 0x80000000 bytes
EAL: Virtual area found at 0x7f2880000000 (size = 0x80000000)
EAL: Requesting 2 pages of size 1024MB from socket 0
EAL: Requesting 2 pages of size 1024MB from socket 1
EAL: TSC frequency is ~2300002 KHz
EAL: Master lcore 0 is ready (tid=f6cdc8a0;cpuset=[0])
EAL: lcore 2 is ready (tid=f58d2700;cpuset=[2])
EAL: lcore 1 is ready (tid=f62d3700;cpuset=[1])
EAL: lcore 3 is ready (tid=f4ed1700;cpuset=[3])
EAL: PCI device 0000:02:00.0 on NUMA socket 0
EAL: probe driver: 15b3:1003 librte_pmd_mlx4
PMD: librte_pmd_mlx4: PCI information matches, using device "mlx4_0" (VF:
false)
PMD: librte_pmd_mlx4: 1 port(s) detected
PMD: librte_pmd_mlx4: ibv_exp_query_device() failed
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL: probe driver: 8086:1528 rte_ixgbe_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:81:00.1 on NUMA socket 1
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: No probed ethernet devices
Best regards,
Masaru Kawashima
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [dpdk-users] HELP: MLX4 problem
2016-01-12 9:43 [dpdk-users] HELP: MLX4 problem Masaru Kawashima
@ 2016-01-12 15:47 ` Adrien Mazarguil
2016-01-14 2:17 ` Masaru Kawashima
0 siblings, 1 reply; 4+ messages in thread
From: Adrien Mazarguil @ 2016-01-12 15:47 UTC (permalink / raw)
To: Masaru Kawashima; +Cc: users
Hi Masaru,
On Tue, Jan 12, 2016 at 06:43:53PM +0900, Masaru Kawashima wrote:
> Hi, all
>
> I'm trying to use dpdk with Mellanox ConnectX-3.
> But, mlnx4 pmd cannot work fine.
>
> Following is my environment:
>
> OS: CentOS 6.5 (x86_64)
> Mellanox OFED: MLNX_OFED_LINUX-3.1-1.1.0.1-rhel6.5-x86_64
> DPDK: MLNX_DPDK-2.1_1.1
>
> I'm trying to run testpmd, but librte_pmd_mlx4 reports
> "ibv_exp_query_device() failed",
> and EAL reports "No probed ethernet devices".
>
> What can I do next to work mlx4 pmd fine?
>
>
> Following is the testpmd's log:
>
> EAL: Detected lcore 0 as core 0 on socket 0
> EAL: Detected lcore 1 as core 1 on socket 0
> EAL: Detected lcore 2 as core 2 on socket 0
> EAL: Detected lcore 3 as core 3 on socket 0
> EAL: Detected lcore 4 as core 4 on socket 0
> EAL: Detected lcore 5 as core 5 on socket 0
> EAL: Detected lcore 6 as core 8 on socket 0
> EAL: Detected lcore 7 as core 9 on socket 0
> EAL: Detected lcore 8 as core 10 on socket 0
> EAL: Detected lcore 9 as core 11 on socket 0
> EAL: Detected lcore 10 as core 12 on socket 0
> EAL: Detected lcore 11 as core 13 on socket 0
> EAL: Detected lcore 12 as core 0 on socket 1
> EAL: Detected lcore 13 as core 1 on socket 1
> EAL: Detected lcore 14 as core 2 on socket 1
> EAL: Detected lcore 15 as core 3 on socket 1
> EAL: Detected lcore 16 as core 4 on socket 1
> EAL: Detected lcore 17 as core 5 on socket 1
> EAL: Detected lcore 18 as core 8 on socket 1
> EAL: Detected lcore 19 as core 9 on socket 1
> EAL: Detected lcore 20 as core 10 on socket 1
> EAL: Detected lcore 21 as core 11 on socket 1
> EAL: Detected lcore 22 as core 12 on socket 1
> EAL: Detected lcore 23 as core 13 on socket 1
> EAL: Support maximum 128 logical core(s) by configuration.
> EAL: Detected 24 lcore(s)
> EAL: Setting up physically contiguous memory...
> EAL: Ask a virtual area of 0x80000000 bytes
> EAL: Virtual area found at 0x7f2940000000 (size = 0x80000000)
> EAL: Ask a virtual area of 0x80000000 bytes
> EAL: Virtual area found at 0x7f2880000000 (size = 0x80000000)
> EAL: Requesting 2 pages of size 1024MB from socket 0
> EAL: Requesting 2 pages of size 1024MB from socket 1
> EAL: TSC frequency is ~2300002 KHz
> EAL: Master lcore 0 is ready (tid=f6cdc8a0;cpuset=[0])
> EAL: lcore 2 is ready (tid=f58d2700;cpuset=[2])
> EAL: lcore 1 is ready (tid=f62d3700;cpuset=[1])
> EAL: lcore 3 is ready (tid=f4ed1700;cpuset=[3])
> EAL: PCI device 0000:02:00.0 on NUMA socket 0
> EAL: probe driver: 15b3:1003 librte_pmd_mlx4
> PMD: librte_pmd_mlx4: PCI information matches, using device "mlx4_0" (VF:
> false)
> PMD: librte_pmd_mlx4: 1 port(s) detected
> PMD: librte_pmd_mlx4: ibv_exp_query_device() failed
Make sure ib_uverbs, mlx4_en, mlx4_ib and mlx4_core are all loaded and
coming from Mellanox OFED (normally not the original modules provided by
your Linux distribution), and that you're using an up-to-date firmware
version, although installing MLNX_OFED should have taken care of these
automatically.
--
Adrien Mazarguil
6WIND
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [dpdk-users] HELP: MLX4 problem
2016-01-12 15:47 ` Adrien Mazarguil
@ 2016-01-14 2:17 ` Masaru Kawashima
2016-01-14 10:04 ` Adrien Mazarguil
0 siblings, 1 reply; 4+ messages in thread
From: Masaru Kawashima @ 2016-01-14 2:17 UTC (permalink / raw)
To: users, Olga Shern
Hi Olga,
Thank you for your advice.
I checked the kernel modules you've shown (ib_uverbs, mlx4_en, mlx4_ib and
mlx4_core) are all loaded fine.
Following is the output of lsmod:
> Module Size Used by
> fuse 73530 0
> vtsspp 362000 0
> sep3_15 546746 0
> pax 4312 0
> blcr 115497 0
> blcr_imports 10683 1 blcr
> autofs4 26481 3
> vboxpci 14851 0
> vboxnetadp 18352 0
> vboxnetflt 18816 0
> vboxdrv 454718 3 vboxpci,vboxnetadp,vboxnetflt
> 8021q 25349 0
> garp 7152 1 8021q
> stp 2218 1 garp
> llc 5546 2 garp,stp
> cpufreq_ondemand 10544 24
> acpi_cpufreq 7763 0
> freq_table 4936 2 cpufreq_ondemand,acpi_cpufreq
> mperf 1557 1 acpi_cpufreq
> cachefiles 36893 1
> fscache 55636 1 cachefiles
> ipt_REJECT 2351 2
> iptable_filter 2793 1
> ipt_MASQUERADE 2466 1
> iptable_nat 6158 1
> nf_nat 22759 2 ipt_MASQUERADE,iptable_nat
> nf_conntrack_ipv4 9506 14 iptable_nat,nf_nat
> nf_defrag_ipv4 1483 1 nf_conntrack_ipv4
> ip_tables 17831 2 iptable_filter,iptable_nat
> nf_conntrack_netbios_ns 1323 0
> nf_conntrack_broadcast 1471 1 nf_conntrack_netbios_ns
> ip6t_REJECT 4628 2
> nf_conntrack_ipv6 8337 10
> nf_defrag_ipv6 11156 1 nf_conntrack_ipv6
> xt_state 1492 21
> nf_conntrack 79758 8
ipt_MASQUERADE,iptable_nat,nf_nat,nf_conntrack_ipv4,nf_conntrack_netbios_ns,nf_conntrack_broadcast,nf_conntrack_ipv6,xt_state
> ip6table_filter 2889 1
> ip6_tables 18732 1 ip6table_filter
> rdma_ucm 16467 0
> ib_ucm 12153 0
> rdma_cm 43237 1 rdma_ucm
> iw_cm 30527 1 rdma_cm
> configfs 29570 2 rdma_cm
> ib_ipoib 124388 0
> ib_cm 38274 3 ib_ucm,rdma_cm,ib_ipoib
> ib_uverbs 65802 2 rdma_ucm,ib_ucm
> ib_umad 14678 0
> mlx5_ib 161275 0
> mlx5_core 325765 1 mlx5_ib
> xfs 1128471 2
> exportfs 4236 1 xfs
> ext3 240548 1
> jbd 80858 1 ext3
> knem 32213 0
> uinput 7992 0
> iTCO_wdt 7115 0
> iTCO_vendor_support 3056 1 iTCO_wdt
> ax88179_178a 19166 0
> usbnet 40894 1 ax88179_178a
> mii 5376 2 ax88179_178a,usbnet
> microcode 112685 0
> igb 197504 0
> raid1 32045 1
> raid0 12417 1
> power_meter 9169 0
> i2c_i801 11359 0
> lpc_ich 12803 0
> mfd_core 1895 1 lpc_ich
> shpchp 32778 0
> sg 29350 0
> mlx4_ib 195595 0
> ib_sa 24561 5 rdma_ucm,rdma_cm,ib_ipoib,ib_cm,mlx4_ib
> ib_mad 47060 4 ib_cm,ib_umad,mlx4_ib,ib_sa
> ib_core 126865 12
rdma_ucm,ib_ucm,rdma_cm,iw_cm,ib_ipoib,ib_cm,ib_uverbs,ib_umad,mlx5_ib,mlx4_ib,ib_sa,ib_mad
> ib_addr 8243 3 rdma_ucm,rdma_cm,ib_core
> ixgbe 243936 0
> mlx4_en 110837 0
> dca 7101 2 igb,ixgbe
> ipv6 318183 97
vboxnetflt,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6,ib_ipoib,ib_core,ib_addr
> ptp 9614 3 igb,ixgbe,mlx4_en
> mdio 4769 1 ixgbe
> pps_core 11458 1 ptp
> mlx4_core 332788 2 mlx4_ib,mlx4_en
> mlx_compat 31376 17
rdma_ucm,ib_ucm,rdma_cm,iw_cm,ib_ipoib,ib_cm,ib_uverbs,ib_umad,mlx5_ib,mlx5_core,mlx4_ib,ib_sa,ib_mad,ib_core,ib_addr,mlx4_en,mlx4_core
> snd_hda_codec_hdmi 28922 1
> snd_hda_codec_realtek 43811 1
> snd_hda_intel 40562 2
> snd_hda_codec 179208 3
snd_hda_codec_hdmi,snd_hda_codec_realtek,snd_hda_intel
> snd_hwdep 6780 1 snd_hda_codec
> snd_seq 55727 0
> snd_seq_device 6500 1 snd_seq
> snd_pcm 87790 3
snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec
> snd_timer 22443 2 snd_seq,snd_pcm
> snd 70745 13
snd_hda_codec_hdmi,snd_hda_codec_realtek,snd_hda_intel,snd_hda_codec,snd_hwdep,snd_seq,snd_seq_device,snd_pcm,snd_timer
> soundcore 7958 1 snd
> snd_page_alloc 8856 2 snd_hda_intel,snd_pcm
> nvidia 8363378 52
> ext4 374405 1
> jbd2 93427 1 ext4
> mbcache 8193 2 ext3,ext4
> sd_mod 40217 16
> crc_t10dif 1541 1 sd_mod
> xhci_hcd 148886 0
> ahci 42247 9
> nouveau 995383 0
> ttm 80590 1 nouveau
> drm_kms_helper 44321 1 nouveau
> drm 280012 3 nouveau,ttm,drm_kms_helper
> i2c_algo_bit 5935 2 igb,nouveau
> i2c_core 31084 7
igb,i2c_i801,nvidia,nouveau,drm_kms_helper,drm,i2c_algo_bit
> mxm_wmi 1967 1 nouveau
> video 20674 1 nouveau
> output 2409 1 video
> wmi 6287 2 nouveau,mxm_wmi
> dm_mirror 14384 0
> dm_region_hash 12085 1 dm_mirror
> dm_log 9930 2 dm_mirror,dm_region_hash
> dm_mod 84337 2 dm_mirror,dm_log
And, I believe they are parts of Mellanox OFED, because the module path
listed in the modules.dep file
are started with "weak-updates/mlnx-ofa_kernel/drivers/".
Firmware version is up to date. Following is the output of flint query
> Image type: FS2
> FW Version: 2.35.5100
> FW Release Date: 6.9.2015
> Product Version: 02.35.51.00
> Rom Info: type=PXE version=3.4.648 devid=4099
> Device ID: 4099
> Description: Node Port1 Port2 Sys
image
> GUIDs: ffffffffffffffff ffffffffffffffff ffffffffffffffff
ffffffffffffffff
> MACs: e41d2d0759a0 e41d2d0759a1
> VSD:
> PSID: MT_1060140023
Now, what can I do next again? I still need more help!
2016-01-13 0:47 GMT+09:00 Adrien Mazarguil <adrien.mazarguil@6wind.com>:
> Hi Masaru,
>
> On Tue, Jan 12, 2016 at 06:43:53PM +0900, Masaru Kawashima wrote:
> > Hi, all
> >
> > I'm trying to use dpdk with Mellanox ConnectX-3.
> > But, mlnx4 pmd cannot work fine.
> >
> > Following is my environment:
> >
> > OS: CentOS 6.5 (x86_64)
> > Mellanox OFED: MLNX_OFED_LINUX-3.1-1.1.0.1-rhel6.5-x86_64
> > DPDK: MLNX_DPDK-2.1_1.1
> >
> > I'm trying to run testpmd, but librte_pmd_mlx4 reports
> > "ibv_exp_query_device() failed",
> > and EAL reports "No probed ethernet devices".
> >
> > What can I do next to work mlx4 pmd fine?
> >
> >
> > Following is the testpmd's log:
> >
> > EAL: Detected lcore 0 as core 0 on socket 0
> > EAL: Detected lcore 1 as core 1 on socket 0
> > EAL: Detected lcore 2 as core 2 on socket 0
> > EAL: Detected lcore 3 as core 3 on socket 0
> > EAL: Detected lcore 4 as core 4 on socket 0
> > EAL: Detected lcore 5 as core 5 on socket 0
> > EAL: Detected lcore 6 as core 8 on socket 0
> > EAL: Detected lcore 7 as core 9 on socket 0
> > EAL: Detected lcore 8 as core 10 on socket 0
> > EAL: Detected lcore 9 as core 11 on socket 0
> > EAL: Detected lcore 10 as core 12 on socket 0
> > EAL: Detected lcore 11 as core 13 on socket 0
> > EAL: Detected lcore 12 as core 0 on socket 1
> > EAL: Detected lcore 13 as core 1 on socket 1
> > EAL: Detected lcore 14 as core 2 on socket 1
> > EAL: Detected lcore 15 as core 3 on socket 1
> > EAL: Detected lcore 16 as core 4 on socket 1
> > EAL: Detected lcore 17 as core 5 on socket 1
> > EAL: Detected lcore 18 as core 8 on socket 1
> > EAL: Detected lcore 19 as core 9 on socket 1
> > EAL: Detected lcore 20 as core 10 on socket 1
> > EAL: Detected lcore 21 as core 11 on socket 1
> > EAL: Detected lcore 22 as core 12 on socket 1
> > EAL: Detected lcore 23 as core 13 on socket 1
> > EAL: Support maximum 128 logical core(s) by configuration.
> > EAL: Detected 24 lcore(s)
> > EAL: Setting up physically contiguous memory...
> > EAL: Ask a virtual area of 0x80000000 bytes
> > EAL: Virtual area found at 0x7f2940000000 (size = 0x80000000)
> > EAL: Ask a virtual area of 0x80000000 bytes
> > EAL: Virtual area found at 0x7f2880000000 (size = 0x80000000)
> > EAL: Requesting 2 pages of size 1024MB from socket 0
> > EAL: Requesting 2 pages of size 1024MB from socket 1
> > EAL: TSC frequency is ~2300002 KHz
> > EAL: Master lcore 0 is ready (tid=f6cdc8a0;cpuset=[0])
> > EAL: lcore 2 is ready (tid=f58d2700;cpuset=[2])
> > EAL: lcore 1 is ready (tid=f62d3700;cpuset=[1])
> > EAL: lcore 3 is ready (tid=f4ed1700;cpuset=[3])
> > EAL: PCI device 0000:02:00.0 on NUMA socket 0
> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4
> > PMD: librte_pmd_mlx4: PCI information matches, using device "mlx4_0" (VF:
> > false)
> > PMD: librte_pmd_mlx4: 1 port(s) detected
> > PMD: librte_pmd_mlx4: ibv_exp_query_device() failed
>
> Make sure ib_uverbs, mlx4_en, mlx4_ib and mlx4_core are all loaded and
> coming from Mellanox OFED (normally not the original modules provided by
> your Linux distribution), and that you're using an up-to-date firmware
> version, although installing MLNX_OFED should have taken care of these
> automatically.
>
> --
> Adrien Mazarguil
> 6WIND
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [dpdk-users] HELP: MLX4 problem
2016-01-14 2:17 ` Masaru Kawashima
@ 2016-01-14 10:04 ` Adrien Mazarguil
0 siblings, 0 replies; 4+ messages in thread
From: Adrien Mazarguil @ 2016-01-14 10:04 UTC (permalink / raw)
To: Masaru Kawashima; +Cc: users
On Thu, Jan 14, 2016 at 11:17:03AM +0900, Masaru Kawashima wrote:
> Hi Olga,
>
> Thank you for your advice.
>
> I checked the kernel modules you've shown (ib_uverbs, mlx4_en, mlx4_ib and
> mlx4_core) are all loaded fine.
[...]
Kernel modules look fine indeed.
There is an additional possibility for the PMD to be linked with the "wrong"
libibverbs (your distribution's version or one installed manually, not the
one coming with OFED) and/or that libibverbs loads the wrong libmlx4 for a
similar reason.
If you have several libibverbs.so* and/or libmlx4*.so* on your system, make
sure DPDK links against OFED's version. I suggest uninstalling all previous
versions to be sure.
Check also that you do not have both /usr/include/infiniband and
/usr/local/include/infiniband.
--
Adrien Mazarguil
6WIND
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2016-01-14 10:04 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-01-12 9:43 [dpdk-users] HELP: MLX4 problem Masaru Kawashima
2016-01-12 15:47 ` Adrien Mazarguil
2016-01-14 2:17 ` Masaru Kawashima
2016-01-14 10:04 ` Adrien Mazarguil
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).