Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies

[Olga Shern <olgas@mellanox.com>]

If you compiled pmd statically with libibverbs and libmlx4 you shouldn't see it in ldd.
Seems that something went wrong.
In any case, ofed doesn't install libraries under /usr/local
Please remove all mlx related libraries under /usr/local/ like libmlx4 and libibverbs





-------- Original message --------
From: Raghav Sethi
Date:05/04/2015 00:29 (GMT+02:00)
To: Olga Shern ,Adrien Mazarguil ,dev@dpdk.org,Xiaozhou Li
Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies


Hi Olga,

Thanks very much for helping.

I tried stopping and starting, after stop lsmod | grep mlx had no matches. After start, it had the output listed below. The device is the same as on the rest of the machines - Mellanox ConnectX3-EN 40G single port. The application parameters are the same as the ones listed on the docs (except of course PCI address is different):

sudo ./testpmd -c 0xff -n 4 -w 0000:06:00.0 -- --rxq=2 --txq=2 -i

Outputs you requested:

1. lspci | grep Mellanox
06:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

2. ibdev2netdev
mlx4_0 port 1 ==> p3p1 (Up)

3. ldd testpmd
        linux-vdso.so.1 =>  (0x00007fff7c1dc000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1a7dc07000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1a7d9ff000)
        libibverbs.so.1 => /usr/local/lib/libibverbs.so.1 (0x00007f1a7d7eb000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1a7d5e7000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1a7d3c9000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1a7d003000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1a7df19000)

4. lsmod | grep mlx
mlx5_ib               138046  0
mlx5_core             108376  1 mlx5_ib
mlx4_ib               199880  0
ib_sa                  43732  5 rdma_cm,ib_cm,mlx4_ib,rdma_ucm,ib_ipoib
mlx4_en               137047  0
ib_mad                 51544  4 ib_cm,ib_sa,mlx4_ib,ib_umad
vxlan                  37619  1 mlx4_en
ib_core               130498  12 rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib
mlx4_core             332375  2 mlx4_en,mlx4_ib
compat                 13738  17 rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_en,mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_addr,ib_core,ib_umad,ib_uverbs,mlx4_core,mlx5_core,rdma_ucm,ib_ipoib
ptp                    18933  3 igb,ixgbe,mlx4_en

Best regards,
Raghav

On Sat, Apr 4, 2015 at 5:04 PM Olga Shern <olgas@mellanox.com<mailto:olgas@mellanox.com>> wrote:
Hi Raghav,

After running /etc/init.d/openibd restart you don’t need to load the modules, they are loaded by the openibd script.
Can you try to run :
/etc/init.d/openibd stop
Verify that modules are not loaded
/etc/init.d/openib start

If it won’t help:
What Mellanox NIC do you have on the “problematic” machine?
What command line do you use to run the application?

Please send output of:

1.       ”lspci | grep Mellanox”

2.       ibdev2netdev

3.       ldd <your application>

4.       lsmod

Thanks
Olga


From: Raghav Sethi [mailto:raghavs@CS.Princeton.EDU<mailto:raghavs@CS.Princeton.EDU>]
Sent: Saturday, April 04, 2015 11:54 PM
To: Olga Shern; Adrien Mazarguil; dev@dpdk.org<mailto:dev@dpdk.org>; Xiaozhou Li

Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies

Hi Olga,

Thanks for your reply. OFED version was 2.4-1.0.4 for ubuntu14.04-x86_64

I tried both variants:
- Compiling with dynamic linking against OFED libraries
- Compiling with static linking against dpdk.org<http://dpdk.org> libraries

Both have the same issue.

Best,
Raghav

On Sat, Apr 4, 2015 at 4:49 PM Olga Shern <olgas@mellanox.com<mailto:olgas@mellanox.com>> wrote:
Hi Raghav,

What OFED version did you install?

There are 2 ways to compile mlx4:
1. You can compile it dynamically with the libibverbs  that are coming with Mellanox_OFED 2.4
2. To get better performance: compile it statically with libibverbs and libmlx4 that can be downloaded from http://dpdk.org/download, the README inside librte_pmd_mlx4 explains how to do it
     If you compile the pmd statically then MLNX_OFED is needed only for kernel modules.

Best Regards,
Olga


-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>] On Behalf Of Raghav Sethi
Sent: Friday, April 03, 2015 9:21 PM
To: Adrien Mazarguil; dev@dpdk.org<mailto:dev@dpdk.org>; Xiaozhou Li
Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies

Hi Adrien,

Quick update: So I just tried the setup again on a new machine with the exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a charm.

Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
>From the error message it appears that the libraries are incorrectly installed/configured. However, the OFED uninstall script removes the libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so,  libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both machines, and they're exactly identical. So it seems either the OFED installation/uninstallation is somehow flawed, or the error message from testpmd is misleading.

Best,
Raghav

On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs@cs.princeton.edu<mailto:raghavs@cs.princeton.edu>>
wrote:

> Hi Adrien,
>
> I've copied dev@dpdk.
>
> I re-installed Mellanox OFED, and then re-compiled DPDK without the
> flags for the static library. I'm running Ubuntu 14.04. However, I
> still get the same error message (below).
>
> I then tried re-installing the Mellanox libraries on dpdk.org<http://dpdk.org>, and
> compiled another copy of DPDK with them statically linked. This
> appears to work perfectly. I no longer get the permission issues I was
> getting before, possibly that was unrelated/fixed by OFED. Now both
> OFED, and OFED+ dpdk.org<http://dpdk.org> libs statically linked have the same issue:
>
> EAL: PCI device 0000:06:00.0 on NUMA socket 0
> EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> version
> IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> reference
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
> ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
> libibverbs.so.1 with link time reference
> libibverbs: Warning: no userspace device-specific driver found for
> /sys/class/infiniband_verbs/uverbs0
> PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> EAL: No probed ethernet devices
>
> The OFED install was successful (and it apparently removes/overwrites
> old versions of the libraries). The installer also checks the firmware
> versions, and it says I'm on the latest. Loading the kernel modules is
> also successful, so I can't for the life of me figure out what this issue is now.
>
> Again, just to be clear, the full list of steps was:
> 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with
> the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel
> modules 5. Run testpmd
>
> Best,
> Raghav
>
> On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <
> adrien.mazarguil@6wind.com<mailto:adrien.mazarguil@6wind.com>> wrote:
>
>> Hi Raghav,
>>
>> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
>> > Hi Adrien,
>> >
>> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
>> I'm
>> > a grad student at Princeton working on a research project.
>>
>> It's fine, but let's discuss on the ML as much as possible next time
>> so the community can benefit from it. Since very few people use this
>> driver at the moment (DPDK 2.0 is not even out yet), I have my own
>> questions for you.
>>
>> > Given your patch (which was very helpful, thanks), here is the
>> > series of all steps that I'm taking to run testpmd on a ConnectX-3
>> > EN 40G
>> >
>> > 1. Compile and install the mellanox-specific libraries listed here:
>> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
>>
>> This is fine, but did you use the provided Makefile to compile these
>> libraries statically? If so, no issues?
>>
>> Also you did not install Mellanox OFED? You should, because updated
>> kernel modules come from that distribution. It won't work properly
>> (or at all) without them.
>>
>> The libraries from DPDK.org are actually optional once you've
>> installed Mellanox OFED. They only improve performance. Remember to
>> update firmwares also.
>>
>> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new
>> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
>> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5.
>> > Set up hugepages using setup.sh 6. Load the kernel modules
>> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right
>> > PCI bus address
>> >
>> > However, I consistently get:
>> > sudo: unable to execute ./testpmd: Permission denied
>>
>> Now that doesn't make sense, isn't that file missing permissions or
>> something? Are you running RHEL with a restrictive SELinux policy?
>> Is sudo configured to allow you to run testpmd?
>>
>> In any case, I don't think it's related to librte_pmd_mlx4.
>>
>> > I tried taking the interface down with ifconfig because
>> > (./tools/dpdk_nic_bind.py --status shows the interface as active),
>> > but
>> that
>> > didn't change anything.
>>
>> No, you should leave the interfaces up, it's not supposed to
>> disappear, the PMD will bring them back up anyway. Actually the Linux
>> kernel interface remains fully functional even while DPDK
>> applications are running. You can send packets out of it but as long
>> as both the kernel and DPDK applications share the same MAC address,
>> only DPDK applications will receive replies.
>> If
>> you change the MAC address on the kernel interface however, you'll be
>> able to use it normally.
>>
>> > I tried installing the mellanox libraries globally, but then I see
>> > these error messages:
>> >
>> > -------
>> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
>> > EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
>> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
>> version
>> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
>> > reference
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
>> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
>> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
>> > libibverbs.so.1 with link time reference
>> > libibverbs: Warning: no userspace device-specific driver found for
>> > /sys/class/infiniband_verbs/uverbs0
>> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
>> > EAL: No probed ethernet devices
>> > Interactive-mode selected
>> > EAL: Error - exiting with code: 1
>> >   Cause: Creation of mbuf pool for socket 0 failed
>> > --------
>>
>> That's because the libibverbs version you're using is not compatible
>> with the userspace drivers installed on your system (libibverbs uses
>> dlopen() to load liblmx4). Even though you've installed both from
>> DPDK.org, seems you have another version installed by your
>> distribution packages.
>>
>> If you intend to use dynamic libraries, you should uninstall
>> libibverbs,
>> libmlx4 and related packages first.
>>
>> Regards,
>>
>> --
>> Adrien Mazarguil
>> 6WIND
>>
>