Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies

[Raghav Sethi <raghavs@CS.Princeton.EDU>]

Hi Olga,

Thanks for your reply. OFED version was 2.4-1.0.4 for ubuntu14.04-x86_64

I tried both variants:
- Compiling with dynamic linking against OFED libraries
- Compiling with static linking against dpdk.org libraries

Both have the same issue.

Best,
Raghav

On Sat, Apr 4, 2015 at 4:49 PM Olga Shern <olgas@mellanox.com> wrote:

> Hi Raghav,
>
> What OFED version did you install?
>
> There are 2 ways to compile mlx4:
> 1. You can compile it dynamically with the libibverbs  that are coming
> with Mellanox_OFED 2.4
> 2. To get better performance: compile it statically with libibverbs and
> libmlx4 that can be downloaded from http://dpdk.org/download, the README
> inside librte_pmd_mlx4 explains how to do it
>      If you compile the pmd statically then MLNX_OFED is needed only for
> kernel modules.
>
> Best Regards,
> Olga
>
>
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Raghav Sethi
> Sent: Friday, April 03, 2015 9:21 PM
> To: Adrien Mazarguil; dev@dpdk.org; Xiaozhou Li
> Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
>
> Hi Adrien,
>
> Quick update: So I just tried the setup again on a new machine with the
> exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a
> charm.
>
> Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
> From the error message it appears that the libraries are incorrectly
> installed/configured. However, the OFED uninstall script removes the
> libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so,
> libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both
> machines, and they're exactly identical. So it seems either the OFED
> installation/uninstallation is somehow flawed, or the error message from
> testpmd is misleading.
>
> Best,
> Raghav
>
> On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs@cs.princeton.edu>
> wrote:
>
> > Hi Adrien,
> >
> > I've copied dev@dpdk.
> >
> > I re-installed Mellanox OFED, and then re-compiled DPDK without the
> > flags for the static library. I'm running Ubuntu 14.04. However, I
> > still get the same error message (below).
> >
> > I then tried re-installing the Mellanox libraries on dpdk.org, and
> > compiled another copy of DPDK with them statically linked. This
> > appears to work perfectly. I no longer get the permission issues I was
> > getting before, possibly that was unrelated/fixed by OFED. Now both
> > OFED, and OFED+ dpdk.org libs statically linked have the same issue:
> >
> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
> > EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> > version
> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> > reference
> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
> > libibverbs.so.1 with link time reference
> > libibverbs: Warning: no userspace device-specific driver found for
> > /sys/class/infiniband_verbs/uverbs0
> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> > EAL: No probed ethernet devices
> >
> > The OFED install was successful (and it apparently removes/overwrites
> > old versions of the libraries). The installer also checks the firmware
> > versions, and it says I'm on the latest. Loading the kernel modules is
> > also successful, so I can't for the life of me figure out what this
> issue is now.
> >
> > Again, just to be clear, the full list of steps was:
> > 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with
> > the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel
> > modules 5. Run testpmd
> >
> > Best,
> > Raghav
> >
> > On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <
> > adrien.mazarguil@6wind.com> wrote:
> >
> >> Hi Raghav,
> >>
> >> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
> >> > Hi Adrien,
> >> >
> >> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
> >> I'm
> >> > a grad student at Princeton working on a research project.
> >>
> >> It's fine, but let's discuss on the ML as much as possible next time
> >> so the community can benefit from it. Since very few people use this
> >> driver at the moment (DPDK 2.0 is not even out yet), I have my own
> >> questions for you.
> >>
> >> > Given your patch (which was very helpful, thanks), here is the
> >> > series of all steps that I'm taking to run testpmd on a ConnectX-3
> >> > EN 40G
> >> >
> >> > 1. Compile and install the mellanox-specific libraries listed here:
> >> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
> >>
> >> This is fine, but did you use the provided Makefile to compile these
> >> libraries statically? If so, no issues?
> >>
> >> Also you did not install Mellanox OFED? You should, because updated
> >> kernel modules come from that distribution. It won't work properly
> >> (or at all) without them.
> >>
> >> The libraries from DPDK.org are actually optional once you've
> >> installed Mellanox OFED. They only improve performance. Remember to
> >> update firmwares also.
> >>
> >> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new
> >> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
> >> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5.
> >> > Set up hugepages using setup.sh 6. Load the kernel modules
> >> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right
> >> > PCI bus address
> >> >
> >> > However, I consistently get:
> >> > sudo: unable to execute ./testpmd: Permission denied
> >>
> >> Now that doesn't make sense, isn't that file missing permissions or
> >> something? Are you running RHEL with a restrictive SELinux policy?
> >> Is sudo configured to allow you to run testpmd?
> >>
> >> In any case, I don't think it's related to librte_pmd_mlx4.
> >>
> >> > I tried taking the interface down with ifconfig because
> >> > (./tools/dpdk_nic_bind.py --status shows the interface as active),
> >> > but
> >> that
> >> > didn't change anything.
> >>
> >> No, you should leave the interfaces up, it's not supposed to
> >> disappear, the PMD will bring them back up anyway. Actually the Linux
> >> kernel interface remains fully functional even while DPDK
> >> applications are running. You can send packets out of it but as long
> >> as both the kernel and DPDK applications share the same MAC address,
> >> only DPDK applications will receive replies.
> >> If
> >> you change the MAC address on the kernel interface however, you'll be
> >> able to use it normally.
> >>
> >> > I tried installing the mellanox libraries globally, but then I see
> >> > these error messages:
> >> >
> >> > -------
> >> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
> >> > EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
> >> > libibverbs: Warning: couldn't load driver
> '/usr/lib/libibverbs/libmlx5':
> >> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> >> version
> >> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> >> > reference
> >> > libibverbs: Warning: couldn't load driver
> '/usr/lib/libibverbs/libmlx4':
> >> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
> >> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
> >> > libibverbs.so.1 with link time reference
> >> > libibverbs: Warning: no userspace device-specific driver found for
> >> > /sys/class/infiniband_verbs/uverbs0
> >> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> >> > EAL: No probed ethernet devices
> >> > Interactive-mode selected
> >> > EAL: Error - exiting with code: 1
> >> >   Cause: Creation of mbuf pool for socket 0 failed
> >> > --------
> >>
> >> That's because the libibverbs version you're using is not compatible
> >> with the userspace drivers installed on your system (libibverbs uses
> >> dlopen() to load liblmx4). Even though you've installed both from
> >> DPDK.org, seems you have another version installed by your
> >> distribution packages.
> >>
> >> If you intend to use dynamic libraries, you should uninstall
> >> libibverbs,
> >> libmlx4 and related packages first.
> >>
> >> Regards,
> >>
> >> --
> >> Adrien Mazarguil
> >> 6WIND
> >>
> >
>