Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies

[Raghav Sethi <raghavs@CS.Princeton.EDU>]

Hi Adrien,

I've copied dev@dpdk.

I re-installed Mellanox OFED, and then re-compiled DPDK without the flags
for the static library. I'm running Ubuntu 14.04. However, I still get the
same error message (below).

I then tried re-installing the Mellanox libraries on dpdk.org, and compiled
another copy of DPDK with them statically linked. This appears to work
perfectly. I no longer get the permission issues I was getting before,
possibly that was unrelated/fixed by OFED. Now both OFED, and OFED+dpdk.org
libs statically linked have the same issue:

EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
/usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, version
IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference
libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
/usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow,
version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
reference
libibverbs: Warning: no userspace device-specific driver found for
/sys/class/infiniband_verbs/uverbs0
PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
EAL: No probed ethernet devices

The OFED install was successful (and it apparently removes/overwrites old
versions of the libraries). The installer also checks the firmware
versions, and it says I'm on the latest. Loading the kernel modules is also
successful, so I can't for the life of me figure out what this issue is now.

Again, just to be clear, the full list of steps was:
1. Install OFED, run /etc/init.d/openibd restart
2. Compile DPDK with the CONFIG_RTE_LIBRTE_MLX4_PMD=y
3. Set up hugepages
4. Load kernel modules
5. Run testpmd

Best,
Raghav

On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <adrien.mazarguil@6wind.com>
wrote:

> Hi Raghav,
>
> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
> > Hi Adrien,
> >
> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
> I'm
> > a grad student at Princeton working on a research project.
>
> It's fine, but let's discuss on the ML as much as possible next time so the
> community can benefit from it. Since very few people use this driver at the
> moment (DPDK 2.0 is not even out yet), I have my own questions for you.
>
> > Given your patch (which was very helpful, thanks), here is the series of
> > all steps that I'm taking to run testpmd on a ConnectX-3 EN 40G
> >
> > 1. Compile and install the mellanox-specific libraries listed here:
> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
>
> This is fine, but did you use the provided Makefile to compile these
> libraries statically? If so, no issues?
>
> Also you did not install Mellanox OFED? You should, because updated kernel
> modules come from that distribution. It won't work properly (or at all)
> without them.
>
> The libraries from DPDK.org are actually optional once you've installed
> Mellanox OFED. They only improve performance. Remember to update firmwares
> also.
>
> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS
> > 3. Create a new configuration which includes
> > the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
> > 4. Compile DPDK 2.0 with the new configuration using setup.sh
> > 5. Set up hugepages using setup.sh
> > 6. Load the kernel modules ib_uverbs mlx4_en mlx4_core mlx4_ib
> > 7. Run testpmd with the right PCI bus address
> >
> > However, I consistently get:
> > sudo: unable to execute ./testpmd: Permission denied
>
> Now that doesn't make sense, isn't that file missing permissions or
> something? Are you running RHEL with a restrictive SELinux policy?
> Is sudo configured to allow you to run testpmd?
>
> In any case, I don't think it's related to librte_pmd_mlx4.
>
> > I tried taking the interface down with ifconfig because
> > (./tools/dpdk_nic_bind.py --status shows the interface as active), but
> that
> > didn't change anything.
>
> No, you should leave the interfaces up, it's not supposed to disappear, the
> PMD will bring them back up anyway. Actually the Linux kernel interface
> remains fully functional even while DPDK applications are running. You can
> send packets out of it but as long as both the kernel and DPDK applications
> share the same MAC address, only DPDK applications will receive replies. If
> you change the MAC address on the kernel interface however, you'll be able
> to use it normally.
>
> > I tried installing the mellanox libraries globally, but then I see these
> > error messages:
> >
> > -------
> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
> > EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> version
> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference
> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow,
> > version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> > reference
> > libibverbs: Warning: no userspace device-specific driver found for
> > /sys/class/infiniband_verbs/uverbs0
> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> > EAL: No probed ethernet devices
> > Interactive-mode selected
> > EAL: Error - exiting with code: 1
> >   Cause: Creation of mbuf pool for socket 0 failed
> > --------
>
> That's because the libibverbs version you're using is not compatible with
> the userspace drivers installed on your system (libibverbs uses dlopen() to
> load liblmx4). Even though you've installed both from DPDK.org, seems you
> have another version installed by your distribution packages.
>
> If you intend to use dynamic libraries, you should uninstall libibverbs,
> libmlx4 and related packages first.
>
> Regards,
>
> --
> Adrien Mazarguil
> 6WIND
>