* [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
@ 2015-04-02 9:33 Adrien Mazarguil
2015-04-03 13:53 ` Thomas Monjalon
[not found] ` <CAC2O_6oqG=3H0R5L6+ZX=1FzNj=mprDzOcUo_e6wezJxzu=ybw@mail.gmail.com>
0 siblings, 2 replies; 10+ messages in thread
From: Adrien Mazarguil @ 2015-04-02 9:33 UTC (permalink / raw)
To: dev
- libmlx4 and libibverbs dependencies distributed with Mellanox OFED are now
also available on DPDK.org to make installation easier.
- Document Mellanox OFED and firmware versions to use.
- Add links to Mellanox and its community websites.
- Add kernel modules parameters section.
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
doc/guides/nics/mlx4.rst | 85 ++++++++++++++++++++++++++++++++++++++++++++----
1 file changed, 78 insertions(+), 7 deletions(-)
diff --git a/doc/guides/nics/mlx4.rst b/doc/guides/nics/mlx4.rst
index b26c219..93a7f57 100644
--- a/doc/guides/nics/mlx4.rst
+++ b/doc/guides/nics/mlx4.rst
@@ -31,8 +31,15 @@ MLX4 poll mode driver library
=============================
The MLX4 poll mode driver library (**librte_pmd_mlx4**) implements support
-for **Mellanox ConnectX-3** 10/40 Gbps adapters (EN 40, EN 10, Pro EN 40) as
-well as their virtual functions (VF) in SR-IOV context.
+for **Mellanox ConnectX-3 EN** 10/40 Gbps adapters as well as their virtual
+functions (VF) in SR-IOV context.
+
+Information and documentation about this family of adapters can be found on
+the `Mellanox website <http://www.mellanox.com>`_. Help is also provided by
+the `Mellanox community <http://community.mellanox.com/welcome>`_.
+
+There is also a `section dedicated to this poll mode driver
+<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`_.
.. note::
@@ -78,7 +85,7 @@ Features and limitations
- Multiple MAC addresses (unicast, multicast) can be configured.
- Scattered packets are supported for TX and RX.
-..
+.. break
- RSS hash key cannot be modified.
- Hardware counters are not implemented (they are software counters).
@@ -90,6 +97,8 @@ Configuration
Compilation options
~~~~~~~~~~~~~~~~~~~
+These options can be modified in the ``.config`` file.
+
- ``CONFIG_RTE_LIBRTE_MLX4_PMD`` (default **n**)
Toggle compilation of librte_pmd_mlx4 itself.
@@ -146,6 +155,30 @@ Run-time configuration
- **ethtool** operations on related kernel interfaces also affect the PMD.
+Kernel module parameters
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The **mlx4_core** kernel module has several parameters that affect the
+behavior and/or the performance of librte_pmd_mlx4. Some of them are described
+below.
+
+- **num_vfs** (integer or triplet, optionally prefixed by device address
+ strings)
+
+ Create the given number of VFs on the specified devices.
+
+- **log_num_mgm_entry_size** (integer)
+
+ Device-managed flow steering (DMFS) is required by DPDK applications. It is
+ enabled by using a negative value, the last four bits of which have a
+ special meaning.
+
+ - **-1**: force device-managed flow steering (DMFS).
+ - **-7**: configure optimized steering mode to improve performance with the
+ following limitation: Ethernet frames with the port MAC address as the
+ destination cannot be received, even in promiscuous mode. Additional MAC
+ addresses can still be set by ``rte_eth_dev_mac_addr_addr()``.
+
Prerequisites
-------------
@@ -185,6 +218,26 @@ DPDK and must be installed separately:
- mlx4_ib: InifiniBand device driver.
- ib_uverbs: user space driver for verbs (entry point for libibverbs).
+- **Firmware update**
+
+ Mellanox OFED releases include firmware updates for ConnectX-3 adapters.
+
+ Because each release provides new features, these updates must be applied to
+ match the kernel modules and libraries they come with.
+
+.. note::
+
+ Both libraries are BSD and GPL licensed. Linux kernel modules are GPL
+ licensed.
+
+Currently supported by DPDK:
+
+- Mellanox OFED **2.4-1**.
+- Firmware version **2.33.5000** and higher.
+
+Getting Mellanox OFED
+~~~~~~~~~~~~~~~~~~~~~
+
While these libraries and kernel modules are available on OpenFabrics
Aliance's `website <https://www.openfabrics.org/>`_ and provided by package
managers on most distributions, this PMD requires Ethernet extensions that
@@ -193,13 +246,31 @@ may not be supported at the moment (this is a work in progress).
`Mellanox OFED
<http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers>`_
includes the necessary support and should be used in the meantime. For DPDK,
-only libibverbs, libmlx4 and mlnx-ofed-kernel packages are required from
-that distribution.
+only libibverbs, libmlx4, mlnx-ofed-kernel packages and firmware updates are
+required from that distribution.
.. note::
- Both libraries are BSD and GPL licensed. Linux kernel modules are GPL
- licensed.
+ Several versions of Mellanox OFED are available. Installing the version
+ this DPDK release was developped and tested against is strongly
+ recommended. Please check the `prerequisites`_.
+
+Getting libibverbs and libmlx4 from DPDK.org
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Based on Mellanox OFED, optimized libibverbs and libmlx4 versions can be
+optionally downloaded from DPDK.org:
+
+`<http://www.dpdk.org/download/mlx4>`_
+
+Some enhancements are done for better performance with DPDK applications and
+are not merged upstream yet.
+
+Since it is partly achieved by tuning compilation options to disable features
+not needed by DPDK, linking these libraries statically and avoid system-wide
+installation is the preferred method.
+
+Installation documentation is available from the above link.
Usage example
-------------
--
2.1.0
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
2015-04-02 9:33 [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies Adrien Mazarguil
@ 2015-04-03 13:53 ` Thomas Monjalon
[not found] ` <CAC2O_6oqG=3H0R5L6+ZX=1FzNj=mprDzOcUo_e6wezJxzu=ybw@mail.gmail.com>
1 sibling, 0 replies; 10+ messages in thread
From: Thomas Monjalon @ 2015-04-03 13:53 UTC (permalink / raw)
To: Adrien Mazarguil; +Cc: dev
2015-04-02 11:33, Adrien Mazarguil:
> - libmlx4 and libibverbs dependencies distributed with Mellanox OFED are now
> also available on DPDK.org to make installation easier.
> - Document Mellanox OFED and firmware versions to use.
> - Add links to Mellanox and its community websites.
> - Add kernel modules parameters section.
>
> Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
Applied, thanks
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
[not found] ` <20150403073423.GY32147@6wind.com>
@ 2015-04-03 16:52 ` Raghav Sethi
2015-04-03 18:21 ` Raghav Sethi
0 siblings, 1 reply; 10+ messages in thread
From: Raghav Sethi @ 2015-04-03 16:52 UTC (permalink / raw)
To: Adrien Mazarguil, dev
Hi Adrien,
I've copied dev@dpdk.
I re-installed Mellanox OFED, and then re-compiled DPDK without the flags
for the static library. I'm running Ubuntu 14.04. However, I still get the
same error message (below).
I then tried re-installing the Mellanox libraries on dpdk.org, and compiled
another copy of DPDK with them statically linked. This appears to work
perfectly. I no longer get the permission issues I was getting before,
possibly that was unrelated/fixed by OFED. Now both OFED, and OFED+dpdk.org
libs statically linked have the same issue:
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL: probe driver: 15b3:1003 librte_pmd_mlx4
libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
/usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, version
IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference
libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
/usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow,
version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
reference
libibverbs: Warning: no userspace device-specific driver found for
/sys/class/infiniband_verbs/uverbs0
PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
EAL: No probed ethernet devices
The OFED install was successful (and it apparently removes/overwrites old
versions of the libraries). The installer also checks the firmware
versions, and it says I'm on the latest. Loading the kernel modules is also
successful, so I can't for the life of me figure out what this issue is now.
Again, just to be clear, the full list of steps was:
1. Install OFED, run /etc/init.d/openibd restart
2. Compile DPDK with the CONFIG_RTE_LIBRTE_MLX4_PMD=y
3. Set up hugepages
4. Load kernel modules
5. Run testpmd
Best,
Raghav
On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <adrien.mazarguil@6wind.com>
wrote:
> Hi Raghav,
>
> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
> > Hi Adrien,
> >
> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
> I'm
> > a grad student at Princeton working on a research project.
>
> It's fine, but let's discuss on the ML as much as possible next time so the
> community can benefit from it. Since very few people use this driver at the
> moment (DPDK 2.0 is not even out yet), I have my own questions for you.
>
> > Given your patch (which was very helpful, thanks), here is the series of
> > all steps that I'm taking to run testpmd on a ConnectX-3 EN 40G
> >
> > 1. Compile and install the mellanox-specific libraries listed here:
> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
>
> This is fine, but did you use the provided Makefile to compile these
> libraries statically? If so, no issues?
>
> Also you did not install Mellanox OFED? You should, because updated kernel
> modules come from that distribution. It won't work properly (or at all)
> without them.
>
> The libraries from DPDK.org are actually optional once you've installed
> Mellanox OFED. They only improve performance. Remember to update firmwares
> also.
>
> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS
> > 3. Create a new configuration which includes
> > the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
> > 4. Compile DPDK 2.0 with the new configuration using setup.sh
> > 5. Set up hugepages using setup.sh
> > 6. Load the kernel modules ib_uverbs mlx4_en mlx4_core mlx4_ib
> > 7. Run testpmd with the right PCI bus address
> >
> > However, I consistently get:
> > sudo: unable to execute ./testpmd: Permission denied
>
> Now that doesn't make sense, isn't that file missing permissions or
> something? Are you running RHEL with a restrictive SELinux policy?
> Is sudo configured to allow you to run testpmd?
>
> In any case, I don't think it's related to librte_pmd_mlx4.
>
> > I tried taking the interface down with ifconfig because
> > (./tools/dpdk_nic_bind.py --status shows the interface as active), but
> that
> > didn't change anything.
>
> No, you should leave the interfaces up, it's not supposed to disappear, the
> PMD will bring them back up anyway. Actually the Linux kernel interface
> remains fully functional even while DPDK applications are running. You can
> send packets out of it but as long as both the kernel and DPDK applications
> share the same MAC address, only DPDK applications will receive replies. If
> you change the MAC address on the kernel interface however, you'll be able
> to use it normally.
>
> > I tried installing the mellanox libraries globally, but then I see these
> > error messages:
> >
> > -------
> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4
> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> version
> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference
> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow,
> > version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> > reference
> > libibverbs: Warning: no userspace device-specific driver found for
> > /sys/class/infiniband_verbs/uverbs0
> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> > EAL: No probed ethernet devices
> > Interactive-mode selected
> > EAL: Error - exiting with code: 1
> > Cause: Creation of mbuf pool for socket 0 failed
> > --------
>
> That's because the libibverbs version you're using is not compatible with
> the userspace drivers installed on your system (libibverbs uses dlopen() to
> load liblmx4). Even though you've installed both from DPDK.org, seems you
> have another version installed by your distribution packages.
>
> If you intend to use dynamic libraries, you should uninstall libibverbs,
> libmlx4 and related packages first.
>
> Regards,
>
> --
> Adrien Mazarguil
> 6WIND
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
2015-04-03 16:52 ` Raghav Sethi
@ 2015-04-03 18:21 ` Raghav Sethi
2015-04-04 20:49 ` Olga Shern
0 siblings, 1 reply; 10+ messages in thread
From: Raghav Sethi @ 2015-04-03 18:21 UTC (permalink / raw)
To: Adrien Mazarguil, dev, Xiaozhou Li
Hi Adrien,
Quick update: So I just tried the setup again on a new machine with the
exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a
charm.
Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
>From the error message it appears that the libraries are incorrectly
installed/configured. However, the OFED uninstall script removes the
libraries in question
(libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so, libibverbs.so), I
checked! In fact, I diffed the libibverbs.so from both machines, and
they're exactly identical. So it seems either the OFED
installation/uninstallation is somehow flawed, or the error message from
testpmd is misleading.
Best,
Raghav
On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs@cs.princeton.edu>
wrote:
> Hi Adrien,
>
> I've copied dev@dpdk.
>
> I re-installed Mellanox OFED, and then re-compiled DPDK without the flags
> for the static library. I'm running Ubuntu 14.04. However, I still get the
> same error message (below).
>
> I then tried re-installing the Mellanox libraries on dpdk.org, and
> compiled another copy of DPDK with them statically linked. This appears to
> work perfectly. I no longer get the permission issues I was getting before,
> possibly that was unrelated/fixed by OFED. Now both OFED, and OFED+
> dpdk.org libs statically linked have the same issue:
>
> EAL: PCI device 0000:06:00.0 on NUMA socket 0
> EAL: probe driver: 15b3:1003 librte_pmd_mlx4
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, version
> IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow,
> version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> reference
> libibverbs: Warning: no userspace device-specific driver found for
> /sys/class/infiniband_verbs/uverbs0
> PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> EAL: No probed ethernet devices
>
> The OFED install was successful (and it apparently removes/overwrites old
> versions of the libraries). The installer also checks the firmware
> versions, and it says I'm on the latest. Loading the kernel modules is also
> successful, so I can't for the life of me figure out what this issue is now.
>
> Again, just to be clear, the full list of steps was:
> 1. Install OFED, run /etc/init.d/openibd restart
> 2. Compile DPDK with the CONFIG_RTE_LIBRTE_MLX4_PMD=y
> 3. Set up hugepages
> 4. Load kernel modules
> 5. Run testpmd
>
> Best,
> Raghav
>
> On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <
> adrien.mazarguil@6wind.com> wrote:
>
>> Hi Raghav,
>>
>> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
>> > Hi Adrien,
>> >
>> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
>> I'm
>> > a grad student at Princeton working on a research project.
>>
>> It's fine, but let's discuss on the ML as much as possible next time so
>> the
>> community can benefit from it. Since very few people use this driver at
>> the
>> moment (DPDK 2.0 is not even out yet), I have my own questions for you.
>>
>> > Given your patch (which was very helpful, thanks), here is the series of
>> > all steps that I'm taking to run testpmd on a ConnectX-3 EN 40G
>> >
>> > 1. Compile and install the mellanox-specific libraries listed here:
>> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
>>
>> This is fine, but did you use the provided Makefile to compile these
>> libraries statically? If so, no issues?
>>
>> Also you did not install Mellanox OFED? You should, because updated kernel
>> modules come from that distribution. It won't work properly (or at all)
>> without them.
>>
>> The libraries from DPDK.org are actually optional once you've installed
>> Mellanox OFED. They only improve performance. Remember to update firmwares
>> also.
>>
>> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS
>> > 3. Create a new configuration which includes
>> > the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
>> > 4. Compile DPDK 2.0 with the new configuration using setup.sh
>> > 5. Set up hugepages using setup.sh
>> > 6. Load the kernel modules ib_uverbs mlx4_en mlx4_core mlx4_ib
>> > 7. Run testpmd with the right PCI bus address
>> >
>> > However, I consistently get:
>> > sudo: unable to execute ./testpmd: Permission denied
>>
>> Now that doesn't make sense, isn't that file missing permissions or
>> something? Are you running RHEL with a restrictive SELinux policy?
>> Is sudo configured to allow you to run testpmd?
>>
>> In any case, I don't think it's related to librte_pmd_mlx4.
>>
>> > I tried taking the interface down with ifconfig because
>> > (./tools/dpdk_nic_bind.py --status shows the interface as active), but
>> that
>> > didn't change anything.
>>
>> No, you should leave the interfaces up, it's not supposed to disappear,
>> the
>> PMD will bring them back up anyway. Actually the Linux kernel interface
>> remains fully functional even while DPDK applications are running. You can
>> send packets out of it but as long as both the kernel and DPDK
>> applications
>> share the same MAC address, only DPDK applications will receive replies.
>> If
>> you change the MAC address on the kernel interface however, you'll be able
>> to use it normally.
>>
>> > I tried installing the mellanox libraries globally, but then I see these
>> > error messages:
>> >
>> > -------
>> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
>> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
>> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
>> version
>> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
>> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow,
>> > version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
>> > reference
>> > libibverbs: Warning: no userspace device-specific driver found for
>> > /sys/class/infiniband_verbs/uverbs0
>> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
>> > EAL: No probed ethernet devices
>> > Interactive-mode selected
>> > EAL: Error - exiting with code: 1
>> > Cause: Creation of mbuf pool for socket 0 failed
>> > --------
>>
>> That's because the libibverbs version you're using is not compatible with
>> the userspace drivers installed on your system (libibverbs uses dlopen()
>> to
>> load liblmx4). Even though you've installed both from DPDK.org, seems you
>> have another version installed by your distribution packages.
>>
>> If you intend to use dynamic libraries, you should uninstall libibverbs,
>> libmlx4 and related packages first.
>>
>> Regards,
>>
>> --
>> Adrien Mazarguil
>> 6WIND
>>
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
2015-04-03 18:21 ` Raghav Sethi
@ 2015-04-04 20:49 ` Olga Shern
2015-04-04 20:53 ` Raghav Sethi
0 siblings, 1 reply; 10+ messages in thread
From: Olga Shern @ 2015-04-04 20:49 UTC (permalink / raw)
To: Raghav Sethi, Adrien Mazarguil, dev, Xiaozhou Li
Hi Raghav,
What OFED version did you install?
There are 2 ways to compile mlx4:
1. You can compile it dynamically with the libibverbs that are coming with Mellanox_OFED 2.4
2. To get better performance: compile it statically with libibverbs and libmlx4 that can be downloaded from http://dpdk.org/download, the README inside librte_pmd_mlx4 explains how to do it
If you compile the pmd statically then MLNX_OFED is needed only for kernel modules.
Best Regards,
Olga
-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Raghav Sethi
Sent: Friday, April 03, 2015 9:21 PM
To: Adrien Mazarguil; dev@dpdk.org; Xiaozhou Li
Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
Hi Adrien,
Quick update: So I just tried the setup again on a new machine with the exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a charm.
Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
From the error message it appears that the libraries are incorrectly installed/configured. However, the OFED uninstall script removes the libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so, libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both machines, and they're exactly identical. So it seems either the OFED installation/uninstallation is somehow flawed, or the error message from testpmd is misleading.
Best,
Raghav
On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs@cs.princeton.edu>
wrote:
> Hi Adrien,
>
> I've copied dev@dpdk.
>
> I re-installed Mellanox OFED, and then re-compiled DPDK without the
> flags for the static library. I'm running Ubuntu 14.04. However, I
> still get the same error message (below).
>
> I then tried re-installing the Mellanox libraries on dpdk.org, and
> compiled another copy of DPDK with them statically linked. This
> appears to work perfectly. I no longer get the permission issues I was
> getting before, possibly that was unrelated/fixed by OFED. Now both
> OFED, and OFED+ dpdk.org libs statically linked have the same issue:
>
> EAL: PCI device 0000:06:00.0 on NUMA socket 0
> EAL: probe driver: 15b3:1003 librte_pmd_mlx4
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> version
> IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> reference
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
> ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
> libibverbs.so.1 with link time reference
> libibverbs: Warning: no userspace device-specific driver found for
> /sys/class/infiniband_verbs/uverbs0
> PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> EAL: No probed ethernet devices
>
> The OFED install was successful (and it apparently removes/overwrites
> old versions of the libraries). The installer also checks the firmware
> versions, and it says I'm on the latest. Loading the kernel modules is
> also successful, so I can't for the life of me figure out what this issue is now.
>
> Again, just to be clear, the full list of steps was:
> 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with
> the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel
> modules 5. Run testpmd
>
> Best,
> Raghav
>
> On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <
> adrien.mazarguil@6wind.com> wrote:
>
>> Hi Raghav,
>>
>> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
>> > Hi Adrien,
>> >
>> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
>> I'm
>> > a grad student at Princeton working on a research project.
>>
>> It's fine, but let's discuss on the ML as much as possible next time
>> so the community can benefit from it. Since very few people use this
>> driver at the moment (DPDK 2.0 is not even out yet), I have my own
>> questions for you.
>>
>> > Given your patch (which was very helpful, thanks), here is the
>> > series of all steps that I'm taking to run testpmd on a ConnectX-3
>> > EN 40G
>> >
>> > 1. Compile and install the mellanox-specific libraries listed here:
>> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
>>
>> This is fine, but did you use the provided Makefile to compile these
>> libraries statically? If so, no issues?
>>
>> Also you did not install Mellanox OFED? You should, because updated
>> kernel modules come from that distribution. It won't work properly
>> (or at all) without them.
>>
>> The libraries from DPDK.org are actually optional once you've
>> installed Mellanox OFED. They only improve performance. Remember to
>> update firmwares also.
>>
>> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new
>> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
>> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5.
>> > Set up hugepages using setup.sh 6. Load the kernel modules
>> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right
>> > PCI bus address
>> >
>> > However, I consistently get:
>> > sudo: unable to execute ./testpmd: Permission denied
>>
>> Now that doesn't make sense, isn't that file missing permissions or
>> something? Are you running RHEL with a restrictive SELinux policy?
>> Is sudo configured to allow you to run testpmd?
>>
>> In any case, I don't think it's related to librte_pmd_mlx4.
>>
>> > I tried taking the interface down with ifconfig because
>> > (./tools/dpdk_nic_bind.py --status shows the interface as active),
>> > but
>> that
>> > didn't change anything.
>>
>> No, you should leave the interfaces up, it's not supposed to
>> disappear, the PMD will bring them back up anyway. Actually the Linux
>> kernel interface remains fully functional even while DPDK
>> applications are running. You can send packets out of it but as long
>> as both the kernel and DPDK applications share the same MAC address,
>> only DPDK applications will receive replies.
>> If
>> you change the MAC address on the kernel interface however, you'll be
>> able to use it normally.
>>
>> > I tried installing the mellanox libraries globally, but then I see
>> > these error messages:
>> >
>> > -------
>> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
>> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
>> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
>> version
>> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
>> > reference
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
>> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
>> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
>> > libibverbs.so.1 with link time reference
>> > libibverbs: Warning: no userspace device-specific driver found for
>> > /sys/class/infiniband_verbs/uverbs0
>> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
>> > EAL: No probed ethernet devices
>> > Interactive-mode selected
>> > EAL: Error - exiting with code: 1
>> > Cause: Creation of mbuf pool for socket 0 failed
>> > --------
>>
>> That's because the libibverbs version you're using is not compatible
>> with the userspace drivers installed on your system (libibverbs uses
>> dlopen() to load liblmx4). Even though you've installed both from
>> DPDK.org, seems you have another version installed by your
>> distribution packages.
>>
>> If you intend to use dynamic libraries, you should uninstall
>> libibverbs,
>> libmlx4 and related packages first.
>>
>> Regards,
>>
>> --
>> Adrien Mazarguil
>> 6WIND
>>
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
2015-04-04 20:49 ` Olga Shern
@ 2015-04-04 20:53 ` Raghav Sethi
2015-04-04 21:04 ` Olga Shern
0 siblings, 1 reply; 10+ messages in thread
From: Raghav Sethi @ 2015-04-04 20:53 UTC (permalink / raw)
To: Olga Shern, Adrien Mazarguil, dev, Xiaozhou Li
Hi Olga,
Thanks for your reply. OFED version was 2.4-1.0.4 for ubuntu14.04-x86_64
I tried both variants:
- Compiling with dynamic linking against OFED libraries
- Compiling with static linking against dpdk.org libraries
Both have the same issue.
Best,
Raghav
On Sat, Apr 4, 2015 at 4:49 PM Olga Shern <olgas@mellanox.com> wrote:
> Hi Raghav,
>
> What OFED version did you install?
>
> There are 2 ways to compile mlx4:
> 1. You can compile it dynamically with the libibverbs that are coming
> with Mellanox_OFED 2.4
> 2. To get better performance: compile it statically with libibverbs and
> libmlx4 that can be downloaded from http://dpdk.org/download, the README
> inside librte_pmd_mlx4 explains how to do it
> If you compile the pmd statically then MLNX_OFED is needed only for
> kernel modules.
>
> Best Regards,
> Olga
>
>
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Raghav Sethi
> Sent: Friday, April 03, 2015 9:21 PM
> To: Adrien Mazarguil; dev@dpdk.org; Xiaozhou Li
> Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
>
> Hi Adrien,
>
> Quick update: So I just tried the setup again on a new machine with the
> exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a
> charm.
>
> Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
> From the error message it appears that the libraries are incorrectly
> installed/configured. However, the OFED uninstall script removes the
> libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so,
> libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both
> machines, and they're exactly identical. So it seems either the OFED
> installation/uninstallation is somehow flawed, or the error message from
> testpmd is misleading.
>
> Best,
> Raghav
>
> On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs@cs.princeton.edu>
> wrote:
>
> > Hi Adrien,
> >
> > I've copied dev@dpdk.
> >
> > I re-installed Mellanox OFED, and then re-compiled DPDK without the
> > flags for the static library. I'm running Ubuntu 14.04. However, I
> > still get the same error message (below).
> >
> > I then tried re-installing the Mellanox libraries on dpdk.org, and
> > compiled another copy of DPDK with them statically linked. This
> > appears to work perfectly. I no longer get the permission issues I was
> > getting before, possibly that was unrelated/fixed by OFED. Now both
> > OFED, and OFED+ dpdk.org libs statically linked have the same issue:
> >
> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4
> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> > version
> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> > reference
> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
> > libibverbs.so.1 with link time reference
> > libibverbs: Warning: no userspace device-specific driver found for
> > /sys/class/infiniband_verbs/uverbs0
> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> > EAL: No probed ethernet devices
> >
> > The OFED install was successful (and it apparently removes/overwrites
> > old versions of the libraries). The installer also checks the firmware
> > versions, and it says I'm on the latest. Loading the kernel modules is
> > also successful, so I can't for the life of me figure out what this
> issue is now.
> >
> > Again, just to be clear, the full list of steps was:
> > 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with
> > the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel
> > modules 5. Run testpmd
> >
> > Best,
> > Raghav
> >
> > On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <
> > adrien.mazarguil@6wind.com> wrote:
> >
> >> Hi Raghav,
> >>
> >> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
> >> > Hi Adrien,
> >> >
> >> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
> >> I'm
> >> > a grad student at Princeton working on a research project.
> >>
> >> It's fine, but let's discuss on the ML as much as possible next time
> >> so the community can benefit from it. Since very few people use this
> >> driver at the moment (DPDK 2.0 is not even out yet), I have my own
> >> questions for you.
> >>
> >> > Given your patch (which was very helpful, thanks), here is the
> >> > series of all steps that I'm taking to run testpmd on a ConnectX-3
> >> > EN 40G
> >> >
> >> > 1. Compile and install the mellanox-specific libraries listed here:
> >> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
> >>
> >> This is fine, but did you use the provided Makefile to compile these
> >> libraries statically? If so, no issues?
> >>
> >> Also you did not install Mellanox OFED? You should, because updated
> >> kernel modules come from that distribution. It won't work properly
> >> (or at all) without them.
> >>
> >> The libraries from DPDK.org are actually optional once you've
> >> installed Mellanox OFED. They only improve performance. Remember to
> >> update firmwares also.
> >>
> >> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new
> >> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
> >> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5.
> >> > Set up hugepages using setup.sh 6. Load the kernel modules
> >> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right
> >> > PCI bus address
> >> >
> >> > However, I consistently get:
> >> > sudo: unable to execute ./testpmd: Permission denied
> >>
> >> Now that doesn't make sense, isn't that file missing permissions or
> >> something? Are you running RHEL with a restrictive SELinux policy?
> >> Is sudo configured to allow you to run testpmd?
> >>
> >> In any case, I don't think it's related to librte_pmd_mlx4.
> >>
> >> > I tried taking the interface down with ifconfig because
> >> > (./tools/dpdk_nic_bind.py --status shows the interface as active),
> >> > but
> >> that
> >> > didn't change anything.
> >>
> >> No, you should leave the interfaces up, it's not supposed to
> >> disappear, the PMD will bring them back up anyway. Actually the Linux
> >> kernel interface remains fully functional even while DPDK
> >> applications are running. You can send packets out of it but as long
> >> as both the kernel and DPDK applications share the same MAC address,
> >> only DPDK applications will receive replies.
> >> If
> >> you change the MAC address on the kernel interface however, you'll be
> >> able to use it normally.
> >>
> >> > I tried installing the mellanox libraries globally, but then I see
> >> > these error messages:
> >> >
> >> > -------
> >> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
> >> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4
> >> > libibverbs: Warning: couldn't load driver
> '/usr/lib/libibverbs/libmlx5':
> >> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> >> version
> >> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> >> > reference
> >> > libibverbs: Warning: couldn't load driver
> '/usr/lib/libibverbs/libmlx4':
> >> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
> >> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
> >> > libibverbs.so.1 with link time reference
> >> > libibverbs: Warning: no userspace device-specific driver found for
> >> > /sys/class/infiniband_verbs/uverbs0
> >> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> >> > EAL: No probed ethernet devices
> >> > Interactive-mode selected
> >> > EAL: Error - exiting with code: 1
> >> > Cause: Creation of mbuf pool for socket 0 failed
> >> > --------
> >>
> >> That's because the libibverbs version you're using is not compatible
> >> with the userspace drivers installed on your system (libibverbs uses
> >> dlopen() to load liblmx4). Even though you've installed both from
> >> DPDK.org, seems you have another version installed by your
> >> distribution packages.
> >>
> >> If you intend to use dynamic libraries, you should uninstall
> >> libibverbs,
> >> libmlx4 and related packages first.
> >>
> >> Regards,
> >>
> >> --
> >> Adrien Mazarguil
> >> 6WIND
> >>
> >
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
2015-04-04 20:53 ` Raghav Sethi
@ 2015-04-04 21:04 ` Olga Shern
2015-04-04 21:29 ` Raghav Sethi
0 siblings, 1 reply; 10+ messages in thread
From: Olga Shern @ 2015-04-04 21:04 UTC (permalink / raw)
To: Raghav Sethi, Adrien Mazarguil, dev, Xiaozhou Li
Hi Raghav,
After running /etc/init.d/openibd restart you don’t need to load the modules, they are loaded by the openibd script.
Can you try to run :
/etc/init.d/openibd stop
Verify that modules are not loaded
/etc/init.d/openib start
If it won’t help:
What Mellanox NIC do you have on the “problematic” machine?
What command line do you use to run the application?
Please send output of:
1. ”lspci | grep Mellanox”
2. ibdev2netdev
3. ldd <your application>
4. lsmod
Thanks
Olga
From: Raghav Sethi [mailto:raghavs@CS.Princeton.EDU]
Sent: Saturday, April 04, 2015 11:54 PM
To: Olga Shern; Adrien Mazarguil; dev@dpdk.org; Xiaozhou Li
Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
Hi Olga,
Thanks for your reply. OFED version was 2.4-1.0.4 for ubuntu14.04-x86_64
I tried both variants:
- Compiling with dynamic linking against OFED libraries
- Compiling with static linking against dpdk.org<http://dpdk.org> libraries
Both have the same issue.
Best,
Raghav
On Sat, Apr 4, 2015 at 4:49 PM Olga Shern <olgas@mellanox.com<mailto:olgas@mellanox.com>> wrote:
Hi Raghav,
What OFED version did you install?
There are 2 ways to compile mlx4:
1. You can compile it dynamically with the libibverbs that are coming with Mellanox_OFED 2.4
2. To get better performance: compile it statically with libibverbs and libmlx4 that can be downloaded from http://dpdk.org/download, the README inside librte_pmd_mlx4 explains how to do it
If you compile the pmd statically then MLNX_OFED is needed only for kernel modules.
Best Regards,
Olga
-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>] On Behalf Of Raghav Sethi
Sent: Friday, April 03, 2015 9:21 PM
To: Adrien Mazarguil; dev@dpdk.org<mailto:dev@dpdk.org>; Xiaozhou Li
Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
Hi Adrien,
Quick update: So I just tried the setup again on a new machine with the exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a charm.
Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
From the error message it appears that the libraries are incorrectly installed/configured. However, the OFED uninstall script removes the libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so, libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both machines, and they're exactly identical. So it seems either the OFED installation/uninstallation is somehow flawed, or the error message from testpmd is misleading.
Best,
Raghav
On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs@cs.princeton.edu<mailto:raghavs@cs.princeton.edu>>
wrote:
> Hi Adrien,
>
> I've copied dev@dpdk.
>
> I re-installed Mellanox OFED, and then re-compiled DPDK without the
> flags for the static library. I'm running Ubuntu 14.04. However, I
> still get the same error message (below).
>
> I then tried re-installing the Mellanox libraries on dpdk.org<http://dpdk.org>, and
> compiled another copy of DPDK with them statically linked. This
> appears to work perfectly. I no longer get the permission issues I was
> getting before, possibly that was unrelated/fixed by OFED. Now both
> OFED, and OFED+ dpdk.org<http://dpdk.org> libs statically linked have the same issue:
>
> EAL: PCI device 0000:06:00.0 on NUMA socket 0
> EAL: probe driver: 15b3:1003 librte_pmd_mlx4
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> version
> IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> reference
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
> ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
> libibverbs.so.1 with link time reference
> libibverbs: Warning: no userspace device-specific driver found for
> /sys/class/infiniband_verbs/uverbs0
> PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> EAL: No probed ethernet devices
>
> The OFED install was successful (and it apparently removes/overwrites
> old versions of the libraries). The installer also checks the firmware
> versions, and it says I'm on the latest. Loading the kernel modules is
> also successful, so I can't for the life of me figure out what this issue is now.
>
> Again, just to be clear, the full list of steps was:
> 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with
> the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel
> modules 5. Run testpmd
>
> Best,
> Raghav
>
> On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <
> adrien.mazarguil@6wind.com<mailto:adrien.mazarguil@6wind.com>> wrote:
>
>> Hi Raghav,
>>
>> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
>> > Hi Adrien,
>> >
>> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
>> I'm
>> > a grad student at Princeton working on a research project.
>>
>> It's fine, but let's discuss on the ML as much as possible next time
>> so the community can benefit from it. Since very few people use this
>> driver at the moment (DPDK 2.0 is not even out yet), I have my own
>> questions for you.
>>
>> > Given your patch (which was very helpful, thanks), here is the
>> > series of all steps that I'm taking to run testpmd on a ConnectX-3
>> > EN 40G
>> >
>> > 1. Compile and install the mellanox-specific libraries listed here:
>> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
>>
>> This is fine, but did you use the provided Makefile to compile these
>> libraries statically? If so, no issues?
>>
>> Also you did not install Mellanox OFED? You should, because updated
>> kernel modules come from that distribution. It won't work properly
>> (or at all) without them.
>>
>> The libraries from DPDK.org are actually optional once you've
>> installed Mellanox OFED. They only improve performance. Remember to
>> update firmwares also.
>>
>> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new
>> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
>> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5.
>> > Set up hugepages using setup.sh 6. Load the kernel modules
>> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right
>> > PCI bus address
>> >
>> > However, I consistently get:
>> > sudo: unable to execute ./testpmd: Permission denied
>>
>> Now that doesn't make sense, isn't that file missing permissions or
>> something? Are you running RHEL with a restrictive SELinux policy?
>> Is sudo configured to allow you to run testpmd?
>>
>> In any case, I don't think it's related to librte_pmd_mlx4.
>>
>> > I tried taking the interface down with ifconfig because
>> > (./tools/dpdk_nic_bind.py --status shows the interface as active),
>> > but
>> that
>> > didn't change anything.
>>
>> No, you should leave the interfaces up, it's not supposed to
>> disappear, the PMD will bring them back up anyway. Actually the Linux
>> kernel interface remains fully functional even while DPDK
>> applications are running. You can send packets out of it but as long
>> as both the kernel and DPDK applications share the same MAC address,
>> only DPDK applications will receive replies.
>> If
>> you change the MAC address on the kernel interface however, you'll be
>> able to use it normally.
>>
>> > I tried installing the mellanox libraries globally, but then I see
>> > these error messages:
>> >
>> > -------
>> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
>> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
>> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
>> version
>> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
>> > reference
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
>> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
>> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
>> > libibverbs.so.1 with link time reference
>> > libibverbs: Warning: no userspace device-specific driver found for
>> > /sys/class/infiniband_verbs/uverbs0
>> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
>> > EAL: No probed ethernet devices
>> > Interactive-mode selected
>> > EAL: Error - exiting with code: 1
>> > Cause: Creation of mbuf pool for socket 0 failed
>> > --------
>>
>> That's because the libibverbs version you're using is not compatible
>> with the userspace drivers installed on your system (libibverbs uses
>> dlopen() to load liblmx4). Even though you've installed both from
>> DPDK.org, seems you have another version installed by your
>> distribution packages.
>>
>> If you intend to use dynamic libraries, you should uninstall
>> libibverbs,
>> libmlx4 and related packages first.
>>
>> Regards,
>>
>> --
>> Adrien Mazarguil
>> 6WIND
>>
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
2015-04-04 21:04 ` Olga Shern
@ 2015-04-04 21:29 ` Raghav Sethi
2015-04-04 22:04 ` Olga Shern
0 siblings, 1 reply; 10+ messages in thread
From: Raghav Sethi @ 2015-04-04 21:29 UTC (permalink / raw)
To: Olga Shern, Adrien Mazarguil, dev, Xiaozhou Li
Hi Olga,
Thanks very much for helping.
I tried stopping and starting, after stop lsmod | grep mlx had no matches.
After start, it had the output listed below. The device is the same as on
the rest of the machines - Mellanox ConnectX3-EN 40G single port. The
application parameters are the same as the ones listed on the docs (except
of course PCI address is different):
sudo ./testpmd -c 0xff -n 4 -w 0000:06:00.0 -- --rxq=2 --txq=2 -i
Outputs you requested:
1. lspci | grep Mellanox
06:00.0 Ethernet controller: Mellanox Technologies MT27500 Family
[ConnectX-3]
2. ibdev2netdev
mlx4_0 port 1 ==> p3p1 (Up)
3. ldd testpmd
linux-vdso.so.1 => (0x00007fff7c1dc000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1a7dc07000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1a7d9ff000)
libibverbs.so.1 => /usr/local/lib/libibverbs.so.1
(0x00007f1a7d7eb000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1a7d5e7000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f1a7d3c9000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1a7d003000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1a7df19000)
4. lsmod | grep mlx
mlx5_ib 138046 0
mlx5_core 108376 1 mlx5_ib
mlx4_ib 199880 0
ib_sa 43732 5 rdma_cm,ib_cm,mlx4_ib,rdma_ucm,ib_ipoib
mlx4_en 137047 0
ib_mad 51544 4 ib_cm,ib_sa,mlx4_ib,ib_umad
vxlan 37619 1 mlx4_en
ib_core 130498 12
rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib
mlx4_core 332375 2 mlx4_en,mlx4_ib
compat 13738 17
rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_en,mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_addr,ib_core,ib_umad,ib_uverbs,mlx4_core,mlx5_core,rdma_ucm,ib_ipoib
ptp 18933 3 igb,ixgbe,mlx4_en
Best regards,
Raghav
On Sat, Apr 4, 2015 at 5:04 PM Olga Shern <olgas@mellanox.com> wrote:
> Hi Raghav,
>
>
>
> After running /etc/init.d/openibd restart you don’t need to load the
> modules, they are loaded by the openibd script.
>
> Can you try to run :
>
> /etc/init.d/openibd stop
>
> Verify that modules are not loaded
>
> /etc/init.d/openib start
>
>
>
> If it won’t help:
>
> What Mellanox NIC do you have on the “problematic” machine?
>
> What command line do you use to run the application?
>
>
>
> Please send output of:
>
> 1. ”lspci | grep Mellanox”
>
> 2. ibdev2netdev
>
> 3. ldd <your application>
>
> 4. lsmod
>
>
>
> Thanks
>
> Olga
>
>
>
>
>
> *From:* Raghav Sethi [mailto:raghavs@CS.Princeton.EDU]
> *Sent:* Saturday, April 04, 2015 11:54 PM
> *To:* Olga Shern; Adrien Mazarguil; dev@dpdk.org; Xiaozhou Li
>
>
> *Subject:* Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
>
>
>
> Hi Olga,
>
> Thanks for your reply. OFED version was 2.4-1.0.4 for ubuntu14.04-x86_64
>
> I tried both variants:
>
> - Compiling with dynamic linking against OFED libraries
>
> - Compiling with static linking against dpdk.org libraries
>
>
>
> Both have the same issue.
>
>
>
> Best,
>
> Raghav
>
>
>
> On Sat, Apr 4, 2015 at 4:49 PM Olga Shern <olgas@mellanox.com> wrote:
>
> Hi Raghav,
>
> What OFED version did you install?
>
> There are 2 ways to compile mlx4:
> 1. You can compile it dynamically with the libibverbs that are coming
> with Mellanox_OFED 2.4
> 2. To get better performance: compile it statically with libibverbs and
> libmlx4 that can be downloaded from http://dpdk.org/download, the README
> inside librte_pmd_mlx4 explains how to do it
> If you compile the pmd statically then MLNX_OFED is needed only for
> kernel modules.
>
> Best Regards,
> Olga
>
>
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Raghav Sethi
> Sent: Friday, April 03, 2015 9:21 PM
> To: Adrien Mazarguil; dev@dpdk.org; Xiaozhou Li
> Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
>
> Hi Adrien,
>
> Quick update: So I just tried the setup again on a new machine with the
> exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a
> charm.
>
> Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
> From the error message it appears that the libraries are incorrectly
> installed/configured. However, the OFED uninstall script removes the
> libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so,
> libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both
> machines, and they're exactly identical. So it seems either the OFED
> installation/uninstallation is somehow flawed, or the error message from
> testpmd is misleading.
>
> Best,
> Raghav
>
> On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs@cs.princeton.edu>
> wrote:
>
> > Hi Adrien,
> >
> > I've copied dev@dpdk.
> >
> > I re-installed Mellanox OFED, and then re-compiled DPDK without the
> > flags for the static library. I'm running Ubuntu 14.04. However, I
> > still get the same error message (below).
> >
> > I then tried re-installing the Mellanox libraries on dpdk.org, and
> > compiled another copy of DPDK with them statically linked. This
> > appears to work perfectly. I no longer get the permission issues I was
> > getting before, possibly that was unrelated/fixed by OFED. Now both
> > OFED, and OFED+ dpdk.org libs statically linked have the same issue:
> >
> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4
> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> > version
> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> > reference
> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
> > libibverbs.so.1 with link time reference
> > libibverbs: Warning: no userspace device-specific driver found for
> > /sys/class/infiniband_verbs/uverbs0
> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> > EAL: No probed ethernet devices
> >
> > The OFED install was successful (and it apparently removes/overwrites
> > old versions of the libraries). The installer also checks the firmware
> > versions, and it says I'm on the latest. Loading the kernel modules is
> > also successful, so I can't for the life of me figure out what this
> issue is now.
> >
> > Again, just to be clear, the full list of steps was:
> > 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with
> > the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel
> > modules 5. Run testpmd
> >
> > Best,
> > Raghav
> >
> > On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <
> > adrien.mazarguil@6wind.com> wrote:
> >
> >> Hi Raghav,
> >>
> >> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
> >> > Hi Adrien,
> >> >
> >> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
> >> I'm
> >> > a grad student at Princeton working on a research project.
> >>
> >> It's fine, but let's discuss on the ML as much as possible next time
> >> so the community can benefit from it. Since very few people use this
> >> driver at the moment (DPDK 2.0 is not even out yet), I have my own
> >> questions for you.
> >>
> >> > Given your patch (which was very helpful, thanks), here is the
> >> > series of all steps that I'm taking to run testpmd on a ConnectX-3
> >> > EN 40G
> >> >
> >> > 1. Compile and install the mellanox-specific libraries listed here:
> >> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
> >>
> >> This is fine, but did you use the provided Makefile to compile these
> >> libraries statically? If so, no issues?
> >>
> >> Also you did not install Mellanox OFED? You should, because updated
> >> kernel modules come from that distribution. It won't work properly
> >> (or at all) without them.
> >>
> >> The libraries from DPDK.org are actually optional once you've
> >> installed Mellanox OFED. They only improve performance. Remember to
> >> update firmwares also.
> >>
> >> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new
> >> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
> >> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5.
> >> > Set up hugepages using setup.sh 6. Load the kernel modules
> >> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right
> >> > PCI bus address
> >> >
> >> > However, I consistently get:
> >> > sudo: unable to execute ./testpmd: Permission denied
> >>
> >> Now that doesn't make sense, isn't that file missing permissions or
> >> something? Are you running RHEL with a restrictive SELinux policy?
> >> Is sudo configured to allow you to run testpmd?
> >>
> >> In any case, I don't think it's related to librte_pmd_mlx4.
> >>
> >> > I tried taking the interface down with ifconfig because
> >> > (./tools/dpdk_nic_bind.py --status shows the interface as active),
> >> > but
> >> that
> >> > didn't change anything.
> >>
> >> No, you should leave the interfaces up, it's not supposed to
> >> disappear, the PMD will bring them back up anyway. Actually the Linux
> >> kernel interface remains fully functional even while DPDK
> >> applications are running. You can send packets out of it but as long
> >> as both the kernel and DPDK applications share the same MAC address,
> >> only DPDK applications will receive replies.
> >> If
> >> you change the MAC address on the kernel interface however, you'll be
> >> able to use it normally.
> >>
> >> > I tried installing the mellanox libraries globally, but then I see
> >> > these error messages:
> >> >
> >> > -------
> >> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
> >> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4
> >> > libibverbs: Warning: couldn't load driver
> '/usr/lib/libibverbs/libmlx5':
> >> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> >> version
> >> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> >> > reference
> >> > libibverbs: Warning: couldn't load driver
> '/usr/lib/libibverbs/libmlx4':
> >> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
> >> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
> >> > libibverbs.so.1 with link time reference
> >> > libibverbs: Warning: no userspace device-specific driver found for
> >> > /sys/class/infiniband_verbs/uverbs0
> >> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> >> > EAL: No probed ethernet devices
> >> > Interactive-mode selected
> >> > EAL: Error - exiting with code: 1
> >> > Cause: Creation of mbuf pool for socket 0 failed
> >> > --------
> >>
> >> That's because the libibverbs version you're using is not compatible
> >> with the userspace drivers installed on your system (libibverbs uses
> >> dlopen() to load liblmx4). Even though you've installed both from
> >> DPDK.org, seems you have another version installed by your
> >> distribution packages.
> >>
> >> If you intend to use dynamic libraries, you should uninstall
> >> libibverbs,
> >> libmlx4 and related packages first.
> >>
> >> Regards,
> >>
> >> --
> >> Adrien Mazarguil
> >> 6WIND
> >>
> >
>
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
2015-04-04 21:29 ` Raghav Sethi
@ 2015-04-04 22:04 ` Olga Shern
2015-04-04 22:47 ` Raghav Sethi
0 siblings, 1 reply; 10+ messages in thread
From: Olga Shern @ 2015-04-04 22:04 UTC (permalink / raw)
To: Raghav Sethi, Adrien Mazarguil, dev, Xiaozhou Li
If you compiled pmd statically with libibverbs and libmlx4 you shouldn't see it in ldd.
Seems that something went wrong.
In any case, ofed doesn't install libraries under /usr/local
Please remove all mlx related libraries under /usr/local/ like libmlx4 and libibverbs
-------- Original message --------
From: Raghav Sethi
Date:05/04/2015 00:29 (GMT+02:00)
To: Olga Shern ,Adrien Mazarguil ,dev@dpdk.org,Xiaozhou Li
Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
Hi Olga,
Thanks very much for helping.
I tried stopping and starting, after stop lsmod | grep mlx had no matches. After start, it had the output listed below. The device is the same as on the rest of the machines - Mellanox ConnectX3-EN 40G single port. The application parameters are the same as the ones listed on the docs (except of course PCI address is different):
sudo ./testpmd -c 0xff -n 4 -w 0000:06:00.0 -- --rxq=2 --txq=2 -i
Outputs you requested:
1. lspci | grep Mellanox
06:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
2. ibdev2netdev
mlx4_0 port 1 ==> p3p1 (Up)
3. ldd testpmd
linux-vdso.so.1 => (0x00007fff7c1dc000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1a7dc07000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1a7d9ff000)
libibverbs.so.1 => /usr/local/lib/libibverbs.so.1 (0x00007f1a7d7eb000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1a7d5e7000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1a7d3c9000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1a7d003000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1a7df19000)
4. lsmod | grep mlx
mlx5_ib 138046 0
mlx5_core 108376 1 mlx5_ib
mlx4_ib 199880 0
ib_sa 43732 5 rdma_cm,ib_cm,mlx4_ib,rdma_ucm,ib_ipoib
mlx4_en 137047 0
ib_mad 51544 4 ib_cm,ib_sa,mlx4_ib,ib_umad
vxlan 37619 1 mlx4_en
ib_core 130498 12 rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib
mlx4_core 332375 2 mlx4_en,mlx4_ib
compat 13738 17 rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_en,mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_addr,ib_core,ib_umad,ib_uverbs,mlx4_core,mlx5_core,rdma_ucm,ib_ipoib
ptp 18933 3 igb,ixgbe,mlx4_en
Best regards,
Raghav
On Sat, Apr 4, 2015 at 5:04 PM Olga Shern <olgas@mellanox.com<mailto:olgas@mellanox.com>> wrote:
Hi Raghav,
After running /etc/init.d/openibd restart you don’t need to load the modules, they are loaded by the openibd script.
Can you try to run :
/etc/init.d/openibd stop
Verify that modules are not loaded
/etc/init.d/openib start
If it won’t help:
What Mellanox NIC do you have on the “problematic” machine?
What command line do you use to run the application?
Please send output of:
1. ”lspci | grep Mellanox”
2. ibdev2netdev
3. ldd <your application>
4. lsmod
Thanks
Olga
From: Raghav Sethi [mailto:raghavs@CS.Princeton.EDU<mailto:raghavs@CS.Princeton.EDU>]
Sent: Saturday, April 04, 2015 11:54 PM
To: Olga Shern; Adrien Mazarguil; dev@dpdk.org<mailto:dev@dpdk.org>; Xiaozhou Li
Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
Hi Olga,
Thanks for your reply. OFED version was 2.4-1.0.4 for ubuntu14.04-x86_64
I tried both variants:
- Compiling with dynamic linking against OFED libraries
- Compiling with static linking against dpdk.org<http://dpdk.org> libraries
Both have the same issue.
Best,
Raghav
On Sat, Apr 4, 2015 at 4:49 PM Olga Shern <olgas@mellanox.com<mailto:olgas@mellanox.com>> wrote:
Hi Raghav,
What OFED version did you install?
There are 2 ways to compile mlx4:
1. You can compile it dynamically with the libibverbs that are coming with Mellanox_OFED 2.4
2. To get better performance: compile it statically with libibverbs and libmlx4 that can be downloaded from http://dpdk.org/download, the README inside librte_pmd_mlx4 explains how to do it
If you compile the pmd statically then MLNX_OFED is needed only for kernel modules.
Best Regards,
Olga
-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>] On Behalf Of Raghav Sethi
Sent: Friday, April 03, 2015 9:21 PM
To: Adrien Mazarguil; dev@dpdk.org<mailto:dev@dpdk.org>; Xiaozhou Li
Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
Hi Adrien,
Quick update: So I just tried the setup again on a new machine with the exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a charm.
Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
>From the error message it appears that the libraries are incorrectly installed/configured. However, the OFED uninstall script removes the libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so, libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both machines, and they're exactly identical. So it seems either the OFED installation/uninstallation is somehow flawed, or the error message from testpmd is misleading.
Best,
Raghav
On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs@cs.princeton.edu<mailto:raghavs@cs.princeton.edu>>
wrote:
> Hi Adrien,
>
> I've copied dev@dpdk.
>
> I re-installed Mellanox OFED, and then re-compiled DPDK without the
> flags for the static library. I'm running Ubuntu 14.04. However, I
> still get the same error message (below).
>
> I then tried re-installing the Mellanox libraries on dpdk.org<http://dpdk.org>, and
> compiled another copy of DPDK with them statically linked. This
> appears to work perfectly. I no longer get the permission issues I was
> getting before, possibly that was unrelated/fixed by OFED. Now both
> OFED, and OFED+ dpdk.org<http://dpdk.org> libs statically linked have the same issue:
>
> EAL: PCI device 0000:06:00.0 on NUMA socket 0
> EAL: probe driver: 15b3:1003 librte_pmd_mlx4
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> version
> IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> reference
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
> ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
> libibverbs.so.1 with link time reference
> libibverbs: Warning: no userspace device-specific driver found for
> /sys/class/infiniband_verbs/uverbs0
> PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> EAL: No probed ethernet devices
>
> The OFED install was successful (and it apparently removes/overwrites
> old versions of the libraries). The installer also checks the firmware
> versions, and it says I'm on the latest. Loading the kernel modules is
> also successful, so I can't for the life of me figure out what this issue is now.
>
> Again, just to be clear, the full list of steps was:
> 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with
> the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel
> modules 5. Run testpmd
>
> Best,
> Raghav
>
> On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <
> adrien.mazarguil@6wind.com<mailto:adrien.mazarguil@6wind.com>> wrote:
>
>> Hi Raghav,
>>
>> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
>> > Hi Adrien,
>> >
>> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
>> I'm
>> > a grad student at Princeton working on a research project.
>>
>> It's fine, but let's discuss on the ML as much as possible next time
>> so the community can benefit from it. Since very few people use this
>> driver at the moment (DPDK 2.0 is not even out yet), I have my own
>> questions for you.
>>
>> > Given your patch (which was very helpful, thanks), here is the
>> > series of all steps that I'm taking to run testpmd on a ConnectX-3
>> > EN 40G
>> >
>> > 1. Compile and install the mellanox-specific libraries listed here:
>> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
>>
>> This is fine, but did you use the provided Makefile to compile these
>> libraries statically? If so, no issues?
>>
>> Also you did not install Mellanox OFED? You should, because updated
>> kernel modules come from that distribution. It won't work properly
>> (or at all) without them.
>>
>> The libraries from DPDK.org are actually optional once you've
>> installed Mellanox OFED. They only improve performance. Remember to
>> update firmwares also.
>>
>> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new
>> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
>> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5.
>> > Set up hugepages using setup.sh 6. Load the kernel modules
>> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right
>> > PCI bus address
>> >
>> > However, I consistently get:
>> > sudo: unable to execute ./testpmd: Permission denied
>>
>> Now that doesn't make sense, isn't that file missing permissions or
>> something? Are you running RHEL with a restrictive SELinux policy?
>> Is sudo configured to allow you to run testpmd?
>>
>> In any case, I don't think it's related to librte_pmd_mlx4.
>>
>> > I tried taking the interface down with ifconfig because
>> > (./tools/dpdk_nic_bind.py --status shows the interface as active),
>> > but
>> that
>> > didn't change anything.
>>
>> No, you should leave the interfaces up, it's not supposed to
>> disappear, the PMD will bring them back up anyway. Actually the Linux
>> kernel interface remains fully functional even while DPDK
>> applications are running. You can send packets out of it but as long
>> as both the kernel and DPDK applications share the same MAC address,
>> only DPDK applications will receive replies.
>> If
>> you change the MAC address on the kernel interface however, you'll be
>> able to use it normally.
>>
>> > I tried installing the mellanox libraries globally, but then I see
>> > these error messages:
>> >
>> > -------
>> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
>> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
>> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
>> version
>> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
>> > reference
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
>> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
>> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
>> > libibverbs.so.1 with link time reference
>> > libibverbs: Warning: no userspace device-specific driver found for
>> > /sys/class/infiniband_verbs/uverbs0
>> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
>> > EAL: No probed ethernet devices
>> > Interactive-mode selected
>> > EAL: Error - exiting with code: 1
>> > Cause: Creation of mbuf pool for socket 0 failed
>> > --------
>>
>> That's because the libibverbs version you're using is not compatible
>> with the userspace drivers installed on your system (libibverbs uses
>> dlopen() to load liblmx4). Even though you've installed both from
>> DPDK.org, seems you have another version installed by your
>> distribution packages.
>>
>> If you intend to use dynamic libraries, you should uninstall
>> libibverbs,
>> libmlx4 and related packages first.
>>
>> Regards,
>>
>> --
>> Adrien Mazarguil
>> 6WIND
>>
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
2015-04-04 22:04 ` Olga Shern
@ 2015-04-04 22:47 ` Raghav Sethi
0 siblings, 0 replies; 10+ messages in thread
From: Raghav Sethi @ 2015-04-04 22:47 UTC (permalink / raw)
To: Olga Shern, Adrien Mazarguil, dev, Xiaozhou Li
Hi Olga,
Thanks so much! I had sent you the output for the case with dynamic
linking. Removing the libraries from /usr/local and re-installing OFED
solved the library issue. However, there is a new problem (below)
An earlier install in the wrong place must have caused the problem. The
OFED uninstaller does not remove the libraries from /usr/local, only
usr/lib apparently. Also, the installer apparently doesn't install them in
/usr/lib if they are present in /usr/local.
Here is the output for testpmd now:
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL: probe driver: 15b3:1003 librte_pmd_mlx4
PMD: librte_pmd_mlx4: PCI information matches, using device "mlx4_0" (VF:
false)
PMD: librte_pmd_mlx4: 1 port(s) detected
PMD: librte_pmd_mlx4: experimental ibv_exp_query_device
EAL: No probed ethernet devices
Any idea what the issue is in this case?
Best,
Raghav
On Sat, Apr 4, 2015 at 6:04 PM Olga Shern <olgas@mellanox.com> wrote:
> If you compiled pmd statically with libibverbs and libmlx4 you
> shouldn't see it in ldd.
> Seems that something went wrong.
> In any case, ofed doesn't install libraries under /usr/local
> Please remove all mlx related libraries under /usr/local/ like libmlx4 and
> libibverbs
>
>
>
>
>
> -------- Original message --------
> From: Raghav Sethi
> Date:05/04/2015 00:29 (GMT+02:00)
> To: Olga Shern ,Adrien Mazarguil ,dev@dpdk.org,Xiaozhou Li
> Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
>
>
> Hi Olga,
>
> Thanks very much for helping.
>
> I tried stopping and starting, after stop lsmod | grep mlx had no
> matches. After start, it had the output listed below. The device is the
> same as on the rest of the machines - Mellanox ConnectX3-EN 40G single
> port. The application parameters are the same as the ones listed on the
> docs (except of course PCI address is different):
>
> sudo ./testpmd -c 0xff -n 4 -w 0000:06:00.0 -- --rxq=2 --txq=2 -i
>
> Outputs you requested:
>
> 1. lspci | grep Mellanox
> 06:00.0 Ethernet controller: Mellanox Technologies MT27500 Family
> [ConnectX-3]
>
> 2. ibdev2netdev
> mlx4_0 port 1 ==> p3p1 (Up)
>
> 3. ldd testpmd
> linux-vdso.so.1 => (0x00007fff7c1dc000)
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1a7dc07000)
> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
> (0x00007f1a7d9ff000)
> libibverbs.so.1 => /usr/local/lib/libibverbs.so.1
> (0x00007f1a7d7eb000)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
> (0x00007f1a7d5e7000)
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x00007f1a7d3c9000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1a7d003000)
> /lib64/ld-linux-x86-64.so.2 (0x00007f1a7df19000)
>
> 4. lsmod | grep mlx
> mlx5_ib 138046 0
> mlx5_core 108376 1 mlx5_ib
> mlx4_ib 199880 0
> ib_sa 43732 5 rdma_cm,ib_cm,mlx4_ib,rdma_ucm,ib_ipoib
> mlx4_en 137047 0
> ib_mad 51544 4 ib_cm,ib_sa,mlx4_ib,ib_umad
> vxlan 37619 1 mlx4_en
> ib_core 130498 12 rdma_cm,ib_cm,ib_sa,iw_cm,
> mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib
> mlx4_core 332375 2 mlx4_en,mlx4_ib
> compat 13738 17 rdma_cm,ib_cm,ib_sa,iw_cm,
> mlx4_en,mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_addr,ib_core,ib_
> umad,ib_uverbs,mlx4_core,mlx5_core,rdma_ucm,ib_ipoib
> ptp 18933 3 igb,ixgbe,mlx4_en
>
> Best regards,
> Raghav
>
> On Sat, Apr 4, 2015 at 5:04 PM Olga Shern <olgas@mellanox.com> wrote:
>
>> Hi Raghav,
>>
>>
>>
>> After running /etc/init.d/openibd restart you don’t need to load the
>> modules, they are loaded by the openibd script.
>>
>> Can you try to run :
>>
>> /etc/init.d/openibd stop
>>
>> Verify that modules are not loaded
>>
>> /etc/init.d/openib start
>>
>>
>>
>> If it won’t help:
>>
>> What Mellanox NIC do you have on the “problematic” machine?
>>
>> What command line do you use to run the application?
>>
>>
>>
>> Please send output of:
>>
>> 1. ”lspci | grep Mellanox”
>>
>> 2. ibdev2netdev
>>
>> 3. ldd <your application>
>>
>> 4. lsmod
>>
>>
>>
>> Thanks
>>
>> Olga
>>
>>
>>
>>
>>
>> *From:* Raghav Sethi [mailto:raghavs@CS.Princeton.EDU]
>> *Sent:* Saturday, April 04, 2015 11:54 PM
>> *To:* Olga Shern; Adrien Mazarguil; dev@dpdk.org; Xiaozhou Li
>>
>>
>> *Subject:* Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
>>
>>
>>
>> Hi Olga,
>>
>> Thanks for your reply. OFED version was 2.4-1.0.4 for ubuntu14.04-x86_64
>>
>> I tried both variants:
>>
>> - Compiling with dynamic linking against OFED libraries
>>
>> - Compiling with static linking against dpdk.org libraries
>>
>>
>>
>> Both have the same issue.
>>
>>
>>
>> Best,
>>
>> Raghav
>>
>>
>>
>> On Sat, Apr 4, 2015 at 4:49 PM Olga Shern <olgas@mellanox.com> wrote:
>>
>> Hi Raghav,
>>
>> What OFED version did you install?
>>
>> There are 2 ways to compile mlx4:
>> 1. You can compile it dynamically with the libibverbs that are coming
>> with Mellanox_OFED 2.4
>> 2. To get better performance: compile it statically with libibverbs and
>> libmlx4 that can be downloaded from http://dpdk.org/download, the README
>> inside librte_pmd_mlx4 explains how to do it
>> If you compile the pmd statically then MLNX_OFED is needed only for
>> kernel modules.
>>
>> Best Regards,
>> Olga
>>
>>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Raghav Sethi
>> Sent: Friday, April 03, 2015 9:21 PM
>> To: Adrien Mazarguil; dev@dpdk.org; Xiaozhou Li
>> Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
>>
>> Hi Adrien,
>>
>> Quick update: So I just tried the setup again on a new machine with the
>> exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a
>> charm.
>>
>> Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
>> From the error message it appears that the libraries are incorrectly
>> installed/configured. However, the OFED uninstall script removes the
>> libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so,
>> libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both
>> machines, and they're exactly identical. So it seems either the OFED
>> installation/uninstallation is somehow flawed, or the error message from
>> testpmd is misleading.
>>
>> Best,
>> Raghav
>>
>> On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs@cs.princeton.edu>
>> wrote:
>>
>> > Hi Adrien,
>> >
>> > I've copied dev@dpdk.
>> >
>> > I re-installed Mellanox OFED, and then re-compiled DPDK without the
>> > flags for the static library. I'm running Ubuntu 14.04. However, I
>> > still get the same error message (below).
>> >
>> > I then tried re-installing the Mellanox libraries on dpdk.org, and
>> > compiled another copy of DPDK with them statically linked. This
>> > appears to work perfectly. I no longer get the permission issues I was
>> > getting before, possibly that was unrelated/fixed by OFED. Now both
>> > OFED, and OFED+ dpdk.org libs statically linked have the same issue:
>> >
>> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
>> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
>> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
>> > version
>> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
>> > reference
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
>> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
>> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
>> > libibverbs.so.1 with link time reference
>> > libibverbs: Warning: no userspace device-specific driver found for
>> > /sys/class/infiniband_verbs/uverbs0
>> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
>> > EAL: No probed ethernet devices
>> >
>> > The OFED install was successful (and it apparently removes/overwrites
>> > old versions of the libraries). The installer also checks the firmware
>> > versions, and it says I'm on the latest. Loading the kernel modules is
>> > also successful, so I can't for the life of me figure out what this
>> issue is now.
>> >
>> > Again, just to be clear, the full list of steps was:
>> > 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with
>> > the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel
>> > modules 5. Run testpmd
>> >
>> > Best,
>> > Raghav
>> >
>> > On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <
>> > adrien.mazarguil@6wind.com> wrote:
>> >
>> >> Hi Raghav,
>> >>
>> >> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
>> >> > Hi Adrien,
>> >> >
>> >> > Thanks for your reply - I hope it's OK that I'm emailing you
>> off-list.
>> >> I'm
>> >> > a grad student at Princeton working on a research project.
>> >>
>> >> It's fine, but let's discuss on the ML as much as possible next time
>> >> so the community can benefit from it. Since very few people use this
>> >> driver at the moment (DPDK 2.0 is not even out yet), I have my own
>> >> questions for you.
>> >>
>> >> > Given your patch (which was very helpful, thanks), here is the
>> >> > series of all steps that I'm taking to run testpmd on a ConnectX-3
>> >> > EN 40G
>> >> >
>> >> > 1. Compile and install the mellanox-specific libraries listed here:
>> >> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
>> >>
>> >> This is fine, but did you use the provided Makefile to compile these
>> >> libraries statically? If so, no issues?
>> >>
>> >> Also you did not install Mellanox OFED? You should, because updated
>> >> kernel modules come from that distribution. It won't work properly
>> >> (or at all) without them.
>> >>
>> >> The libraries from DPDK.org are actually optional once you've
>> >> installed Mellanox OFED. They only improve performance. Remember to
>> >> update firmwares also.
>> >>
>> >> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new
>> >> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
>> >> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5.
>> >> > Set up hugepages using setup.sh 6. Load the kernel modules
>> >> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right
>> >> > PCI bus address
>> >> >
>> >> > However, I consistently get:
>> >> > sudo: unable to execute ./testpmd: Permission denied
>> >>
>> >> Now that doesn't make sense, isn't that file missing permissions or
>> >> something? Are you running RHEL with a restrictive SELinux policy?
>> >> Is sudo configured to allow you to run testpmd?
>> >>
>> >> In any case, I don't think it's related to librte_pmd_mlx4.
>> >>
>> >> > I tried taking the interface down with ifconfig because
>> >> > (./tools/dpdk_nic_bind.py --status shows the interface as active),
>> >> > but
>> >> that
>> >> > didn't change anything.
>> >>
>> >> No, you should leave the interfaces up, it's not supposed to
>> >> disappear, the PMD will bring them back up anyway. Actually the Linux
>> >> kernel interface remains fully functional even while DPDK
>> >> applications are running. You can send packets out of it but as long
>> >> as both the kernel and DPDK applications share the same MAC address,
>> >> only DPDK applications will receive replies.
>> >> If
>> >> you change the MAC address on the kernel interface however, you'll be
>> >> able to use it normally.
>> >>
>> >> > I tried installing the mellanox libraries globally, but then I see
>> >> > these error messages:
>> >> >
>> >> > -------
>> >> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
>> >> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4
>> >> > libibverbs: Warning: couldn't load driver
>> '/usr/lib/libibverbs/libmlx5':
>> >> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
>> >> version
>> >> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
>> >> > reference
>> >> > libibverbs: Warning: couldn't load driver
>> '/usr/lib/libibverbs/libmlx4':
>> >> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
>> >> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
>> >> > libibverbs.so.1 with link time reference
>> >> > libibverbs: Warning: no userspace device-specific driver found for
>> >> > /sys/class/infiniband_verbs/uverbs0
>> >> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
>> >> > EAL: No probed ethernet devices
>> >> > Interactive-mode selected
>> >> > EAL: Error - exiting with code: 1
>> >> > Cause: Creation of mbuf pool for socket 0 failed
>> >> > --------
>> >>
>> >> That's because the libibverbs version you're using is not compatible
>> >> with the userspace drivers installed on your system (libibverbs uses
>> >> dlopen() to load liblmx4). Even though you've installed both from
>> >> DPDK.org, seems you have another version installed by your
>> >> distribution packages.
>> >>
>> >> If you intend to use dynamic libraries, you should uninstall
>> >> libibverbs,
>> >> libmlx4 and related packages first.
>> >>
>> >> Regards,
>> >>
>> >> --
>> >> Adrien Mazarguil
>> >> 6WIND
>> >>
>> >
>>
>>
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2015-04-04 22:47 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-04-02 9:33 [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies Adrien Mazarguil
2015-04-03 13:53 ` Thomas Monjalon
[not found] ` <CAC2O_6oqG=3H0R5L6+ZX=1FzNj=mprDzOcUo_e6wezJxzu=ybw@mail.gmail.com>
[not found] ` <20150403073423.GY32147@6wind.com>
2015-04-03 16:52 ` Raghav Sethi
2015-04-03 18:21 ` Raghav Sethi
2015-04-04 20:49 ` Olga Shern
2015-04-04 20:53 ` Raghav Sethi
2015-04-04 21:04 ` Olga Shern
2015-04-04 21:29 ` Raghav Sethi
2015-04-04 22:04 ` Olga Shern
2015-04-04 22:47 ` Raghav Sethi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).