From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from redflag.CS.Princeton.EDU (redflag.CS.Princeton.EDU [128.112.136.68]) by dpdk.org (Postfix) with ESMTP id 2E2659A8E for ; Sat, 4 Apr 2015 22:53:36 +0200 (CEST) Received: from mail-ig0-f170.google.com (mail-ig0-f170.google.com [209.85.213.170]) (authenticated bits=0) by redflag.CS.Princeton.EDU (8.13.8/8.13.8) with ESMTP id t34KrWt6011791 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=NOT) for ; Sat, 4 Apr 2015 16:53:34 -0400 Received: by igcau2 with SMTP id au2so990781igc.1 for ; Sat, 04 Apr 2015 13:53:32 -0700 (PDT) X-Received: by 10.42.10.198 with SMTP id r6mr5194568icr.10.1428180812311; Sat, 04 Apr 2015 13:53:32 -0700 (PDT) MIME-Version: 1.0 References: <1427967223-22918-1-git-send-email-adrien.mazarguil@6wind.com> <20150403073423.GY32147@6wind.com> In-Reply-To: From: Raghav Sethi Date: Sat, 04 Apr 2015 20:53:31 +0000 Message-ID: To: Olga Shern , Adrien Mazarguil , "dev@dpdk.org" , Xiaozhou Li Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Apr 2015 20:53:36 -0000 Hi Olga, Thanks for your reply. OFED version was 2.4-1.0.4 for ubuntu14.04-x86_64 I tried both variants: - Compiling with dynamic linking against OFED libraries - Compiling with static linking against dpdk.org libraries Both have the same issue. Best, Raghav On Sat, Apr 4, 2015 at 4:49 PM Olga Shern wrote: > Hi Raghav, > > What OFED version did you install? > > There are 2 ways to compile mlx4: > 1. You can compile it dynamically with the libibverbs that are coming > with Mellanox_OFED 2.4 > 2. To get better performance: compile it statically with libibverbs and > libmlx4 that can be downloaded from http://dpdk.org/download, the README > inside librte_pmd_mlx4 explains how to do it > If you compile the pmd statically then MLNX_OFED is needed only for > kernel modules. > > Best Regards, > Olga > > > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Raghav Sethi > Sent: Friday, April 03, 2015 9:21 PM > To: Adrien Mazarguil; dev@dpdk.org; Xiaozhou Li > Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies > > Hi Adrien, > > Quick update: So I just tried the setup again on a new machine with the > exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a > charm. > > Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW). > From the error message it appears that the libraries are incorrectly > installed/configured. However, the OFED uninstall script removes the > libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so, > libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both > machines, and they're exactly identical. So it seems either the OFED > installation/uninstallation is somehow flawed, or the error message from > testpmd is misleading. > > Best, > Raghav > > On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi > wrote: > > > Hi Adrien, > > > > I've copied dev@dpdk. > > > > I re-installed Mellanox OFED, and then re-compiled DPDK without the > > flags for the static library. I'm running Ubuntu 14.04. However, I > > still get the same error message (below). > > > > I then tried re-installing the Mellanox libraries on dpdk.org, and > > compiled another copy of DPDK with them statically linked. This > > appears to work perfectly. I no longer get the permission issues I was > > getting before, possibly that was unrelated/fixed by OFED. Now both > > OFED, and OFED+ dpdk.org libs statically linked have the same issue: > > > > EAL: PCI device 0000:06:00.0 on NUMA socket 0 > > EAL: probe driver: 15b3:1003 librte_pmd_mlx4 > > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5': > > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, > > version > > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time > > reference > > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4': > > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol > > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file > > libibverbs.so.1 with link time reference > > libibverbs: Warning: no userspace device-specific driver found for > > /sys/class/infiniband_verbs/uverbs0 > > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded? > > EAL: No probed ethernet devices > > > > The OFED install was successful (and it apparently removes/overwrites > > old versions of the libraries). The installer also checks the firmware > > versions, and it says I'm on the latest. Loading the kernel modules is > > also successful, so I can't for the life of me figure out what this > issue is now. > > > > Again, just to be clear, the full list of steps was: > > 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with > > the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel > > modules 5. Run testpmd > > > > Best, > > Raghav > > > > On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil < > > adrien.mazarguil@6wind.com> wrote: > > > >> Hi Raghav, > >> > >> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote: > >> > Hi Adrien, > >> > > >> > Thanks for your reply - I hope it's OK that I'm emailing you off-list. > >> I'm > >> > a grad student at Princeton working on a research project. > >> > >> It's fine, but let's discuss on the ML as much as possible next time > >> so the community can benefit from it. Since very few people use this > >> driver at the moment (DPDK 2.0 is not even out yet), I have my own > >> questions for you. > >> > >> > Given your patch (which was very helpful, thanks), here is the > >> > series of all steps that I'm taking to run testpmd on a ConnectX-3 > >> > EN 40G > >> > > >> > 1. Compile and install the mellanox-specific libraries listed here: > >> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/ > >> > >> This is fine, but did you use the provided Makefile to compile these > >> libraries statically? If so, no issues? > >> > >> Also you did not install Mellanox OFED? You should, because updated > >> kernel modules come from that distribution. It won't work properly > >> (or at all) without them. > >> > >> The libraries from DPDK.org are actually optional once you've > >> installed Mellanox OFED. They only improve performance. Remember to > >> update firmwares also. > >> > >> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new > >> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag. > >> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5. > >> > Set up hugepages using setup.sh 6. Load the kernel modules > >> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right > >> > PCI bus address > >> > > >> > However, I consistently get: > >> > sudo: unable to execute ./testpmd: Permission denied > >> > >> Now that doesn't make sense, isn't that file missing permissions or > >> something? Are you running RHEL with a restrictive SELinux policy? > >> Is sudo configured to allow you to run testpmd? > >> > >> In any case, I don't think it's related to librte_pmd_mlx4. > >> > >> > I tried taking the interface down with ifconfig because > >> > (./tools/dpdk_nic_bind.py --status shows the interface as active), > >> > but > >> that > >> > didn't change anything. > >> > >> No, you should leave the interfaces up, it's not supposed to > >> disappear, the PMD will bring them back up anyway. Actually the Linux > >> kernel interface remains fully functional even while DPDK > >> applications are running. You can send packets out of it but as long > >> as both the kernel and DPDK applications share the same MAC address, > >> only DPDK applications will receive replies. > >> If > >> you change the MAC address on the kernel interface however, you'll be > >> able to use it normally. > >> > >> > I tried installing the mellanox libraries globally, but then I see > >> > these error messages: > >> > > >> > ------- > >> > EAL: PCI device 0000:06:00.0 on NUMA socket 0 > >> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4 > >> > libibverbs: Warning: couldn't load driver > '/usr/lib/libibverbs/libmlx5': > >> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, > >> version > >> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time > >> > reference > >> > libibverbs: Warning: couldn't load driver > '/usr/lib/libibverbs/libmlx4': > >> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol > >> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file > >> > libibverbs.so.1 with link time reference > >> > libibverbs: Warning: no userspace device-specific driver found for > >> > /sys/class/infiniband_verbs/uverbs0 > >> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded? > >> > EAL: No probed ethernet devices > >> > Interactive-mode selected > >> > EAL: Error - exiting with code: 1 > >> > Cause: Creation of mbuf pool for socket 0 failed > >> > -------- > >> > >> That's because the libibverbs version you're using is not compatible > >> with the userspace drivers installed on your system (libibverbs uses > >> dlopen() to load liblmx4). Even though you've installed both from > >> DPDK.org, seems you have another version installed by your > >> distribution packages. > >> > >> If you intend to use dynamic libraries, you should uninstall > >> libibverbs, > >> libmlx4 and related packages first. > >> > >> Regards, > >> > >> -- > >> Adrien Mazarguil > >> 6WIND > >> > > >