From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from bluebox.CS.Princeton.EDU (bluebox.CS.Princeton.EDU [128.112.136.38]) by dpdk.org (Postfix) with ESMTP id CFB49B6AB for ; Fri, 3 Apr 2015 18:52:11 +0200 (CEST) Received: from mail-ig0-f178.google.com (mail-ig0-f178.google.com [209.85.213.178]) (authenticated bits=0) by bluebox.CS.Princeton.EDU (8.13.8/8.13.8) with ESMTP id t33Gq7Bw014332 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=NOT) for ; Fri, 3 Apr 2015 12:52:10 -0400 Received: by igcau2 with SMTP id au2so67811824igc.1 for ; Fri, 03 Apr 2015 09:52:06 -0700 (PDT) X-Received: by 10.107.155.13 with SMTP id d13mr5138647ioe.29.1428079926825; Fri, 03 Apr 2015 09:52:06 -0700 (PDT) MIME-Version: 1.0 References: <1427967223-22918-1-git-send-email-adrien.mazarguil@6wind.com> <20150403073423.GY32147@6wind.com> In-Reply-To: <20150403073423.GY32147@6wind.com> From: Raghav Sethi Date: Fri, 03 Apr 2015 16:52:05 +0000 Message-ID: To: Adrien Mazarguil , "dev@dpdk.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2015 16:52:12 -0000 Hi Adrien, I've copied dev@dpdk. I re-installed Mellanox OFED, and then re-compiled DPDK without the flags for the static library. I'm running Ubuntu 14.04. However, I still get the same error message (below). I then tried re-installing the Mellanox libraries on dpdk.org, and compiled another copy of DPDK with them statically linked. This appears to work perfectly. I no longer get the permission issues I was getting before, possibly that was unrelated/fixed by OFED. Now both OFED, and OFED+dpdk.org libs statically linked have the same issue: EAL: PCI device 0000:06:00.0 on NUMA socket 0 EAL: probe driver: 15b3:1003 librte_pmd_mlx4 libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5': /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4': /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0 PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded? EAL: No probed ethernet devices The OFED install was successful (and it apparently removes/overwrites old versions of the libraries). The installer also checks the firmware versions, and it says I'm on the latest. Loading the kernel modules is also successful, so I can't for the life of me figure out what this issue is now. Again, just to be clear, the full list of steps was: 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel modules 5. Run testpmd Best, Raghav On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil wrote: > Hi Raghav, > > On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote: > > Hi Adrien, > > > > Thanks for your reply - I hope it's OK that I'm emailing you off-list. > I'm > > a grad student at Princeton working on a research project. > > It's fine, but let's discuss on the ML as much as possible next time so the > community can benefit from it. Since very few people use this driver at the > moment (DPDK 2.0 is not even out yet), I have my own questions for you. > > > Given your patch (which was very helpful, thanks), here is the series of > > all steps that I'm taking to run testpmd on a ConnectX-3 EN 40G > > > > 1. Compile and install the mellanox-specific libraries listed here: > > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/ > > This is fine, but did you use the provided Makefile to compile these > libraries statically? If so, no issues? > > Also you did not install Mellanox OFED? You should, because updated kernel > modules come from that distribution. It won't work properly (or at all) > without them. > > The libraries from DPDK.org are actually optional once you've installed > Mellanox OFED. They only improve performance. Remember to update firmwares > also. > > > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS > > 3. Create a new configuration which includes > > the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag. > > 4. Compile DPDK 2.0 with the new configuration using setup.sh > > 5. Set up hugepages using setup.sh > > 6. Load the kernel modules ib_uverbs mlx4_en mlx4_core mlx4_ib > > 7. Run testpmd with the right PCI bus address > > > > However, I consistently get: > > sudo: unable to execute ./testpmd: Permission denied > > Now that doesn't make sense, isn't that file missing permissions or > something? Are you running RHEL with a restrictive SELinux policy? > Is sudo configured to allow you to run testpmd? > > In any case, I don't think it's related to librte_pmd_mlx4. > > > I tried taking the interface down with ifconfig because > > (./tools/dpdk_nic_bind.py --status shows the interface as active), but > that > > didn't change anything. > > No, you should leave the interfaces up, it's not supposed to disappear, the > PMD will bring them back up anyway. Actually the Linux kernel interface > remains fully functional even while DPDK applications are running. You can > send packets out of it but as long as both the kernel and DPDK applications > share the same MAC address, only DPDK applications will receive replies. If > you change the MAC address on the kernel interface however, you'll be able > to use it normally. > > > I tried installing the mellanox libraries globally, but then I see these > > error messages: > > > > ------- > > EAL: PCI device 0000:06:00.0 on NUMA socket 0 > > EAL: probe driver: 15b3:1003 librte_pmd_mlx4 > > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5': > > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, > version > > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference > > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4': > > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow, > > version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time > > reference > > libibverbs: Warning: no userspace device-specific driver found for > > /sys/class/infiniband_verbs/uverbs0 > > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded? > > EAL: No probed ethernet devices > > Interactive-mode selected > > EAL: Error - exiting with code: 1 > > Cause: Creation of mbuf pool for socket 0 failed > > -------- > > That's because the libibverbs version you're using is not compatible with > the userspace drivers installed on your system (libibverbs uses dlopen() to > load liblmx4). Even though you've installed both from DPDK.org, seems you > have another version installed by your distribution packages. > > If you intend to use dynamic libraries, you should uninstall libibverbs, > libmlx4 and related packages first. > > Regards, > > -- > Adrien Mazarguil > 6WIND >