From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from bluebox.CS.Princeton.EDU (bluebox.CS.Princeton.EDU [128.112.136.38]) by dpdk.org (Postfix) with ESMTP id 91FCA6A95 for ; Fri, 3 Apr 2015 20:21:19 +0200 (CEST) Received: from mail-ig0-f171.google.com (mail-ig0-f171.google.com [209.85.213.171]) (authenticated bits=0) by bluebox.CS.Princeton.EDU (8.13.8/8.13.8) with ESMTP id t33ILGBP021725 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=NOT) for ; Fri, 3 Apr 2015 14:21:18 -0400 Received: by ignm3 with SMTP id m3so68875377ign.0 for ; Fri, 03 Apr 2015 11:21:16 -0700 (PDT) X-Received: by 10.51.17.40 with SMTP id gb8mr6530202igd.44.1428085276510; Fri, 03 Apr 2015 11:21:16 -0700 (PDT) MIME-Version: 1.0 References: <1427967223-22918-1-git-send-email-adrien.mazarguil@6wind.com> <20150403073423.GY32147@6wind.com> In-Reply-To: From: Raghav Sethi Date: Fri, 03 Apr 2015 18:21:15 +0000 Message-ID: To: Adrien Mazarguil , "dev@dpdk.org" , Xiaozhou Li Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2015 18:21:20 -0000 Hi Adrien, Quick update: So I just tried the setup again on a new machine with the exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a charm. Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW). >>From the error message it appears that the libraries are incorrectly installed/configured. However, the OFED uninstall script removes the libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so, libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both machines, and they're exactly identical. So it seems either the OFED installation/uninstallation is somehow flawed, or the error message from testpmd is misleading. Best, Raghav On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi wrote: > Hi Adrien, > > I've copied dev@dpdk. > > I re-installed Mellanox OFED, and then re-compiled DPDK without the flags > for the static library. I'm running Ubuntu 14.04. However, I still get the > same error message (below). > > I then tried re-installing the Mellanox libraries on dpdk.org, and > compiled another copy of DPDK with them statically linked. This appears to > work perfectly. I no longer get the permission issues I was getting before, > possibly that was unrelated/fixed by OFED. Now both OFED, and OFED+ > dpdk.org libs statically linked have the same issue: > > EAL: PCI device 0000:06:00.0 on NUMA socket 0 > EAL: probe driver: 15b3:1003 librte_pmd_mlx4 > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5': > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, version > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4': > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow, > version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time > reference > libibverbs: Warning: no userspace device-specific driver found for > /sys/class/infiniband_verbs/uverbs0 > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded? > EAL: No probed ethernet devices > > The OFED install was successful (and it apparently removes/overwrites old > versions of the libraries). The installer also checks the firmware > versions, and it says I'm on the latest. Loading the kernel modules is also > successful, so I can't for the life of me figure out what this issue is now. > > Again, just to be clear, the full list of steps was: > 1. Install OFED, run /etc/init.d/openibd restart > 2. Compile DPDK with the CONFIG_RTE_LIBRTE_MLX4_PMD=y > 3. Set up hugepages > 4. Load kernel modules > 5. Run testpmd > > Best, > Raghav > > On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil < > adrien.mazarguil@6wind.com> wrote: > >> Hi Raghav, >> >> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote: >> > Hi Adrien, >> > >> > Thanks for your reply - I hope it's OK that I'm emailing you off-list. >> I'm >> > a grad student at Princeton working on a research project. >> >> It's fine, but let's discuss on the ML as much as possible next time so >> the >> community can benefit from it. Since very few people use this driver at >> the >> moment (DPDK 2.0 is not even out yet), I have my own questions for you. >> >> > Given your patch (which was very helpful, thanks), here is the series of >> > all steps that I'm taking to run testpmd on a ConnectX-3 EN 40G >> > >> > 1. Compile and install the mellanox-specific libraries listed here: >> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/ >> >> This is fine, but did you use the provided Makefile to compile these >> libraries statically? If so, no issues? >> >> Also you did not install Mellanox OFED? You should, because updated kernel >> modules come from that distribution. It won't work properly (or at all) >> without them. >> >> The libraries from DPDK.org are actually optional once you've installed >> Mellanox OFED. They only improve performance. Remember to update firmwares >> also. >> >> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS >> > 3. Create a new configuration which includes >> > the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag. >> > 4. Compile DPDK 2.0 with the new configuration using setup.sh >> > 5. Set up hugepages using setup.sh >> > 6. Load the kernel modules ib_uverbs mlx4_en mlx4_core mlx4_ib >> > 7. Run testpmd with the right PCI bus address >> > >> > However, I consistently get: >> > sudo: unable to execute ./testpmd: Permission denied >> >> Now that doesn't make sense, isn't that file missing permissions or >> something? Are you running RHEL with a restrictive SELinux policy? >> Is sudo configured to allow you to run testpmd? >> >> In any case, I don't think it's related to librte_pmd_mlx4. >> >> > I tried taking the interface down with ifconfig because >> > (./tools/dpdk_nic_bind.py --status shows the interface as active), but >> that >> > didn't change anything. >> >> No, you should leave the interfaces up, it's not supposed to disappear, >> the >> PMD will bring them back up anyway. Actually the Linux kernel interface >> remains fully functional even while DPDK applications are running. You can >> send packets out of it but as long as both the kernel and DPDK >> applications >> share the same MAC address, only DPDK applications will receive replies. >> If >> you change the MAC address on the kernel interface however, you'll be able >> to use it normally. >> >> > I tried installing the mellanox libraries globally, but then I see these >> > error messages: >> > >> > ------- >> > EAL: PCI device 0000:06:00.0 on NUMA socket 0 >> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4 >> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5': >> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, >> version >> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference >> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4': >> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow, >> > version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time >> > reference >> > libibverbs: Warning: no userspace device-specific driver found for >> > /sys/class/infiniband_verbs/uverbs0 >> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded? >> > EAL: No probed ethernet devices >> > Interactive-mode selected >> > EAL: Error - exiting with code: 1 >> > Cause: Creation of mbuf pool for socket 0 failed >> > -------- >> >> That's because the libibverbs version you're using is not compatible with >> the userspace drivers installed on your system (libibverbs uses dlopen() >> to >> load liblmx4). Even though you've installed both from DPDK.org, seems you >> have another version installed by your distribution packages. >> >> If you intend to use dynamic libraries, you should uninstall libibverbs, >> libmlx4 and related packages first. >> >> Regards, >> >> -- >> Adrien Mazarguil >> 6WIND >> >