From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from bluebox.CS.Princeton.EDU (bluebox.CS.Princeton.EDU [128.112.136.38]) by dpdk.org (Postfix) with ESMTP id 03BD7B6AB for ; Sun, 5 Apr 2015 00:47:42 +0200 (CEST) Received: from mail-ie0-f176.google.com (mail-ie0-f176.google.com [209.85.223.176]) (authenticated bits=0) by bluebox.CS.Princeton.EDU (8.13.8/8.13.8) with ESMTP id t34MlesJ021385 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=NOT) for ; Sat, 4 Apr 2015 18:47:42 -0400 Received: by ierf6 with SMTP id f6so729540ier.2 for ; Sat, 04 Apr 2015 15:47:39 -0700 (PDT) X-Received: by 10.107.15.129 with SMTP id 1mr12603100iop.20.1428187659829; Sat, 04 Apr 2015 15:47:39 -0700 (PDT) MIME-Version: 1.0 References: <1427967223-22918-1-git-send-email-adrien.mazarguil@6wind.com> <20150403073423.GY32147@6wind.com> In-Reply-To: From: Raghav Sethi Date: Sat, 04 Apr 2015 22:47:39 +0000 Message-ID: To: Olga Shern , Adrien Mazarguil , "dev@dpdk.org" , Xiaozhou Li Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Apr 2015 22:47:43 -0000 Hi Olga, Thanks so much! I had sent you the output for the case with dynamic linking. Removing the libraries from /usr/local and re-installing OFED solved the library issue. However, there is a new problem (below) An earlier install in the wrong place must have caused the problem. The OFED uninstaller does not remove the libraries from /usr/local, only usr/lib apparently. Also, the installer apparently doesn't install them in /usr/lib if they are present in /usr/local. Here is the output for testpmd now: EAL: PCI device 0000:06:00.0 on NUMA socket 0 EAL: probe driver: 15b3:1003 librte_pmd_mlx4 PMD: librte_pmd_mlx4: PCI information matches, using device "mlx4_0" (VF: false) PMD: librte_pmd_mlx4: 1 port(s) detected PMD: librte_pmd_mlx4: experimental ibv_exp_query_device EAL: No probed ethernet devices Any idea what the issue is in this case? Best, Raghav On Sat, Apr 4, 2015 at 6:04 PM Olga Shern wrote: > If you compiled pmd statically with libibverbs and libmlx4 you > shouldn't see it in ldd. > Seems that something went wrong. > In any case, ofed doesn't install libraries under /usr/local > Please remove all mlx related libraries under /usr/local/ like libmlx4 an= d > libibverbs > > > > > > -------- Original message -------- > From: Raghav Sethi > Date:05/04/2015 00:29 (GMT+02:00) > To: Olga Shern ,Adrien Mazarguil ,dev@dpdk.org,Xiaozhou Li > Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies > > > Hi Olga, > > Thanks very much for helping. > > I tried stopping and starting, after stop lsmod | grep mlx had no > matches. After start, it had the output listed below. The device is the > same as on the rest of the machines - Mellanox ConnectX3-EN 40G single > port. The application parameters are the same as the ones listed on the > docs (except of course PCI address is different): > > sudo ./testpmd -c 0xff -n 4 -w 0000:06:00.0 -- --rxq=3D2 --txq=3D2 -i > > Outputs you requested: > > 1. lspci | grep Mellanox > 06:00.0 Ethernet controller: Mellanox Technologies MT27500 Family > [ConnectX-3] > > 2. ibdev2netdev > mlx4_0 port 1 =3D=3D> p3p1 (Up) > > 3. ldd testpmd > linux-vdso.so.1 =3D> (0x00007fff7c1dc000) > libm.so.6 =3D> /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1a7dc0700= 0) > librt.so.1 =3D> /lib/x86_64-linux-gnu/librt.so.1 > (0x00007f1a7d9ff000) > libibverbs.so.1 =3D> /usr/local/lib/libibverbs.so.1 > (0x00007f1a7d7eb000) > libdl.so.2 =3D> /lib/x86_64-linux-gnu/libdl.so.2 > (0x00007f1a7d5e7000) > libpthread.so.0 =3D> /lib/x86_64-linux-gnu/libpthread.so.0 > (0x00007f1a7d3c9000) > libc.so.6 =3D> /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1a7d00300= 0) > /lib64/ld-linux-x86-64.so.2 (0x00007f1a7df19000) > > 4. lsmod | grep mlx > mlx5_ib 138046 0 > mlx5_core 108376 1 mlx5_ib > mlx4_ib 199880 0 > ib_sa 43732 5 rdma_cm,ib_cm,mlx4_ib,rdma_ucm,ib_ipoib > mlx4_en 137047 0 > ib_mad 51544 4 ib_cm,ib_sa,mlx4_ib,ib_umad > vxlan 37619 1 mlx4_en > ib_core 130498 12 rdma_cm,ib_cm,ib_sa,iw_cm, > mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib > mlx4_core 332375 2 mlx4_en,mlx4_ib > compat 13738 17 rdma_cm,ib_cm,ib_sa,iw_cm, > mlx4_en,mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_addr,ib_core,ib_ > umad,ib_uverbs,mlx4_core,mlx5_core,rdma_ucm,ib_ipoib > ptp 18933 3 igb,ixgbe,mlx4_en > > Best regards, > Raghav > > On Sat, Apr 4, 2015 at 5:04 PM Olga Shern wrote: > >> Hi Raghav, >> >> >> >> After running /etc/init.d/openibd restart you don=E2=80=99t need to load= the >> modules, they are loaded by the openibd script. >> >> Can you try to run : >> >> /etc/init.d/openibd stop >> >> Verify that modules are not loaded >> >> /etc/init.d/openib start >> >> >> >> If it won=E2=80=99t help: >> >> What Mellanox NIC do you have on the =E2=80=9Cproblematic=E2=80=9D machi= ne? >> >> What command line do you use to run the application? >> >> >> >> Please send output of: >> >> 1. =E2=80=9Dlspci | grep Mellanox=E2=80=9D >> >> 2. ibdev2netdev >> >> 3. ldd >> >> 4. lsmod >> >> >> >> Thanks >> >> Olga >> >> >> >> >> >> *From:* Raghav Sethi [mailto:raghavs@CS.Princeton.EDU] >> *Sent:* Saturday, April 04, 2015 11:54 PM >> *To:* Olga Shern; Adrien Mazarguil; dev@dpdk.org; Xiaozhou Li >> >> >> *Subject:* Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencie= s >> >> >> >> Hi Olga, >> >> Thanks for your reply. OFED version was 2.4-1.0.4 for ubuntu14.04-x86_64 >> >> I tried both variants: >> >> - Compiling with dynamic linking against OFED libraries >> >> - Compiling with static linking against dpdk.org libraries >> >> >> >> Both have the same issue. >> >> >> >> Best, >> >> Raghav >> >> >> >> On Sat, Apr 4, 2015 at 4:49 PM Olga Shern wrote: >> >> Hi Raghav, >> >> What OFED version did you install? >> >> There are 2 ways to compile mlx4: >> 1. You can compile it dynamically with the libibverbs that are coming >> with Mellanox_OFED 2.4 >> 2. To get better performance: compile it statically with libibverbs and >> libmlx4 that can be downloaded from http://dpdk.org/download, the README >> inside librte_pmd_mlx4 explains how to do it >> If you compile the pmd statically then MLNX_OFED is needed only for >> kernel modules. >> >> Best Regards, >> Olga >> >> >> -----Original Message----- >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Raghav Sethi >> Sent: Friday, April 03, 2015 9:21 PM >> To: Adrien Mazarguil; dev@dpdk.org; Xiaozhou Li >> Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies >> >> Hi Adrien, >> >> Quick update: So I just tried the setup again on a new machine with the >> exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like= a >> charm. >> >> Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW)= . >> From the error message it appears that the libraries are incorrectly >> installed/configured. However, the OFED uninstall script removes the >> libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so, >> libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both >> machines, and they're exactly identical. So it seems either the OFED >> installation/uninstallation is somehow flawed, or the error message from >> testpmd is misleading. >> >> Best, >> Raghav >> >> On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi >> wrote: >> >> > Hi Adrien, >> > >> > I've copied dev@dpdk. >> > >> > I re-installed Mellanox OFED, and then re-compiled DPDK without the >> > flags for the static library. I'm running Ubuntu 14.04. However, I >> > still get the same error message (below). >> > >> > I then tried re-installing the Mellanox libraries on dpdk.org, and >> > compiled another copy of DPDK with them statically linked. This >> > appears to work perfectly. I no longer get the permission issues I was >> > getting before, possibly that was unrelated/fixed by OFED. Now both >> > OFED, and OFED+ dpdk.org libs statically linked have the same issue: >> > >> > EAL: PCI device 0000:06:00.0 on NUMA socket 0 >> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4 >> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5= ': >> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, >> > version >> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time >> > reference >> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4= ': >> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol >> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file >> > libibverbs.so.1 with link time reference >> > libibverbs: Warning: no userspace device-specific driver found for >> > /sys/class/infiniband_verbs/uverbs0 >> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded? >> > EAL: No probed ethernet devices >> > >> > The OFED install was successful (and it apparently removes/overwrites >> > old versions of the libraries). The installer also checks the firmware >> > versions, and it says I'm on the latest. Loading the kernel modules is >> > also successful, so I can't for the life of me figure out what this >> issue is now. >> > >> > Again, just to be clear, the full list of steps was: >> > 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with >> > the CONFIG_RTE_LIBRTE_MLX4_PMD=3Dy 3. Set up hugepages 4. Load kernel >> > modules 5. Run testpmd >> > >> > Best, >> > Raghav >> > >> > On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil < >> > adrien.mazarguil@6wind.com> wrote: >> > >> >> Hi Raghav, >> >> >> >> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote: >> >> > Hi Adrien, >> >> > >> >> > Thanks for your reply - I hope it's OK that I'm emailing you >> off-list. >> >> I'm >> >> > a grad student at Princeton working on a research project. >> >> >> >> It's fine, but let's discuss on the ML as much as possible next time >> >> so the community can benefit from it. Since very few people use this >> >> driver at the moment (DPDK 2.0 is not even out yet), I have my own >> >> questions for you. >> >> >> >> > Given your patch (which was very helpful, thanks), here is the >> >> > series of all steps that I'm taking to run testpmd on a ConnectX-3 >> >> > EN 40G >> >> > >> >> > 1. Compile and install the mellanox-specific libraries listed here: >> >> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/ >> >> >> >> This is fine, but did you use the provided Makefile to compile these >> >> libraries statically? If so, no issues? >> >> >> >> Also you did not install Mellanox OFED? You should, because updated >> >> kernel modules come from that distribution. It won't work properly >> >> (or at all) without them. >> >> >> >> The libraries from DPDK.org are actually optional once you've >> >> installed Mellanox OFED. They only improve performance. Remember to >> >> update firmwares also. >> >> >> >> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new >> >> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=3Dy fla= g. >> >> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5. >> >> > Set up hugepages using setup.sh 6. Load the kernel modules >> >> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right >> >> > PCI bus address >> >> > >> >> > However, I consistently get: >> >> > sudo: unable to execute ./testpmd: Permission denied >> >> >> >> Now that doesn't make sense, isn't that file missing permissions or >> >> something? Are you running RHEL with a restrictive SELinux policy? >> >> Is sudo configured to allow you to run testpmd? >> >> >> >> In any case, I don't think it's related to librte_pmd_mlx4. >> >> >> >> > I tried taking the interface down with ifconfig because >> >> > (./tools/dpdk_nic_bind.py --status shows the interface as active), >> >> > but >> >> that >> >> > didn't change anything. >> >> >> >> No, you should leave the interfaces up, it's not supposed to >> >> disappear, the PMD will bring them back up anyway. Actually the Linux >> >> kernel interface remains fully functional even while DPDK >> >> applications are running. You can send packets out of it but as long >> >> as both the kernel and DPDK applications share the same MAC address, >> >> only DPDK applications will receive replies. >> >> If >> >> you change the MAC address on the kernel interface however, you'll be >> >> able to use it normally. >> >> >> >> > I tried installing the mellanox libraries globally, but then I see >> >> > these error messages: >> >> > >> >> > ------- >> >> > EAL: PCI device 0000:06:00.0 on NUMA socket 0 >> >> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4 >> >> > libibverbs: Warning: couldn't load driver >> '/usr/lib/libibverbs/libmlx5': >> >> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, >> >> version >> >> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time >> >> > reference >> >> > libibverbs: Warning: couldn't load driver >> '/usr/lib/libibverbs/libmlx4': >> >> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol >> >> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file >> >> > libibverbs.so.1 with link time reference >> >> > libibverbs: Warning: no userspace device-specific driver found for >> >> > /sys/class/infiniband_verbs/uverbs0 >> >> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded? >> >> > EAL: No probed ethernet devices >> >> > Interactive-mode selected >> >> > EAL: Error - exiting with code: 1 >> >> > Cause: Creation of mbuf pool for socket 0 failed >> >> > -------- >> >> >> >> That's because the libibverbs version you're using is not compatible >> >> with the userspace drivers installed on your system (libibverbs uses >> >> dlopen() to load liblmx4). Even though you've installed both from >> >> DPDK.org, seems you have another version installed by your >> >> distribution packages. >> >> >> >> If you intend to use dynamic libraries, you should uninstall >> >> libibverbs, >> >> libmlx4 and related packages first. >> >> >> >> Regards, >> >> >> >> -- >> >> Adrien Mazarguil >> >> 6WIND >> >> >> > >> >>