From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from emea01-am1-obe.outbound.protection.outlook.com (mail-am1on0072.outbound.protection.outlook.com [157.56.112.72]) by dpdk.org (Postfix) with ESMTP id 50C2B1396 for ; Sun, 5 Apr 2015 00:04:22 +0200 (CEST) Received: from AM2PR05MB0996.eurprd05.prod.outlook.com (25.161.234.146) by AM2PR05MB1124.eurprd05.prod.outlook.com (25.163.147.16) with Microsoft SMTP Server (TLS) id 15.1.125.19; Sat, 4 Apr 2015 22:04:21 +0000 Received: from AM2PR05MB0995.eurprd05.prod.outlook.com (25.161.234.145) by AM2PR05MB0996.eurprd05.prod.outlook.com (25.161.234.146) with Microsoft SMTP Server (TLS) id 15.1.125.19; Sat, 4 Apr 2015 22:04:20 +0000 Received: from AM2PR05MB0995.eurprd05.prod.outlook.com ([25.161.234.145]) by AM2PR05MB0995.eurprd05.prod.outlook.com ([25.161.234.145]) with mapi id 15.01.0125.002; Sat, 4 Apr 2015 22:04:19 +0000 From: Olga Shern To: Raghav Sethi , Adrien Mazarguil , "dev@dpdk.org" , Xiaozhou Li Thread-Topic: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies Thread-Index: AQHQbSgiO+fUkAfob06pxc+pO8bsip07gsmigAAY14CAAbliQIAAA36AgAAAknCAAAlpAIAACc6L Date: Sat, 4 Apr 2015 22:04:19 +0000 Message-ID: References: <1427967223-22918-1-git-send-email-adrien.mazarguil@6wind.com> <20150403073423.GY32147@6wind.com> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [84.229.208.242] authentication-results: CS.Princeton.EDU; dkim=none (message not signed) header.d=none; x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:; SRVR:AM2PR05MB0996; UriScan:; BCL:0; PCL:0; RULEID:; SRVR:AM2PR05MB1124; x-microsoft-antispam-prvs: x-forefront-antispam-report: BMV:1; SFV:NSPM; SFS:(10009020)(377454003)(479174004)(51704005)(13464003)(24454002)(2501003)(40100003)(122556002)(2420400003)(16601075003)(102836002)(15975445007)(62966003)(2900100001)(2950100001)(2171001)(77156002)(92566002)(87936001)(2656002)(93886004)(66066001)(16236675004)(15395725005)(50986999)(54356999)(76176999)(16297215004)(63666004)(107886001)(19617315012)(46102003)(86362001)(33646002)(19580395003)(19580405001); DIR:OUT; SFP:1101; SCL:1; SRVR:AM2PR05MB0996; H:AM2PR05MB0995.eurprd05.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(5005006)(5002010); SRVR:AM2PR05MB0996; BCL:0; PCL:0; RULEID:; SRVR:AM2PR05MB0996; x-forefront-prvs: 0536638EAC MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Apr 2015 22:04:19.6929 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM2PR05MB0996 X-OriginatorOrg: Mellanox.com Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Apr 2015 22:04:22 -0000 If you compiled pmd statically with libibverbs and libmlx4 you shouldn't se= e it in ldd. Seems that something went wrong. In any case, ofed doesn't install libraries under /usr/local Please remove all mlx related libraries under /usr/local/ like libmlx4 and = libibverbs -------- Original message -------- From: Raghav Sethi Date:05/04/2015 00:29 (GMT+02:00) To: Olga Shern ,Adrien Mazarguil ,dev@dpdk.org,Xiaozhou Li Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies Hi Olga, Thanks very much for helping. I tried stopping and starting, after stop lsmod | grep mlx had no matches. = After start, it had the output listed below. The device is the same as on t= he rest of the machines - Mellanox ConnectX3-EN 40G single port. The applic= ation parameters are the same as the ones listed on the docs (except of cou= rse PCI address is different): sudo ./testpmd -c 0xff -n 4 -w 0000:06:00.0 -- --rxq=3D2 --txq=3D2 -i Outputs you requested: 1. lspci | grep Mellanox 06:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX= -3] 2. ibdev2netdev mlx4_0 port 1 =3D=3D> p3p1 (Up) 3. ldd testpmd linux-vdso.so.1 =3D> (0x00007fff7c1dc000) libm.so.6 =3D> /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1a7dc07000) librt.so.1 =3D> /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1a7d9ff00= 0) libibverbs.so.1 =3D> /usr/local/lib/libibverbs.so.1 (0x00007f1a7d7e= b000) libdl.so.2 =3D> /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1a7d5e700= 0) libpthread.so.0 =3D> /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007= f1a7d3c9000) libc.so.6 =3D> /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1a7d003000) /lib64/ld-linux-x86-64.so.2 (0x00007f1a7df19000) 4. lsmod | grep mlx mlx5_ib 138046 0 mlx5_core 108376 1 mlx5_ib mlx4_ib 199880 0 ib_sa 43732 5 rdma_cm,ib_cm,mlx4_ib,rdma_ucm,ib_ipoib mlx4_en 137047 0 ib_mad 51544 4 ib_cm,ib_sa,mlx4_ib,ib_umad vxlan 37619 1 mlx4_en ib_core 130498 12 rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_ib,mlx5_ib,= ib_mad,ib_ucm,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib mlx4_core 332375 2 mlx4_en,mlx4_ib compat 13738 17 rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_en,mlx4_ib,= mlx5_ib,ib_mad,ib_ucm,ib_addr,ib_core,ib_umad,ib_uverbs,mlx4_core,mlx5_core= ,rdma_ucm,ib_ipoib ptp 18933 3 igb,ixgbe,mlx4_en Best regards, Raghav On Sat, Apr 4, 2015 at 5:04 PM Olga Shern > wrote: Hi Raghav, After running /etc/init.d/openibd restart you don=92t need to load the modu= les, they are loaded by the openibd script. Can you try to run : /etc/init.d/openibd stop Verify that modules are not loaded /etc/init.d/openib start If it won=92t help: What Mellanox NIC do you have on the =93problematic=94 machine? What command line do you use to run the application? Please send output of: 1. =94lspci | grep Mellanox=94 2. ibdev2netdev 3. ldd 4. lsmod Thanks Olga From: Raghav Sethi [mailto:raghavs@CS.Princeton.EDU] Sent: Saturday, April 04, 2015 11:54 PM To: Olga Shern; Adrien Mazarguil; dev@dpdk.org; Xiaozh= ou Li Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies Hi Olga, Thanks for your reply. OFED version was 2.4-1.0.4 for ubuntu14.04-x86_64 I tried both variants: - Compiling with dynamic linking against OFED libraries - Compiling with static linking against dpdk.org libraries Both have the same issue. Best, Raghav On Sat, Apr 4, 2015 at 4:49 PM Olga Shern > wrote: Hi Raghav, What OFED version did you install? There are 2 ways to compile mlx4: 1. You can compile it dynamically with the libibverbs that are coming with= Mellanox_OFED 2.4 2. To get better performance: compile it statically with libibverbs and lib= mlx4 that can be downloaded from http://dpdk.org/download, the README insid= e librte_pmd_mlx4 explains how to do it If you compile the pmd statically then MLNX_OFED is needed only for ke= rnel modules. Best Regards, Olga -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Beh= alf Of Raghav Sethi Sent: Friday, April 03, 2015 9:21 PM To: Adrien Mazarguil; dev@dpdk.org; Xiaozhou Li Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies Hi Adrien, Quick update: So I just tried the setup again on a new machine with the exa= ct same hardware - installed OFED and compiled DPDK 2.0.0. Works like a cha= rm. Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW). >>From the error message it appears that the libraries are incorrectly instal= led/configured. However, the OFED uninstall script removes the libraries in= question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so, libibverbs.so)= , I checked! In fact, I diffed the libibverbs.so from both machines, and th= ey're exactly identical. So it seems either the OFED installation/uninstall= ation is somehow flawed, or the error message from testpmd is misleading. Best, Raghav On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi > wrote: > Hi Adrien, > > I've copied dev@dpdk. > > I re-installed Mellanox OFED, and then re-compiled DPDK without the > flags for the static library. I'm running Ubuntu 14.04. However, I > still get the same error message (below). > > I then tried re-installing the Mellanox libraries on dpdk.org, and > compiled another copy of DPDK with them statically linked. This > appears to work perfectly. I no longer get the permission issues I was > getting before, possibly that was unrelated/fixed by OFED. Now both > OFED, and OFED+ dpdk.org libs statically linked have the= same issue: > > EAL: PCI device 0000:06:00.0 on NUMA socket 0 > EAL: probe driver: 15b3:1003 librte_pmd_mlx4 > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5': > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, > version > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time > reference > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4': > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file > libibverbs.so.1 with link time reference > libibverbs: Warning: no userspace device-specific driver found for > /sys/class/infiniband_verbs/uverbs0 > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded? > EAL: No probed ethernet devices > > The OFED install was successful (and it apparently removes/overwrites > old versions of the libraries). The installer also checks the firmware > versions, and it says I'm on the latest. Loading the kernel modules is > also successful, so I can't for the life of me figure out what this issue= is now. > > Again, just to be clear, the full list of steps was: > 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with > the CONFIG_RTE_LIBRTE_MLX4_PMD=3Dy 3. Set up hugepages 4. Load kernel > modules 5. Run testpmd > > Best, > Raghav > > On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil < > adrien.mazarguil@6wind.com> wrote: > >> Hi Raghav, >> >> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote: >> > Hi Adrien, >> > >> > Thanks for your reply - I hope it's OK that I'm emailing you off-list. >> I'm >> > a grad student at Princeton working on a research project. >> >> It's fine, but let's discuss on the ML as much as possible next time >> so the community can benefit from it. Since very few people use this >> driver at the moment (DPDK 2.0 is not even out yet), I have my own >> questions for you. >> >> > Given your patch (which was very helpful, thanks), here is the >> > series of all steps that I'm taking to run testpmd on a ConnectX-3 >> > EN 40G >> > >> > 1. Compile and install the mellanox-specific libraries listed here: >> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/ >> >> This is fine, but did you use the provided Makefile to compile these >> libraries statically? If so, no issues? >> >> Also you did not install Mellanox OFED? You should, because updated >> kernel modules come from that distribution. It won't work properly >> (or at all) without them. >> >> The libraries from DPDK.org are actually optional once you've >> installed Mellanox OFED. They only improve performance. Remember to >> update firmwares also. >> >> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new >> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=3Dy flag. >> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5. >> > Set up hugepages using setup.sh 6. Load the kernel modules >> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right >> > PCI bus address >> > >> > However, I consistently get: >> > sudo: unable to execute ./testpmd: Permission denied >> >> Now that doesn't make sense, isn't that file missing permissions or >> something? Are you running RHEL with a restrictive SELinux policy? >> Is sudo configured to allow you to run testpmd? >> >> In any case, I don't think it's related to librte_pmd_mlx4. >> >> > I tried taking the interface down with ifconfig because >> > (./tools/dpdk_nic_bind.py --status shows the interface as active), >> > but >> that >> > didn't change anything. >> >> No, you should leave the interfaces up, it's not supposed to >> disappear, the PMD will bring them back up anyway. Actually the Linux >> kernel interface remains fully functional even while DPDK >> applications are running. You can send packets out of it but as long >> as both the kernel and DPDK applications share the same MAC address, >> only DPDK applications will receive replies. >> If >> you change the MAC address on the kernel interface however, you'll be >> able to use it normally. >> >> > I tried installing the mellanox libraries globally, but then I see >> > these error messages: >> > >> > ------- >> > EAL: PCI device 0000:06:00.0 on NUMA socket 0 >> > EAL: probe driver: 15b3:1003 librte_pmd_mlx4 >> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5= ': >> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, >> version >> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time >> > reference >> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4= ': >> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol >> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file >> > libibverbs.so.1 with link time reference >> > libibverbs: Warning: no userspace device-specific driver found for >> > /sys/class/infiniband_verbs/uverbs0 >> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded? >> > EAL: No probed ethernet devices >> > Interactive-mode selected >> > EAL: Error - exiting with code: 1 >> > Cause: Creation of mbuf pool for socket 0 failed >> > -------- >> >> That's because the libibverbs version you're using is not compatible >> with the userspace drivers installed on your system (libibverbs uses >> dlopen() to load liblmx4). Even though you've installed both from >> DPDK.org, seems you have another version installed by your >> distribution packages. >> >> If you intend to use dynamic libraries, you should uninstall >> libibverbs, >> libmlx4 and related packages first. >> >> Regards, >> >> -- >> Adrien Mazarguil >> 6WIND >> >