From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from emea01-am1-obe.outbound.protection.outlook.com (mail-am1on0089.outbound.protection.outlook.com [157.56.112.89]) by dpdk.org (Postfix) with ESMTP id 3596EC43C for ; Mon, 27 Apr 2015 11:20:28 +0200 (CEST) Received: from AM2PR05MB0996.eurprd05.prod.outlook.com (25.161.234.146) by AM2PR05MB0674.eurprd05.prod.outlook.com (25.161.20.153) with Microsoft SMTP Server (TLS) id 15.1.148.16; Mon, 27 Apr 2015 09:20:28 +0000 Received: from AM2PR05MB0995.eurprd05.prod.outlook.com (25.161.234.145) by AM2PR05MB0996.eurprd05.prod.outlook.com (25.161.234.146) with Microsoft SMTP Server (TLS) id 15.1.148.15; Mon, 27 Apr 2015 09:20:26 +0000 Received: from AM2PR05MB0995.eurprd05.prod.outlook.com ([25.161.234.145]) by AM2PR05MB0995.eurprd05.prod.outlook.com ([25.161.234.145]) with mapi id 15.01.0148.008; Mon, 27 Apr 2015 09:20:26 +0000 From: Olga Shern To: Arthas , dev Thread-Topic: RE: [dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3problem! Thread-Index: AQHQgJcx4qlFeQ2T7kO1XmiUdcxpqp1gimcggAAKPICAAADmEA== Date: Mon, 27 Apr 2015 09:20:26 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: qq.com; dkim=none (message not signed) header.d=none; x-originating-ip: [193.47.165.251] x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:; SRVR:AM2PR05MB0996; UriScan:; BCL:0; PCL:0; RULEID:; SRVR:AM2PR05MB0674; x-microsoft-antispam-prvs: x-forefront-antispam-report: BMV:1; SFV:NSPM; SFS:(10009020)(50944005)(377454003)(19625215002)(54356999)(2656002)(33656002)(40100003)(16236675004)(19580405001)(86362001)(19580395003)(76576001)(5001770100001)(46102003)(87936001)(19617315012)(74316001)(107886001)(5890100001)(92566002)(2950100001)(15975445007)(102836002)(76176999)(62966003)(19300405004)(226693001)(106116001)(2900100001)(93886004)(66066001)(77156002)(122556002)(50986999); DIR:OUT; SFP:1101; SCL:1; SRVR:AM2PR05MB0996; H:AM2PR05MB0995.eurprd05.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(5005006)(5002010)(3002001); SRVR:AM2PR05MB0996; BCL:0; PCL:0; RULEID:; SRVR:AM2PR05MB0996; x-forefront-prvs: 0559FB9674 MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Apr 2015 09:20:26.5584 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM2PR05MB0996 X-OriginatorOrg: Mellanox.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3problem! X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Apr 2015 09:20:28 -0000 OK, great it is clear :) From: Arthas [mailto:kangzy1982@qq.com] Sent: Monday, April 27, 2015 12:17 PM To: Olga Shern; dev Subject: Re: RE: [dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3probl= em! Hi, Olga Yes, you are right! "Mellanox PMD can also be statically compiled with optimized libibverbs and= libmlx4 that can be downloaded from dpdk.org site." Only libibverbs tools need compile with shared! :) and because my environment is busybox with my custom kernel, so the kernel = version is very important yet. the MLNX_OFED package contain a ofed-docs indicate it! :) Best Regards! Kang ------------------ Original ------------------ From: "Olga Shern";>; Date: Mon, Apr 27, 2015 04:50 PM To: "Arthas">; "dev">; Subject: RE: [dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3problem! Hi Kang, I am glad you were able to fix the problem. But I think there is some confusion here, it means that we need have better= documentation :) First step to work with Mellanox NIC is to install proper MLNX_OFED and FW = from Mellanox.com Seems that you did it. Then you need to run /etc/init.d/openibd restart to load the needed modules= . For DPDK, you need to configure your card to work as Ethernet, Mellanox VP= I cards can be either Infiniband or Ethernet. Use connectx_port_config to configure it, as you did After that you can run ibdev2netdev and ifconfig and verify that you have i= nterfaces for Mellanox NIC ports. Then you can compile DPDK with Mellanox PMD enabled. Mellanox PMD can be dy= namically linked with the MLNX_OFED installed libibverbs and libmlx4 - but = the performance you get is not the best one. Mellanox PMD can also be statically compiled with optimized libibverbs and = libmlx4 that can be downloaded from dpdk.org site. Let me know if it clear. Best Regards, Olga From: Arthas [mailto:kangzy1982@qq.com] Sent: Monday, April 27, 2015 6:07 AM To: Arthas; Olga Shern; dev Subject: Re: [dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3problem! Hi, Olga, I think I have solved this problem. libibverbs & libmlx4 can't use dpdp.org and can't compile with static linki= ng. I try use libibverbs-1.1.8mlnx1-OFED.2.4.45.ga305acd.src.rpm & libmlx4-1.0.= 6mlnx1-OFED.2.4.0.1.2.src and compile with dynamic. This two package in MLN= X_OFED version 2.4-1.0.0 source package. and I use ofed_script connectx_port_config config Card with Ethernet mode= . The testpmd was started with no error! :) Thanks a million!! :) Best Regards! Kang ------------------ Original ------------------ From: "Arthas";>; Date: Mon, Apr 27, 2015 10:37 AM To: "Olga Shern">; "dev">; Subject: Re: [dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3problem! 3KS, Olga, My MLNX_OFED version 2.4-1.0.0 and libibverbs & libmlx4 download from dpd= k.org. when i try to run ibdev2netdev , it don't output any message! :( Here is my environment. # uname -r 3.10.0-mlx # lsmod mlx4_en 92824 0 - Live 0xffffffffa010f000 (O) mlx4_ib 135033 0 - Live 0xffffffffa00e3000 (O) ib_sa 24721 1 mlx4_ib, Live 0xffffffffa00d7000 (O) ib_mad 31664 2 mlx4_ib,ib_sa, Live 0xffffffffa00ca000 (O) mlx4_core 226516 2 mlx4_en,mlx4_ib, Live 0xffffffffa007c000 (O) ib_uverbs 43309 0 - Live 0xffffffffa003f000 (O) ib_core 79534 4 mlx4_ib,ib_sa,ib_mad,ib_uverbs, Live 0xffffffffa001d000 (O) ib_addr 4273 2 ib_uverbs,ib_core, Live 0xffffffffa0017000 (O) compat 4948 8 mlx4_en,mlx4_ib,ib_sa,ib_mad,mlx4_core,ib_uverbs,ib_core,ib_a= ddr, Live 0xffffffffa0011000 (O) # mstflint -d 02:00.0 q Image type: FS2 FW Version: 2.33.5100 FW Release Date: 25.1.2015 Product Version: 02.33.51.00 Rom Info: type=3DPXE version=3D3.4.460 devid=3D4099 Device ID: 4099 Description: Node Port1 Port2 Sys ima= ge GUIDs: 0002c90300faa480 0002c90300faa481 0002c90300faa482 0002c90= 300faa483 MACs: 0002c9faa481 0002c9faa482 VSD: PSID: MT_1090110019 # ibv_devices libibverbs: Warning: dlopen(NULL) failed, assuming static linking. libibverbs: Warning: no userspace device-specific driver found for /sys/cla= ss/infiniband_verbs/uverbs0 When linking libibverbs statically, driver must be statically linke= d too. device node GUID ------ ---------------- # ibv_devinfo -l libibverbs: Warning: dlopen(NULL) failed, assuming static linking. libibverbs: Warning: no userspace device-specific driver found for /sys/cla= ss/infiniband_verbs/uverbs0 When linking libibverbs statically, driver must be statically linke= d too. 0 HCAs found: /sys/class/infiniband_verbs # ls abi_version uverbs0 # find . -name "mlx4*" ./proc/irq/148/mlx4-32@0000:02:00.0 ./proc/irq/147/mlx4-31@0000:02:00.0 ./proc/irq/146/mlx4-30@0000:02:00.0 ./proc/irq/145/mlx4-29@0000:02:00.0 ./proc/irq/144/mlx4-28@0000:02:00.0 ./proc/irq/143/mlx4-27@0000:02:00.0 ./proc/irq/142/mlx4-26@0000:02:00.0 ./proc/irq/141/mlx4-25@0000:02:00.0 ./proc/irq/140/mlx4-24@0000:02:00.0 ./proc/irq/139/mlx4-23@0000:02:00.0 ./proc/irq/138/mlx4-22@0000:02:00.0 ./proc/irq/137/mlx4-21@0000:02:00.0 ./proc/irq/136/mlx4-20@0000:02:00.0 ./proc/irq/135/mlx4-19@0000:02:00.0 ./proc/irq/134/mlx4-18@0000:02:00.0 ./proc/irq/133/mlx4-17@0000:02:00.0 ./proc/irq/132/mlx4-16@0000:02:00.0 ./proc/irq/131/mlx4-15@0000:02:00.0 ./proc/irq/130/mlx4-14@0000:02:00.0 ./proc/irq/129/mlx4-13@0000:02:00.0 ./proc/irq/128/mlx4-12@0000:02:00.0 ./proc/irq/127/mlx4-11@0000:02:00.0 ./proc/irq/126/mlx4-10@0000:02:00.0 ./proc/irq/125/mlx4-9@0000:02:00.0 ./proc/irq/124/mlx4-8@0000:02:00.0 ./proc/irq/123/mlx4-7@0000:02:00.0 ./proc/irq/122/mlx4-6@0000:02:00.0 ./proc/irq/121/mlx4-5@0000:02:00.0 ./proc/irq/120/mlx4-4@0000:02:00.0 ./proc/irq/119/mlx4-3@0000:02:00.0 ./proc/irq/118/mlx4-2@0000:02:00.0 ./proc/irq/117/mlx4-1@0000:02:00.0 ./proc/irq/116/mlx4-async@pci:0000:02:00.0 ./proc/driver/mlx4_ib ./sys/bus/pci/drivers/mlx4_core ./sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/mlx4_port1_mtu ./sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/mlx4_port2_mtu ./sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/infiniband/mlx4_0 ./sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/mlx4_port1 ./sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/mlx4_port2 ./sys/class/infiniband/mlx4_0 ./sys/module/ib_sa/holders/mlx4_ib ./sys/module/compat/holders/mlx4_en ./sys/module/compat/holders/mlx4_ib ./sys/module/compat/holders/mlx4_core ./sys/module/mlx4_en ./sys/module/mlx4_ib ./sys/module/ib_mad/holders/mlx4_ib ./sys/module/ib_core/holders/mlx4_ib ./sys/module/mlx4_core ./sys/module/mlx4_core/holders/mlx4_en ./sys/module/mlx4_core/holders/mlx4_ib ./lib/modules/3.10.0-mlx/extra/mlnx-ofa_kernel/drivers/net/ethernet/mellano= x/mlx4 ./lib/modules/3.10.0-mlx/extra/mlnx-ofa_kernel/drivers/net/ethernet/mellano= x/mlx4/mlx4_en.ko ./lib/modules/3.10.0-mlx/extra/mlnx-ofa_kernel/drivers/net/ethernet/mellano= x/mlx4/mlx4_core.ko ./lib/modules/3.10.0-mlx/extra/mlnx-ofa_kernel/drivers/infiniband/hw/mlx4 ./lib/modules/3.10.0-mlx/extra/mlnx-ofa_kernel/drivers/infiniband/hw/mlx4/m= lx4_ib.ko ./sdd/tmp/mlx4.driver ./sdd/tmp/mlx4 ./sdd/tmp/mlx4/mlx4.hw ./sdd/tmp/mlx4/mlx4.config ./sdd/tmp/mlx4/mlx4.img.2.10.700 ./sdd/tmp/etc/modprobe.d/mlx4.conf ./etc/libibverbs.d/mlx4.driver ./etc/modprobe.d/mlx4.conf ------------------ Original ------------------ From: "Olga Shern";>; Date: Sun, Apr 26, 2015 03:28 PM To: "Arthas">; "dev">; Subject: RE: RE: [dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3prob= lem! Hi Kang, Sorry for the late response. ConnectX-3 VPI card that you have is supported. So there shouldn't be any p= roblem. Please make sure to follow the instructions in http://www.dpdk.org/doc/guid= es/nics/mlx4.html What MLNX_OFED did you install? What is the output of ibdev2netdev? Make sure, mlx4_core parameter log_num_mgm_entry_size =3D -1 Best Regards, Olga From: Arthas [mailto:kangzy1982@qq.com] Sent: Wednesday, April 22, 2015 6:56 AM To: Olga Shern; dev Subject: Re: RE: [dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3 prob= lem! 3KS, Olga, I try you suggestion and it can't start yet. My Mellanox ConnectX-3 car= d is "ConnectX(r)-3 VPI adapter card, dual-port QSFP, FDR IB (56Gb/s) and= 40/56GbE, PCIe3.0 x8 8GT/s, tall bracket,RoHS R6". May be the problem? # mstflint -d 02:00.0 q Image type: FS2 FW Version: 2.33.5100 FW Release Date: 25.1.2015 Product Version: 02.33.51.00 Rom Info: type=3DPXE version=3D3.4.460 devid=3D4099 Device ID: 4099 Description: Node Port1 Port2 Sys ima= ge GUIDs: 0002c90300faa480 0002c90300faa481 0002c90300faa482 0002c90= 300faa483 MACs: 0002c9faa481 0002c9faa482 VSD: PSID: MT_1090110019 Best Regards, Kang ------------------ Original ------------------ From: "Olga Shern";>; Date: Tue, Apr 21, 2015 09:34 PM To: "Arthas">; "dev">; Subject: RE: [dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3 problem= ! Hi Kang, You probably compiled the downloaded libraries using dynamic linkage, prio= r to the static one you specified here. You need to remove them from /usr/local/libs and also please delete /usr/l= ocal/include/infiniband Let me know if it solves your problem. Best Regards, Olga -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of ZhanYing Kang Sent: Tuesday, April 21, 2015 11:45 AM To: dev Subject: [dpdk-dev] DPDK v2.0 testpmd with Mellanox ConnectX-3 problem! When I'm run testpmd with DPDK 2.0 and the Mellanox ConnectX-3. I get a e= rror. Here are my questions: 1> lspci |grep Mell 02:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-= 3] 2> compile libibverbs-1.1.7mlnx1 && libmlx4-1.0.5mlnx1 export EXTRA_CFLAGS= =3D-I/DPDK/mlx4/install/usr/local/include export EXTRA_LDFLAGS=3D-L/DPDK/mlx4/install/usr/local/lib 3> compile dpdk v2.0 with x86_64-native-linuxapp-gcc config and 3> CONFIG_RTE_LIBRTE_MLX4_PMD=3Dy 4> modprobe -a ib_uverbs mlx4_en mlx4_core mlx4_ib lsmod mlx4_ib 135033 0 = - Live 0xffffffffa0215000 (O) ib_sa 24721 1 mlx4_ib, Live 0xffffffffa020900= 0 (O) ib_mad 31664 2 mlx4_ib,ib_sa, Live 0xffffffffa01fc000 (O) mlx4_en 928= 08 0 - Live 0xffffffffa01b1000 (O) mlx4_core 226516 2 mlx4_ib,mlx4_en, Live= 0xffffffffa0163000 (O) ib_uverbs 43309 0 - Live 0xffffffffa0133000 (O) ib_= core 79534 4 mlx4_ib,ib_sa,ib_mad,ib_uverbs, Live 0xffffffffa0100000 (O) ib= _addr 4273 2 ib_uverbs,ib_core, Live 0xffffffffa00fa000 (O) compat 4948 8 m= lx4_ib,ib_sa,ib_mad,mlx4_en,mlx4_core,ib_uverbs,ib_core,ib_addr, Live 0xfff= fffffa00ad000 (O) 5> run testpmd # ./testpmd-mlx -c 0xff00 -n 4 -w 0000:02:00.0 -- --rxq=3D2 --txq=3D2 -i EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 1 on socket 0 EAL: Detected lcore 2 as core 2 on socket 0 EAL: Detected lcore 3 as core 3 on socket 0 EAL: Detected lcore 4 as core 4 on socket 0 EAL: Detected lcore 5 as core 5 on socket 0 EAL: Detected lcore 6 as core 6 on socket 0 EAL: Detected lcore 7 as core 7 on socket 0 EAL: Detected lcore 8 as core 0 on socket 0 EAL: Detected lcore 9 as core 1 on socket 0 EAL: Detected lcore 10 as core 2 on socket 0 EAL: Detected lcore 11 as core 3 on socket 0 EAL: Detected lcore 12 as core 4 on socket 0 EAL: Detected lcore 13 as core 5 on socket 0 EAL: Detected lcore 14 as core 6 on socket 0 EAL: Detected lcore 15 as core 7 on socket 0 EAL: Support maximum 32 logical core(s) by configuration. EAL: Detected 16 lcore(s) EAL: Setting up memory... EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7f5a2b800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x31000000 bytes EAL: Virtual area found at 0x7f59fa600000 (size =3D 0x31000000) EAL: Ask a virtual area of 0x7ec00000 bytes EAL: Virtual area found at 0x7f597b800000 (size =3D 0x7ec00000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7f597b400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x250000000 bytes EAL: Virtual area found at 0x7f572b200000 (size =3D 0x250000000) EAL: Requesting 6144 pages of size 2MB from socket 0 EAL: TSC frequency is ~2394227 KHz EAL: Master lcore 8 is ready (tid=3D2e1b7900;cpuset=3D[8]) EAL: lcore 11 is ready (tid=3D299b7700;cpuset=3D[11]) EAL: lcore 14 is ready (tid=3D281b4700;cpuset=3D[14]) EAL: lcore 15 is ready (tid=3D279b3700;cpuset=3D[15]) EAL: lcore 12 is ready (tid=3D291b6700;cpuset=3D[12]) EAL: lcore 9 is ready (tid=3D2caaf700;cpuset=3D[9]) EAL: lcore 10 is ready (tid=3D2c2ae700;cpuset=3D[10]) EAL: lcore 13 is ready (tid=3D289b5700;cpuset=3D[13]) EAL: PCI device 0000:02:00.0 on NUMA socket -1 EAL: probe driver: 15b3:1003 librte_pmd_mlx4 PMD: librte_pmd_mlx4: PCI information matches, using device "mlx4_0" (VF: f= alse) PMD: librte_pmd_mlx4: 2 port(s) detected PMD: librte_pmd_mlx4: bad state for port 1: "down" (1) PMD: librte_pmd_mlx4: port 1 MAC address is 02:02:c9:fa:a4:81 PMD: librte_pmd_mlx4: bad state for port 2: "down" (1) PMD: librte_pmd_mlx4: port 2 MAC address is 00:02:c9:fa:a4:82 Interactive-m= ode selected Configuring Port 0 (socket 0) PMD: librte_pmd_mlx4: 0x7dffe0: TX queues number update: 0 -> 2 PMD: librte_pmd_mlx4: 0x7dffe0: RX queues number update: 0 -> 2 PMD: librte_pmd_mlx4: 0x7fffc81e67e0: flow configuration failed, errno=3D38= : Function not implemented PMD: librte_pmd_mlx4: 0x7fffc81e67e0: flow configuration failed, errno=3D38= : Function not implemented PMD: librte_pmd_mlx4: 0x7dffe0: QP flow attachment failed: Function not imp= lemented Fail to configure port 0 EAL: Error - exiting with code: 1 Cause: Start ports failed Best, Kang!