From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt0-f177.google.com (mail-qt0-f177.google.com [209.85.216.177]) by dpdk.org (Postfix) with ESMTP id 249D57D5B for ; Tue, 1 Aug 2017 11:49:46 +0200 (CEST) Received: by mail-qt0-f177.google.com with SMTP id p3so5873453qtg.2 for ; Tue, 01 Aug 2017 02:49:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=kRxG08dwFox+d4+eneiIRpq4lrfM9osHtsAyqjsOyp8=; b=ApIlqtqywsZr59Zd+KALmBnzbEIby5qAztR4SlV1j/Tp7zqcJ4usBkeHn82+yu/Zmz r5APf7k8DjJ3BSlxXg9X9NaowuQWMPsj5yQ9crIgAc1SHAfE69eJP2XkB5cVZHcG/EhS E/AXcDrb9FssL0dW+gzkiaLN+65YPGZPXGB/kZkzYqhA70XojhY1bLsgnEIQLtKwZNfj eOPYG3qf/ixAPX/0s5UJ20i0uBrccE9MhgCqgKlJp9ZwXomlIcEwW/G0W3sYV4adPeI4 827sDDeC2NL/icagJfz/n3BxN4gcvWp7cVs4Znkhjo43B1Owjsjp63eozTnAPPNKC0pN wahg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=kRxG08dwFox+d4+eneiIRpq4lrfM9osHtsAyqjsOyp8=; b=lBGHLrxoyXL2FC72fQK3RpRMe38rprwA3R5nKNwt/IpnH8aeen6I7+ud3SNdnAh7Bh rIWdPPxA0jUhYBGDlqF/O7Yrx7ubXbTCIDtuPSY4w6OG9zd+VsOndP2aZTXXCRLTEcvJ Jt2DRgqbXVtaNdR8/vwHHC/32B+wYz83kmR9Q0fDQ5vHEzaKDiH4CSxX5/Kb9A7JzHvb 2MmB1+NIltdha7hm4rXYXXx+Z4+TKLzOva83h2d26obaedDevEx1ZvSjNqZFST7LXtPR N0Y3rV0fVaq1ePbg2nYytkgkjNCawlm6ddfCaL9Mx91cMJMywgoUXCwtJl0I4LajzxWq VwPA== X-Gm-Message-State: AIVw111bUl4fRLElFyj9KvcmcbaBzcSgbD+9wwmLXmJkq34NxvrbJMDT bxrhha9AtDGYdiSoSC/f3PKb2YrD0w== X-Received: by 10.237.46.38 with SMTP id j35mr27443740qtd.285.1501580986359; Tue, 01 Aug 2017 02:49:46 -0700 (PDT) MIME-Version: 1.0 Received: by 10.200.37.201 with HTTP; Tue, 1 Aug 2017 02:49:45 -0700 (PDT) In-Reply-To: References: From: Zhilong Zheng Date: Tue, 1 Aug 2017 17:49:45 +0800 Message-ID: To: Paul Tsvika Cc: "users@dpdk.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] [dpdk][pktgen] - execute pktgen with mlx4 (40G) with some errors X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Aug 2017 09:49:47 -0000 I think you can check the device status using command "ibstat" on the sender and receiver. If both are down, it seems you should check the link connection. =E5=9C=A8 2017=E5=B9=B48=E6=9C=881=E6=97=A5=EF=BC=8C17:37=EF=BC=8CPaul Tsvi= ka =E5=86=99=E9=81=93=EF=BC=9A I found the issue with the output log when executed pktgen. | Ports 0-1 of 2
Copyright (c) <2010-2017>, Intel Corporatio= n Flags:Port : P--------------:0 Link State : * <--Down--> * ----TotalRate---- Pkts/s Max/Rx : 0/0 0/0 The link is down. This looks not right. Do you have any idea to check this? P.T 2017-08-01 17:12 GMT+08:00 Zhilong Zheng : > ~35Gbps and ~40Gbps when setting to 128 and 256. I use 4 cores to tx > packets. > > =E5=9C=A8 2017=E5=B9=B48=E6=9C=881=E6=97=A5=EF=BC=8C17:04=EF=BC=8CPaul Ts= vika =E5=86=99=E9=81=93=EF=BC=9A > > Hi Zheng, > > What throughput you had when you tried 128 or 256B packets? > > > P.T > > 2017-08-01 16:53 GMT+08:00 Zhilong Zheng : > >> Hi Paul, >> >> You can refer to the website (https://community.mellanox.co >> m/docs/DOC-2489#jive_content_id_Getting_started) to tune performance. By >> the way, it looks like that you can not get ~40Gbps throughputs when >> generating 64B packets as @Adrien mentioned (Actually I am also confused >> about this problem before, http://dpdk.org/ml/archives/us >> ers/2017-July/002226.html). >> >> Hope it can help you. >> >> Regards, >> >> Zhilong >> >> >> =E5=9C=A8 2017=E5=B9=B48=E6=9C=881=E6=97=A5=EF=BC=8C16:15=EF=BC=8CPaul T= svika =E5=86=99=E9=81=93=EF=BC=9A >> >> Hi zheng, >> >> Thanks for the feedback. >> >> You are right. To get pktgen work with mlx4, it needs to recompile DPDK >> and pktgen as well. >> >> I am able to run pktgen now. >> >> By following the document from DPDK website for this NIC, >> I did and change the following variables >> >> I set *MLX4_INLINE_RECV_SIZE=3D64* as the environment variable in bashrc= . >> set l*og_num_mgm_entry_size=3D-7* >> >> Disable pause frames: >> *ethtool -A ens2d1 rx off tx 0ff* >> >> I did not change the NIC register ( *setpci* ) at this time. >> >> >> and then I run the commands below in order: >> >> >> *test$: modprobe -a ib_uverbs mlx4_en mlx4_core mlx4_ib* >> *test$: /etc/init.d/openibd restart* >> *test:$: sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 0-6 -n 1 >> --proc-type auto --file-prefix pg -b 03:00.0 -b 03:00.1 -b 05:00.0 -b >> 05:00.1 -- -P -m "[1-5.0]"* >> >> | Ports 0-1 of 2
Copyright (c) <2010-2017>, Intel >> Corporation >> Flags:Port : P--------------:0 >> Link State : <--Down--> ----TotalRate---- >> Pkts/s Max/Rx : 0/0 0/0 >> Max/Tx : 844160/732319 844160/732319 >> MBits/s Rx/Tx : 0/*492 * 0/*492* >> Broadcast : 0 >> Multicast : 0 >> 64 Bytes : 0 >> 65-127 : 0 >> 128-255 : 0 >> 256-511 : 0 >> 512-1023 : 0 >> 1024-1518 : 0 >> Runts/Jumbos : 0/0 >> Errors Rx/Tx : 0/0 >> Total Rx Pkts : 0 >> Tx Pkts : 59289215 >> Rx MBs : 0 >> Tx MBs : 39842 >> ARP/ICMP Pkts : 0/0 >> Tx Overrun : 0 >> Cycles per Tx : 14880952/3200 >> Missed Rx : 0 >> mcasts Rx : None >> No Mbuf Rx : 0 >> : >> Pattern Type : abcd... >> Tx Count/% Rate : Forever /100% >> PktSize/Tx Burst : 64 / 32 >> Src/Dest Port : 1234 / 5678 >> Pkt Type:VLAN ID : IPv4 / TCP:0001 >> Dst IP Address : 192.168.1.1 >> Src IP Address : 192.168.0.1/24 >> Src MAC Address : e4:1d:2d:02:44:81 >> Powered by DPDK -------------------------- >> >> The transmitting rate should be around 40000 since it is 40G NIC. >> However, the rate is much lower than it and it is probably related to >> configurations. >> >> I actually have two mother boards and each one has 40G NICs installed. >> My intention is to do the tx / rx experiment. >> My first step is to check tx rate can reach as close to 40000 as possibl= e. >> >> Anyone has related experience please advise. >> >> >> >> P.T >> >> >> 2017-08-01 12:15 GMT+08:00 zhilong zheng : >> >>> >>> Hi, >>> >>> After enabling the MLX4 flag and recompiling the DPDK source code, Do >>> you recompile the code of pktgen? I think you can recompile it by runni= ng >>> make clean and make. >>> >>> Regards, >>> Zhilong >>> >>> > =E5=9C=A8 2017=E5=B9=B48=E6=9C=881=E6=97=A5=EF=BC=8C11:38=EF=BC=8CPau= l Tsvika =E5=86=99=E9=81=93=EF=BC=9A >>> > >>> > Hi Olga, >>> > >>> > 2017-08-01 0:11 GMT+08:00 Olga Shern : >>> > >>> >> Hi Paul, >>> >> >>> >> Looks like your ConnectX-3 configured to be IB and not ETH >>> >> >>> >>> pci@0000:04:00.0 ib0 network MT27500 Family [ConnectX-3= ] >>> >> >>> >> Use connectx_port_config script. >>> >> Set all the ports to Ethernet, and follow the instructions on the >>> screen. >>> >> >>> > >>> > Thanks. I followed the script and configured both ports to Ethernet. >>> > >>> > ibstat info below: >>> > >>> > CA 'mlx4_0' >>> > CA type: MT4099 >>> > Number of ports: 2 >>> > Firmware version: 2.40.7000 >>> > Hardware version: 1 >>> > Node GUID: 0xe41d2d0300024480 >>> > System image GUID: 0xe41d2d0300024483 >>> > Port 1: >>> > State: Down >>> > Physical state: Disabled >>> > Rate: 10 >>> > Base lid: 0 >>> > LMC: 0 >>> > SM lid: 0 >>> > Capability mask: 0x04010000 >>> > Port GUID: 0xe61d2dfffe024481 >>> > Link layer: Ethernet >>> > Port 2: >>> > State: Active >>> > Physical state: LinkUp >>> > Rate: 40 >>> > Base lid: 0 >>> > LMC: 0 >>> > SM lid: 0 >>> > Capability mask: 0x04010000 >>> > Port GUID: 0xe61d2dfffe024482 >>> > Link layer: Ethernet >>> > >>> > I also change the default value from n to >>> CONFIG_RTE_LIBRTE_MLX4_PMD=3Dy. >>> > Recompiled DPDK and it all worked. >>> > >>> > And Below is my ifconfig info: >>> > >>> > ens2 Link encap:Ethernet HWaddr e4:1d:2d:02:44:81 >>> > UP BROADCAST MULTICAST MTU:1500 Metric:1 >>> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >>> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >>> > collisions:0 txqueuelen:1000 >>> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >>> > >>> > ens2d1 Link encap:Ethernet HWaddr e4:1d:2d:02:44:82 >>> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>> > RX packets:126 errors:0 dropped:0 overruns:0 frame:0 >>> > TX packets:220 errors:0 dropped:0 overruns:0 carrier:0 >>> > collisions:0 txqueuelen:1000 >>> > RX bytes:29494 (29.4 KB) TX bytes:38063 (38.0 KB) >>> > >>> > >>> > Since I assume that I don't have to bind the device manually, I >>> executed >>> > ./dpdk-devbind.py --status with the following info popped out: >>> > >>> > Network devices using kernel driver >>> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>> > 0000:03:00.0 'Ethernet Connection X552/X557-AT 10GBASE-T 15ad' if=3De= no3 >>> > drv=3Dixgbe unused=3D >>> > 0000:03:00.1 'Ethernet Connection X552/X557-AT 10GBASE-T 15ad' if=3De= no4 >>> > drv=3Dixgbe unused=3D >>> > 0000:04:00.0 'MT27500 Family [ConnectX-3] 1003' if=3Dens2,ens2d1 >>> > drv=3Dmlx4_core unused=3D >>> > 0000:05:00.0 'I350 Gigabit Network Connection 1521' if=3Deno1 drv=3Di= gb >>> unused=3D >>> > 0000:05:00.1 'I350 Gigabit Network Connection 1521' if=3Deno2 drv=3Di= gb >>> unused=3D >>> > *Active* >>> > >>> > >>> > With these configurations, I run commands in order but still had the >>> same >>> > result: >>> > >>> > $test: modprobe -a ib_uverbs mlx4_en mlx4_core mlx4_ib >>> > $test: /etc/init.d/openibd restart >>> > $test: sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 0-6 -n 1 >>> --proc-type >>> > auto --file-prefix pg -b 03:00.0 -b 03:00.1 -b 05:00.0 -b 05:00.1 -- >>> -P -m >>> > "[1-5.0]" >>> > >>> > >>> >>>> Packet Burst 32, RX Desc 512, TX Desc 1024, mbufs/port 8192, mbuf >>> cache >>> > 1024 >>> > !PANIC!: *** Did not find any ports to use *** >>> > PANIC in pktgen_config_ports(): >>> > *** Did not find any ports to use ***6: >>> > [./app/x86_64-native-linuxapp-gcc/pktgen(_start+0x29) >>> > [0x447239]] >>> > 5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) >>> > [0x7f0ec0931830]] >>> > 4: [./app/x86_64-native-linuxapp-gcc/pktgen(main+0x580) [0x444710]] >>> > 3: [./app/x86_64-native-linuxapp-gcc/pktgen(pktgen_config_ports >>> +0x1614) >>> > [0x46e324]] >>> > 2: [./app/x86_64-native-linuxapp-gcc/pktgen(__rte_panic+0xc3) >>> [0x43ead4]] >>> > 1: [./app/x86_64-native-linuxapp-gcc/pktgen(rte_dump_stack+0x2b) >>> [0x4b7ecb]] >>> > Aborted (core dumped) >>> > >>> > >>> > >>> > Is there anything I've done wrong here? >>> > >>> > >>> > Thanks >>> > >>> > >>> > P.T >>> > >>> > >>> >> You can use -w to specify ConnectX-3 PCI address, you don't need to >>> use -b >>> >> >>> >> Make sure you compile DPDK with mlx4 PMD, it is not enabled by defau= lt >>> >> >>> >> Best Regards, >>> >> Olga >>> >> >>> >> >>> >> >>> > >>> > -- >>> > P.T >>> >> >> >> >> -- >> P.T >> >> >> > > > -- > P.T > > > --=20 P.T