From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt0-f182.google.com (mail-qt0-f182.google.com [209.85.216.182]) by dpdk.org (Postfix) with ESMTP id 5F84B915D for ; Tue, 1 Aug 2017 11:12:07 +0200 (CEST) Received: by mail-qt0-f182.google.com with SMTP id t37so5415950qtg.5 for ; Tue, 01 Aug 2017 02:12:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=oJifLt19UMdZKV4U3pkzp5b637toUjTjyzBVqfMsNOA=; b=SwwVpg0RHkHJ0gKhSNFc+jtgfm7/CBO4Dw5b++WtZ/Qf7ViY3SCe9NmoIk01Tq8YZa xau2DOZvwAvU+BUvBozdqSNYPYKfcidwkZX0kWn3M/rliAilEg1kAhA9obUyvOx6tKbF wuEbZMQixSY6LZcwHthV2xYMV7mRMhCg8QeHEr3A+2ow9YSwh81J3hj5jZyJa8kdfS93 hRV3nxcO3wOoqAJ7BFBbbq0Pf3qSYPVX2n+M9jpeZ8EjNptGp+ztS2FyzVZpl8QNW8nd w5Pl02RuAQhDxX3DvUyd8WEYzakfvSRnws9pfPWyzgF4M3FKWS1omUp/ex7GXSgraKZ9 LOzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=oJifLt19UMdZKV4U3pkzp5b637toUjTjyzBVqfMsNOA=; b=aX+NUA1FcN8HjKKWOzAOdf6L2ZA/i1PKBJHCi3Rw9s/1/nCEGIHqnjdujKCaTueBMi Tz2Q1sw5KX4Yha1vRBsZb1E8zI/PkAbi68OvSYGJzZkJVjyjeR+Fa6hWdEp/ujEVuqKl QlA3rcBic+vyk6T9W3NlDbAk/d6C0QVqMWG8lis2hxGLpccKCsfVdWkOENd7jCQCWOBL wqOKtCTtra5n8cDbOjm66YY9PuLLU5jixtO1ItwXy6kxacfeWVZsKZgC/pXKXyjfnXm6 75DFI7WSW/01HK63JY/pKDbINg5gtiYU88W2dT7G6WU8/f2JdweJDool8bcLNsaCzf57 h6vg== X-Gm-Message-State: AIVw110IaQVBDuPJcEyRBExYaxo1AJ8JH73cmVS2l69ZuS/+k1EpHNpn ADJOzOtv5V+0W2Xc/v9C6bIuM7K8KQ== X-Received: by 10.200.48.66 with SMTP id g2mr25579360qte.119.1501578726552; Tue, 01 Aug 2017 02:12:06 -0700 (PDT) MIME-Version: 1.0 Received: by 10.200.37.201 with HTTP; Tue, 1 Aug 2017 02:12:06 -0700 (PDT) In-Reply-To: References: From: Zhilong Zheng Date: Tue, 1 Aug 2017 17:12:06 +0800 Message-ID: To: Paul Tsvika Cc: "users@dpdk.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] [dpdk][pktgen] - execute pktgen with mlx4 (40G) with some errors X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Aug 2017 09:12:07 -0000 ~35Gbps and ~40Gbps when setting to 128 and 256. I use 4 cores to tx packets. =E5=9C=A8 2017=E5=B9=B48=E6=9C=881=E6=97=A5=EF=BC=8C17:04=EF=BC=8CPaul Tsvi= ka =E5=86=99=E9=81=93=EF=BC=9A Hi Zheng, What throughput you had when you tried 128 or 256B packets? P.T 2017-08-01 16:53 GMT+08:00 Zhilong Zheng : > Hi Paul, > > You can refer to the website (https://community.mellanox.co > m/docs/DOC-2489#jive_content_id_Getting_started) to tune performance. By > the way, it looks like that you can not get ~40Gbps throughputs when > generating 64B packets as @Adrien mentioned (Actually I am also confused > about this problem before, http://dpdk.org/ml/archives/us > ers/2017-July/002226.html). > > Hope it can help you. > > Regards, > > Zhilong > > > =E5=9C=A8 2017=E5=B9=B48=E6=9C=881=E6=97=A5=EF=BC=8C16:15=EF=BC=8CPaul Ts= vika =E5=86=99=E9=81=93=EF=BC=9A > > Hi zheng, > > Thanks for the feedback. > > You are right. To get pktgen work with mlx4, it needs to recompile DPDK > and pktgen as well. > > I am able to run pktgen now. > > By following the document from DPDK website for this NIC, > I did and change the following variables > > I set *MLX4_INLINE_RECV_SIZE=3D64* as the environment variable in bashrc. > set l*og_num_mgm_entry_size=3D-7* > > Disable pause frames: > *ethtool -A ens2d1 rx off tx 0ff* > > I did not change the NIC register ( *setpci* ) at this time. > > > and then I run the commands below in order: > > > *test$: modprobe -a ib_uverbs mlx4_en mlx4_core mlx4_ib* > *test$: /etc/init.d/openibd restart* > *test:$: sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 0-6 -n 1 > --proc-type auto --file-prefix pg -b 03:00.0 -b 03:00.1 -b 05:00.0 -b > 05:00.1 -- -P -m "[1-5.0]"* > > | Ports 0-1 of 2
Copyright (c) <2010-2017>, Intel > Corporation > Flags:Port : P--------------:0 > Link State : <--Down--> ----TotalRate---- > Pkts/s Max/Rx : 0/0 0/0 > Max/Tx : 844160/732319 844160/732319 > MBits/s Rx/Tx : 0/*492 * 0/*492* > Broadcast : 0 > Multicast : 0 > 64 Bytes : 0 > 65-127 : 0 > 128-255 : 0 > 256-511 : 0 > 512-1023 : 0 > 1024-1518 : 0 > Runts/Jumbos : 0/0 > Errors Rx/Tx : 0/0 > Total Rx Pkts : 0 > Tx Pkts : 59289215 > Rx MBs : 0 > Tx MBs : 39842 > ARP/ICMP Pkts : 0/0 > Tx Overrun : 0 > Cycles per Tx : 14880952/3200 > Missed Rx : 0 > mcasts Rx : None > No Mbuf Rx : 0 > : > Pattern Type : abcd... > Tx Count/% Rate : Forever /100% > PktSize/Tx Burst : 64 / 32 > Src/Dest Port : 1234 / 5678 > Pkt Type:VLAN ID : IPv4 / TCP:0001 > Dst IP Address : 192.168.1.1 > Src IP Address : 192.168.0.1/24 > Src MAC Address : e4:1d:2d:02:44:81 > Powered by DPDK -------------------------- > > The transmitting rate should be around 40000 since it is 40G NIC. > However, the rate is much lower than it and it is probably related to > configurations. > > I actually have two mother boards and each one has 40G NICs installed. > My intention is to do the tx / rx experiment. > My first step is to check tx rate can reach as close to 40000 as possible= . > > Anyone has related experience please advise. > > > > P.T > > > 2017-08-01 12:15 GMT+08:00 zhilong zheng : > >> >> Hi, >> >> After enabling the MLX4 flag and recompiling the DPDK source code, Do yo= u >> recompile the code of pktgen? I think you can recompile it by running ma= ke >> clean and make. >> >> Regards, >> Zhilong >> >> > =E5=9C=A8 2017=E5=B9=B48=E6=9C=881=E6=97=A5=EF=BC=8C11:38=EF=BC=8CPaul= Tsvika =E5=86=99=E9=81=93=EF=BC=9A >> > >> > Hi Olga, >> > >> > 2017-08-01 0:11 GMT+08:00 Olga Shern : >> > >> >> Hi Paul, >> >> >> >> Looks like your ConnectX-3 configured to be IB and not ETH >> >> >> >>> pci@0000:04:00.0 ib0 network MT27500 Family [ConnectX-3] >> >> >> >> Use connectx_port_config script. >> >> Set all the ports to Ethernet, and follow the instructions on the >> screen. >> >> >> > >> > Thanks. I followed the script and configured both ports to Ethernet. >> > >> > ibstat info below: >> > >> > CA 'mlx4_0' >> > CA type: MT4099 >> > Number of ports: 2 >> > Firmware version: 2.40.7000 >> > Hardware version: 1 >> > Node GUID: 0xe41d2d0300024480 >> > System image GUID: 0xe41d2d0300024483 >> > Port 1: >> > State: Down >> > Physical state: Disabled >> > Rate: 10 >> > Base lid: 0 >> > LMC: 0 >> > SM lid: 0 >> > Capability mask: 0x04010000 >> > Port GUID: 0xe61d2dfffe024481 >> > Link layer: Ethernet >> > Port 2: >> > State: Active >> > Physical state: LinkUp >> > Rate: 40 >> > Base lid: 0 >> > LMC: 0 >> > SM lid: 0 >> > Capability mask: 0x04010000 >> > Port GUID: 0xe61d2dfffe024482 >> > Link layer: Ethernet >> > >> > I also change the default value from n to CONFIG_RTE_LIBRTE_MLX4_PMD= =3Dy. >> > Recompiled DPDK and it all worked. >> > >> > And Below is my ifconfig info: >> > >> > ens2 Link encap:Ethernet HWaddr e4:1d:2d:02:44:81 >> > UP BROADCAST MULTICAST MTU:1500 Metric:1 >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> > collisions:0 txqueuelen:1000 >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >> > >> > ens2d1 Link encap:Ethernet HWaddr e4:1d:2d:02:44:82 >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> > RX packets:126 errors:0 dropped:0 overruns:0 frame:0 >> > TX packets:220 errors:0 dropped:0 overruns:0 carrier:0 >> > collisions:0 txqueuelen:1000 >> > RX bytes:29494 (29.4 KB) TX bytes:38063 (38.0 KB) >> > >> > >> > Since I assume that I don't have to bind the device manually, I execut= ed >> > ./dpdk-devbind.py --status with the following info popped out: >> > >> > Network devices using kernel driver >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> > 0000:03:00.0 'Ethernet Connection X552/X557-AT 10GBASE-T 15ad' if=3Den= o3 >> > drv=3Dixgbe unused=3D >> > 0000:03:00.1 'Ethernet Connection X552/X557-AT 10GBASE-T 15ad' if=3Den= o4 >> > drv=3Dixgbe unused=3D >> > 0000:04:00.0 'MT27500 Family [ConnectX-3] 1003' if=3Dens2,ens2d1 >> > drv=3Dmlx4_core unused=3D >> > 0000:05:00.0 'I350 Gigabit Network Connection 1521' if=3Deno1 drv=3Dig= b >> unused=3D >> > 0000:05:00.1 'I350 Gigabit Network Connection 1521' if=3Deno2 drv=3Dig= b >> unused=3D >> > *Active* >> > >> > >> > With these configurations, I run commands in order but still had the >> same >> > result: >> > >> > $test: modprobe -a ib_uverbs mlx4_en mlx4_core mlx4_ib >> > $test: /etc/init.d/openibd restart >> > $test: sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 0-6 -n 1 >> --proc-type >> > auto --file-prefix pg -b 03:00.0 -b 03:00.1 -b 05:00.0 -b 05:00.1 -- -= P >> -m >> > "[1-5.0]" >> > >> > >> >>>> Packet Burst 32, RX Desc 512, TX Desc 1024, mbufs/port 8192, mbuf >> cache >> > 1024 >> > !PANIC!: *** Did not find any ports to use *** >> > PANIC in pktgen_config_ports(): >> > *** Did not find any ports to use ***6: >> > [./app/x86_64-native-linuxapp-gcc/pktgen(_start+0x29) >> > [0x447239]] >> > 5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) >> > [0x7f0ec0931830]] >> > 4: [./app/x86_64-native-linuxapp-gcc/pktgen(main+0x580) [0x444710]] >> > 3: [./app/x86_64-native-linuxapp-gcc/pktgen(pktgen_config_ports+0x1614= ) >> > [0x46e324]] >> > 2: [./app/x86_64-native-linuxapp-gcc/pktgen(__rte_panic+0xc3) >> [0x43ead4]] >> > 1: [./app/x86_64-native-linuxapp-gcc/pktgen(rte_dump_stack+0x2b) >> [0x4b7ecb]] >> > Aborted (core dumped) >> > >> > >> > >> > Is there anything I've done wrong here? >> > >> > >> > Thanks >> > >> > >> > P.T >> > >> > >> >> You can use -w to specify ConnectX-3 PCI address, you don't need to >> use -b >> >> >> >> Make sure you compile DPDK with mlx4 PMD, it is not enabled by defaul= t >> >> >> >> Best Regards, >> >> Olga >> >> >> >> >> >> >> > >> > -- >> > P.T >> > > > > -- > P.T > > > --=20 P.T