From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yk0-f172.google.com (mail-yk0-f172.google.com [209.85.160.172]) by dpdk.org (Postfix) with ESMTP id DAD1C2C50 for ; Mon, 7 Mar 2016 18:14:12 +0100 (CET) Received: by mail-yk0-f172.google.com with SMTP id h197so30556753yke.2 for ; Mon, 07 Mar 2016 09:14:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=iUdD/z2z+FhQb+4EMpkIMXERgCAA1eZZHVnJZ0o+3F0=; b=vlDQmqgqYohuNeWYRosa2/rkhNplXq4xut3TNJ763opmEKZylzU8wFlcGbo8XqFgMq 4afrvLKnOWVGJGjfUxcqIg/JsC2l5Ogw3BIwuSuELgSENrQ2x89ywPVyNhjTi3XeWbe6 xR5F8jIUOhoZ35s72hsGyQgAwF/gJbh2BxD+KdIuHFLYgAZ7lZlOePkf5hH02HaOv2H8 2HvaeqlVNdTr2/p+xPke4tZwA5Gk73ljmmnO6IR8uOGThfaM7Ru+i/n3+o8eQ9x/dCgT IPeQtmraHvjVLotvspq7WcPMcfVcN9jrREAdrQV/g6uExYg20O2ZSGm8BKmrShcyj5iZ 1jOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=iUdD/z2z+FhQb+4EMpkIMXERgCAA1eZZHVnJZ0o+3F0=; b=gO4jxfCqBSyBGl5na7niNUnocats+pXbH5cKIB5xAFaUf6uHxXQDrGo9cLkLizoxr1 9P7NcblARrfQNljKOcPw0Dwj2WGL0ETeNk/Re9KuI3Hvcli4E8ABic1l5hoFnB2WKdPA maklc2U70pO90uBsPkJygfng7K/e6jta6/wWHSR6flLMKyvDSUNt2UxosV/L8s6nz+mg H0Hd5ZYxujVOpYDbhE4CU2JRQJB7SnU3eAYSyV40ql2NKnZYGFP5K5C8qaQC20vZjvxZ 2W5ysqlo9cNuVemVXxr3PSQZQx1o6WLvtqhZs8aclhunhe+KtB6n20CC8pxPphOtEAo1 uBYw== X-Gm-Message-State: AD7BkJJS3cr8HvA9q5WQLuFnTlZ8MIxyU9ABXTwMV8KXg0dVGRPFCfySJV9EpDMiS2rzTQ== X-Received: by 10.37.42.143 with SMTP id q137mr13074273ybq.46.1457370852359; Mon, 07 Mar 2016 09:14:12 -0800 (PST) Received: from mail-yk0-f169.google.com (mail-yk0-f169.google.com. [209.85.160.169]) by smtp.gmail.com with ESMTPSA id f126sm12460066ywc.22.2016.03.07.09.14.11 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 07 Mar 2016 09:14:11 -0800 (PST) Received: by mail-yk0-f169.google.com with SMTP id z7so12737438yka.0 for ; Mon, 07 Mar 2016 09:14:11 -0800 (PST) MIME-Version: 1.0 X-Received: by 10.37.33.84 with SMTP id h81mr14108700ybh.166.1457370851005; Mon, 07 Mar 2016 09:14:11 -0800 (PST) Received: by 10.37.115.200 with HTTP; Mon, 7 Mar 2016 09:14:10 -0800 (PST) In-Reply-To: References: <116CFAF2-CD73-41C4-96C5-D748E90205F7@intel.com> <01E1F718-CF3F-4692-B94A-B1FB19F55F5C@intel.com> Date: Tue, 8 Mar 2016 01:14:10 +0800 X-Gmail-Original-Message-ID: Message-ID: From: Royce Niu To: "Wiles, Keith" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: "users@dpdk.org" Subject: Re: [dpdk-users] Performance Problem of DPDK pkt-gen X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Mar 2016 17:14:13 -0000 You are so cool. Thanks! The problem is how to disable pause frame now. I searched and tried sudo ethtool -A eth4 autoneg off tx off rx off sudo ethtool -A eth4 autoneg off tx off rx off sudo ethtool -A eth5 autoneg off tx off rx off sudo ethtool -A eth5 autoneg off tx off rx off sudo ethtool -K eth4 tso off gro off gso off tx off rx off sudo ethtool -K eth5 tso off gro off gso off tx off rx off But, it is not working... I think it is not working in DPDK. On Tue, Mar 8, 2016 at 12:41 AM, Wiles, Keith wrote: > From: Royce Niu > Date: Monday, March 7, 2016 at 10:35 AM > To: Keith Wiles > Cc: Royce Niu , "users@dpdk.org" > Subject: Re: [dpdk-users] Performance Problem of DPDK pkt-gen > > > >Dear Keith, > > > > > >I started both PCs pkt-gen and sending 64K packets at the same time: > > > >PC1 PC2 > >NIC0 -> NIC0 ( 12Mpps) > > > >NIC1 <- NIC1 ( 12Mpps ) > > > > > >Although it is less than 14Mpps, but it is ok? > > > >So, the problem is the pause packet? > > > > > > > > I am guessing that is that problem, maybe someone else has a better idea. > I assume the problem is with L2FWD code having to move packets between numa > zones or across the QPI bus. In Pktgen each port has its own set of mbufs > and does not need to interact between sockets. > > > > > >On Tue, Mar 8, 2016 at 12:25 AM, Wiles, Keith > > wrote: > > > >From: Royce Niu > >Date: Monday, March 7, 2016 at 10:01 AM > >To: Keith Wiles > >Cc: Royce Niu , "users@dpdk.org" > >Subject: Re: [dpdk-users] Performance Problem of DPDK pkt-gen > > > > > >>Dear Keith, > >> > >>I am doing the measurement works. The two PCs are same in > software/physical configuration with two 10Gb/s link. > >> > >>The L2FWD actually is in a virtual machine in L2FWD pc. I don't mind > packet drops if L2FWD in VM on L2FWD do its best. > >> > >>Is there an solution cancel rate limiting in linux/dpdk? So, I can know > how many packets in lost in 14.4Mpps environments. > > > >Sorry, I am not sure about how to turn off pause frames. But you should > try running Pktgen on both machines to verify the problem. > >> > >> > >> > >> > >> > >>On Mon, Mar 7, 2016 at 11:49 PM, Wiles, Keith > >> wrote: > >> > >>From: Royce Niu > >>Date: Monday, March 7, 2016 at 9:41 AM > >>To: Keith Wiles > >>Cc: Royce Niu , "users@dpdk.org" > >>Subject: Re: [dpdk-users] Performance Problem of DPDK pkt-gen > >> > >> > >> > >>Yes. > >> > >>The problem is the sending rate is not 14Mpps when L2FWD is working. > >> > >>When L2FWD is working, the sending rate is about 4Mpps, instead of > 14Mpps. When I close the L2FWD pc, the sending rate recovers to 14Mpps... > >> > >>I think there is something wrong with my Pktgen PC. Could you help me > check my commands? Or is there anything wrong? > >> > >> > >> > >> > >> > >> > >>I do not think Pktgen has a problem or the PC on which it runs. I expect > the problem is the second PC is not able to keep up with the RX rate and is > sending pause packets to the TX machine. The pause packets will reduce the > TX rate on the Pktgen PC. > >> > >>Try running Pktgen on the L2FWD PC and see if the rate drops. If the > rate does not drop then the second PC is sending pause frame back to the > first PC to do rate limiting on the TX side. > >> > >> > >> > >> > >> > >> > >>On Mon, Mar 7, 2016 at 11:35 PM, Wiles, Keith > >> wrote: > >> > >>From: Royce Niu > >>Date: Monday, March 7, 2016 at 9:30 AM > >>To: Keith Wiles > >>Cc: Royce Niu , "users@dpdk.org" > >>Subject: Re: [dpdk-users] Performance Problem of DPDK pkt-gen > >> > >> > >> > >>Hi, Keith > >> > >>Maybe, since I didn't configure the CPU affinity in the L2FWD pc on > purpose so far. > >> > >>But, my question is the first PC have a poor sending rate when L2FWD is > working in second PC. > >> > >>You mean the problem is related to L2FWD? > >> > >> > >> > >> > >> > >> > >>The sending rate of the Pktgen PC should be constant, but the forwarding > rate of the second PC maybe the problem because sending packets received on > one socket and then being send by another socket is a problem as the QPI > bus between sockets is not as > >> fast. > >> > >> > >> > >>On Mon, Mar 7, 2016 at 11:26 PM, Wiles, Keith > >> wrote: > >> > >>>Dear all, > >>> > >>>I am using an server with 4 cpus (4 x 8 core CPUs with HT) and NICs > (X520). > >>> > >>>When I use pkt-gen on NIC1 or NIC2, the speed of generating 64Byte is > >>>14Mpps. > >>> > >>>If I generating both on NIC1 and NIC2, the speed of generating 64Byte > on > >>>both are more than 13Mpps. > >>> > >>>However, I use same configuration PC (with DPDK L2FWD) to bridge NIC1 > and > >>>NIC2, so I can generate packet on NIC1 and receive these packets on > NIC2 in > >>>pkt-gen. The speed of generating is decreased to 4Mpps and the receive > rate > >>>is 3Mpps. > >> > >>I am not sure how you configured the second PC for L2FWD, but I suspect > the L2FWD is having to receive packets on socket 0 and send the packets on > socket 1, this means the QPI bus gets involved here. Is this the case? > >> > >>> > >>> > >>>I want to know why generating speed is slower than the situation without > >>>the bridge of NIC1 and NIC2? how to solve these problem? > >>> > >>>The detailed information is as following. > >>> > >>>sudo sysctl vm.nr_hugepages=4096 > >>>echo 1024 | sudo tee > >>>/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages > >>>echo 1024 | sudo tee > >>>/sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages > >>>echo 1024 | sudo tee > >>>/sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages > >>>echo 1024 | sudo tee > >>>/sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages > >>> > >>> > >>>sudo mkdir -p /dev/hugepages > >>>sudo mount -t hugetlbfs nodev /dev/hugepages > >>> > >>> > >>>sudo dpdk-2.2.0/tools/dpdk_nic_bind.py --status > >>>sudo modprobe uio > >>>sudo insmod dpdk-2.2.0/build/kmod/igb_uio.ko > >>> > >>>sudo dpdk-2.2.0/tools/dpdk_nic_bind.py -b igb_uio 04:00.0 04:00.1 > >>>sudo dpdk-2.2.0/tools/dpdk_nic_bind.py --status > >>> > >>>cd pktgen-2.9.12/ > >>> > >>>sudo app/build/pktgen -c 0x1f -n 3 --proc-type auto --socket-mem > >>>128,128,128,128 -- -P -m "[1:3].0, [2:4].1" -f test/set_seq.pkt > >>> > >>>I tried to change -m, but, sometime there is no packet generated by > >>>pkt-gen. > >>> > >>> > >>>The core map is : > >>> > >>>EAL: Detected lcore 0 as core 0 on socket 0 > >>>EAL: Detected lcore 1 as core 0 on socket 1 > >>>EAL: Detected lcore 2 as core 0 on socket 2 > >>>EAL: Detected lcore 3 as core 0 on socket 3 > >>>EAL: Detected lcore 4 as core 1 on socket 0 > >>>EAL: Detected lcore 5 as core 1 on socket 1 > >>>EAL: Detected lcore 6 as core 1 on socket 2 > >>>EAL: Detected lcore 7 as core 1 on socket 3 > >>>EAL: Detected lcore 8 as core 2 on socket 0 > >>>EAL: Detected lcore 9 as core 2 on socket 1 > >>>EAL: Detected lcore 10 as core 2 on socket 2 > >>>EAL: Detected lcore 11 as core 2 on socket 3 > >>>EAL: Detected lcore 12 as core 3 on socket 0 > >>>EAL: Detected lcore 13 as core 3 on socket 1 > >>>EAL: Detected lcore 14 as core 3 on socket 2 > >>>EAL: Detected lcore 15 as core 3 on socket 3 > >>>EAL: Detected lcore 16 as core 4 on socket 0 > >>>EAL: Detected lcore 17 as core 4 on socket 1 > >>>EAL: Detected lcore 18 as core 4 on socket 2 > >>>EAL: Detected lcore 19 as core 4 on socket 3 > >>>EAL: Detected lcore 20 as core 5 on socket 0 > >>>EAL: Detected lcore 21 as core 5 on socket 1 > >>>EAL: Detected lcore 22 as core 5 on socket 2 > >>>EAL: Detected lcore 23 as core 5 on socket 3 > >>>EAL: Detected lcore 24 as core 6 on socket 0 > >>>EAL: Detected lcore 25 as core 6 on socket 1 > >>>EAL: Detected lcore 26 as core 6 on socket 2 > >>>EAL: Detected lcore 27 as core 6 on socket 3 > >>>EAL: Detected lcore 28 as core 7 on socket 0 > >>>EAL: Detected lcore 29 as core 7 on socket 1 > >>>EAL: Detected lcore 30 as core 7 on socket 2 > >>>EAL: Detected lcore 31 as core 7 on socket 3 > >>>EAL: Detected lcore 32 as core 0 on socket 0 > >>>EAL: Detected lcore 33 as core 0 on socket 1 > >>>EAL: Detected lcore 34 as core 0 on socket 2 > >>>EAL: Detected lcore 35 as core 0 on socket 3 > >>>EAL: Detected lcore 36 as core 1 on socket 0 > >>>EAL: Detected lcore 37 as core 1 on socket 1 > >>>EAL: Detected lcore 38 as core 1 on socket 2 > >>>EAL: Detected lcore 39 as core 1 on socket 3 > >>>EAL: Detected lcore 40 as core 2 on socket 0 > >>>EAL: Detected lcore 41 as core 2 on socket 1 > >>>EAL: Detected lcore 42 as core 2 on socket 2 > >>>EAL: Detected lcore 43 as core 2 on socket 3 > >>>EAL: Detected lcore 44 as core 3 on socket 0 > >>>EAL: Detected lcore 45 as core 3 on socket 1 > >>>EAL: Detected lcore 46 as core 3 on socket 2 > >>>EAL: Detected lcore 47 as core 3 on socket 3 > >>>EAL: Detected lcore 48 as core 4 on socket 0 > >>>EAL: Detected lcore 49 as core 4 on socket 1 > >>>EAL: Detected lcore 50 as core 4 on socket 2 > >>>EAL: Detected lcore 51 as core 4 on socket 3 > >>>EAL: Detected lcore 52 as core 5 on socket 0 > >>>EAL: Detected lcore 53 as core 5 on socket 1 > >>>EAL: Detected lcore 54 as core 5 on socket 2 > >>>EAL: Detected lcore 55 as core 5 on socket 3 > >>>EAL: Detected lcore 56 as core 6 on socket 0 > >>>EAL: Detected lcore 57 as core 6 on socket 1 > >>>EAL: Detected lcore 58 as core 6 on socket 2 > >>>EAL: Detected lcore 59 as core 6 on socket 3 > >>>EAL: Detected lcore 60 as core 7 on socket 0 > >>>EAL: Detected lcore 61 as core 7 on socket 1 > >>>EAL: Detected lcore 62 as core 7 on socket 2 > >>>EAL: Detected lcore 63 as core 7 on socket 3 > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>>-- > >>>Regards, > >>> > >>>Royce Niu > >>> > >> > >> > >> > >> > >>Regards, > >>Keith > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >>-- > >>Regards, > >> > >>Royce Niu > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >>Regards, > >>Keith > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >>-- > >>Regards, > >> > >>Royce Niu > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >>Regards, > >>Keith > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >>-- > >>Regards, > >> > >>Royce Niu > >> > >> > >> > >> > >> > >> > >> > > > > > > > > > >Regards, > >Keith > > > > > > > > > > > > > > > > > >-- > >Regards, > > > >Royce Niu > > > > > > > > > > > > > > > > > Regards, > Keith > > > -- Regards, Royce Niu