From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ob0-f169.google.com (mail-ob0-f169.google.com [209.85.214.169]) by dpdk.org (Postfix) with ESMTP id D20A0B346 for ; Thu, 10 Jul 2014 10:39:55 +0200 (CEST) Received: by mail-ob0-f169.google.com with SMTP id nu7so2413335obb.28 for ; Thu, 10 Jul 2014 01:40:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=Uu+bhPn0oEszD0AGNcMrEq4H7BXHfUn4xaUJSYcjjrI=; b=PJpR5j7bdVjtIVKkx8tTR28Jpi4B6nVAQXvLobKdLASoU7v1FYMcd8wNKZWkck5v9G ofdw13ZHJwbLFuAfo9a/yaNTJCXdMTa0vloBu9S87TdTes02aK6zl6qjzY6lcVJ8d98G usDrn/O5Edu/c7Kf+oXP/yvb02khOAZwCG7fL8zZKS0Ua+xB4xw4HoNxcLDD3RFcs98A 3kchkHyo800hNO+uBhbh6an/IIkX7rsD/nCxnfwDipIlC52cvNNyCoHI3HPp7Iv3/Nmr V+2T/8rR+wT603NmKkbSFwNsICEkOr4jFseV74Vmo03UCjdYYMqEPjcYt++xyD0JAWrK ZWnQ== X-Gm-Message-State: ALoCoQnJX8U7D1FK7kCm3egTcm864tEv42oOhptNUJyp8C/7RbbGHO1bYBGS52/nCeHBSl8AXUO9 MIME-Version: 1.0 X-Received: by 10.60.115.67 with SMTP id jm3mr52615981oeb.8.1404981617189; Thu, 10 Jul 2014 01:40:17 -0700 (PDT) Received: by 10.202.65.131 with HTTP; Thu, 10 Jul 2014 01:40:17 -0700 (PDT) In-Reply-To: <53BE495E.2000900@cas-well.com> References: <53BE495E.2000900@cas-well.com> Date: Thu, 10 Jul 2014 11:40:17 +0300 Message-ID: From: Alex Markuze To: Zachary.Jen@cas-well.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: "dev@dpdk.org" , Alan.Yu@cas-well.com Subject: Re: [dpdk-dev] DPDK Performance issue with l2fwd X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 08:39:56 -0000 Hi Zachary, Your issue may be with the PCI-e 3, with 16 lanes Each slot is limited to 128Gb/s[3]. Now, AFAIK[1] the CPU is connected to the I/O with a single PCI-E slot. Several thoughts that may help you: 1. You can figure out the max b/w by running netsurf over the kernel interfaces (w/o DPDK). Each CPU can handle the Netperf and the Completion interrupts with grace (packets of 64K and all offloads on) for 10Gb nics. With more then 10 Nics I would disable the IRQ balancer and make sure interrupts are spread evenly by setting the IRQ affinity manually [2]. As long as you have a physical core(NO hyperthreading) per NIC port you can figure out the MAX B/W you can get with all the nics. 2. You can try using (If available to you , obviously) 40Gb and 56Gb Nics (Mellanox), In this case for each Netperf flow you will need to separate each Netperf Stream and the interrupts to different Cores to Reach wire speed as long as both cores are on the same NUMA node(lscpu). Hope this helps. [1] http://komposter.com.ua/documents/PCI_Express_Base_Specification_Revision_3= .0.pdf [2] http://h50146.www5.hp.com/products/software/oe/linux/mainstream/support/whi= tepaper/pdfs/4AA4-9294ENW.pdf [3]http://en.wikipedia.org/wiki/PCI_Express#PCI_Express_3.x On Thu, Jul 10, 2014 at 11:07 AM, wrote: > Hey Guys, > > Recently, I have used l2fwd to test 160G (82599 10G * 16 ports), but I > got a strange pheromone in my test. > > When I used 12 ports to test the performance of l2fwd, it can work fine > and achieve 120G. > But it got abnormal when I using over than 12 port. Part of ports seems > something wrong and no any Tx/Rx. > Has anyone know about this? > > My testing Environment. > 1. E5-2658 v2 (10 cores) * 2 > > http://ark.intel.com/zh-tw/products/76160/Intel-Xeon-Processor-E5-2658-v2= -25M-Cache-2_40-GHz > 2. one core handle one port. (In order to get best performance.) > 3. No any QPI crossing issue. > 4. l2fwd parameters > 4.1 -c 0xF0FF -- -P 0xF00FF =3D> 120G get! > 4.2 -c 0xFF0FF -- -P 0xFF0FF =3D> Failed! Only first 10 ports can > work well. > 4.3 -c 0x3F3FF -- -P 0x3F3FF =3D> Failed! Only first 10 ports can > work well. > > BTW, I have tried lots of parameter sets and if I set the ports number > over than 12 ports, it only first 10 ports got work. > Else, everything got well. > > Can anyone help me to solve the issue? Or DPDK only can set less equal > than 12 ports? > Or DPDK max throughput is 120G? > > =E6=9C=AC=E4=BF=A1=E4=BB=B6=E5=8F=AF=E8=83=BD=E5=8C=85=E5=90=AB=E7=91=9E= =E7=A5=BA=E9=9B=BB=E9=80=9A=E6=A9=9F=E5=AF=86=E8=B3=87=E8=A8=8A=EF=BC=8C=E9= =9D=9E=E6=8C=87=E5=AE=9A=E4=B9=8B=E6=94=B6=E4=BB=B6=E8=80=85=EF=BC=8C=E8=AB= =8B=E5=8B=BF=E4=BD=BF=E7=94=A8=E6=88=96=E6=8F=AD=E9=9C=B2=E6=9C=AC=E4=BF=A1= =E4=BB=B6=E5=85=A7=E5=AE=B9=EF=BC=8C=E4=B8=A6=E8=AB=8B=E9=8A=B7=E6=AF=80=E6= =AD=A4=E4=BF=A1=E4=BB=B6=E3=80=82 This email may contain > confidential information. Please do not use or disclose it in any way and > delete it if you are not the intended recipient. >