From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qc0-f173.google.com (mail-qc0-f173.google.com [209.85.216.173]) by dpdk.org (Postfix) with ESMTP id 2FDA312A1 for ; Wed, 7 May 2014 21:43:17 +0200 (CEST) Received: by mail-qc0-f173.google.com with SMTP id i8so1676036qcq.18 for ; Wed, 07 May 2014 12:43:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=zrzGxOwgoRj3wWOguFVsX0dGPdhvhZi03ou3s+OEvLQ=; b=gBfoIH6twsYIDGhZSWN32NWs9ChfVtlwHNRDNS6E7E0Yp7rX/hKDkVYVvysCW7ZS8Y xQw5CgwJNdwt2bNPbr7ulsfsXK5X1tqaACB7v/QL3PD1OOpLMtElMPreuPKDlTmuo4Zf 5XjZBbHPSX3LRi9X9//N/KzWKYqR6dEoHz5/WfKeO0feUm3+oYHVu5LKPmNBfTDcQZqb VTYhJSkBI4YoWJaXWq0sSGrPI9IV/VPWn9q+OFNUNFXwFbmeMnxkwBAmgBp4qlIqIWZQ OVps6kOXctsXfKpryuu9UuUeOG+9FoJ9m3lfw8z6EkXJNBlgsvdpnQAPfYddsijaadj4 svMw== X-Gm-Message-State: ALoCoQlr+0bb2tnPyNE14w7vJj6DoNdtHoxIGqC+ynZefkdJd8dg1bWsdXMsc/Pk+y7Xu6ZbWrq3 MIME-Version: 1.0 X-Received: by 10.140.92.37 with SMTP id a34mr7385635qge.91.1399491802975; Wed, 07 May 2014 12:43:22 -0700 (PDT) Received: by 10.140.51.38 with HTTP; Wed, 7 May 2014 12:43:22 -0700 (PDT) X-Originating-IP: [84.109.83.45] In-Reply-To: <5D695A7F6F10504DBD9B9187395A21797D0BB105@ORSMSX112.amr.corp.intel.com> References: <5D695A7F6F10504DBD9B9187395A21797D0BB105@ORSMSX112.amr.corp.intel.com> Date: Wed, 7 May 2014 22:43:22 +0300 Message-ID: From: Daniel Kaminsky To: "Jayakumar, Muthurajan" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] packet loss: multi-queue (RSS enabled) X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 May 2014 19:43:17 -0000 Hi Hamid, I didn't see any attachment but I think there is a solution. My first question is, when you created the mempool, how much did you defined the memory cache per CPU? In the example program this number is fairly small (32) and I think that increasing it to something much bigger (e.g. 512) will significantly improve the CPU scalability. Regards, Daniel Kaminsky On Wed, Apr 30, 2014 at 7:56 AM, Jayakumar, Muthurajan < muthurajan.jayakumar@intel.com> wrote: > Hi, > Please find the attached paper http://kfall.net/ucbpage/papers/snc.pdf > Figures 4 and 5 refers about the degradation when the # of queues are > increased. > It refers sweet spot as 2 to 4 queues. > > Have you please verified with smaller # of queues please? > > Thanks, > > > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Hamid Ramazani > Sent: Tuesday, April 29, 2014 9:42 PM > To: Thomas Monjalon; dev@dpdk.org > Subject: [dpdk-dev] packet loss: multi-queue (RSS enabled) > > Hi, > I tried a lot (more than a week) to solve the problem myself and not to > bother the list, but I didn't succeed. Maybe other people have the same > problem. > > I have a simple program attached. It is intended for simple packet > capturing; captures from interface and writes to memory, and frees the > memory in the next loop. > > I have a 10G 82599EB Intel SFI/SFP+ network interface for capturing > packets. > As you may know, this network card supports up to 128 RSS queues. > > This is just a test, so the packets being sent at 820Kpps (kilo packet per > second). Each packet is 1500B (fixed size); it is 9.16 Gbit per second. > Of course when the packet per seconds goes up and packet size goes down > (e.g. 400B per packet), it gets much worse. > > When using one queue to receive, I receive all the packets, with no loss. > When I use more than one queue (e.g. 8 queues), with each thread running > on a dedicated core, I have a considerable amount of loss. > > Please note that: > 1. The computer has 12 * 2.67GHz cores, and it does nothing else but > capturing packets. The CPU is Intel Xeon X5650. > 2. The operating system is Ubuntu 12.04.3 LTS > > Attached file includes: > main.h > main.c > Makefile > ./run.sh > > It is configured to be run with 8 queues. > > If you want to change the number of receive queues, please: > 1. in main.c, change the value assigned to nb_rx_q_of_dev to the desired > value. > 2. change core mask in run.sh file (since there is SKIP_MASTER, you should > give a core containing one more CPU than given number of queues). > > I think there might be following problems: > 1. the port configuration is not fine. > 2. freeing memory has a considerable amount of overhead, and may be I > shouldn't do that. But If I don't the pool will be full, won't be? Is there > any other way? > > Please help. > Thanks a lot in advance for your help and comments. > > All the Best, > Hamid >