From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-la0-x230.google.com (mail-la0-x230.google.com [IPv6:2a00:1450:4010:c03::230]) by dpdk.org (Postfix) with ESMTP id EE55C1F3 for ; Thu, 19 Sep 2013 09:50:01 +0200 (CEST) Received: by mail-la0-f48.google.com with SMTP id er20so6574833lab.35 for ; Thu, 19 Sep 2013 00:50:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=+tQquqvSS/gs9uEM6aNFswI2ZrrS4qDcpgnpnfnlVyU=; b=KvbzkrApEirc1OAyc4qR2kFHdAElsuIClqt9a3vanGwChtjOwGqakoI83m+L+WxbSH tlXXOFKVXjgz9I7XHpm7kY97GvJEzR6pjF77T2GQzkVec+sRhXmWkFEUOyqXdse+1INS OUgFzGIyew/XxB/2VHZ2pfba0/HIhMlEghprDyl/vVkducxMew6Gb1LSwZS7lIoG+o9B bAGqTYmM4u/QrAsELje6BwptptfCAviTB5iQNR0S5rIljwKZkaTu6z0YdEnlY0KWa9jG FxyNxfkOFljE3WXhRfDkPXp4JpguvmV8qhH10igAgPBXDvtYWyeZVwO/o0f72EL6SH1p +ROw== X-Received: by 10.112.72.229 with SMTP id g5mr788734lbv.10.1379577035199; Thu, 19 Sep 2013 00:50:35 -0700 (PDT) Received: from [192.168.15.100] ([91.246.87.40]) by mx.google.com with ESMTPSA id k6sm2846940lae.9.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 19 Sep 2013 00:50:34 -0700 (PDT) Message-ID: <523AACC9.8010304@gmail.com> Date: Thu, 19 Sep 2013 11:50:33 +0400 From: Dmitry Vyal User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130623 Thunderbird/17.0.7 MIME-Version: 1.0 To: dev@dpdk.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Sep 2013 07:50:02 -0000 Good day everyone, While working on IP packet defragmenting I had to enlarge mempool size. I did this to provide large enough time window for assembling a fragment sequence. Unfortunately, I got a performance regression: if I enlarge mempool size from 2**12 to 2**20 MBufs, packet performance for not fragmented packets drops from ~8.5mpps to ~5.5mpps for single core. I made only a single measure, so the data are noisy, but the trend is evident: SIZE 4096 - 8.47mpps SIZE 8192 - 8.26mpps SIZE 16384 - 8.29mpps SIZE 32768 - 8.31mpps SIZE 65536 - 8.12mpps SIZE 131072 - 7.93mpps SIZE 262144 - 6.22mpps SIZE 524288 - 5.72mpps SIZE 1048576 - 5.63mpps And I need even larger sizes. I want to ask for an advice, how best to tackle this? One way I'm thinking about is to make two mempools, one large for fragments (we may accumulate a big number of them) and one small for full packets, we just forward them burst by burst. Is it possible to configure RSS to distribute packets between queues according to this scheme? Perhaps, there are better ways? Thanks, Dmitry