From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ee0-f46.google.com (mail-ee0-f46.google.com [74.125.83.46]) by dpdk.org (Postfix) with ESMTP id 7843F30E for ; Wed, 29 May 2013 16:07:28 +0200 (CEST) Received: by mail-ee0-f46.google.com with SMTP id e49so5444077eek.33 for ; Wed, 29 May 2013 07:07:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding :x-gm-message-state; bh=8SSKCfuXqKWixhNDahPGWADg1Z2oSOBQ3mKw614w7qw=; b=ecMiLUEqBZbqtcGYJZaEhw/GVI2zVm3NhdM8w5/XWR8Ewf2oAWJHxNgkP7rqapiAI/ /yXVnmPfTo8c1c5iYfuhZu/MBWX4dYtJtwgykRzk/vJL/pwJN1z4GCpR/VEt0kHVWcFJ nxxKxRxGzPcloXo9wA5a9RcYnLCqLe1acbbdy7L98QgaJCkNz9O3ztC14lKaqS4yT0qK ssknuzE64YyN7LLmCgKRuquD7V+xBO05C6dPNi33gr1wrQ7M3SRggqy7R2osN308Ibbu EUnWCk0Qg0CiU0qpoMhkPZyRJMISOpDz4e7FYnEh0jbeKQ6pCK5CRAsAK1ZsKnV9Y5Yc XaIQ== X-Received: by 10.14.110.194 with SMTP id u42mr3573350eeg.117.1369836454242; Wed, 29 May 2013 07:07:34 -0700 (PDT) Received: from [10.16.0.97] (6wind.net2.nerim.net. [213.41.180.237]) by mx.google.com with ESMTPSA id g7sm54442582eew.15.2013.05.29.07.07.32 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 29 May 2013 07:07:33 -0700 (PDT) Message-ID: <51A60BA0.7000700@6wind.com> Date: Wed, 29 May 2013 16:07:28 +0200 From: Damien Millescamps User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.15) Gecko/20110303 Lightning/1.0b2 Thunderbird/3.1.9 MIME-Version: 1.0 To: "dev@dpdk.org" References: <519F74F6.3000903@mahan.org> <201305241641.38896.thomas.monjalon@6wind.com> <201305241745.25844.thomas.monjalon@6wind.com> <5BBC85C7-B39F-4200-AB7B-CD5464BDA431@mahan.org> <51A10FC3.5050703@6wind.com> <51A12618.3040509@6wind.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Gm-Message-State: ALoCoQn2PUvJ4UC3TO6AO66u5hPgvSBWe7Ct4oLO+grAMh5AG/6fjCc0Ri1oiibwJPZUO6VX7lsz Subject: Re: [dpdk-dev] Best example for showing throughput? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 May 2013 14:07:28 -0000 On 05/28/2013 09:15 PM, Patrick Mahan wrote: > So the overhead cost is almost 70%? > > Can this ever do line rate? Under what conditions? It has been my experience that the industry standard is testing throughput using these 64 byte packets. This overhead can actually be explained considering the PCIe 2.1[1] standard and 82599 Specifications[2]. To sum up, for each packet the adapter needs to first send a read request on a 16 Bytes packet descriptor (cf. [2]), to which it will receive a read answer. Then the adapter must issue either a read or write request to the packet physical address for the size of the packet. The frame format for PCIe read and write request is composed with a Start of frame, a Sequence Number, a Header, the Data, an LRC and an End of frame (cf. [1]). The overhead we are talking about here is more than 16 Bytes per PCIe message. In addition to that, the PCIe physical layer uses a 10bits per bytes encoding, thus adding to the overhead. Now if you apply this to the 64 Bytes packet, you should notice that the overhead is way above 70% (4 messages plus descriptor and data size times 10b/8b encoding which should be around 83% if I didn't miss anything). However, if we end up with a limited overhead it is because the 82599 implements thresholds in order to be able to batch the packet descriptor reading / writing back (cf. [2] WTHRESH for example) thus reducing the overhead to a little more than 70% with the default DPDK parameters. You can achieve line-rate for 64 Bytes packets on each port independently. When using both port simultaneously you can achieve line-rate using packet size above 64Bytes. In the post to which I redirected you, Alexander talked about 256Bytes packets. But if you take the time to compute the total throughput needed on the PCIe as a function of the packet size, you'll probably end up with a lower minimum packet size than 256B to achieve line-rate simultaneously on both port. [1] http://www.pcisig.com/members/downloads/specifications/pciexpress/PCI_Express_Base_r2_1_04Mar09.pdf [2] http://www.intel.com/content/dam/doc/datasheet/82599-10-gbe-controller-datasheet.pdf -- Damien Millescamps