From: "Venkatesan, Venky" <venky.venkatesan@intel.com>
To: dev@dpdk.org
Subject: Re: [dpdk-dev] IXGBE RX packet loss with 5+ cores
Date: Tue, 13 Oct 2015 08:34:48 -0700 [thread overview]
Message-ID: <561D2498.1060707@intel.com> (raw)
In-Reply-To: <D2428901.CE78%rsanford@akamai.com>
On 10/13/2015 7:47 AM, Sanford, Robert wrote:
>>>> [Robert:]
>>>> 1. The 82599 device supports up to 128 queues. Why do we see trouble
>>>> with as few as 5 queues? What could limit the system (and one port
>>>> controlled by 5+ cores) from receiving at line-rate without loss?
>>>>
>>>> 2. As far as we can tell, the RX path only touches the device
>>>> registers when it updates a Receive Descriptor Tail register (RDT[n]),
>>>> roughly every rx_free_thresh packets. Is there a big difference
>>>> between one core doing this and N cores doing it 1/N as often?
>>> [Stephen:]
>>> As you add cores, there is more traffic on the PCI bus from each core
>>> polling. There is a fix number of PCI bus transactions per second
>>> possible.
>>> Each core is increasing the number of useless (empty) transactions.
>> [Bruce:]
>> The polling for packets by the core should not be using PCI bandwidth
>> directly,
>> as the ixgbe driver (and other drivers) check for the DD bit being set on
>> the
>> descriptor in memory/cache.
> I was preparing to reply with the same point.
>
>>> [Stephen:] Why do you think adding more cores will help?
> We're using run-to-completion and sometimes spend too many cycles per pkt.
> We realize that we need to move to io+workers model, but wanted a better
> understanding of the dynamics involved here.
>
>> [Bruce:] However, using an increased number of queues can
>> use PCI bandwidth in other ways, for instance, with more queues you
>> reduce the
>> amount of descriptor coalescing that can be done by the NICs, so that
>> instead of
>> having a single transaction of 4 descriptors to one queue, the NIC may
>> instead
>> have to do 4 transactions each writing 1 descriptor to 4 different
>> queues. This
>> is possibly why sending all traffic to a single queue works ok - the
>> polling on
>> the other queues is still being done, but has little effect.
> Brilliant! This idea did not occur to me.
To add a little more detail - this ends up being both a bandwidth and a
transaction bottleneck. Not only do you add an increased transaction
count, you also add a huge amount of bandwidth overhead (each 16 byte
descriptor is preceded by a PCI-E TLP which is about the same size). So
what ends up happening in the case where the incoming packets are
bifurcated to different queues (1 per queue) is that you have 2x the
number of transactions (1 for the packet and one for the descriptor) and
then we essentially double the bandwidth used because you now have the
TLP overhead per descriptor write.
There is a second issue that also pops up when coalescing breaks down -
testpmd essentially in iofwd mode simply transmits the number of packets
it receives (i.e. Rx (n) -> Tx (n)). This means that the transmit side
also suffers from writing one descriptor at a time for output (i.e. when
the NIC pulls a descriptor cache line to transmit, it finds 1 valid
descriptor). When a second descriptor is transmitted on the same it will
again pull and find only one valid descriptor. That is another 2x
increase in transaction count as well as PCI-E TLP overhead.
The third hit actually comes from the transmit side when transmitting
one packet at a time. The last part of the transmit process is a MMIO
write to the tail pointer. This is a costly operation (since it is a
un-cacheable memory operation) in terms of cycles, not to mention again
with heavy PCI-E overhead (TLP + 4 byte write) and increased transaction
counts on PCI-E.
Hope that explains all the touch-points as to why you see the drop off
in performance you see.
>
>
>
> --
> Thanks guys,
> Robert
>
next prev parent reply other threads:[~2015-10-13 15:35 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-10-13 2:57 Sanford, Robert
2015-10-13 5:18 ` Stephen Hemminger
2015-10-13 13:59 ` Bruce Richardson
2015-10-13 14:47 ` Sanford, Robert
2015-10-13 15:34 ` Venkatesan, Venky [this message]
2018-11-01 6:42 ` Saber Rezvani
2015-10-13 20:24 ` Alexander Duyck
2015-10-14 9:29 ` Bruce Richardson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=561D2498.1060707@intel.com \
--to=venky.venkatesan@intel.com \
--cc=dev@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).