DPDK patches and discussions
 help / color / mirror / Atom feed
From: Tom Barbette <barbette@kth.se>
To: "dev@dpdk.org" <dev@dpdk.org>
Subject: [dpdk-dev] Performance impact of "declaring" more CPU cores
Date: Thu, 24 Oct 2019 19:32:18 +0200	[thread overview]
Message-ID: <f46a3505-a4cd-d4a3-7429-ea5b29ca4b28@kth.se> (raw)

Hi all,

We're experiencing a very strange problem. The code of our application 
is modified to always use 8 cores. However, when running with "-l 
0-MAXCPU", with MAXCPU varying between 7 and 15 (therefore "allocating" 
more *unused* cores), the performance of the application change. It can 
be multiple Gbps at very high speed (100G) with a large number of cores, 
and it is not linear in the sense that more cores will not necessarily 
increase, nor degrade performance.

An example can be seen here : 
https://kth.box.com/s/v5v1hyidd51ebd7b8ixmcw513lfqwj4a . Note the 
errorbars (10 runs per points) show that it is not "global" variance 
that create the difference between cores. Once you end up in a "class" 
of performance, you stay there even when changing some parameters (such 
as the number of cores you actually use). From a research perspective 
this is problematic as we cannot trust results with X cores are indeed a 
certain %age better than Y core because it could be due to this problem, 
not what we "improved" in the application.

That application is a NAT and a FW, but we could observe that with other 
VNFs, at different scales. The link is saturated at 100G with real 
packets (avg size 1040B).

We have Mellanox ConnectX 5, DPDK 19.02. This could be observed on 
between Skylake and Cascade Lake machines. The example is a 
single-socket machine with 8 cores and HT. But this happened with a NUMA 
machine, or using only variation of the 18 cores of one sockets.

The only useful observation we made is that when we are in a "bad case", 
the LLC has more cache misses.

We could not really reproduce with XL710, but at 40G the problem can be 
barely seen with MLX, so that path is not very conclusive. We do not 
have other 100G NICs (hurry up Intel! :p). Similarly reproducing with 
testpmd is hard, as the problem really shows with something like 8 
cores, and we need the load to be less than the NIC wire speed, but high 
enough to observe the effect...

To rule out that problem, we hardcoded the number of cores to be used 
everywhere inside the application. So except for DPDK internals, all 
resources are allocated for 8 cores.

And that is still the best lead. Could it be that we simply get 
lucky/unlucky buffer allocations for "something per lcore" and therefore 
packet evict each others because of the limited associativity of the 
cache? At those speeds, failure of DDIO is death...
Would there be a way to fix/verify the memory allocation? base-virtaddr 
did not work though. ASLR disabled (enabled does not change anything).


Thanks for the help,

Tom

             reply	other threads:[~2019-10-24 17:32 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-24 17:32 Tom Barbette [this message]
2019-10-25 17:35 ` David Christensen
2019-10-30 17:20   ` Tom Barbette

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=f46a3505-a4cd-d4a3-7429-ea5b29ca4b28@kth.se \
    --to=barbette@kth.se \
    --cc=dev@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).