From: Stephen Hemminger <stephen@networkplumber.org>
To: bugzilla@dpdk.org
Cc: dev@dpdk.org
Subject: Re: [Bug 1137] CPU affinity set incorrectly when lcore_id 0 is not the master-lcore
Date: Wed, 30 Nov 2022 11:10:44 -0800 [thread overview]
Message-ID: <20221130111044.11850141@hermes.local> (raw)
In-Reply-To: <bug-1137-3@http.bugs.dpdk.org/>
On Wed, 30 Nov 2022 18:41:16 +0000
bugzilla@dpdk.org wrote:
> https://bugs.dpdk.org/show_bug.cgi?id=1137
>
> Bug ID: 1137
> Summary: CPU affinity set incorrectly when lcore_id 0 is not
> the master-lcore
> Product: DPDK
> Version: 22.11
> Hardware: All
> OS: Linux
> Status: UNCONFIRMED
> Severity: normal
> Priority: Normal
> Component: core
> Assignee: dev@dpdk.org
> Reporter: ltroup@cisco.com
> Target Milestone: ---
>
> Created attachment 233
> --> https://bugs.dpdk.org/attachment.cgi?id=233&action=edit
> Logs showing incorrect CPU set assignment
>
> When a range of CPUs are used (e.g. 0-3), and the master-lcore is set to
> non-zero, the CPU affinity for lcore-id 0 is set incorrectly, due to its cpuset
> being overwritten by the control-thread creation.
>
> CPU arguments passed are '-c f --master-lcore 3', to indicate that CPUs 0-3
> should be used, with the master on CPU 3. In particular, DPDK itself is
> initialized from CPU 3.
>
> When the control threads (eal-intr-thread, rte_mp_handle) are created, they are
> initialized from CPU3 - so inherit the cpuset containing just this CPU. When
> calling __rte_thread_init(), ctrl_thread_init() passes the result of
> rte_lcore_id() - but this is not yet initialized for this thread - so is set to
> 0.
>
> This means that internally, the lcore_id for the control-thread is set to 0 -
> and
> in particular, the call to thread_update_affinity() overwrites the cpuset for
> lcore_id=0 with the cpuset of CPU3:
>
> memmove(&lcore_config[lcore_id].cpuset, cpusetp,
> sizeof(rte_cpuset_t));
>
> This all occurs before the main __rte_thread_init() call for each Slave thread
> - so that the slave thread associated with lcore_id, which should be running on
> CPU0, instead has its affinity incorrectly set to CPU3.
>
> RTE logs are attached showing this behavior (and including some additional logs
> added locally to print the lcore-id and cpusets being passed).
>
> The fix for this should be to make ctrl_thread_init() more similar to
> rte_thread_register(), so that it calls eal_lcore_non_eal_allocate() to assign
> an lcore-id, then passes this to __rte_thread_init(). I have tested a fix for
> this locally to confirm.
>
Side note: using CPU 0 with DPDK is not recommended for any real application.
It is impossible to fully isolate CPU 0 and therefore you will get poor performance
and mystery latency spikes.
prev parent reply other threads:[~2022-11-30 19:10 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-11-30 18:41 bugzilla
2022-11-30 19:10 ` Stephen Hemminger [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20221130111044.11850141@hermes.local \
--to=stephen@networkplumber.org \
--cc=bugzilla@dpdk.org \
--cc=dev@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).