DPDK patches and discussions
 help / color / mirror / Atom feed
From: Stephen Hemminger <stephen@networkplumber.org>
To: "Morten Brørup" <mb@smartsharesystems.com>
Cc: "Thomas Monjalon" <thomas@monjalon.net>, <dev@dpdk.org>,
	"David Marchand" <david.marchand@redhat.com>, <stable@dpdk.org>,
	"Anatoly Burakov" <anatoly.burakov@intel.com>,
	"Dmitry Kozlyuk" <dmitry.kozliuk@gmail.com>,
	"Narcisa Ana Maria Vasile" <navasile@linux.microsoft.com>,
	"Dmitry Malloy" <dmitrym@microsoft.com>,
	"Pallavi Kadam" <pallavi.kadam@intel.com>,
	"Tyler Retzlaff" <roretzla@linux.microsoft.com>,
	"Andrew Rybchenko" <andrew.rybchenko@oktetlabs.ru>,
	"Konstantin Ananyev" <konstantin.v.ananyev@yandex.ru>
Subject: Re: [PATCH v2] eal/unix: allow creating thread with real-time priority
Date: Wed, 25 Oct 2023 14:33:18 -0700	[thread overview]
Message-ID: <20231025143318.3be26bb3@hermes.local> (raw)
In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35E9EF8B@smartserver.smartshare.dk>

On Wed, 25 Oct 2023 19:54:06 +0200
Morten Brørup <mb@smartsharesystems.com> wrote:

> I agree with Thomas on this.
> 
> If you want the log message, please degrade it to INFO or DEBUG level. It is only relevant when chasing problems, not for normal production - and thus NOTICE is too high.

I don't want the message to be hidden.
If we get any bug reports want to be able to say "read the log, don't do that".

> Someone might build a kernel with options to keep non-dataplane threads off some dedicated CPU cores, so they can be used for guaranteed low-latency dataplane threads. We do. We don't use real-time priority, though.

This is really, hard to do. Isolated CPU's are not isolated from interrupts and other sources which end up scheduling work as kernel threads. Plus there is the behavior where kernel decides to turn a soft irq into a kernel thread, then starve
itself. Under starvation, disk corruption is likely if interrupts never get processed :-(

> For reference, we did some experiments (using this custom built kernel) with a dedicated thread doing nothing but a loop calling rte_rdtsc_precise() and registering the delta. Although the overwhelming majority is ca. CPU 80 cycles, there are some big outliers at ca. 9,000 CPU cycles. (Order of magnitude: ca. 45 of these big outliers per minute.) Apparently some kernel threads steal some cycles from this thread, regardless of our customizations. We haven't bothered analyzing and optimizing it further.

Was this on isolated CPU?
Did you check that that CPU was excluded from the smp_affinty mask on all devices?
Did you enable the kernel feature to avoid clock ticks if CPU is dedicated?
Same thing for RCU, need to adjust parameters?

Also, on many systems there can be SMI BIOS hidden execution that will cause big outliers.

Lastly never try and use CPU 0. The kernel uses CPU 0 as catch all in lots of places.

> I think our experiment supports the need to allow kernel threads to run, e.g. by calling sleep() or similar, when an EAL thread has real-time priority.


  reply	other threads:[~2023-10-25 21:33 UTC|newest]

Thread overview: 54+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-10-24 12:54 [PATCH] " Thomas Monjalon
2023-10-24 13:55 ` Morten Brørup
2023-10-24 16:04   ` Stephen Hemminger
2023-10-25 13:15     ` Thomas Monjalon
2023-10-25 13:34       ` Bruce Richardson
2023-10-25 13:44         ` Thomas Monjalon
2023-10-25 15:08           ` Stephen Hemminger
2023-10-25 15:14             ` Bruce Richardson
2023-10-25 15:18               ` Thomas Monjalon
2023-10-25 15:32                 ` Thomas Monjalon
2023-10-25 15:13 ` [PATCH v2] " Thomas Monjalon
2023-10-25 15:37   ` Stephen Hemminger
2023-10-25 16:46     ` Thomas Monjalon
2023-10-25 17:54       ` Morten Brørup
2023-10-25 21:33         ` Stephen Hemminger [this message]
2023-10-26  7:33           ` Morten Brørup
2023-10-26 16:32             ` Stephen Hemminger
2023-10-26 17:07               ` Morten Brørup
2023-10-26  0:00         ` Stephen Hemminger
2023-10-25 16:31 ` [PATCH v3 0/2] " Thomas Monjalon
2023-10-25 16:31   ` [PATCH v3 1/2] eal: add thread yield functions Thomas Monjalon
2023-10-25 16:40     ` Bruce Richardson
2023-10-25 17:06       ` Thomas Monjalon
2023-10-25 18:07     ` Morten Brørup
2023-10-25 16:31   ` [PATCH v3 2/2] eal/unix: allow creating thread with real-time priority Thomas Monjalon
2023-10-26 13:36   ` [PATCH v3 0/2] " Thomas Monjalon
2023-10-26 13:44     ` Morten Brørup
2023-10-26 13:57       ` Morten Brørup
2023-10-26 14:04         ` Thomas Monjalon
2023-10-26 14:08           ` Morten Brørup
2023-10-26 14:30             ` Thomas Monjalon
2023-10-26 14:50               ` Morten Brørup
2023-10-26 14:59                 ` Morten Brørup
2023-10-26 15:54                   ` Bruce Richardson
2023-10-26 16:07                     ` Thomas Monjalon
2023-10-26 16:50                       ` Morten Brørup
2023-10-26 19:51                         ` Thomas Monjalon
2023-10-27  7:19                           ` Morten Brørup
2023-10-26 16:28             ` Stephen Hemminger
2023-10-26 19:55               ` Thomas Monjalon
2023-10-26 21:10                 ` Stephen Hemminger
2023-10-26 14:10         ` David Marchand
2023-10-26 13:37 ` [PATCH v4 " Thomas Monjalon
2023-10-26 13:37   ` [PATCH v4 1/2] eal: add thread yield functions Thomas Monjalon
2023-10-26 13:37   ` [PATCH v4 2/2] eal/unix: allow creating thread with real-time priority Thomas Monjalon
2023-10-26 14:19 ` [PATCH v5 0/2] " Thomas Monjalon
2023-10-26 14:19   ` [PATCH v5 1/2] eal: add thread yield functions Thomas Monjalon
2023-10-26 14:19   ` [PATCH v5 2/2] eal/unix: allow creating thread with real-time priority Thomas Monjalon
2023-10-26 20:00   ` [PATCH v5 0/2] " Thomas Monjalon
2023-10-26 23:35     ` Tyler Retzlaff
2023-10-27  8:08 ` [PATCH v6 1/1] eal/unix: " Thomas Monjalon
2023-10-27  8:45   ` Morten Brørup
2023-10-27  9:11     ` Thomas Monjalon
2023-10-27 18:15     ` Stephen Hemminger

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20231025143318.3be26bb3@hermes.local \
    --to=stephen@networkplumber.org \
    --cc=anatoly.burakov@intel.com \
    --cc=andrew.rybchenko@oktetlabs.ru \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=dmitry.kozliuk@gmail.com \
    --cc=dmitrym@microsoft.com \
    --cc=konstantin.v.ananyev@yandex.ru \
    --cc=mb@smartsharesystems.com \
    --cc=navasile@linux.microsoft.com \
    --cc=pallavi.kadam@intel.com \
    --cc=roretzla@linux.microsoft.com \
    --cc=stable@dpdk.org \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).