Large interruptions for EAL thread running on isol core

Large interruptions for EAL thread running on isol core

[Antonio Di Bacco]

I'm running a DPDK thread on an isolated core. I also set some  flags
that could help keeping the core at rest on linux like: nosoftlockup
nohz_full rcu_nocbs irqaffinity.

Unfortunately the thread gets some interruptions that stop the thread
for about 20-30 micro seconds. This seems smal but my application
suffers a lot.

I also tried to use  rte_thread_set_priority that indeed has a strong
effect but unfortunately creates problems to Linux (like network not
working).

Is there any other knob that could help running the DPDK thread with
minimum or no interruptions at all?

Re: Large interruptions for EAL thread running on isol core

[Omer Yamac]

Hi Antonio,

Did you try to disable logical cores on CPU?

On 23.06.2022 21:03, Antonio Di Bacco wrote:
> I'm running a DPDK thread on an isolated core. I also set some  flags
> that could help keeping the core at rest on linux like: nosoftlockup
> nohz_full rcu_nocbs irqaffinity.
> 
> Unfortunately the thread gets some interruptions that stop the thread
> for about 20-30 micro seconds. This seems smal but my application
> suffers a lot.
> 
> I also tried to use  rte_thread_set_priority that indeed has a strong
> effect but unfortunately creates problems to Linux (like network not
> working).
> 
> Is there any other knob that could help running the DPDK thread with
> minimum or no interruptions at all?

Re: Large interruptions for EAL thread running on isol core

[Stephen Hemminger]

On Thu, 23 Jun 2022 20:03:02 +0200
Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:

> I'm running a DPDK thread on an isolated core. I also set some  flags
> that could help keeping the core at rest on linux like: nosoftlockup
> nohz_full rcu_nocbs irqaffinity.
> 
> Unfortunately the thread gets some interruptions that stop the thread
> for about 20-30 micro seconds. This seems smal but my application
> suffers a lot.
> 
> I also tried to use  rte_thread_set_priority that indeed has a strong
> effect but unfortunately creates problems to Linux (like network not
> working).
> 
> Is there any other knob that could help running the DPDK thread with
> minimum or no interruptions at all?

Look with perf and see what is happening.
First check for interrupt affinity.
Don't try real time priority.

The other thing to look for would be any BIOS settings.
Some system management units can take away CPU silently for polling
some internal housekeeping.

Re: Large interruptions for EAL thread running on isol core

[Carsten Andrich]

[-- Attachment #1.1: Type: text/plain, Size: 3705 bytes --]

I've been working on this issue occasionally for a 2+ years now. 
Unfortunately, it is not easy to solve and substantiated information on 
it is also hard to come by.

My current, **empirically founded** solution to get interfering 
interrupts down as much as possible is the following (last tested about 
a year ago with kernel ~5.10):

 1. Don't run anything on CPU 0. It's the boot CPU and may run some
    Linux voodoo that cannot be transferred to other CPUs. Read this
    somewhere, but haven't been able to substantiate that. Also, disable
    SMT or keep the SMT-siblings isolated, too, to avoid contention on
    cache and memory. May both be superstition, but I don't see any
    potential down-side to it.
 2. Use real-time priority (SCHED_FIFO w/ priority 99) for the DPDK
    threads and
    echo -1 > /proc/sys/kernel/sched_rt_runtime_us
    to disable the runtime limit. With the runtime limit in place, the
    SCHED_FIFO performance will be significantly worse than SCHED_OTHER.
 3. Use the following kernel command line (similar to yours; isolates
    all but 2 cores on a 12 core machine):
    isolcpus=nohz,domain,2-11,14-23 nohz_full=2-11,14-23
    rcu_nocbs=2-11,14-23 rcu_nocb_poll irqaffinity=0-1,12-13
    Note that this only works properly with a NO_HZ kernel. The kernel
    boot log should contain an error message if above parameters are
    used on a non-NO_HZ kernel.
    I compile my kernel with the following config enabled: NO_HZ_FULL,
    RCU_NOCB_CPU, CONTEXT_TRACKING_FORCE
 4. Force remaining interrupts off the isolated CPUs by first stopping
    all of them and subsequently starting them again (some interrupts
    may remain on the isolated CPUs regardless of irqaffinit=; mileage
    may vary):
    for CPU in $CPU_LIST ;
         do echo 0 > /sys/devices/system/cpu/cpu$CPU/online
    done
    for CPU in $CPU_LIST ;
         do echo 1 > /sys/devices/system/cpu/cpu$CPU/online
    done
 5. Check which interrupts still occur on the isolated CPUs while
    running your DPDK progam via /proc/interrupts. I've had issues with
    some hardware drivers' interrupts (e.g., RAID controllers) refusing
    to be kicked off the isolated CPUs despite all of the above. Try to
    move your sensitive threads to different CPUs.
 6. Despair. Even if you succeed in getting all hardware interrupts
    disabled, the kernel will still occasionally interrupt your program,
    e.g., for some accounting business.

The last point is where intervention from the kernel side is required. 
Work on that has been underway for several years [1,2,3], but nothing 
has been mainlined yet.

Hope this helps. We've been able to get worst case packet forwarding 
jitter down to less than 10 µs, with anything above 3 µs being very rare 
(see attached histogram; your mileage may vary; measured by comparing 
TSC values between DPDK rx and tx calls).

Best regards,
Carsten

[1] https://lwn.net/Articles/659490/
[2] https://lwn.net/Articles/816298/
[3] 
https://lore.kernel.org/lkml/20220315153132.717153751@fedora.localdomain/

On 23.06.22 20:03, Antonio Di Bacco wrote:
> I'm running a DPDK thread on an isolated core. I also set some  flags
> that could help keeping the core at rest on linux like: nosoftlockup
> nohz_full rcu_nocbs irqaffinity.
>
> Unfortunately the thread gets some interruptions that stop the thread
> for about 20-30 micro seconds. This seems smal but my application
> suffers a lot.
>
> I also tried to use  rte_thread_set_priority that indeed has a strong
> effect but unfortunately creates problems to Linux (like network not
> working).
>
> Is there any other knob that could help running the DPDK thread with
> minimum or no interruptions at all?

[-- Attachment #1.2: Type: text/html, Size: 4809 bytes --]

[-- Attachment #2: DPDK_isol_rx_tx_delay.png --]
[-- Type: image/png, Size: 24571 bytes --]

Re: Large interruptions for EAL thread running on isol core

[Antonio Di Bacco]

Yes, Hyperthreading is disabled.

On Thu, Jun 23, 2022 at 8:07 PM Omer Yamac <omer.yamac@ceng.metu.edu.tr> wrote:
>
> Hi Antonio,
>
> Did you try to disable logical cores on CPU?
>
> On 23.06.2022 21:03, Antonio Di Bacco wrote:
> > I'm running a DPDK thread on an isolated core. I also set some  flags
> > that could help keeping the core at rest on linux like: nosoftlockup
> > nohz_full rcu_nocbs irqaffinity.
> >
> > Unfortunately the thread gets some interruptions that stop the thread
> > for about 20-30 micro seconds. This seems smal but my application
> > suffers a lot.
> >
> > I also tried to use  rte_thread_set_priority that indeed has a strong
> > effect but unfortunately creates problems to Linux (like network not
> > working).
> >
> > Is there any other knob that could help running the DPDK thread with
> > minimum or no interruptions at all?

Re: Large interruptions for EAL thread running on isol core

[Antonio Di Bacco]

Thank you, didn't know that "System management units" could steal
CPU!!! That is scary

On Thu, Jun 23, 2022 at 8:42 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Thu, 23 Jun 2022 20:03:02 +0200
> Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
>
> > I'm running a DPDK thread on an isolated core. I also set some  flags
> > that could help keeping the core at rest on linux like: nosoftlockup
> > nohz_full rcu_nocbs irqaffinity.
> >
> > Unfortunately the thread gets some interruptions that stop the thread
> > for about 20-30 micro seconds. This seems smal but my application
> > suffers a lot.
> >
> > I also tried to use  rte_thread_set_priority that indeed has a strong
> > effect but unfortunately creates problems to Linux (like network not
> > working).
> >
> > Is there any other knob that could help running the DPDK thread with
> > minimum or no interruptions at all?
>
> Look with perf and see what is happening.
> First check for interrupt affinity.
> Don't try real time priority.
>
> The other thing to look for would be any BIOS settings.
> Some system management units can take away CPU silently for polling
> some internal housekeeping.

RE: Large interruptions for EAL thread running on isol core

[Kinsella, Ray]

Have you tried testing with Jitter?

Looking through the CSIT data, will give an idea of the reasonable ranges for results.

Intel ICELAKE
https://docs.fd.io/csit/master/report/introduction/test_environment_sut_calib_icx.html

Intel CASCADE LAKE
https://docs.fd.io/csit/master/report/introduction/test_environment_sut_calib_clx.html

Ray K

-----Original Message-----
From: Antonio Di Bacco <a.dibacco.ks@gmail.com> 
Sent: Friday 24 June 2022 10:45
To: Stephen Hemminger <stephen@networkplumber.org>
Cc: users@dpdk.org
Subject: Re: Large interruptions for EAL thread running on isol core

Thank you, didn't know that "System management units" could steal CPU!!! That is scary

On Thu, Jun 23, 2022 at 8:42 PM Stephen Hemminger <stephen@networkplumber.org> wrote:
>
> On Thu, 23 Jun 2022 20:03:02 +0200
> Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
>
> > I'm running a DPDK thread on an isolated core. I also set some  
> > flags that could help keeping the core at rest on linux like: 
> > nosoftlockup nohz_full rcu_nocbs irqaffinity.
> >
> > Unfortunately the thread gets some interruptions that stop the 
> > thread for about 20-30 micro seconds. This seems smal but my 
> > application suffers a lot.
> >
> > I also tried to use  rte_thread_set_priority that indeed has a 
> > strong effect but unfortunately creates problems to Linux (like 
> > network not working).
> >
> > Is there any other knob that could help running the DPDK thread with 
> > minimum or no interruptions at all?
>
> Look with perf and see what is happening.
> First check for interrupt affinity.
> Don't try real time priority.
>
> The other thing to look for would be any BIOS settings.
> Some system management units can take away CPU silently for polling 
> some internal housekeeping.

Re: Large interruptions for EAL thread running on isol core

[Stephen Hemminger]

On Thu, 23 Jun 2022 21:03:49 +0200
Carsten Andrich <carsten.andrich@tu-ilmenau.de> wrote:

>  2. Use real-time priority (SCHED_FIFO w/ priority 99) for the DPDK
>     threads and
>     echo -1 > /proc/sys/kernel/sched_rt_runtime_us
>     to disable the runtime limit. With the runtime limit in place, the
>     SCHED_FIFO performance will be significantly worse than SCHED_OTHER.

This can cause major issues if application is normal DPDK application (never does system calls).
If an interrupt or other event happens on your isolated CPU, the work that it would
do in soft irq is never performed. FIFO has higher priority than kernel threads.
This can lead to mystery lockups from other applications (reads not completing, network timeouts, etc).

In any semi-recent kernel using SCHED_NO_HZ_FULL will keep clock ticks from happening
on the isolated cores.

Re: Large interruptions for EAL thread running on isol core

[Carsten Andrich]

I cannot find SCHED_NO_HZ_FULL in v5.18:
https://elixir.bootlin.com/linux/latest/A/ident/SCHED_NO_HZ_FULL

Could you elaborate?

Thanks and best regards,
Carsten

On 24.06.22 17:01, Stephen Hemminger wrote:
> On Thu, 23 Jun 2022 21:03:49 +0200
> Carsten Andrich <carsten.andrich@tu-ilmenau.de> wrote:
>
>>   2. Use real-time priority (SCHED_FIFO w/ priority 99) for the DPDK
>>      threads and
>>      echo -1 > /proc/sys/kernel/sched_rt_runtime_us
>>      to disable the runtime limit. With the runtime limit in place, the
>>      SCHED_FIFO performance will be significantly worse than SCHED_OTHER.
> This can cause major issues if application is normal DPDK application (never does system calls).
> If an interrupt or other event happens on your isolated CPU, the work that it would
> do in soft irq is never performed. FIFO has higher priority than kernel threads.
> This can lead to mystery lockups from other applications (reads not completing, network timeouts, etc).
>
> In any semi-recent kernel using SCHED_NO_HZ_FULL will keep clock ticks from happening
> on the isolated cores.

Re: Large interruptions for EAL thread running on isol core

[Stephen Hemminger]

On Fri, 24 Jun 2022 17:41:44 +0200
Carsten Andrich <carsten.andrich@tu-ilmenau.de> wrote:

> I cannot find SCHED_NO_HZ_FULL in v5.18:
> https://elixir.bootlin.com/linux/latest/A/ident/SCHED_NO_HZ_FULL
> 
Sorry, wrong option.

Full docs here:
https://docs.kernel.org/timers/no_hz.html

Re: Large interruptions for EAL thread running on isol core

[Carsten Andrich]

On 24.06.22 17:01, Stephen Hemminger wrote:
> On Thu, 23 Jun 2022 21:03:49 +0200
> Carsten Andrich <carsten.andrich@tu-ilmenau.de> wrote:
>
>>   2. Use real-time priority (SCHED_FIFO w/ priority 99) for the DPDK
>>      threads and
>>      echo -1 > /proc/sys/kernel/sched_rt_runtime_us
>>      to disable the runtime limit. With the runtime limit in place, the
>>      SCHED_FIFO performance will be significantly worse than SCHED_OTHER.
> This can cause major issues if application is normal DPDK application (never does system calls).
> If an interrupt or other event happens on your isolated CPU, the work that it would
> do in soft irq is never performed. FIFO has higher priority than kernel threads.
> This can lead to mystery lockups from other applications (reads not completing, network timeouts, etc).

Thanks for pointing that out. Do you know of any official kernel 
documentation that could shed some light on that? I haven't had any 
serious issues like the ones you list, but maybe I've been lucky. My 
DPDK applications typically run on fairly minimal systems used 
exclusively for DPDK tasks, which require minimal latency/jitter. Minor 
side-effects from using SCHED_FIFO are tolerable in my case, if it 
improves performance.


> In any semi-recent kernel using SCHED_NO_HZ_FULL will keep clock ticks from happening
> on the isolated cores.
On 24.06.22 18:42, Stephen Hemminger wrote:
> Sorry, wrong option.
>
> Full docs here:
> https://docs.kernel.org/timers/no_hz.html

Thanks. nohz_full is already in the list of kernel cmdline options I use.

Best regards,
Carsten

Re: Large interruptions for EAL thread running on isol core

[Stephen Hemminger]

On Tue, 28 Jun 2022 09:25:50 +0200
Carsten Andrich <carsten.andrich@tu-ilmenau.de> wrote:

> On 24.06.22 17:01, Stephen Hemminger wrote:
> > On Thu, 23 Jun 2022 21:03:49 +0200
> > Carsten Andrich <carsten.andrich@tu-ilmenau.de> wrote:
> >  
> >>   2. Use real-time priority (SCHED_FIFO w/ priority 99) for the DPDK
> >>      threads and
> >>      echo -1 > /proc/sys/kernel/sched_rt_runtime_us
> >>      to disable the runtime limit. With the runtime limit in place, the
> >>      SCHED_FIFO performance will be significantly worse than SCHED_OTHER.  
> > This can cause major issues if application is normal DPDK application (never does system calls).
> > If an interrupt or other event happens on your isolated CPU, the work that it would
> > do in soft irq is never performed. FIFO has higher priority than kernel threads.
> > This can lead to mystery lockups from other applications (reads not completing, network timeouts, etc).  
> 
> Thanks for pointing that out. Do you know of any official kernel 
> documentation that could shed some light on that? I haven't had any 
> serious issues like the ones you list, but maybe I've been lucky. My 
> DPDK applications typically run on fairly minimal systems used 
> exclusively for DPDK tasks, which require minimal latency/jitter. Minor 
> side-effects from using SCHED_FIFO are tolerable in my case, if it 
> improves performance.

Do some looking around and  you will find good documentation like:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_real_time/7/html/tuning_guide/real_time_throttling


This characteristic of real-time threads means that it is quite easy to
write an application which monopolizes 100% of a given CPU. At first
glance this sounds like it might be a good idea, but in reality it
causes lots of headaches for the operating system. The OS is
responsible for managing both system-wide and per-CPU resources and
must periodically examine data structures describing these resources
and perform housekeeping activities with them. If a core is monopolized
by a SCHED_FIFO thread, it cannot perform the housekeeping tasks and
eventually the entire system becomes unstable, potentially causing a
crash.