From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <users-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id D272CA057F
	for <public@inbox.dpdk.org>; Tue, 28 Jun 2022 17:19:22 +0200 (CEST)
Received: from [217.70.189.124] (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id A5843400D7;
	Tue, 28 Jun 2022 17:19:22 +0200 (CEST)
Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com
 [209.85.214.180])
 by mails.dpdk.org (Postfix) with ESMTP id DE44640042
 for <users@dpdk.org>; Tue, 28 Jun 2022 17:19:21 +0200 (CEST)
Received: by mail-pl1-f180.google.com with SMTP id jh14so11380996plb.1
 for <users@dpdk.org>; Tue, 28 Jun 2022 08:19:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=networkplumber-org.20210112.gappssmtp.com; s=20210112;
 h=date:from:to:cc:subject:message-id:in-reply-to:references
 :mime-version:content-transfer-encoding;
 bh=KXpQjfX/A2rdJNuM8t+aOWKUOkchDYEgFS9Mlwa04Ik=;
 b=mPsAszxFp2hG1V37xdJAF2JdaIGowLALQC6Uj/iS0XQz+I0HOHSc1KFEIwbZAupzfR
 ktrZ8Bh4Kf2abg460X75Azh/Ozsf4meBgKjC/yTVr2U2J2CbdDMAvXguaCk8MMQf/hQw
 7SErunmooaYGNzlFv2Gb1FL6FzgSwU7HSfKgdHHoF4FAtPOt78InBNPG5R0R7lXWcd0W
 BXcl6MaLDwvRnXgbGyk2oS7ZC6fL7wSYgreOz6ljHvg07fJ2myT0BHExjyqqT5VNh3rT
 /h5x10jK3xsq0+tRKCaIs5gwWdQ6mtsx74elGf7bByBMzO5CJVWHgLRTyILhmVdz1QHP
 4oEQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to
 :references:mime-version:content-transfer-encoding;
 bh=KXpQjfX/A2rdJNuM8t+aOWKUOkchDYEgFS9Mlwa04Ik=;
 b=W9L7vthNI+mA+EuO4kXb4IkNT4f7/q2XY72AeEs+InwVGSK3ZHdQsCIGEohnDV0lL7
 s+z2JpgwrfuxHqgvcqPupt/NRiQpQZU4+NqdQhC/P5QjRlxqv3aSbRoV8CpnzNK8RMu4
 LiiH2a8WFQXvYtXe7xNmY8pP2Str/ROPR9mfwC8AR57TsEQwPlKrFd6bDkamm1E8bA1v
 Za2BQ58aRLtTwrR2bTWDXN2x2rLqzol9TAxZrrxScnZx0MJxAQfzvXk6JpNgrjMC99Nr
 p5iAec2hN+jqghSqIzLQg8zwdEAh7bb9UEqkRl82VnuLYO9IWPdG2cvDCTK/aIsT7Aac
 raRw==
X-Gm-Message-State: AJIora/z8eOVeoyX2DxWtYZucLqf4Cn+LsdCq8k06SKNmbqkLs7vBoRm
 1YTOdBHSXx7jQPlAnnIoOUGsk2T0ii+msYv4
X-Google-Smtp-Source: AGRyM1v5A3OsMyO4kapfOeBlNypzZghnuaDLZ2+wCL1cWwlhd7q99DjQ/nonwhl32y7e0AwyJ7kKXg==
X-Received: by 2002:a17:90b:3b85:b0:1ec:d979:ca42 with SMTP id
 pc5-20020a17090b3b8500b001ecd979ca42mr92896pjb.209.1656429561005; 
 Tue, 28 Jun 2022 08:19:21 -0700 (PDT)
Received: from hermes.local (204-195-112-199.wavecable.com. [204.195.112.199])
 by smtp.gmail.com with ESMTPSA id
 n22-20020a17090a161600b001ecd48b80a2sm11879229pja.5.2022.06.28.08.19.20
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Tue, 28 Jun 2022 08:19:20 -0700 (PDT)
Date: Tue, 28 Jun 2022 08:19:18 -0700
From: Stephen Hemminger <stephen@networkplumber.org>
To: Carsten Andrich <carsten.andrich@tu-ilmenau.de>
Cc: users@dpdk.org
Subject: Re: Large interruptions for EAL thread running on isol core
Message-ID: <20220628081918.0fc0f31c@hermes.local>
In-Reply-To: <0906411a-90ad-5651-0153-e34a29027f66@tu-ilmenau.de>
References: <CAO8pfF=yo37Ng5QAcopL=6LgKMHr4+=pPcONPATyS+OD0GLgfg@mail.gmail.com>
 <9a95a560-c7c6-d4fa-1041-836fa400f497@tu-ilmenau.de>
 <20220624080124.244a01a9@hermes.local>
 <ef44e436-8e3b-7083-10d8-0dc1602581c8@tu-ilmenau.de>
 <20220624094243.232f955e@hermes.local>
 <0906411a-90ad-5651-0153-e34a29027f66@tu-ilmenau.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-BeenThere: users@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK usage discussions <users.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/users>,
 <mailto:users-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/users/>
List-Post: <mailto:users@dpdk.org>
List-Help: <mailto:users-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/users>,
 <mailto:users-request@dpdk.org?subject=subscribe>
Errors-To: users-bounces@dpdk.org

On Tue, 28 Jun 2022 09:25:50 +0200
Carsten Andrich <carsten.andrich@tu-ilmenau.de> wrote:

> On 24.06.22 17:01, Stephen Hemminger wrote:
> > On Thu, 23 Jun 2022 21:03:49 +0200
> > Carsten Andrich <carsten.andrich@tu-ilmenau.de> wrote:
> >  
> >>   2. Use real-time priority (SCHED_FIFO w/ priority 99) for the DPDK
> >>      threads and
> >>      echo -1 > /proc/sys/kernel/sched_rt_runtime_us
> >>      to disable the runtime limit. With the runtime limit in place, the
> >>      SCHED_FIFO performance will be significantly worse than SCHED_OTHER.  
> > This can cause major issues if application is normal DPDK application (never does system calls).
> > If an interrupt or other event happens on your isolated CPU, the work that it would
> > do in soft irq is never performed. FIFO has higher priority than kernel threads.
> > This can lead to mystery lockups from other applications (reads not completing, network timeouts, etc).  
> 
> Thanks for pointing that out. Do you know of any official kernel 
> documentation that could shed some light on that? I haven't had any 
> serious issues like the ones you list, but maybe I've been lucky. My 
> DPDK applications typically run on fairly minimal systems used 
> exclusively for DPDK tasks, which require minimal latency/jitter. Minor 
> side-effects from using SCHED_FIFO are tolerable in my case, if it 
> improves performance.

Do some looking around and  you will find good documentation like:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_real_time/7/html/tuning_guide/real_time_throttling


This characteristic of real-time threads means that it is quite easy to
write an application which monopolizes 100% of a given CPU. At first
glance this sounds like it might be a good idea, but in reality it
causes lots of headaches for the operating system. The OS is
responsible for managing both system-wide and per-CPU resources and
must periodically examine data structures describing these resources
and perform housekeeping activities with them. If a core is monopolized
by a SCHED_FIFO thread, it cannot perform the housekeeping tasks and
eventually the entire system becomes unstable, potentially causing a
crash.