From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D272CA057F for ; Tue, 28 Jun 2022 17:19:22 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A5843400D7; Tue, 28 Jun 2022 17:19:22 +0200 (CEST) Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by mails.dpdk.org (Postfix) with ESMTP id DE44640042 for ; Tue, 28 Jun 2022 17:19:21 +0200 (CEST) Received: by mail-pl1-f180.google.com with SMTP id jh14so11380996plb.1 for ; Tue, 28 Jun 2022 08:19:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KXpQjfX/A2rdJNuM8t+aOWKUOkchDYEgFS9Mlwa04Ik=; b=mPsAszxFp2hG1V37xdJAF2JdaIGowLALQC6Uj/iS0XQz+I0HOHSc1KFEIwbZAupzfR ktrZ8Bh4Kf2abg460X75Azh/Ozsf4meBgKjC/yTVr2U2J2CbdDMAvXguaCk8MMQf/hQw 7SErunmooaYGNzlFv2Gb1FL6FzgSwU7HSfKgdHHoF4FAtPOt78InBNPG5R0R7lXWcd0W BXcl6MaLDwvRnXgbGyk2oS7ZC6fL7wSYgreOz6ljHvg07fJ2myT0BHExjyqqT5VNh3rT /h5x10jK3xsq0+tRKCaIs5gwWdQ6mtsx74elGf7bByBMzO5CJVWHgLRTyILhmVdz1QHP 4oEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KXpQjfX/A2rdJNuM8t+aOWKUOkchDYEgFS9Mlwa04Ik=; b=W9L7vthNI+mA+EuO4kXb4IkNT4f7/q2XY72AeEs+InwVGSK3ZHdQsCIGEohnDV0lL7 s+z2JpgwrfuxHqgvcqPupt/NRiQpQZU4+NqdQhC/P5QjRlxqv3aSbRoV8CpnzNK8RMu4 LiiH2a8WFQXvYtXe7xNmY8pP2Str/ROPR9mfwC8AR57TsEQwPlKrFd6bDkamm1E8bA1v Za2BQ58aRLtTwrR2bTWDXN2x2rLqzol9TAxZrrxScnZx0MJxAQfzvXk6JpNgrjMC99Nr p5iAec2hN+jqghSqIzLQg8zwdEAh7bb9UEqkRl82VnuLYO9IWPdG2cvDCTK/aIsT7Aac raRw== X-Gm-Message-State: AJIora/z8eOVeoyX2DxWtYZucLqf4Cn+LsdCq8k06SKNmbqkLs7vBoRm 1YTOdBHSXx7jQPlAnnIoOUGsk2T0ii+msYv4 X-Google-Smtp-Source: AGRyM1v5A3OsMyO4kapfOeBlNypzZghnuaDLZ2+wCL1cWwlhd7q99DjQ/nonwhl32y7e0AwyJ7kKXg== X-Received: by 2002:a17:90b:3b85:b0:1ec:d979:ca42 with SMTP id pc5-20020a17090b3b8500b001ecd979ca42mr92896pjb.209.1656429561005; Tue, 28 Jun 2022 08:19:21 -0700 (PDT) Received: from hermes.local (204-195-112-199.wavecable.com. [204.195.112.199]) by smtp.gmail.com with ESMTPSA id n22-20020a17090a161600b001ecd48b80a2sm11879229pja.5.2022.06.28.08.19.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Jun 2022 08:19:20 -0700 (PDT) Date: Tue, 28 Jun 2022 08:19:18 -0700 From: Stephen Hemminger To: Carsten Andrich Cc: users@dpdk.org Subject: Re: Large interruptions for EAL thread running on isol core Message-ID: <20220628081918.0fc0f31c@hermes.local> In-Reply-To: <0906411a-90ad-5651-0153-e34a29027f66@tu-ilmenau.de> References: <9a95a560-c7c6-d4fa-1041-836fa400f497@tu-ilmenau.de> <20220624080124.244a01a9@hermes.local> <20220624094243.232f955e@hermes.local> <0906411a-90ad-5651-0153-e34a29027f66@tu-ilmenau.de> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Tue, 28 Jun 2022 09:25:50 +0200 Carsten Andrich wrote: > On 24.06.22 17:01, Stephen Hemminger wrote: > > On Thu, 23 Jun 2022 21:03:49 +0200 > > Carsten Andrich wrote: > > > >> 2. Use real-time priority (SCHED_FIFO w/ priority 99) for the DPDK > >> threads and > >> echo -1 > /proc/sys/kernel/sched_rt_runtime_us > >> to disable the runtime limit. With the runtime limit in place, the > >> SCHED_FIFO performance will be significantly worse than SCHED_OTHER. > > This can cause major issues if application is normal DPDK application (never does system calls). > > If an interrupt or other event happens on your isolated CPU, the work that it would > > do in soft irq is never performed. FIFO has higher priority than kernel threads. > > This can lead to mystery lockups from other applications (reads not completing, network timeouts, etc). > > Thanks for pointing that out. Do you know of any official kernel > documentation that could shed some light on that? I haven't had any > serious issues like the ones you list, but maybe I've been lucky. My > DPDK applications typically run on fairly minimal systems used > exclusively for DPDK tasks, which require minimal latency/jitter. Minor > side-effects from using SCHED_FIFO are tolerable in my case, if it > improves performance. Do some looking around and you will find good documentation like: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_real_time/7/html/tuning_guide/real_time_throttling This characteristic of real-time threads means that it is quite easy to write an application which monopolizes 100% of a given CPU. At first glance this sounds like it might be a good idea, but in reality it causes lots of headaches for the operating system. The OS is responsible for managing both system-wide and per-CPU resources and must periodically examine data structures describing these resources and perform housekeeping activities with them. If a core is monopolized by a SCHED_FIFO thread, it cannot perform the housekeeping tasks and eventually the entire system becomes unstable, potentially causing a crash.