From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 26E9EA0544 for ; Tue, 28 Jun 2022 09:25:53 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id ABCF240691; Tue, 28 Jun 2022 09:25:52 +0200 (CEST) Received: from smail.rz.tu-ilmenau.de (smail.rz.tu-ilmenau.de [141.24.186.67]) by mails.dpdk.org (Postfix) with ESMTP id F06BE400D7 for ; Tue, 28 Jun 2022 09:25:50 +0200 (CEST) Received: from [141.24.218.14] (unknown [141.24.218.14]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smail.rz.tu-ilmenau.de (Postfix) with ESMTPSA id B7631580093; Tue, 28 Jun 2022 09:25:50 +0200 (CEST) Message-ID: <0906411a-90ad-5651-0153-e34a29027f66@tu-ilmenau.de> Date: Tue, 28 Jun 2022 09:25:50 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.10.0 Subject: Re: Large interruptions for EAL thread running on isol core Content-Language: en-US To: Stephen Hemminger Cc: users@dpdk.org References: <9a95a560-c7c6-d4fa-1041-836fa400f497@tu-ilmenau.de> <20220624080124.244a01a9@hermes.local> <20220624094243.232f955e@hermes.local> From: Carsten Andrich In-Reply-To: <20220624094243.232f955e@hermes.local> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On 24.06.22 17:01, Stephen Hemminger wrote: > On Thu, 23 Jun 2022 21:03:49 +0200 > Carsten Andrich wrote: > >> 2. Use real-time priority (SCHED_FIFO w/ priority 99) for the DPDK >> threads and >> echo -1 > /proc/sys/kernel/sched_rt_runtime_us >> to disable the runtime limit. With the runtime limit in place, the >> SCHED_FIFO performance will be significantly worse than SCHED_OTHER. > This can cause major issues if application is normal DPDK application (never does system calls). > If an interrupt or other event happens on your isolated CPU, the work that it would > do in soft irq is never performed. FIFO has higher priority than kernel threads. > This can lead to mystery lockups from other applications (reads not completing, network timeouts, etc). Thanks for pointing that out. Do you know of any official kernel documentation that could shed some light on that? I haven't had any serious issues like the ones you list, but maybe I've been lucky. My DPDK applications typically run on fairly minimal systems used exclusively for DPDK tasks, which require minimal latency/jitter. Minor side-effects from using SCHED_FIFO are tolerable in my case, if it improves performance. > In any semi-recent kernel using SCHED_NO_HZ_FULL will keep clock ticks from happening > on the isolated cores. On 24.06.22 18:42, Stephen Hemminger wrote: > Sorry, wrong option. > > Full docs here: > https://docs.kernel.org/timers/no_hz.html Thanks. nohz_full is already in the list of kernel cmdline options I use. Best regards, Carsten