From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9D28A431FE for ; Wed, 25 Oct 2023 23:33:24 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 933E440E54; Wed, 25 Oct 2023 23:33:24 +0200 (CEST) Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by mails.dpdk.org (Postfix) with ESMTP id 3E9994029A for ; Wed, 25 Oct 2023 23:33:22 +0200 (CEST) Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-1bf55a81eeaso1404905ad.0 for ; Wed, 25 Oct 2023 14:33:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1698269601; x=1698874401; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=Nxd/VgLRKX9UlFTIi6U0jsgIvsP3nRKCCpiXd1Jo4v4=; b=MRNjs5NxXjWMUkTJomPqReQZZGH5tAG0HBV/ibO6x0AWxn21Ib/a3hsL3fdqFk0T54 OctRtYhBfp4L8EciEDzH6XGfky0xrEYAnHNffeirhuJctrlQMnD7KnIdykW4594MaQxn 1S6cQXL3TXXmt0J9hqGCwopDdFq2roeT953lSofVxLE54Biicuewbn6T//YZgBKShNQK Jjig84EJRX5mo+lXnBPckzxzBXA8FLSsuurzKWx5pLv6UDwCRNN5XGTTZbaevmmIRk2+ VgGkvgKC/XA6jxMVcJiylUMjS03/3npMneTsz8rCxwPCw98+wFPsdAF1JFWvTc1xdIBH BMpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698269601; x=1698874401; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Nxd/VgLRKX9UlFTIi6U0jsgIvsP3nRKCCpiXd1Jo4v4=; b=nLIyddUOdSyWah0jwhfVfrJETGm6X/x3iPWnOR/u4uQPWzxBlBfmavIH3XGjFD6ZHC DzEoOY9lUKFfAZ/t3PQui5Z2ZtRpmPLm5W/KRnRsmpDUWfFAjZbF2cvZixgJIjDVFL8/ C3Qp37brMHQQyaMh6QgJUwEjEGNCEhbn7RH8VLwCkvwe5lZgk93p/yV/5pmWdUJ/85rs WdqzpxcAx7/fV/vgSmevCeobRavC+Vf0d41j6KJMVDrC3hARb0LREPf4im8jq4/l6zB8 yEcIm+X9mV/Oz5Fj1PvuTpq6Uddt/Cacfm2DaSD92mYkE/TnjqE1VC19rzmrozYvTv6H moyA== X-Gm-Message-State: AOJu0Yz9x8QpRbyKLTdpghGp8x0Pp1vou/0n9btq4fq+IOx/QkljQHM+ C3GdQIrUXZVeBUkxfYKB59eAAA== X-Google-Smtp-Source: AGHT+IHLuxLvauIl281BLHV1mJ+55ceWytzw4nqyevCvLLLeah3LZZDMzRAwEpNEjOpxIrp0e/ePug== X-Received: by 2002:a17:90a:e00d:b0:27d:9b5:d6f8 with SMTP id u13-20020a17090ae00d00b0027d09b5d6f8mr12196048pjy.49.1698269601149; Wed, 25 Oct 2023 14:33:21 -0700 (PDT) Received: from hermes.local (204-195-126-68.wavecable.com. [204.195.126.68]) by smtp.gmail.com with ESMTPSA id pb6-20020a17090b3c0600b0027d06ddc06bsm344566pjb.33.2023.10.25.14.33.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 14:33:21 -0700 (PDT) Date: Wed, 25 Oct 2023 14:33:18 -0700 From: Stephen Hemminger To: Morten =?UTF-8?B?QnLDuHJ1cA==?= Cc: "Thomas Monjalon" , , "David Marchand" , , "Anatoly Burakov" , "Dmitry Kozlyuk" , "Narcisa Ana Maria Vasile" , "Dmitry Malloy" , "Pallavi Kadam" , "Tyler Retzlaff" , "Andrew Rybchenko" , "Konstantin Ananyev" Subject: Re: [PATCH v2] eal/unix: allow creating thread with real-time priority Message-ID: <20231025143318.3be26bb3@hermes.local> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35E9EF8B@smartserver.smartshare.dk> References: <20231024125416.798897-1-thomas@monjalon.net> <20231025151352.995318-1-thomas@monjalon.net> <20231025083700.4e3e274c@hermes.local> <23265462.6Emhk5qWAg@thomas> <98CBD80474FA8B44BF855DF32C47DC35E9EF8B@smartserver.smartshare.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org On Wed, 25 Oct 2023 19:54:06 +0200 Morten Br=C3=B8rup wrote: > I agree with Thomas on this. >=20 > If you want the log message, please degrade it to INFO or DEBUG level. It= is only relevant when chasing problems, not for normal production - and th= us NOTICE is too high. I don't want the message to be hidden. If we get any bug reports want to be able to say "read the log, don't do th= at". > Someone might build a kernel with options to keep non-dataplane threads o= ff some dedicated CPU cores, so they can be used for guaranteed low-latency= dataplane threads. We do. We don't use real-time priority, though. This is really, hard to do. Isolated CPU's are not isolated from interrupts= and other sources which end up scheduling work as kernel threads. Plus the= re is the behavior where kernel decides to turn a soft irq into a kernel th= read, then starve itself. Under starvation, disk corruption is likely if interrupts never get= processed :-( > For reference, we did some experiments (using this custom built kernel) w= ith a dedicated thread doing nothing but a loop calling rte_rdtsc_precise()= and registering the delta. Although the overwhelming majority is ca. CPU 8= 0 cycles, there are some big outliers at ca. 9,000 CPU cycles. (Order of ma= gnitude: ca. 45 of these big outliers per minute.) Apparently some kernel t= hreads steal some cycles from this thread, regardless of our customizations= . We haven't bothered analyzing and optimizing it further. Was this on isolated CPU? Did you check that that CPU was excluded from the smp_affinty mask on all d= evices? Did you enable the kernel feature to avoid clock ticks if CPU is dedicated? Same thing for RCU, need to adjust parameters? Also, on many systems there can be SMI BIOS hidden execution that will caus= e big outliers. Lastly never try and use CPU 0. The kernel uses CPU 0 as catch all in lots = of places. > I think our experiment supports the need to allow kernel threads to run, = e.g. by calling sleep() or similar, when an EAL thread has real-time priori= ty.