From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A85CA431FE; Wed, 25 Oct 2023 23:33:23 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 647BA4029A; Wed, 25 Oct 2023 23:33:23 +0200 (CEST) Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by mails.dpdk.org (Postfix) with ESMTP id 1F8A64014F for ; Wed, 25 Oct 2023 23:33:22 +0200 (CEST) Received: by mail-pj1-f48.google.com with SMTP id 98e67ed59e1d1-27d087c4276so147768a91.0 for ; Wed, 25 Oct 2023 14:33:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1698269601; x=1698874401; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=Nxd/VgLRKX9UlFTIi6U0jsgIvsP3nRKCCpiXd1Jo4v4=; b=MRNjs5NxXjWMUkTJomPqReQZZGH5tAG0HBV/ibO6x0AWxn21Ib/a3hsL3fdqFk0T54 OctRtYhBfp4L8EciEDzH6XGfky0xrEYAnHNffeirhuJctrlQMnD7KnIdykW4594MaQxn 1S6cQXL3TXXmt0J9hqGCwopDdFq2roeT953lSofVxLE54Biicuewbn6T//YZgBKShNQK Jjig84EJRX5mo+lXnBPckzxzBXA8FLSsuurzKWx5pLv6UDwCRNN5XGTTZbaevmmIRk2+ VgGkvgKC/XA6jxMVcJiylUMjS03/3npMneTsz8rCxwPCw98+wFPsdAF1JFWvTc1xdIBH BMpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698269601; x=1698874401; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Nxd/VgLRKX9UlFTIi6U0jsgIvsP3nRKCCpiXd1Jo4v4=; b=Sp5WMzHvuA1cFqbsFFsVYcJj3udidJigbgZvx8DwowoXVbYbw3arLFHMIdZg/HK9wI SwsRAwzsQQbACSG1CIDja6AHLDRm02yCUbjAXO+d6ZJbG2m45a1Hem8Ap0c6HJl3HnPU qpjgpb24HitJOWVWuG0de9toxRmcGHnSp9+fK34zhW3OXqnTLKFDbWkp2/ct/47PoFFR co1b5sqsFiDqgi6p8vLTfSRfVFtrCDh8Kpk6yG6sqpVRMMSB+iMrOmJQup6awfB8rK/6 zviBD6rRKM4OoGNFaxHnXFsGu9eCr053wb3VIsWaf4f3bEPwi+SZ91dHuRpKU+u/GRGP Ifhw== X-Gm-Message-State: AOJu0YxNtA3C/13vGUuGcTz/UwYNcnXRk0S50W+qsxW4UvKUq3zkJhGu tmrD5swLNP84hjfSCOwgUEMiDw== X-Google-Smtp-Source: AGHT+IHLuxLvauIl281BLHV1mJ+55ceWytzw4nqyevCvLLLeah3LZZDMzRAwEpNEjOpxIrp0e/ePug== X-Received: by 2002:a17:90a:e00d:b0:27d:9b5:d6f8 with SMTP id u13-20020a17090ae00d00b0027d09b5d6f8mr12196048pjy.49.1698269601149; Wed, 25 Oct 2023 14:33:21 -0700 (PDT) Received: from hermes.local (204-195-126-68.wavecable.com. [204.195.126.68]) by smtp.gmail.com with ESMTPSA id pb6-20020a17090b3c0600b0027d06ddc06bsm344566pjb.33.2023.10.25.14.33.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 14:33:21 -0700 (PDT) Date: Wed, 25 Oct 2023 14:33:18 -0700 From: Stephen Hemminger To: Morten =?UTF-8?B?QnLDuHJ1cA==?= Cc: "Thomas Monjalon" , , "David Marchand" , , "Anatoly Burakov" , "Dmitry Kozlyuk" , "Narcisa Ana Maria Vasile" , "Dmitry Malloy" , "Pallavi Kadam" , "Tyler Retzlaff" , "Andrew Rybchenko" , "Konstantin Ananyev" Subject: Re: [PATCH v2] eal/unix: allow creating thread with real-time priority Message-ID: <20231025143318.3be26bb3@hermes.local> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35E9EF8B@smartserver.smartshare.dk> References: <20231024125416.798897-1-thomas@monjalon.net> <20231025151352.995318-1-thomas@monjalon.net> <20231025083700.4e3e274c@hermes.local> <23265462.6Emhk5qWAg@thomas> <98CBD80474FA8B44BF855DF32C47DC35E9EF8B@smartserver.smartshare.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Wed, 25 Oct 2023 19:54:06 +0200 Morten Br=C3=B8rup wrote: > I agree with Thomas on this. >=20 > If you want the log message, please degrade it to INFO or DEBUG level. It= is only relevant when chasing problems, not for normal production - and th= us NOTICE is too high. I don't want the message to be hidden. If we get any bug reports want to be able to say "read the log, don't do th= at". > Someone might build a kernel with options to keep non-dataplane threads o= ff some dedicated CPU cores, so they can be used for guaranteed low-latency= dataplane threads. We do. We don't use real-time priority, though. This is really, hard to do. Isolated CPU's are not isolated from interrupts= and other sources which end up scheduling work as kernel threads. Plus the= re is the behavior where kernel decides to turn a soft irq into a kernel th= read, then starve itself. Under starvation, disk corruption is likely if interrupts never get= processed :-( > For reference, we did some experiments (using this custom built kernel) w= ith a dedicated thread doing nothing but a loop calling rte_rdtsc_precise()= and registering the delta. Although the overwhelming majority is ca. CPU 8= 0 cycles, there are some big outliers at ca. 9,000 CPU cycles. (Order of ma= gnitude: ca. 45 of these big outliers per minute.) Apparently some kernel t= hreads steal some cycles from this thread, regardless of our customizations= . We haven't bothered analyzing and optimizing it further. Was this on isolated CPU? Did you check that that CPU was excluded from the smp_affinty mask on all d= evices? Did you enable the kernel feature to avoid clock ticks if CPU is dedicated? Same thing for RCU, need to adjust parameters? Also, on many systems there can be SMI BIOS hidden execution that will caus= e big outliers. Lastly never try and use CPU 0. The kernel uses CPU 0 as catch all in lots = of places. > I think our experiment supports the need to allow kernel threads to run, = e.g. by calling sleep() or similar, when an EAL thread has real-time priori= ty.