From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 99D65A034F; Thu, 1 Apr 2021 00:09:33 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2ED4A40142; Thu, 1 Apr 2021 00:09:33 +0200 (CEST) Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by mails.dpdk.org (Postfix) with ESMTP id 901364013F for ; Thu, 1 Apr 2021 00:09:32 +0200 (CEST) Received: by mail-pj1-f42.google.com with SMTP id cl21-20020a17090af695b02900c61ac0f0e9so3685215pjb.1 for ; Wed, 31 Mar 2021 15:09:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DNUEAnktC2EOmV3HloI/m01iqRfd8PurWz67I4gaz1A=; b=kN2wabt+wuIfYwKPmy7jjpFyVFSBIZhos+Qd4do7eErRHgixXQzXo4DHQ+KOuTOiys 3THhgigUUSwHjEsR9BLlwRoejSAwgQFvn4lIZPKbwv27wlR8sOZIlKW/ypbRoklgueG9 eH4fNADMcv8E8/5+MWo2ah1J4Tk+0+ebVhK+qxWvUCKUmDOR9Dkc9xqg6wbNbraMTr0i R1Dh9G/E40A7P8fB4OeL3/Sm29bHLVl1PVsDi50tMszj/r0lwkEDk1TeXB58IEE0DAe4 BjP1mJtRql8t29ckTtlnVaAX8ThiU59DgUwCY6wSWhos/yRQ1b3qbRmFGMHghvLoV7zF 4Yaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DNUEAnktC2EOmV3HloI/m01iqRfd8PurWz67I4gaz1A=; b=RfcY4+vYfhrM0DU1fZXjm/iofsIaUJJsLWIdRjH/Ky6Uf3R0rj712a2a1UNxCK+T/6 I5LJ0IcQ/mZHE6AnasAKVRIPWcQ85oTHOUFmsHMEq3O6oDNswOBpelXPCp2LDw/TNThz 0qQuePm/61CReF+T0vnOX6IveCN6IQzG/qv0hds3od/i8XFnuuyYBtwteLalU1WRASU+ qzz2w1oTmHkIop7VbpXOUGCpBMPOo0q58KkgWBCkTFJXV4awAR4BW+fKdZsE7mm2b4CM /sf/CPdWQIwBVnASPVCfQ6dv3JLXLvFkQg5t7bQR9lg7F6lUtTE+7hJCRkKzJKZSxuaa Uk+w== X-Gm-Message-State: AOAM531NzovhBBmMp+6Ff5qowuQxaNqi4VH6s7SBtfoOYdtYtSeTclgg njRzRYjL+xW/YUsJlf4mfQ53LQ== X-Google-Smtp-Source: ABdhPJxWbOTGDVc1G6oh3lKa6pw1RWZ427lCFMuFYx6YPxy82bJtRwDFaKaxzmpDUepFXHWZc6c6ew== X-Received: by 2002:a17:90a:a88d:: with SMTP id h13mr5339199pjq.61.1617228571747; Wed, 31 Mar 2021 15:09:31 -0700 (PDT) Received: from hermes.local (76-14-218-44.or.wavecable.com. [76.14.218.44]) by smtp.gmail.com with ESMTPSA id f15sm3417858pgg.84.2021.03.31.15.09.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Mar 2021 15:09:31 -0700 (PDT) Date: Wed, 31 Mar 2021 15:09:27 -0700 From: Stephen Hemminger To: Dmitry Kozlyuk Cc: Narcisa Ana Maria Vasile , dev@dpdk.org, thomas@monjalon.net, khot@microsoft.com, navasile@microsoft.com, dmitrym@microsoft.com, roretzla@microsoft.com, talshn@nvidia.com, ocardona@microsoft.com, bruce.richardson@intel.com, david.marchand@redhat.com, pallavi.kadam@intel.com Message-ID: <20210331150927.77663caf@hermes.local> In-Reply-To: <20210401001254.2a6db477@sovereign> References: <1616802771-31578-10-git-send-email-navasile@linux.microsoft.com> <1617057640-24301-1-git-send-email-navasile@linux.microsoft.com> <1617057640-24301-10-git-send-email-navasile@linux.microsoft.com> <20210330141139.728ccea2@hermes.local> <20210401001254.2a6db477@sovereign> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v5 09/10] eal: add EAL argument for setting thread priority X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Thu, 1 Apr 2021 00:12:54 +0300 Dmitry Kozlyuk wrote: > 2021-03-30 14:11 (UTC-0700), Stephen Hemminger: > > On Mon, 29 Mar 2021 15:40:39 -0700 > > Narcisa Ana Maria Vasile wrote: > > > > > From: Narcisa Vasile > > > > > > Allow the user to choose the thread priority through an EAL > > > command line argument. > > > > > > The user can select the thread priority to be either 'normal' > > > or 'critical': > > > --thread-prio normal > > > --thread-prio realtime > > > > > > Signed-off-by: Narcisa Vasile > > > > The discussion internally was that this was intended to resolve issues on Windows. > > So it makes sense for Windows, but it is not something that we want to have on Linux. > > Could you make this Windows only, and add update the documentation please. > > > > I just don't want Linux users discovering it, trying it, then reporting more bugs. > > Can you share more details of that discussion? > Is realtime-critical needed not for busy-polling apps (which indeed cause > starvation), but for interrupt-driven ones to process packets ASAP? > > If it's true, then maybe NetUIO can instead give priority boost to these > threads when notifying them about interrupts (Omar? DmitryM?). This can be > configurable via devargs. One downside is that every kernel driver has to > support it, currently Mellanox bifurcated driver and NetUIO. But they will > need some interrupt-related IOCTLs anyway. A DPDK application typically has cores detected to polling for packets. The temptation is to set those cores to have a real time scheduling policy (SCHED_FIFO, or SCH_RR). The problem is that those priorities run in preference to required kernel functions. So the polling-for-packets threads will starve out the Linux kernel RCU and softirq completion of I/O. This starvation will lead to memory loss (no RCU cleanup) and potential deadlocks (disk I/O never completing). It is possible to use real time priority on Linux but it requires lots of tuning to make sure that the kernel never runs work queues, interrupts or soft irqs on those cores. Lots of changes to /proc, kernel command line, and sysfs tunables. Which is possible on embedded systems but not for general purpose applications. This is already a problem that shows up, but it only happens if the DPDK application writer explcitly calls the setscheduler on those threads. At that point, it is the case where the user has started to manipulate threads, and we have to assume they know the consequences and are ready to deal with them. On Windows, the situation is different so yes, this is necessary.