From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DA53443200 for ; Thu, 26 Oct 2023 02:00:42 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B7836402A3; Thu, 26 Oct 2023 02:00:42 +0200 (CEST) Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) by mails.dpdk.org (Postfix) with ESMTP id F3DBA402A3 for ; Thu, 26 Oct 2023 02:00:40 +0200 (CEST) Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-5a9bf4fbd3fso283637a12.1 for ; Wed, 25 Oct 2023 17:00:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1698278440; x=1698883240; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=W5RlOvmFYzYOdreqard2kHBwB214ThFcEUc5bEX+oes=; b=UA7Jo78KhGqR5D8AAmocXgdjd8GVuymiMEBwKu9gJhOsScRtGA5yGOY0TI/QvlnjlT nRHoSGVddl4pR+THlc7f/ndpmxbTUyWbhsgg7q1LtQcBklerjg6ei60zV3JlDwEegCJB ceinagv7zvkkxwQddKjyZ6C7tROUj3pK7QeI2LK7zjpkhkzdxX/8txtpJuWCf1b/LcHu dRyPsbNiVIBOvOkswsq1XKuM+vmVHXZTlH72jN9IaaWw5uJuy284Iu2K9s6P0CdlY2G2 RvcQyLppUn7/I2ItfzWdU75Mi5kE5ck5pZZEPxutGGIqacBsosbb7955cyRuy045v2Em XmkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698278440; x=1698883240; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W5RlOvmFYzYOdreqard2kHBwB214ThFcEUc5bEX+oes=; b=hwtlQzT7MNZtbtUO5U2fH6ge3Qi84/VleW/AvR1qVP/Q8NOXZEn+BP12uEyVdfYzhV xvc5fWeRUm9cXVG17DfeHv0dfwQapNsGs1D0AtWrSKVFXXwC8hA3MdQS0xyg0dLekkbB lQnY2YKxmF3QIpvuZiVQRheKaiDCvcLl4uC+1IszZWXSOVvDxBUX65E+xM0PKNfFj6vV pIze8KOWLwuy7mFYEAMtWW9MlqUb2cYlCbwoyNWqhyYIMvawTNJeGXfP4ns2bkIlBWiu yOjWOSr4IzA+vNyUdlwJlX+ilMAs2el5qEQ9gHc1ojmS1IrkImY/7BKYGuE854elkgXV hSnA== X-Gm-Message-State: AOJu0YyXjyCVKSTEcU1Zf/WAtBGZc3nnKWlwC1mv7LR8VOiMBrLC+nmH 36QC3XxjR00btrlc0XarA58wOw== X-Google-Smtp-Source: AGHT+IHAX9SobDFKfgWFOPslivTO4Yh82UTpN+caNwjwqMlUp9xjyWvAcpDNGYYXwIG0Q2EgJuVPVg== X-Received: by 2002:a05:6a20:c19b:b0:15d:4a2b:b513 with SMTP id bg27-20020a056a20c19b00b0015d4a2bb513mr8230829pzb.36.1698278439958; Wed, 25 Oct 2023 17:00:39 -0700 (PDT) Received: from hermes.local (204-195-126-68.wavecable.com. [204.195.126.68]) by smtp.gmail.com with ESMTPSA id u22-20020a63d356000000b005af9dcb4756sm1795133pgi.42.2023.10.25.17.00.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 17:00:39 -0700 (PDT) Date: Wed, 25 Oct 2023 17:00:37 -0700 From: Stephen Hemminger To: Morten =?UTF-8?B?QnLDuHJ1cA==?= Cc: "Thomas Monjalon" , , "David Marchand" , , "Anatoly Burakov" , "Dmitry Kozlyuk" , "Narcisa Ana Maria Vasile" , "Dmitry Malloy" , "Pallavi Kadam" , "Tyler Retzlaff" , "Andrew Rybchenko" , "Konstantin Ananyev" Subject: Re: [PATCH v2] eal/unix: allow creating thread with real-time priority Message-ID: <20231025170037.70d7471c@hermes.local> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35E9EF8B@smartserver.smartshare.dk> References: <20231024125416.798897-1-thomas@monjalon.net> <20231025151352.995318-1-thomas@monjalon.net> <20231025083700.4e3e274c@hermes.local> <23265462.6Emhk5qWAg@thomas> <98CBD80474FA8B44BF855DF32C47DC35E9EF8B@smartserver.smartshare.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org On Wed, 25 Oct 2023 19:54:06 +0200 Morten Br=C3=B8rup wrote: > Someone might build a kernel with options to keep non-dataplane threads o= ff some dedicated CPU cores, so they can be used for guaranteed low-latency= dataplane threads. We do. We don't use real-time priority, though. >=20 > For reference, we did some experiments (using this custom built kernel) w= ith a dedicated thread doing nothing but a loop calling rte_rdtsc_precise()= and registering the delta. Although the overwhelming majority is ca. CPU 8= 0 cycles, there are some big outliers at ca. 9,000 CPU cycles. (Order of ma= gnitude: ca. 45 of these big outliers per minute.) Apparently some kernel t= hreads steal some cycles from this thread, regardless of our customizations= . We haven't bothered analyzing and optimizing it further. >=20 > I think our experiment supports the need to allow kernel threads to run, = e.g. by calling sleep() or similar, when an EAL thread has real-time priori= ty. First. We need to dispel the myth that on Linux real time is faster. It isn't, you can ask the RT kernel developers if you need more support. The purpose of RT is to have user process respond in a deterministic fashion to a kernel event (ie. usually HW interrupt or IPC). In most cases, this is not how DPDK applications are written. It is possible to use hardware inter= rupts in DPDK, but getting it right is hard, and the latency of HW to kernel to D= PDK in userspace is a long time. RT will make it more consistent, but it won't remove the overhead; i.e less long tail with RT, but average latency will still be too long for most network applications. Users say "but my polling thread is getting latency", then you need to make= sure your application is running on dedicated cores and the system is configured= to not use those cores for HW events. Doing RT won't fix that. Then some user say "but I want to run multiple polling threads on a single = core". That plain won't work no matter what the priority.