From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D268543200; Thu, 26 Oct 2023 02:00:43 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E766040E54; Thu, 26 Oct 2023 02:00:42 +0200 (CEST) Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) by mails.dpdk.org (Postfix) with ESMTP id EE6924027F for ; Thu, 26 Oct 2023 02:00:40 +0200 (CEST) Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-578b4997decso290097a12.0 for ; Wed, 25 Oct 2023 17:00:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1698278440; x=1698883240; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=W5RlOvmFYzYOdreqard2kHBwB214ThFcEUc5bEX+oes=; b=UA7Jo78KhGqR5D8AAmocXgdjd8GVuymiMEBwKu9gJhOsScRtGA5yGOY0TI/QvlnjlT nRHoSGVddl4pR+THlc7f/ndpmxbTUyWbhsgg7q1LtQcBklerjg6ei60zV3JlDwEegCJB ceinagv7zvkkxwQddKjyZ6C7tROUj3pK7QeI2LK7zjpkhkzdxX/8txtpJuWCf1b/LcHu dRyPsbNiVIBOvOkswsq1XKuM+vmVHXZTlH72jN9IaaWw5uJuy284Iu2K9s6P0CdlY2G2 RvcQyLppUn7/I2ItfzWdU75Mi5kE5ck5pZZEPxutGGIqacBsosbb7955cyRuy045v2Em XmkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698278440; x=1698883240; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W5RlOvmFYzYOdreqard2kHBwB214ThFcEUc5bEX+oes=; b=JsCJMLHmy3GBIZ3Im6CqViSMmpGitJbM+xGuOURv2eEQftW8fS+C4jS8R72b+dKS1d NIYn/flBh/RLVhf9di3zBWH6l2ODtnq4yywSehWBhEjhAsmi3XhoEvkmppu6XzoIY3pk GoTgmhiwdFHSzx7BUuQ5raGVeByIk82uI2Bhrh9SWTsUoruMI3+KLuphCH8bIJraYSU8 gmrKEy2ViajRKvHQkWtyZFaxewObbMEx23C3CRwEZ0k2l9cY2DLV5IpZYKjUF9opjkjk BbSqHZo7BIFd6LedxaHoQ9u2DQJ/YTbFr0Kzgje2t8olaMgQpGXs5x8PxxBs95VYR3dw kN6A== X-Gm-Message-State: AOJu0Ywfn3d86SllMXOeI/XYSNZEGPs75ZXWo0SX8WC1ilZ6jqMn8p5A TxfNxv/lvEVcObD+VhOmxB8Z0A== X-Google-Smtp-Source: AGHT+IHAX9SobDFKfgWFOPslivTO4Yh82UTpN+caNwjwqMlUp9xjyWvAcpDNGYYXwIG0Q2EgJuVPVg== X-Received: by 2002:a05:6a20:c19b:b0:15d:4a2b:b513 with SMTP id bg27-20020a056a20c19b00b0015d4a2bb513mr8230829pzb.36.1698278439958; Wed, 25 Oct 2023 17:00:39 -0700 (PDT) Received: from hermes.local (204-195-126-68.wavecable.com. [204.195.126.68]) by smtp.gmail.com with ESMTPSA id u22-20020a63d356000000b005af9dcb4756sm1795133pgi.42.2023.10.25.17.00.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 17:00:39 -0700 (PDT) Date: Wed, 25 Oct 2023 17:00:37 -0700 From: Stephen Hemminger To: Morten =?UTF-8?B?QnLDuHJ1cA==?= Cc: "Thomas Monjalon" , , "David Marchand" , , "Anatoly Burakov" , "Dmitry Kozlyuk" , "Narcisa Ana Maria Vasile" , "Dmitry Malloy" , "Pallavi Kadam" , "Tyler Retzlaff" , "Andrew Rybchenko" , "Konstantin Ananyev" Subject: Re: [PATCH v2] eal/unix: allow creating thread with real-time priority Message-ID: <20231025170037.70d7471c@hermes.local> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35E9EF8B@smartserver.smartshare.dk> References: <20231024125416.798897-1-thomas@monjalon.net> <20231025151352.995318-1-thomas@monjalon.net> <20231025083700.4e3e274c@hermes.local> <23265462.6Emhk5qWAg@thomas> <98CBD80474FA8B44BF855DF32C47DC35E9EF8B@smartserver.smartshare.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Wed, 25 Oct 2023 19:54:06 +0200 Morten Br=C3=B8rup wrote: > Someone might build a kernel with options to keep non-dataplane threads o= ff some dedicated CPU cores, so they can be used for guaranteed low-latency= dataplane threads. We do. We don't use real-time priority, though. >=20 > For reference, we did some experiments (using this custom built kernel) w= ith a dedicated thread doing nothing but a loop calling rte_rdtsc_precise()= and registering the delta. Although the overwhelming majority is ca. CPU 8= 0 cycles, there are some big outliers at ca. 9,000 CPU cycles. (Order of ma= gnitude: ca. 45 of these big outliers per minute.) Apparently some kernel t= hreads steal some cycles from this thread, regardless of our customizations= . We haven't bothered analyzing and optimizing it further. >=20 > I think our experiment supports the need to allow kernel threads to run, = e.g. by calling sleep() or similar, when an EAL thread has real-time priori= ty. First. We need to dispel the myth that on Linux real time is faster. It isn't, you can ask the RT kernel developers if you need more support. The purpose of RT is to have user process respond in a deterministic fashion to a kernel event (ie. usually HW interrupt or IPC). In most cases, this is not how DPDK applications are written. It is possible to use hardware inter= rupts in DPDK, but getting it right is hard, and the latency of HW to kernel to D= PDK in userspace is a long time. RT will make it more consistent, but it won't remove the overhead; i.e less long tail with RT, but average latency will still be too long for most network applications. Users say "but my polling thread is getting latency", then you need to make= sure your application is running on dedicated cores and the system is configured= to not use those cores for HW events. Doing RT won't fix that. Then some user say "but I want to run multiple polling threads on a single = core". That plain won't work no matter what the priority.