From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 31AF745A6B for ; Mon, 30 Sep 2024 17:57:31 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A78724027C; Mon, 30 Sep 2024 17:57:30 +0200 (CEST) Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by mails.dpdk.org (Postfix) with ESMTP id 8AB794014F for ; Mon, 30 Sep 2024 17:57:29 +0200 (CEST) Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2e09f67bc39so3653980a91.1 for ; Mon, 30 Sep 2024 08:57:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1727711848; x=1728316648; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=eSpxEPFUuFka3AnOlIibL5Z3q44kfULOWn7bTxv4b9c=; b=TwJRoVc8um2lq/CDlUin/4Te4br7LEHTWmRmwLWFspte2qa2cz59pPwOkzPTAzOz7g Isj6rXzJ749R6aodkYHPVBamK6ERTBScXs8PdV5xyZtYFiMc90udlRQikqp/Rq4LX8vn L8wSCkhSs2N2Bw0xq7zOAqI7dms556k5euDI7vAqpOJ2GETBoIx3zBfmxqeP9em28KKm S2uSA1GbD1nmVR4bCpzFGEilRqsmllUMncJQG84I00fp/GAgicIO1B4VjVAly+QqGhEj 0YR8ovPtwGnc3KnxS+NKpo7JcoyfdrOlQ7m585q/mZ6Q2GOUTqvOQqR9V9ZW5sDfWz2F hprA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727711848; x=1728316648; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eSpxEPFUuFka3AnOlIibL5Z3q44kfULOWn7bTxv4b9c=; b=wni2zshxG1mMm9GI7X6X7KY4IM3KvM0JA0k1jwRsu/Qt6nu4v/Vvpd6PcpjLSTse+p 80g5qX3NQlp6DHi6+6VtOdcO2rf981R/7mMlj36uKRrCYW9AlbIYAKCg0+n38nscXbI2 54ZNxFBv09lEXKrDV6v71CLqcJWU3fT68s5bqZ4RLJ7cgo/Gz31r1NvXKX1Rit9hBol/ /gRRCA2U7Pp7De3grop/G4k+5sdq0/nKY9SppM6KWVTjJj9hvq//iTl9Sp8COR9daP90 ZZImtiu6uTi1t8fR676csx+hN4ZDvl8iemnxgS9dLnF6x3/amycIK583bU8EGKnTFmHr NTOw== X-Gm-Message-State: AOJu0Yx3euSGr5VbV9u3IqNoZ2sdlKtQgdueuB1hdcfohuLTNjwhQYkI kwvZxQ5TICycsy6/Pi9hQP75gMDfUFiIa/dJ0WJ9jRsk0V2PwnORuqWZgYI4lpY= X-Google-Smtp-Source: AGHT+IGIEgwkETDoCmzQgMKbC1rlFNGe6HSm2izoezhrJzdcCdiikFVZIglsdSdVF7Vk8yYE7vzYFw== X-Received: by 2002:a17:90a:cb84:b0:2d3:c4cd:245f with SMTP id 98e67ed59e1d1-2e0b8b1785cmr14291655a91.17.1727711848604; Mon, 30 Sep 2024 08:57:28 -0700 (PDT) Received: from hermes.local (204-195-96-226.wavecable.com. [204.195.96.226]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e0b6e22cbbsm8129021a91.49.2024.09.30.08.57.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Sep 2024 08:57:28 -0700 (PDT) Date: Mon, 30 Sep 2024 08:57:26 -0700 From: Stephen Hemminger To: amit sehas Cc: "users@dpdk.org" Subject: Re: core performance Message-ID: <20240930085726.6df70a01@hermes.local> In-Reply-To: <595544330.11681349.1727123476579@mail.yahoo.com> References: <595544330.11681349.1727123476579.ref@mail.yahoo.com> <595544330.11681349.1727123476579@mail.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Mon, 23 Sep 2024 20:31:16 +0000 (UTC) amit sehas wrote: > We are seeing different dpdk threads (launched via=C2=A0rte_eal_remote_la= unch()), demonstrate very different performance. Are the DPDK threads running on isolated cpus? Are the DPDK threads doing any system calls (use strace to check)? >=20 > After placing counters all over the code, we realize that some threads ar= e uniformly slow, in other words there is no application level issue that i= s throttling one thread over the other. We come to the conculsion that eith= er the Cores on which they are running are not at the same frequency which = seems doubtful or the threads are not getting a chance to execute on the co= res uniformly. >=20 > It seems that isolcpus has been deprecated in recent versions of linux. >=20 > What is the recommended approach to prevent the kernel from utilizing som= e CPU threads, for anything other than the threads that are launched on the= m. On modern Linux systems, CPU isolation can be achieved with cgroups. >=20 > Is there some API in dpdk which also helps us determine which CPU core th= e thread is pinned to?=C2=A0 > I did not find any code in dpdk which actually performed pinning of a thr= ead to a CPU core. It is here in lib/eal/linux/eal.c /* Launch threads, called at application init(). */ int rte_eal_init(int argc, char **argv) { ... RTE_LCORE_FOREACH_WORKER(i) { ... ret =3D rte_thread_set_affinity_by_id(lcore_config[i].thread_id, &lcore_config[i].cpuset); if (ret !=3D 0) rte_panic("Cannot set affinity\n"); } >=20 > In our case it is more or less certain that the different threads are sim= ply not getting the same CPU core time, as a result some are demonstrating = higher throughput than the others ... >=20 > how do we fix this? Did you get profiling info? I would start by getting flame graph using perf.