From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <users-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id 31AF745A6B
for <public@inbox.dpdk.org>; Mon, 30 Sep 2024 17:57:31 +0200 (CEST)
Received: from mails.dpdk.org (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id A78724027C;
Mon, 30 Sep 2024 17:57:30 +0200 (CEST)
Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com
[209.85.216.43]) by mails.dpdk.org (Postfix) with ESMTP id 8AB794014F
for <users@dpdk.org>; Mon, 30 Sep 2024 17:57:29 +0200 (CEST)
Received: by mail-pj1-f43.google.com with SMTP id
98e67ed59e1d1-2e09f67bc39so3653980a91.1
for <users@dpdk.org>; Mon, 30 Sep 2024 08:57:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1727711848;
x=1728316648; darn=dpdk.org;
h=content-transfer-encoding:mime-version:references:in-reply-to
:message-id:subject:cc:to:from:date:from:to:cc:subject:date
:message-id:reply-to;
bh=eSpxEPFUuFka3AnOlIibL5Z3q44kfULOWn7bTxv4b9c=;
b=TwJRoVc8um2lq/CDlUin/4Te4br7LEHTWmRmwLWFspte2qa2cz59pPwOkzPTAzOz7g
Isj6rXzJ749R6aodkYHPVBamK6ERTBScXs8PdV5xyZtYFiMc90udlRQikqp/Rq4LX8vn
L8wSCkhSs2N2Bw0xq7zOAqI7dms556k5euDI7vAqpOJ2GETBoIx3zBfmxqeP9em28KKm
S2uSA1GbD1nmVR4bCpzFGEilRqsmllUMncJQG84I00fp/GAgicIO1B4VjVAly+QqGhEj
0YR8ovPtwGnc3KnxS+NKpo7JcoyfdrOlQ7m585q/mZ6Q2GOUTqvOQqR9V9ZW5sDfWz2F
hprA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1727711848; x=1728316648;
h=content-transfer-encoding:mime-version:references:in-reply-to
:message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc
:subject:date:message-id:reply-to;
bh=eSpxEPFUuFka3AnOlIibL5Z3q44kfULOWn7bTxv4b9c=;
b=wni2zshxG1mMm9GI7X6X7KY4IM3KvM0JA0k1jwRsu/Qt6nu4v/Vvpd6PcpjLSTse+p
80g5qX3NQlp6DHi6+6VtOdcO2rf981R/7mMlj36uKRrCYW9AlbIYAKCg0+n38nscXbI2
54ZNxFBv09lEXKrDV6v71CLqcJWU3fT68s5bqZ4RLJ7cgo/Gz31r1NvXKX1Rit9hBol/
/gRRCA2U7Pp7De3grop/G4k+5sdq0/nKY9SppM6KWVTjJj9hvq//iTl9Sp8COR9daP90
ZZImtiu6uTi1t8fR676csx+hN4ZDvl8iemnxgS9dLnF6x3/amycIK583bU8EGKnTFmHr
NTOw==
X-Gm-Message-State: AOJu0Yx3euSGr5VbV9u3IqNoZ2sdlKtQgdueuB1hdcfohuLTNjwhQYkI
kwvZxQ5TICycsy6/Pi9hQP75gMDfUFiIa/dJ0WJ9jRsk0V2PwnORuqWZgYI4lpY=
X-Google-Smtp-Source: AGHT+IGIEgwkETDoCmzQgMKbC1rlFNGe6HSm2izoezhrJzdcCdiikFVZIglsdSdVF7Vk8yYE7vzYFw==
X-Received: by 2002:a17:90a:cb84:b0:2d3:c4cd:245f with SMTP id
98e67ed59e1d1-2e0b8b1785cmr14291655a91.17.1727711848604;
Mon, 30 Sep 2024 08:57:28 -0700 (PDT)
Received: from hermes.local (204-195-96-226.wavecable.com. [204.195.96.226])
by smtp.gmail.com with ESMTPSA id
98e67ed59e1d1-2e0b6e22cbbsm8129021a91.49.2024.09.30.08.57.27
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Mon, 30 Sep 2024 08:57:28 -0700 (PDT)
Date: Mon, 30 Sep 2024 08:57:26 -0700
From: Stephen Hemminger <stephen@networkplumber.org>
To: amit sehas <cun23@yahoo.com>
Cc: "users@dpdk.org" <users@dpdk.org>
Subject: Re: core performance
Message-ID: <20240930085726.6df70a01@hermes.local>
In-Reply-To: <595544330.11681349.1727123476579@mail.yahoo.com>
References: <595544330.11681349.1727123476579.ref@mail.yahoo.com>
<595544330.11681349.1727123476579@mail.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-BeenThere: users@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK usage discussions <users.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/users>,
<mailto:users-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/users/>
List-Post: <mailto:users@dpdk.org>
List-Help: <mailto:users-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/users>,
<mailto:users-request@dpdk.org?subject=subscribe>
Errors-To: users-bounces@dpdk.org
On Mon, 23 Sep 2024 20:31:16 +0000 (UTC)
amit sehas <cun23@yahoo.com> wrote:
> We are seeing different dpdk threads (launched via=C2=A0rte_eal_remote_la=
unch()), demonstrate very different performance.
Are the DPDK threads running on isolated cpus?
Are the DPDK threads doing any system calls (use strace to check)?
>=20
> After placing counters all over the code, we realize that some threads ar=
e uniformly slow, in other words there is no application level issue that i=
s throttling one thread over the other. We come to the conculsion that eith=
er the Cores on which they are running are not at the same frequency which =
seems doubtful or the threads are not getting a chance to execute on the co=
res uniformly.
>=20
> It seems that isolcpus has been deprecated in recent versions of linux.
>=20
> What is the recommended approach to prevent the kernel from utilizing som=
e CPU threads, for anything other than the threads that are launched on the=
m.
On modern Linux systems, CPU isolation can be achieved with cgroups.
>=20
> Is there some API in dpdk which also helps us determine which CPU core th=
e thread is pinned to?=C2=A0
> I did not find any code in dpdk which actually performed pinning of a thr=
ead to a CPU core.
It is here in lib/eal/linux/eal.c
/* Launch threads, called at application init(). */
int
rte_eal_init(int argc, char **argv)
{
...
RTE_LCORE_FOREACH_WORKER(i) {
...
ret =3D rte_thread_set_affinity_by_id(lcore_config[i].thread_id,
&lcore_config[i].cpuset);
if (ret !=3D 0)
rte_panic("Cannot set affinity\n");
}
>=20
> In our case it is more or less certain that the different threads are sim=
ply not getting the same CPU core time, as a result some are demonstrating =
higher throughput than the others ...
>=20
> how do we fix this?
Did you get profiling info? I would start by getting flame graph using perf.