From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8672045A05 for ; Tue, 24 Sep 2024 03:14:16 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1F7D74027C; Tue, 24 Sep 2024 03:14:16 +0200 (CEST) Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) by mails.dpdk.org (Postfix) with ESMTP id DF36F4025D for ; Tue, 24 Sep 2024 03:14:14 +0200 (CEST) Received: by mail-ej1-f49.google.com with SMTP id a640c23a62f3a-a910860e4dcso242022366b.3 for ; Mon, 23 Sep 2024 18:14:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1727140454; x=1727745254; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=060TZY/oMJ0y50T80FB8ZCng/UqVYPyXL+B8UaB3r4A=; b=naW4bsAhxyaGq6IEWZ4MMGYN/u8tcAA1XGOliN5+IPjPm3lTjSHjeO2On0HQKgdpBI OfwD7n3eWbh+UgGeYRBCwuhzSifxxkIapBeiFO7M4I1eAkfWFWGZc2E73kSE15iudPAR +6sPe1GY7shbVr6lqVNDAxZ6Wptv76lfjOKXcyCi2RLV//sQl7QmqSO4TmeA0m6u3lEn UukPQNEmaCvSjHsv4Owi0vDL92WK5D/f3YFtVuEyKgAKuEn4g6zWLXSOptxy/iNZ3sXo TcZiBUqTbftgfB+16cKXrA+IrpfuTHYydUvlmwb+Y0xh7fa8twDb3owRCo5hz8Rdxk1R i+Bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727140454; x=1727745254; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=060TZY/oMJ0y50T80FB8ZCng/UqVYPyXL+B8UaB3r4A=; b=raha1L5ufI3pscAK2NF3Mtb96Hv2wUNSYWDyg1HLUPqolrr4dUF64cZV+ilwssYm7b cCzuVmlMkvuKRpRsfzlb5PKb+XOO8umNWwsMSuhRLqr8/VIMsOHbkPdkbliCWZub6eq5 pYcazM7tUqPvZ0TjI+7DSVReJTvdc9TzHcMd4FPi9EdE/AsL45dBxbm0psfnP39PPxbq dxM5zXtL3G9Sla1Bm7mugqWq6BK9gwyZTz2pvFxEhRTFIQBd1QGGVsuihJG0RTTHpW28 AaE/ktMG1rEB2QFR4RyGxucwiP72KxSFrzEaM+PN4FO0L5Ky3gnRkCbtcmkwxHAkqJ2M tgyQ== X-Forwarded-Encrypted: i=1; AJvYcCWJDnCOzOj7fGTUUXjDY1FBR2GteK9XXi0sPK6w5LMK7vSKTaCiippj+z6/lvk89Pg20FEHXQ==@dpdk.org X-Gm-Message-State: AOJu0YzEYrdosULyV8Kqc2sbriYnBKYjyaT8rLpepy2sBejJT+2duZMM J8sEPWJyHnsBXWOXAnpABWJ+npsEJa6Cj5P1vOmMhDvfX5fdpw/o5TFXonVAK4O7yeZpYwCPasM 9rkxcaAlEbTkLcX7qcDxpgaQaogeYUQ== X-Google-Smtp-Source: AGHT+IFFIiSmw8I7v3olGS2MCPraJBJDh2NenMFHjlFP+qofkjcOQ4ZdGibs5MwgdbYBBjyHJ4IuKrZm3PHQiAxyM0E= X-Received: by 2002:a17:907:7295:b0:a8d:11c2:2b4 with SMTP id a640c23a62f3a-a90d517ef1fmr1568116566b.56.1727140454198; Mon, 23 Sep 2024 18:14:14 -0700 (PDT) MIME-Version: 1.0 References: <1987164393.11670398.1727125003663.ref@mail.yahoo.com> <1987164393.11670398.1727125003663@mail.yahoo.com> <1299564509.11731667.1727133474900@mail.yahoo.com> In-Reply-To: <1299564509.11731667.1727133474900@mail.yahoo.com> From: Nishant Verma Date: Mon, 23 Sep 2024 21:14:02 -0400 Message-ID: Subject: Re: core performance To: amit sehas Cc: Wisam Jaddo , "users@dpdk.org" Content-Type: multipart/alternative; boundary="0000000000003ff7950622d33954" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --0000000000003ff7950622d33954 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Can you share output of lscpu and command you are using to execute the app? . Regards, Nishant Verma On Mon, Sep 23, 2024 at 7:17=E2=80=AFPM amit sehas wrote: > Thanks for the responses, this is on AWS, which is utilizing Xeon with > hyperthreading. Not utilizing hyperthreading is not an option. > > After trying a few things i am narrowing down on the following approach: > > only for the critical threads we could utilize: rte_thread_set_priority > to RTE_THREAD_PRIORITY_REALTIME_CRITICAL > > however this API requires a rte_thread_t parameter, if we utilize > rte_eal_remote_launch, we are not provided with this parameter. > I am searching through the code to see if there is an API where i can > obtain the rte_thread_t for the current thread that was launched with > rte_eal_remote_launch. > > regards > > > > > > > On Monday, September 23, 2024 at 03:18:11 PM PDT, Nishant Verma < > vnish11@gmail.com> wrote: > > > > > > Also make sure all core you are using are physical core not the logical > core. > Secondly, check your core isolation options and apply them accordingly. > > > . > > Regards, > Nishant Verma > > > On Mon, Sep 23, 2024 at 6:04=E2=80=AFPM Wisam Jaddo w= rote: > > Hello Amit, > > > >> -----Original Message----- > >> From: amit sehas > >> Sent: Monday, September 23, 2024 11:57 PM > >> To: users@dpdk.org > >> Subject: core performance > >> > >> We are seeing different dpdk threads (launched > via rte_eal_remote_launch()), > >> demonstrate very different performance. > >> > >> > >> > >> After placing counters all over the code, we realize that some threads > are > >> uniformly slow, in other words there is no application level issue tha= t > is > >> throttling one thread over the other. We come to the conculsion that > either > >> the Cores on which they are running are not at the same frequency whic= h > >> seems doubtful or the threads are not getting a chance to execute on > the cores > >> uniformly. > >> > >> > >> > >> It seems that isolcpus has been deprecated in recent versions of linux= . > >> > >> > >> > >> What is the recommended approach to prevent the kernel from utilizing > some > >> CPU threads, for anything other than the threads that are launched on > them. > > > > If you are wishing to run each thread on separate core, try to use > rte_eal_mp_remote_launch() > > instead of rte_eal_remote_launch(), make sure that your CPU is isolated= , > and you are passing correct > > Cores that were isolated to your app using -c, -l. > > > > > >> > >> > >> > >> Is there some API in dpdk which also helps us determine which CPU core > the > >> thread is pinned to? > >> > >> I did not find any code in dpdk which actually performed pinning of a > thread to > >> a CPU core. > >> > >> > >> > >> In our case it is more or less certain that the different threads are > simply not > >> getting the same CPU core time, as a result some are demonstrating > higher > >> throughput than the others ... > >> > >> > >> > >> how do we fix this? > > > > BRs, > > Wisam Jaddo > > > > --0000000000003ff7950622d33954 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Can you share output of lscpu and command you are using t= o execute the app?

.

Regards,
Nishant Verma


On Mon, Sep 23, 2024 at 7:17=E2=80=AFPM amit = sehas <cun23@yahoo.com> wrote:=
Thanks for the responses, this is on AWS, which = is utilizing Xeon with hyperthreading. Not utilizing hyperthreading is not = an option.

After trying a few things i am narrowing down on the following approach:
only for the critical threads we could utilize:=C2=A0 rte_thread_set_priori= ty to=C2=A0RTE_THREAD_PRIORITY_REALTIME_CRITICAL

however this API requires a rte_thread_t parameter, if we utilize rte_eal_r= emote_launch, we are not provided with this parameter.
I am searching through the code to see if there is an API where i can obtai= n the rte_thread_t for the current thread that was launched with rte_eal_re= mote_launch.

regards






On Monday, September 23, 2024 at 03:18:11 PM PDT, Nishant Verma <vnish11@gmail.com> w= rote:





Also make sure all core you are using are physical core not the logical cor= e.=C2=A0
Secondly, check your core isolation options and apply them accordingly.


.

Regards,
Nishant Verma


On Mon, Sep 23, 2024 at 6:04=E2=80=AFPM Wisam Jaddo <wisamm@nvidia.com> wrote:
> Hello Amit,
>
>> -----Original Message-----
>> From: amit sehas <cun23@yahoo.com>
>> Sent: Monday, September 23, 2024 11:57 PM
>> To: users@dpdk= .org
>> Subject: core performance
>>
>> We are seeing different dpdk threads (launched via=C2=A0rte_eal_re= mote_launch()),
>> demonstrate very different performance.
>>
>>
>>
>> After placing counters all over the code, we realize that some thr= eads are
>> uniformly slow, in other words there is no application level issue= that is
>> throttling one thread over the other. We come to the conculsion th= at either
>> the Cores on which they are running are not at the same frequency = which
>> seems doubtful or the threads are not getting a chance to execute = on the cores
>> uniformly.
>>
>>
>>
>> It seems that isolcpus has been deprecated in recent versions of l= inux.
>>
>>
>>
>> What is the recommended approach to prevent the kernel from utiliz= ing some
>> CPU threads, for anything other than the threads that are launched= on them.
>
> If you are wishing to run each thread on separate core, try to use rte= _eal_mp_remote_launch()
> instead of rte_eal_remote_launch(), make sure that your CPU is isolate= d, and you are passing correct
> Cores that were isolated to your app using -c, -l.
>
>
>>
>>
>>
>> Is there some API in dpdk which also helps us determine which CPU = core the
>> thread is pinned to?
>>
>> I did not find any code in dpdk which actually performed pinning o= f a thread to
>> a CPU core.
>>
>>
>>
>> In our case it is more or less certain that the different threads = are simply not
>> getting the same CPU core time, as a result some are demonstrating= higher
>> throughput than the others ...
>>
>>
>>
>> how do we fix this?
>
> BRs,
> Wisam Jaddo
>

--0000000000003ff7950622d33954--