Re: core performance

[amit sehas <cun23@yahoo.com>]

Thanks for your response, and thanks for your input on the set_priority, 

The best guess we have at this point is that this is not a dpdk performance issue. This is an issue with some threads running into more context switches than the others and hence not getting the same slice of the CPU. We are certain that this is not a dpdk performance issue, the code
is uniformly slow in one thread versus the other and the threads are doing a very large amount of work including accessing databases. The threads in question are not really doing packet processing as much as other work.

So this is certainly not a dpdk performance issue. This is an issue of kernel threads not being scheduled properly or in the worse case the cores running on different frequency (which is quite unlikely no the AWS Xeons we are running this on).

If you are asking for the dpdk config files to check for dpdk related performance issue then we are quite certain the issue is not with dpdk performance ...


regards






On Tuesday, September 24, 2024 at 06:23:39 AM PDT, Nishant Verma <vnish11@gmail.com> wrote: 

I assume you are using any variant of linux. So execute command "lscpu" and provide the output.
Also share your command or config file that provides application to know which core to use and how many memory channels and ports.

Thanks.

Regards,
Nishant Verma


On Mon, Sep 23, 2024 at 10:06 PM amit sehas <cun23@yahoo.com> wrote:
> Thanks for your response, i am not sure i understand your question ... we have our product that utilizes dpdk ... the commands are just our server commands and parameters ... and the lscpu is the hyperthreaded 8 thread Xeon instance in AWS ...
> 
> regards
> 
> 
> 
> 
> 
> 
> On Monday, September 23, 2024 at 06:14:16 PM PDT, Nishant Verma <vnish11@gmail.com> wrote: 
> 
> 
> 
> 
> 
> Can you share output of lscpu and command you are using to execute the app?
> 
> .
> 
> Regards,
> Nishant Verma
> 
> 
> On Mon, Sep 23, 2024 at 7:17 PM amit sehas <cun23@yahoo.com> wrote:
>> Thanks for the responses, this is on AWS, which is utilizing Xeon with hyperthreading. Not utilizing hyperthreading is not an option.
>> 
>> After trying a few things i am narrowing down on the following approach:
>> 
>> only for the critical threads we could utilize:  rte_thread_set_priority to RTE_THREAD_PRIORITY_REALTIME_CRITICAL
>> 
>> however this API requires a rte_thread_t parameter, if we utilize rte_eal_remote_launch, we are not provided with this parameter.
>> I am searching through the code to see if there is an API where i can obtain the rte_thread_t for the current thread that was launched with rte_eal_remote_launch.
>> 
>> regards
>> 
>> 
>> 
>> 
>> 
>> 
>> On Monday, September 23, 2024 at 03:18:11 PM PDT, Nishant Verma <vnish11@gmail.com> wrote: 
>> 
>> 
>> 
>> 
>> 
>> Also make sure all core you are using are physical core not the logical core. 
>> Secondly, check your core isolation options and apply them accordingly.
>> 
>> 
>> .
>> 
>> Regards,
>> Nishant Verma
>> 
>> 
>> On Mon, Sep 23, 2024 at 6:04 PM Wisam Jaddo <wisamm@nvidia.com> wrote:
>>> Hello Amit,
>>> 
>>>> -----Original Message-----
>>>> From: amit sehas <cun23@yahoo.com>
>>>> Sent: Monday, September 23, 2024 11:57 PM
>>>> To: users@dpdk.org
>>>> Subject: core performance
>>>> 
>>>> We are seeing different dpdk threads (launched via rte_eal_remote_launch()),
>>>> demonstrate very different performance.
>>>> 
>>>> 
>>>> 
>>>> After placing counters all over the code, we realize that some threads are
>>>> uniformly slow, in other words there is no application level issue that is
>>>> throttling one thread over the other. We come to the conculsion that either
>>>> the Cores on which they are running are not at the same frequency which
>>>> seems doubtful or the threads are not getting a chance to execute on the cores
>>>> uniformly.
>>>> 
>>>> 
>>>> 
>>>> It seems that isolcpus has been deprecated in recent versions of linux.
>>>> 
>>>> 
>>>> 
>>>> What is the recommended approach to prevent the kernel from utilizing some
>>>> CPU threads, for anything other than the threads that are launched on them.
>>> 
>>> If you are wishing to run each thread on separate core, try to use rte_eal_mp_remote_launch()
>>> instead of rte_eal_remote_launch(), make sure that your CPU is isolated, and you are passing correct
>>> Cores that were isolated to your app using -c, -l.
>>> 
>>> 
>>>> 
>>>> 
>>>> 
>>>> Is there some API in dpdk which also helps us determine which CPU core the
>>>> thread is pinned to?
>>>> 
>>>> I did not find any code in dpdk which actually performed pinning of a thread to
>>>> a CPU core.
>>>> 
>>>> 
>>>> 
>>>> In our case it is more or less certain that the different threads are simply not
>>>> getting the same CPU core time, as a result some are demonstrating higher
>>>> throughput than the others ...
>>>> 
>>>> 
>>>> 
>>>> how do we fix this?
>>> 
>>> BRs,
>>> Wisam Jaddo
>>> 
>> 
>> 
> 
>