Re: core performance

[amit sehas <cun23@yahoo.com>]

Simply reordering the launch of different threads brings back a lot of the lost performance, this is a clear evidence that some CPU threads are more predisposed to context switches than the others.

This is a thread scheduling issue at the CPU level as we have expected. In a previous exchange someone has suggested that utilizing rte_thread_set_priority to RTE_THREAD_PRIORITY_REALTIME_CRITICAL is not a good idea 

we should be able to prioritize some threads over the other threads ... since we are utilizing rte_eal_remote_launch, one would think that such a functonality should be a part of the library ...

any ideas folks?

regards






On Tuesday, September 24, 2024 at 01:47:05 PM PDT, amit sehas <cun23@yahoo.com> wrote: 





Thanks for the suggestions, so this is a database server which is doing lots of stuff, not every thread is heavily involved in dpdk packet processing. As a result the guidelines for attaining the most dpdk performance are applicable to only a few threads.

In this particular issue we are specificially looking at CPU scheduling of threads that are primarily heavily processing database queries. These threads, from our measurements, are not being uniformly scheduled on the CPU ...

This is our primary concern, since we utilized rte_eal_remote_launch to spawn the threads, we are wondering if there are any options in this API that will allow us to more uniformly allocate the CPU to threads that are critical...

regards


On Tuesday, September 24, 2024 at 09:38:16 AM PDT, Stephen Hemminger <stephen@networkplumber.org> wrote: 





On Tue, 24 Sep 2024 14:40:49 +0000 (UTC)

amit sehas <cun23@yahoo.com> wrote:

> Thanks for your response, and thanks for your input on the set_priority, 
> 
> The best guess we have at this point is that this is not a dpdk performance issue. This is an issue with some threads running into more context switches than the others and hence not getting the same slice of the CPU. We are certain that this is not a dpdk performance issue, the code
> is uniformly slow in one thread versus the other and the threads are doing a very large amount of work including accessing databases. The threads in question are not really doing packet processing as much as other work.
> 
> So this is certainly not a dpdk performance issue. This is an issue of kernel threads not being scheduled properly or in the worse case the cores running on different frequency (which is quite unlikely no the AWS Xeons we are running this on).
> 
> If you are asking for the dpdk config files to check for dpdk related performance issue then we are quite certain the issue is not with dpdk performance ...

> On Mon, Sep 23, 2024 at 10:06 PM amit sehas <cun23@yahoo.com> wrote:
> > Thanks for your response, i am not sure i understand your question ... we have our product that utilizes dpdk ... the commands are just our server commands and parameters ... and the lscpu is the hyperthreaded 8 thread Xeon instance in AWS ...



The rules of getting performance in DPDK:
  - use DPDK threads (pinned) for datapath
  - use isolated CPU's for those DPDK threads
  - do not do any system calls
  - avoid floating point

You can use tracing tools like strace or BPF to see what the thread is doing.