Re: core performance

[Stephen Hemminger <stephen@networkplumber.org>]

On Mon, 23 Sep 2024 20:31:16 +0000 (UTC)
amit sehas <cun23@yahoo.com> wrote:

> We are seeing different dpdk threads (launched via rte_eal_remote_launch()), demonstrate very different performance.


Are the DPDK threads running on isolated cpus?
Are the DPDK threads doing any system calls (use strace to check)?

> 
> After placing counters all over the code, we realize that some threads are uniformly slow, in other words there is no application level issue that is throttling one thread over the other. We come to the conculsion that either the Cores on which they are running are not at the same frequency which seems doubtful or the threads are not getting a chance to execute on the cores uniformly.
> 
> It seems that isolcpus has been deprecated in recent versions of linux.
> 
> What is the recommended approach to prevent the kernel from utilizing some CPU threads, for anything other than the threads that are launched on them.

On modern Linux systems, CPU isolation can be achieved with cgroups.

> 
> Is there some API in dpdk which also helps us determine which CPU core the thread is pinned to? 
> I did not find any code in dpdk which actually performed pinning of a thread to a CPU core.

It is here in lib/eal/linux/eal.c

/* Launch threads, called at application init(). */
int
rte_eal_init(int argc, char **argv)
{
...
	RTE_LCORE_FOREACH_WORKER(i) {
...
		ret = rte_thread_set_affinity_by_id(lcore_config[i].thread_id,
			&lcore_config[i].cpuset);
		if (ret != 0)
			rte_panic("Cannot set affinity\n");
	}



> 
> In our case it is more or less certain that the different threads are simply not getting the same CPU core time, as a result some are demonstrating higher throughput than the others ...
> 
> how do we fix this?

Did you get profiling info? I would start by getting flame graph using perf.