Re: core performance

[amit sehas <cun23@yahoo.com>]

If there is a way to determine:

vCPU thread utilization numbers over a period of time, such as a few hours

or which processes are consuming the most CPU

top always indicates that the server is consuming the most CPU.

Now i am begining to wonder if 8 vCPU threads really are capable of running 6 high intensity threads or only 4 such threads? Dont know

Also tried to utilize pthread_setschedparam() explicitly on some of the threads, it made no difference to the performance. But if we do it on more than 1-2 threads then it hangs the whole system.

This is primarily a matter of CPU scheduling, and if we restirct context switching on even 2 critical threads we have a win.


regards






On Thursday, September 26, 2024 at 09:56:04 AM PDT, amit sehas <cun23@yahoo.com> wrote: 





Belos is the lscpu that was requested, it appears to suggest an 8 vCPU thread setup ... if am reading it correctly:

$ lscpu
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          46 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   8
  On-line CPU(s) list:    0-7
Vendor ID:                GenuineIntel
  BIOS Vendor ID:         Intel(R) Corporation
  Model name:             Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
    BIOS Model name:      Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
    CPU family:           6
    Model:                85
    Thread(s) per core:   2
    Core(s) per socket:   4
    Socket(s):            1
    Stepping:             7
    BogoMIPS:             4999.99
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 cl
                          flush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc re
                          p_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclm
                          ulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_t
                          imer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpci
                          d_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx5
                          12f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xs
                          aveopt xsavec xgetbv1 xsaves ida arat pku ospke
Virtualization features:
  Hypervisor vendor:      KVM
  Virtualization type:    full
Caches (sum of all):
  L1d:                    128 KiB (4 instances)
  L1i:                    128 KiB (4 instances)
  L2:                     4 MiB (4 instances)
  L3:                     35.8 MiB (1 instance)
NUMA:
  NUMA node(s):           1
  NUMA node0 CPU(s):      0-7
Vulnerabilities:
  Gather data sampling:   Unknown: Dependent on hypervisor status
  Itlb multihit:          KVM: Mitigation: VMX unsupported
  L1tf:                   Mitigation; PTE Inversion
  Mds:                    Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unkn
                          own
  Meltdown:               Mitigation; PTI
  Mmio stale data:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unkn
                          own
  Reg file data sampling: Not affected
  Retbleed:               Vulnerable
  Spec rstack overflow:   Not affected
  Spec store bypass:      Vulnerable
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affec
                          ted; BHI Retpoline
  Srbds:                  Not affected
  Tsx async abort:        Not affected








On Thursday, September 26, 2024 at 05:32:52 AM PDT, amit sehas <cun23@yahoo.com> wrote: 





Simply reordering the launch of different threads brings back a lot of the lost performance, this is a clear evidence that some CPU threads are more predisposed to context switches than the others.

This is a thread scheduling issue at the CPU level as we have expected. In a previous exchange someone has suggested that utilizing rte_thread_set_priority to RTE_THREAD_PRIORITY_REALTIME_CRITICAL is not a good idea 

we should be able to prioritize some threads over the other threads ... since we are utilizing rte_eal_remote_launch, one would think that such a functonality should be a part of the library ...

any ideas folks?

regards






On Tuesday, September 24, 2024 at 01:47:05 PM PDT, amit sehas <cun23@yahoo.com> wrote: 





Thanks for the suggestions, so this is a database server which is doing lots of stuff, not every thread is heavily involved in dpdk packet processing. As a result the guidelines for attaining the most dpdk performance are applicable to only a few threads.

In this particular issue we are specificially looking at CPU scheduling of threads that are primarily heavily processing database queries. These threads, from our measurements, are not being uniformly scheduled on the CPU ...

This is our primary concern, since we utilized rte_eal_remote_launch to spawn the threads, we are wondering if there are any options in this API that will allow us to more uniformly allocate the CPU to threads that are critical...

regards


On Tuesday, September 24, 2024 at 09:38:16 AM PDT, Stephen Hemminger <stephen@networkplumber.org> wrote: 





On Tue, 24 Sep 2024 14:40:49 +0000 (UTC)

amit sehas <cun23@yahoo.com> wrote:

> Thanks for your response, and thanks for your input on the set_priority, 
> 
> The best guess we have at this point is that this is not a dpdk performance issue. This is an issue with some threads running into more context switches than the others and hence not getting the same slice of the CPU. We are certain that this is not a dpdk performance issue, the code
> is uniformly slow in one thread versus the other and the threads are doing a very large amount of work including accessing databases. The threads in question are not really doing packet processing as much as other work.
> 
> So this is certainly not a dpdk performance issue. This is an issue of kernel threads not being scheduled properly or in the worse case the cores running on different frequency (which is quite unlikely no the AWS Xeons we are running this on).
> 
> If you are asking for the dpdk config files to check for dpdk related performance issue then we are quite certain the issue is not with dpdk performance ...

> On Mon, Sep 23, 2024 at 10:06 PM amit sehas <cun23@yahoo.com> wrote:
> > Thanks for your response, i am not sure i understand your question ... we have our product that utilizes dpdk ... the commands are just our server commands and parameters ... and the lscpu is the hyperthreaded 8 thread Xeon instance in AWS ...



The rules of getting performance in DPDK:
  - use DPDK threads (pinned) for datapath
  - use isolated CPU's for those DPDK threads
  - do not do any system calls
  - avoid floating point

You can use tracing tools like strace or BPF to see what the thread is doing.