RE: ARM power-saving measurements

[Feifei Wang <Feifei.Wang2@arm.com>]

Hi, Alex

Thanks for your attention.
Following I will share how to measure the power savings in Arm:

1.Test Setup:
1.1  ensure the dpdk version has been applied the wfe patch:
http://patches.dpdk.org/project/dpdk/cover/20230220085109.3463640-1-feifei.wang2@arm.com/
This is due to dpdk cannot support l3fwd-power  'power monitor' in arm before this patch

1.2 enable wfe instructions in arm:
dpdk/config/arm/meson.build:
'RTE_ARM_USE_WFE', false -> 'RTE_ARM_USE_WFE', true

1.3 build l3fwd-power
meson -Dexamples=l3fwd-power build  &&  cd build  &&  ninja

2. Test Command:
L3fwd-power uses 'power monitor' mode
sudo ./examples/dpdk-l3fwd-power -l 14-15 -n 4 -a 0000:13:00.0 -a 0000:13:00.1 -- -p 0x3 -P --config='(0,0,14),(1,0,15)' --pmd-mgmt=monitor
note: power monitor can just support more than 1 cores.

3. Power saving count:
For arm, wfe instructions can let CPU go into sleeping mode if it is always idle.
Thus, we have 2 ways to collect power saving info.

3.1 Using perf to record dynamic instruction number within 1 second:
$sudo perf stat -C <cpu id>, and then we can get CPU instructions number during a fixed time slot.
For examples:
Performance counter stats for 'CPU(s) 14':
          3,506.56 msec cpu-clock                 #    1.000 CPUs utilized
                 0      context-switches          #    0.000 K/sec
                 0      cpu-migrations            #    0.000 K/sec
                 0      page-faults               #    0.000 K/sec
     8,768,815,778      cycles                    #    2.501 GHz
        26,071,802      instructions              #    0.00  insn per cycle
   <not supported>      branches
           104,982      branch-misses

       3.506978873 seconds time elapsed

Dynamic instructions number for 1 second = 26,071,802/3.506978873

If we disable wfe in arm, we can find 'dynamic instructions number for 1 second'
increases significantly. This also means CPU is running at full load.

3.2 Get CPU power form BMC:
For ampere altra and thunderx2 server, we can look up CPU timely power from BMC.
For example, when we run l3fwd-power in thunderx2, server information->power->consumption:
-----------------------------------------------------------------------------------
CPU0 Current Power Consumption	69 W | 235 BTU/hr
CPU1 Current Power Consumption	0 W | 0 BTU/hr
CPU Power Limit Management Activated	
CPU Power Limit in Watts(1-150)	
-----------------------------------------------------------------------------------
If we disable wfe, CPU0 Current Power Consumption will increase, then we can know how much power we can save.

Best Regards
Feifei

> -----Original Message-----
> From: Alexander Kozyrev <akozyrev@nvidia.com>
> Sent: Saturday, April 1, 2023 1:29 AM
> To: Feifei Wang <Feifei.Wang2@arm.com>; Ruifeng Wang
> <Ruifeng.Wang@arm.com>
> Cc: dev@dpdk.org
> Subject: RE: ARM power-saving measurements
> 
> Hi Feifei/Ruifeng, any advice on power saving measurements on ARM?
> 
> Regards,
> Alex
> 
> > -----Original Message-----
> > From: Alexander Kozyrev
> > Sent: March 13, 2023 18:08
> > To: Feifei Wang <feifei.wang2@arm.com>; ruifeng.wang@arm.com
> > Subject: ARM power-saving measurements
> >
> > Hi Feifei/ Ruifeng, I can see that power management support on ARM has
> > been integrated successfully.
> > Could you please share the knowledge on how to measure the power
> > savings on this architecture?
> > What are the tools available there? Unfortunately, turbostat is not
> > supported on ARM. How did you test your feature?
> >
> > Regards,
> > Alex