[dpdk-dev] [PATCH] doc: arm64: document DPDK application profiling methods

[dpdk-dev] [PATCH] doc: arm64: document DPDK application profiling methods

[Jerin Jacob]

Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 doc/guides/prog_guide/profile_app.rst | 58 +++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/doc/guides/prog_guide/profile_app.rst b/doc/guides/prog_guide/profile_app.rst
index 3226187..bb78623 100644
--- a/doc/guides/prog_guide/profile_app.rst
+++ b/doc/guides/prog_guide/profile_app.rst
@@ -31,6 +31,14 @@
 Profile Your Application
 ========================
 
+Introduction
+------------
+
+The following sections describe the methods to profile DPDK applications on
+different architectures.
+
+x86
+~~~
 Intel processors provide performance counters to monitor events.
 Some tools provided by Intel can be used to profile and benchmark an application.
 See the *VTune Performance Analyzer Essentials* publication from Intel Press for more information.
@@ -50,3 +58,53 @@ The main situations that should be monitored through event counters are:
 Refer to the
 `Intel Performance Analysis Guide <http://software.intel.com/sites/products/collateral/hpc/vtune/performance_analysis_guide.pdf>`_
 for details about application profiling.
+
+ARM64
+~~~~~
+
+Perf
+^^^^
+ARM64 architecture provide performance counters to monitor events.
+The Linux perf tool can be used to profile and benchmark an application.
+In addition to the standard events, perf can be used to profile arm64 specific
+PMU events through raw events(-e -rXX)
+
+Refer to the
+`ARM64 specific PMU events enumeration <http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.100095_0002_04_en/way1382543438508.html>`_
+
+High-resolution cycle counter
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The default cntvct_el0 based rte_rdtsc() provides portable means to get wall
+clock counter at user space. Typically it runs at <= 100MHz.
+
+The alternative method to enable rte_rdtsc() for high resolution
+wall clock counter is through armv8 PMU subsystem.
+The PMU cycle counter runs at CPU frequency, However, access to PMU cycle
+counter from user space is not enabled by default in the arm64 linux kernel.
+It is possible to enable cycle counter at user space access
+by configuring the PMU from the privileged mode (kernel space).
+
+by default rte_rdtsc() implementation uses portable cntvct_el0 scheme.
+Application can choose the PMU based implementation with
+CONFIG_RTE_ARM_EAL_RDTSC_USE_PMU
+
+The PMU based scheme useful for high accuracy performance profiling.
+Find below the example steps to configure the PMU based cycle counter on an
+armv8 machine.
+
+.. code-block:: console
+
+    git clone https://github.com/jerinjacobk/armv8_pmu_cycle_counter_el0
+    cd armv8_pmu_cycle_counter_el0
+    make
+    sudo insmod pmu_el0_cycle_counter.ko
+    cd $DPDK_DIR
+    make config T=arm64-armv8a-linuxapp-gcc
+    echo "CONFIG_RTE_ARM_EAL_RDTSC_USE_PMU=y" >> build/.config
+    make
+
+.. warning::
+
+    This method can not be used in production systems as this may alter PMU
+    state used by standard Linux user space tool like perf.
+
-- 
2.5.5

Re: [dpdk-dev] [PATCH] doc: arm64: document DPDK application profiling methods

[Thomas Monjalon]

Thanks for providing a patch so quickly :)

2016-10-04 16:10, Jerin Jacob:
> +The PMU based scheme useful for high accuracy performance profiling.

A verb is missing.

> +Find below the example steps to configure the PMU based cycle counter on an
> +armv8 machine.
> +
> +.. code-block:: console
> +
> +    git clone https://github.com/jerinjacobk/armv8_pmu_cycle_counter_el0
> +    cd armv8_pmu_cycle_counter_el0
> +    make
> +    sudo insmod pmu_el0_cycle_counter.ko
> +    cd $DPDK_DIR
> +    make config T=arm64-armv8a-linuxapp-gcc
> +    echo "CONFIG_RTE_ARM_EAL_RDTSC_USE_PMU=y" >> build/.config
> +    make

What about the ARM 32 code that Jan is using?

> +.. warning::
> +
> +    This method can not be used in production systems as this may alter PMU
> +    state used by standard Linux user space tool like perf.

More details please?

Re: [dpdk-dev] [PATCH] doc: arm64: document DPDK application profiling methods

[Jan Viktorin]

On Tue, 04 Oct 2016 14:40:47 +0200
Thomas Monjalon <thomas.monjalon@6wind.com> wrote:

> Thanks for providing a patch so quickly :)
> 
> 2016-10-04 16:10, Jerin Jacob:
> > +The PMU based scheme useful for high accuracy performance profiling.  
> 
> A verb is missing.
> 
> > +Find below the example steps to configure the PMU based cycle counter on an
> > +armv8 machine.
> > +
> > +.. code-block:: console
> > +
> > +    git clone https://github.com/jerinjacobk/armv8_pmu_cycle_counter_el0
> > +    cd armv8_pmu_cycle_counter_el0
> > +    make
> > +    sudo insmod pmu_el0_cycle_counter.ko
> > +    cd $DPDK_DIR
> > +    make config T=arm64-armv8a-linuxapp-gcc
> > +    echo "CONFIG_RTE_ARM_EAL_RDTSC_USE_PMU=y" >> build/.config
> > +    make  
> 
> What about the ARM 32 code that Jan is using?

Hi, I didn't have time for this yet. The basic description is here:

 lib/librte_eal/common/include/arch/arm/rte_cycles_32.h

In the Linux Kernel, it is used here:

 arch/arm/kernel/perf_event_v7.c (see registers c12, c13 and c14)

Regards
Jan

> 
> > +.. warning::
> > +
> > +    This method can not be used in production systems as this may alter PMU
> > +    state used by standard Linux user space tool like perf.  
> 
> More details please?
> 



-- 
   Jan Viktorin                  E-mail: Viktorin@RehiveTech.com
   System Architect              Web:    www.RehiveTech.com
   RehiveTech
   Brno, Czech Republic

[dpdk-dev] [PATCH v2] doc: arm64: document DPDK application profiling methods

[Jerin Jacob]

Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
v2:
-Addressed ARM64 specific review comments(Suggested by Thomas)
http://dpdk.org/dev/patchwork/patch/16362/
---
 doc/guides/prog_guide/profile_app.rst | 58 +++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/doc/guides/prog_guide/profile_app.rst b/doc/guides/prog_guide/profile_app.rst
index 3226187..9f1b7ee 100644
--- a/doc/guides/prog_guide/profile_app.rst
+++ b/doc/guides/prog_guide/profile_app.rst
@@ -31,6 +31,14 @@
 Profile Your Application
 ========================
 
+Introduction
+------------
+
+The following sections describe the methods to profile DPDK applications on
+different architectures.
+
+x86
+~~~
 Intel processors provide performance counters to monitor events.
 Some tools provided by Intel can be used to profile and benchmark an application.
 See the *VTune Performance Analyzer Essentials* publication from Intel Press for more information.
@@ -50,3 +58,53 @@ The main situations that should be monitored through event counters are:
 Refer to the
 `Intel Performance Analysis Guide <http://software.intel.com/sites/products/collateral/hpc/vtune/performance_analysis_guide.pdf>`_
 for details about application profiling.
+
+ARM64
+~~~~~
+
+Perf
+^^^^
+ARM64 architecture provide performance counters to monitor events.
+The Linux perf tool can be used to profile and benchmark an application.
+In addition to the standard events, perf can be used to profile arm64 specific
+PMU events through raw events(-e -rXX)
+
+Refer to the
+`ARM64 specific PMU events enumeration <http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.100095_0002_04_en/way1382543438508.html>`_
+
+High-resolution cycle counter
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The default cntvct_el0 based rte_rdtsc() provides portable means to get wall
+clock counter at user space. Typically it runs at <= 100MHz.
+
+The alternative method to enable rte_rdtsc() for high resolution
+wall clock counter is through armv8 PMU subsystem.
+The PMU cycle counter runs at CPU frequency, However, access to PMU cycle
+counter from user space is not enabled by default in the arm64 linux kernel.
+It is possible to enable cycle counter at user space access
+by configuring the PMU from the privileged mode (kernel space).
+
+by default rte_rdtsc() implementation uses portable cntvct_el0 scheme.
+Application can choose the PMU based implementation with
+CONFIG_RTE_ARM_EAL_RDTSC_USE_PMU
+
+Find below the example steps to configure the PMU based cycle counter on an
+armv8 machine.
+
+.. code-block:: console
+
+    git clone https://github.com/jerinjacobk/armv8_pmu_cycle_counter_el0
+    cd armv8_pmu_cycle_counter_el0
+    make
+    sudo insmod pmu_el0_cycle_counter.ko
+    cd $DPDK_DIR
+    make config T=arm64-armv8a-linuxapp-gcc
+    echo "CONFIG_RTE_ARM_EAL_RDTSC_USE_PMU=y" >> build/.config
+    make
+
+.. warning::
+
+   The PMU based scheme is useful for high accuracy performance profiling with
+   rte_rdtsc(). However, This method can not be used in conjunction with Linux
+   userspace profiling tools like perf as this scheme alters the PMU registers
+   state.
-- 
2.5.5

Re: [dpdk-dev] [PATCH v2] doc: arm64: document DPDK application profiling methods

[Thomas Monjalon]

2016-10-05 14:13, Jerin Jacob:
> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> ---
> v2:
> -Addressed ARM64 specific review comments(Suggested by Thomas)

I feel more comments could be done, especially about formatting.
You are adding a chapter Introduction without any other section
of the same level.
Some technical terms should be enclosed in backquotes.

Please John, could you guide Jerin or provide an updated version?
Thanks

Re: [dpdk-dev] [PATCH v2] doc: arm64: document DPDK application profiling methods

[Mcnamara, John]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> Sent: Wednesday, October 26, 2016 9:07 PM
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>; Mcnamara, John
> <john.mcnamara@intel.com>
> Cc: dev@dpdk.org; jianbo.liu@linaro.org; viktorin@rehivetech.com
> Subject: Re: [dpdk-dev] [PATCH v2] doc: arm64: document DPDK application
> profiling methods
> 
> 2016-10-05 14:13, Jerin Jacob:
> > Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > ---
> > v2:
> > -Addressed ARM64 specific review comments(Suggested by Thomas)
> 
> I feel more comments could be done, especially about formatting.
> You are adding a chapter Introduction without any other section of the
> same level.
> Some technical terms should be enclosed in backquotes.
> 
> Please John, could you guide Jerin or provide an updated version?

Will do.

Re: [dpdk-dev] [PATCH v2] doc: arm64: document DPDK application profiling methods

[Mcnamara, John]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jerin Jacob
> Sent: Wednesday, October 5, 2016 9:43 AM
> To: dev@dpdk.org
> Cc: thomas.monjalon@6wind.com; jianbo.liu@linaro.org;
> viktorin@rehivetech.com; Jerin Jacob <jerin.jacob@caviumnetworks.com>
> Subject: [dpdk-dev] [PATCH v2] doc: arm64: document DPDK application
> profiling methods
> 

Hi Jerin,

Thanks for this. It is probably something that should be extended from
the Intel side as well.

...



>  Profile Your Application
>  ========================
> 
> +Introduction
> +------------
> +
> +The following sections describe the methods to profile DPDK
> +applications on different architectures.

You could probably just skip the introduction heading and put this
text under the main heading.

As Thomas pointed out there are a few other RST formatting issues
as well. I'll forward some suggested changes.

John

[dpdk-dev] [PATCH v3] doc: arm64: document DPDK application profiling methods

[Jerin Jacob]

Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
Signed-off-by: John McNamara <john.mcnamara@intel.com>
---
v3:
Fixed formatting issues:
- Remove the introduction heading and put intro text under the main heading(Thomas)
- Fixed RST formatting issues such as enclosing technical terms in backquotes(John)
Thanks, John for providing the updated version
v2:
-Addressed ARM64 specific review comments(Suggested by Thomas)
http://dpdk.org/dev/patchwork/patch/16362/
---
 doc/guides/prog_guide/profile_app.rst | 64 ++++++++++++++++++++++++++++++++++-
 1 file changed, 63 insertions(+), 1 deletion(-)

diff --git a/doc/guides/prog_guide/profile_app.rst b/doc/guides/prog_guide/profile_app.rst
index 3226187..54b546a 100644
--- a/doc/guides/prog_guide/profile_app.rst
+++ b/doc/guides/prog_guide/profile_app.rst
@@ -31,8 +31,15 @@
 Profile Your Application
 ========================
 
+The following sections describe methods of profiling DPDK applications on
+different architectures.
+
+
+Profiling on x86
+----------------
+
 Intel processors provide performance counters to monitor events.
-Some tools provided by Intel can be used to profile and benchmark an application.
+Some tools provided by Intel, such as VTune, can be used to profile and benchmark an application.
 See the *VTune Performance Analyzer Essentials* publication from Intel Press for more information.
 
 For a DPDK application, this can be done in a Linux* application environment only.
@@ -50,3 +57,58 @@ The main situations that should be monitored through event counters are:
 Refer to the
 `Intel Performance Analysis Guide <http://software.intel.com/sites/products/collateral/hpc/vtune/performance_analysis_guide.pdf>`_
 for details about application profiling.
+
+
+Profiling on ARM64
+------------------
+
+Using Linux perf
+~~~~~~~~~~~~~~~~
+
+The ARM64 architecture provide performance counters to monitor events.  The
+Linux ``perf`` tool can be used to profile and benchmark an application.  In
+addition to the standard events, ``perf`` can be used to profile arm64
+specific PMU (Performance Monitor Unit) events through raw events (``-e``
+``-rXX``).
+
+For more derails refer to the
+`ARM64 specific PMU events enumeration <http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.100095_0002_04_en/way1382543438508.html>`_.
+
+
+High-resolution cycle counter
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The default ``cntvct_el0`` based ``rte_rdtsc()`` provides a portable means to
+get a wall clock counter in user space. Typically it runs at <= 100MHz.
+
+The alternative method to enable ``rte_rdtsc()`` for a high resolution wall
+clock counter is through the armv8 PMU subsystem. The PMU cycle counter runs
+at CPU frequency. However, access to the PMU cycle counter from user space is
+not enabled by default in the arm64 linux kernel. It is possible to enable
+cycle counter for user space access by configuring the PMU from the privileged
+mode (kernel space).
+
+By default the ``rte_rdtsc()`` implementation uses a portable ``cntvct_el0``
+scheme.  Application can choose the PMU based implementation with
+``CONFIG_RTE_ARM_EAL_RDTSC_USE_PMU``.
+
+The example below shows the steps to configure the PMU based cycle counter on
+an armv8 machine.
+
+.. code-block:: console
+
+    git clone https://github.com/jerinjacobk/armv8_pmu_cycle_counter_el0
+    cd armv8_pmu_cycle_counter_el0
+    make
+    sudo insmod pmu_el0_cycle_counter.ko
+    cd $DPDK_DIR
+    make config T=arm64-armv8a-linuxapp-gcc
+    echo "CONFIG_RTE_ARM_EAL_RDTSC_USE_PMU=y" >> build/.config
+    make
+
+.. warning::
+
+   The PMU based scheme is useful for high accuracy performance profiling with
+   ``rte_rdtsc()``. However, this method can not be used in conjunction with
+   Linux userspace profiling tools like ``perf`` as this scheme alters the PMU
+   registers state.
-- 
2.5.5

Re: [dpdk-dev] [PATCH v3] doc: arm64: document DPDK application profiling methods

[Jianbo Liu]

On 8 November 2016 at 11:32, Jerin Jacob <jerin.jacob@caviumnetworks.com> wrote:
> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> Signed-off-by: John McNamara <john.mcnamara@intel.com>
> ---
> v3:
> Fixed formatting issues:
> - Remove the introduction heading and put intro text under the main heading(Thomas)
> - Fixed RST formatting issues such as enclosing technical terms in backquotes(John)
> Thanks, John for providing the updated version
> v2:
> -Addressed ARM64 specific review comments(Suggested by Thomas)
> http://dpdk.org/dev/patchwork/patch/16362/
> ---
>  doc/guides/prog_guide/profile_app.rst | 64 ++++++++++++++++++++++++++++++++++-
>  1 file changed, 63 insertions(+), 1 deletion(-)
>

Acked-by: Jianbo Liu <jianbo.liu@linaro.org>

Re: [dpdk-dev] [PATCH v3] doc: arm64: document DPDK application profiling methods

[Thomas Monjalon]

> > Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > Signed-off-by: John McNamara <john.mcnamara@intel.com>
> 
> Acked-by: Jianbo Liu <jianbo.liu@linaro.org>

Applied, thanks