[PATCH 0/2] Improve docs on getting info on running process

[PATCH 0/2] Improve docs on getting info on running process

[Bruce Richardson]

Add details to our documentation on how to register logging for a component.
Also provide guidelines on when to use logs vs tracing vs telemetry.

Feedback welcome on the guidelines.

Bruce Richardson (2):
  doc/contributing: provide coding details for dynamic logging
  doc/contributing: guidelines for logging, tracing and telemetry

 doc/guides/contributing/coding_style.rst | 19 +++++++++++++
 doc/guides/contributing/design.rst       | 34 ++++++++++++++++++++++++
 doc/guides/prog_guide/telemetry_lib.rst  |  2 ++
 doc/guides/prog_guide/trace_lib.rst      |  2 ++
 lib/cfgfile/rte_cfgfile.c                |  2 ++
 5 files changed, 59 insertions(+)

--
2.39.2

[PATCH 1/2] doc/contributing: provide coding details for dynamic logging

[Bruce Richardson]

While the section on dynamic logging in the contributors guide covered
the details of the logging naming scheme, it failed to cover exactly how
the component developer, i.e. the contributor, could actually use
dynamic logging in their component.

Fix this by splitting the details of the naming scheme into a separate
subsection, and adding to the introduction on logging, a recommendation
(and example) to use RTE_LOG_REGISTER_DEFAULT.

Similarly, when discussing specialization, include a reference to the
RTE_LOG_REGISTER_SUFFIX macro.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/contributing/coding_style.rst | 17 +++++++++++++++++
 lib/cfgfile/rte_cfgfile.c                |  2 ++
 2 files changed, 19 insertions(+)

diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
index 00d6270624..f50a78a346 100644
--- a/doc/guides/contributing/coding_style.rst
+++ b/doc/guides/contributing/coding_style.rst
@@ -803,6 +803,21 @@ logging of a particular topic, the ``--log-level`` parameter can be provided
 to EAL, which will change the log level. DPDK code can register topics,
 which allows the user to adjust the log verbosity for that specific topic.
 
+To register a library or driver for dynamic logging,
+using the standardized naming scheme described below,
+use ``RTE_LOG_REGISTER_DEFAULT`` macro to define a log-type variable inside your component's main C file.
+Thereafter, it is usual to define a macro or macros inside your component to make logging more convenient.
+
+For example, the ``rte_cfgfile`` library defines:
+
+.. literalinclude:: ../../../lib/cfgfile/rte_cfgfile.c
+    :language: c
+    :start-after: Setting up dynamic logging 8<
+    :end-before: >8 End of setting up dynamic logging
+
+Dynamic Logging Naming Scheme
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 In general, the naming scheme is as follows: ``type.section.name``
 
  * Type is the type of component, where ``lib``, ``pmd``, ``bus`` and ``user``
@@ -837,6 +852,8 @@ A specialization looks like this:
  * Initialization output: ``type.section.name.init``
  * PF/VF mailbox output: ``type.section.name.mbox``
 
+These specializations are created using the ``RTE_LOG_REGISTER_SUFFIX`` macro.
+
 A real world example is the i40e poll mode driver which exposes two
 specializations, one for initialization ``pmd.net.i40e.init`` and the other for
 the remaining driver logs ``pmd.net.i40e.driver``.
diff --git a/lib/cfgfile/rte_cfgfile.c b/lib/cfgfile/rte_cfgfile.c
index 9fa7d010ef..eefba6e408 100644
--- a/lib/cfgfile/rte_cfgfile.c
+++ b/lib/cfgfile/rte_cfgfile.c
@@ -27,11 +27,13 @@ struct rte_cfgfile {
 	struct rte_cfgfile_section *sections;
 };
 
+/* Setting up dynamic logging 8< */
 RTE_LOG_REGISTER_DEFAULT(cfgfile_logtype, INFO);
 
 #define CFG_LOG(level, fmt, args...)					\
 	rte_log(RTE_LOG_ ## level, cfgfile_logtype, "%s(): " fmt "\n",	\
 		__func__, ## args)
+/* >8 End of setting up dynamic logging */
 
 /** when we resize a file structure, how many extra entries
  * for new sections do we add in */
-- 
2.39.2

[PATCH 2/2] doc/contributing: guidelines for logging, tracing and telemetry

[Bruce Richardson]

As discussed by DPDK technical board [1], out contributor guide should
include some details as to when to use logging vs tracing vs telemetry
to provide the end user with information about the running process and
the DPDK libraries it uses.

[1] https://mails.dpdk.org/archives/dev/2023-March/265204.html

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/contributing/coding_style.rst |  2 ++
 doc/guides/contributing/design.rst       | 34 ++++++++++++++++++++++++
 doc/guides/prog_guide/telemetry_lib.rst  |  2 ++
 doc/guides/prog_guide/trace_lib.rst      |  2 ++
 4 files changed, 40 insertions(+)

diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
index f50a78a346..13b2390d9e 100644
--- a/doc/guides/contributing/coding_style.rst
+++ b/doc/guides/contributing/coding_style.rst
@@ -794,6 +794,8 @@ Control Statements
                  /* NOTREACHED */
          }
 
+.. _dynamic_logging:
+
 Dynamic Logging
 ---------------
 
diff --git a/doc/guides/contributing/design.rst b/doc/guides/contributing/design.rst
index d24a7ff6a0..9edf3ea43b 100644
--- a/doc/guides/contributing/design.rst
+++ b/doc/guides/contributing/design.rst
@@ -4,6 +4,40 @@
 Design
 ======
 
+Runtime Information - Logging, Tracing and Telemetry
+-------------------------------------------------------
+
+It is often desirable to provide information to the end-user as to what is happening to the application at runtime.
+DPDK provides a number of built-in mechanisms to provide this introspection:
+
+* :ref:`Logging<dynamic_logging>`
+* :ref:`Tracing<trace_library>`
+* :ref:`Telemetry<telemetry_library>`
+
+Each of these has it's own strengths and suitabilities for use within DPDK components.
+
+Below are some guidelines for when each should be used:
+
+* For reporting error conditions, or other abnormal runtime issues, *logging* should be used.
+  Depending on the severity of the issue, the appropriate log level, for example,
+  ``ERROR``, ``WARNING`` or ``NOTICE``, should be used.
+* In general, it is not recommended that the DPDK logging functions should be used for debugging.
+  Although the ``DEBUG`` log level may be used in components, it should only be used sparingly,
+  and the *tracing* functionality used instead.
+  More specifically:
+
+  * for cases where a path through the code is only likely to be taken once,
+    for example, initialization code, either *logging* at ``DEBUG`` level or *tracing* may be used -
+    though tracing is preferred.
+  * where a path is to be taken multiple times within a short timeframe, for example,
+    datapath or regular control plane code, *tracing* should be used.
+
+* For numerical or statistical data generated by a component, for example, per-packet statistics,
+  *telemetry* should be used.
+* For any data where the data may need to be gathered at any point in the execution to help assess the state of the application component,
+  for example, core configuration, device information, *telemetry* should be used.
+
+
 Environment or Architecture-specific Sources
 --------------------------------------------
 
diff --git a/doc/guides/prog_guide/telemetry_lib.rst b/doc/guides/prog_guide/telemetry_lib.rst
index 32f525a67f..71f8bd735e 100644
--- a/doc/guides/prog_guide/telemetry_lib.rst
+++ b/doc/guides/prog_guide/telemetry_lib.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright(c) 2020 Intel Corporation.
 
+.. _telemetry_library:
+
 Telemetry Library
 =================
 
diff --git a/doc/guides/prog_guide/trace_lib.rst b/doc/guides/prog_guide/trace_lib.rst
index e5718feddc..a3b8a7c2eb 100644
--- a/doc/guides/prog_guide/trace_lib.rst
+++ b/doc/guides/prog_guide/trace_lib.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright(C) 2020 Marvell International Ltd.
 
+.. _trace_library:
+
 Trace Library
 =============
 
-- 
2.39.2

Re: [PATCH 2/2] doc/contributing: guidelines for logging, tracing and telemetry

[Stephen Hemminger]

On Tue, 13 Jun 2023 15:33:55 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:

> +Below are some guidelines for when each should be used:
> +
> +* For reporting error conditions, or other abnormal runtime issues, *logging* should be used.
> +  Depending on the severity of the issue, the appropriate log level, for example,
> +  ``ERROR``, ``WARNING`` or ``NOTICE``, should be used.
> +* In general, it is not recommended that the DPDK logging functions should be used for debugging.
> +  Although the ``DEBUG`` log level may be used in components, it should only be used sparingly,
> +  and the *tracing* functionality used instead.
> +  More specifically:
> +
> +  * for cases where a path through the code is only likely to be taken once,
> +    for example, initialization code, either *logging* at ``DEBUG`` level or *tracing* may be used -
> +    though tracing is preferred.
> +  * where a path is to be taken multiple times within a short timeframe, for example,
> +    datapath or regular control plane code, *tracing* should be used.

When a new feature, bus or device type is added it *must* not log when not present.
No new messages for existing users please.

RE: [PATCH 2/2] doc/contributing: guidelines for logging, tracing and telemetry

[Morten Brørup]

> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Tuesday, 13 June 2023 16.34
> 
> As discussed by DPDK technical board [1], out contributor guide should

Typo: out -> our

> include some details as to when to use logging vs tracing vs telemetry
> to provide the end user with information about the running process and
> the DPDK libraries it uses.
> 
> [1] https://mails.dpdk.org/archives/dev/2023-March/265204.html
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---

Debug logging can be more verbose than trace in the one-off cases, so I have a slightly different personal preference. But perhaps this is mainly relevant in the abnormal execution paths, and thus already covered.

Anyway, I agree with everything in this patch.

Acked-by: Morten Brørup <mb@smartsharesystems.com>

You might also consider mentioning the xyz_dump(FILE *f) functions such as rte_mempool_list_dump(FILE *f).

I find this type of semi-structured verbose dumping very useful for investigating the state of systems in production. Our debug CLI provides access to these _dump() functions, including some of our own, from various subsystems in the application.

Re: [PATCH 2/2] doc/contributing: guidelines for logging, tracing and telemetry

[Bruce Richardson]

On Tue, Jun 13, 2023 at 09:38:55PM +0200, Morten Brørup wrote:
> > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Sent: Tuesday, 13 June 2023 16.34
> > 
> > As discussed by DPDK technical board [1], out contributor guide should
> 
> Typo: out -> our
> 
> > include some details as to when to use logging vs tracing vs telemetry
> > to provide the end user with information about the running process and
> > the DPDK libraries it uses.
> > 
> > [1] https://mails.dpdk.org/archives/dev/2023-March/265204.html
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > ---
> 
> Debug logging can be more verbose than trace in the one-off cases, so I have a slightly different personal preference. But perhaps this is mainly relevant in the abnormal execution paths, and thus already covered.
> 

Feel free to suggest alternate wordings if you have any. That particular
item is hard to come up with good guidelines for - especially ones that are
not pages long! :-)

> Anyway, I agree with everything in this patch.
> 
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> 

Thanks.

> You might also consider mentioning the xyz_dump(FILE *f) functions such as rte_mempool_list_dump(FILE *f).
> 
> I find this type of semi-structured verbose dumping very useful for investigating the state of systems in production. Our debug CLI provides access to these _dump() functions, including some of our own, from various subsystems in the application.
> 

I'll try and put some mention of that in V2.

/Bruce

RE: [PATCH 2/2] doc/contributing: guidelines for logging, tracing and telemetry

[Morten Brørup]

> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Wednesday, 14 June 2023 10.37
> 
> On Tue, Jun 13, 2023 at 09:38:55PM +0200, Morten Brørup wrote:
> > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > Sent: Tuesday, 13 June 2023 16.34
> > >
> > > As discussed by DPDK technical board [1], out contributor guide should
> >
> > Typo: out -> our
> >
> > > include some details as to when to use logging vs tracing vs telemetry
> > > to provide the end user with information about the running process and
> > > the DPDK libraries it uses.
> > >
> > > [1] https://mails.dpdk.org/archives/dev/2023-March/265204.html
> > >
> > > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > > ---
> >
> > Debug logging can be more verbose than trace in the one-off cases, so I have
> a slightly different personal preference. But perhaps this is mainly relevant
> in the abnormal execution paths, and thus already covered.
> >
> 
> Feel free to suggest alternate wordings if you have any. That particular
> item is hard to come up with good guidelines for - especially ones that are
> not pages long! :-)

+  * For cases where a path through the code is only likely to be taken once,
+    for example, initialization code, either *logging* at ``DEBUG`` level or *tracing* may be used -
+    though tracing is preferred.

How about adding this sentence at the end of the bullet:
+    though tracing is preferred. When conveying a lot of information, *logging* with more details can supplement *tracing* with the basic details.

+  * Where a path is to be taken multiple times within a short timeframe, for example,
+    datapath or regular control plane code, *tracing* should be used.

Capitalize the first letter at the bullets: "* for"->"* For" and "* where"->"* Where"

> 
> > Anyway, I agree with everything in this patch.
> >
> > Acked-by: Morten Brørup <mb@smartsharesystems.com>
> >
> 
> Thanks.
> 
> > You might also consider mentioning the xyz_dump(FILE *f) functions such as
> rte_mempool_list_dump(FILE *f).
> >
> > I find this type of semi-structured verbose dumping very useful for
> investigating the state of systems in production. Our debug CLI provides
> access to these _dump() functions, including some of our own, from various
> subsystems in the application.
> >
> 
> I'll try and put some mention of that in V2.
> 
> /Bruce

Re: [PATCH 2/2] doc/contributing: guidelines for logging, tracing and telemetry

[Ferruh Yigit]

On 6/13/2023 3:33 PM, Bruce Richardson wrote:
> As discussed by DPDK technical board [1], out contributor guide should
> include some details as to when to use logging vs tracing vs telemetry
> to provide the end user with information about the running process and
> the DPDK libraries it uses.
> 
> [1] https://mails.dpdk.org/archives/dev/2023-March/265204.html
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
>  doc/guides/contributing/coding_style.rst |  2 ++
>  doc/guides/contributing/design.rst       | 34 ++++++++++++++++++++++++
>  doc/guides/prog_guide/telemetry_lib.rst  |  2 ++
>  doc/guides/prog_guide/trace_lib.rst      |  2 ++
>  4 files changed, 40 insertions(+)
> 
> diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
> index f50a78a346..13b2390d9e 100644
> --- a/doc/guides/contributing/coding_style.rst
> +++ b/doc/guides/contributing/coding_style.rst
> @@ -794,6 +794,8 @@ Control Statements
>                   /* NOTREACHED */
>           }
>  
> +.. _dynamic_logging:
> +
>  Dynamic Logging
>  ---------------
>  
> diff --git a/doc/guides/contributing/design.rst b/doc/guides/contributing/design.rst
> index d24a7ff6a0..9edf3ea43b 100644
> --- a/doc/guides/contributing/design.rst
> +++ b/doc/guides/contributing/design.rst
> @@ -4,6 +4,40 @@
>  Design
>  ======
>  
> +Runtime Information - Logging, Tracing and Telemetry
> +-------------------------------------------------------
> +
> +It is often desirable to provide information to the end-user as to what is happening to the application at runtime.
>

I think target audience matters, who is consumer of the runtime
information, possible candidates I can think of:
- dev-ops engineer deploys and maintains a dpdk application
- DPDK based application developer
- DPDK developer

Should we distinguish them? Like I guess telemetry output can be more
useful to dev-ops engineer, but debug output can be more useful to
application developer.

To roughly set target consumer for each method may help developer to
decide which one to use, but not sure if we can narrow it like this.

> +DPDK provides a number of built-in mechanisms to provide this introspection:
> +
> +* :ref:`Logging<dynamic_logging>`
> +* :ref:`Tracing<trace_library>`
> +* :ref:`Telemetry<telemetry_library>`
> +
> +Each of these has it's own strengths and suitabilities for use within DPDK components.
> +
> +Below are some guidelines for when each should be used:
> +
> +* For reporting error conditions, or other abnormal runtime issues, *logging* should be used.
> +  Depending on the severity of the issue, the appropriate log level, for example,
> +  ``ERROR``, ``WARNING`` or ``NOTICE``, should be used.

+1

> +* In general, it is not recommended that the DPDK logging functions should be used for debugging.
> +  Although the ``DEBUG`` log level may be used in components, it should only be used sparingly,
> +  and the *tracing* functionality used instead.

I think DEBUG logging is still useful for control path, 'tracing' is
missing the custom message, so I found it more useful for to analyze
order and frequency of a function call, which suits more to datapath.

> +  More specifically:
> +
> +  * for cases where a path through the code is only likely to be taken once,
> +    for example, initialization code, either *logging* at ``DEBUG`` level or *tracing* may be used -
> +    though tracing is preferred.>

I am for preferring 'logging' for this case, because there is extra
effort required to analyze the trace output, but logs are simple and
just there.

> +  * where a path is to be taken multiple times within a short timeframe, for example,
> +    datapath or regular control plane code, *tracing* should be used.
> +
>

+1 to use tracing mainly for datapath

Also there is logging support for datapath, it would be nice to clarify
that usage too. Datapath logging used:
a) RTE_LOG_DP macros where code compiled out based on selected log level
b) 'RTE_ETHDEV_DEBUG_RX' & 'RTE_ETHDEV_DEBUG_TX' macros

ethdev layer use b for logging,
drivers use combination of a & b

Tracing helps on datapath to get profiling etc, but there are cases
driver needs to log some variables, like descriptor details to ensure
*functionality* is correct.


I suggest to remove/deprecate RTE_LOG_DP [1], and use
'RTE_ETHDEV_DEBUG_x' macros for developer logging, and use tracing for
profiling. What do you think?


[1]
Intention is good with RTE_LOG_DP macro to set the desired log level and
compile out not used logs so that they won't take extra cycle.
But every time log level changed, it requires compiling again anyway,
and in practice there is no multiple level logging need for datapath,
most of the time single macro to enable/disable logging is sufficient.
That why I believe simpler to use an explicit macro to enable/disable
datapath logging, like 'RTE_ETHDEV_DEBUG_RX' & 'RTE_ETHDEV_DEBUG_TX'.



> +* For numerical or statistical data generated by a component, for example, per-packet statistics,
> +  *telemetry* should be used.
>

+1


> +* For any data where the data may need to be gathered at any point in the execution to help assess the state of the application component,
> +  for example, core configuration, device information, *telemetry* should be used.
> +

+1

Should we explicitly state that telemetry only used as readonly?
I think it implicitly states, but since there was a patch in the past, I
wonder if this needs to be highlighted?

Re: [PATCH 2/2] doc/contributing: guidelines for logging, tracing and telemetry

[Jerin Jacob]

On Tue, Jun 13, 2023 at 8:04 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> As discussed by DPDK technical board [1], out contributor guide should
> include some details as to when to use logging vs tracing vs telemetry
> to provide the end user with information about the running process and
> the DPDK libraries it uses.
>
> [1] https://mails.dpdk.org/archives/dev/2023-March/265204.html
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
>  doc/guides/contributing/coding_style.rst |  2 ++
>  doc/guides/contributing/design.rst       | 34 ++++++++++++++++++++++++
>  doc/guides/prog_guide/telemetry_lib.rst  |  2 ++
>  doc/guides/prog_guide/trace_lib.rst      |  2 ++
>  4 files changed, 40 insertions(+)
>
> diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
> index f50a78a346..13b2390d9e 100644
> --- a/doc/guides/contributing/coding_style.rst
> +++ b/doc/guides/contributing/coding_style.rst
> @@ -794,6 +794,8 @@ Control Statements
>                   /* NOTREACHED */
>           }
>
> +.. _dynamic_logging:
> +
>  Dynamic Logging
>  ---------------
>
> diff --git a/doc/guides/contributing/design.rst b/doc/guides/contributing/design.rst
> index d24a7ff6a0..9edf3ea43b 100644
> --- a/doc/guides/contributing/design.rst
> +++ b/doc/guides/contributing/design.rst
> @@ -4,6 +4,40 @@
>  Design
>  ======
>
> +Runtime Information - Logging, Tracing and Telemetry
> +-------------------------------------------------------
> +
> +It is often desirable to provide information to the end-user as to what is happening to the application at runtime.
> +DPDK provides a number of built-in mechanisms to provide this introspection:
> +
> +* :ref:`Logging<dynamic_logging>`
> +* :ref:`Tracing<trace_library>`
> +* :ref:`Telemetry<telemetry_library>`
> +
> +Each of these has it's own strengths and suitabilities for use within DPDK components.
> +
> +Below are some guidelines for when each should be used:
> +
> +* For reporting error conditions, or other abnormal runtime issues, *logging* should be used.
> +  Depending on the severity of the issue, the appropriate log level, for example,
> +  ``ERROR``, ``WARNING`` or ``NOTICE``, should be used.
> +* In general, it is not recommended that the DPDK logging functions should be used for debugging.
> +  Although the ``DEBUG`` log level may be used in components, it should only be used sparingly,
> +  and the *tracing* functionality used instead.
> +  More specifically:
> +
> +  * for cases where a path through the code is only likely to be taken once,
> +    for example, initialization code, either *logging* at ``DEBUG`` level or *tracing* may be used -
> +    though tracing is preferred.
> +  * where a path is to be taken multiple times within a short timeframe, for example,
> +    datapath or regular control plane code, *tracing* should be used.


If you think it makes sense, the following point may be added for trace.
# Since trace is using CommonTraceFormat, it is possible to write
_reusable_ post-processing tools with
https://babeltrace.org/docs/v1.5/python/babeltrace/ like finding idle
CPU time etc.

Rest looks good to me.



> +
> +* For numerical or statistical data generated by a component, for example, per-packet statistics,
> +  *telemetry* should be used.
> +* For any data where the data may need to be gathered at any point in the execution to help assess the state of the application component,
> +  for example, core configuration, device information, *telemetry* should be used.
> +
> +
>  Environment or Architecture-specific Sources
>  --------------------------------------------
>
> diff --git a/doc/guides/prog_guide/telemetry_lib.rst b/doc/guides/prog_guide/telemetry_lib.rst
> index 32f525a67f..71f8bd735e 100644
> --- a/doc/guides/prog_guide/telemetry_lib.rst
> +++ b/doc/guides/prog_guide/telemetry_lib.rst
> @@ -1,6 +1,8 @@
>  ..  SPDX-License-Identifier: BSD-3-Clause
>      Copyright(c) 2020 Intel Corporation.
>
> +.. _telemetry_library:
> +
>  Telemetry Library
>  =================
>
> diff --git a/doc/guides/prog_guide/trace_lib.rst b/doc/guides/prog_guide/trace_lib.rst
> index e5718feddc..a3b8a7c2eb 100644
> --- a/doc/guides/prog_guide/trace_lib.rst
> +++ b/doc/guides/prog_guide/trace_lib.rst
> @@ -1,6 +1,8 @@
>  ..  SPDX-License-Identifier: BSD-3-Clause
>      Copyright(C) 2020 Marvell International Ltd.
>
> +.. _trace_library:
> +
>  Trace Library
>  =============
>
> --
> 2.39.2
>

[PATCH v2 0/2] Improve docs on getting info on running process

[Bruce Richardson]

Add details to our documentation on how to register logging for a
component. Also provide guidelines on when to use logs vs tracing
vs telemetry.

V2: extended second patch based on the feedback provided.
    Hopefully, we can keep this section short and clear as the
    revisions roll! :-)

Bruce Richardson (2):
  doc/contributing: provide coding details for dynamic logging
  doc/contributing: guidelines for logging, tracing and telemetry

 doc/guides/contributing/coding_style.rst | 19 ++++++++++
 doc/guides/contributing/design.rst       | 47 ++++++++++++++++++++++++
 doc/guides/prog_guide/telemetry_lib.rst  |  2 +
 doc/guides/prog_guide/trace_lib.rst      |  2 +
 lib/cfgfile/rte_cfgfile.c                |  2 +
 5 files changed, 72 insertions(+)

--
2.39.2

[PATCH v2 1/2] doc/contributing: provide coding details for dynamic logging

[Bruce Richardson]

While the section on dynamic logging in the contributors guide covered
the details of the logging naming scheme, it failed to cover exactly how
the component developer, i.e. the contributor, could actually use
dynamic logging in their component.

Fix this by splitting the details of the naming scheme into a separate
subsection, and adding to the introduction on logging, a recommendation
(and example) to use RTE_LOG_REGISTER_DEFAULT.

Similarly, when discussing specialization, include a reference to the
RTE_LOG_REGISTER_SUFFIX macro.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 doc/guides/contributing/coding_style.rst | 17 +++++++++++++++++
 lib/cfgfile/rte_cfgfile.c                |  2 ++
 2 files changed, 19 insertions(+)

diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
index 89db6260cf..307c7deb9a 100644
--- a/doc/guides/contributing/coding_style.rst
+++ b/doc/guides/contributing/coding_style.rst
@@ -803,6 +803,21 @@ logging of a particular topic, the ``--log-level`` parameter can be provided
 to EAL, which will change the log level. DPDK code can register topics,
 which allows the user to adjust the log verbosity for that specific topic.
 
+To register a library or driver for dynamic logging,
+using the standardized naming scheme described below,
+use ``RTE_LOG_REGISTER_DEFAULT`` macro to define a log-type variable inside your component's main C file.
+Thereafter, it is usual to define a macro or macros inside your component to make logging more convenient.
+
+For example, the ``rte_cfgfile`` library defines:
+
+.. literalinclude:: ../../../lib/cfgfile/rte_cfgfile.c
+    :language: c
+    :start-after: Setting up dynamic logging 8<
+    :end-before: >8 End of setting up dynamic logging
+
+Dynamic Logging Naming Scheme
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 In general, the naming scheme is as follows: ``type.section.name``
 
  * Type is the type of component, where ``lib``, ``pmd``, ``bus`` and ``user``
@@ -837,6 +852,8 @@ A specialization looks like this:
  * Initialization output: ``type.section.name.init``
  * PF/VF mailbox output: ``type.section.name.mbox``
 
+These specializations are created using the ``RTE_LOG_REGISTER_SUFFIX`` macro.
+
 A real world example is the i40e poll mode driver which exposes two
 specializations, one for initialization ``pmd.net.i40e.init`` and the other for
 the remaining driver logs ``pmd.net.i40e.driver``.
diff --git a/lib/cfgfile/rte_cfgfile.c b/lib/cfgfile/rte_cfgfile.c
index 9fa7d010ef..eefba6e408 100644
--- a/lib/cfgfile/rte_cfgfile.c
+++ b/lib/cfgfile/rte_cfgfile.c
@@ -27,11 +27,13 @@ struct rte_cfgfile {
 	struct rte_cfgfile_section *sections;
 };
 
+/* Setting up dynamic logging 8< */
 RTE_LOG_REGISTER_DEFAULT(cfgfile_logtype, INFO);
 
 #define CFG_LOG(level, fmt, args...)					\
 	rte_log(RTE_LOG_ ## level, cfgfile_logtype, "%s(): " fmt "\n",	\
 		__func__, ## args)
+/* >8 End of setting up dynamic logging */
 
 /** when we resize a file structure, how many extra entries
  * for new sections do we add in */
-- 
2.39.2

[PATCH v2 2/2] doc/contributing: guidelines for logging, tracing and telemetry

[Bruce Richardson]

As discussed by DPDK technical board [1], our contributor guide should
include some details as to when to use logging vs tracing vs telemetry
to provide the end user with information about the running process and
the DPDK libraries it uses.

[1] https://mails.dpdk.org/archives/dev/2023-March/265204.html

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>

---
V2:
* Added note about not logging from unused drivers
* Reworked bullets/sub-bullets on tracing vs debug logs for debugging.
  - Consensus in replies was that people liked having debug logs for
    single-use, e.g. init cases.
  - Kept recommendation for tracing for data-path only
* Added short discussion of *_dump() functions at end of section
* Added sentence to indicate that telemetry should be read-only
* Added mention of common trace format and that other tools are
  available for it.
---
 doc/guides/contributing/coding_style.rst |  2 +
 doc/guides/contributing/design.rst       | 47 ++++++++++++++++++++++++
 doc/guides/prog_guide/telemetry_lib.rst  |  2 +
 doc/guides/prog_guide/trace_lib.rst      |  2 +
 4 files changed, 53 insertions(+)

diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
index 307c7deb9a..0861305dc6 100644
--- a/doc/guides/contributing/coding_style.rst
+++ b/doc/guides/contributing/coding_style.rst
@@ -794,6 +794,8 @@ Control Statements
                  /* NOTREACHED */
          }
 
+.. _dynamic_logging:
+
 Dynamic Logging
 ---------------
 
diff --git a/doc/guides/contributing/design.rst b/doc/guides/contributing/design.rst
index d24a7ff6a0..30104e0bfb 100644
--- a/doc/guides/contributing/design.rst
+++ b/doc/guides/contributing/design.rst
@@ -4,6 +4,53 @@
 Design
 ======
 
+Runtime Information - Logging, Tracing and Telemetry
+-------------------------------------------------------
+
+It is often desirable to provide information to the end-user as to what is happening to the application at runtime.
+DPDK provides a number of built-in mechanisms to provide this introspection:
+
+* :ref:`Logging<dynamic_logging>`
+* :ref:`Tracing<trace_library>`
+* :ref:`Telemetry<telemetry_library>`
+
+Each of these has it's own strengths and suitabilities for use within DPDK components.
+
+Below are some guidelines for when each should be used:
+
+* For reporting error conditions, or other abnormal runtime issues, *logging* should be used.
+  Depending on the severity of the issue, the appropriate log level, for example,
+  ``ERROR``, ``WARNING`` or ``NOTICE``, should be used.
+
+.. note::
+
+    Drivers off all classes, including both bus and device drivers,
+    should not output any log information if the hardware they support is not present.
+    This is to avoid any changes in output for existing users when a new driver is added to DPDK. 
+
+* For component initialization, or other cases where a path through the code is only likely to be taken once,
+  either *logging* at ``DEBUG`` level or *tracing* may be used, or potentially both.
+  In the latter case, tracing can provide basic information as to the code path taken,
+  with debug-level logging providing additional details on internal state,
+  not possible to emit via tracing.
+
+* For a component's data-path, where a path is to be taken multiple times within a short timeframe,
+  *tracing* should be used.
+  Since DPDK tracing uses `Common Trace Format <https://diamon.org/ctf/>`_ for its tracing logs,
+  post-analysis can be done using a range of external tools.
+
+* For numerical or statistical data generated by a component, for example, per-packet statistics,
+  *telemetry* should be used.
+
+* For any data where the data may need to be gathered at any point in the execution to help assess the state of the application component,
+  for example, core configuration, device information, *telemetry* should be used.
+  Telemetry callbacks should not modify any program state, but be "read-only".
+
+Many libraries also include a ``rte_<libname>_dump()`` function as part of their API,
+writing verbose internal details to a given file-handle.
+New libraries are encouraged to provide such functions where it makes sense to do so,
+as they provide an additional application-controlled mechanism to get details of the internals of a DPDK component.
+
 Environment or Architecture-specific Sources
 --------------------------------------------
 
diff --git a/doc/guides/prog_guide/telemetry_lib.rst b/doc/guides/prog_guide/telemetry_lib.rst
index 32f525a67f..71f8bd735e 100644
--- a/doc/guides/prog_guide/telemetry_lib.rst
+++ b/doc/guides/prog_guide/telemetry_lib.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright(c) 2020 Intel Corporation.
 
+.. _telemetry_library:
+
 Telemetry Library
 =================
 
diff --git a/doc/guides/prog_guide/trace_lib.rst b/doc/guides/prog_guide/trace_lib.rst
index e5718feddc..a3b8a7c2eb 100644
--- a/doc/guides/prog_guide/trace_lib.rst
+++ b/doc/guides/prog_guide/trace_lib.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright(C) 2020 Marvell International Ltd.
 
+.. _trace_library:
+
 Trace Library
 =============
 
-- 
2.39.2

Re: [PATCH v2 1/2] doc/contributing: provide coding details for dynamic logging

[fengchengwen]

Acked-by: Chengwen Feng <fengchengwen@huawei.com>

On 2023/6/21 1:07, Bruce Richardson wrote:
> While the section on dynamic logging in the contributors guide covered
> the details of the logging naming scheme, it failed to cover exactly how
> the component developer, i.e. the contributor, could actually use
> dynamic logging in their component.
> 
> Fix this by splitting the details of the naming scheme into a separate
> subsection, and adding to the introduction on logging, a recommendation
> (and example) to use RTE_LOG_REGISTER_DEFAULT.
> 
> Similarly, when discussing specialization, include a reference to the
> RTE_LOG_REGISTER_SUFFIX macro.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---

...

Re: [PATCH v2 1/2] doc/contributing: provide coding details for dynamic logging

[David Marchand]

On Tue, Jun 20, 2023 at 7:08 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> While the section on dynamic logging in the contributors guide covered
> the details of the logging naming scheme, it failed to cover exactly how
> the component developer, i.e. the contributor, could actually use
> dynamic logging in their component.
>
> Fix this by splitting the details of the naming scheme into a separate
> subsection, and adding to the introduction on logging, a recommendation
> (and example) to use RTE_LOG_REGISTER_DEFAULT.
>
> Similarly, when discussing specialization, include a reference to the
> RTE_LOG_REGISTER_SUFFIX macro.
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
>  doc/guides/contributing/coding_style.rst | 17 +++++++++++++++++
>  lib/cfgfile/rte_cfgfile.c                |  2 ++
>  2 files changed, 19 insertions(+)
>
> diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
> index 89db6260cf..307c7deb9a 100644
> --- a/doc/guides/contributing/coding_style.rst
> +++ b/doc/guides/contributing/coding_style.rst
> @@ -803,6 +803,21 @@ logging of a particular topic, the ``--log-level`` parameter can be provided
>  to EAL, which will change the log level. DPDK code can register topics,
>  which allows the user to adjust the log verbosity for that specific topic.
>
> +To register a library or driver for dynamic logging,

Nit: no need for a line break here, it can be fixed when applying.


> +using the standardized naming scheme described below,
> +use ``RTE_LOG_REGISTER_DEFAULT`` macro to define a log-type variable inside your component's main C file.
> +Thereafter, it is usual to define a macro or macros inside your component to make logging more convenient.
> +
> +For example, the ``rte_cfgfile`` library defines:
> +
> +.. literalinclude:: ../../../lib/cfgfile/rte_cfgfile.c
> +    :language: c
> +    :start-after: Setting up dynamic logging 8<
> +    :end-before: >8 End of setting up dynamic logging
> +
> +Dynamic Logging Naming Scheme
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
>  In general, the naming scheme is as follows: ``type.section.name``
>
>   * Type is the type of component, where ``lib``, ``pmd``, ``bus`` and ``user``
> @@ -837,6 +852,8 @@ A specialization looks like this:
>   * Initialization output: ``type.section.name.init``
>   * PF/VF mailbox output: ``type.section.name.mbox``
>
> +These specializations are created using the ``RTE_LOG_REGISTER_SUFFIX`` macro.
> +
>  A real world example is the i40e poll mode driver which exposes two
>  specializations, one for initialization ``pmd.net.i40e.init`` and the other for
>  the remaining driver logs ``pmd.net.i40e.driver``.
> diff --git a/lib/cfgfile/rte_cfgfile.c b/lib/cfgfile/rte_cfgfile.c
> index 9fa7d010ef..eefba6e408 100644
> --- a/lib/cfgfile/rte_cfgfile.c
> +++ b/lib/cfgfile/rte_cfgfile.c
> @@ -27,11 +27,13 @@ struct rte_cfgfile {
>         struct rte_cfgfile_section *sections;
>  };
>
> +/* Setting up dynamic logging 8< */
>  RTE_LOG_REGISTER_DEFAULT(cfgfile_logtype, INFO);
>
>  #define CFG_LOG(level, fmt, args...)                                   \
>         rte_log(RTE_LOG_ ## level, cfgfile_logtype, "%s(): " fmt "\n",  \
>                 __func__, ## args)
> +/* >8 End of setting up dynamic logging */
>
>  /** when we resize a file structure, how many extra entries
>   * for new sections do we add in */
> --
> 2.39.2
>

Thanks, this doc patch is a good addition in its current form.

Reviewed-by: David Marchand <david.marchand@redhat.com>


Stephen used a little trick with a #define RTE_LOGTYPE_$type
${type}_logtype so that RTE_LOG() is directly usable.
This makes conversion from static to dynamic logtypes rather easy.
Example: https://patchwork.dpdk.org/project/dpdk/patch/20230329234049.11071-5-stephen@networkplumber.org/
It could be something we advertise in this doc later.

Maybe some consistency cleanup (and guidance) would be good too, as
many components have close-yet-somewhat-different log formats (some
log the function name, others write the line number etc...).


-- 
David Marchand

Re: [PATCH v2 2/2] doc/contributing: guidelines for logging, tracing and telemetry

[David Marchand]

On Tue, Jun 20, 2023 at 7:08 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> As discussed by DPDK technical board [1], our contributor guide should
> include some details as to when to use logging vs tracing vs telemetry
> to provide the end user with information about the running process and
> the DPDK libraries it uses.
>
> [1] https://mails.dpdk.org/archives/dev/2023-March/265204.html
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
>
> ---
> V2:
> * Added note about not logging from unused drivers
> * Reworked bullets/sub-bullets on tracing vs debug logs for debugging.
>   - Consensus in replies was that people liked having debug logs for
>     single-use, e.g. init cases.
>   - Kept recommendation for tracing for data-path only
> * Added short discussion of *_dump() functions at end of section
> * Added sentence to indicate that telemetry should be read-only
> * Added mention of common trace format and that other tools are
>   available for it.
> ---
>  doc/guides/contributing/coding_style.rst |  2 +
>  doc/guides/contributing/design.rst       | 47 ++++++++++++++++++++++++
>  doc/guides/prog_guide/telemetry_lib.rst  |  2 +
>  doc/guides/prog_guide/trace_lib.rst      |  2 +
>  4 files changed, 53 insertions(+)
>
> diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
> index 307c7deb9a..0861305dc6 100644
> --- a/doc/guides/contributing/coding_style.rst
> +++ b/doc/guides/contributing/coding_style.rst
> @@ -794,6 +794,8 @@ Control Statements
>                   /* NOTREACHED */
>           }
>
> +.. _dynamic_logging:
> +
>  Dynamic Logging
>  ---------------
>
> diff --git a/doc/guides/contributing/design.rst b/doc/guides/contributing/design.rst
> index d24a7ff6a0..30104e0bfb 100644
> --- a/doc/guides/contributing/design.rst
> +++ b/doc/guides/contributing/design.rst
> @@ -4,6 +4,53 @@
>  Design
>  ======
>
> +Runtime Information - Logging, Tracing and Telemetry
> +-------------------------------------------------------

I would put this section right before the statistics section (since
telemetry and stats are somehow related).


> +
> +It is often desirable to provide information to the end-user as to what is happening to the application at runtime.
> +DPDK provides a number of built-in mechanisms to provide this introspection:
> +
> +* :ref:`Logging<dynamic_logging>`
> +* :ref:`Tracing<trace_library>`
> +* :ref:`Telemetry<telemetry_library>`
> +
> +Each of these has it's own strengths and suitabilities for use within DPDK components.

Nit: its* ?


> +
> +Below are some guidelines for when each should be used:
> +
> +* For reporting error conditions, or other abnormal runtime issues, *logging* should be used.
> +  Depending on the severity of the issue, the appropriate log level, for example,
> +  ``ERROR``, ``WARNING`` or ``NOTICE``, should be used.
> +
> +.. note::
> +
> +    Drivers off all classes, including both bus and device drivers,

Nit: of*

> +    should not output any log information if the hardware they support is not present.
> +    This is to avoid any changes in output for existing users when a new driver is added to DPDK.
> +
> +* For component initialization, or other cases where a path through the code is only likely to be taken once,
> +  either *logging* at ``DEBUG`` level or *tracing* may be used, or potentially both.
> +  In the latter case, tracing can provide basic information as to the code path taken,
> +  with debug-level logging providing additional details on internal state,
> +  not possible to emit via tracing.
> +
> +* For a component's data-path, where a path is to be taken multiple times within a short timeframe,
> +  *tracing* should be used.
> +  Since DPDK tracing uses `Common Trace Format <https://diamon.org/ctf/>`_ for its tracing logs,
> +  post-analysis can be done using a range of external tools.
> +
> +* For numerical or statistical data generated by a component, for example, per-packet statistics,
> +  *telemetry* should be used.
> +
> +* For any data where the data may need to be gathered at any point in the execution to help assess the state of the application component,
> +  for example, core configuration, device information, *telemetry* should be used.
> +  Telemetry callbacks should not modify any program state, but be "read-only".
> +
> +Many libraries also include a ``rte_<libname>_dump()`` function as part of their API,
> +writing verbose internal details to a given file-handle.
> +New libraries are encouraged to provide such functions where it makes sense to do so,
> +as they provide an additional application-controlled mechanism to get details of the internals of a DPDK component.
> +
>  Environment or Architecture-specific Sources
>  --------------------------------------------
>
> diff --git a/doc/guides/prog_guide/telemetry_lib.rst b/doc/guides/prog_guide/telemetry_lib.rst
> index 32f525a67f..71f8bd735e 100644
> --- a/doc/guides/prog_guide/telemetry_lib.rst
> +++ b/doc/guides/prog_guide/telemetry_lib.rst
> @@ -1,6 +1,8 @@
>  ..  SPDX-License-Identifier: BSD-3-Clause
>      Copyright(c) 2020 Intel Corporation.
>
> +.. _telemetry_library:
> +
>  Telemetry Library
>  =================
>
> diff --git a/doc/guides/prog_guide/trace_lib.rst b/doc/guides/prog_guide/trace_lib.rst
> index e5718feddc..a3b8a7c2eb 100644
> --- a/doc/guides/prog_guide/trace_lib.rst
> +++ b/doc/guides/prog_guide/trace_lib.rst
> @@ -1,6 +1,8 @@
>  ..  SPDX-License-Identifier: BSD-3-Clause
>      Copyright(C) 2020 Marvell International Ltd.
>
> +.. _trace_library:
> +
>  Trace Library
>  =============
>
> --
> 2.39.2
>

Reviewed-by: David Marchand <david.marchand@redhat.com>


-- 
David Marchand

[PATCH v3 0/2] Improve docs on getting info on running process

[Bruce Richardson]

Add details to our documentation on how to register logging for a
component. Also provide guidelines on when to use logs vs tracing
vs telemetry.

V3: Moved new doc section on runtime info, and other minor changes flagged
    by David on review.

V2: extended second patch based on the feedback provided.
    Hopefully, we can keep this section short and clear as the
    revisions roll! :-)

Bruce Richardson (2):
  doc/contributing: provide coding details for dynamic logging
  doc/contributing: guidelines for logging, tracing and telemetry

 doc/guides/contributing/coding_style.rst | 18 +++++++++
 doc/guides/contributing/design.rst       | 49 ++++++++++++++++++++++++
 doc/guides/prog_guide/telemetry_lib.rst  |  2 +
 doc/guides/prog_guide/trace_lib.rst      |  2 +
 lib/cfgfile/rte_cfgfile.c                |  2 +
 5 files changed, 73 insertions(+)

--
2.39.2

[PATCH v3 1/2] doc/contributing: provide coding details for dynamic logging

[Bruce Richardson]

While the section on dynamic logging in the contributors guide covered
the details of the logging naming scheme, it failed to cover exactly how
the component developer, i.e. the contributor, could actually use
dynamic logging in their component.

Fix this by splitting the details of the naming scheme into a separate
subsection, and adding to the introduction on logging, a recommendation
(and example) to use RTE_LOG_REGISTER_DEFAULT.

Similarly, when discussing specialization, include a reference to the
RTE_LOG_REGISTER_SUFFIX macro.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Chengwen Feng <fengchengwen@huawei.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>
---
 doc/guides/contributing/coding_style.rst | 16 ++++++++++++++++
 lib/cfgfile/rte_cfgfile.c                |  2 ++
 2 files changed, 18 insertions(+)

diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
index 00d6270624..f41f88c632 100644
--- a/doc/guides/contributing/coding_style.rst
+++ b/doc/guides/contributing/coding_style.rst
@@ -803,6 +803,20 @@ logging of a particular topic, the ``--log-level`` parameter can be provided
 to EAL, which will change the log level. DPDK code can register topics,
 which allows the user to adjust the log verbosity for that specific topic.
 
+To register a library or driver for dynamic logging, using the standardized naming scheme described below,
+use ``RTE_LOG_REGISTER_DEFAULT`` macro to define a log-type variable inside your component's main C file.
+Thereafter, it is usual to define a macro or macros inside your component to make logging more convenient.
+
+For example, the ``rte_cfgfile`` library defines:
+
+.. literalinclude:: ../../../lib/cfgfile/rte_cfgfile.c
+    :language: c
+    :start-after: Setting up dynamic logging 8<
+    :end-before: >8 End of setting up dynamic logging
+
+Dynamic Logging Naming Scheme
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 In general, the naming scheme is as follows: ``type.section.name``
 
  * Type is the type of component, where ``lib``, ``pmd``, ``bus`` and ``user``
@@ -837,6 +851,8 @@ A specialization looks like this:
  * Initialization output: ``type.section.name.init``
  * PF/VF mailbox output: ``type.section.name.mbox``
 
+These specializations are created using the ``RTE_LOG_REGISTER_SUFFIX`` macro.
+
 A real world example is the i40e poll mode driver which exposes two
 specializations, one for initialization ``pmd.net.i40e.init`` and the other for
 the remaining driver logs ``pmd.net.i40e.driver``.
diff --git a/lib/cfgfile/rte_cfgfile.c b/lib/cfgfile/rte_cfgfile.c
index 9fa7d010ef..eefba6e408 100644
--- a/lib/cfgfile/rte_cfgfile.c
+++ b/lib/cfgfile/rte_cfgfile.c
@@ -27,11 +27,13 @@ struct rte_cfgfile {
 	struct rte_cfgfile_section *sections;
 };
 
+/* Setting up dynamic logging 8< */
 RTE_LOG_REGISTER_DEFAULT(cfgfile_logtype, INFO);
 
 #define CFG_LOG(level, fmt, args...)					\
 	rte_log(RTE_LOG_ ## level, cfgfile_logtype, "%s(): " fmt "\n",	\
 		__func__, ## args)
+/* >8 End of setting up dynamic logging */
 
 /** when we resize a file structure, how many extra entries
  * for new sections do we add in */
-- 
2.39.2

[PATCH v3 2/2] doc/contributing: guidelines for logging, tracing and telemetry

[Bruce Richardson]

As discussed by DPDK technical board [1], our contributor guide should
include some details as to when to use logging vs tracing vs telemetry
to provide the end user with information about the running process and
the DPDK libraries it uses.

[1] https://mails.dpdk.org/archives/dev/2023-March/265204.html

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>
---
 doc/guides/contributing/coding_style.rst |  2 +
 doc/guides/contributing/design.rst       | 49 ++++++++++++++++++++++++
 doc/guides/prog_guide/telemetry_lib.rst  |  2 +
 doc/guides/prog_guide/trace_lib.rst      |  2 +
 4 files changed, 55 insertions(+)

diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
index f41f88c632..2d8d9a1469 100644
--- a/doc/guides/contributing/coding_style.rst
+++ b/doc/guides/contributing/coding_style.rst
@@ -794,6 +794,8 @@ Control Statements
                  /* NOTREACHED */
          }
 
+.. _dynamic_logging:
+
 Dynamic Logging
 ---------------
 
diff --git a/doc/guides/contributing/design.rst b/doc/guides/contributing/design.rst
index d24a7ff6a0..5a4c4b7ed1 100644
--- a/doc/guides/contributing/design.rst
+++ b/doc/guides/contributing/design.rst
@@ -4,6 +4,7 @@
 Design
 ======
 
+
 Environment or Architecture-specific Sources
 --------------------------------------------
 
@@ -68,6 +69,54 @@ Adding a new static field or flag must be an exception matching many criteria
 like (non exhaustive): wide usage, performance, size.
 
 
+Runtime Information - Logging, Tracing and Telemetry
+-------------------------------------------------------
+
+It is often desirable to provide information to the end-user as to what is happening to the application at runtime.
+DPDK provides a number of built-in mechanisms to provide this introspection:
+
+* :ref:`Logging<dynamic_logging>`
+* :ref:`Tracing<trace_library>`
+* :ref:`Telemetry<telemetry_library>`
+
+Each of these has its own strengths and suitabilities for use within DPDK components.
+
+Below are some guidelines for when each should be used:
+
+* For reporting error conditions, or other abnormal runtime issues, *logging* should be used.
+  Depending on the severity of the issue, the appropriate log level, for example,
+  ``ERROR``, ``WARNING`` or ``NOTICE``, should be used.
+
+.. note::
+
+    Drivers of all classes, including both bus and device drivers,
+    should not output any log information if the hardware they support is not present.
+    This is to avoid any changes in output for existing users when a new driver is added to DPDK.
+
+* For component initialization, or other cases where a path through the code is only likely to be taken once,
+  either *logging* at ``DEBUG`` level or *tracing* may be used, or potentially both.
+  In the latter case, tracing can provide basic information as to the code path taken,
+  with debug-level logging providing additional details on internal state,
+  not possible to emit via tracing.
+
+* For a component's data-path, where a path is to be taken multiple times within a short timeframe,
+  *tracing* should be used.
+  Since DPDK tracing uses `Common Trace Format <https://diamon.org/ctf/>`_ for its tracing logs,
+  post-analysis can be done using a range of external tools.
+
+* For numerical or statistical data generated by a component, for example, per-packet statistics,
+  *telemetry* should be used.
+
+* For any data where the data may need to be gathered at any point in the execution to help assess the state of the application component,
+  for example, core configuration, device information, *telemetry* should be used.
+  Telemetry callbacks should not modify any program state, but be "read-only".
+
+Many libraries also include a ``rte_<libname>_dump()`` function as part of their API,
+writing verbose internal details to a given file-handle.
+New libraries are encouraged to provide such functions where it makes sense to do so,
+as they provide an additional application-controlled mechanism to get details of the internals of a DPDK component.
+
+
 Library Statistics
 ------------------
 
diff --git a/doc/guides/prog_guide/telemetry_lib.rst b/doc/guides/prog_guide/telemetry_lib.rst
index 32f525a67f..71f8bd735e 100644
--- a/doc/guides/prog_guide/telemetry_lib.rst
+++ b/doc/guides/prog_guide/telemetry_lib.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright(c) 2020 Intel Corporation.
 
+.. _telemetry_library:
+
 Telemetry Library
 =================
 
diff --git a/doc/guides/prog_guide/trace_lib.rst b/doc/guides/prog_guide/trace_lib.rst
index e5718feddc..a3b8a7c2eb 100644
--- a/doc/guides/prog_guide/trace_lib.rst
+++ b/doc/guides/prog_guide/trace_lib.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright(C) 2020 Marvell International Ltd.
 
+.. _trace_library:
+
 Trace Library
 =============
 
-- 
2.39.2

Re: [PATCH v3 0/2] Improve docs on getting info on running process

[Stephen Hemminger]

On Tue, 18 Jul 2023 17:48:00 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:

> Add details to our documentation on how to register logging for a
> component. Also provide guidelines on when to use logs vs tracing
> vs telemetry.
> 
> V3: Moved new doc section on runtime info, and other minor changes flagged
>     by David on review.
> 
> V2: extended second patch based on the feedback provided.
>     Hopefully, we can keep this section short and clear as the
>     revisions roll! :-)
> 
> Bruce Richardson (2):
>   doc/contributing: provide coding details for dynamic logging
>   doc/contributing: guidelines for logging, tracing and telemetry
> 
>  doc/guides/contributing/coding_style.rst | 18 +++++++++
>  doc/guides/contributing/design.rst       | 49 ++++++++++++++++++++++++
>  doc/guides/prog_guide/telemetry_lib.rst  |  2 +
>  doc/guides/prog_guide/trace_lib.rst      |  2 +
>  lib/cfgfile/rte_cfgfile.c                |  2 +
>  5 files changed, 73 insertions(+)
> 
> --
> 2.39.2
> 

Should we add explicit statements about:
  - Static logtypes (especially LOGTYPE_PMD) should not be used.
  - New static logtypes must not be added.

Re: [PATCH v3 2/2] doc/contributing: guidelines for logging, tracing and telemetry

[Ferruh Yigit]

On 7/18/2023 5:48 PM, Bruce Richardson wrote:
> As discussed by DPDK technical board [1], our contributor guide should
> include some details as to when to use logging vs tracing vs telemetry
> to provide the end user with information about the running process and
> the DPDK libraries it uses.
> 
> [1] https://mails.dpdk.org/archives/dev/2023-March/265204.html
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> Reviewed-by: David Marchand <david.marchand@redhat.com>
>

Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>

> ---
>  doc/guides/contributing/coding_style.rst |  2 +
>  doc/guides/contributing/design.rst       | 49 ++++++++++++++++++++++++
>  doc/guides/prog_guide/telemetry_lib.rst  |  2 +
>  doc/guides/prog_guide/trace_lib.rst      |  2 +
>  4 files changed, 55 insertions(+)
> 
> diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
> index f41f88c632..2d8d9a1469 100644
> --- a/doc/guides/contributing/coding_style.rst
> +++ b/doc/guides/contributing/coding_style.rst
> @@ -794,6 +794,8 @@ Control Statements
>                   /* NOTREACHED */
>           }
>  
> +.. _dynamic_logging:
> +
>  Dynamic Logging
>  ---------------
>  
> diff --git a/doc/guides/contributing/design.rst b/doc/guides/contributing/design.rst
> index d24a7ff6a0..5a4c4b7ed1 100644
> --- a/doc/guides/contributing/design.rst
> +++ b/doc/guides/contributing/design.rst
> @@ -4,6 +4,7 @@
>  Design
>  ======
>  
> +
>  Environment or Architecture-specific Sources
>  --------------------------------------------
>  
> @@ -68,6 +69,54 @@ Adding a new static field or flag must be an exception matching many criteria
>  like (non exhaustive): wide usage, performance, size.
>  
>  
> +Runtime Information - Logging, Tracing and Telemetry
> +-------------------------------------------------------
> +
> +It is often desirable to provide information to the end-user as to what is happening to the application at runtime.
> +DPDK provides a number of built-in mechanisms to provide this introspection:
> +
> +* :ref:`Logging<dynamic_logging>`
> +* :ref:`Tracing<trace_library>`
> +* :ref:`Telemetry<telemetry_library>`
> +
> +Each of these has its own strengths and suitabilities for use within DPDK components.
> +
> +Below are some guidelines for when each should be used:
> +
> +* For reporting error conditions, or other abnormal runtime issues, *logging* should be used.
> +  Depending on the severity of the issue, the appropriate log level, for example,
> +  ``ERROR``, ``WARNING`` or ``NOTICE``, should be used.
> +
> +.. note::
> +
> +    Drivers of all classes, including both bus and device drivers,
> +    should not output any log information if the hardware they support is not present.
> +    This is to avoid any changes in output for existing users when a new driver is added to DPDK.
> +

+1 to note, this is a good addition.

> +* For component initialization, or other cases where a path through the code is only likely to be taken once,
> +  either *logging* at ``DEBUG`` level or *tracing* may be used, or potentially both.
> +  In the latter case, tracing can provide basic information as to the code path taken,
> +  with debug-level logging providing additional details on internal state,
> +  not possible to emit via tracing.
> +
> +* For a component's data-path, where a path is to be taken multiple times within a short timeframe,
> +  *tracing* should be used.
> +  Since DPDK tracing uses `Common Trace Format <https://diamon.org/ctf/>`_ for its tracing logs,
> +  post-analysis can be done using a range of external tools.
> +
> +* For numerical or statistical data generated by a component, for example, per-packet statistics,
> +  *telemetry* should be used.
> +
> +* For any data where the data may need to be gathered at any point in the execution to help assess the state of the application component,
> +  for example, core configuration, device information, *telemetry* should be used.
> +  Telemetry callbacks should not modify any program state, but be "read-only".
> +

+1

> +Many libraries also include a ``rte_<libname>_dump()`` function as part of their API,
> +writing verbose internal details to a given file-handle.
> +New libraries are encouraged to provide such functions where it makes sense to do so,
> +as they provide an additional application-controlled mechanism to get details of the internals of a DPDK component.
> +
> +
>  Library Statistics
>  ------------------
>  
> diff --git a/doc/guides/prog_guide/telemetry_lib.rst b/doc/guides/prog_guide/telemetry_lib.rst
> index 32f525a67f..71f8bd735e 100644
> --- a/doc/guides/prog_guide/telemetry_lib.rst
> +++ b/doc/guides/prog_guide/telemetry_lib.rst
> @@ -1,6 +1,8 @@
>  ..  SPDX-License-Identifier: BSD-3-Clause
>      Copyright(c) 2020 Intel Corporation.
>  
> +.. _telemetry_library:
> +
>  Telemetry Library
>  =================
>  
> diff --git a/doc/guides/prog_guide/trace_lib.rst b/doc/guides/prog_guide/trace_lib.rst
> index e5718feddc..a3b8a7c2eb 100644
> --- a/doc/guides/prog_guide/trace_lib.rst
> +++ b/doc/guides/prog_guide/trace_lib.rst
> @@ -1,6 +1,8 @@
>  ..  SPDX-License-Identifier: BSD-3-Clause
>      Copyright(C) 2020 Marvell International Ltd.
>  
> +.. _trace_library:
> +
>  Trace Library
>  =============
>  

Re: [PATCH v3 1/2] doc/contributing: provide coding details for dynamic logging

[Ferruh Yigit]

On 7/18/2023 5:48 PM, Bruce Richardson wrote:
> While the section on dynamic logging in the contributors guide covered
> the details of the logging naming scheme, it failed to cover exactly how
> the component developer, i.e. the contributor, could actually use
> dynamic logging in their component.
> 
> Fix this by splitting the details of the naming scheme into a separate
> subsection, and adding to the introduction on logging, a recommendation
> (and example) to use RTE_LOG_REGISTER_DEFAULT.
> 
> Similarly, when discussing specialization, include a reference to the
> RTE_LOG_REGISTER_SUFFIX macro.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Chengwen Feng <fengchengwen@huawei.com>
> Reviewed-by: David Marchand <david.marchand@redhat.com>
>

Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>

Re: [PATCH v3 2/2] doc/contributing: guidelines for logging, tracing and telemetry

[lihuisong (C)]

在 2023/7/19 0:48, Bruce Richardson 写道:
> As discussed by DPDK technical board [1], our contributor guide should
> include some details as to when to use logging vs tracing vs telemetry
> to provide the end user with information about the running process and
> the DPDK libraries it uses.
>
> [1] https://mails.dpdk.org/archives/dev/2023-March/265204.html
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> Reviewed-by: David Marchand <david.marchand@redhat.com>
> ---
>   doc/guides/contributing/coding_style.rst |  2 +
>   doc/guides/contributing/design.rst       | 49 ++++++++++++++++++++++++
>   doc/guides/prog_guide/telemetry_lib.rst  |  2 +
>   doc/guides/prog_guide/trace_lib.rst      |  2 +
>   4 files changed, 55 insertions(+)
>
> diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
> index f41f88c632..2d8d9a1469 100644
> --- a/doc/guides/contributing/coding_style.rst
> +++ b/doc/guides/contributing/coding_style.rst
> @@ -794,6 +794,8 @@ Control Statements
>                    /* NOTREACHED */
>            }
>   
> +.. _dynamic_logging:
> +
>   Dynamic Logging
>   ---------------
>   
> diff --git a/doc/guides/contributing/design.rst b/doc/guides/contributing/design.rst
> index d24a7ff6a0..5a4c4b7ed1 100644
> --- a/doc/guides/contributing/design.rst
> +++ b/doc/guides/contributing/design.rst
> @@ -4,6 +4,7 @@
>   Design
>   ======
>   
> +
>   Environment or Architecture-specific Sources
>   --------------------------------------------
>   
> @@ -68,6 +69,54 @@ Adding a new static field or flag must be an exception matching many criteria
>   like (non exhaustive): wide usage, performance, size.
>   
>   
> +Runtime Information - Logging, Tracing and Telemetry
> +-------------------------------------------------------
> +
> +It is often desirable to provide information to the end-user as to what is happening to the application at runtime.
> +DPDK provides a number of built-in mechanisms to provide this introspection:
> +
> +* :ref:`Logging<dynamic_logging>`
> +* :ref:`Tracing<trace_library>`
> +* :ref:`Telemetry<telemetry_library>`
> +
> +Each of these has its own strengths and suitabilities for use within DPDK components.
> +
> +Below are some guidelines for when each should be used:
> +
> +* For reporting error conditions, or other abnormal runtime issues, *logging* should be used.
> +  Depending on the severity of the issue, the appropriate log level, for example,
> +  ``ERROR``, ``WARNING`` or ``NOTICE``, should be used.
> +
> +.. note::
> +
> +    Drivers of all classes, including both bus and device drivers,
> +    should not output any log information if the hardware they support is not present.
> +    This is to avoid any changes in output for existing users when a new driver is added to DPDK.
> +
> +* For component initialization, or other cases where a path through the code is only likely to be taken once,
> +  either *logging* at ``DEBUG`` level or *tracing* may be used, or potentially both.
> +  In the latter case, tracing can provide basic information as to the code path taken,
> +  with debug-level logging providing additional details on internal state,
> +  not possible to emit via tracing.
> +
> +* For a component's data-path, where a path is to be taken multiple times within a short timeframe,
> +  *tracing* should be used.
> +  Since DPDK tracing uses `Common Trace Format <https://diamon.org/ctf/>`_ for its tracing logs,
> +  post-analysis can be done using a range of external tools.
> +
> +* For numerical or statistical data generated by a component, for example, per-packet statistics,
> +  *telemetry* should be used.
> +
> +* For any data where the data may need to be gathered at any point in the execution to help assess the state of the application component,
> +  for example, core configuration, device information, *telemetry* should be used.
> +  Telemetry callbacks should not modify any program state, but be "read-only".
> +
> +Many libraries also include a ``rte_<libname>_dump()`` function as part of their API,
> +writing verbose internal details to a given file-handle.
> +New libraries are encouraged to provide such functions where it makes sense to do so,
> +as they provide an additional application-controlled mechanism to get details of the internals of a DPDK component.
> +
+1 now more clear
Acked-by: Huisong Li <lihuisong@huawei.com>
> +
>   Library Statistics
>   ------------------
>   
> diff --git a/doc/guides/prog_guide/telemetry_lib.rst b/doc/guides/prog_guide/telemetry_lib.rst
> index 32f525a67f..71f8bd735e 100644
> --- a/doc/guides/prog_guide/telemetry_lib.rst
> +++ b/doc/guides/prog_guide/telemetry_lib.rst
> @@ -1,6 +1,8 @@
>   ..  SPDX-License-Identifier: BSD-3-Clause
>       Copyright(c) 2020 Intel Corporation.
>   
> +.. _telemetry_library:
> +
>   Telemetry Library
>   =================
>   
> diff --git a/doc/guides/prog_guide/trace_lib.rst b/doc/guides/prog_guide/trace_lib.rst
> index e5718feddc..a3b8a7c2eb 100644
> --- a/doc/guides/prog_guide/trace_lib.rst
> +++ b/doc/guides/prog_guide/trace_lib.rst
> @@ -1,6 +1,8 @@
>   ..  SPDX-License-Identifier: BSD-3-Clause
>       Copyright(C) 2020 Marvell International Ltd.
>   
> +.. _trace_library:
> +
>   Trace Library
>   =============
>   

Re: [PATCH v3 0/2] Improve docs on getting info on running process

[Bruce Richardson]

On Tue, Jul 18, 2023 at 10:40:20AM -0700, Stephen Hemminger wrote:
> On Tue, 18 Jul 2023 17:48:00 +0100
> Bruce Richardson <bruce.richardson@intel.com> wrote:
> 
> > Add details to our documentation on how to register logging for a
> > component. Also provide guidelines on when to use logs vs tracing
> > vs telemetry.
> > 
> > V3: Moved new doc section on runtime info, and other minor changes flagged
> >     by David on review.
> > 
> > V2: extended second patch based on the feedback provided.
> >     Hopefully, we can keep this section short and clear as the
> >     revisions roll! :-)
> > 
> > Bruce Richardson (2):
> >   doc/contributing: provide coding details for dynamic logging
> >   doc/contributing: guidelines for logging, tracing and telemetry
> > 
> >  doc/guides/contributing/coding_style.rst | 18 +++++++++
> >  doc/guides/contributing/design.rst       | 49 ++++++++++++++++++++++++
> >  doc/guides/prog_guide/telemetry_lib.rst  |  2 +
> >  doc/guides/prog_guide/trace_lib.rst      |  2 +
> >  lib/cfgfile/rte_cfgfile.c                |  2 +
> >  5 files changed, 73 insertions(+)
> > 
> > --
> > 2.39.2
> > 
> 
> Should we add explicit statements about:
>   - Static logtypes (especially LOGTYPE_PMD) should not be used.
>   - New static logtypes must not be added.

I would hope that should not be necessary especially since this section
talks about dynamic logging only. However, no harm in adding a
one-line note about it. I'll see if I can get a few minutes to spin a V4
with an extra note, but if not I think this set is good to go as-is.

/Bruce

Re: [PATCH v3 0/2] Improve docs on getting info on running process

[Bruce Richardson]

On Tue, Jul 18, 2023 at 10:40:20AM -0700, Stephen Hemminger wrote:
> On Tue, 18 Jul 2023 17:48:00 +0100
> Bruce Richardson <bruce.richardson@intel.com> wrote:
> 
> > Add details to our documentation on how to register logging for a
> > component. Also provide guidelines on when to use logs vs tracing
> > vs telemetry.
> > 
> > V3: Moved new doc section on runtime info, and other minor changes flagged
> >     by David on review.
> > 
> > V2: extended second patch based on the feedback provided.
> >     Hopefully, we can keep this section short and clear as the
> >     revisions roll! :-)
> > 
> > Bruce Richardson (2):
> >   doc/contributing: provide coding details for dynamic logging
> >   doc/contributing: guidelines for logging, tracing and telemetry
> > 
> >  doc/guides/contributing/coding_style.rst | 18 +++++++++
> >  doc/guides/contributing/design.rst       | 49 ++++++++++++++++++++++++
> >  doc/guides/prog_guide/telemetry_lib.rst  |  2 +
> >  doc/guides/prog_guide/trace_lib.rst      |  2 +
> >  lib/cfgfile/rte_cfgfile.c                |  2 +
> >  5 files changed, 73 insertions(+)
> > 
> > --
> > 2.39.2
> > 
> 
> Should we add explicit statements about:
>   - Static logtypes (especially LOGTYPE_PMD) should not be used.
>   - New static logtypes must not be added.

How about:

+.. note::
+
+   The statically-defined log types defined in ``rte_log.h`` are for legacy components,
+   and they will likely be removed in a future release.
+   Do not add new entries to this file.
+

Any suggested rewording before I spin a new patch?

/Bruce

Re: [PATCH v3 2/2] doc/contributing: guidelines for logging, tracing and telemetry

[Jerin Jacob]

On Wed, Jul 19, 2023 at 6:38 AM lihuisong (C) <lihuisong@huawei.com> wrote:
>
>
> 在 2023/7/19 0:48, Bruce Richardson 写道:
> > As discussed by DPDK technical board [1], our contributor guide should
> > include some details as to when to use logging vs tracing vs telemetry
> > to provide the end user with information about the running process and
> > the DPDK libraries it uses.
> >
> > [1] https://mails.dpdk.org/archives/dev/2023-March/265204.html
> >
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > Acked-by: Morten Brørup <mb@smartsharesystems.com>
> > Reviewed-by: David Marchand <david.marchand@redhat.com>

Acked-by: Jerin Jacob <jerinj@marvell.com>

[PATCH v4 0/2] Improve docs on getting info on running process

[Bruce Richardson]

Add details to our documentation on how to register logging for a
component. Also provide guidelines on when to use logs vs tracing
vs telemetry.

V4: Added note to patch 1 saying not to use the old static logging method

V3: Moved new doc section on runtime info, and other minor changes flagged
    by David on review.

V2: extended second patch based on the feedback provided.

Bruce Richardson (2):
  doc/contributing: provide coding details for dynamic logging
  doc/contributing: guidelines for logging, tracing and telemetry

 doc/guides/contributing/coding_style.rst | 24 ++++++++++++
 doc/guides/contributing/design.rst       | 49 ++++++++++++++++++++++++
 doc/guides/prog_guide/telemetry_lib.rst  |  2 +
 doc/guides/prog_guide/trace_lib.rst      |  2 +
 lib/cfgfile/rte_cfgfile.c                |  2 +
 5 files changed, 79 insertions(+)

--
2.39.2

[PATCH v4 1/2] doc/contributing: provide coding details for dynamic logging

[Bruce Richardson]

While the section on dynamic logging in the contributors guide covered
the details of the logging naming scheme, it failed to cover exactly how
the component developer, i.e. the contributor, could actually use
dynamic logging in their component.

Fix this by splitting the details of the naming scheme into a separate
subsection, and adding to the introduction on logging, a recommendation
(and example) to use RTE_LOG_REGISTER_DEFAULT.

Similarly, when discussing specialization, include a reference to the
RTE_LOG_REGISTER_SUFFIX macro.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Chengwen Feng <fengchengwen@huawei.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>
---
 doc/guides/contributing/coding_style.rst | 22 ++++++++++++++++++++++
 lib/cfgfile/rte_cfgfile.c                |  2 ++
 2 files changed, 24 insertions(+)

diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
index 00d6270624..383942a601 100644
--- a/doc/guides/contributing/coding_style.rst
+++ b/doc/guides/contributing/coding_style.rst
@@ -803,6 +803,26 @@ logging of a particular topic, the ``--log-level`` parameter can be provided
 to EAL, which will change the log level. DPDK code can register topics,
 which allows the user to adjust the log verbosity for that specific topic.
 
+To register a library or driver for dynamic logging, using the standardized naming scheme described below,
+use ``RTE_LOG_REGISTER_DEFAULT`` macro to define a log-type variable inside your component's main C file.
+Thereafter, it is usual to define a macro or macros inside your component to make logging more convenient.
+
+For example, the ``rte_cfgfile`` library defines:
+
+.. literalinclude:: ../../../lib/cfgfile/rte_cfgfile.c
+    :language: c
+    :start-after: Setting up dynamic logging 8<
+    :end-before: >8 End of setting up dynamic logging
+
+.. note::
+
+   The statically-defined log types defined in ``rte_log.h`` are for legacy components,
+   and they will likely be removed in a future release.
+   Do not add new entries to this file.
+
+Dynamic Logging Naming Scheme
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 In general, the naming scheme is as follows: ``type.section.name``
 
  * Type is the type of component, where ``lib``, ``pmd``, ``bus`` and ``user``
@@ -837,6 +857,8 @@ A specialization looks like this:
  * Initialization output: ``type.section.name.init``
  * PF/VF mailbox output: ``type.section.name.mbox``
 
+These specializations are created using the ``RTE_LOG_REGISTER_SUFFIX`` macro.
+
 A real world example is the i40e poll mode driver which exposes two
 specializations, one for initialization ``pmd.net.i40e.init`` and the other for
 the remaining driver logs ``pmd.net.i40e.driver``.
diff --git a/lib/cfgfile/rte_cfgfile.c b/lib/cfgfile/rte_cfgfile.c
index 9fa7d010ef..eefba6e408 100644
--- a/lib/cfgfile/rte_cfgfile.c
+++ b/lib/cfgfile/rte_cfgfile.c
@@ -27,11 +27,13 @@ struct rte_cfgfile {
 	struct rte_cfgfile_section *sections;
 };
 
+/* Setting up dynamic logging 8< */
 RTE_LOG_REGISTER_DEFAULT(cfgfile_logtype, INFO);
 
 #define CFG_LOG(level, fmt, args...)					\
 	rte_log(RTE_LOG_ ## level, cfgfile_logtype, "%s(): " fmt "\n",	\
 		__func__, ## args)
+/* >8 End of setting up dynamic logging */
 
 /** when we resize a file structure, how many extra entries
  * for new sections do we add in */
-- 
2.39.2

[PATCH v4 2/2] doc/contributing: guidelines for logging, tracing and telemetry

[Bruce Richardson]

As discussed by DPDK technical board [1], our contributor guide should
include some details as to when to use logging vs tracing vs telemetry
to provide the end user with information about the running process and
the DPDK libraries it uses.

[1] https://mails.dpdk.org/archives/dev/2023-March/265204.html

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>
---
 doc/guides/contributing/coding_style.rst |  2 +
 doc/guides/contributing/design.rst       | 49 ++++++++++++++++++++++++
 doc/guides/prog_guide/telemetry_lib.rst  |  2 +
 doc/guides/prog_guide/trace_lib.rst      |  2 +
 4 files changed, 55 insertions(+)

diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
index 383942a601..37e2cc5efb 100644
--- a/doc/guides/contributing/coding_style.rst
+++ b/doc/guides/contributing/coding_style.rst
@@ -794,6 +794,8 @@ Control Statements
                  /* NOTREACHED */
          }
 
+.. _dynamic_logging:
+
 Dynamic Logging
 ---------------
 
diff --git a/doc/guides/contributing/design.rst b/doc/guides/contributing/design.rst
index d24a7ff6a0..5a4c4b7ed1 100644
--- a/doc/guides/contributing/design.rst
+++ b/doc/guides/contributing/design.rst
@@ -4,6 +4,7 @@
 Design
 ======
 
+
 Environment or Architecture-specific Sources
 --------------------------------------------
 
@@ -68,6 +69,54 @@ Adding a new static field or flag must be an exception matching many criteria
 like (non exhaustive): wide usage, performance, size.
 
 
+Runtime Information - Logging, Tracing and Telemetry
+-------------------------------------------------------
+
+It is often desirable to provide information to the end-user as to what is happening to the application at runtime.
+DPDK provides a number of built-in mechanisms to provide this introspection:
+
+* :ref:`Logging<dynamic_logging>`
+* :ref:`Tracing<trace_library>`
+* :ref:`Telemetry<telemetry_library>`
+
+Each of these has its own strengths and suitabilities for use within DPDK components.
+
+Below are some guidelines for when each should be used:
+
+* For reporting error conditions, or other abnormal runtime issues, *logging* should be used.
+  Depending on the severity of the issue, the appropriate log level, for example,
+  ``ERROR``, ``WARNING`` or ``NOTICE``, should be used.
+
+.. note::
+
+    Drivers of all classes, including both bus and device drivers,
+    should not output any log information if the hardware they support is not present.
+    This is to avoid any changes in output for existing users when a new driver is added to DPDK.
+
+* For component initialization, or other cases where a path through the code is only likely to be taken once,
+  either *logging* at ``DEBUG`` level or *tracing* may be used, or potentially both.
+  In the latter case, tracing can provide basic information as to the code path taken,
+  with debug-level logging providing additional details on internal state,
+  not possible to emit via tracing.
+
+* For a component's data-path, where a path is to be taken multiple times within a short timeframe,
+  *tracing* should be used.
+  Since DPDK tracing uses `Common Trace Format <https://diamon.org/ctf/>`_ for its tracing logs,
+  post-analysis can be done using a range of external tools.
+
+* For numerical or statistical data generated by a component, for example, per-packet statistics,
+  *telemetry* should be used.
+
+* For any data where the data may need to be gathered at any point in the execution to help assess the state of the application component,
+  for example, core configuration, device information, *telemetry* should be used.
+  Telemetry callbacks should not modify any program state, but be "read-only".
+
+Many libraries also include a ``rte_<libname>_dump()`` function as part of their API,
+writing verbose internal details to a given file-handle.
+New libraries are encouraged to provide such functions where it makes sense to do so,
+as they provide an additional application-controlled mechanism to get details of the internals of a DPDK component.
+
+
 Library Statistics
 ------------------
 
diff --git a/doc/guides/prog_guide/telemetry_lib.rst b/doc/guides/prog_guide/telemetry_lib.rst
index 32f525a67f..71f8bd735e 100644
--- a/doc/guides/prog_guide/telemetry_lib.rst
+++ b/doc/guides/prog_guide/telemetry_lib.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright(c) 2020 Intel Corporation.
 
+.. _telemetry_library:
+
 Telemetry Library
 =================
 
diff --git a/doc/guides/prog_guide/trace_lib.rst b/doc/guides/prog_guide/trace_lib.rst
index e5718feddc..a3b8a7c2eb 100644
--- a/doc/guides/prog_guide/trace_lib.rst
+++ b/doc/guides/prog_guide/trace_lib.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright(C) 2020 Marvell International Ltd.
 
+.. _trace_library:
+
 Trace Library
 =============
 
-- 
2.39.2

Re: [PATCH v4 0/2] Improve docs on getting info on running process

[Ferruh Yigit]

On 7/20/2023 12:05 PM, Bruce Richardson wrote:
> Add details to our documentation on how to register logging for a
> component. Also provide guidelines on when to use logs vs tracing
> vs telemetry.
> 
> V4: Added note to patch 1 saying not to use the old static logging method
> 
> V3: Moved new doc section on runtime info, and other minor changes flagged
>     by David on review.
> 
> V2: extended second patch based on the feedback provided.
> 
> Bruce Richardson (2):
>   doc/contributing: provide coding details for dynamic logging
>   doc/contributing: guidelines for logging, tracing and telemetry
> 

For series,
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>

Re: [PATCH v4 2/2] doc/contributing: guidelines for logging, tracing and telemetry

[Thomas Monjalon]

20/07/2023 13:05, Bruce Richardson:
> --- a/doc/guides/prog_guide/telemetry_lib.rst
> +++ b/doc/guides/prog_guide/telemetry_lib.rst
> @@ -1,6 +1,8 @@
>  ..  SPDX-License-Identifier: BSD-3-Clause
>      Copyright(c) 2020 Intel Corporation.
>  
> +.. _telemetry_library:
> +
>  Telemetry Library
>  =================

No need to add such anchor at the beginning of a file.
The syntax :doc: can be used to reference such page.
I fix it while applying.

Applied, thanks.