[RFC] Dynamic log/trace control via telemetry

[Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>]

When debugging a live app, useful info can be obtained from logs or traces
that were not enabled when it was started and it is undesirable to restart.
Furthermore, unless the app authors have considered tracing,
rte_trace_save() is only called on exit, i.e. a shutdown is required again.

What if the telemetry socket gave the missing control?
For example:

--> /eal/log/set_level,debug
{"/eal/log/set_level": {"status": "success"}}
--> /eal/log/set_level,pmd.net.*:debug
{"/eal/log/set_level": {"status": "success"}}
--> /eal/log/set_level,lib.[a-d],debug
{"/eal/log/set_level": {"status": "success"}}
--> /eal/log/set_level,foobar
{"/eal/log/set_level": {"status": "fail", "message": "Invalid log level: foobar"}}

Tracing is more complicated because it requires resource allocation.
It could be:

/eal/trace/set_mode,discard
/eal/trace/set_mode,overwrite
    rte_trace_mode_set()

/eal/trace/enable_pattern,<pattern>
/eal/trace/enable_regex,<regex>
/eal/trace/disable_pattern,<pattern>
/eal/trace/disable_regex,<regex>
    rte_trace_enable_pattern()
    rte_trace_enable_regex()

/eal/trace/rearm
    Clear the trace buffer.
    Apply the new settings accumulated by the previous commands.

/eal/trace/save
    rte_trace_save()

An open question is how to deal with multi-process.
Only the primary process listens to the telemetry socket.
Log and trace settings are not shared between processes.
OTOH, when enabling some log source, it is desirable to do in all processes.

In general, telemetry is not for changing the state of the app,
but logs and traces are diagnostic information that seems to be in-scope.
A similar suggestion was not opposed recently:

    http://inbox.dpdk.org/dev/24c49429394294cfbf0d9c506b205029bac77c8b.1657890378.git.anatoly.burakov@intel.com/