Re: [RFC] Dynamic log/trace control via telemetry

[Stephen Hemminger <stephen@networkplumber.org>]

On Tue, 16 Aug 2022 02:17:38 +0300
Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:

> When debugging a live app, useful info can be obtained from logs or traces
> that were not enabled when it was started and it is undesirable to restart.
> Furthermore, unless the app authors have considered tracing,
> rte_trace_save() is only called on exit, i.e. a shutdown is required again.
> 
> What if the telemetry socket gave the missing control?
> For example:
> 
> --> /eal/log/set_level,debug  
> {"/eal/log/set_level": {"status": "success"}}
> --> /eal/log/set_level,pmd.net.*:debug  
> {"/eal/log/set_level": {"status": "success"}}
> --> /eal/log/set_level,lib.[a-d],debug  
> {"/eal/log/set_level": {"status": "success"}}
> --> /eal/log/set_level,foobar  
> {"/eal/log/set_level": {"status": "fail", "message": "Invalid log level: foobar"}}
> 
> Tracing is more complicated because it requires resource allocation.
> It could be:
> 
> /eal/trace/set_mode,discard
> /eal/trace/set_mode,overwrite
>     rte_trace_mode_set()
> 
> /eal/trace/enable_pattern,<pattern>
> /eal/trace/enable_regex,<regex>
> /eal/trace/disable_pattern,<pattern>
> /eal/trace/disable_regex,<regex>
>     rte_trace_enable_pattern()
>     rte_trace_enable_regex()
> 
> /eal/trace/rearm
>     Clear the trace buffer.
>     Apply the new settings accumulated by the previous commands.
> 
> /eal/trace/save
>     rte_trace_save()
> 
> An open question is how to deal with multi-process.
> Only the primary process listens to the telemetry socket.
> Log and trace settings are not shared between processes.
> OTOH, when enabling some log source, it is desirable to do in all processes.
> 
> In general, telemetry is not for changing the state of the app,
> but logs and traces are diagnostic information that seems to be in-scope.
> A similar suggestion was not opposed recently:
> 
>     http://inbox.dpdk.org/dev/24c49429394294cfbf0d9c506b205029bac77c8b.1657890378.git.anatoly.burakov@intel.com/

Not sure if turning telemetry into a do all control api makes sense.
This seems like a different API.
Also, the default would have to be disabled for application safety reasons.