From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 96A49A0032; Wed, 17 Aug 2022 04:08:43 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3726D40150; Wed, 17 Aug 2022 04:08:43 +0200 (CEST) Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) by mails.dpdk.org (Postfix) with ESMTP id 47FD5400D6 for ; Wed, 17 Aug 2022 04:08:41 +0200 (CEST) Received: by mail-pg1-f179.google.com with SMTP id c24so10818052pgg.11 for ; Tue, 16 Aug 2022 19:08:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc; bh=k3PxcdoAg4tAEZdtjF/IT3hhSkQLDYvh8fX6+QNwdvo=; b=UXaZhbK5lgovhFM+f+t5hqV0mFjx1eS6VkOmv3Vga0skyF3Ktj8OE7nUVqAeYPL4ME GWcjqDgRYXYD6SkHSL0YjjVe9EKZtLn0usZZIgb03ayoR8na3mEqKP/YbfLYmjqxEAqD QC5T0WDQ5WDYvJz3Jespsy61jkAMSZdhZUv/fFwobzmjyfT/SzT3cWld+xH5UFa6Nlev aXbQgHFA6EV29nQNz0vXF3lKufoiUcaPek8j4ohTquzRkvuA8XmTZDk2sUaU8uMNKVNx 1NbYEZX/o+/x26wrh7rLekIerbH7oZSIZ0NCYfWfUsPkpHXWLqOPfbGssKY8Cn9cE62F zlzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc; bh=k3PxcdoAg4tAEZdtjF/IT3hhSkQLDYvh8fX6+QNwdvo=; b=uXtUaMZbhnU/iALvtbkRi2ZdgpCFbj1Yimc8D/b2MtSpB06zoCMp4BTUn0KoXp6oON /KKVgRsKsHGu27nLwzw5OB1Lk1aTg9Z2xOvcrFWSBoCkEuiyW5Zb8anKUca/fX8cAk8W Hoj5KL/5b9H2i0B0rVo0wvF3pNdrCfwUOgo1uA/NcPVdkbaKCCXl9iTvBehgeeAzOfLo Iq2TDkYUt8d8wUUR/ubZwBXzx1MxhgwffFehXlCRUk8B7MExc774okfZjgEX2Vl6Hbqn +pmiVrMZ2/PVrURHsVaJofQVzpBmX26Y5i8ksFhI8IZXmx0LmBgZpnZY5QT+j74/6z8A 0pzg== X-Gm-Message-State: ACgBeo17TnEBrENMNDK2yO48G0ozKqa/BTMvtfGNQGfrJnqXZ/hEXraI p923Jepq0K+jq7AA0HclKetEkOb+hRQMGQ== X-Google-Smtp-Source: AA6agR60WNW4W7tOHhXqMbq1nprLO+A2+7QlvSwef0ElKbJLJpuje96xfVCXMlcY1RzAhd8TsuC7mA== X-Received: by 2002:a05:6a00:b8d:b0:52d:dead:5570 with SMTP id g13-20020a056a000b8d00b0052ddead5570mr23264513pfj.56.1660702120339; Tue, 16 Aug 2022 19:08:40 -0700 (PDT) Received: from hermes.local (204-195-120-218.wavecable.com. [204.195.120.218]) by smtp.gmail.com with ESMTPSA id kx11-20020a17090b228b00b001faa4a6691asm231384pjb.30.2022.08.16.19.08.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Aug 2022 19:08:40 -0700 (PDT) Date: Tue, 16 Aug 2022 19:08:37 -0700 From: Stephen Hemminger To: Dmitry Kozlyuk Cc: dev@dpdk.org Subject: Re: [RFC] Dynamic log/trace control via telemetry Message-ID: <20220816190837.40c557bb@hermes.local> In-Reply-To: <20220816021738.5498f802@sovereign> References: <20220816021738.5498f802@sovereign> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Tue, 16 Aug 2022 02:17:38 +0300 Dmitry Kozlyuk wrote: > When debugging a live app, useful info can be obtained from logs or traces > that were not enabled when it was started and it is undesirable to restart. > Furthermore, unless the app authors have considered tracing, > rte_trace_save() is only called on exit, i.e. a shutdown is required again. > > What if the telemetry socket gave the missing control? > For example: > > --> /eal/log/set_level,debug > {"/eal/log/set_level": {"status": "success"}} > --> /eal/log/set_level,pmd.net.*:debug > {"/eal/log/set_level": {"status": "success"}} > --> /eal/log/set_level,lib.[a-d],debug > {"/eal/log/set_level": {"status": "success"}} > --> /eal/log/set_level,foobar > {"/eal/log/set_level": {"status": "fail", "message": "Invalid log level: foobar"}} > > Tracing is more complicated because it requires resource allocation. > It could be: > > /eal/trace/set_mode,discard > /eal/trace/set_mode,overwrite > rte_trace_mode_set() > > /eal/trace/enable_pattern, > /eal/trace/enable_regex, > /eal/trace/disable_pattern, > /eal/trace/disable_regex, > rte_trace_enable_pattern() > rte_trace_enable_regex() > > /eal/trace/rearm > Clear the trace buffer. > Apply the new settings accumulated by the previous commands. > > /eal/trace/save > rte_trace_save() > > An open question is how to deal with multi-process. > Only the primary process listens to the telemetry socket. > Log and trace settings are not shared between processes. > OTOH, when enabling some log source, it is desirable to do in all processes. > > In general, telemetry is not for changing the state of the app, > but logs and traces are diagnostic information that seems to be in-scope. > A similar suggestion was not opposed recently: > > http://inbox.dpdk.org/dev/24c49429394294cfbf0d9c506b205029bac77c8b.1657890378.git.anatoly.burakov@intel.com/ Not sure if turning telemetry into a do all control api makes sense. This seems like a different API. Also, the default would have to be disabled for application safety reasons.