From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 30766A0548; Mon, 26 Apr 2021 19:49:06 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 27D58411F9; Mon, 26 Apr 2021 19:47:29 +0200 (CEST) Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by mails.dpdk.org (Postfix) with ESMTP id 91426411F9 for ; Mon, 26 Apr 2021 19:47:26 +0200 (CEST) Received: by mail-pl1-f180.google.com with SMTP id y1so13694396plg.11 for ; Mon, 26 Apr 2021 10:47:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RfxfX+b5ciIB9S854YGzyrTVT5ZF3GEqtwIBpynpVhQ=; b=AFuryPSHf297j/eA/bR8+RcaWzQb0ycNss3q0VanEIdwiNJ1nFC07/V+ogrGfRvJLx hl4zSUguqSHsmuP8aAOrC7/hV19wj9D9KV1A5ENRvnyQ90p7ANFpaXxtxuaL/CgJdesD hN5Xm4mTRU5YyQaCDjQ82SJnJCZ/XRdEyFLV0SmBBcFNvIR5SD4QcvpjT1/F1Z8uQyWY 1TyvnGs1FqlBlGZEJEjvjwcbrRguoZUAs+th4Dc7C0GDdDLOcZ23lzRJArvv0L6adai5 uEouLFjmiqkFXG28AuaZy2H6iIrijOmMHNGhtnVT5VcDmhg6RNFwmlF4QYsCCiOnwBMZ sKug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RfxfX+b5ciIB9S854YGzyrTVT5ZF3GEqtwIBpynpVhQ=; b=o8eCum72zZEosaQMLj2nfJMEHaGCBNvH817ahZ1ds0WItxK95u6w2GayH9PXj+H9MW gKGceM5NRcgdXCNMAhEX5RnTDDweKY2KdFQnoRgJNlXnLgJY3WQvP/Beh5w53xrG3+ew PKUQez5hVxrhaP6YJ9q5fIrPQ2Kz8IfUHYQIfeOsHm3mXFfjHhYmQE26e1SIjC+X61aP NJHfM5jxHcv8FvOax+RRfAuM8uk6em+b29BKPNNwARG+k9VOI6PR8ArEGInKcvKBZv2H FbifsmwXfta0W1WS7/QNWUOyZpPRm5OYoam7l01RVFNqF8tiBK9FGH3Ymgt73NtJ9Y9u gC+g== X-Gm-Message-State: AOAM533D43Mr7Sibxw1g4/QdpuQJ7QcKJEFeImwIUh7yjF7t5F5ZIQ/7 U0gPt5uLjYJx/+tll67iv0jZAQ== X-Google-Smtp-Source: ABdhPJwb9C6Eyc+6E4ltSu5Q6JxUyuDliyWaoFawXoInoMiam8Jo5LW4kpegwDLnrfeK1l7YzH/FTA== X-Received: by 2002:a17:902:b18f:b029:ec:7ac0:fd1a with SMTP id s15-20020a170902b18fb02900ec7ac0fd1amr19869513plr.84.1619459245707; Mon, 26 Apr 2021 10:47:25 -0700 (PDT) Received: from hermes.local (76-14-218-44.or.wavecable.com. [76.14.218.44]) by smtp.gmail.com with ESMTPSA id k15sm336971pfi.0.2021.04.26.10.47.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Apr 2021 10:47:25 -0700 (PDT) Date: Mon, 26 Apr 2021 10:47:20 -0700 From: Stephen Hemminger To: Jerin Jacob Cc: Honnappa Nagarahalli , Kathleen Capella , "thomas@monjalon.net" , "dev@dpdk.org" , Dharmik Thakkar , Ruifeng Wang , "david.marchand@redhat.com" , Bruce Richardson , "jerinj@marvell.com" , "hemant.agrawal@nxp.com" , Ferruh Yigit , "Ananyev, Konstantin" , Stephen Hemminger , nd Message-ID: <20210426104720.2b892045@hermes.local> In-Reply-To: References: <2273212.ItJIoklBD0@thomas> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] L3fwd mode in testpmd X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Mon, 26 Apr 2021 15:14:59 +0530 Jerin Jacob wrote: > On Sat, Apr 24, 2021 at 5:56 AM Honnappa Nagarahalli > wrote: > > > > > > > > > > > > > > > > On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > Performance of L3fwd example application is one > > > > > > > > > > > of the key > > > > > > > > > > benchmarks in DPDK. However, the application does not have > > > > > > > > > > many debugging statistics to understand the performance > > > > > > > > > > issues. We have added L3fwd as another mode/stream to > > > > > > > > > > testpmd which provides > > > > > > > > enough > > > > > > > > > > statistics at various levels. This has allowed us to debug > > > > > > > > > > the performance issues effectively. > > > > > > > > > > > > > > > > > > > > > > There is more work to be done to get it to upstreamable > > > > > > > > > > > state. I am > > > > > > > > > > wondering if such a patch is helpful for others and if the > > > > > > > > > > community would be interested in taking a look. Please let > > > > > > > > > > me know > > > > > > > what you think. > > > > > > > > > > > > > > > > > > > > We are using app/proc-info/ to attach and analyze the > > > performance. > > > > > > > > > > That helps to analyze the unmodified application. I think, > > > > > > > > > > if something is missing in proc-info app, in my opinion it > > > > > > > > > > is better to enhance proc-info so that it can help other > > > > > > > > > > third-party > > > > > applications. > > > > > > > > > > > > > > > > > > > > Just my 2c. > > > > > > > > > Thanks Jerin. We will explore that. > > > > > > > > > > > > > > > > I agree it is dangerous to rely too much on testpmd for everything. > > > > > > > > Please tell us what in testpmd could be useful out of it. > > > > > > > > > > > > > > > Things that are very helpful in testpmd are: 1) HW statistics > > > > > > > from the NIC 2) Forwarding stats 3) Burst stats (indication of > > > > > > > headroom > > > > > > > availability) 4) Easy to set parameters like RX and TX queue > > > > > > > depths (among others) without having to recompile. > > > > > > > > > > > > [Kathleen Capella] > > > > > > Thank you for the suggestion of app/proc-info. I've tried it out > > > > > > with l3fwd and see that it does have the HW stats from the NIC and > > > > > > the forwarding > > > > > stats. > > > > > > However, it does not have the burst stats testpmd offers, nor the > > > > > > > > > > One option to see such level of debugging would be to have > > > > > - Create a memzone in the primary process > > > > > - Application under test can update the stats in memzone based on > > > > > the code flow > > > > > - proc-info can read the counters updated by application under test > > > > > using the memzone object got through rte_memzone_lookup() > > > > Agreed. Currently, using app/proc-info does not provide this ability. We > > > cannot add this capability to app/proc-info as these stats would be specific to > > > L3fwd application. > > > > > > I meant creating generic counter-read/write infra via memzone to not make it > > > as l3fwd specific. > > Currently, app/proc-info is able to print the stats as they are standardized via the API. But for statistics that are generated in the application, they are very specific to that application. For ex: burst stats in testpmd are very specific to it and another application might implement the same in a very different manner. > > > > In needs to be something like the app/proc-info just needs to be a dumb displaying utility and the application has to do all the heavy lifting of copying the exact display strings to the memory. > > Yes. > > > > > > > > > > > > > Another approach will be using rte_trace()[1] for debugging/tracing > > > > > by adding tracepoints in l3fwd for such events. > > > > > It has a timestamp and the trace format is opensource trace > > > > > format(CTF(Common trace format)), so that we can use post posting > > > > > tools to analyze. > > > > > [1] > > > > > https://doc.dpdk.org/guides/prog_guide/trace_lib.html > > > > This is good for analyzing an incident. I think it is an overhead for > > > development purposes. > > > > > > Consider if one wants to add burst stats, one can add stats increment under > > > RTE_TRACE_POINT_FP, it will be emitted whenever code flow through that > > > path. Set of events of can be viewed in trace viewer[1]. Would that be > > > enough? > > > Adding traces to l3fwd can be upstreamed as it is useful for others for > > > debugging. > > > > > > [1] > > > https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG > > This needs post processing of the trace info to derive the information, is it correct? For ex: for burst stats, there will be several traces generated collecting the number of packets returned by rte_eth_rx_burst which needs to be post processed. > > Or You can have an additional variable to acculate it. > > > Also, adding traces is equivalent to adding statistics in L3fwd. > > Yes. > > If the sole purpose only stats then it is better to add status in > l3fwd without performance impact. I thought some thing else. > > > > > > > > > > > > > > ability to easily change parameters without having to recompile, > > > > > > which helps reduce debugging time significantly. > > We will not be able to fix this above issue. > > It depends on what you want to debug. Trace can be disabled at runtime. DPDK has existing API's for application metrics but they are rarely used. Why not implement rte_metrics in l3fwd and proc-info?