DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Varghese, Vipin" <vipin.varghese@intel.com>
To: "Hunt, David" <david.hunt@intel.com>,
	Hideyuki Yamashita <yamashita.hideyuki@ntt-tx.co.jp>
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH 0/5] add apistats function
Date: Sat, 5 Dec 2020 13:23:50 +0000	[thread overview]
Message-ID: <MWHPR11MB15814F74D177EE49FB5BD16D90F00@MWHPR11MB1581.namprd11.prod.outlook.com> (raw)
In-Reply-To: <2a9a170b-c351-e3f2-36b5-6db9d4c29da5@intel.com>

Sharing an alternate approach, if RX-TX callbacks are enabled in DPDK (which is by default). One can register a callback handler to update counters with the following information as `port-queue pair, lcoreid, total rx burst request, total empty rx burst, 1-8 pks, 9-16 pkts, 16-32 pkts`. Callback handlers can be selectively enabled or disabled too.

Can you please help me understand how `rte_apistats` would be different or pros of having it as library (that needs to be added to linking and running in case of using DPDK applications)?

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of David Hunt
> Sent: Friday, December 4, 2020 3:51 PM
> To: Hideyuki Yamashita <yamashita.hideyuki@ntt-tx.co.jp>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH 0/5] add apistats function
> 
> 
> On 4/12/2020 7:51 AM, Hideyuki Yamashita wrote:
> > In general, DPDK application consumes CPU usage because it polls
> > incoming packets using rx_burst API in infinite loop.
> > This makes difficult to estimate how much CPU usage is really used to
> > send/receive packets by the DPDK application.
> >
> > For example, even if no incoming packets arriving, CPU usage looks
> > nearly 100% when observed by top command.
> >
> > It is beneficial if developers can observe real CPU usage of the DPDK
> > application.
> > Such information can be exported to monitoring application like
> > prometheus/graphana and shows CPU usage graphically.
> >
> > To achieve above, this patch set provides apistats functionality.
> > apistats provides the followiing two counters for each lcore.
> > - rx_burst_counts[RTE_MAX_LCORE]
> > - tx_burst_counts[RTE_MAX_LCORE]
> > Those accumulates rx_burst/tx_burst counts since the application starts.
> >
> > By using those values, developers can roughly estimate CPU usage.
> > Let us assume a DPDK application is simply forwarding packets.
> > It calls tx_burst only if it receive packets.
> > If rx_burst_counts=1000 and tx_burst_count=1000 during certain period
> > of time, one can assume CPU usage is 100%.
> > If rx_burst_counts=1000 and tx_burst_count=100 during certain period
> > of time, one can assume CPU usage is 10%.
> > Here we assumes that tx_burst_count equals counts which rx_burst
> > function really receives incoming packets.
> >
> >
> > This patch set provides the following.
> > - basic API counting functionality(apistats) into librte_ethdev
> > - add code to testpmd to accumulate counter information
> > - add code to proc-info to retrieve above mentioned counter
> > information
> > - add description in proc-info document about --apistats parameter
> > - modify MAINTAINERS file for apistats.c and apistats.h
> >
> > Hideyuki Yamashita (5):
> >    maintainers: update maintainers file for apistats
> >    app/proc-info: add to use apistats
> >    app/test-pmd: add to use apistats
> >    docs: add description of apistats parameter into proc-info
> >    librte_ethdev: add to use apistats
> >
> >   MAINTAINERS                      |  3 ++
> >   app/proc-info/main.c             | 46 +++++++++++++++++++++++
> >   app/test-pmd/testpmd.c           |  4 ++
> >   doc/guides/tools/proc_info.rst   | 10 ++++-
> >   lib/librte_ethdev/meson.build    |  6 ++-
> >   lib/librte_ethdev/rte_apistats.c | 64 ++++++++++++++++++++++++++++++++
> >   lib/librte_ethdev/rte_apistats.h | 64 ++++++++++++++++++++++++++++++++
> >   lib/librte_ethdev/rte_ethdev.h   |  7 ++++
> >   lib/librte_ethdev/version.map    |  5 +++
> >   9 files changed, 205 insertions(+), 4 deletions(-)
> >   create mode 100644 lib/librte_ethdev/rte_apistats.c
> >   create mode 100644 lib/librte_ethdev/rte_apistats.h
> 
> 
> Hi Hideyuki,
> 
> I have a few questions on the patch set.
> 
> How does this compare to the mechanism added to l3fwd-power which counts
> the number of empty, partial and full polls, and uses them to calculate
> busyness? We saw pretty good tracking of busyness using those metrics. I
> would be concerned that just looking at the numebr of rx_bursts and tx_bursts
> may be limited to only a few use-cases. The l3fwd-power example uses
> branchless increments to capture empty, non-empty, full, and non-full polls.
> 
> Why not use the existing telemetry library to store the stats? It would be good
> if whatever metrics were counted were made available in a standard way, so
> that external entities such as collectd could pick them up, rather than having to
> parse the new struct. The l3fwd-power example registers the necessary new
> metrics, and exposes them through the telemetry library.
> 
> And a comment on the patch set in general: The order of the patch set seems
> reversed. The earlier patch do not compile, because they depend on
> rte_apistats.h, which is introduced in the final patch. Each patch as it is applied
> needs to build successfully.
> 
> Also, I would suggest a different name. rte_apistats seems very generic and
> could apply to anything. How about something like rte_ethdev_stats.h or
> rte_burst_stats.h?
> 
> Rgds,
> Dave.
> 
> 
> 
> 
> 
> 


  reply	other threads:[~2020-12-05 13:24 UTC|newest]

Thread overview: 29+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-12-04  7:51 Hideyuki Yamashita
2020-12-04  7:51 ` [dpdk-dev] [PATCH 1/5] maintainers: update maintainers file for apistats Hideyuki Yamashita
2020-12-05 13:27   ` Varghese, Vipin
2020-12-24  6:36     ` Hideyuki Yamashita
2020-12-04  7:51 ` [dpdk-dev] [PATCH 2/5] app/proc-info: add to use apistats Hideyuki Yamashita
2020-12-05 13:15   ` Varghese, Vipin
2020-12-04  7:51 ` [dpdk-dev] [PATCH 3/5] app/test-pmd: " Hideyuki Yamashita
2020-12-05 13:04   ` Varghese, Vipin
2020-12-04  7:51 ` [dpdk-dev] [PATCH 4/5] docs: add description of apistats parameter into proc-info Hideyuki Yamashita
2020-12-05 13:02   ` Varghese, Vipin
2020-12-07  9:48     ` Pattan, Reshma
2020-12-04  7:51 ` [dpdk-dev] [PATCH 5/5] librte_ethdev: add to use apistats Hideyuki Yamashita
2020-12-05 13:01   ` Varghese, Vipin
2020-12-06 17:47   ` Stephen Hemminger
2020-12-24  6:33     ` Hideyuki Yamashita
2020-12-07 12:38   ` Ananyev, Konstantin
2020-12-22  2:48     ` Hideyuki Yamashita
2020-12-22  2:50     ` Hideyuki Yamashita
2020-12-22  9:04       ` Morten Brørup
2020-12-22 13:05       ` Ananyev, Konstantin
2020-12-04  9:09 ` [dpdk-dev] [PATCH 0/5] add apistats function Ferruh Yigit
2020-12-04 10:20 ` David Hunt
2020-12-05 13:23   ` Varghese, Vipin [this message]
2020-12-24  6:43     ` Hideyuki Yamashita
2020-12-24 12:35       ` Varghese, Vipin
2020-12-22  2:16   ` Hideyuki Yamashita
2020-12-07  9:46 ` Thomas Monjalon
2020-12-22  2:22   ` Hideyuki Yamashita
2021-02-22 15:10     ` Ferruh Yigit

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=MWHPR11MB15814F74D177EE49FB5BD16D90F00@MWHPR11MB1581.namprd11.prod.outlook.com \
    --to=vipin.varghese@intel.com \
    --cc=david.hunt@intel.com \
    --cc=dev@dpdk.org \
    --cc=yamashita.hideyuki@ntt-tx.co.jp \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).