Re: [dpdk-dev] [PATCH 0/2] new headroom stats library and example application

[Neil Horman <nhorman@tuxdriver.com>]

On Thu, Jan 29, 2015 at 05:10:36PM +0000, Wodkowski, PawelX wrote:
> > -----Original Message-----
> > From: Neil Horman [mailto:nhorman@tuxdriver.com]
> > Sent: Thursday, January 29, 2015 2:25 PM
> > To: Wodkowski, PawelX
> > Cc: dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH 0/2] new headroom stats library and example
> > application
> > 
> > On Thu, Jan 29, 2015 at 12:50:04PM +0100, Pawel Wodkowski wrote:
> > > Hi community,
> > > I would like to introduce library for measuring load of some arbitrary jobs. It
> > > can be used to profile every kind of job sets on any arbitrary execution unit.
> > > In provided l2fwd-headroom example I demonstrate how to use this library to
> > > profile packet forwarding (job set is froward, flush and stats) on LCores
> > > (execution unit). This example does no limit possible schemes on which this
> > > library can be used.
> > >
> > > Pawel Wodkowski (2):
> > >   librte_headroom: New library for checking core/system/app load
> > >   examples: introduce new l2fwd-headroom example
> > >
> > >  config/common_bsdapp               |    6 +
> > >  config/common_linuxapp             |    6 +
> > >  examples/Makefile                  |    1 +
> > >  examples/l2fwd-headroom/Makefile   |   51 +++
> > >  examples/l2fwd-headroom/main.c     |  875
> > ++++++++++++++++++++++++++++++++++++
> > >  lib/Makefile                       |    1 +
> > >  lib/librte_headroom/Makefile       |   50 +++
> > >  lib/librte_headroom/rte_headroom.c |  368 +++++++++++++++
> > >  lib/librte_headroom/rte_headroom.h |  481 ++++++++++++++++++++
> > >  mk/rte.app.mk                      |    4 +
> > >  10 files changed, 1843 insertions(+)
> > >  create mode 100644 examples/l2fwd-headroom/Makefile
> > >  create mode 100644 examples/l2fwd-headroom/main.c
> > >  create mode 100644 lib/librte_headroom/Makefile
> > >  create mode 100644 lib/librte_headroom/rte_headroom.c
> > >  create mode 100644 lib/librte_headroom/rte_headroom.h
> > >
> > > --
> > > 1.7.9.5
> > >
> > >
> > 
> > Whats the advantage of this library over the other tools to preform the same
> > function.
> 
> Hi Neil,
> 
> Good point, what is advantage over perf. Answer is: this library does not 
> supposed to be a perf competition and is not for profiling app in the way perf does.
> It is an small and fast extension. It's main task is to manage job list to invoke
> them exactly when needed and provide some basic stats about application idle 
> time (whatever programmer will consider the idle) and busy time.
> 
> For example:
> application might decide to add remove some jobs to/from LCore(s) dynamically
> basing on current idle time (ex: move job from one core to another).
> 
> Also application might have some information's about traffic type it handles
> and provide own algorithm to calculate invocation time (it can also dynamically
> switch between those algorithms only replacing handlers).
> 
> >  Perf can provide all the information in this library, and do so
> > without having to directly modify the source for the execution unit under test
> 
> Yes, perf can provide those information's  but it can't handle the case when 
> you are poling for packets too fast or too slow and waist time getting only couple 
> of them. Library will adjust time when it execute job basing on value this job
> returned previously. Code modifications are not so deep, as you can see comparing 
> l2wf vs l2fwd-headroom app.
> 
> For example in  application I introduced, when forward job return less than
> MAX_PKT_BURST execution period will be increased. If it return more it will decrease
> execution period. Stats provided for that can be used to determine if application is
> behaving correctly and if there is a time for handling another port (what did for tests).
> 
You're still re-inventing the wheel here, and I don't see any advantage to doing
so.  If the goal of the library is to profile the run time of a task, then you
have perf and systemtap for such purposes.  If the goal is to create a job
scheduler that allows you to track multiple parallel tasks, and adjust their
execution, there are several pre-existing libraries that any application
programmer can already leverage to do just that (Berkely UPC or libtask to name
just two examples).  Truthfully, on a dedicated cpu, you could just as easily
create multiple child processes runnnig at SCHED_RR and set their priorities
accordingly.

I don't see why we need another library to do what several other tools/libraries
can do quite well.

Neil

> Pawel
>