Re: [dpdk-dev] L3fwd mode in testpmd

["Ananyev, Konstantin" <konstantin.ananyev@intel.com>]

 
> > > >>>>>>>>>>>> On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli
> > > >>>>>>>>>>>> <Honnappa.Nagarahalli@arm.com> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hello,
> > > >>>>>>>>>>>>>         Performance of L3fwd example application is one
> > > >>>>>>>>>>>>> of the key
> > > >>>>>>>>>>>> benchmarks in DPDK. However, the application does not
> > > >>>>>>>>>>>> have many debugging statistics to understand the
> > > >>>>>>>>>>>> performance issues. We have added L3fwd as another
> > > >>>>>>>>>>>> mode/stream to testpmd which provides
> > > >>>>>>>>>> enough
> > > >>>>>>>>>>>> statistics at various levels. This has allowed us to
> > > >>>>>>>>>>>> debug the performance issues effectively.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> There is more work to be done to get it to upstreamable
> > > >>>>>>>>>>>>> state. I am
> > > >>>>>>>>>>>> wondering if such a patch is helpful for others and if
> > > >>>>>>>>>>>> the community would be interested in taking a look.
> > > >>>>>>>>>>>> Please let me know
> > > >>>>>>>>> what you think.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> We are using app/proc-info/ to attach and analyze the
> > > >>>>> performance.
> > > >>>>>>>>>>>> That helps to analyze the unmodified application. I
> > > >>>>>>>>>>>> think, if something is missing in proc-info app, in my
> > > >>>>>>>>>>>> opinion it is better to enhance proc-info so that it can
> > > >>>>>>>>>>>> help other third-party
> > > >>>>>>> applications.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Just my 2c.
> > > >>>>>>>>>>> Thanks Jerin. We will explore that.
> > > >>>>>>>>>>
> > > >>>>>>>>>> I agree it is dangerous to rely too much on testpmd for
> > > >> everything.
> > > >>>>>>>>>> Please tell us what in testpmd could be useful out of it.
> > > >>>>>>>>>>
> > > >>>>>>>>> Things that are very helpful in testpmd are: 1) HW
> > > >>>>>>>>> statistics from the NIC 2) Forwarding stats 3) Burst stats
> > > >>>>>>>>> (indication of headroom
> > > >>>>>>>>> availability) 4) Easy to set parameters like RX and TX queue
> > > >>>>>>>>> depths (among others) without having to recompile.
> > > >>>>>>>>
> > > >>>>>>>> [Kathleen Capella]
> > > >>>>>>>> Thank you for the suggestion of app/proc-info. I've tried it
> > > >>>>>>>> out with l3fwd and see that it does have the HW stats from
> > > >>>>>>>> the NIC and the forwarding
> > > >>>>>>> stats.
> > > >>>>>>>> However, it does not have the burst stats testpmd offers, nor
> > > >>>>>>>> the
> > > >>>>>>>
> > > >>>>>>> One option to see such  level of debugging would be to have
> > > >>>>>>> - Create a memzone in the primary process
> > > >>>>>>> - Application under test can update the stats in memzone based
> > > >>>>>>> on the code flow
> > > >>>>>>> - proc-info can read the counters updated by application under
> > > >>>>>>> test using the memzone object got through
> > > >> rte_memzone_lookup()
> > > >>>>>> Agreed. Currently, using app/proc-info does not provide this
> > > >>>>>> ability. We
> > > >>>>> cannot add this capability to app/proc-info as these stats would
> > > >>>>> be specific to L3fwd application.
> > > >>>>>
> > > >>>>> I meant creating generic counter-read/write infra via memzone to
> > > >>>>> not make it as l3fwd specific.
> > > >>>> Currently, app/proc-info is able to print the stats as they are
> > > >>>> standardized
> > > >> via the API. But for statistics that are generated in the
> > > >> application, they are very specific to that application. For ex:
> > > >> burst stats in testpmd are very specific to it and another
> > > >> application might implement the same in a very different manner.
> > > >>>>
> > > >>>> In needs to be something like the app/proc-info just needs to be
> > > >>>> a dumb
> > > >> displaying utility and the application has to do all the heavy
> > > >> lifting of copying the exact display strings to the memory.
> > > >>>
> > > >>> Yes.
> > > >>>
> > > >>>>
> > > >>>>>>>
> > > >>>>>>> Another approach will be using rte_trace()[1] for
> > > >>>>>>> debugging/tracing by adding tracepoints in l3fwd for such events.
> > > >>>>>>> It has a timestamp and the trace format is opensource trace
> > > >>>>>>> format(CTF(Common trace format)), so that we can use post
> > > >>>>>>> posting tools to analyze.
> > > >>>>>>> [1]
> > > >>>>>>> https://doc.dpdk.org/guides/prog_guide/trace_lib.html
> > > >>>>>> This is good for analyzing an incident. I think it is an
> > > >>>>>> overhead for
> > > >>>>> development purposes.
> > > >>>>>
> > > >>>>> Consider if one wants to add burst stats, one can add stats
> > > >>>>> increment under RTE_TRACE_POINT_FP, it will be emitted
> > whenever
> > > >>>>> code flow through that path. Set of events of can be viewed in
> > > >>>>> trace viewer[1]. Would that be enough?
> > > >>>>> Adding traces to l3fwd can be upstreamed as it is useful for
> > > >>>>> others for debugging.
> > > >>>>>
> > > >>>>> [1]
> > > >>>>> https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG
> > > >>>> This needs post processing of the trace info to derive the
> > > >>>> information, is it
> > > >> correct? For ex: for burst stats, there will be several traces
> > > >> generated collecting the number of packets returned by
> > > >> rte_eth_rx_burst which needs to be post processed.
> > > >>>
> > > >>> Or You can have an additional variable to acculate it.
> > > >>>
> > > >>>> Also, adding traces is equivalent to adding statistics in L3fwd.
> > > >>>
> > > >>> Yes.
> > > >>>
> > > >>> If the sole purpose only stats then it is better to add status in
> > > >>> l3fwd without performance impact. I thought some thing else.
> > > >>>
> > > >>>>
> > > >>>>>>>
> > > >>>>>>>> ability to easily change parameters without having to
> > > >>>>>>>> recompile, which helps reduce debugging time significantly.
> > > >>>> We will not be able to fix this above issue.
> > > >>>
> > > >>> It depends on what you want to debug. Trace can be disabled at
> > runtime.
> > > >>
> > > >>
> > > >> DPDK has existing API's for application metrics but they are rarely used.
> > > >>
> > > >> Why not implement rte_metrics in l3fwd and proc-info?
> > > > This discussion has ended up as a stats discussion. But, we also need to
> > be able to change the configurable parameters easily.
> > > > If we implement the stats and ability to change the configurable
> > > > parameters, then it is essentially bringing in some of the
> > > > capabilities from
> > > testpmd to the sample application. I think that will result in lot more code in
> > the sample application and will make it complicated.
> > > >
> > > > Instead our proposal is to take L3fwd to testpmd and use all the
> > > > infra code that testpmd provides. We see that this approach results
> > > > in less
> > > amount of code added to DPDK overall.
> > > >
> > >
> > > Agree that it may help testing to have l3fwd support on the testpmd.
> > >
> > > Two concerns,
> > > 1) Testpmd already too complex.
> > > 2) Code duplication.
> > >
> > > For 1), if the l3fwd can be implemented in testpmd as new, independent
> > > forwarding mode, without touching rest of the testpmd, I think it can be
> > OK.
> Yes, this is what we have done. It is a new forwarding mode.
> We could remove some forwarding modes from testpmd. For ex: macfwd, macswap seem very similar to iofwd mode.

Not really, iowfd doesn't touch packet data at all, while macfwd and macwap change L2 headers.
In fact I found all of them quite helpful (just for different cases), so please keep them.

> 
> >
> > In fact, l3fwd is also quite big and complex:
> > $ wc -l examples/l3fwd/*.[h,c] |grep total
> >   6969 total
> >
> > Plus it will introduce extra dependencies (fib, lpm, hash, might-be acl?) I am
> > not sure it is a good idea to pull all these complexities into test-pmd.
> I do not suggest pulling all these in. In our case, I see that the ask is only on LPM. I am open to hearing what others see as the requirement.

Ok, but l3fwd forwarding model is quite different from current PMD one
(egress queue selection, TX packets buffering, etc.).
I suppose you'll need to pull all that too from l3fwd?

> 
> > I can't imagine that l3fwd app need ability to configure each and every PMD
> > parameter.
> > From my experience in l3fwd most of cycles are spent not in PMD itself, but
> > in actual packet processing: header parsing and checking, classification,
> > routing table lookup, etc.
> During our work, we had to experiment with burst size, rx/tx queue depths along with other PMD specific configuration parameters. The
> packet processing code remains the same and there is not much to optimize.

I think burst-size and rx/tx queue size can be added into l3fwd as new config parameters.
Doesn't look like a major issue to me.
PMD specific parameters could be a problem... anything particular you plan to use?
 
> >
> > > Not sure how to address 2), also lets say we want to add new feature
> > > to l3fwd, where it should go, to the sample or to the testpmd?
> L3fwd example will remain as the example. We have to duplicate the code into testpmd. If L3fwd example is changed, it needs to be
> changed in testpmd as well.

Usually code duplication is not a good sign.
I understand that sometimes it is unavoidable, but why we have to do it here?