[dpdk-dev] L3fwd mode in testpmd

[dpdk-dev] L3fwd mode in testpmd

[Honnappa Nagarahalli]

Hello,
	Performance of L3fwd example application is one of the key benchmarks in DPDK. However, the application does not have many debugging statistics to understand the performance issues. We have added L3fwd as another mode/stream to testpmd which provides enough statistics at various levels. This has allowed us to debug the performance issues effectively.

There is more work to be done to get it to upstreamable state. I am wondering if such a patch is helpful for others and if the community would be interested in taking a look. Please let me know what you think.

Thanks,
Honnappa

Re: [dpdk-dev] L3fwd mode in testpmd

[Jerin Jacob]

On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli
<Honnappa.Nagarahalli@arm.com> wrote:
>
> Hello,
>         Performance of L3fwd example application is one of the key benchmarks in DPDK. However, the application does not have many debugging statistics to understand the performance issues. We have added L3fwd as another mode/stream to testpmd which provides enough statistics at various levels. This has allowed us to debug the performance issues effectively.
>
> There is more work to be done to get it to upstreamable state. I am wondering if such a patch is helpful for others and if the community would be interested in taking a look. Please let me know what you think.

We are using app/proc-info/ to attach and analyze the performance.
That helps to analyze the unmodified application. I think, if
something is missing in proc-info app, in my opinion it is better to
enhance proc-info so that it can help other third-party applications.

Just my 2c.


>
> Thanks,
> Honnappa

Re: [dpdk-dev] L3fwd mode in testpmd

[Honnappa Nagarahalli]

<snip>

> 
> On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com> wrote:
> >
> > Hello,
> >         Performance of L3fwd example application is one of the key
> benchmarks in DPDK. However, the application does not have many
> debugging statistics to understand the performance issues. We have added
> L3fwd as another mode/stream to testpmd which provides enough statistics
> at various levels. This has allowed us to debug the performance issues
> effectively.
> >
> > There is more work to be done to get it to upstreamable state. I am
> wondering if such a patch is helpful for others and if the community would
> be interested in taking a look. Please let me know what you think.
> 
> We are using app/proc-info/ to attach and analyze the performance.
> That helps to analyze the unmodified application. I think, if something is
> missing in proc-info app, in my opinion it is better to enhance proc-info so
> that it can help other third-party applications.
> 
> Just my 2c.
Thanks Jerin. We will explore that.

> 
> 
> >
> > Thanks,
> > Honnappa

Re: [dpdk-dev] L3fwd mode in testpmd

[Thomas Monjalon]

11/03/2021 16:18, Honnappa Nagarahalli:
> <snip>
> 
> > 
> > On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli
> > <Honnappa.Nagarahalli@arm.com> wrote:
> > >
> > > Hello,
> > >         Performance of L3fwd example application is one of the key
> > benchmarks in DPDK. However, the application does not have many
> > debugging statistics to understand the performance issues. We have added
> > L3fwd as another mode/stream to testpmd which provides enough statistics
> > at various levels. This has allowed us to debug the performance issues
> > effectively.
> > >
> > > There is more work to be done to get it to upstreamable state. I am
> > wondering if such a patch is helpful for others and if the community would
> > be interested in taking a look. Please let me know what you think.
> > 
> > We are using app/proc-info/ to attach and analyze the performance.
> > That helps to analyze the unmodified application. I think, if something is
> > missing in proc-info app, in my opinion it is better to enhance proc-info so
> > that it can help other third-party applications.
> > 
> > Just my 2c.
> Thanks Jerin. We will explore that.

I agree it is dangerous to rely too much on testpmd for everything.
Please tell us what in testpmd could be useful out of it.

Re: [dpdk-dev] L3fwd mode in testpmd

[Honnappa Nagarahalli]

<snip>

> >
> > >
> > > On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli
> > > <Honnappa.Nagarahalli@arm.com> wrote:
> > > >
> > > > Hello,
> > > >         Performance of L3fwd example application is one of the key
> > > benchmarks in DPDK. However, the application does not have many
> > > debugging statistics to understand the performance issues. We have
> > > added L3fwd as another mode/stream to testpmd which provides
> enough
> > > statistics at various levels. This has allowed us to debug the
> > > performance issues effectively.
> > > >
> > > > There is more work to be done to get it to upstreamable state. I
> > > > am
> > > wondering if such a patch is helpful for others and if the community
> > > would be interested in taking a look. Please let me know what you think.
> > >
> > > We are using app/proc-info/ to attach and analyze the performance.
> > > That helps to analyze the unmodified application. I think, if
> > > something is missing in proc-info app, in my opinion it is better to
> > > enhance proc-info so that it can help other third-party applications.
> > >
> > > Just my 2c.
> > Thanks Jerin. We will explore that.
> 
> I agree it is dangerous to rely too much on testpmd for everything.
> Please tell us what in testpmd could be useful out of it.
> 
Things that are very helpful in testpmd are: 1) HW statistics from the NIC 2) Forwarding stats 3) Burst stats (indication of headroom availability) 4) Easy to set parameters like RX and TX queue depths (among others) without having to recompile.

Re: [dpdk-dev] L3fwd mode in testpmd

[Kathleen Capella]

> -----Original Message-----
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Thursday, March 11, 2021 11:00 AM
> To: thomas@monjalon.net
> Cc: Jerin Jacob <jerinjacobk@gmail.com>; dev@dpdk.org; Kathleen Capella
> <Kathleen.Capella@arm.com>; Dharmik Thakkar
> <Dharmik.Thakkar@arm.com>; Ruifeng Wang <Ruifeng.Wang@arm.com>;
> david.marchand@redhat.com; Bruce Richardson
> <bruce.richardson@intel.com>; jerinj@marvell.com;
> hemant.agrawal@nxp.com; Ferruh Yigit <ferruh.yigit@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; Stephen Hemminger
> <sthemmin@microsoft.com>; nd <nd@arm.com>; nd <nd@arm.com>
> Subject: RE: [dpdk-dev] L3fwd mode in testpmd
> 
> <snip>
> 
> > >
> > > >
> > > > On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli
> > > > <Honnappa.Nagarahalli@arm.com> wrote:
> > > > >
> > > > > Hello,
> > > > >         Performance of L3fwd example application is one of the
> > > > > key
> > > > benchmarks in DPDK. However, the application does not have many
> > > > debugging statistics to understand the performance issues. We have
> > > > added L3fwd as another mode/stream to testpmd which provides
> > enough
> > > > statistics at various levels. This has allowed us to debug the
> > > > performance issues effectively.
> > > > >
> > > > > There is more work to be done to get it to upstreamable state. I
> > > > > am
> > > > wondering if such a patch is helpful for others and if the
> > > > community would be interested in taking a look. Please let me know
> what you think.
> > > >
> > > > We are using app/proc-info/ to attach and analyze the performance.
> > > > That helps to analyze the unmodified application. I think, if
> > > > something is missing in proc-info app, in my opinion it is better
> > > > to enhance proc-info so that it can help other third-party applications.
> > > >
> > > > Just my 2c.
> > > Thanks Jerin. We will explore that.
> >
> > I agree it is dangerous to rely too much on testpmd for everything.
> > Please tell us what in testpmd could be useful out of it.
> >
> Things that are very helpful in testpmd are: 1) HW statistics from the NIC 2)
> Forwarding stats 3) Burst stats (indication of headroom availability) 4) Easy to
> set parameters like RX and TX queue depths (among others) without having
> to recompile.

[Kathleen Capella] 
Thank you for the suggestion of app/proc-info. I've tried it out with 
l3fwd and see that it does have the HW stats from the NIC and the forwarding stats.
However, it does not have the burst stats testpmd offers, nor the 
ability to easily change parameters without having to recompile, 
which helps reduce debugging time significantly.

Re: [dpdk-dev] L3fwd mode in testpmd

[Jerin Jacob]

On Thu, Apr 1, 2021 at 2:05 AM Kathleen Capella
<Kathleen.Capella@arm.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> > Sent: Thursday, March 11, 2021 11:00 AM
> > To: thomas@monjalon.net
> > Cc: Jerin Jacob <jerinjacobk@gmail.com>; dev@dpdk.org; Kathleen Capella
> > <Kathleen.Capella@arm.com>; Dharmik Thakkar
> > <Dharmik.Thakkar@arm.com>; Ruifeng Wang <Ruifeng.Wang@arm.com>;
> > david.marchand@redhat.com; Bruce Richardson
> > <bruce.richardson@intel.com>; jerinj@marvell.com;
> > hemant.agrawal@nxp.com; Ferruh Yigit <ferruh.yigit@intel.com>; Ananyev,
> > Konstantin <konstantin.ananyev@intel.com>; Stephen Hemminger
> > <sthemmin@microsoft.com>; nd <nd@arm.com>; nd <nd@arm.com>
> > Subject: RE: [dpdk-dev] L3fwd mode in testpmd
> >
> > <snip>
> >
> > > >
> > > > >
> > > > > On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli
> > > > > <Honnappa.Nagarahalli@arm.com> wrote:
> > > > > >
> > > > > > Hello,
> > > > > >         Performance of L3fwd example application is one of the
> > > > > > key
> > > > > benchmarks in DPDK. However, the application does not have many
> > > > > debugging statistics to understand the performance issues. We have
> > > > > added L3fwd as another mode/stream to testpmd which provides
> > > enough
> > > > > statistics at various levels. This has allowed us to debug the
> > > > > performance issues effectively.
> > > > > >
> > > > > > There is more work to be done to get it to upstreamable state. I
> > > > > > am
> > > > > wondering if such a patch is helpful for others and if the
> > > > > community would be interested in taking a look. Please let me know
> > what you think.
> > > > >
> > > > > We are using app/proc-info/ to attach and analyze the performance.
> > > > > That helps to analyze the unmodified application. I think, if
> > > > > something is missing in proc-info app, in my opinion it is better
> > > > > to enhance proc-info so that it can help other third-party applications.
> > > > >
> > > > > Just my 2c.
> > > > Thanks Jerin. We will explore that.
> > >
> > > I agree it is dangerous to rely too much on testpmd for everything.
> > > Please tell us what in testpmd could be useful out of it.
> > >
> > Things that are very helpful in testpmd are: 1) HW statistics from the NIC 2)
> > Forwarding stats 3) Burst stats (indication of headroom availability) 4) Easy to
> > set parameters like RX and TX queue depths (among others) without having
> > to recompile.
>
> [Kathleen Capella]
> Thank you for the suggestion of app/proc-info. I've tried it out with
> l3fwd and see that it does have the HW stats from the NIC and the forwarding stats.
> However, it does not have the burst stats testpmd offers, nor the

One option to see such  level of debugging would be to have
- Create a memzone in the primary process
- Application under test can update the stats in memzone based on the code flow
- proc-info can read the counters updated by application under test
using the memzone object got through rte_memzone_lookup()

Another approach will be using rte_trace()[1] for debugging/tracing by
adding tracepoints in l3fwd for such events.
It has a timestamp and the trace format is opensource trace
format(CTF(Common trace format)), so that we can use
post posting tools to analyze.
[1]
https://doc.dpdk.org/guides/prog_guide/trace_lib.html


> ability to easily change parameters without having to recompile,
> which helps reduce debugging time significantly.

Re: [dpdk-dev] L3fwd mode in testpmd

[Honnappa Nagarahalli]

<snip>

> > > > > >
> > > > > > On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli
> > > > > > <Honnappa.Nagarahalli@arm.com> wrote:
> > > > > > >
> > > > > > > Hello,
> > > > > > >         Performance of L3fwd example application is one of
> > > > > > > the key
> > > > > > benchmarks in DPDK. However, the application does not have
> > > > > > many debugging statistics to understand the performance
> > > > > > issues. We have added L3fwd as another mode/stream to testpmd
> > > > > > which provides
> > > > enough
> > > > > > statistics at various levels. This has allowed us to debug the
> > > > > > performance issues effectively.
> > > > > > >
> > > > > > > There is more work to be done to get it to upstreamable
> > > > > > > state. I am
> > > > > > wondering if such a patch is helpful for others and if the
> > > > > > community would be interested in taking a look. Please let me
> > > > > > know
> > > what you think.
> > > > > >
> > > > > > We are using app/proc-info/ to attach and analyze the performance.
> > > > > > That helps to analyze the unmodified application. I think, if
> > > > > > something is missing in proc-info app, in my opinion it is
> > > > > > better to enhance proc-info so that it can help other third-party
> applications.
> > > > > >
> > > > > > Just my 2c.
> > > > > Thanks Jerin. We will explore that.
> > > >
> > > > I agree it is dangerous to rely too much on testpmd for everything.
> > > > Please tell us what in testpmd could be useful out of it.
> > > >
> > > Things that are very helpful in testpmd are: 1) HW statistics from
> > > the NIC 2) Forwarding stats 3) Burst stats (indication of headroom
> > > availability) 4) Easy to set parameters like RX and TX queue depths
> > > (among others) without having to recompile.
> >
> > [Kathleen Capella]
> > Thank you for the suggestion of app/proc-info. I've tried it out with
> > l3fwd and see that it does have the HW stats from the NIC and the forwarding
> stats.
> > However, it does not have the burst stats testpmd offers, nor the
> 
> One option to see such  level of debugging would be to have
> - Create a memzone in the primary process
> - Application under test can update the stats in memzone based on the code
> flow
> - proc-info can read the counters updated by application under test using the
> memzone object got through rte_memzone_lookup()
Agreed. Currently, using app/proc-info does not provide this ability. We cannot add this capability to app/proc-info as these stats would be specific to L3fwd application.

> 
> Another approach will be using rte_trace()[1] for debugging/tracing by adding
> tracepoints in l3fwd for such events.
> It has a timestamp and the trace format is opensource trace
> format(CTF(Common trace format)), so that we can use post posting tools to
> analyze.
> [1]
> https://doc.dpdk.org/guides/prog_guide/trace_lib.html
This is good for analyzing an incident. I think it is an overhead for development purposes.

> 
> 
> > ability to easily change parameters without having to recompile, which
> > helps reduce debugging time significantly.

Re: [dpdk-dev] L3fwd mode in testpmd

[Jerin Jacob]

On Thu, Apr 1, 2021 at 5:51 AM Honnappa Nagarahalli
<Honnappa.Nagarahalli@arm.com> wrote:
>
> <snip>
>
> > > > > > >
> > > > > > > On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli
> > > > > > > <Honnappa.Nagarahalli@arm.com> wrote:
> > > > > > > >
> > > > > > > > Hello,
> > > > > > > >         Performance of L3fwd example application is one of
> > > > > > > > the key
> > > > > > > benchmarks in DPDK. However, the application does not have
> > > > > > > many debugging statistics to understand the performance
> > > > > > > issues. We have added L3fwd as another mode/stream to testpmd
> > > > > > > which provides
> > > > > enough
> > > > > > > statistics at various levels. This has allowed us to debug the
> > > > > > > performance issues effectively.
> > > > > > > >
> > > > > > > > There is more work to be done to get it to upstreamable
> > > > > > > > state. I am
> > > > > > > wondering if such a patch is helpful for others and if the
> > > > > > > community would be interested in taking a look. Please let me
> > > > > > > know
> > > > what you think.
> > > > > > >
> > > > > > > We are using app/proc-info/ to attach and analyze the performance.
> > > > > > > That helps to analyze the unmodified application. I think, if
> > > > > > > something is missing in proc-info app, in my opinion it is
> > > > > > > better to enhance proc-info so that it can help other third-party
> > applications.
> > > > > > >
> > > > > > > Just my 2c.
> > > > > > Thanks Jerin. We will explore that.
> > > > >
> > > > > I agree it is dangerous to rely too much on testpmd for everything.
> > > > > Please tell us what in testpmd could be useful out of it.
> > > > >
> > > > Things that are very helpful in testpmd are: 1) HW statistics from
> > > > the NIC 2) Forwarding stats 3) Burst stats (indication of headroom
> > > > availability) 4) Easy to set parameters like RX and TX queue depths
> > > > (among others) without having to recompile.
> > >
> > > [Kathleen Capella]
> > > Thank you for the suggestion of app/proc-info. I've tried it out with
> > > l3fwd and see that it does have the HW stats from the NIC and the forwarding
> > stats.
> > > However, it does not have the burst stats testpmd offers, nor the
> >
> > One option to see such  level of debugging would be to have
> > - Create a memzone in the primary process
> > - Application under test can update the stats in memzone based on the code
> > flow
> > - proc-info can read the counters updated by application under test using the
> > memzone object got through rte_memzone_lookup()
> Agreed. Currently, using app/proc-info does not provide this ability. We cannot add this capability to app/proc-info as these stats would be specific to L3fwd application.

I meant creating generic counter-read/write infra via memzone to not
make it as l3fwd specific.

>
> >
> > Another approach will be using rte_trace()[1] for debugging/tracing by adding
> > tracepoints in l3fwd for such events.
> > It has a timestamp and the trace format is opensource trace
> > format(CTF(Common trace format)), so that we can use post posting tools to
> > analyze.
> > [1]
> > https://doc.dpdk.org/guides/prog_guide/trace_lib.html
> This is good for analyzing an incident. I think it is an overhead for development purposes.

Consider if one wants to add burst stats, one can add stats increment
under RTE_TRACE_POINT_FP, it will be emitted
whenever code flow through that path. Set of events of can be viewed
in trace viewer[1]. Would that be enough?
Adding traces to l3fwd can be upstreamed as it is useful for others
for debugging.

[1]
https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG

>
> >
> >
> > > ability to easily change parameters without having to recompile, which
> > > helps reduce debugging time significantly.

Re: [dpdk-dev] L3fwd mode in testpmd

[Honnappa Nagarahalli]

<snip>

> > > > > > > > On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli
> > > > > > > > <Honnappa.Nagarahalli@arm.com> wrote:
> > > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > >         Performance of L3fwd example application is one
> > > > > > > > > of the key
> > > > > > > > benchmarks in DPDK. However, the application does not have
> > > > > > > > many debugging statistics to understand the performance
> > > > > > > > issues. We have added L3fwd as another mode/stream to
> > > > > > > > testpmd which provides
> > > > > > enough
> > > > > > > > statistics at various levels. This has allowed us to debug
> > > > > > > > the performance issues effectively.
> > > > > > > > >
> > > > > > > > > There is more work to be done to get it to upstreamable
> > > > > > > > > state. I am
> > > > > > > > wondering if such a patch is helpful for others and if the
> > > > > > > > community would be interested in taking a look. Please let
> > > > > > > > me know
> > > > > what you think.
> > > > > > > >
> > > > > > > > We are using app/proc-info/ to attach and analyze the
> performance.
> > > > > > > > That helps to analyze the unmodified application. I think,
> > > > > > > > if something is missing in proc-info app, in my opinion it
> > > > > > > > is better to enhance proc-info so that it can help other
> > > > > > > > third-party
> > > applications.
> > > > > > > >
> > > > > > > > Just my 2c.
> > > > > > > Thanks Jerin. We will explore that.
> > > > > >
> > > > > > I agree it is dangerous to rely too much on testpmd for everything.
> > > > > > Please tell us what in testpmd could be useful out of it.
> > > > > >
> > > > > Things that are very helpful in testpmd are: 1) HW statistics
> > > > > from the NIC 2) Forwarding stats 3) Burst stats (indication of
> > > > > headroom
> > > > > availability) 4) Easy to set parameters like RX and TX queue
> > > > > depths (among others) without having to recompile.
> > > >
> > > > [Kathleen Capella]
> > > > Thank you for the suggestion of app/proc-info. I've tried it out
> > > > with l3fwd and see that it does have the HW stats from the NIC and
> > > > the forwarding
> > > stats.
> > > > However, it does not have the burst stats testpmd offers, nor the
> > >
> > > One option to see such  level of debugging would be to have
> > > - Create a memzone in the primary process
> > > - Application under test can update the stats in memzone based on
> > > the code flow
> > > - proc-info can read the counters updated by application under test
> > > using the memzone object got through rte_memzone_lookup()
> > Agreed. Currently, using app/proc-info does not provide this ability. We
> cannot add this capability to app/proc-info as these stats would be specific to
> L3fwd application.
> 
> I meant creating generic counter-read/write infra via memzone to not make it
> as l3fwd specific.
Currently, app/proc-info is able to print the stats as they are standardized via the API. But for statistics that are generated in the application, they are very specific to that application. For ex: burst stats in testpmd are very specific to it and another application might implement the same in a very different manner.

In needs to be something like the app/proc-info just needs to be a dumb displaying utility and the application has to do all the heavy lifting of copying the exact display strings to the memory.

> > >
> > > Another approach will be using rte_trace()[1] for debugging/tracing
> > > by adding tracepoints in l3fwd for such events.
> > > It has a timestamp and the trace format is opensource trace
> > > format(CTF(Common trace format)), so that we can use post posting
> > > tools to analyze.
> > > [1]
> > > https://doc.dpdk.org/guides/prog_guide/trace_lib.html
> > This is good for analyzing an incident. I think it is an overhead for
> development purposes.
> 
> Consider if one wants to add burst stats, one can add stats increment under
> RTE_TRACE_POINT_FP, it will be emitted whenever code flow through that
> path. Set of events of can be viewed in trace viewer[1]. Would that be
> enough?
> Adding traces to l3fwd can be upstreamed as it is useful for others for
> debugging.
> 
> [1]
> https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG
This needs post processing of the trace info to derive the information, is it correct? For ex: for burst stats, there will be several traces generated collecting the number of packets returned by rte_eth_rx_burst which needs to be post processed.
Also, adding traces is equivalent to adding statistics in L3fwd.

> > >
> > > > ability to easily change parameters without having to recompile,
> > > > which helps reduce debugging time significantly.
We will not be able to fix this above issue.

Re: [dpdk-dev] L3fwd mode in testpmd

[Jerin Jacob]

On Sat, Apr 24, 2021 at 5:56 AM Honnappa Nagarahalli
<Honnappa.Nagarahalli@arm.com> wrote:
>
> <snip>
>
> > > > > > > > > On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli
> > > > > > > > > <Honnappa.Nagarahalli@arm.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Hello,
> > > > > > > > > >         Performance of L3fwd example application is one
> > > > > > > > > > of the key
> > > > > > > > > benchmarks in DPDK. However, the application does not have
> > > > > > > > > many debugging statistics to understand the performance
> > > > > > > > > issues. We have added L3fwd as another mode/stream to
> > > > > > > > > testpmd which provides
> > > > > > > enough
> > > > > > > > > statistics at various levels. This has allowed us to debug
> > > > > > > > > the performance issues effectively.
> > > > > > > > > >
> > > > > > > > > > There is more work to be done to get it to upstreamable
> > > > > > > > > > state. I am
> > > > > > > > > wondering if such a patch is helpful for others and if the
> > > > > > > > > community would be interested in taking a look. Please let
> > > > > > > > > me know
> > > > > > what you think.
> > > > > > > > >
> > > > > > > > > We are using app/proc-info/ to attach and analyze the
> > performance.
> > > > > > > > > That helps to analyze the unmodified application. I think,
> > > > > > > > > if something is missing in proc-info app, in my opinion it
> > > > > > > > > is better to enhance proc-info so that it can help other
> > > > > > > > > third-party
> > > > applications.
> > > > > > > > >
> > > > > > > > > Just my 2c.
> > > > > > > > Thanks Jerin. We will explore that.
> > > > > > >
> > > > > > > I agree it is dangerous to rely too much on testpmd for everything.
> > > > > > > Please tell us what in testpmd could be useful out of it.
> > > > > > >
> > > > > > Things that are very helpful in testpmd are: 1) HW statistics
> > > > > > from the NIC 2) Forwarding stats 3) Burst stats (indication of
> > > > > > headroom
> > > > > > availability) 4) Easy to set parameters like RX and TX queue
> > > > > > depths (among others) without having to recompile.
> > > > >
> > > > > [Kathleen Capella]
> > > > > Thank you for the suggestion of app/proc-info. I've tried it out
> > > > > with l3fwd and see that it does have the HW stats from the NIC and
> > > > > the forwarding
> > > > stats.
> > > > > However, it does not have the burst stats testpmd offers, nor the
> > > >
> > > > One option to see such  level of debugging would be to have
> > > > - Create a memzone in the primary process
> > > > - Application under test can update the stats in memzone based on
> > > > the code flow
> > > > - proc-info can read the counters updated by application under test
> > > > using the memzone object got through rte_memzone_lookup()
> > > Agreed. Currently, using app/proc-info does not provide this ability. We
> > cannot add this capability to app/proc-info as these stats would be specific to
> > L3fwd application.
> >
> > I meant creating generic counter-read/write infra via memzone to not make it
> > as l3fwd specific.
> Currently, app/proc-info is able to print the stats as they are standardized via the API. But for statistics that are generated in the application, they are very specific to that application. For ex: burst stats in testpmd are very specific to it and another application might implement the same in a very different manner.
>
> In needs to be something like the app/proc-info just needs to be a dumb displaying utility and the application has to do all the heavy lifting of copying the exact display strings to the memory.

Yes.

>
> > > >
> > > > Another approach will be using rte_trace()[1] for debugging/tracing
> > > > by adding tracepoints in l3fwd for such events.
> > > > It has a timestamp and the trace format is opensource trace
> > > > format(CTF(Common trace format)), so that we can use post posting
> > > > tools to analyze.
> > > > [1]
> > > > https://doc.dpdk.org/guides/prog_guide/trace_lib.html
> > > This is good for analyzing an incident. I think it is an overhead for
> > development purposes.
> >
> > Consider if one wants to add burst stats, one can add stats increment under
> > RTE_TRACE_POINT_FP, it will be emitted whenever code flow through that
> > path. Set of events of can be viewed in trace viewer[1]. Would that be
> > enough?
> > Adding traces to l3fwd can be upstreamed as it is useful for others for
> > debugging.
> >
> > [1]
> > https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG
> This needs post processing of the trace info to derive the information, is it correct? For ex: for burst stats, there will be several traces generated collecting the number of packets returned by rte_eth_rx_burst which needs to be post processed.

Or You can have an additional variable to acculate it.

> Also, adding traces is equivalent to adding statistics in L3fwd.

Yes.

If the sole purpose only stats then it is better to add status in
l3fwd without performance impact. I thought some thing else.

>
> > > >
> > > > > ability to easily change parameters without having to recompile,
> > > > > which helps reduce debugging time significantly.
> We will not be able to fix this above issue.

It depends on what you want to debug. Trace can be disabled at runtime.

Re: [dpdk-dev] L3fwd mode in testpmd

[Stephen Hemminger]

On Mon, 26 Apr 2021 15:14:59 +0530
Jerin Jacob <jerinjacobk@gmail.com> wrote:

> On Sat, Apr 24, 2021 at 5:56 AM Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com> wrote:
> >
> > <snip>
> >  
> > > > > > > > > > On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli
> > > > > > > > > > <Honnappa.Nagarahalli@arm.com> wrote:  
> > > > > > > > > > >
> > > > > > > > > > > Hello,
> > > > > > > > > > >         Performance of L3fwd example application is one
> > > > > > > > > > > of the key  
> > > > > > > > > > benchmarks in DPDK. However, the application does not have
> > > > > > > > > > many debugging statistics to understand the performance
> > > > > > > > > > issues. We have added L3fwd as another mode/stream to
> > > > > > > > > > testpmd which provides  
> > > > > > > > enough  
> > > > > > > > > > statistics at various levels. This has allowed us to debug
> > > > > > > > > > the performance issues effectively.  
> > > > > > > > > > >
> > > > > > > > > > > There is more work to be done to get it to upstreamable
> > > > > > > > > > > state. I am  
> > > > > > > > > > wondering if such a patch is helpful for others and if the
> > > > > > > > > > community would be interested in taking a look. Please let
> > > > > > > > > > me know  
> > > > > > > what you think.  
> > > > > > > > > >
> > > > > > > > > > We are using app/proc-info/ to attach and analyze the  
> > > performance.  
> > > > > > > > > > That helps to analyze the unmodified application. I think,
> > > > > > > > > > if something is missing in proc-info app, in my opinion it
> > > > > > > > > > is better to enhance proc-info so that it can help other
> > > > > > > > > > third-party  
> > > > > applications.  
> > > > > > > > > >
> > > > > > > > > > Just my 2c.  
> > > > > > > > > Thanks Jerin. We will explore that.  
> > > > > > > >
> > > > > > > > I agree it is dangerous to rely too much on testpmd for everything.
> > > > > > > > Please tell us what in testpmd could be useful out of it.
> > > > > > > >  
> > > > > > > Things that are very helpful in testpmd are: 1) HW statistics
> > > > > > > from the NIC 2) Forwarding stats 3) Burst stats (indication of
> > > > > > > headroom
> > > > > > > availability) 4) Easy to set parameters like RX and TX queue
> > > > > > > depths (among others) without having to recompile.  
> > > > > >
> > > > > > [Kathleen Capella]
> > > > > > Thank you for the suggestion of app/proc-info. I've tried it out
> > > > > > with l3fwd and see that it does have the HW stats from the NIC and
> > > > > > the forwarding  
> > > > > stats.  
> > > > > > However, it does not have the burst stats testpmd offers, nor the  
> > > > >
> > > > > One option to see such  level of debugging would be to have
> > > > > - Create a memzone in the primary process
> > > > > - Application under test can update the stats in memzone based on
> > > > > the code flow
> > > > > - proc-info can read the counters updated by application under test
> > > > > using the memzone object got through rte_memzone_lookup()  
> > > > Agreed. Currently, using app/proc-info does not provide this ability. We  
> > > cannot add this capability to app/proc-info as these stats would be specific to
> > > L3fwd application.
> > >
> > > I meant creating generic counter-read/write infra via memzone to not make it
> > > as l3fwd specific.  
> > Currently, app/proc-info is able to print the stats as they are standardized via the API. But for statistics that are generated in the application, they are very specific to that application. For ex: burst stats in testpmd are very specific to it and another application might implement the same in a very different manner.
> >
> > In needs to be something like the app/proc-info just needs to be a dumb displaying utility and the application has to do all the heavy lifting of copying the exact display strings to the memory.  
> 
> Yes.
> 
> >  
> > > > >
> > > > > Another approach will be using rte_trace()[1] for debugging/tracing
> > > > > by adding tracepoints in l3fwd for such events.
> > > > > It has a timestamp and the trace format is opensource trace
> > > > > format(CTF(Common trace format)), so that we can use post posting
> > > > > tools to analyze.
> > > > > [1]
> > > > > https://doc.dpdk.org/guides/prog_guide/trace_lib.html  
> > > > This is good for analyzing an incident. I think it is an overhead for  
> > > development purposes.
> > >
> > > Consider if one wants to add burst stats, one can add stats increment under
> > > RTE_TRACE_POINT_FP, it will be emitted whenever code flow through that
> > > path. Set of events of can be viewed in trace viewer[1]. Would that be
> > > enough?
> > > Adding traces to l3fwd can be upstreamed as it is useful for others for
> > > debugging.
> > >
> > > [1]
> > > https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG  
> > This needs post processing of the trace info to derive the information, is it correct? For ex: for burst stats, there will be several traces generated collecting the number of packets returned by rte_eth_rx_burst which needs to be post processed.  
> 
> Or You can have an additional variable to acculate it.
> 
> > Also, adding traces is equivalent to adding statistics in L3fwd.  
> 
> Yes.
> 
> If the sole purpose only stats then it is better to add status in
> l3fwd without performance impact. I thought some thing else.
> 
> >  
> > > > >  
> > > > > > ability to easily change parameters without having to recompile,
> > > > > > which helps reduce debugging time significantly.  
> > We will not be able to fix this above issue.  
> 
> It depends on what you want to debug. Trace can be disabled at runtime.


DPDK has existing API's for application metrics but they are rarely used.

Why not implement rte_metrics in l3fwd and proc-info?

Re: [dpdk-dev] L3fwd mode in testpmd

[Honnappa Nagarahalli]

<snip>

> > >
> > > > > > > > > > > On Thu, Mar 11, 2021 at 12:01 AM Honnappa
> > > > > > > > > > > Nagarahalli <Honnappa.Nagarahalli@arm.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hello,
> > > > > > > > > > > >         Performance of L3fwd example application
> > > > > > > > > > > > is one of the key
> > > > > > > > > > > benchmarks in DPDK. However, the application does
> > > > > > > > > > > not have many debugging statistics to understand the
> > > > > > > > > > > performance issues. We have added L3fwd as another
> > > > > > > > > > > mode/stream to testpmd which provides
> > > > > > > > > enough
> > > > > > > > > > > statistics at various levels. This has allowed us to
> > > > > > > > > > > debug the performance issues effectively.
> > > > > > > > > > > >
> > > > > > > > > > > > There is more work to be done to get it to
> > > > > > > > > > > > upstreamable state. I am
> > > > > > > > > > > wondering if such a patch is helpful for others and
> > > > > > > > > > > if the community would be interested in taking a
> > > > > > > > > > > look. Please let me know
> > > > > > > > what you think.
> > > > > > > > > > >
> > > > > > > > > > > We are using app/proc-info/ to attach and analyze
> > > > > > > > > > > the
> > > > performance.
> > > > > > > > > > > That helps to analyze the unmodified application. I
> > > > > > > > > > > think, if something is missing in proc-info app, in
> > > > > > > > > > > my opinion it is better to enhance proc-info so that
> > > > > > > > > > > it can help other third-party
> > > > > > applications.
> > > > > > > > > > >
> > > > > > > > > > > Just my 2c.
> > > > > > > > > > Thanks Jerin. We will explore that.
> > > > > > > > >
> > > > > > > > > I agree it is dangerous to rely too much on testpmd for
> everything.
> > > > > > > > > Please tell us what in testpmd could be useful out of it.
> > > > > > > > >
> > > > > > > > Things that are very helpful in testpmd are: 1) HW
> > > > > > > > statistics from the NIC 2) Forwarding stats 3) Burst stats
> > > > > > > > (indication of headroom
> > > > > > > > availability) 4) Easy to set parameters like RX and TX
> > > > > > > > queue depths (among others) without having to recompile.
> > > > > > >
> > > > > > > [Kathleen Capella]
> > > > > > > Thank you for the suggestion of app/proc-info. I've tried it
> > > > > > > out with l3fwd and see that it does have the HW stats from
> > > > > > > the NIC and the forwarding
> > > > > > stats.
> > > > > > > However, it does not have the burst stats testpmd offers,
> > > > > > > nor the
> > > > > >
> > > > > > One option to see such  level of debugging would be to have
> > > > > > - Create a memzone in the primary process
> > > > > > - Application under test can update the stats in memzone based
> > > > > > on the code flow
> > > > > > - proc-info can read the counters updated by application under
> > > > > > test using the memzone object got through
> rte_memzone_lookup()
> > > > > Agreed. Currently, using app/proc-info does not provide this
> > > > > ability. We
> > > > cannot add this capability to app/proc-info as these stats would
> > > > be specific to L3fwd application.
> > > >
> > > > I meant creating generic counter-read/write infra via memzone to
> > > > not make it as l3fwd specific.
> > > Currently, app/proc-info is able to print the stats as they are standardized
> via the API. But for statistics that are generated in the application, they are
> very specific to that application. For ex: burst stats in testpmd are very
> specific to it and another application might implement the same in a very
> different manner.
> > >
> > > In needs to be something like the app/proc-info just needs to be a dumb
> displaying utility and the application has to do all the heavy lifting of copying
> the exact display strings to the memory.
> >
> > Yes.
> >
> > >
> > > > > >
> > > > > > Another approach will be using rte_trace()[1] for
> > > > > > debugging/tracing by adding tracepoints in l3fwd for such events.
> > > > > > It has a timestamp and the trace format is opensource trace
> > > > > > format(CTF(Common trace format)), so that we can use post
> > > > > > posting tools to analyze.
> > > > > > [1]
> > > > > > https://doc.dpdk.org/guides/prog_guide/trace_lib.html
> > > > > This is good for analyzing an incident. I think it is an
> > > > > overhead for
> > > > development purposes.
> > > >
> > > > Consider if one wants to add burst stats, one can add stats
> > > > increment under RTE_TRACE_POINT_FP, it will be emitted whenever
> > > > code flow through that path. Set of events of can be viewed in
> > > > trace viewer[1]. Would that be enough?
> > > > Adding traces to l3fwd can be upstreamed as it is useful for
> > > > others for debugging.
> > > >
> > > > [1]
> > > > https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG
> > > This needs post processing of the trace info to derive the information, is it
> correct? For ex: for burst stats, there will be several traces generated
> collecting the number of packets returned by rte_eth_rx_burst which needs
> to be post processed.
> >
> > Or You can have an additional variable to acculate it.
> >
> > > Also, adding traces is equivalent to adding statistics in L3fwd.
> >
> > Yes.
> >
> > If the sole purpose only stats then it is better to add status in
> > l3fwd without performance impact. I thought some thing else.
> >
> > >
> > > > > >
> > > > > > > ability to easily change parameters without having to
> > > > > > > recompile, which helps reduce debugging time significantly.
> > > We will not be able to fix this above issue.
> >
> > It depends on what you want to debug. Trace can be disabled at runtime.
> 
> 
> DPDK has existing API's for application metrics but they are rarely used.
> 
> Why not implement rte_metrics in l3fwd and proc-info?
This discussion has ended up as a stats discussion. But, we also need to be able to change the configurable parameters easily.
If we implement the stats and ability to change the configurable parameters, then it is essentially bringing in some of the capabilities from testpmd to the sample application. I think that will result in lot more code in the sample application and will make it complicated.

Instead our proposal is to take L3fwd to testpmd and use all the infra code that testpmd provides. We see that this approach results in less amount of code added to DPDK overall.

Re: [dpdk-dev] L3fwd mode in testpmd

[Andrew Rybchenko]

On 4/26/21 11:46 PM, Honnappa Nagarahalli wrote:
> <snip>
> 
>>>>
>>>>>>>>>>>> On Thu, Mar 11, 2021 at 12:01 AM Honnappa
>>>>>>>>>>>> Nagarahalli <Honnappa.Nagarahalli@arm.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>         Performance of L3fwd example application
>>>>>>>>>>>>> is one of the key
>>>>>>>>>>>> benchmarks in DPDK. However, the application does
>>>>>>>>>>>> not have many debugging statistics to understand the
>>>>>>>>>>>> performance issues. We have added L3fwd as another
>>>>>>>>>>>> mode/stream to testpmd which provides
>>>>>>>>>> enough
>>>>>>>>>>>> statistics at various levels. This has allowed us to
>>>>>>>>>>>> debug the performance issues effectively.
>>>>>>>>>>>>>
>>>>>>>>>>>>> There is more work to be done to get it to
>>>>>>>>>>>>> upstreamable state. I am
>>>>>>>>>>>> wondering if such a patch is helpful for others and
>>>>>>>>>>>> if the community would be interested in taking a
>>>>>>>>>>>> look. Please let me know
>>>>>>>>> what you think.
>>>>>>>>>>>>
>>>>>>>>>>>> We are using app/proc-info/ to attach and analyze
>>>>>>>>>>>> the
>>>>> performance.
>>>>>>>>>>>> That helps to analyze the unmodified application. I
>>>>>>>>>>>> think, if something is missing in proc-info app, in
>>>>>>>>>>>> my opinion it is better to enhance proc-info so that
>>>>>>>>>>>> it can help other third-party
>>>>>>> applications.
>>>>>>>>>>>>
>>>>>>>>>>>> Just my 2c.
>>>>>>>>>>> Thanks Jerin. We will explore that.
>>>>>>>>>>
>>>>>>>>>> I agree it is dangerous to rely too much on testpmd for
>> everything.
>>>>>>>>>> Please tell us what in testpmd could be useful out of it.
>>>>>>>>>>
>>>>>>>>> Things that are very helpful in testpmd are: 1) HW
>>>>>>>>> statistics from the NIC 2) Forwarding stats 3) Burst stats
>>>>>>>>> (indication of headroom
>>>>>>>>> availability) 4) Easy to set parameters like RX and TX
>>>>>>>>> queue depths (among others) without having to recompile.
>>>>>>>>
>>>>>>>> [Kathleen Capella]
>>>>>>>> Thank you for the suggestion of app/proc-info. I've tried it
>>>>>>>> out with l3fwd and see that it does have the HW stats from
>>>>>>>> the NIC and the forwarding
>>>>>>> stats.
>>>>>>>> However, it does not have the burst stats testpmd offers,
>>>>>>>> nor the
>>>>>>>
>>>>>>> One option to see such  level of debugging would be to have
>>>>>>> - Create a memzone in the primary process
>>>>>>> - Application under test can update the stats in memzone based
>>>>>>> on the code flow
>>>>>>> - proc-info can read the counters updated by application under
>>>>>>> test using the memzone object got through
>> rte_memzone_lookup()
>>>>>> Agreed. Currently, using app/proc-info does not provide this
>>>>>> ability. We
>>>>> cannot add this capability to app/proc-info as these stats would
>>>>> be specific to L3fwd application.
>>>>>
>>>>> I meant creating generic counter-read/write infra via memzone to
>>>>> not make it as l3fwd specific.
>>>> Currently, app/proc-info is able to print the stats as they are standardized
>> via the API. But for statistics that are generated in the application, they are
>> very specific to that application. For ex: burst stats in testpmd are very
>> specific to it and another application might implement the same in a very
>> different manner.
>>>>
>>>> In needs to be something like the app/proc-info just needs to be a dumb
>> displaying utility and the application has to do all the heavy lifting of copying
>> the exact display strings to the memory.
>>>
>>> Yes.
>>>
>>>>
>>>>>>>
>>>>>>> Another approach will be using rte_trace()[1] for
>>>>>>> debugging/tracing by adding tracepoints in l3fwd for such events.
>>>>>>> It has a timestamp and the trace format is opensource trace
>>>>>>> format(CTF(Common trace format)), so that we can use post
>>>>>>> posting tools to analyze.
>>>>>>> [1]
>>>>>>> https://doc.dpdk.org/guides/prog_guide/trace_lib.html
>>>>>> This is good for analyzing an incident. I think it is an
>>>>>> overhead for
>>>>> development purposes.
>>>>>
>>>>> Consider if one wants to add burst stats, one can add stats
>>>>> increment under RTE_TRACE_POINT_FP, it will be emitted whenever
>>>>> code flow through that path. Set of events of can be viewed in
>>>>> trace viewer[1]. Would that be enough?
>>>>> Adding traces to l3fwd can be upstreamed as it is useful for
>>>>> others for debugging.
>>>>>
>>>>> [1]
>>>>> https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG
>>>> This needs post processing of the trace info to derive the information, is it
>> correct? For ex: for burst stats, there will be several traces generated
>> collecting the number of packets returned by rte_eth_rx_burst which needs
>> to be post processed.
>>>
>>> Or You can have an additional variable to acculate it.
>>>
>>>> Also, adding traces is equivalent to adding statistics in L3fwd.
>>>
>>> Yes.
>>>
>>> If the sole purpose only stats then it is better to add status in
>>> l3fwd without performance impact. I thought some thing else.
>>>
>>>>
>>>>>>>
>>>>>>>> ability to easily change parameters without having to
>>>>>>>> recompile, which helps reduce debugging time significantly.
>>>> We will not be able to fix this above issue.
>>>
>>> It depends on what you want to debug. Trace can be disabled at runtime.
>>
>>
>> DPDK has existing API's for application metrics but they are rarely used.
>>
>> Why not implement rte_metrics in l3fwd and proc-info?
> This discussion has ended up as a stats discussion. But, we also need to be able to change the configurable parameters easily.
> If we implement the stats and ability to change the configurable parameters, then it is essentially bringing in some of the capabilities from testpmd to the sample application. I think that will result in lot more code in the sample application and will make it complicated.
> 
> Instead our proposal is to take L3fwd to testpmd and use all the infra code that testpmd provides. We see that this approach results in less amount of code added to DPDK overall.
> 

+1

Re: [dpdk-dev] L3fwd mode in testpmd

[Ferruh Yigit]

On 4/26/2021 9:46 PM, Honnappa Nagarahalli wrote:
> <snip>
> 
>>>>
>>>>>>>>>>>> On Thu, Mar 11, 2021 at 12:01 AM Honnappa
>>>>>>>>>>>> Nagarahalli <Honnappa.Nagarahalli@arm.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>         Performance of L3fwd example application
>>>>>>>>>>>>> is one of the key
>>>>>>>>>>>> benchmarks in DPDK. However, the application does
>>>>>>>>>>>> not have many debugging statistics to understand the
>>>>>>>>>>>> performance issues. We have added L3fwd as another
>>>>>>>>>>>> mode/stream to testpmd which provides
>>>>>>>>>> enough
>>>>>>>>>>>> statistics at various levels. This has allowed us to
>>>>>>>>>>>> debug the performance issues effectively.
>>>>>>>>>>>>>
>>>>>>>>>>>>> There is more work to be done to get it to
>>>>>>>>>>>>> upstreamable state. I am
>>>>>>>>>>>> wondering if such a patch is helpful for others and
>>>>>>>>>>>> if the community would be interested in taking a
>>>>>>>>>>>> look. Please let me know
>>>>>>>>> what you think.
>>>>>>>>>>>>
>>>>>>>>>>>> We are using app/proc-info/ to attach and analyze
>>>>>>>>>>>> the
>>>>> performance.
>>>>>>>>>>>> That helps to analyze the unmodified application. I
>>>>>>>>>>>> think, if something is missing in proc-info app, in
>>>>>>>>>>>> my opinion it is better to enhance proc-info so that
>>>>>>>>>>>> it can help other third-party
>>>>>>> applications.
>>>>>>>>>>>>
>>>>>>>>>>>> Just my 2c.
>>>>>>>>>>> Thanks Jerin. We will explore that.
>>>>>>>>>>
>>>>>>>>>> I agree it is dangerous to rely too much on testpmd for
>> everything.
>>>>>>>>>> Please tell us what in testpmd could be useful out of it.
>>>>>>>>>>
>>>>>>>>> Things that are very helpful in testpmd are: 1) HW
>>>>>>>>> statistics from the NIC 2) Forwarding stats 3) Burst stats
>>>>>>>>> (indication of headroom
>>>>>>>>> availability) 4) Easy to set parameters like RX and TX
>>>>>>>>> queue depths (among others) without having to recompile.
>>>>>>>>
>>>>>>>> [Kathleen Capella]
>>>>>>>> Thank you for the suggestion of app/proc-info. I've tried it
>>>>>>>> out with l3fwd and see that it does have the HW stats from
>>>>>>>> the NIC and the forwarding
>>>>>>> stats.
>>>>>>>> However, it does not have the burst stats testpmd offers,
>>>>>>>> nor the
>>>>>>>
>>>>>>> One option to see such  level of debugging would be to have
>>>>>>> - Create a memzone in the primary process
>>>>>>> - Application under test can update the stats in memzone based
>>>>>>> on the code flow
>>>>>>> - proc-info can read the counters updated by application under
>>>>>>> test using the memzone object got through
>> rte_memzone_lookup()
>>>>>> Agreed. Currently, using app/proc-info does not provide this
>>>>>> ability. We
>>>>> cannot add this capability to app/proc-info as these stats would
>>>>> be specific to L3fwd application.
>>>>>
>>>>> I meant creating generic counter-read/write infra via memzone to
>>>>> not make it as l3fwd specific.
>>>> Currently, app/proc-info is able to print the stats as they are standardized
>> via the API. But for statistics that are generated in the application, they are
>> very specific to that application. For ex: burst stats in testpmd are very
>> specific to it and another application might implement the same in a very
>> different manner.
>>>>
>>>> In needs to be something like the app/proc-info just needs to be a dumb
>> displaying utility and the application has to do all the heavy lifting of copying
>> the exact display strings to the memory.
>>>
>>> Yes.
>>>
>>>>
>>>>>>>
>>>>>>> Another approach will be using rte_trace()[1] for
>>>>>>> debugging/tracing by adding tracepoints in l3fwd for such events.
>>>>>>> It has a timestamp and the trace format is opensource trace
>>>>>>> format(CTF(Common trace format)), so that we can use post
>>>>>>> posting tools to analyze.
>>>>>>> [1]
>>>>>>> https://doc.dpdk.org/guides/prog_guide/trace_lib.html
>>>>>> This is good for analyzing an incident. I think it is an
>>>>>> overhead for
>>>>> development purposes.
>>>>>
>>>>> Consider if one wants to add burst stats, one can add stats
>>>>> increment under RTE_TRACE_POINT_FP, it will be emitted whenever
>>>>> code flow through that path. Set of events of can be viewed in
>>>>> trace viewer[1]. Would that be enough?
>>>>> Adding traces to l3fwd can be upstreamed as it is useful for
>>>>> others for debugging.
>>>>>
>>>>> [1]
>>>>> https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG
>>>> This needs post processing of the trace info to derive the information, is it
>> correct? For ex: for burst stats, there will be several traces generated
>> collecting the number of packets returned by rte_eth_rx_burst which needs
>> to be post processed.
>>>
>>> Or You can have an additional variable to acculate it.
>>>
>>>> Also, adding traces is equivalent to adding statistics in L3fwd.
>>>
>>> Yes.
>>>
>>> If the sole purpose only stats then it is better to add status in
>>> l3fwd without performance impact. I thought some thing else.
>>>
>>>>
>>>>>>>
>>>>>>>> ability to easily change parameters without having to
>>>>>>>> recompile, which helps reduce debugging time significantly.
>>>> We will not be able to fix this above issue.
>>>
>>> It depends on what you want to debug. Trace can be disabled at runtime.
>>
>>
>> DPDK has existing API's for application metrics but they are rarely used.
>>
>> Why not implement rte_metrics in l3fwd and proc-info?
> This discussion has ended up as a stats discussion. But, we also need to be able to change the configurable parameters easily.
> If we implement the stats and ability to change the configurable parameters, then it is essentially bringing in some of the capabilities from testpmd to the sample application. I think that will result in lot more code in the sample application and will make it complicated.
> 
> Instead our proposal is to take L3fwd to testpmd and use all the infra code that testpmd provides. We see that this approach results in less amount of code added to DPDK overall.
> 

Agree that it may help testing to have l3fwd support on the testpmd.

Two concerns,
1) Testpmd already too complex.
2) Code duplication.

For 1), if the l3fwd can be implemented in testpmd as new, independent
forwarding mode, without touching rest of the testpmd, I think it can be OK.

Not sure how to address 2), also lets say we want to add new feature to l3fwd,
where it should go, to the sample or to the testpmd?

Re: [dpdk-dev] L3fwd mode in testpmd

[Ananyev, Konstantin]

> -----Original Message-----
> From: Yigit, Ferruh <ferruh.yigit@intel.com>
> Sent: Tuesday, April 27, 2021 10:50 AM
> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Stephen Hemminger <stephen@networkplumber.org>; Jerin Jacob
> <jerinjacobk@gmail.com>
> Cc: Kathleen Capella <Kathleen.Capella@arm.com>; thomas@monjalon.net; dev@dpdk.org; Dharmik Thakkar
> <Dharmik.Thakkar@arm.com>; Ruifeng Wang <Ruifeng.Wang@arm.com>; david.marchand@redhat.com; Richardson, Bruce
> <bruce.richardson@intel.com>; jerinj@marvell.com; hemant.agrawal@nxp.com; Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> Stephen Hemminger <sthemmin@microsoft.com>; nd <nd@arm.com>
> Subject: Re: [dpdk-dev] L3fwd mode in testpmd
> 
> On 4/26/2021 9:46 PM, Honnappa Nagarahalli wrote:
> > <snip>
> >
> >>>>
> >>>>>>>>>>>> On Thu, Mar 11, 2021 at 12:01 AM Honnappa
> >>>>>>>>>>>> Nagarahalli <Honnappa.Nagarahalli@arm.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hello,
> >>>>>>>>>>>>>         Performance of L3fwd example application
> >>>>>>>>>>>>> is one of the key
> >>>>>>>>>>>> benchmarks in DPDK. However, the application does
> >>>>>>>>>>>> not have many debugging statistics to understand the
> >>>>>>>>>>>> performance issues. We have added L3fwd as another
> >>>>>>>>>>>> mode/stream to testpmd which provides
> >>>>>>>>>> enough
> >>>>>>>>>>>> statistics at various levels. This has allowed us to
> >>>>>>>>>>>> debug the performance issues effectively.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> There is more work to be done to get it to
> >>>>>>>>>>>>> upstreamable state. I am
> >>>>>>>>>>>> wondering if such a patch is helpful for others and
> >>>>>>>>>>>> if the community would be interested in taking a
> >>>>>>>>>>>> look. Please let me know
> >>>>>>>>> what you think.
> >>>>>>>>>>>>
> >>>>>>>>>>>> We are using app/proc-info/ to attach and analyze
> >>>>>>>>>>>> the
> >>>>> performance.
> >>>>>>>>>>>> That helps to analyze the unmodified application. I
> >>>>>>>>>>>> think, if something is missing in proc-info app, in
> >>>>>>>>>>>> my opinion it is better to enhance proc-info so that
> >>>>>>>>>>>> it can help other third-party
> >>>>>>> applications.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Just my 2c.
> >>>>>>>>>>> Thanks Jerin. We will explore that.
> >>>>>>>>>>
> >>>>>>>>>> I agree it is dangerous to rely too much on testpmd for
> >> everything.
> >>>>>>>>>> Please tell us what in testpmd could be useful out of it.
> >>>>>>>>>>
> >>>>>>>>> Things that are very helpful in testpmd are: 1) HW
> >>>>>>>>> statistics from the NIC 2) Forwarding stats 3) Burst stats
> >>>>>>>>> (indication of headroom
> >>>>>>>>> availability) 4) Easy to set parameters like RX and TX
> >>>>>>>>> queue depths (among others) without having to recompile.
> >>>>>>>>
> >>>>>>>> [Kathleen Capella]
> >>>>>>>> Thank you for the suggestion of app/proc-info. I've tried it
> >>>>>>>> out with l3fwd and see that it does have the HW stats from
> >>>>>>>> the NIC and the forwarding
> >>>>>>> stats.
> >>>>>>>> However, it does not have the burst stats testpmd offers,
> >>>>>>>> nor the
> >>>>>>>
> >>>>>>> One option to see such  level of debugging would be to have
> >>>>>>> - Create a memzone in the primary process
> >>>>>>> - Application under test can update the stats in memzone based
> >>>>>>> on the code flow
> >>>>>>> - proc-info can read the counters updated by application under
> >>>>>>> test using the memzone object got through
> >> rte_memzone_lookup()
> >>>>>> Agreed. Currently, using app/proc-info does not provide this
> >>>>>> ability. We
> >>>>> cannot add this capability to app/proc-info as these stats would
> >>>>> be specific to L3fwd application.
> >>>>>
> >>>>> I meant creating generic counter-read/write infra via memzone to
> >>>>> not make it as l3fwd specific.
> >>>> Currently, app/proc-info is able to print the stats as they are standardized
> >> via the API. But for statistics that are generated in the application, they are
> >> very specific to that application. For ex: burst stats in testpmd are very
> >> specific to it and another application might implement the same in a very
> >> different manner.
> >>>>
> >>>> In needs to be something like the app/proc-info just needs to be a dumb
> >> displaying utility and the application has to do all the heavy lifting of copying
> >> the exact display strings to the memory.
> >>>
> >>> Yes.
> >>>
> >>>>
> >>>>>>>
> >>>>>>> Another approach will be using rte_trace()[1] for
> >>>>>>> debugging/tracing by adding tracepoints in l3fwd for such events.
> >>>>>>> It has a timestamp and the trace format is opensource trace
> >>>>>>> format(CTF(Common trace format)), so that we can use post
> >>>>>>> posting tools to analyze.
> >>>>>>> [1]
> >>>>>>> https://doc.dpdk.org/guides/prog_guide/trace_lib.html
> >>>>>> This is good for analyzing an incident. I think it is an
> >>>>>> overhead for
> >>>>> development purposes.
> >>>>>
> >>>>> Consider if one wants to add burst stats, one can add stats
> >>>>> increment under RTE_TRACE_POINT_FP, it will be emitted whenever
> >>>>> code flow through that path. Set of events of can be viewed in
> >>>>> trace viewer[1]. Would that be enough?
> >>>>> Adding traces to l3fwd can be upstreamed as it is useful for
> >>>>> others for debugging.
> >>>>>
> >>>>> [1]
> >>>>> https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG
> >>>> This needs post processing of the trace info to derive the information, is it
> >> correct? For ex: for burst stats, there will be several traces generated
> >> collecting the number of packets returned by rte_eth_rx_burst which needs
> >> to be post processed.
> >>>
> >>> Or You can have an additional variable to acculate it.
> >>>
> >>>> Also, adding traces is equivalent to adding statistics in L3fwd.
> >>>
> >>> Yes.
> >>>
> >>> If the sole purpose only stats then it is better to add status in
> >>> l3fwd without performance impact. I thought some thing else.
> >>>
> >>>>
> >>>>>>>
> >>>>>>>> ability to easily change parameters without having to
> >>>>>>>> recompile, which helps reduce debugging time significantly.
> >>>> We will not be able to fix this above issue.
> >>>
> >>> It depends on what you want to debug. Trace can be disabled at runtime.
> >>
> >>
> >> DPDK has existing API's for application metrics but they are rarely used.
> >>
> >> Why not implement rte_metrics in l3fwd and proc-info?
> > This discussion has ended up as a stats discussion. But, we also need to be able to change the configurable parameters easily.
> > If we implement the stats and ability to change the configurable parameters, then it is essentially bringing in some of the capabilities from
> testpmd to the sample application. I think that will result in lot more code in the sample application and will make it complicated.
> >
> > Instead our proposal is to take L3fwd to testpmd and use all the infra code that testpmd provides. We see that this approach results in less
> amount of code added to DPDK overall.
> >
> 
> Agree that it may help testing to have l3fwd support on the testpmd.
> 
> Two concerns,
> 1) Testpmd already too complex.
> 2) Code duplication.
> 
> For 1), if the l3fwd can be implemented in testpmd as new, independent
> forwarding mode, without touching rest of the testpmd, I think it can be OK.

In fact, l3fwd is also quite big and complex:
$ wc -l examples/l3fwd/*.[h,c] |grep total
  6969 total

Plus it will introduce extra dependencies (fib, lpm, hash, might-be acl?)
I am not sure it is a good idea to pull all these complexities into test-pmd.
I can't imagine that l3fwd app need ability to configure each and every
PMD parameter.
From my experience in l3fwd most of cycles are spent not in PMD itself,
but in actual packet processing: header parsing and checking, classification,
routing table lookup, etc. 

> Not sure how to address 2), also lets say we want to add new feature to l3fwd,
> where it should go, to the sample or to the testpmd?

Re: [dpdk-dev] L3fwd mode in testpmd

[Thomas Monjalon]

27/04/2021 11:57, Ananyev, Konstantin:
> From: Yigit, Ferruh <ferruh.yigit@intel.com>
> > On 4/26/2021 9:46 PM, Honnappa Nagarahalli wrote:
> > >>>>>>>>>>>> On Thu, Mar 11, 2021 at 12:01 AM Honnappa
> > >>>>>>>>>>>> Nagarahalli <Honnappa.Nagarahalli@arm.com> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hello,
> > >>>>>>>>>>>>>         Performance of L3fwd example application
> > >>>>>>>>>>>>> is one of the key
> > >>>>>>>>>>>> benchmarks in DPDK. However, the application does
> > >>>>>>>>>>>> not have many debugging statistics to understand the
> > >>>>>>>>>>>> performance issues. We have added L3fwd as another
> > >>>>>>>>>>>> mode/stream to testpmd which provides
> > >>>>>>>>>> enough
> > >>>>>>>>>>>> statistics at various levels. This has allowed us to
> > >>>>>>>>>>>> debug the performance issues effectively.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> There is more work to be done to get it to
> > >>>>>>>>>>>>> upstreamable state. I am
> > >>>>>>>>>>>> wondering if such a patch is helpful for others and
> > >>>>>>>>>>>> if the community would be interested in taking a
> > >>>>>>>>>>>> look. Please let me know
> > >>>>>>>>> what you think.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> We are using app/proc-info/ to attach and analyze
> > >>>>>>>>>>>> the
> > >>>>> performance.
> > >>>>>>>>>>>> That helps to analyze the unmodified application. I
> > >>>>>>>>>>>> think, if something is missing in proc-info app, in
> > >>>>>>>>>>>> my opinion it is better to enhance proc-info so that
> > >>>>>>>>>>>> it can help other third-party
> > >>>>>>> applications.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Just my 2c.
> > >>>>>>>>>>> Thanks Jerin. We will explore that.
> > >>>>>>>>>>
> > >>>>>>>>>> I agree it is dangerous to rely too much on testpmd for
> > >> everything.
> > >>>>>>>>>> Please tell us what in testpmd could be useful out of it.
> > >>>>>>>>>>
> > >>>>>>>>> Things that are very helpful in testpmd are: 1) HW
> > >>>>>>>>> statistics from the NIC 2) Forwarding stats 3) Burst stats
> > >>>>>>>>> (indication of headroom
> > >>>>>>>>> availability) 4) Easy to set parameters like RX and TX
> > >>>>>>>>> queue depths (among others) without having to recompile.
> > >>>>>>>>
> > >>>>>>>> [Kathleen Capella]
> > >>>>>>>> Thank you for the suggestion of app/proc-info. I've tried it
> > >>>>>>>> out with l3fwd and see that it does have the HW stats from
> > >>>>>>>> the NIC and the forwarding
> > >>>>>>> stats.
> > >>>>>>>> However, it does not have the burst stats testpmd offers,
> > >>>>>>>> nor the
> > >>>>>>>
> > >>>>>>> One option to see such  level of debugging would be to have
> > >>>>>>> - Create a memzone in the primary process
> > >>>>>>> - Application under test can update the stats in memzone based
> > >>>>>>> on the code flow
> > >>>>>>> - proc-info can read the counters updated by application under
> > >>>>>>> test using the memzone object got through
> > >> rte_memzone_lookup()
> > >>>>>> Agreed. Currently, using app/proc-info does not provide this
> > >>>>>> ability. We
> > >>>>> cannot add this capability to app/proc-info as these stats would
> > >>>>> be specific to L3fwd application.
> > >>>>>
> > >>>>> I meant creating generic counter-read/write infra via memzone to
> > >>>>> not make it as l3fwd specific.
> > >>>> Currently, app/proc-info is able to print the stats as they are standardized
> > >> via the API. But for statistics that are generated in the application, they are
> > >> very specific to that application. For ex: burst stats in testpmd are very
> > >> specific to it and another application might implement the same in a very
> > >> different manner.
> > >>>>
> > >>>> In needs to be something like the app/proc-info just needs to be a dumb
> > >> displaying utility and the application has to do all the heavy lifting of copying
> > >> the exact display strings to the memory.
> > >>>
> > >>> Yes.
> > >>>
> > >>>>
> > >>>>>>>
> > >>>>>>> Another approach will be using rte_trace()[1] for
> > >>>>>>> debugging/tracing by adding tracepoints in l3fwd for such events.
> > >>>>>>> It has a timestamp and the trace format is opensource trace
> > >>>>>>> format(CTF(Common trace format)), so that we can use post
> > >>>>>>> posting tools to analyze.
> > >>>>>>> [1]
> > >>>>>>> https://doc.dpdk.org/guides/prog_guide/trace_lib.html
> > >>>>>> This is good for analyzing an incident. I think it is an
> > >>>>>> overhead for
> > >>>>> development purposes.
> > >>>>>
> > >>>>> Consider if one wants to add burst stats, one can add stats
> > >>>>> increment under RTE_TRACE_POINT_FP, it will be emitted whenever
> > >>>>> code flow through that path. Set of events of can be viewed in
> > >>>>> trace viewer[1]. Would that be enough?
> > >>>>> Adding traces to l3fwd can be upstreamed as it is useful for
> > >>>>> others for debugging.
> > >>>>>
> > >>>>> [1]
> > >>>>> https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG
> > >>>> This needs post processing of the trace info to derive the information, is it
> > >> correct? For ex: for burst stats, there will be several traces generated
> > >> collecting the number of packets returned by rte_eth_rx_burst which needs
> > >> to be post processed.
> > >>>
> > >>> Or You can have an additional variable to acculate it.
> > >>>
> > >>>> Also, adding traces is equivalent to adding statistics in L3fwd.
> > >>>
> > >>> Yes.
> > >>>
> > >>> If the sole purpose only stats then it is better to add status in
> > >>> l3fwd without performance impact. I thought some thing else.
> > >>>
> > >>>>
> > >>>>>>>
> > >>>>>>>> ability to easily change parameters without having to
> > >>>>>>>> recompile, which helps reduce debugging time significantly.
> > >>>> We will not be able to fix this above issue.
> > >>>
> > >>> It depends on what you want to debug. Trace can be disabled at runtime.
> > >>
> > >>
> > >> DPDK has existing API's for application metrics but they are rarely used.
> > >>
> > >> Why not implement rte_metrics in l3fwd and proc-info?
> > > This discussion has ended up as a stats discussion. But, we also need to be able to change the configurable parameters easily.
> > > If we implement the stats and ability to change the configurable parameters, then it is essentially bringing in some of the capabilities from
> > testpmd to the sample application. I think that will result in lot more code in the sample application and will make it complicated.
> > >
> > > Instead our proposal is to take L3fwd to testpmd and use all the infra code that testpmd provides. We see that this approach results in less
> > amount of code added to DPDK overall.
> > >
> > 
> > Agree that it may help testing to have l3fwd support on the testpmd.
> > 
> > Two concerns,
> > 1) Testpmd already too complex.
> > 2) Code duplication.
> > 
> > For 1), if the l3fwd can be implemented in testpmd as new, independent
> > forwarding mode, without touching rest of the testpmd, I think it can be OK.
> 
> In fact, l3fwd is also quite big and complex:
> $ wc -l examples/l3fwd/*.[h,c] |grep total
>   6969 total
> 
> Plus it will introduce extra dependencies (fib, lpm, hash, might-be acl?)
> I am not sure it is a good idea to pull all these complexities into test-pmd.
> I can't imagine that l3fwd app need ability to configure each and every
> PMD parameter.
> From my experience in l3fwd most of cycles are spent not in PMD itself,
> but in actual packet processing: header parsing and checking, classification,
> routing table lookup, etc. 

testpmd goal is to test the driver, not the libraries.
 
> > Not sure how to address 2), also lets say we want to add new feature to l3fwd,
> > where it should go, to the sample or to the testpmd?

l3fwd is not targetted for testing.

Maybe we just lack a new test application for routing libraries?

Re: [dpdk-dev] L3fwd mode in testpmd

[Bruce Richardson]

On Tue, Apr 27, 2021 at 01:11:43PM +0200, Thomas Monjalon wrote:
> 27/04/2021 11:57, Ananyev, Konstantin:
> > From: Yigit, Ferruh <ferruh.yigit@intel.com>
> > > On 4/26/2021 9:46 PM, Honnappa Nagarahalli wrote:
> > > >>>>>>>>>>>> On Thu, Mar 11, 2021 at 12:01 AM Honnappa
> > > >>>>>>>>>>>> Nagarahalli <Honnappa.Nagarahalli@arm.com> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hello,
> > > >>>>>>>>>>>>>         Performance of L3fwd example application
> > > >>>>>>>>>>>>> is one of the key
> > > >>>>>>>>>>>> benchmarks in DPDK. However, the application does
> > > >>>>>>>>>>>> not have many debugging statistics to understand the
> > > >>>>>>>>>>>> performance issues. We have added L3fwd as another
> > > >>>>>>>>>>>> mode/stream to testpmd which provides
> > > >>>>>>>>>> enough
> > > >>>>>>>>>>>> statistics at various levels. This has allowed us to
> > > >>>>>>>>>>>> debug the performance issues effectively.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> There is more work to be done to get it to
> > > >>>>>>>>>>>>> upstreamable state. I am
> > > >>>>>>>>>>>> wondering if such a patch is helpful for others and
> > > >>>>>>>>>>>> if the community would be interested in taking a
> > > >>>>>>>>>>>> look. Please let me know
> > > >>>>>>>>> what you think.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> We are using app/proc-info/ to attach and analyze
> > > >>>>>>>>>>>> the
> > > >>>>> performance.
> > > >>>>>>>>>>>> That helps to analyze the unmodified application. I
> > > >>>>>>>>>>>> think, if something is missing in proc-info app, in
> > > >>>>>>>>>>>> my opinion it is better to enhance proc-info so that
> > > >>>>>>>>>>>> it can help other third-party
> > > >>>>>>> applications.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Just my 2c.
> > > >>>>>>>>>>> Thanks Jerin. We will explore that.
> > > >>>>>>>>>>
> > > >>>>>>>>>> I agree it is dangerous to rely too much on testpmd for
> > > >> everything.
> > > >>>>>>>>>> Please tell us what in testpmd could be useful out of it.
> > > >>>>>>>>>>
> > > >>>>>>>>> Things that are very helpful in testpmd are: 1) HW
> > > >>>>>>>>> statistics from the NIC 2) Forwarding stats 3) Burst stats
> > > >>>>>>>>> (indication of headroom
> > > >>>>>>>>> availability) 4) Easy to set parameters like RX and TX
> > > >>>>>>>>> queue depths (among others) without having to recompile.
> > > >>>>>>>>
> > > >>>>>>>> [Kathleen Capella]
> > > >>>>>>>> Thank you for the suggestion of app/proc-info. I've tried it
> > > >>>>>>>> out with l3fwd and see that it does have the HW stats from
> > > >>>>>>>> the NIC and the forwarding
> > > >>>>>>> stats.
> > > >>>>>>>> However, it does not have the burst stats testpmd offers,
> > > >>>>>>>> nor the
> > > >>>>>>>
> > > >>>>>>> One option to see such  level of debugging would be to have
> > > >>>>>>> - Create a memzone in the primary process
> > > >>>>>>> - Application under test can update the stats in memzone based
> > > >>>>>>> on the code flow
> > > >>>>>>> - proc-info can read the counters updated by application under
> > > >>>>>>> test using the memzone object got through
> > > >> rte_memzone_lookup()
> > > >>>>>> Agreed. Currently, using app/proc-info does not provide this
> > > >>>>>> ability. We
> > > >>>>> cannot add this capability to app/proc-info as these stats would
> > > >>>>> be specific to L3fwd application.
> > > >>>>>
> > > >>>>> I meant creating generic counter-read/write infra via memzone to
> > > >>>>> not make it as l3fwd specific.
> > > >>>> Currently, app/proc-info is able to print the stats as they are standardized
> > > >> via the API. But for statistics that are generated in the application, they are
> > > >> very specific to that application. For ex: burst stats in testpmd are very
> > > >> specific to it and another application might implement the same in a very
> > > >> different manner.
> > > >>>>
> > > >>>> In needs to be something like the app/proc-info just needs to be a dumb
> > > >> displaying utility and the application has to do all the heavy lifting of copying
> > > >> the exact display strings to the memory.
> > > >>>
> > > >>> Yes.
> > > >>>
> > > >>>>
> > > >>>>>>>
> > > >>>>>>> Another approach will be using rte_trace()[1] for
> > > >>>>>>> debugging/tracing by adding tracepoints in l3fwd for such events.
> > > >>>>>>> It has a timestamp and the trace format is opensource trace
> > > >>>>>>> format(CTF(Common trace format)), so that we can use post
> > > >>>>>>> posting tools to analyze.
> > > >>>>>>> [1]
> > > >>>>>>> https://doc.dpdk.org/guides/prog_guide/trace_lib.html
> > > >>>>>> This is good for analyzing an incident. I think it is an
> > > >>>>>> overhead for
> > > >>>>> development purposes.
> > > >>>>>
> > > >>>>> Consider if one wants to add burst stats, one can add stats
> > > >>>>> increment under RTE_TRACE_POINT_FP, it will be emitted whenever
> > > >>>>> code flow through that path. Set of events of can be viewed in
> > > >>>>> trace viewer[1]. Would that be enough?
> > > >>>>> Adding traces to l3fwd can be upstreamed as it is useful for
> > > >>>>> others for debugging.
> > > >>>>>
> > > >>>>> [1]
> > > >>>>> https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG
> > > >>>> This needs post processing of the trace info to derive the information, is it
> > > >> correct? For ex: for burst stats, there will be several traces generated
> > > >> collecting the number of packets returned by rte_eth_rx_burst which needs
> > > >> to be post processed.
> > > >>>
> > > >>> Or You can have an additional variable to acculate it.
> > > >>>
> > > >>>> Also, adding traces is equivalent to adding statistics in L3fwd.
> > > >>>
> > > >>> Yes.
> > > >>>
> > > >>> If the sole purpose only stats then it is better to add status in
> > > >>> l3fwd without performance impact. I thought some thing else.
> > > >>>
> > > >>>>
> > > >>>>>>>
> > > >>>>>>>> ability to easily change parameters without having to
> > > >>>>>>>> recompile, which helps reduce debugging time significantly.
> > > >>>> We will not be able to fix this above issue.
> > > >>>
> > > >>> It depends on what you want to debug. Trace can be disabled at runtime.
> > > >>
> > > >>
> > > >> DPDK has existing API's for application metrics but they are rarely used.
> > > >>
> > > >> Why not implement rte_metrics in l3fwd and proc-info?
> > > > This discussion has ended up as a stats discussion. But, we also need to be able to change the configurable parameters easily.
> > > > If we implement the stats and ability to change the configurable parameters, then it is essentially bringing in some of the capabilities from
> > > testpmd to the sample application. I think that will result in lot more code in the sample application and will make it complicated.
> > > >
> > > > Instead our proposal is to take L3fwd to testpmd and use all the infra code that testpmd provides. We see that this approach results in less
> > > amount of code added to DPDK overall.
> > > >
> > > 
> > > Agree that it may help testing to have l3fwd support on the testpmd.
> > > 
> > > Two concerns,
> > > 1) Testpmd already too complex.
> > > 2) Code duplication.
> > > 
> > > For 1), if the l3fwd can be implemented in testpmd as new, independent
> > > forwarding mode, without touching rest of the testpmd, I think it can be OK.
> > 
> > In fact, l3fwd is also quite big and complex:
> > $ wc -l examples/l3fwd/*.[h,c] |grep total
> >   6969 total
> > 
> > Plus it will introduce extra dependencies (fib, lpm, hash, might-be acl?)
> > I am not sure it is a good idea to pull all these complexities into test-pmd.
> > I can't imagine that l3fwd app need ability to configure each and every
> > PMD parameter.
> > From my experience in l3fwd most of cycles are spent not in PMD itself,
> > but in actual packet processing: header parsing and checking, classification,
> > routing table lookup, etc. 
> 
> testpmd goal is to test the driver, not the libraries.
>  
> > > Not sure how to address 2), also lets say we want to add new feature to l3fwd,
> > > where it should go, to the sample or to the testpmd?
> 
> l3fwd is not targetted for testing.
> 
> Maybe we just lack a new test application for routing libraries?
> 
Yes, I think we do. However, I also think that there are quite a few
advantages to having l3fwding supported in testpmd - particularly in terms
of code reuse, since testpmd already has a lot of the functionality that
one would look for. Furthermore, since testpmd has multiple forwarding
engine support, it makes it very easy to add lpm, hash etc. as separate
forwarding engines, rather than trying to mash them all together into a
single one.

The main downsides are as you point out:
1. repurposing a PMD-testing app for also helping testing libs. The counter
point here is that for much testing, the key perf metric for a PMD that
will be looked at is the l3fwd'ing one rather than an iofwd one.
2. the extra dependencies for testpmd. I think that if we do look to merge
in the extra functionality, we can make the presence of the new forwarding
engine dependent on the presence of the required libs.

Overall, I'm cautiously in favour of this work, since I believe the
benefits outweigh the disadvantages. Having l3fwd testing in testpmd would
also allow us to consider simplifying l3fwd example so it is more of an
example and less of a "testing-app-masquerading-as-an-example".

/Bruce

Re: [dpdk-dev] L3fwd mode in testpmd

[Honnappa Nagarahalli]

<snip>

> > > > > This discussion has ended up as a stats discussion. But, we also need
> to be able to change the configurable parameters easily.
> > > > > If we implement the stats and ability to change the configurable
> > > > > parameters, then it is essentially bringing in some of the
> > > > > capabilities from
> > > > testpmd to the sample application. I think that will result in lot more
> code in the sample application and will make it complicated.
> > > > >
> > > > > Instead our proposal is to take L3fwd to testpmd and use all the
> > > > > infra code that testpmd provides. We see that this approach
> > > > > results in less
> > > > amount of code added to DPDK overall.
> > > > >
> > > >
> > > > Agree that it may help testing to have l3fwd support on the testpmd.
> > > >
> > > > Two concerns,
> > > > 1) Testpmd already too complex.
> > > > 2) Code duplication.
> > > >
> > > > For 1), if the l3fwd can be implemented in testpmd as new,
> > > > independent forwarding mode, without touching rest of the testpmd, I
> think it can be OK.
> > >
> > > In fact, l3fwd is also quite big and complex:
> > > $ wc -l examples/l3fwd/*.[h,c] |grep total
> > >   6969 total
> > >
> > > Plus it will introduce extra dependencies (fib, lpm, hash, might-be
> > > acl?) I am not sure it is a good idea to pull all these complexities into test-
> pmd.
> > > I can't imagine that l3fwd app need ability to configure each and
> > > every PMD parameter.
> > > From my experience in l3fwd most of cycles are spent not in PMD
> > > itself, but in actual packet processing: header parsing and
> > > checking, classification, routing table lookup, etc.
> >
> > testpmd goal is to test the driver, not the libraries.
Agree. I think the L3fwd should be an exception as the performance of this application is a key metric for DPDK.

> >
> > > > Not sure how to address 2), also lets say we want to add new
> > > > feature to l3fwd, where it should go, to the sample or to the testpmd?
> >
> > l3fwd is not targetted for testing.
> >
> > Maybe we just lack a new test application for routing libraries?
May be. But, I would think the unit tests for the routing libraries should be enough.

> >
> Yes, I think we do. However, I also think that there are quite a few
> advantages to having l3fwding supported in testpmd - particularly in terms of
> code reuse, since testpmd already has a lot of the functionality that one
> would look for. Furthermore, since testpmd has multiple forwarding engine
> support, it makes it very easy to add lpm, hash etc. as separate forwarding
> engines, rather than trying to mash them all together into a single one.
+1

> 
> The main downsides are as you point out:
> 1. repurposing a PMD-testing app for also helping testing libs. The counter
> point here is that for much testing, the key perf metric for a PMD that will be
> looked at is the l3fwd'ing one rather than an iofwd one.
> 2. the extra dependencies for testpmd. I think that if we do look to merge in
> the extra functionality, we can make the presence of the new forwarding
> engine dependent on the presence of the required libs.
+1

> 
> Overall, I'm cautiously in favour of this work, since I believe the benefits
> outweigh the disadvantages. Having l3fwd testing in testpmd would also
> allow us to consider simplifying l3fwd example so it is more of an example
> and less of a "testing-app-masquerading-as-an-example".
> 
> /Bruce

Re: [dpdk-dev] L3fwd mode in testpmd

[Honnappa Nagarahalli]

<snip>

> > >>>>>>>>>>>> On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli
> > >>>>>>>>>>>> <Honnappa.Nagarahalli@arm.com> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hello,
> > >>>>>>>>>>>>>         Performance of L3fwd example application is one
> > >>>>>>>>>>>>> of the key
> > >>>>>>>>>>>> benchmarks in DPDK. However, the application does not
> > >>>>>>>>>>>> have many debugging statistics to understand the
> > >>>>>>>>>>>> performance issues. We have added L3fwd as another
> > >>>>>>>>>>>> mode/stream to testpmd which provides
> > >>>>>>>>>> enough
> > >>>>>>>>>>>> statistics at various levels. This has allowed us to
> > >>>>>>>>>>>> debug the performance issues effectively.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> There is more work to be done to get it to upstreamable
> > >>>>>>>>>>>>> state. I am
> > >>>>>>>>>>>> wondering if such a patch is helpful for others and if
> > >>>>>>>>>>>> the community would be interested in taking a look.
> > >>>>>>>>>>>> Please let me know
> > >>>>>>>>> what you think.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> We are using app/proc-info/ to attach and analyze the
> > >>>>> performance.
> > >>>>>>>>>>>> That helps to analyze the unmodified application. I
> > >>>>>>>>>>>> think, if something is missing in proc-info app, in my
> > >>>>>>>>>>>> opinion it is better to enhance proc-info so that it can
> > >>>>>>>>>>>> help other third-party
> > >>>>>>> applications.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Just my 2c.
> > >>>>>>>>>>> Thanks Jerin. We will explore that.
> > >>>>>>>>>>
> > >>>>>>>>>> I agree it is dangerous to rely too much on testpmd for
> > >> everything.
> > >>>>>>>>>> Please tell us what in testpmd could be useful out of it.
> > >>>>>>>>>>
> > >>>>>>>>> Things that are very helpful in testpmd are: 1) HW
> > >>>>>>>>> statistics from the NIC 2) Forwarding stats 3) Burst stats
> > >>>>>>>>> (indication of headroom
> > >>>>>>>>> availability) 4) Easy to set parameters like RX and TX queue
> > >>>>>>>>> depths (among others) without having to recompile.
> > >>>>>>>>
> > >>>>>>>> [Kathleen Capella]
> > >>>>>>>> Thank you for the suggestion of app/proc-info. I've tried it
> > >>>>>>>> out with l3fwd and see that it does have the HW stats from
> > >>>>>>>> the NIC and the forwarding
> > >>>>>>> stats.
> > >>>>>>>> However, it does not have the burst stats testpmd offers, nor
> > >>>>>>>> the
> > >>>>>>>
> > >>>>>>> One option to see such  level of debugging would be to have
> > >>>>>>> - Create a memzone in the primary process
> > >>>>>>> - Application under test can update the stats in memzone based
> > >>>>>>> on the code flow
> > >>>>>>> - proc-info can read the counters updated by application under
> > >>>>>>> test using the memzone object got through
> > >> rte_memzone_lookup()
> > >>>>>> Agreed. Currently, using app/proc-info does not provide this
> > >>>>>> ability. We
> > >>>>> cannot add this capability to app/proc-info as these stats would
> > >>>>> be specific to L3fwd application.
> > >>>>>
> > >>>>> I meant creating generic counter-read/write infra via memzone to
> > >>>>> not make it as l3fwd specific.
> > >>>> Currently, app/proc-info is able to print the stats as they are
> > >>>> standardized
> > >> via the API. But for statistics that are generated in the
> > >> application, they are very specific to that application. For ex:
> > >> burst stats in testpmd are very specific to it and another
> > >> application might implement the same in a very different manner.
> > >>>>
> > >>>> In needs to be something like the app/proc-info just needs to be
> > >>>> a dumb
> > >> displaying utility and the application has to do all the heavy
> > >> lifting of copying the exact display strings to the memory.
> > >>>
> > >>> Yes.
> > >>>
> > >>>>
> > >>>>>>>
> > >>>>>>> Another approach will be using rte_trace()[1] for
> > >>>>>>> debugging/tracing by adding tracepoints in l3fwd for such events.
> > >>>>>>> It has a timestamp and the trace format is opensource trace
> > >>>>>>> format(CTF(Common trace format)), so that we can use post
> > >>>>>>> posting tools to analyze.
> > >>>>>>> [1]
> > >>>>>>> https://doc.dpdk.org/guides/prog_guide/trace_lib.html
> > >>>>>> This is good for analyzing an incident. I think it is an
> > >>>>>> overhead for
> > >>>>> development purposes.
> > >>>>>
> > >>>>> Consider if one wants to add burst stats, one can add stats
> > >>>>> increment under RTE_TRACE_POINT_FP, it will be emitted
> whenever
> > >>>>> code flow through that path. Set of events of can be viewed in
> > >>>>> trace viewer[1]. Would that be enough?
> > >>>>> Adding traces to l3fwd can be upstreamed as it is useful for
> > >>>>> others for debugging.
> > >>>>>
> > >>>>> [1]
> > >>>>> https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG
> > >>>> This needs post processing of the trace info to derive the
> > >>>> information, is it
> > >> correct? For ex: for burst stats, there will be several traces
> > >> generated collecting the number of packets returned by
> > >> rte_eth_rx_burst which needs to be post processed.
> > >>>
> > >>> Or You can have an additional variable to acculate it.
> > >>>
> > >>>> Also, adding traces is equivalent to adding statistics in L3fwd.
> > >>>
> > >>> Yes.
> > >>>
> > >>> If the sole purpose only stats then it is better to add status in
> > >>> l3fwd without performance impact. I thought some thing else.
> > >>>
> > >>>>
> > >>>>>>>
> > >>>>>>>> ability to easily change parameters without having to
> > >>>>>>>> recompile, which helps reduce debugging time significantly.
> > >>>> We will not be able to fix this above issue.
> > >>>
> > >>> It depends on what you want to debug. Trace can be disabled at
> runtime.
> > >>
> > >>
> > >> DPDK has existing API's for application metrics but they are rarely used.
> > >>
> > >> Why not implement rte_metrics in l3fwd and proc-info?
> > > This discussion has ended up as a stats discussion. But, we also need to
> be able to change the configurable parameters easily.
> > > If we implement the stats and ability to change the configurable
> > > parameters, then it is essentially bringing in some of the
> > > capabilities from
> > testpmd to the sample application. I think that will result in lot more code in
> the sample application and will make it complicated.
> > >
> > > Instead our proposal is to take L3fwd to testpmd and use all the
> > > infra code that testpmd provides. We see that this approach results
> > > in less
> > amount of code added to DPDK overall.
> > >
> >
> > Agree that it may help testing to have l3fwd support on the testpmd.
> >
> > Two concerns,
> > 1) Testpmd already too complex.
> > 2) Code duplication.
> >
> > For 1), if the l3fwd can be implemented in testpmd as new, independent
> > forwarding mode, without touching rest of the testpmd, I think it can be
> OK.
Yes, this is what we have done. It is a new forwarding mode.
We could remove some forwarding modes from testpmd. For ex: macfwd, macswap seem very similar to iofwd mode.

> 
> In fact, l3fwd is also quite big and complex:
> $ wc -l examples/l3fwd/*.[h,c] |grep total
>   6969 total
> 
> Plus it will introduce extra dependencies (fib, lpm, hash, might-be acl?) I am
> not sure it is a good idea to pull all these complexities into test-pmd.
I do not suggest pulling all these in. In our case, I see that the ask is only on LPM. I am open to hearing what others see as the requirement.

> I can't imagine that l3fwd app need ability to configure each and every PMD
> parameter.
> From my experience in l3fwd most of cycles are spent not in PMD itself, but
> in actual packet processing: header parsing and checking, classification,
> routing table lookup, etc.
During our work, we had to experiment with burst size, rx/tx queue depths along with other PMD specific configuration parameters. The packet processing code remains the same and there is not much to optimize.

> 
> > Not sure how to address 2), also lets say we want to add new feature
> > to l3fwd, where it should go, to the sample or to the testpmd?
L3fwd example will remain as the example. We have to duplicate the code into testpmd. If L3fwd example is changed, it needs to be changed in testpmd as well.

Re: [dpdk-dev] L3fwd mode in testpmd

[Bruce Richardson]

On Tue, Apr 27, 2021 at 11:17:24PM +0000, Honnappa Nagarahalli wrote:
<snip>
> Yes, this is what we have done. It is a new forwarding mode.
> We could remove some forwarding modes from testpmd. For ex: macfwd, macswap seem very similar to iofwd mode.

I'd suggest removing one of those two, but not both. Sometimes one is
testing with a system behind a switch where correct mac addresses need to
be used to ensure proper packet return.

> 
> > 
> > In fact, l3fwd is also quite big and complex:
> > $ wc -l examples/l3fwd/*.[h,c] |grep total
> >   6969 total
> > 
> > Plus it will introduce extra dependencies (fib, lpm, hash, might-be acl?) I am
> > not sure it is a good idea to pull all these complexities into test-pmd.
> I do not suggest pulling all these in. In our case, I see that the ask is only on LPM. I am open to hearing what others see as the requirement.
> 
I think fib is the planned long-term replacement for lpm, and implements
the same algorithm, so I think first versions should use it.

Re: [dpdk-dev] L3fwd mode in testpmd

[Stanisław Kardach]

On Wed, Apr 28, 2021 at 12:48 PM Bruce Richardson <
bruce.richardson@intel.com> wrote:
<snip>

> > I do not suggest pulling all these in. In our case, I see that the ask
> is only on LPM. I am open to hearing what others see as the requirement.
> >
> I think fib is the planned long-term replacement for lpm, and implements
> the same algorithm, so I think first versions should use it.
>
> If I may add my 2 cents. If a decision is made to implement any FIB/LPM
logic in testpmd, it would be great if a vector-only approach that is used
in l3fwd could be avoided.
While working on RISC-V port I had to disable l3fwd compilation for RISC-V
as it requires a vector engine (in l3fwd_em.c). The board I'm currently
using has rv64gc ISA which does not have vector extensions and I use
testpmd extensively for verification, hence it would be a shame to lose it.

Re: [dpdk-dev] L3fwd mode in testpmd

[Thomas Monjalon]

28/04/2021 13:04, Stanisław Kardach:
> On Wed, Apr 28, 2021 at 12:48 PM Bruce Richardson <
> bruce.richardson@intel.com> wrote:
> <snip>
> 
> > > I do not suggest pulling all these in. In our case, I see that the ask
> > is only on LPM. I am open to hearing what others see as the requirement.
> > >
> > I think fib is the planned long-term replacement for lpm, and implements
> > the same algorithm, so I think first versions should use it.
> 
> If I may add my 2 cents. If a decision is made to implement any FIB/LPM
> logic in testpmd, it would be great if a vector-only approach that is used
> in l3fwd could be avoided.
> While working on RISC-V port I had to disable l3fwd compilation for RISC-V
> as it requires a vector engine (in l3fwd_em.c). The board I'm currently
> using has rv64gc ISA which does not have vector extensions and I use
> testpmd extensively for verification, hence it would be a shame to lose it.

Good point.
That's why an example is not a test application,
they have different goals.

Re: [dpdk-dev] L3fwd mode in testpmd

[Honnappa Nagarahalli]

<snip>

On Wed, Apr 28, 2021 at 12:48 PM Bruce Richardson <bruce.richardson@intel.com<mailto:bruce.richardson@intel.com>> wrote:
<snip>
> I do not suggest pulling all these in. In our case, I see that the ask is only on LPM. I am open to hearing what others see as the requirement.
>
I think fib is the planned long-term replacement for lpm, and implements
the same algorithm, so I think first versions should use it.
If I may add my 2 cents. If a decision is made to implement any FIB/LPM logic in testpmd, it would be great if a vector-only approach that is used in l3fwd could be avoided.
While working on RISC-V port I had to disable l3fwd compilation for RISC-V as it requires a vector engine (in l3fwd_em.c). The board I'm currently using has rv64gc ISA which does not have vector extensions and I use testpmd extensively for verification, hence it would be a shame to lose it.
[Honnappa] Sorry, I do not understand this. I see that vector code is under compile time flag as below

#if defined RTE_ARCH_X86 || defined __ARM_NEON
                        l3fwd_em_send_packets(nb_rx, pkts_burst,
                                                        portid, qconf);
#else
                       l3fwd_em_no_opt_send_packets(nb_rx, pkts_burst,
                                                        portid, qconf);
#endif

Re: [dpdk-dev] L3fwd mode in testpmd

[Stanislaw Kardach]

On Wed, Apr 28, 2021 at 09:44:54PM +0000, Honnappa Nagarahalli wrote:
<snip>
> [Honnappa] Sorry, I do not understand this. I see that vector code is under compile time flag as below
> 
> #if defined RTE_ARCH_X86 || defined __ARM_NEON
>                         l3fwd_em_send_packets(nb_rx, pkts_burst,
>                                                         portid, qconf);
> #else
>                        l3fwd_em_no_opt_send_packets(nb_rx, pkts_burst,
>                                                         portid, qconf);
> #endif
Take a look at the ifdef tree at the top of l3fwd_em.c, here:
http://git.dpdk.org/dpdk/tree/examples/l3fwd/l3fwd_em.c#n218

#if defined(__SSE2__)
...
#else
#error No vector engine (SSE, NEON, ALTIVEC) available, check your toolchain
#endif

-- 
Best Regards,
Stanislaw Kardach

Re: [dpdk-dev] L3fwd mode in testpmd

[Ananyev, Konstantin]

Hi Stanislaw,

> 
> On Wed, Apr 28, 2021 at 09:44:54PM +0000, Honnappa Nagarahalli wrote:
> <snip>
> > [Honnappa] Sorry, I do not understand this. I see that vector code is under compile time flag as below
> >
> > #if defined RTE_ARCH_X86 || defined __ARM_NEON
> >                         l3fwd_em_send_packets(nb_rx, pkts_burst,
> >                                                         portid, qconf);
> > #else
> >                        l3fwd_em_no_opt_send_packets(nb_rx, pkts_burst,
> >                                                         portid, qconf);
> > #endif
> Take a look at the ifdef tree at the top of l3fwd_em.c, here:
> http://git.dpdk.org/dpdk/tree/examples/l3fwd/l3fwd_em.c#n218
> 
> #if defined(__SSE2__)
> ...
> #else
> #error No vector engine (SSE, NEON, ALTIVEC) available, check your toolchain
> #endif
> 

I think it is just a flaw and needs to be fixed.
Patch would help here 😊
Konstantin

Re: [dpdk-dev] L3fwd mode in testpmd

[Stanislaw Kardach]

On Thu, Apr 29, 2021 at 08:31:03AM +0000, Ananyev, Konstantin wrote:
> Hi Stanislaw,
> 
> > 
> > On Wed, Apr 28, 2021 at 09:44:54PM +0000, Honnappa Nagarahalli wrote:
> > <snip>
> > > [Honnappa] Sorry, I do not understand this. I see that vector code is under compile time flag as below
> > >
> > > #if defined RTE_ARCH_X86 || defined __ARM_NEON
> > >                         l3fwd_em_send_packets(nb_rx, pkts_burst,
> > >                                                         portid, qconf);
> > > #else
> > >                        l3fwd_em_no_opt_send_packets(nb_rx, pkts_burst,
> > >                                                         portid, qconf);
> > > #endif
> > Take a look at the ifdef tree at the top of l3fwd_em.c, here:
> > http://git.dpdk.org/dpdk/tree/examples/l3fwd/l3fwd_em.c#n218
> > 
> > #if defined(__SSE2__)
> > ...
> > #else
> > #error No vector engine (SSE, NEON, ALTIVEC) available, check your toolchain
> > #endif
> > 
> 
> I think it is just a flaw and needs to be fixed.
> Patch would help here 😊
> Konstantin

It looks as if implementing em_mask_key() is enough to get l3fwd
working. However to me this ifdef seems tricky. How should a scalar
implementation handle the xmm_t type? rte_xmm_t looks like an API
type/union, but both are not mentioned in documentation and are in
platform dependent rte_vect.h only.
So either I add another case for RISC-V or (what seems more proper) add
an else clause implementation. However then should I change this function
to take rte_xmm_t? If not is casting xmm_t to i.e. int32_t[] always
valid? Even if I change to rte_xmm_t, it's not a stable API type, is it?
So what guarantee do I have that it maps to int32_t bit-wise on every
platform?

I think the semantic requirements of xmm_t typedef are a bit undefined as
well as the vector handling across the architectures (being something
rather arch specific). I don't have a clear idea on how to solve this
yet and I would not like to hijack this discussion with vector stuff.

Though I may be missing some obvious solution here. Any idea is welcome.
:)

-- 
Best Regards,
Stanislaw Kardach

Re: [dpdk-dev] L3fwd mode in testpmd

[Ananyev, Konstantin]

> 
> On Thu, Apr 29, 2021 at 08:31:03AM +0000, Ananyev, Konstantin wrote:
> > Hi Stanislaw,
> >
> > >
> > > On Wed, Apr 28, 2021 at 09:44:54PM +0000, Honnappa Nagarahalli wrote:
> > > <snip>
> > > > [Honnappa] Sorry, I do not understand this. I see that vector code is under compile time flag as below
> > > >
> > > > #if defined RTE_ARCH_X86 || defined __ARM_NEON
> > > >                         l3fwd_em_send_packets(nb_rx, pkts_burst,
> > > >                                                         portid, qconf);
> > > > #else
> > > >                        l3fwd_em_no_opt_send_packets(nb_rx, pkts_burst,
> > > >                                                         portid, qconf);
> > > > #endif
> > > Take a look at the ifdef tree at the top of l3fwd_em.c, here:
> > > http://git.dpdk.org/dpdk/tree/examples/l3fwd/l3fwd_em.c#n218
> > >
> > > #if defined(__SSE2__)
> > > ...
> > > #else
> > > #error No vector engine (SSE, NEON, ALTIVEC) available, check your toolchain
> > > #endif
> > >
> >
> > I think it is just a flaw and needs to be fixed.
> > Patch would help here 😊
> > Konstantin
> 
> It looks as if implementing em_mask_key() is enough to get l3fwd
> working. However to me this ifdef seems tricky. How should a scalar
> implementation handle the xmm_t type? rte_xmm_t looks like an API
> type/union, but both are not mentioned in documentation and are in
> platform dependent rte_vect.h only.
> So either I add another case for RISC-V or (what seems more proper) add
> an else clause implementation. However then should I change this function
> to take rte_xmm_t? If not is casting xmm_t to i.e. int32_t[] always
> valid? Even if I change to rte_xmm_t, it's not a stable API type, is it?
> So what guarantee do I have that it maps to int32_t bit-wise on every
> platform?
> 
> I think the semantic requirements of xmm_t typedef are a bit undefined as
> well as the vector handling across the architectures (being something
> rather arch specific). I don't have a clear idea on how to solve this
> yet and I would not like to hijack this discussion with vector stuff.
> 
> Though I may be missing some obvious solution here. Any idea is welcome.
> :)

I think it should be possible to replace xmm_t with rte_xmm_t in ipv(4|6)_5tuple_host
and make em_mask_key to take 'rte_xmm_t *' as a parameter/return value instead of xmm_t.
With that in place scalar version seems straightforward.
Of course perf regression test would be needed after such changes,
but I think with '-O3' it should be no difference.

Re: [dpdk-dev] L3fwd mode in testpmd

[Stanislaw Kardach]

On Thu, Apr 29, 2021 at 11:47:30AM +0000, Ananyev, Konstantin wrote:
<snip>
> > 
> > It looks as if implementing em_mask_key() is enough to get l3fwd
> > working. However to me this ifdef seems tricky. How should a scalar
> > implementation handle the xmm_t type? rte_xmm_t looks like an API
> > type/union, but both are not mentioned in documentation and are in
> > platform dependent rte_vect.h only.
> > So either I add another case for RISC-V or (what seems more proper) add
> > an else clause implementation. However then should I change this function
> > to take rte_xmm_t? If not is casting xmm_t to i.e. int32_t[] always
> > valid? Even if I change to rte_xmm_t, it's not a stable API type, is it?
> > So what guarantee do I have that it maps to int32_t bit-wise on every
> > platform?
> > 
> > I think the semantic requirements of xmm_t typedef are a bit undefined as
> > well as the vector handling across the architectures (being something
> > rather arch specific). I don't have a clear idea on how to solve this
> > yet and I would not like to hijack this discussion with vector stuff.
> > 
> > Though I may be missing some obvious solution here. Any idea is welcome.
> > :)
> 
> I think it should be possible to replace xmm_t with rte_xmm_t in ipv(4|6)_5tuple_host
> and make em_mask_key to take 'rte_xmm_t *' as a parameter/return value instead of xmm_t.
> With that in place scalar version seems straightforward.
> Of course perf regression test would be needed after such changes,
> but I think with '-O3' it should be no difference.
> 
I did that and it works in practice. I'm more asking about the lack of
definition in rte_xmm_t semantics. Because once it's in an example,
people may start assuming it's OK to use it this way.
If it is OK, then I'll just post a patch, otherwise we need a separate
discussion.

-- 
Best Regards,
Stanislaw Kardach

Re: [dpdk-dev] L3fwd mode in testpmd

[Ananyev, Konstantin]

> 
> On Thu, Apr 29, 2021 at 11:47:30AM +0000, Ananyev, Konstantin wrote:
> <snip>
> > >
> > > It looks as if implementing em_mask_key() is enough to get l3fwd
> > > working. However to me this ifdef seems tricky. How should a scalar
> > > implementation handle the xmm_t type? rte_xmm_t looks like an API
> > > type/union, but both are not mentioned in documentation and are in
> > > platform dependent rte_vect.h only.
> > > So either I add another case for RISC-V or (what seems more proper) add
> > > an else clause implementation. However then should I change this function
> > > to take rte_xmm_t? If not is casting xmm_t to i.e. int32_t[] always
> > > valid? Even if I change to rte_xmm_t, it's not a stable API type, is it?
> > > So what guarantee do I have that it maps to int32_t bit-wise on every
> > > platform?
> > >
> > > I think the semantic requirements of xmm_t typedef are a bit undefined as
> > > well as the vector handling across the architectures (being something
> > > rather arch specific). I don't have a clear idea on how to solve this
> > > yet and I would not like to hijack this discussion with vector stuff.
> > >
> > > Though I may be missing some obvious solution here. Any idea is welcome.
> > > :)
> >
> > I think it should be possible to replace xmm_t with rte_xmm_t in ipv(4|6)_5tuple_host
> > and make em_mask_key to take 'rte_xmm_t *' as a parameter/return value instead of xmm_t.
> > With that in place scalar version seems straightforward.
> > Of course perf regression test would be needed after such changes,
> > but I think with '-O3' it should be no difference.
> >
> I did that and it works in practice. I'm more asking about the lack of
> definition in rte_xmm_t semantics. Because once it's in an example,
> people may start assuming it's OK to use it this way.
> If it is OK, then I'll just post a patch, otherwise we need a separate
> discussion.

From my perspective: rte_xmm_t is a union used to simplify SIMD-related
code development. It contains HW specific field (xmm_t) and common ones.
It is not used in public DPDK API, but it is used quite extensively inside various libs.
As a public structure - so it can be used by examples and user code
(as long as it is defined for the given architecture).
So I suppose it is up to you guys to decide do you want to define it for your architecture or not.
If not, but you would still like to run l3fwd, then probably l3fwd_em.c needs to be split
Into l3fwd_em_scalar.c and l3fwd_em_vect.c.  

Konstantin

Re: [dpdk-dev] L3fwd mode in testpmd

[Dharmik Thakkar]

Hi,
Kathleen has submitted an RFC patch [1] in this regard. We’d appreciate your comments.

[1]
https://patches.dpdk.org/project/dpdk/patch/20210430213747.41530-2-kathleen.capella@arm.com/

Thank you!

> On Apr 30, 2021, at 4:28 AM, Ananyev, Konstantin <konstantin.ananyev@intel.com> wrote:
> 
>> 
>> 
>> On Thu, Apr 29, 2021 at 11:47:30AM +0000, Ananyev, Konstantin wrote:
>> <snip>
>>>> 
>>>> It looks as if implementing em_mask_key() is enough to get l3fwd
>>>> working. However to me this ifdef seems tricky. How should a scalar
>>>> implementation handle the xmm_t type? rte_xmm_t looks like an API
>>>> type/union, but both are not mentioned in documentation and are in
>>>> platform dependent rte_vect.h only.
>>>> So either I add another case for RISC-V or (what seems more proper) add
>>>> an else clause implementation. However then should I change this function
>>>> to take rte_xmm_t? If not is casting xmm_t to i.e. int32_t[] always
>>>> valid? Even if I change to rte_xmm_t, it's not a stable API type, is it?
>>>> So what guarantee do I have that it maps to int32_t bit-wise on every
>>>> platform?
>>>> 
>>>> I think the semantic requirements of xmm_t typedef are a bit undefined as
>>>> well as the vector handling across the architectures (being something
>>>> rather arch specific). I don't have a clear idea on how to solve this
>>>> yet and I would not like to hijack this discussion with vector stuff.
>>>> 
>>>> Though I may be missing some obvious solution here. Any idea is welcome.
>>>> :)
>>> 
>>> I think it should be possible to replace xmm_t with rte_xmm_t in ipv(4|6)_5tuple_host
>>> and make em_mask_key to take 'rte_xmm_t *' as a parameter/return value instead of xmm_t.
>>> With that in place scalar version seems straightforward.
>>> Of course perf regression test would be needed after such changes,
>>> but I think with '-O3' it should be no difference.
>>> 
>> I did that and it works in practice. I'm more asking about the lack of
>> definition in rte_xmm_t semantics. Because once it's in an example,
>> people may start assuming it's OK to use it this way.
>> If it is OK, then I'll just post a patch, otherwise we need a separate
>> discussion.
> 
> From my perspective: rte_xmm_t is a union used to simplify SIMD-related
> code development. It contains HW specific field (xmm_t) and common ones.
> It is not used in public DPDK API, but it is used quite extensively inside various libs.
> As a public structure - so it can be used by examples and user code
> (as long as it is defined for the given architecture).
> So I suppose it is up to you guys to decide do you want to define it for your architecture or not.
> If not, but you would still like to run l3fwd, then probably l3fwd_em.c needs to be split
> Into l3fwd_em_scalar.c and l3fwd_em_vect.c.  
> 
> Konstantin

Re: [dpdk-dev] L3fwd mode in testpmd

[Thomas Monjalon]

28/04/2021 12:48, Bruce Richardson:
> On Tue, Apr 27, 2021 at 11:17:24PM +0000, Honnappa Nagarahalli wrote:
> <snip>
> > Yes, this is what we have done. It is a new forwarding mode.
> > We could remove some forwarding modes from testpmd. For ex: macfwd, macswap seem very similar to iofwd mode.
> 
> I'd suggest removing one of those two, but not both. Sometimes one is
> testing with a system behind a switch where correct mac addresses need to
> be used to ensure proper packet return.

For info, there is also 5tswap since 20.08.

Re: [dpdk-dev] L3fwd mode in testpmd

[Ananyev, Konstantin]

 
> > > >>>>>>>>>>>> On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli
> > > >>>>>>>>>>>> <Honnappa.Nagarahalli@arm.com> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hello,
> > > >>>>>>>>>>>>>         Performance of L3fwd example application is one
> > > >>>>>>>>>>>>> of the key
> > > >>>>>>>>>>>> benchmarks in DPDK. However, the application does not
> > > >>>>>>>>>>>> have many debugging statistics to understand the
> > > >>>>>>>>>>>> performance issues. We have added L3fwd as another
> > > >>>>>>>>>>>> mode/stream to testpmd which provides
> > > >>>>>>>>>> enough
> > > >>>>>>>>>>>> statistics at various levels. This has allowed us to
> > > >>>>>>>>>>>> debug the performance issues effectively.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> There is more work to be done to get it to upstreamable
> > > >>>>>>>>>>>>> state. I am
> > > >>>>>>>>>>>> wondering if such a patch is helpful for others and if
> > > >>>>>>>>>>>> the community would be interested in taking a look.
> > > >>>>>>>>>>>> Please let me know
> > > >>>>>>>>> what you think.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> We are using app/proc-info/ to attach and analyze the
> > > >>>>> performance.
> > > >>>>>>>>>>>> That helps to analyze the unmodified application. I
> > > >>>>>>>>>>>> think, if something is missing in proc-info app, in my
> > > >>>>>>>>>>>> opinion it is better to enhance proc-info so that it can
> > > >>>>>>>>>>>> help other third-party
> > > >>>>>>> applications.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Just my 2c.
> > > >>>>>>>>>>> Thanks Jerin. We will explore that.
> > > >>>>>>>>>>
> > > >>>>>>>>>> I agree it is dangerous to rely too much on testpmd for
> > > >> everything.
> > > >>>>>>>>>> Please tell us what in testpmd could be useful out of it.
> > > >>>>>>>>>>
> > > >>>>>>>>> Things that are very helpful in testpmd are: 1) HW
> > > >>>>>>>>> statistics from the NIC 2) Forwarding stats 3) Burst stats
> > > >>>>>>>>> (indication of headroom
> > > >>>>>>>>> availability) 4) Easy to set parameters like RX and TX queue
> > > >>>>>>>>> depths (among others) without having to recompile.
> > > >>>>>>>>
> > > >>>>>>>> [Kathleen Capella]
> > > >>>>>>>> Thank you for the suggestion of app/proc-info. I've tried it
> > > >>>>>>>> out with l3fwd and see that it does have the HW stats from
> > > >>>>>>>> the NIC and the forwarding
> > > >>>>>>> stats.
> > > >>>>>>>> However, it does not have the burst stats testpmd offers, nor
> > > >>>>>>>> the
> > > >>>>>>>
> > > >>>>>>> One option to see such  level of debugging would be to have
> > > >>>>>>> - Create a memzone in the primary process
> > > >>>>>>> - Application under test can update the stats in memzone based
> > > >>>>>>> on the code flow
> > > >>>>>>> - proc-info can read the counters updated by application under
> > > >>>>>>> test using the memzone object got through
> > > >> rte_memzone_lookup()
> > > >>>>>> Agreed. Currently, using app/proc-info does not provide this
> > > >>>>>> ability. We
> > > >>>>> cannot add this capability to app/proc-info as these stats would
> > > >>>>> be specific to L3fwd application.
> > > >>>>>
> > > >>>>> I meant creating generic counter-read/write infra via memzone to
> > > >>>>> not make it as l3fwd specific.
> > > >>>> Currently, app/proc-info is able to print the stats as they are
> > > >>>> standardized
> > > >> via the API. But for statistics that are generated in the
> > > >> application, they are very specific to that application. For ex:
> > > >> burst stats in testpmd are very specific to it and another
> > > >> application might implement the same in a very different manner.
> > > >>>>
> > > >>>> In needs to be something like the app/proc-info just needs to be
> > > >>>> a dumb
> > > >> displaying utility and the application has to do all the heavy
> > > >> lifting of copying the exact display strings to the memory.
> > > >>>
> > > >>> Yes.
> > > >>>
> > > >>>>
> > > >>>>>>>
> > > >>>>>>> Another approach will be using rte_trace()[1] for
> > > >>>>>>> debugging/tracing by adding tracepoints in l3fwd for such events.
> > > >>>>>>> It has a timestamp and the trace format is opensource trace
> > > >>>>>>> format(CTF(Common trace format)), so that we can use post
> > > >>>>>>> posting tools to analyze.
> > > >>>>>>> [1]
> > > >>>>>>> https://doc.dpdk.org/guides/prog_guide/trace_lib.html
> > > >>>>>> This is good for analyzing an incident. I think it is an
> > > >>>>>> overhead for
> > > >>>>> development purposes.
> > > >>>>>
> > > >>>>> Consider if one wants to add burst stats, one can add stats
> > > >>>>> increment under RTE_TRACE_POINT_FP, it will be emitted
> > whenever
> > > >>>>> code flow through that path. Set of events of can be viewed in
> > > >>>>> trace viewer[1]. Would that be enough?
> > > >>>>> Adding traces to l3fwd can be upstreamed as it is useful for
> > > >>>>> others for debugging.
> > > >>>>>
> > > >>>>> [1]
> > > >>>>> https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG
> > > >>>> This needs post processing of the trace info to derive the
> > > >>>> information, is it
> > > >> correct? For ex: for burst stats, there will be several traces
> > > >> generated collecting the number of packets returned by
> > > >> rte_eth_rx_burst which needs to be post processed.
> > > >>>
> > > >>> Or You can have an additional variable to acculate it.
> > > >>>
> > > >>>> Also, adding traces is equivalent to adding statistics in L3fwd.
> > > >>>
> > > >>> Yes.
> > > >>>
> > > >>> If the sole purpose only stats then it is better to add status in
> > > >>> l3fwd without performance impact. I thought some thing else.
> > > >>>
> > > >>>>
> > > >>>>>>>
> > > >>>>>>>> ability to easily change parameters without having to
> > > >>>>>>>> recompile, which helps reduce debugging time significantly.
> > > >>>> We will not be able to fix this above issue.
> > > >>>
> > > >>> It depends on what you want to debug. Trace can be disabled at
> > runtime.
> > > >>
> > > >>
> > > >> DPDK has existing API's for application metrics but they are rarely used.
> > > >>
> > > >> Why not implement rte_metrics in l3fwd and proc-info?
> > > > This discussion has ended up as a stats discussion. But, we also need to
> > be able to change the configurable parameters easily.
> > > > If we implement the stats and ability to change the configurable
> > > > parameters, then it is essentially bringing in some of the
> > > > capabilities from
> > > testpmd to the sample application. I think that will result in lot more code in
> > the sample application and will make it complicated.
> > > >
> > > > Instead our proposal is to take L3fwd to testpmd and use all the
> > > > infra code that testpmd provides. We see that this approach results
> > > > in less
> > > amount of code added to DPDK overall.
> > > >
> > >
> > > Agree that it may help testing to have l3fwd support on the testpmd.
> > >
> > > Two concerns,
> > > 1) Testpmd already too complex.
> > > 2) Code duplication.
> > >
> > > For 1), if the l3fwd can be implemented in testpmd as new, independent
> > > forwarding mode, without touching rest of the testpmd, I think it can be
> > OK.
> Yes, this is what we have done. It is a new forwarding mode.
> We could remove some forwarding modes from testpmd. For ex: macfwd, macswap seem very similar to iofwd mode.

Not really, iowfd doesn't touch packet data at all, while macfwd and macwap change L2 headers.
In fact I found all of them quite helpful (just for different cases), so please keep them.

> 
> >
> > In fact, l3fwd is also quite big and complex:
> > $ wc -l examples/l3fwd/*.[h,c] |grep total
> >   6969 total
> >
> > Plus it will introduce extra dependencies (fib, lpm, hash, might-be acl?) I am
> > not sure it is a good idea to pull all these complexities into test-pmd.
> I do not suggest pulling all these in. In our case, I see that the ask is only on LPM. I am open to hearing what others see as the requirement.

Ok, but l3fwd forwarding model is quite different from current PMD one
(egress queue selection, TX packets buffering, etc.).
I suppose you'll need to pull all that too from l3fwd?

> 
> > I can't imagine that l3fwd app need ability to configure each and every PMD
> > parameter.
> > From my experience in l3fwd most of cycles are spent not in PMD itself, but
> > in actual packet processing: header parsing and checking, classification,
> > routing table lookup, etc.
> During our work, we had to experiment with burst size, rx/tx queue depths along with other PMD specific configuration parameters. The
> packet processing code remains the same and there is not much to optimize.

I think burst-size and rx/tx queue size can be added into l3fwd as new config parameters.
Doesn't look like a major issue to me.
PMD specific parameters could be a problem... anything particular you plan to use?
 
> >
> > > Not sure how to address 2), also lets say we want to add new feature
> > > to l3fwd, where it should go, to the sample or to the testpmd?
> L3fwd example will remain as the example. We have to duplicate the code into testpmd. If L3fwd example is changed, it needs to be
> changed in testpmd as well.

Usually code duplication is not a good sign.
I understand that sometimes it is unavoidable, but why we have to do it here?

Re: [dpdk-dev] L3fwd mode in testpmd

[Honnappa Nagarahalli]

<snip>

> 
> >
> > >
> > > In fact, l3fwd is also quite big and complex:
> > > $ wc -l examples/l3fwd/*.[h,c] |grep total
> > >   6969 total
> > >
> > > Plus it will introduce extra dependencies (fib, lpm, hash, might-be
> > > acl?) I am not sure it is a good idea to pull all these complexities into test-
> pmd.
> > I do not suggest pulling all these in. In our case, I see that the ask is only on
> LPM. I am open to hearing what others see as the requirement.
> 
> Ok, but l3fwd forwarding model is quite different from current PMD one
> (egress queue selection, TX packets buffering, etc.).
> I suppose you'll need to pull all that too from l3fwd?
We will send an RFC to show what it looks like.

> 
> >
> > > I can't imagine that l3fwd app need ability to configure each and
> > > every PMD parameter.
> > > From my experience in l3fwd most of cycles are spent not in PMD
> > > itself, but in actual packet processing: header parsing and
> > > checking, classification, routing table lookup, etc.
> > During our work, we had to experiment with burst size, rx/tx queue
> > depths along with other PMD specific configuration parameters. The packet
> processing code remains the same and there is not much to optimize.
> 
> I think burst-size and rx/tx queue size can be added into l3fwd as new config
> parameters.
> Doesn't look like a major issue to me.
> PMD specific parameters could be a problem... anything particular you plan
> to use?
We are talking about debugging the performance problem which needs all the flexibility possible.

> 
> > >
> > > > Not sure how to address 2), also lets say we want to add new
> > > > feature to l3fwd, where it should go, to the sample or to the testpmd?
> > L3fwd example will remain as the example. We have to duplicate the
> > code into testpmd. If L3fwd example is changed, it needs to be changed in
> testpmd as well.
> 
> Usually code duplication is not a good sign.
> I understand that sometimes it is unavoidable, but why we have to do it
> here?

Re: [dpdk-dev] L3fwd mode in testpmd

[Stephen Hemminger]

On Tue, 27 Apr 2021 10:50:20 +0100
Ferruh Yigit <ferruh.yigit@intel.com> wrote:

> Agree that it may help testing to have l3fwd support on the testpmd.
> 
> Two concerns,
> 1) Testpmd already too complex.
> 2) Code duplication.
> 
> For 1), if the l3fwd can be implemented in testpmd as new, independent
> forwarding mode, without touching rest of the testpmd, I think it can be OK.
> 
> Not sure how to address 2), also lets say we want to add new feature to l3fwd,
> where it should go, to the sample or to the testpmd?

The original purpose of l3fwd seems to be getting lost here.
It was intended as an example, not a complete test or real life application.

Re: [dpdk-dev] L3fwd mode in testpmd

[Honnappa Nagarahalli]

<snip>

> 
> On Tue, 27 Apr 2021 10:50:20 +0100
> Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> 
> > Agree that it may help testing to have l3fwd support on the testpmd.
> >
> > Two concerns,
> > 1) Testpmd already too complex.
> > 2) Code duplication.
> >
> > For 1), if the l3fwd can be implemented in testpmd as new, independent
> > forwarding mode, without touching rest of the testpmd, I think it can be
> OK.
> >
> > Not sure how to address 2), also lets say we want to add new feature
> > to l3fwd, where it should go, to the sample or to the testpmd?
> 
> The original purpose of l3fwd seems to be getting lost here.
> It was intended as an example, not a complete test or real life application.
The issue is, this app has become an industry standard for performance comparison between platforms (whether we like it or not). But, it does not have a whole lot of debugging capabilities. I think adding a L3fwd mode to testpmd will be helpful to keep the sample application simpler.

Re: [dpdk-dev] L3fwd mode in testpmd

[Ananyev, Konstantin]

> >
> > On Tue, 27 Apr 2021 10:50:20 +0100
> > Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> >
> > > Agree that it may help testing to have l3fwd support on the testpmd.
> > >
> > > Two concerns,
> > > 1) Testpmd already too complex.
> > > 2) Code duplication.
> > >
> > > For 1), if the l3fwd can be implemented in testpmd as new, independent
> > > forwarding mode, without touching rest of the testpmd, I think it can be
> > OK.
> > >
> > > Not sure how to address 2), also lets say we want to add new feature
> > > to l3fwd, where it should go, to the sample or to the testpmd?
> >
> > The original purpose of l3fwd seems to be getting lost here.
> > It was intended as an example, not a complete test or real life application.
> The issue is, this app has become an industry standard for performance comparison between platforms (whether we like it or not). 
> But, it
> does not have a whole lot of debugging capabilities.

Ok, could you list what exactly you think is missed?
If we are talking about extra stats, then I still think it is probably easier to add it into l3fwd,
then pull whole l3fwd code into test-pmd.

> I think adding a L3fwd mode to testpmd will be helpful to keep the sample application
> simpler.

Hmm, not sure how these 2 things are linked...
Why let say adding fib-fwd into test-pmd will help to decrease l3fwd code complexity? 

Re: [dpdk-dev] L3fwd mode in testpmd

[Honnappa Nagarahalli]

<snip>

> 
> > >
> > > On Tue, 27 Apr 2021 10:50:20 +0100
> > > Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> > >
> > > > Agree that it may help testing to have l3fwd support on the testpmd.
> > > >
> > > > Two concerns,
> > > > 1) Testpmd already too complex.
> > > > 2) Code duplication.
> > > >
> > > > For 1), if the l3fwd can be implemented in testpmd as new,
> > > > independent forwarding mode, without touching rest of the testpmd,
> > > > I think it can be
> > > OK.
> > > >
> > > > Not sure how to address 2), also lets say we want to add new
> > > > feature to l3fwd, where it should go, to the sample or to the testpmd?
> > >
> > > The original purpose of l3fwd seems to be getting lost here.
> > > It was intended as an example, not a complete test or real life application.
> > The issue is, this app has become an industry standard for performance
> comparison between platforms (whether we like it or not).
> > But, it
> > does not have a whole lot of debugging capabilities.
> 
> Ok, could you list what exactly you think is missed?
> If we are talking about extra stats, then I still think it is probably easier to add
> it into l3fwd, then pull whole l3fwd code into test-pmd.
In stats it is missing hardware, software stats, burst stats.
From run time configuration capability, it is missing burst size, rx/tx queue depth, number of queues we can assign to a core.

> 
> > I think adding a L3fwd mode to testpmd will be helpful to keep the
> > sample application simpler.
> 
> Hmm, not sure how these 2 things are linked...
> Why let say adding fib-fwd into test-pmd will help to decrease l3fwd code
> complexity?
From my perspective, L3fwd example application should showcase how to use DPDK, various packet processing models in DPDK. Anyone new to DPDK should be able to take this code and start creating their application.
However, this application has LPM, Exact match and FIB. I am not sure how Exact Match and FIB modes help the user.
I suggest removing LPM from L3fwd example application as FIB is added (or vice versa).

Re: [dpdk-dev] L3fwd mode in testpmd

[Ananyev, Konstantin]

> <snip>
> 
> >
> > > >
> > > > On Tue, 27 Apr 2021 10:50:20 +0100
> > > > Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> > > >
> > > > > Agree that it may help testing to have l3fwd support on the testpmd.
> > > > >
> > > > > Two concerns,
> > > > > 1) Testpmd already too complex.
> > > > > 2) Code duplication.
> > > > >
> > > > > For 1), if the l3fwd can be implemented in testpmd as new,
> > > > > independent forwarding mode, without touching rest of the testpmd,
> > > > > I think it can be
> > > > OK.
> > > > >
> > > > > Not sure how to address 2), also lets say we want to add new
> > > > > feature to l3fwd, where it should go, to the sample or to the testpmd?
> > > >
> > > > The original purpose of l3fwd seems to be getting lost here.
> > > > It was intended as an example, not a complete test or real life application.
> > > The issue is, this app has become an industry standard for performance
> > comparison between platforms (whether we like it or not).
> > > But, it
> > > does not have a whole lot of debugging capabilities.
> >
> > Ok, could you list what exactly you think is missed?
> > If we are talking about extra stats, then I still think it is probably easier to add
> > it into l3fwd, then pull whole l3fwd code into test-pmd.
> In stats it is missing hardware, software stats, burst stats.
> From run time configuration capability, it is missing burst size, rx/tx queue depth, number of queues we can assign to a core.

AFAIK l3fwd is quite flexible in rx queues to cores assignments, so not sure what else do you have in mind here.
About other things - in general I am ok for adding these extra functionality into l3fwd.
I don't see any major issues here.  
I think it will be less effort then pulling l3fwd into test-pmd.

> 
> >
> > > I think adding a L3fwd mode to testpmd will be helpful to keep the
> > > sample application simpler.
> >
> > Hmm, not sure how these 2 things are linked...
> > Why let say adding fib-fwd into test-pmd will help to decrease l3fwd code
> > complexity?
> From my perspective, L3fwd example application should showcase how to use DPDK, various packet processing models in DPDK. Anyone
> new to DPDK should be able to take this code and start creating their application.
> However, this application has LPM, Exact match and FIB. I am not sure how Exact Match and FIB modes help the user.

About exact match - routing decisions are not always made on des tip address.
Sometimes you need to look at whole addrs+proto+ports tuple.
Exact match provides example for such case.
I also think it would be really good to add ACL mode into l3fwd too (merge l3fwd and l3fd-acl apps into one).

> I suggest removing LPM from L3fwd example application as FIB is added (or vice versa).

AFAIK, as a stretch goal LPM library has to be replaced with FIB one.
Recently FIB was added into l3fwd, so I suppose deprecating and removing LPM mode here
is natural next step (if there would be no performance impact off-course).
CC-ing to Vladimir (FIB maintainer) and Conor to learn their opinions.
 

Re: [dpdk-dev] L3fwd mode in testpmd

[Honnappa Nagarahalli]

<snip>

> > > > > > > > > > On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli
> > > > > > > > > > <Honnappa.Nagarahalli@arm.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hello,
> > > > > > > > > > >         Performance of L3fwd example application is
> > > > > > > > > > > one of the key
> > > > > > > > > > benchmarks in DPDK. However, the application does not
> > > > > > > > > > have many debugging statistics to understand the
> > > > > > > > > > performance issues. We have added L3fwd as another
> > > > > > > > > > mode/stream to testpmd which provides
> > > > > > > > enough
> > > > > > > > > > statistics at various levels. This has allowed us to
> > > > > > > > > > debug the performance issues effectively.
> > > > > > > > > > >
> > > > > > > > > > > There is more work to be done to get it to
> > > > > > > > > > > upstreamable state. I am
> > > > > > > > > > wondering if such a patch is helpful for others and if
> > > > > > > > > > the community would be interested in taking a look.
> > > > > > > > > > Please let me know
> > > > > > > what you think.
> > > > > > > > > >
> > > > > > > > > > We are using app/proc-info/ to attach and analyze the
> > > performance.
> > > > > > > > > > That helps to analyze the unmodified application. I
> > > > > > > > > > think, if something is missing in proc-info app, in my
> > > > > > > > > > opinion it is better to enhance proc-info so that it
> > > > > > > > > > can help other third-party
> > > > > applications.
> > > > > > > > > >
> > > > > > > > > > Just my 2c.
> > > > > > > > > Thanks Jerin. We will explore that.
> > > > > > > >
> > > > > > > > I agree it is dangerous to rely too much on testpmd for
> everything.
> > > > > > > > Please tell us what in testpmd could be useful out of it.
> > > > > > > >
> > > > > > > Things that are very helpful in testpmd are: 1) HW
> > > > > > > statistics from the NIC 2) Forwarding stats 3) Burst stats
> > > > > > > (indication of headroom
> > > > > > > availability) 4) Easy to set parameters like RX and TX queue
> > > > > > > depths (among others) without having to recompile.
> > > > > >
> > > > > > [Kathleen Capella]
> > > > > > Thank you for the suggestion of app/proc-info. I've tried it
> > > > > > out with l3fwd and see that it does have the HW stats from the
> > > > > > NIC and the forwarding
> > > > > stats.
> > > > > > However, it does not have the burst stats testpmd offers, nor
> > > > > > the
> > > > >
> > > > > One option to see such  level of debugging would be to have
> > > > > - Create a memzone in the primary process
> > > > > - Application under test can update the stats in memzone based
> > > > > on the code flow
> > > > > - proc-info can read the counters updated by application under
> > > > > test using the memzone object got through rte_memzone_lookup()
> > > > Agreed. Currently, using app/proc-info does not provide this
> > > > ability. We
> > > cannot add this capability to app/proc-info as these stats would be
> > > specific to L3fwd application.
> > >
> > > I meant creating generic counter-read/write infra via memzone to not
> > > make it as l3fwd specific.
> > Currently, app/proc-info is able to print the stats as they are standardized
> via the API. But for statistics that are generated in the application, they are
> very specific to that application. For ex: burst stats in testpmd are very
> specific to it and another application might implement the same in a very
> different manner.
> >
> > In needs to be something like the app/proc-info just needs to be a dumb
> displaying utility and the application has to do all the heavy lifting of copying
> the exact display strings to the memory.
> 
> Yes.
> 
> >
> > > > >
> > > > > Another approach will be using rte_trace()[1] for
> > > > > debugging/tracing by adding tracepoints in l3fwd for such events.
> > > > > It has a timestamp and the trace format is opensource trace
> > > > > format(CTF(Common trace format)), so that we can use post
> > > > > posting tools to analyze.
> > > > > [1]
> > > > > https://doc.dpdk.org/guides/prog_guide/trace_lib.html
> > > > This is good for analyzing an incident. I think it is an overhead
> > > > for
> > > development purposes.
> > >
> > > Consider if one wants to add burst stats, one can add stats
> > > increment under RTE_TRACE_POINT_FP, it will be emitted whenever
> code
> > > flow through that path. Set of events of can be viewed in trace
> > > viewer[1]. Would that be enough?
> > > Adding traces to l3fwd can be upstreamed as it is useful for others
> > > for debugging.
> > >
> > > [1]
> > > https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG
> > This needs post processing of the trace info to derive the information, is it
> correct? For ex: for burst stats, there will be several traces generated
> collecting the number of packets returned by rte_eth_rx_burst which needs
> to be post processed.
> 
> Or You can have an additional variable to acculate it.
> 
> > Also, adding traces is equivalent to adding statistics in L3fwd.
> 
> Yes.
> 
> If the sole purpose only stats then it is better to add status in l3fwd without
> performance impact. I thought some thing else.
Stats as well as ability to change the configuration parameters without having to recompile.

> 
> >
> > > > >
> > > > > > ability to easily change parameters without having to
> > > > > > recompile, which helps reduce debugging time significantly.
> > We will not be able to fix this above issue.
> 
> It depends on what you want to debug. Trace can be disabled at runtime.
We need to be able to identify the best configurations for various parameters like RX/TX queue depths, burst size etc