Re: [dpdk-dev] [PATCH v3] pipeline: add statistics for librte_pipeline

Re: [dpdk-dev] [PATCH v3] pipeline: add statistics for librte_pipeline

[Rajagopalan Sivaramakrishnan]

> > You also reiterate that you would like to have the stats always enabled. You
> can definitely do this, it is one of the available choices, but why not also
> accommodate the users that want to pick the opposite choice? Why force
> apps to spend cycles on stats if the app either does not want these counters
> (library counters not relevant for that app, maybe the app is only interested
> in maintaining some other stats that it implements itself) or do not want
> them anymore (maybe they only needed them during debug phase), etc?
> Jay asked this question, and I did my best in my reply to describe our
> motivation (http://www.dpdk.org/ml/archives/dev/2015-May/017992.html).
> Maybe you missed that post, it would be good to get your reply on this one
> too.
>
> I want to see DPDK get out of the config madness.
> This is real code, not an Intel benchmark special.


I agree that statistics will definitely be required in most real-world production environments and the overhead
from per-core stats gathering will be minimal if the data structures are such that CPU cache thrashing is avoided.
However, if there are scenarios where it is desirable to turn stats off, I think we can live with a config option.
I am not comfortable with using the log level to enable/disable statistics as they are not really related. A
separate config option for stats collection seems like a reasonable compromise.

Raja

Re: [dpdk-dev] [PATCH v3] pipeline: add statistics for librte_pipeline

[Dumitrescu, Cristian]

Hi Raja,

Thanks for your input.

I think we have the following options identified so far for stats collection configuration:

1. Stats configuration through the RTE_LOG_LEVEL
2. Single configuration flag global for all DPDK libraries
3. Single configuration flag per DPDK library

It would be good if Thomas and Stephen, as well as others, would reply with their preference order.

My personal preference order is: 3., 2., 1., but I can work with any of the above that is identified by the majority of the replies. My goal right now is reaching a conclusion on this item as soon as we can.

Regards,
Cristian



> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rajagopalan
> Sivaramakrishnan
> Sent: Wednesday, May 27, 2015 11:45 PM
> To: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v3] pipeline: add statistics for librte_pipeline
> 
> 
> > > You also reiterate that you would like to have the stats always enabled.
> You
> > can definitely do this, it is one of the available choices, but why not also
> > accommodate the users that want to pick the opposite choice? Why force
> > apps to spend cycles on stats if the app either does not want these
> counters
> > (library counters not relevant for that app, maybe the app is only
> interested
> > in maintaining some other stats that it implements itself) or do not want
> > them anymore (maybe they only needed them during debug phase), etc?
> > Jay asked this question, and I did my best in my reply to describe our
> > motivation (http://www.dpdk.org/ml/archives/dev/2015-
> May/017992.html).
> > Maybe you missed that post, it would be good to get your reply on this one
> > too.
> >
> > I want to see DPDK get out of the config madness.
> > This is real code, not an Intel benchmark special.
> 
> 
> I agree that statistics will definitely be required in most real-world production
> environments and the overhead
> from per-core stats gathering will be minimal if the data structures are such
> that CPU cache thrashing is avoided.
> However, if there are scenarios where it is desirable to turn stats off, I think
> we can live with a config option.
> I am not comfortable with using the log level to enable/disable statistics as
> they are not really related. A
> separate config option for stats collection seems like a reasonable
> compromise.
> 
> Raja

Re: [dpdk-dev] [PATCH v3] pipeline: add statistics for librte_pipeline

[Rajagopalan Sivaramakrishnan]

My first preference would be to enable stats always. However, if the
majority feels that it should be optional,
your preference of 3, 2, 1 seems fine to me. I hope the same decision will
apply to port/table/other stats.

Raja

On 5/28/15, 12:26 PM, "Dumitrescu, Cristian"
<cristian.dumitrescu@intel.com> wrote:

>Hi Raja,
>
>Thanks for your input.
>
>I think we have the following options identified so far for stats
>collection configuration:
>
>1. Stats configuration through the RTE_LOG_LEVEL
>2. Single configuration flag global for all DPDK libraries
>3. Single configuration flag per DPDK library
>
>It would be good if Thomas and Stephen, as well as others, would reply
>with their preference order.
>
>My personal preference order is: 3., 2., 1., but I can work with any of
>the above that is identified by the majority of the replies. My goal
>right now is reaching a conclusion on this item as soon as we can.
>
>Regards,
>Cristian
>
>
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rajagopalan
>> Sivaramakrishnan
>> Sent: Wednesday, May 27, 2015 11:45 PM
>> To: dev@dpdk.org
>> Subject: Re: [dpdk-dev] [PATCH v3] pipeline: add statistics for
>>librte_pipeline
>> 
>> 
>> > > You also reiterate that you would like to have the stats always
>>enabled.
>> You
>> > can definitely do this, it is one of the available choices, but why
>>not also
>> > accommodate the users that want to pick the opposite choice? Why force
>> > apps to spend cycles on stats if the app either does not want these
>> counters
>> > (library counters not relevant for that app, maybe the app is only
>> interested
>> > in maintaining some other stats that it implements itself) or do not
>>want
>> > them anymore (maybe they only needed them during debug phase), etc?
>> > Jay asked this question, and I did my best in my reply to describe our
>> > motivation (http://www.dpdk.org/ml/archives/dev/2015-
>> May/017992.html).
>> > Maybe you missed that post, it would be good to get your reply on
>>this one
>> > too.
>> >
>> > I want to see DPDK get out of the config madness.
>> > This is real code, not an Intel benchmark special.
>> 
>> 
>> I agree that statistics will definitely be required in most real-world
>>production
>> environments and the overhead
>> from per-core stats gathering will be minimal if the data structures
>>are such
>> that CPU cache thrashing is avoided.
>> However, if there are scenarios where it is desirable to turn stats
>>off, I think
>> we can live with a config option.
>> I am not comfortable with using the log level to enable/disable
>>statistics as
>> they are not really related. A
>> separate config option for stats collection seems like a reasonable
>> compromise.
>> 
>> Raja

Re: [dpdk-dev] [PATCH v3] pipeline: add statistics for librte_pipeline

[Ramia, Kannan Babu]

The confusion is due to whether you consider stats as a library feature or Debug feature. Mostly log levels are considered as debug features in the production system and controlled system wide flag not per library flags. While statistics could be considered as a library feature which could be turned on and off depends on the application needs.  I am with Cristian to have per library feature configuration flag for statistics. 

Regards
Kannan Babu

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Dumitrescu, Cristian
Sent: Friday, May 29, 2015 12:56 AM
To: Rajagopalan Sivaramakrishnan; dev@dpdk.org
Subject: Re: [dpdk-dev] [PATCH v3] pipeline: add statistics for librte_pipeline

Hi Raja,

Thanks for your input.

I think we have the following options identified so far for stats collection configuration:

1. Stats configuration through the RTE_LOG_LEVEL 2. Single configuration flag global for all DPDK libraries 3. Single configuration flag per DPDK library

It would be good if Thomas and Stephen, as well as others, would reply with their preference order.

My personal preference order is: 3., 2., 1., but I can work with any of the above that is identified by the majority of the replies. My goal right now is reaching a conclusion on this item as soon as we can.

Regards,
Cristian



> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rajagopalan 
> Sivaramakrishnan
> Sent: Wednesday, May 27, 2015 11:45 PM
> To: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v3] pipeline: add statistics for 
> librte_pipeline
> 
> 
> > > You also reiterate that you would like to have the stats always enabled.
> You
> > can definitely do this, it is one of the available choices, but why 
> > not also accommodate the users that want to pick the opposite 
> > choice? Why force apps to spend cycles on stats if the app either 
> > does not want these
> counters
> > (library counters not relevant for that app, maybe the app is only
> interested
> > in maintaining some other stats that it implements itself) or do not 
> > want them anymore (maybe they only needed them during debug phase), etc?
> > Jay asked this question, and I did my best in my reply to describe 
> > our motivation (http://www.dpdk.org/ml/archives/dev/2015-
> May/017992.html).
> > Maybe you missed that post, it would be good to get your reply on 
> > this one too.
> >
> > I want to see DPDK get out of the config madness.
> > This is real code, not an Intel benchmark special.
> 
> 
> I agree that statistics will definitely be required in most real-world 
> production environments and the overhead from per-core stats gathering 
> will be minimal if the data structures are such that CPU cache 
> thrashing is avoided.
> However, if there are scenarios where it is desirable to turn stats 
> off, I think we can live with a config option.
> I am not comfortable with using the log level to enable/disable 
> statistics as they are not really related. A separate config option 
> for stats collection seems like a reasonable compromise.
> 
> Raja

Re: [dpdk-dev] [PATCH v3] pipeline: add statistics for librte_pipeline

[Thomas Monjalon]

2015-05-28 19:26, Dumitrescu, Cristian:
> I think we have the following options identified so far for stats collection configuration:
> 
> 1. Stats configuration through the RTE_LOG_LEVEL
> 2. Single configuration flag global for all DPDK libraries
> 3. Single configuration flag per DPDK library
> 
> It would be good if Thomas and Stephen, as well as others, would reply with their preference order.

There are important design questions in these threads.
I think that the best way to come to a conclusion is to submit a design rule
	- to state whether statistics must be considered as a feature or as debug,
	- and to decide whether stats must be always available or disabled
globally or per-library.
It should be written in a new doc. I suggest docs/guidelines/design.rst.
Then we'll have to discuss and vote on this base. It will avoid future
discussions.
The underlying discussion is to decide if every cycle is important even if
there is a real usability drawback.

In order to reach a conclusion, it seems reasonnable to target a consensus
2 weeks after the first submission of these design rules.
Thanks Cristian to follow up.