[dpdk-dev] Question about telemetry on 18.11 release

[dpdk-dev] Question about telemetry on 18.11 release

[Hideyuki Yamashita]

Hello,

I have some basic questions about telemetry API
which is planned to be relaesed in 18.11.

Note that I have read the follwoing document.
https://doc.dpdk.org/guides/howto/telemetry.html

Q1. In general dpdk application use polling when
receiving packets. So CPU usage is always looks 100%
when measured using e.g. vmstat even if there is no packet receiving.
(CPU is busy for polling packets)
Is that correct?

Q2. Is it correct understanding that dpdk application 
can send "any" value(or metric) to client if client
requested via "telemetry" framework?

Q3.Is it possible to query "real cpu usage" from dpdk
application using "telemetry" framework?
("real cpu usage" I mean if 0 pakcket incoming,
cpu usage indicates almost 0 %)

Q4. Is it possible to inform client about fault or some
trouble from dpdk application?

Background  I ask above is that 
- I am relative new to DPDK world and have almost no knowledge about
"telemetry"
- I am interested in how dpdk applications can "scales" on platform
like OpenStack. I think some mesurement mechanism required
and I thought it might be "telemetry" APIs.

Thanks in advance.

BR,
Hideyuki Yamashita
NTT TechnoCross

Re: [dpdk-dev] Question about telemetry on 18.11 release

[Ferruh Yigit]

On 11/21/2018 7:48 AM, Hideyuki Yamashita wrote:
> Hello,
> 
> I have some basic questions about telemetry API
> which is planned to be relaesed in 18.11.
> 
> Note that I have read the follwoing document.
> https://doc.dpdk.org/guides/howto/telemetry.html
> 
> Q1. In general dpdk application use polling when
> receiving packets. So CPU usage is always looks 100%
> when measured using e.g. vmstat even if there is no packet receiving.
> (CPU is busy for polling packets)
> Is that correct?
> 
> Q2. Is it correct understanding that dpdk application 
> can send "any" value(or metric) to client if client
> requested via "telemetry" framework?
> 
> Q3.Is it possible to query "real cpu usage" from dpdk
> application using "telemetry" framework?
> ("real cpu usage" I mean if 0 pakcket incoming,
> cpu usage indicates almost 0 %)
> 
> Q4. Is it possible to inform client about fault or some
> trouble from dpdk application?
> 
> Background  I ask above is that 
> - I am relative new to DPDK world and have almost no knowledge about
> "telemetry"
> - I am interested in how dpdk applications can "scales" on platform
> like OpenStack. I think some mesurement mechanism required
> and I thought it might be "telemetry" APIs.

Briefly I think yes, telemetry can be measurement mechanism and interface to
external application. Cc'ed Harry & Kevin for more detailed answers.

> 
> Thanks in advance.
> 
> BR,
> Hideyuki Yamashita
> NTT TechnoCross
> 
> 

Re: [dpdk-dev] Question about telemetry on 18.11 release

[Laatz, Kevin]

On 21/11/2018 11:38, Ferruh Yigit wrote:
> On 11/21/2018 7:48 AM, Hideyuki Yamashita wrote:
>> Hello,
>>
>> I have some basic questions about telemetry API
>> which is planned to be relaesed in 18.11.
>>
>> Note that I have read the follwoing document.
>> https://doc.dpdk.org/guides/howto/telemetry.html
>>
>> Q1. In general dpdk application use polling when
>> receiving packets. So CPU usage is always looks 100%
>> when measured using e.g. vmstat even if there is no packet receiving.
>> (CPU is busy for polling packets)
>> Is that correct?
This is a general DPDK question and not specific to telemetry. The above 
is correct,
it will always show as 100% busy.
>>
>> Q2. Is it correct understanding that dpdk application
>> can send "any" value(or metric) to client if client
>> requested via "telemetry" framework?
Correct, assuming the metrics are being collected in the metrics library.
The telemetry library grabs everything in the metrics library and sends
it to the client. If an application exposes its metrics to the metrics 
library,
then telemetry can pick this up.
>>
>> Q3.Is it possible to query "real cpu usage" from dpdk
>> application using "telemetry" framework?
>> ("real cpu usage" I mean if 0 pakcket incoming,
>> cpu usage indicates almost 0 %)
In short, no. The telemetry library only exposes what is available in the
metrics library. Getting the "real DPDK cpu usage" is very difficult in 
general.
>>
>> Q4. Is it possible to inform client about fault or some
>> trouble from dpdk application?

Telemetry can provide the metrics but doesn't have any monitoring
capabilities in terms of alerting the client of, let's say, an enormous 
burst
of dropped packets. The telemetry library is purely a mechanism to 
transport
metrics from DPDK to an external client. It is up to the client to 
interpret
these metrics.
>>
>> Background  I ask above is that
>> - I am relative new to DPDK world and have almost no knowledge about
>> "telemetry"
>> - I am interested in how dpdk applications can "scales" on platform
>> like OpenStack. I think some mesurement mechanism required
>> and I thought it might be "telemetry" APIs.
> Briefly I think yes, telemetry can be measurement mechanism and interface to
> external application. Cc'ed Harry & Kevin for more detailed answers.
>
>> Thanks in advance.
>>
>> BR,
>> Hideyuki Yamashita
>> NTT TechnoCross
>>
>>

Re: [dpdk-dev] Question about telemetry on 18.11 release

[Hideyuki Yamashita]

Hello Kevin,

Thanks for your answering.
Please see inline tagged with [Hideyuki].
> On 21/11/2018 11:38, Ferruh Yigit wrote:
> > On 11/21/2018 7:48 AM, Hideyuki Yamashita wrote:
> >> Hello,
> >>
> >> I have some basic questions about telemetry API
> >> which is planned to be relaesed in 18.11.
> >>
> >> Note that I have read the follwoing document.
> >> https://doc.dpdk.org/guides/howto/telemetry.html
> >>
> >> Q1. In general dpdk application use polling when
> >> receiving packets. So CPU usage is always looks 100%
> >> when measured using e.g. vmstat even if there is no packet receiving.
> >> (CPU is busy for polling packets)
> >> Is that correct?
> This is a general DPDK question and not specific to telemetry. The above is correct,
> it will always show as 100% busy.
> >>
> >> Q2. Is it correct understanding that dpdk application
> >> can send "any" value(or metric) to client if client
> >> requested via "telemetry" framework?
> Correct, assuming the metrics are being collected in the metrics library.
> The telemetry library grabs everything in the metrics library and sends
> it to the client. If an application exposes its metrics to the metrics library,
> then telemetry can pick this up.
[Hideyuki]
OK. Thanks. Now I think I understand.
Both "metrics" and "telemetry" are framework to
transmit information from its producer to consumer
and that's all.
Framework itself does NOT care about the content
of the information. Right?

> >> Q3.Is it possible to query "real cpu usage" from dpdk
> >> application using "telemetry" framework?
> >> ("real cpu usage" I mean if 0 pakcket incoming,
> >> cpu usage indicates almost 0 %)
> In short, no. The telemetry library only exposes what is available in the
> metrics library. Getting the "real DPDK cpu usage" is very difficult in general.
[Hideyuki]
I would like to know the reason why getting "real DPDK cpu usage" is
very difficult in general.
I've learned there already exist mechanism like jobstats to get 
statistics of port and/or lcores.
I thought that by combining those (jobstats,metrics,telemetry)
it might be possible to get cpu usage.


> >> Q4. Is it possible to inform client about fault or some
> >> trouble from dpdk application?
> 
> Telemetry can provide the metrics but doesn't have any monitoring
> capabilities in terms of alerting the client of, let's say, an enormous burst
> of dropped packets. The telemetry library is purely a mechanism to transport
> metrics from DPDK to an external client. It is up to the client to interpret
> these metrics.
[Hideyuki]
Thanks for your information.
I understand that using metrics/telemetry, client can get 
information about burst, dropped packets and is able to
assume "there is a fault" if the client is smart enough.
BTW are there any mechanism in dpdk library to detect fault of dpdk
application except metrics/telemetry?

Thanks!

BR,
Hideyuki Yamashita
NTT TechnoCross

> >> Background  I ask above is that
> >> - I am relative new to DPDK world and have almost no knowledge about
> >> "telemetry"
> >> - I am interested in how dpdk applications can "scales" on platform
> >> like OpenStack. I think some mesurement mechanism required
> >> and I thought it might be "telemetry" APIs.
> > Briefly I think yes, telemetry can be measurement mechanism and interface to
> > external application. Cc'ed Harry & Kevin for more detailed answers.
> >
> >> Thanks in advance.
> >>
> >> BR,
> >> Hideyuki Yamashita
> >> NTT TechnoCross
> >>
> >>

Re: [dpdk-dev] Question about telemetry on 18.11 release

[Rami Rosen]

Hi Hideyuki,

Regarding your questions about DPDK CPU usage, etc:
I believe that due to the way PMDs are implemented, maybe one
should consider using DPDK RTE jobstats API for collecting stats for a DPDK
app. This API reflects better the actual DPDK app stats.
See an example in:
https://doc.dpdk.org/guides/sample_app_ug/l2_forward_job_stats.html
And also see:
https://doc.dpdk.org/api/rte__jobstats_8h_source.html

Regards,
Rami Rosen

בתאריך יום ד׳, 21 בנוב׳ 2018, 09:50, מאת Hideyuki Yamashita <
yamashita.hideyuki@po.ntt-tx.co.jp>:

> Hello,
>
> I have some basic questions about telemetry API
> which is planned to be relaesed in 18.11.
>
> Note that I have read the follwoing document.
> https://doc.dpdk.org/guides/howto/telemetry.html
>
> Q1. In general dpdk application use polling when
> receiving packets. So CPU usage is always looks 100%
> when measured using e.g. vmstat even if there is no packet receiving.
> (CPU is busy for polling packets)
> Is that correct?
>
> Q2. Is it correct understanding that dpdk application
> can send "any" value(or metric) to client if client
> requested via "telemetry" framework?
>
> Q3.Is it possible to query "real cpu usage" from dpdk
> application using "telemetry" framework?
> ("real cpu usage" I mean if 0 pakcket incoming,
> cpu usage indicates almost 0 %)
>
> Q4. Is it possible to inform client about fault or some
> trouble from dpdk application?
>
> Background  I ask above is that
> - I am relative new to DPDK world and have almost no knowledge about
> "telemetry"
> - I am interested in how dpdk applications can "scales" on platform
> like OpenStack. I think some mesurement mechanism required
> and I thought it might be "telemetry" APIs.
>
> Thanks in advance.
>
> BR,
> Hideyuki Yamashita
> NTT TechnoCross
>
>
>

Re: [dpdk-dev] Question about telemetry on 18.11 release

[Hideyuki Yamashita]

Hi Ferruh, Rami

Thanks for your answering me at this busy time during 
release of 18.11.

I will wait response from your colleague.
>Ferruh 

Rami, thanks for your advice.
If I understand you correctly, then 
there already exist APIs to collect statistic 
information inside dpdk including CPU usage.

I've read the l2fwd-jobstats code in addition to documents 
you pointed out and found that there exists code sample showing
statistics per lcore. (You are right)

So my next questions are 

Q5.Are there any API document for jobstats?
I've already read header file definition.
I've already read sample coce.
But API documenation will be a great help to 
understand semantics(or usage) of each APIs 
within jobstats.

Q6. Is it possible to use those (jobstats and telemetry) together?

If I understand l2fwd-jobstats sample code correctly,
statistic information is periodicaly stored into memory
(variables), so at that timing application can pass 
statistics information to teremetry API (server side).
But that is only my guess.

Q7.Are there any samples implementing above?
(My guess is 'no'. jobstats sample only use 'jobstats'
and 'telemetry' sample only use 'telemetry' though.)

Have a nice day and thanks again.
BR,
Hideyuki Yamashita
NTT TechnoCross

> Hi Hideyuki,
> 
> Regarding your questions about DPDK CPU usage, etc:
> I believe that due to the way PMDs are implemented, maybe one
> should consider using DPDK RTE jobstats API for collecting stats for a DPDK
> app. This API reflects better the actual DPDK app stats.
> See an example in:
> https://doc.dpdk.org/guides/sample_app_ug/l2_forward_job_stats.html
> And also see:
> https://doc.dpdk.org/api/rte__jobstats_8h_source.html
> 
> Regards,
> Rami Rosen
> 
> ?????? ??? ??, 21 ????? 2018, 09:50, ??? Hideyuki Yamashita <
> yamashita.hideyuki@po.ntt-tx.co.jp>:
> 
> > Hello,
> >
> > I have some basic questions about telemetry API
> > which is planned to be relaesed in 18.11.
> >
> > Note that I have read the follwoing document.
> > https://doc.dpdk.org/guides/howto/telemetry.html
> >
> > Q1. In general dpdk application use polling when
> > receiving packets. So CPU usage is always looks 100%
> > when measured using e.g. vmstat even if there is no packet receiving.
> > (CPU is busy for polling packets)
> > Is that correct?
> >
> > Q2. Is it correct understanding that dpdk application
> > can send "any" value(or metric) to client if client
> > requested via "telemetry" framework?
> >
> > Q3.Is it possible to query "real cpu usage" from dpdk
> > application using "telemetry" framework?
> > ("real cpu usage" I mean if 0 pakcket incoming,
> > cpu usage indicates almost 0 %)
> >
> > Q4. Is it possible to inform client about fault or some
> > trouble from dpdk application?
> >
> > Background  I ask above is that
> > - I am relative new to DPDK world and have almost no knowledge about
> > "telemetry"
> > - I am interested in how dpdk applications can "scales" on platform
> > like OpenStack. I think some mesurement mechanism required
> > and I thought it might be "telemetry" APIs.
> >
> > Thanks in advance.
> >
> > BR,
> > Hideyuki Yamashita
> > NTT TechnoCross
> >
> >
> >

Re: [dpdk-dev] Question about telemetry on 18.11 release

[Rami Rosen]

Hi, Hideyuki,

>Rami, thanks for your advice.
>If I understand you correctly, then
>there already exist APIs to collect statistic
>information inside dpdk including CPU usage.

Yup. I want to also note that the librte jobstats is very veteran, it
exists in DPDK versions prior to 16.04 (like in 2.2.0; see:
https://github.com/DPDK/dpdk/tree/v2.2.0/lib/librte_jobstats )
The telemetry library is relatively new.

Following are my answers to your new queries (Q5-Q7):

>Q5.Are there any API document for jobstats?
AFAIK, there is no API document apart from what I already sent (namely
the sample guide and
the API link, https://doc.dpdk.org/api/rte__jobstats_8h_source.html)

>Q6. Is it possible to use those (jobstats and telemetry) together?
It could be, never tried it though.

>Q7.Are there any samples implementing above?

Assuming you mean implementing both APIs together: No, AFAIK. It will
be great if someone will write documentation and implement such a
thing, if it is doable.

Regards,
Rami Rosen

Re: [dpdk-dev] Question about telemetry on 18.11 release

[Hideyuki Yamashita]

Hello Rami,

Thanks for your response.
Please see inline.

> Hi, Hideyuki,
> 
> >Rami, thanks for your advice.
> >If I understand you correctly, then
> >there already exist APIs to collect statistic
> >information inside dpdk including CPU usage.
> 
> Yup. I want to also note that the librte jobstats is very veteran, it
> exists in DPDK versions prior to 16.04 (like in 2.2.0; see:
> https://github.com/DPDK/dpdk/tree/v2.2.0/lib/librte_jobstats )
> The telemetry library is relatively new.
> 
> Following are my answers to your new queries (Q5-Q7):
> 
> >Q5.Are there any API document for jobstats?
> AFAIK, there is no API document apart from what I already sent (namely
> the sample guide and
> the API link, https://doc.dpdk.org/api/rte__jobstats_8h_source.html)
[Hideyuki]
Are there any reason why accompanied document does NOT exit?
(if you know the reason why)

> >Q6. Is it possible to use those (jobstats and telemetry) together?
> It could be, never tried it though.
> 
> >Q7.Are there any samples implementing above?
> 
> Assuming you mean implementing both APIs together: No, AFAIK. It will
> be great if someone will write documentation and implement such a
> thing, if it is doable.
[Hideyuki]
Is it possible to contribute such a sample application by my side?
(Just idea, I need permission from my boss though..)

In that case, is it correct that what I should do is 
create/verify my contribution and post patches to dev list?
Or are there any steps(procedures) to add such new sample?

Q8.
BTW, when I read jobstats sample, I felt that 
application programmers should be fully aware of the jobstats
from the first place of software design (many timers in main function) 
and it is NOT "opt-in" "opt-out" feature.

When considering many NFV applications are already 
there, I think it is more preferable that such statistics features
can be "opt-in" or "opt-out".
Is it possible to realize above using current DPDK framework?

Thanks again.

BR,
Hideyuki Yamashita
NTT TechnoCross

> Regards,
> Rami Rosen

Re: [dpdk-dev] Question about telemetry on 18.11 release

[Rami Rosen]

HI, Hideyuki,

>Are there any reason why accompanied document does NOT exit?
>> (if you know the reason why)

exit->exist; I don't know.

> Is it possible to contribute such a sample application by my side?
> (Just idea, I need permission from my boss though..)
>
You may try, DPDK is an Open Source project.

> Q8.
> BTW, when I read jobstats sample, I felt that
> application programmers should be fully aware of the jobstats
> from the first place of software design (many timers in main function)
> and it is NOT "opt-in" "opt-out" feature.

I believe many people think the same.

Regards,
Rami Rosen