[dpdk-dev] DPDK Telemetry library enhancement

[dpdk-dev] DPDK Telemetry library enhancement

[Harman Kalra]

Hi All,

We have a use case where we need to gather statistics over network. Current implementation of telemetry library is based on Unix socket, we would like to enhance the scope of library to use network sockets. We understand security challenges with network sockets, to overcome them can we can think of two steps:
1. By default library will be using Unix sockets, it will be user decision to run library with network sockets by passing respective eal flags.
2. We can introduce some key/password authentication mechanism to the library, where only authorized clients can get connected to the server. Password can be passed by the user as eal flags, something similar to vf token which is uuid based.
Kindly provide us suggestions/challenges over this enhancements.

One more query, can we register driver specific telemetry commands to collect driver stats? Is this upstream able?

Thanks
Harman

Re: [dpdk-dev] DPDK Telemetry library enhancement

[Thomas Monjalon]

05/05/2021 09:49, Harman Kalra:
> Hi All,
> 
> We have a use case where we need to gather statistics over network. Current implementation of telemetry library is based on Unix socket, we would like to enhance the scope of library to use network sockets. We understand security challenges with network sockets, to overcome them can we can think of two steps:
> 1. By default library will be using Unix sockets, it will be user decision to run library with network sockets by passing respective eal flags.
> 2. We can introduce some key/password authentication mechanism to the library, where only authorized clients can get connected to the server. Password can be passed by the user as eal flags, something similar to vf token which is uuid based.
> Kindly provide us suggestions/challenges over this enhancements.

Not sure it should be part of the telemetry lib.
In any case, when implementing network communication,
I encourage you to look at ZeroMQ.

It reminds me this discussion for IF proxy:
http://inbox.dpdk.org/dev/2580933.jp2sp48Hzj@xps/

> 
> One more query, can we register driver specific telemetry commands to collect driver stats? Is this upstream able?

Re: [dpdk-dev] DPDK Telemetry library enhancement

[David Marchand]

On Wed, May 5, 2021 at 9:49 AM Harman Kalra <hkalra@marvell.com> wrote:
> We have a use case where we need to gather statistics over network. Current implementation of telemetry library is based on Unix socket, we would like to enhance the scope of library to use network sockets. We understand security challenges with network sockets, to overcome them can we can think of two steps:
> 1. By default library will be using Unix sockets, it will be user decision to run library with network sockets by passing respective eal flags.
> 2. We can introduce some key/password authentication mechanism to the library, where only authorized clients can get connected to the server. Password can be passed by the user as eal flags, something similar to vf token which is uuid based.
> Kindly provide us suggestions/challenges over this enhancements.

Rather than implement a new protocol, and code a new client, would
collectd fulfill your need?

Info currently retrieved by the telemetry collecd plugin is hardwired.
Maybe some configuration could be added to this plugin for your new stats.


>
> One more query, can we register driver specific telemetry commands to collect driver stats? Is this upstream able?

I don't remember a limitation/guideline on this subject.
Any command can be registered, this is up to the telemetry client to invoke it.


-- 
David Marchand

Re: [dpdk-dev] DPDK Telemetry library enhancement

[Jerin Jacob]

On Wed, May 5, 2021 at 2:13 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 05/05/2021 09:49, Harman Kalra:
> > Hi All,
> >
> > We have a use case where we need to gather statistics over network. Current implementation of telemetry library is based on Unix socket, we would like to enhance the scope of library to use network sockets. We understand security challenges with network sockets, to overcome them can we can think of two steps:
> > 1. By default library will be using Unix sockets, it will be user decision to run library with network sockets by passing respective eal flags.
> > 2. We can introduce some key/password authentication mechanism to the library, where only authorized clients can get connected to the server. Password can be passed by the user as eal flags, something similar to vf token which is uuid based.
> > Kindly provide us suggestions/challenges over this enhancements.
>
> Not sure it should be part of the telemetry lib.
> In any case, when implementing network communication,
> I encourage you to look at ZeroMQ.

ZeroMQ is a good option for Transport to hide the underlying transport
variants like In-process, Intra-process, TCP.
Also, it has various different options for security backend like
http://curvezmq.org/

if we pick ZeroMQ for transport then it will translate to

1) Remove unix file socket from telemetry
2) Use ZeroMQ for local and remote messaging
3) Needs to make telemetry or dpdk depends on ZeroMQ library(Since
telemetry is experimental, I hope, we can change this)

Thoughts from others including telemetry maintainers


>
> It reminds me this discussion for IF proxy:
> http://inbox.dpdk.org/dev/2580933.jp2sp48Hzj@xps/
>
> >
> > One more query, can we register driver specific telemetry commands to collect driver stats? Is this upstream able?
>
>
>

Re: [dpdk-dev] DPDK Telemetry library enhancement

[Bruce Richardson]

On Wed, May 05, 2021 at 03:07:02PM +0530, Jerin Jacob wrote:
> On Wed, May 5, 2021 at 2:13 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > 05/05/2021 09:49, Harman Kalra:
> > > Hi All,
> > >
> > > We have a use case where we need to gather statistics over network. Current implementation of telemetry library is based on Unix socket, we would like to enhance the scope of library to use network sockets. We understand security challenges with network sockets, to overcome them can we can think of two steps:
> > > 1. By default library will be using Unix sockets, it will be user decision to run library with network sockets by passing respective eal flags.
> > > 2. We can introduce some key/password authentication mechanism to the library, where only authorized clients can get connected to the server. Password can be passed by the user as eal flags, something similar to vf token which is uuid based.
> > > Kindly provide us suggestions/challenges over this enhancements.
> >
> > Not sure it should be part of the telemetry lib.
> > In any case, when implementing network communication,
> > I encourage you to look at ZeroMQ.
> 
> ZeroMQ is a good option for Transport to hide the underlying transport
> variants like In-process, Intra-process, TCP.
> Also, it has various different options for security backend like
> http://curvezmq.org/
> 

Sounds reasonable - I'm in favour of any scheme that means that we don't
need to implement out own authentication or security mechanisms for this.

> if we pick ZeroMQ for transport then it will translate to
> 
> 1) Remove unix file socket from telemetry
> 2) Use ZeroMQ for local and remote messaging
> 3) Needs to make telemetry or dpdk depends on ZeroMQ library(Since
> telemetry is experimental, I hope, we can change this)
> 
> Thoughts from others including telemetry maintainers
> 
I'd like to keep the existing Unix socket around, as well as any extra
zeromq interface, rather than replacing one with the other. Then rather
than introducing a hard dependency on zeromq, it can be an optional one,
where support is compiled in if available. There may be monitoring
applications such as collectd, which run their own local monitoring process
and for which the local unix interface may be as good.

/Bruce

Re: [dpdk-dev] DPDK Telemetry library enhancement

[Bruce Richardson]

On Wed, May 05, 2021 at 10:57:37AM +0200, David Marchand wrote:
> On Wed, May 5, 2021 at 9:49 AM Harman Kalra <hkalra@marvell.com> wrote:
> > We have a use case where we need to gather statistics over network. Current implementation of telemetry library is based on Unix socket, we would like to enhance the scope of library to use network sockets. We understand security challenges with network sockets, to overcome them can we can think of two steps:
> > 1. By default library will be using Unix sockets, it will be user decision to run library with network sockets by passing respective eal flags.
> > 2. We can introduce some key/password authentication mechanism to the library, where only authorized clients can get connected to the server. Password can be passed by the user as eal flags, something similar to vf token which is uuid based.
> > Kindly provide us suggestions/challenges over this enhancements.
> 
> Rather than implement a new protocol, and code a new client, would
> collectd fulfill your need?
> 
> Info currently retrieved by the telemetry collecd plugin is hardwired.
> Maybe some configuration could be added to this plugin for your new stats.
> 
> 
> >
> > One more query, can we register driver specific telemetry commands to collect driver stats? Is this upstream able?
> 
> I don't remember a limitation/guideline on this subject.
> Any command can be registered, this is up to the telemetry client to invoke it.
> 
+1

Re: [dpdk-dev] [EXT] Re: DPDK Telemetry library enhancement

[Harman Kalra]

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Wednesday, May 5, 2021 2:28 PM
> To: Harman Kalra <hkalra@marvell.com>
> Cc: bruce.richardson@intel.com; kevin.laatz@intel.com;
> thomas@monjalon.net; stephen@networkplumber.org; dev@dpdk.org; Luca
> Boccassi <bluca@debian.org>; Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> Subject: [EXT] Re: DPDK Telemetry library enhancement
> 
> External Email
> 
> ----------------------------------------------------------------------
> On Wed, May 5, 2021 at 9:49 AM Harman Kalra <hkalra@marvell.com> wrote:
> > We have a use case where we need to gather statistics over network. Current
> implementation of telemetry library is based on Unix socket, we would like to
> enhance the scope of library to use network sockets. We understand security
> challenges with network sockets, to overcome them can we can think of two
> steps:
> > 1. By default library will be using Unix sockets, it will be user decision to run
> library with network sockets by passing respective eal flags.
> > 2. We can introduce some key/password authentication mechanism to the
> library, where only authorized clients can get connected to the server. Password
> can be passed by the user as eal flags, something similar to vf token which is
> uuid based.
> > Kindly provide us suggestions/challenges over this enhancements.
> 
> Rather than implement a new protocol, and code a new client, would collectd
> fulfill your need?
> 
> Info currently retrieved by the telemetry collecd plugin is hardwired.
> Maybe some configuration could be added to this plugin for your new stats.

Thanks David, but our requirement is to get statistics remotely over network. DPDK process and daemon may not be running on the same platform.

> 
> 
> >
> > One more query, can we register driver specific telemetry commands to
> collect driver stats? Is this upstream able?
> 
> I don't remember a limitation/guideline on this subject.
> Any command can be registered, this is up to the telemetry client to invoke it.

Sure, we will register new commands to our driver. 

Thanks
Harman

> 
> 
> --
> David Marchand

Re: [dpdk-dev] DPDK Telemetry library enhancement

[Jerin Jacob]

On Wed, May 5, 2021 at 3:31 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Wed, May 05, 2021 at 03:07:02PM +0530, Jerin Jacob wrote:
> > On Wed, May 5, 2021 at 2:13 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > >
> > > 05/05/2021 09:49, Harman Kalra:
> > > > Hi All,
> > > >
> > > > We have a use case where we need to gather statistics over network. Current implementation of telemetry library is based on Unix socket, we would like to enhance the scope of library to use network sockets. We understand security challenges with network sockets, to overcome them can we can think of two steps:
> > > > 1. By default library will be using Unix sockets, it will be user decision to run library with network sockets by passing respective eal flags.
> > > > 2. We can introduce some key/password authentication mechanism to the library, where only authorized clients can get connected to the server. Password can be passed by the user as eal flags, something similar to vf token which is uuid based.
> > > > Kindly provide us suggestions/challenges over this enhancements.
> > >
> > > Not sure it should be part of the telemetry lib.
> > > In any case, when implementing network communication,
> > > I encourage you to look at ZeroMQ.
> >
> > ZeroMQ is a good option for Transport to hide the underlying transport
> > variants like In-process, Intra-process, TCP.
> > Also, it has various different options for security backend like
> > http://curvezmq.org/
> >
>
> Sounds reasonable - I'm in favour of any scheme that means that we don't
> need to implement out own authentication or security mechanisms for this.
>
> > if we pick ZeroMQ for transport then it will translate to
> >
> > 1) Remove unix file socket from telemetry
> > 2) Use ZeroMQ for local and remote messaging
> > 3) Needs to make telemetry or dpdk depends on ZeroMQ library(Since
> > telemetry is experimental, I hope, we can change this)
> >
> > Thoughts from others including telemetry maintainers
> >
> I'd like to keep the existing Unix socket around, as well as any extra
> zeromq interface, rather than replacing one with the other. Then rather

I think, the purpose of zeromq is to unify the different transport
mechanisms to avoid multiple code
path for different transport.IMHO, at some point in time it needs to unify.


> than introducing a hard dependency on zeromq, it can be an optional one,
> where support is compiled in if available. There may be monitoring
> applications such as collectd, which run their own local monitoring process
> and for which the local unix interface may be as good.

Collectd has zeromq plugin for transport[1]. So, I think, we can meet
that use case too with zeromq

[1]
https://collectd.org/wiki/index.php/Plugin:ZeroMQ


>
> /Bruce

Re: [dpdk-dev] [EXT] Re: DPDK Telemetry library enhancement

[David Marchand]

On Wed, May 5, 2021 at 12:06 PM Harman Kalra <hkalra@marvell.com> wrote:
> > > We have a use case where we need to gather statistics over network. Current
> > implementation of telemetry library is based on Unix socket, we would like to
> > enhance the scope of library to use network sockets. We understand security
> > challenges with network sockets, to overcome them can we can think of two
> > steps:
> > > 1. By default library will be using Unix sockets, it will be user decision to run
> > library with network sockets by passing respective eal flags.
> > > 2. We can introduce some key/password authentication mechanism to the
> > library, where only authorized clients can get connected to the server. Password
> > can be passed by the user as eal flags, something similar to vf token which is
> > uuid based.
> > > Kindly provide us suggestions/challenges over this enhancements.
> >
> > Rather than implement a new protocol, and code a new client, would collectd
> > fulfill your need?
> >
> > Info currently retrieved by the telemetry collecd plugin is hardwired.
> > Maybe some configuration could be added to this plugin for your new stats.
>
> Thanks David, but our requirement is to get statistics remotely over network. DPDK process and daemon may not be running on the same platform.

collectd can act as a local agent for the system hosting the dpdk
process and forward the retrieved data to another "central" collectd,
using the "network"
https://collectd.org/wiki/index.php/Networking_introduction

With collectd plugins, it also opens the possibility to gather other
metrics (out of dpdk) from this system and/or publish in different
forms the collected data.


-- 
David Marchand