DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] DPDK Telemetry library enhancement
@ 2021-05-05  7:49 Harman Kalra
  2021-05-05  8:43 ` Thomas Monjalon
  2021-05-05  8:57 ` David Marchand
  0 siblings, 2 replies; 9+ messages in thread
From: Harman Kalra @ 2021-05-05  7:49 UTC (permalink / raw)
  To: bruce.richardson, kevin.laatz, David Marchand, thomas, stephen; +Cc: dev

Hi All,

We have a use case where we need to gather statistics over network. Current implementation of telemetry library is based on Unix socket, we would like to enhance the scope of library to use network sockets. We understand security challenges with network sockets, to overcome them can we can think of two steps:
1. By default library will be using Unix sockets, it will be user decision to run library with network sockets by passing respective eal flags.
2. We can introduce some key/password authentication mechanism to the library, where only authorized clients can get connected to the server. Password can be passed by the user as eal flags, something similar to vf token which is uuid based.
Kindly provide us suggestions/challenges over this enhancements.

One more query, can we register driver specific telemetry commands to collect driver stats? Is this upstream able?

Thanks
Harman

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [dpdk-dev] DPDK Telemetry library enhancement
  2021-05-05  7:49 [dpdk-dev] DPDK Telemetry library enhancement Harman Kalra
@ 2021-05-05  8:43 ` Thomas Monjalon
  2021-05-05  9:37   ` Jerin Jacob
  2021-05-05  8:57 ` David Marchand
  1 sibling, 1 reply; 9+ messages in thread
From: Thomas Monjalon @ 2021-05-05  8:43 UTC (permalink / raw)
  To: Harman Kalra
  Cc: bruce.richardson, kevin.laatz, David Marchand, stephen, dev,
	bluca, jerinj

05/05/2021 09:49, Harman Kalra:
> Hi All,
> 
> We have a use case where we need to gather statistics over network. Current implementation of telemetry library is based on Unix socket, we would like to enhance the scope of library to use network sockets. We understand security challenges with network sockets, to overcome them can we can think of two steps:
> 1. By default library will be using Unix sockets, it will be user decision to run library with network sockets by passing respective eal flags.
> 2. We can introduce some key/password authentication mechanism to the library, where only authorized clients can get connected to the server. Password can be passed by the user as eal flags, something similar to vf token which is uuid based.
> Kindly provide us suggestions/challenges over this enhancements.

Not sure it should be part of the telemetry lib.
In any case, when implementing network communication,
I encourage you to look at ZeroMQ.

It reminds me this discussion for IF proxy:
http://inbox.dpdk.org/dev/2580933.jp2sp48Hzj@xps/

> 
> One more query, can we register driver specific telemetry commands to collect driver stats? Is this upstream able?




^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [dpdk-dev] DPDK Telemetry library enhancement
  2021-05-05  7:49 [dpdk-dev] DPDK Telemetry library enhancement Harman Kalra
  2021-05-05  8:43 ` Thomas Monjalon
@ 2021-05-05  8:57 ` David Marchand
  2021-05-05 10:05   ` Bruce Richardson
  2021-05-05 10:06   ` [dpdk-dev] [EXT] " Harman Kalra
  1 sibling, 2 replies; 9+ messages in thread
From: David Marchand @ 2021-05-05  8:57 UTC (permalink / raw)
  To: Harman Kalra
  Cc: bruce.richardson, kevin.laatz, thomas, stephen, dev,
	Luca Boccassi, Jerin Jacob Kollanukkaran

On Wed, May 5, 2021 at 9:49 AM Harman Kalra <hkalra@marvell.com> wrote:
> We have a use case where we need to gather statistics over network. Current implementation of telemetry library is based on Unix socket, we would like to enhance the scope of library to use network sockets. We understand security challenges with network sockets, to overcome them can we can think of two steps:
> 1. By default library will be using Unix sockets, it will be user decision to run library with network sockets by passing respective eal flags.
> 2. We can introduce some key/password authentication mechanism to the library, where only authorized clients can get connected to the server. Password can be passed by the user as eal flags, something similar to vf token which is uuid based.
> Kindly provide us suggestions/challenges over this enhancements.

Rather than implement a new protocol, and code a new client, would
collectd fulfill your need?

Info currently retrieved by the telemetry collecd plugin is hardwired.
Maybe some configuration could be added to this plugin for your new stats.


>
> One more query, can we register driver specific telemetry commands to collect driver stats? Is this upstream able?

I don't remember a limitation/guideline on this subject.
Any command can be registered, this is up to the telemetry client to invoke it.


-- 
David Marchand


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [dpdk-dev] DPDK Telemetry library enhancement
  2021-05-05  8:43 ` Thomas Monjalon
@ 2021-05-05  9:37   ` Jerin Jacob
  2021-05-05 10:01     ` Bruce Richardson
  0 siblings, 1 reply; 9+ messages in thread
From: Jerin Jacob @ 2021-05-05  9:37 UTC (permalink / raw)
  To: Thomas Monjalon
  Cc: Harman Kalra, bruce.richardson, kevin.laatz, David Marchand,
	stephen, dev, Luca Boccassi, Jerin Jacob

On Wed, May 5, 2021 at 2:13 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 05/05/2021 09:49, Harman Kalra:
> > Hi All,
> >
> > We have a use case where we need to gather statistics over network. Current implementation of telemetry library is based on Unix socket, we would like to enhance the scope of library to use network sockets. We understand security challenges with network sockets, to overcome them can we can think of two steps:
> > 1. By default library will be using Unix sockets, it will be user decision to run library with network sockets by passing respective eal flags.
> > 2. We can introduce some key/password authentication mechanism to the library, where only authorized clients can get connected to the server. Password can be passed by the user as eal flags, something similar to vf token which is uuid based.
> > Kindly provide us suggestions/challenges over this enhancements.
>
> Not sure it should be part of the telemetry lib.
> In any case, when implementing network communication,
> I encourage you to look at ZeroMQ.

ZeroMQ is a good option for Transport to hide the underlying transport
variants like In-process, Intra-process, TCP.
Also, it has various different options for security backend like
http://curvezmq.org/

if we pick ZeroMQ for transport then it will translate to

1) Remove unix file socket from telemetry
2) Use ZeroMQ for local and remote messaging
3) Needs to make telemetry or dpdk depends on ZeroMQ library(Since
telemetry is experimental, I hope, we can change this)

Thoughts from others including telemetry maintainers


>
> It reminds me this discussion for IF proxy:
> http://inbox.dpdk.org/dev/2580933.jp2sp48Hzj@xps/
>
> >
> > One more query, can we register driver specific telemetry commands to collect driver stats? Is this upstream able?
>
>
>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [dpdk-dev] DPDK Telemetry library enhancement
  2021-05-05  9:37   ` Jerin Jacob
@ 2021-05-05 10:01     ` Bruce Richardson
  2021-05-05 10:14       ` Jerin Jacob
  0 siblings, 1 reply; 9+ messages in thread
From: Bruce Richardson @ 2021-05-05 10:01 UTC (permalink / raw)
  To: Jerin Jacob
  Cc: Thomas Monjalon, Harman Kalra, kevin.laatz, David Marchand,
	stephen, dev, Luca Boccassi, Jerin Jacob

On Wed, May 05, 2021 at 03:07:02PM +0530, Jerin Jacob wrote:
> On Wed, May 5, 2021 at 2:13 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > 05/05/2021 09:49, Harman Kalra:
> > > Hi All,
> > >
> > > We have a use case where we need to gather statistics over network. Current implementation of telemetry library is based on Unix socket, we would like to enhance the scope of library to use network sockets. We understand security challenges with network sockets, to overcome them can we can think of two steps:
> > > 1. By default library will be using Unix sockets, it will be user decision to run library with network sockets by passing respective eal flags.
> > > 2. We can introduce some key/password authentication mechanism to the library, where only authorized clients can get connected to the server. Password can be passed by the user as eal flags, something similar to vf token which is uuid based.
> > > Kindly provide us suggestions/challenges over this enhancements.
> >
> > Not sure it should be part of the telemetry lib.
> > In any case, when implementing network communication,
> > I encourage you to look at ZeroMQ.
> 
> ZeroMQ is a good option for Transport to hide the underlying transport
> variants like In-process, Intra-process, TCP.
> Also, it has various different options for security backend like
> http://curvezmq.org/
> 

Sounds reasonable - I'm in favour of any scheme that means that we don't
need to implement out own authentication or security mechanisms for this.

> if we pick ZeroMQ for transport then it will translate to
> 
> 1) Remove unix file socket from telemetry
> 2) Use ZeroMQ for local and remote messaging
> 3) Needs to make telemetry or dpdk depends on ZeroMQ library(Since
> telemetry is experimental, I hope, we can change this)
> 
> Thoughts from others including telemetry maintainers
> 
I'd like to keep the existing Unix socket around, as well as any extra
zeromq interface, rather than replacing one with the other. Then rather
than introducing a hard dependency on zeromq, it can be an optional one,
where support is compiled in if available. There may be monitoring
applications such as collectd, which run their own local monitoring process
and for which the local unix interface may be as good.

/Bruce

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [dpdk-dev] DPDK Telemetry library enhancement
  2021-05-05  8:57 ` David Marchand
@ 2021-05-05 10:05   ` Bruce Richardson
  2021-05-05 10:06   ` [dpdk-dev] [EXT] " Harman Kalra
  1 sibling, 0 replies; 9+ messages in thread
From: Bruce Richardson @ 2021-05-05 10:05 UTC (permalink / raw)
  To: David Marchand
  Cc: Harman Kalra, kevin.laatz, thomas, stephen, dev, Luca Boccassi,
	Jerin Jacob Kollanukkaran

On Wed, May 05, 2021 at 10:57:37AM +0200, David Marchand wrote:
> On Wed, May 5, 2021 at 9:49 AM Harman Kalra <hkalra@marvell.com> wrote:
> > We have a use case where we need to gather statistics over network. Current implementation of telemetry library is based on Unix socket, we would like to enhance the scope of library to use network sockets. We understand security challenges with network sockets, to overcome them can we can think of two steps:
> > 1. By default library will be using Unix sockets, it will be user decision to run library with network sockets by passing respective eal flags.
> > 2. We can introduce some key/password authentication mechanism to the library, where only authorized clients can get connected to the server. Password can be passed by the user as eal flags, something similar to vf token which is uuid based.
> > Kindly provide us suggestions/challenges over this enhancements.
> 
> Rather than implement a new protocol, and code a new client, would
> collectd fulfill your need?
> 
> Info currently retrieved by the telemetry collecd plugin is hardwired.
> Maybe some configuration could be added to this plugin for your new stats.
> 
> 
> >
> > One more query, can we register driver specific telemetry commands to collect driver stats? Is this upstream able?
> 
> I don't remember a limitation/guideline on this subject.
> Any command can be registered, this is up to the telemetry client to invoke it.
> 
+1

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [dpdk-dev] [EXT] Re: DPDK Telemetry library enhancement
  2021-05-05  8:57 ` David Marchand
  2021-05-05 10:05   ` Bruce Richardson
@ 2021-05-05 10:06   ` Harman Kalra
  2021-05-05 11:26     ` David Marchand
  1 sibling, 1 reply; 9+ messages in thread
From: Harman Kalra @ 2021-05-05 10:06 UTC (permalink / raw)
  To: David Marchand
  Cc: bruce.richardson, kevin.laatz, thomas, stephen, dev,
	Luca Boccassi, Jerin Jacob Kollanukkaran



> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Wednesday, May 5, 2021 2:28 PM
> To: Harman Kalra <hkalra@marvell.com>
> Cc: bruce.richardson@intel.com; kevin.laatz@intel.com;
> thomas@monjalon.net; stephen@networkplumber.org; dev@dpdk.org; Luca
> Boccassi <bluca@debian.org>; Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> Subject: [EXT] Re: DPDK Telemetry library enhancement
> 
> External Email
> 
> ----------------------------------------------------------------------
> On Wed, May 5, 2021 at 9:49 AM Harman Kalra <hkalra@marvell.com> wrote:
> > We have a use case where we need to gather statistics over network. Current
> implementation of telemetry library is based on Unix socket, we would like to
> enhance the scope of library to use network sockets. We understand security
> challenges with network sockets, to overcome them can we can think of two
> steps:
> > 1. By default library will be using Unix sockets, it will be user decision to run
> library with network sockets by passing respective eal flags.
> > 2. We can introduce some key/password authentication mechanism to the
> library, where only authorized clients can get connected to the server. Password
> can be passed by the user as eal flags, something similar to vf token which is
> uuid based.
> > Kindly provide us suggestions/challenges over this enhancements.
> 
> Rather than implement a new protocol, and code a new client, would collectd
> fulfill your need?
> 
> Info currently retrieved by the telemetry collecd plugin is hardwired.
> Maybe some configuration could be added to this plugin for your new stats.

Thanks David, but our requirement is to get statistics remotely over network. DPDK process and daemon may not be running on the same platform.

> 
> 
> >
> > One more query, can we register driver specific telemetry commands to
> collect driver stats? Is this upstream able?
> 
> I don't remember a limitation/guideline on this subject.
> Any command can be registered, this is up to the telemetry client to invoke it.

Sure, we will register new commands to our driver. 

Thanks
Harman

> 
> 
> --
> David Marchand


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [dpdk-dev] DPDK Telemetry library enhancement
  2021-05-05 10:01     ` Bruce Richardson
@ 2021-05-05 10:14       ` Jerin Jacob
  0 siblings, 0 replies; 9+ messages in thread
From: Jerin Jacob @ 2021-05-05 10:14 UTC (permalink / raw)
  To: Bruce Richardson
  Cc: Thomas Monjalon, Harman Kalra, kevin.laatz, David Marchand,
	stephen, dev, Luca Boccassi, Jerin Jacob

On Wed, May 5, 2021 at 3:31 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Wed, May 05, 2021 at 03:07:02PM +0530, Jerin Jacob wrote:
> > On Wed, May 5, 2021 at 2:13 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > >
> > > 05/05/2021 09:49, Harman Kalra:
> > > > Hi All,
> > > >
> > > > We have a use case where we need to gather statistics over network. Current implementation of telemetry library is based on Unix socket, we would like to enhance the scope of library to use network sockets. We understand security challenges with network sockets, to overcome them can we can think of two steps:
> > > > 1. By default library will be using Unix sockets, it will be user decision to run library with network sockets by passing respective eal flags.
> > > > 2. We can introduce some key/password authentication mechanism to the library, where only authorized clients can get connected to the server. Password can be passed by the user as eal flags, something similar to vf token which is uuid based.
> > > > Kindly provide us suggestions/challenges over this enhancements.
> > >
> > > Not sure it should be part of the telemetry lib.
> > > In any case, when implementing network communication,
> > > I encourage you to look at ZeroMQ.
> >
> > ZeroMQ is a good option for Transport to hide the underlying transport
> > variants like In-process, Intra-process, TCP.
> > Also, it has various different options for security backend like
> > http://curvezmq.org/
> >
>
> Sounds reasonable - I'm in favour of any scheme that means that we don't
> need to implement out own authentication or security mechanisms for this.
>
> > if we pick ZeroMQ for transport then it will translate to
> >
> > 1) Remove unix file socket from telemetry
> > 2) Use ZeroMQ for local and remote messaging
> > 3) Needs to make telemetry or dpdk depends on ZeroMQ library(Since
> > telemetry is experimental, I hope, we can change this)
> >
> > Thoughts from others including telemetry maintainers
> >
> I'd like to keep the existing Unix socket around, as well as any extra
> zeromq interface, rather than replacing one with the other. Then rather

I think, the purpose of zeromq is to unify the different transport
mechanisms to avoid multiple code
path for different transport.IMHO, at some point in time it needs to unify.


> than introducing a hard dependency on zeromq, it can be an optional one,
> where support is compiled in if available. There may be monitoring
> applications such as collectd, which run their own local monitoring process
> and for which the local unix interface may be as good.

Collectd has zeromq plugin for transport[1]. So, I think, we can meet
that use case too with zeromq

[1]
https://collectd.org/wiki/index.php/Plugin:ZeroMQ


>
> /Bruce

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [dpdk-dev] [EXT] Re: DPDK Telemetry library enhancement
  2021-05-05 10:06   ` [dpdk-dev] [EXT] " Harman Kalra
@ 2021-05-05 11:26     ` David Marchand
  0 siblings, 0 replies; 9+ messages in thread
From: David Marchand @ 2021-05-05 11:26 UTC (permalink / raw)
  To: Harman Kalra
  Cc: bruce.richardson, kevin.laatz, thomas, stephen, dev,
	Luca Boccassi, Jerin Jacob Kollanukkaran

On Wed, May 5, 2021 at 12:06 PM Harman Kalra <hkalra@marvell.com> wrote:
> > > We have a use case where we need to gather statistics over network. Current
> > implementation of telemetry library is based on Unix socket, we would like to
> > enhance the scope of library to use network sockets. We understand security
> > challenges with network sockets, to overcome them can we can think of two
> > steps:
> > > 1. By default library will be using Unix sockets, it will be user decision to run
> > library with network sockets by passing respective eal flags.
> > > 2. We can introduce some key/password authentication mechanism to the
> > library, where only authorized clients can get connected to the server. Password
> > can be passed by the user as eal flags, something similar to vf token which is
> > uuid based.
> > > Kindly provide us suggestions/challenges over this enhancements.
> >
> > Rather than implement a new protocol, and code a new client, would collectd
> > fulfill your need?
> >
> > Info currently retrieved by the telemetry collecd plugin is hardwired.
> > Maybe some configuration could be added to this plugin for your new stats.
>
> Thanks David, but our requirement is to get statistics remotely over network. DPDK process and daemon may not be running on the same platform.

collectd can act as a local agent for the system hosting the dpdk
process and forward the retrieved data to another "central" collectd,
using the "network"
https://collectd.org/wiki/index.php/Networking_introduction

With collectd plugins, it also opens the possibility to gather other
metrics (out of dpdk) from this system and/or publish in different
forms the collected data.


-- 
David Marchand


^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2021-05-05 11:26 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-05  7:49 [dpdk-dev] DPDK Telemetry library enhancement Harman Kalra
2021-05-05  8:43 ` Thomas Monjalon
2021-05-05  9:37   ` Jerin Jacob
2021-05-05 10:01     ` Bruce Richardson
2021-05-05 10:14       ` Jerin Jacob
2021-05-05  8:57 ` David Marchand
2021-05-05 10:05   ` Bruce Richardson
2021-05-05 10:06   ` [dpdk-dev] [EXT] " Harman Kalra
2021-05-05 11:26     ` David Marchand

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).