[dpdk-dev] [PATCH] metrics/windows: build rte_metrics library

[dpdk-dev] [PATCH] metrics/windows: build rte_metrics library

[Tyler Retzlaff]

From: Tyler Retzlaff <roretzla@microsoft.com>

declare reduced set of exports for rte_metrics on windows and include
it in the set of libraries to build and install.

rte_metrics_tel* exports depend on conditionally available component.
this change minimally allows enabling the building of rte_metrics on
windows.

Signed-off-by: Tyler Retzlaff <roretzla@microsoft.com>
---
 lib/librte_metrics/rte_metrics_exports.def | 8 ++++++++
 lib/meson.build                            | 1 +
 2 files changed, 9 insertions(+)
 create mode 100644 lib/librte_metrics/rte_metrics_exports.def

diff --git a/lib/librte_metrics/rte_metrics_exports.def b/lib/librte_metrics/rte_metrics_exports.def
new file mode 100644
index 000000000..40ec342a0
--- /dev/null
+++ b/lib/librte_metrics/rte_metrics_exports.def
@@ -0,0 +1,8 @@
+EXPORTS
+	rte_metrics_get_names
+	rte_metrics_get_values
+	rte_metrics_init
+	rte_metrics_reg_name
+	rte_metrics_reg_names
+	rte_metrics_update_value
+	rte_metrics_update_values
diff --git a/lib/meson.build b/lib/meson.build
index ed00f8914..845397fba 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -45,6 +45,7 @@ if is_windows
 		'mempool', 'mbuf', 'net', 'meter', 'ethdev', 'pci',
 		'cmdline',
 		'hash',
+		'metrics',
 	] # only supported libraries for windows
 endif
 
-- 
2.29.0.vfs.0.0

Re: [dpdk-dev] [PATCH] metrics/windows: build rte_metrics library

[Dmitry Kozlyuk]

+ Pallavi, Khoa

On Mon, 11 Jan 2021 15:37:26 -0800, Tyler Retzlaff wrote:
> From: Tyler Retzlaff <roretzla@microsoft.com>
> 
> declare reduced set of exports for rte_metrics on windows and include
> it in the set of libraries to build and install.
> 
> rte_metrics_tel* exports depend on conditionally available component.
> this change minimally allows enabling the building of rte_metrics on
> windows.

What's the future plan? Once we talked about a script that would inspect
objects and filter missing functions from the list in .map to build .def. Is
MSFT working on it? Or do we choose another way?

> --- /dev/null
> +++ b/lib/librte_metrics/rte_metrics_exports.def
> @@ -0,0 +1,8 @@
> +EXPORTS
> +	rte_metrics_get_names
> +	rte_metrics_get_values
> +	rte_metrics_init

+ rte_metrics_deinit

> +	rte_metrics_reg_name
> +	rte_metrics_reg_names
> +	rte_metrics_update_value
> +	rte_metrics_update_values

Re: [dpdk-dev] [PATCH] metrics/windows: build rte_metrics library

[Tyler Retzlaff]

On Tue, Jan 12, 2021 at 04:15:34AM +0300, Dmitry Kozlyuk wrote:
> + Pallavi, Khoa
> 
> What's the future plan? Once we talked about a script that would inspect
> objects and filter missing functions from the list in .map to build .def. Is
> MSFT working on it? Or do we choose another way?

MSFT has a task tracking work to resolve this. We agree it is cumbersome to
try and keep the .map and .def files in sync by hand.

Re: [dpdk-dev] [PATCH] metrics/windows: build rte_metrics library

[Tal Shnaiderman]

> Subject: Re: [dpdk-dev] [PATCH] metrics/windows: build rte_metrics library
> 
> On Tue, Jan 12, 2021 at 04:15:34AM +0300, Dmitry Kozlyuk wrote:
> > + Pallavi, Khoa
> >
> > What's the future plan? Once we talked about a script that would
> > inspect objects and filter missing functions from the list in .map to
> > build .def. Is MSFT working on it? Or do we choose another way?
> 
> MSFT has a task tracking work to resolve this. We agree it is cumbersome to
> try and keep the .map and .def files in sync by hand.

Is there a DD to complete this task? Dmitry already sent an email with the details on how to resolve it using DUMPBIN.

[dpdk-dev] [PATCH v2] metrics/windows: build rte_metrics library

[Tyler Retzlaff]

From: Tyler Retzlaff <roretzla@microsoft.com>

declare reduced set of exports for rte_metrics on windows and include
it in the set of libraries to build and install.

rte_metrics_tel* exports depend on conditionally available component.
this change minimally allows enabling the building of rte_metrics on
windows.

Signed-off-by: Tyler Retzlaff <roretzla@microsoft.com>
---
 lib/librte_metrics/rte_metrics_exports.def | 9 +++++++++
 lib/meson.build                            | 1 +
 2 files changed, 10 insertions(+)
 create mode 100644 lib/librte_metrics/rte_metrics_exports.def

diff --git a/lib/librte_metrics/rte_metrics_exports.def b/lib/librte_metrics/rte_metrics_exports.def
new file mode 100644
index 000000000..4cb7352ae
--- /dev/null
+++ b/lib/librte_metrics/rte_metrics_exports.def
@@ -0,0 +1,9 @@
+EXPORTS
+	rte_metrics_get_names
+	rte_metrics_get_values
+	rte_metrics_init
+	rte_metrics_deinit
+	rte_metrics_reg_name
+	rte_metrics_reg_names
+	rte_metrics_update_value
+	rte_metrics_update_values
diff --git a/lib/meson.build b/lib/meson.build
index ed00f8914..845397fba 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -45,6 +45,7 @@ if is_windows
 		'mempool', 'mbuf', 'net', 'meter', 'ethdev', 'pci',
 		'cmdline',
 		'hash',
+		'metrics',
 	] # only supported libraries for windows
 endif
 
-- 
2.29.0.vfs.0.0

Re: [dpdk-dev] [PATCH v2] metrics/windows: build rte_metrics library

[Thomas Monjalon]

12/01/2021 02:30, Tyler Retzlaff:
> From: Tyler Retzlaff <roretzla@microsoft.com>
> 
> declare reduced set of exports for rte_metrics on windows and include
> it in the set of libraries to build and install.
> 
> rte_metrics_tel* exports depend on conditionally available component.
> this change minimally allows enabling the building of rte_metrics on
> windows.
> 
> Signed-off-by: Tyler Retzlaff <roretzla@microsoft.com>

Not sure it makes sense without the new telemetry feature.
Please focus on telemetry lib instead of half-enabling
the old metrics lib.

Re: [dpdk-dev] [PATCH v2] metrics/windows: build rte_metrics library

[Tyler Retzlaff]

On Sun, Jan 17, 2021 at 11:19:55PM +0100, Thomas Monjalon wrote:
> 
> Not sure it makes sense without the new telemetry feature.
> Please focus on telemetry lib instead of half-enabling
> the old metrics lib.
> 

can you elaborate? (or reference a mailing list discussion) that gives some
guidance?

is the telemetry lib a replacement for metrics? the component we have now
relies on the non-telemetry functions exported from metrics but does not
use the telemetry functions.

also, i notice that the meson.build for telemetry lib has an include path
that references rte_metrics but does not appear to actually include any of
the headers from rte_metrics (vestigial? missed in previous cleanup perhaps?)

thanks!

Re: [dpdk-dev] [PATCH v2] metrics/windows: build rte_metrics library

[Thomas Monjalon]

19/01/2021 22:31, Tyler Retzlaff:
> On Sun, Jan 17, 2021 at 11:19:55PM +0100, Thomas Monjalon wrote:
> > 
> > Not sure it makes sense without the new telemetry feature.
> > Please focus on telemetry lib instead of half-enabling
> > the old metrics lib.
> > 
> 
> can you elaborate? (or reference a mailing list discussion) that gives some
> guidance?
> 
> is the telemetry lib a replacement for metrics? the component we have now
> relies on the non-telemetry functions exported from metrics but does not
> use the telemetry functions.
> 
> also, i notice that the meson.build for telemetry lib has an include path
> that references rte_metrics but does not appear to actually include any of
> the headers from rte_metrics (vestigial? missed in previous cleanup perhaps?)

I think Bruce and Ciara will better explain than me
the intent of the telemetry lib and the compatibility path with the metrics lib.

Re: [dpdk-dev] [PATCH v2] metrics/windows: build rte_metrics library

[Bruce Richardson]

On Tue, Jan 19, 2021 at 10:52:03PM +0100, Thomas Monjalon wrote:
> 19/01/2021 22:31, Tyler Retzlaff:
> > On Sun, Jan 17, 2021 at 11:19:55PM +0100, Thomas Monjalon wrote:
> > > 
> > > Not sure it makes sense without the new telemetry feature.
> > > Please focus on telemetry lib instead of half-enabling
> > > the old metrics lib.
> > > 
> > 
> > can you elaborate? (or reference a mailing list discussion) that gives some
> > guidance?
> > 
> > is the telemetry lib a replacement for metrics? the component we have now
> > relies on the non-telemetry functions exported from metrics but does not
> > use the telemetry functions.
> > 
> > also, i notice that the meson.build for telemetry lib has an include path
> > that references rte_metrics but does not appear to actually include any of
> > the headers from rte_metrics (vestigial? missed in previous cleanup perhaps?)
> 
> I think Bruce and Ciara will better explain than me
> the intent of the telemetry lib and the compatibility path with the metrics lib.
> 

The include path addition for metrics to the telemetry library does indeed
look like it was just missed being removed, since I can compile things up
successfully with it removed.

With regards to interaction between telemetry library and metrics library,
they are complementary but not replacements for each other. The metrics
library provides support for tracking metrics, mostly on a per-port basis.
The original telemetry library implementation was based on top of the
metrics library and supported reporting out data from that library. More
recent versions of the telemetry library have reworked that support to
allow the reporting of arbitrary telemetry data, not just from the metrics
library, but the old interface is still supported.

/Bruce

Re: [dpdk-dev] [PATCH v2] metrics/windows: build rte_metrics library

[Thomas Monjalon]

20/01/2021 11:37, Bruce Richardson:
> On Tue, Jan 19, 2021 at 10:52:03PM +0100, Thomas Monjalon wrote:
> > 19/01/2021 22:31, Tyler Retzlaff:
> > > On Sun, Jan 17, 2021 at 11:19:55PM +0100, Thomas Monjalon wrote:
> > > > 
> > > > Not sure it makes sense without the new telemetry feature.
> > > > Please focus on telemetry lib instead of half-enabling
> > > > the old metrics lib.
> > > > 
> > > 
> > > can you elaborate? (or reference a mailing list discussion) that gives some
> > > guidance?
> > > 
> > > is the telemetry lib a replacement for metrics? the component we have now
> > > relies on the non-telemetry functions exported from metrics but does not
> > > use the telemetry functions.
> > > 
> > > also, i notice that the meson.build for telemetry lib has an include path
> > > that references rte_metrics but does not appear to actually include any of
> > > the headers from rte_metrics (vestigial? missed in previous cleanup perhaps?)
> > 
> > I think Bruce and Ciara will better explain than me
> > the intent of the telemetry lib and the compatibility path with the metrics lib.
> > 
> 
> The include path addition for metrics to the telemetry library does indeed
> look like it was just missed being removed, since I can compile things up
> successfully with it removed.
> 
> With regards to interaction between telemetry library and metrics library,
> they are complementary but not replacements for each other. The metrics
> library provides support for tracking metrics, mostly on a per-port basis.
> The original telemetry library implementation was based on top of the
> metrics library and supported reporting out data from that library. More
> recent versions of the telemetry library have reworked that support to
> allow the reporting of arbitrary telemetry data, not just from the metrics
> library, but the old interface is still supported.

The question is what is available in rte_metrics which is not already
available with rte_telemetry only?
I think rte_metrics is used only for rte_bitratestats and rte_latencystats.
Can we use rte_telemetry for this and forget rte_metrics?

Re: [dpdk-dev] [PATCH v2] metrics/windows: build rte_metrics library

[Bruce Richardson]

On Wed, Jan 20, 2021 at 12:09:19PM +0100, Thomas Monjalon wrote:
> 20/01/2021 11:37, Bruce Richardson:
> > On Tue, Jan 19, 2021 at 10:52:03PM +0100, Thomas Monjalon wrote:
> > > 19/01/2021 22:31, Tyler Retzlaff:
> > > > On Sun, Jan 17, 2021 at 11:19:55PM +0100, Thomas Monjalon wrote:
> > > > > 
> > > > > Not sure it makes sense without the new telemetry feature.
> > > > > Please focus on telemetry lib instead of half-enabling
> > > > > the old metrics lib.
> > > > > 
> > > > 
> > > > can you elaborate? (or reference a mailing list discussion) that gives some
> > > > guidance?
> > > > 
> > > > is the telemetry lib a replacement for metrics? the component we have now
> > > > relies on the non-telemetry functions exported from metrics but does not
> > > > use the telemetry functions.
> > > > 
> > > > also, i notice that the meson.build for telemetry lib has an include path
> > > > that references rte_metrics but does not appear to actually include any of
> > > > the headers from rte_metrics (vestigial? missed in previous cleanup perhaps?)
> > > 
> > > I think Bruce and Ciara will better explain than me
> > > the intent of the telemetry lib and the compatibility path with the metrics lib.
> > > 
> > 
> > The include path addition for metrics to the telemetry library does indeed
> > look like it was just missed being removed, since I can compile things up
> > successfully with it removed.
> > 
> > With regards to interaction between telemetry library and metrics library,
> > they are complementary but not replacements for each other. The metrics
> > library provides support for tracking metrics, mostly on a per-port basis.
> > The original telemetry library implementation was based on top of the
> > metrics library and supported reporting out data from that library. More
> > recent versions of the telemetry library have reworked that support to
> > allow the reporting of arbitrary telemetry data, not just from the metrics
> > library, but the old interface is still supported.
> 
> The question is what is available in rte_metrics which is not already
> available with rte_telemetry only?
> I think rte_metrics is used only for rte_bitratestats and rte_latencystats.
> Can we use rte_telemetry for this and forget rte_metrics?
>
No, we can't.
In short, metrics is a data store. Telemetry is a data output mechanism.

Telemetry cannot directly replace metrics because it does not store any
data, it simply matches incoming requests against callbacks to retrieve and
export data. The actual management of the underlying data sources is the
responsibility of the library/app which has registered the callback.

Metrics, on the other hand, is a data storage library, and allows the
registration of groups of statictics which can be tracked and updated. It
does also provide some callbacks to allow the data to be output by
telemetry, but this functionality is not the primary goal of the lib. Both
bitratestats and latencystats use metrics to manage their stats data -
again any output provided is incidental to the primary purpose.

Regards,
/Bruce

Re: [dpdk-dev] [PATCH v2] metrics/windows: build rte_metrics library

[Thomas Monjalon]

20/01/2021 12:54, Bruce Richardson:
> On Wed, Jan 20, 2021 at 12:09:19PM +0100, Thomas Monjalon wrote:
> > 20/01/2021 11:37, Bruce Richardson:
> > > On Tue, Jan 19, 2021 at 10:52:03PM +0100, Thomas Monjalon wrote:
> > > > 19/01/2021 22:31, Tyler Retzlaff:
> > > > > On Sun, Jan 17, 2021 at 11:19:55PM +0100, Thomas Monjalon wrote:
> > > > > > 
> > > > > > Not sure it makes sense without the new telemetry feature.
> > > > > > Please focus on telemetry lib instead of half-enabling
> > > > > > the old metrics lib.
> > > > > > 
> > > > > 
> > > > > can you elaborate? (or reference a mailing list discussion) that gives some
> > > > > guidance?
> > > > > 
> > > > > is the telemetry lib a replacement for metrics? the component we have now
> > > > > relies on the non-telemetry functions exported from metrics but does not
> > > > > use the telemetry functions.
> > > > > 
> > > > > also, i notice that the meson.build for telemetry lib has an include path
> > > > > that references rte_metrics but does not appear to actually include any of
> > > > > the headers from rte_metrics (vestigial? missed in previous cleanup perhaps?)
> > > > 
> > > > I think Bruce and Ciara will better explain than me
> > > > the intent of the telemetry lib and the compatibility path with the metrics lib.
> > > > 
> > > 
> > > The include path addition for metrics to the telemetry library does indeed
> > > look like it was just missed being removed, since I can compile things up
> > > successfully with it removed.
> > > 
> > > With regards to interaction between telemetry library and metrics library,
> > > they are complementary but not replacements for each other. The metrics
> > > library provides support for tracking metrics, mostly on a per-port basis.
> > > The original telemetry library implementation was based on top of the
> > > metrics library and supported reporting out data from that library. More
> > > recent versions of the telemetry library have reworked that support to
> > > allow the reporting of arbitrary telemetry data, not just from the metrics
> > > library, but the old interface is still supported.
> > 
> > The question is what is available in rte_metrics which is not already
> > available with rte_telemetry only?
> > I think rte_metrics is used only for rte_bitratestats and rte_latencystats.
> > Can we use rte_telemetry for this and forget rte_metrics?
> >
> No, we can't.
> In short, metrics is a data store. Telemetry is a data output mechanism.
> 
> Telemetry cannot directly replace metrics because it does not store any
> data, it simply matches incoming requests against callbacks to retrieve and
> export data. The actual management of the underlying data sources is the
> responsibility of the library/app which has registered the callback.
> 
> Metrics, on the other hand, is a data storage library, and allows the
> registration of groups of statictics which can be tracked and updated. It
> does also provide some callbacks to allow the data to be output by
> telemetry, but this functionality is not the primary goal of the lib. Both
> bitratestats and latencystats use metrics to manage their stats data -
> again any output provided is incidental to the primary purpose.

When you say "we can't", you mean it is not a desirable addition?
I am fine with keeping the design as is, split in libs.

Re: [dpdk-dev] [PATCH v2] metrics/windows: build rte_metrics library

[Bruce Richardson]

On Wed, Jan 20, 2021 at 01:13:46PM +0100, Thomas Monjalon wrote:
> 20/01/2021 12:54, Bruce Richardson:
> > On Wed, Jan 20, 2021 at 12:09:19PM +0100, Thomas Monjalon wrote:
> > > 20/01/2021 11:37, Bruce Richardson:
> > > > On Tue, Jan 19, 2021 at 10:52:03PM +0100, Thomas Monjalon wrote:
> > > > > 19/01/2021 22:31, Tyler Retzlaff:
> > > > > > On Sun, Jan 17, 2021 at 11:19:55PM +0100, Thomas Monjalon wrote:
> > > > > > > 
> > > > > > > Not sure it makes sense without the new telemetry feature.
> > > > > > > Please focus on telemetry lib instead of half-enabling
> > > > > > > the old metrics lib.
> > > > > > > 
> > > > > > 
> > > > > > can you elaborate? (or reference a mailing list discussion) that gives some
> > > > > > guidance?
> > > > > > 
> > > > > > is the telemetry lib a replacement for metrics? the component we have now
> > > > > > relies on the non-telemetry functions exported from metrics but does not
> > > > > > use the telemetry functions.
> > > > > > 
> > > > > > also, i notice that the meson.build for telemetry lib has an include path
> > > > > > that references rte_metrics but does not appear to actually include any of
> > > > > > the headers from rte_metrics (vestigial? missed in previous cleanup perhaps?)
> > > > > 
> > > > > I think Bruce and Ciara will better explain than me
> > > > > the intent of the telemetry lib and the compatibility path with the metrics lib.
> > > > > 
> > > > 
> > > > The include path addition for metrics to the telemetry library does indeed
> > > > look like it was just missed being removed, since I can compile things up
> > > > successfully with it removed.
> > > > 
> > > > With regards to interaction between telemetry library and metrics library,
> > > > they are complementary but not replacements for each other. The metrics
> > > > library provides support for tracking metrics, mostly on a per-port basis.
> > > > The original telemetry library implementation was based on top of the
> > > > metrics library and supported reporting out data from that library. More
> > > > recent versions of the telemetry library have reworked that support to
> > > > allow the reporting of arbitrary telemetry data, not just from the metrics
> > > > library, but the old interface is still supported.
> > > 
> > > The question is what is available in rte_metrics which is not already
> > > available with rte_telemetry only?
> > > I think rte_metrics is used only for rte_bitratestats and rte_latencystats.
> > > Can we use rte_telemetry for this and forget rte_metrics?
> > >
> > No, we can't.
> > In short, metrics is a data store. Telemetry is a data output mechanism.
> > 
> > Telemetry cannot directly replace metrics because it does not store any
> > data, it simply matches incoming requests against callbacks to retrieve and
> > export data. The actual management of the underlying data sources is the
> > responsibility of the library/app which has registered the callback.
> > 
> > Metrics, on the other hand, is a data storage library, and allows the
> > registration of groups of statictics which can be tracked and updated. It
> > does also provide some callbacks to allow the data to be output by
> > telemetry, but this functionality is not the primary goal of the lib. Both
> > bitratestats and latencystats use metrics to manage their stats data -
> > again any output provided is incidental to the primary purpose.
> 
> When you say "we can't", you mean it is not a desirable addition?
> I am fine with keeping the design as is, split in libs.
> 

Well, I primarily meant "we can't" as in the library is not designed to do
that right now, and so cannot do what is asked. For the second interpretation
of it being a desirable addition or not, I also agree that expanding the
scope of telemetry to also cover data management is not something we want.
Any rework of metrics and associated library should be kept separate from
telemetry, and allow each lib to try and focus on just doing one thing.
So +1 for keeping things split.

/Bruce