Re: [PATCH v2] usertools: add telemetry exporter

["Anthony Harivel" <aharivel@redhat.com>]

Hi Robin,

I've tested your patch and this is all good for me. 
The errors are better handled. 

The most common one is when the socket telemetry is closed and you 
get:

2024-04-19 15:22:00 ERROR 192.168.122.116 GET /metrics HTTP/1.1: telemetry socket not available
Traceback (most recent call last):                            
  File "/usr/bin/dpdk-telemetry-exporter.py", line 312, in do_GET
    with TelemetrySocket(self.server.dpdk_socket_path) as sock: 
  File "/usr/bin/dpdk-telemetry-exporter.py", line 165, in __init__
    self.sock.connect(path)                                   
FileNotFoundError: [Errno 2] No such file or directory

You get the Traceback of Python which is a bit useless for the user but 
at least you have at the first line the root cause: 
"telemetry socket not available" which is IMO the most important.

Thanks for you patch !

Tested-by: Anthony Harivel <aharivel@redhat.com>

Regards,
Anthony

Robin Jarry, Apr 16, 2024 at 15:46:
> For now the telemetry socket is local to the machine running a DPDK
> application. Also, there is no official "schema" for the exposed
> metrics. Add a framework and a script to collect and expose these
> metrics to telemetry and observability agree gators such as Prometheus,
> Carbon or Influxdb. The exposed data must be done with end-users in
> mind, some DPDK terminology or internals may not make sense to everyone.
>
> The script only serves as an entry point and does not know anything
> about any specific metrics nor JSON data structures exposed in the
> telemetry socket.
>
> It uses dynamically loaded endpoint exporters which are basic python
> files that must implement two functions:
>
>  def info() -> dict[MetricName, MetricInfo]:
>      Mapping of metric names to their description and type.
>
>  def metrics(sock: TelemetrySocket) -> list[MetricValue]:
>      Request data from sock and return it as metric values. A metric
>      value is a 3-tuple: (name: str, value: any, labels: dict). Each
>      name must be present in info().
>
> The sock argument passed to metrics() has a single method:
>
>  def cmd(self, uri: str, arg: any = None) -> dict | list:
>      Request JSON data to the telemetry socket and parse it to python
>      values.
>
> The main script invokes endpoints and exports the data into an output
> format. For now, only two formats are implemented:
>
> * openmetrics/prometheus: text based format exported via a local HTTP
>   server.
> * carbon/graphite: binary (python pickle) format exported to a distant
>   carbon TCP server.
>
> As a starting point, 3 built-in endpoints are implemented:
>
> * counters: ethdev hardware counters
> * cpu: lcore usage
> * memory: overall memory usage
>
> The goal is to keep all built-in endpoints in the DPDK repository so
> that they can be updated along with the telemetry JSON data structures.
>
> Example output for the openmetrics:// format:
>
>  ~# dpdk-telemetry-exporter.py -o openmetrics://:9876 &
>  INFO using endpoint: counters (from .../telemetry-endpoints/counters.py)
>  INFO using endpoint: cpu (from .../telemetry-endpoints/cpu.py)
>  INFO using endpoint: memory (from .../telemetry-endpoints/memory.py)
>  INFO listening on port 9876
>  [1] 838829
>
>  ~$ curl http://127.0.0.1:9876/
>  # HELP dpdk_cpu_total_cycles Total number of CPU cycles.
>  # TYPE dpdk_cpu_total_cycles counter
>  # HELP dpdk_cpu_busy_cycles Number of busy CPU cycles.
>  # TYPE dpdk_cpu_busy_cycles counter
>  dpdk_cpu_total_cycles{cpu="73", numa="0"} 4353385274702980
>  dpdk_cpu_busy_cycles{cpu="73", numa="0"} 6215932860
>  dpdk_cpu_total_cycles{cpu="9", numa="0"} 4353385274745740
>  dpdk_cpu_busy_cycles{cpu="9", numa="0"} 6215932860
>  dpdk_cpu_total_cycles{cpu="8", numa="0"} 4353383451895540
>  dpdk_cpu_busy_cycles{cpu="8", numa="0"} 6171923160
>  dpdk_cpu_total_cycles{cpu="72", numa="0"} 4353385274817320
>  dpdk_cpu_busy_cycles{cpu="72", numa="0"} 6215932860
>  # HELP dpdk_memory_total_bytes The total size of reserved memory in bytes.
>  # TYPE dpdk_memory_total_bytes gauge
>  # HELP dpdk_memory_used_bytes The currently used memory in bytes.
>  # TYPE dpdk_memory_used_bytes gauge
>  dpdk_memory_total_bytes 1073741824
>  dpdk_memory_used_bytes 794197376
>
> Link: https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format
> Link: https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#text-format
> Link: https://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-pickle-protocol
> Link: https://github.com/influxdata/telegraf/tree/master/plugins/inputs/prometheus
> Signed-off-by: Robin Jarry <rjarry@redhat.com>
> ---
>
> Notes:
>     v2:
>     
>     * Refuse to run if no endpoints are enabled.
>     * Handle endpoint errors gracefully without failing the whole query.
>
>  usertools/dpdk-telemetry-exporter.py      | 405 ++++++++++++++++++++++
>  usertools/meson.build                     |   6 +
>  usertools/telemetry-endpoints/counters.py |  47 +++
>  usertools/telemetry-endpoints/cpu.py      |  29 ++
>  usertools/telemetry-endpoints/memory.py   |  37 ++
>  5 files changed, 524 insertions(+)
>  create mode 100755 usertools/dpdk-telemetry-exporter.py
>  create mode 100644 usertools/telemetry-endpoints/counters.py
>  create mode 100644 usertools/telemetry-endpoints/cpu.py
>  create mode 100644 usertools/telemetry-endpoints/memory.py
>
> diff --git a/usertools/dpdk-telemetry-exporter.py b/usertools/dpdk-telemetry-exporter.py
> new file mode 100755
> index 000000000000..f8d873ad856c
> --- /dev/null
> +++ b/usertools/dpdk-telemetry-exporter.py
> @@ -0,0 +1,405 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2023 Robin Jarry
> +
> +r'''
> +DPDK telemetry exporter.
> +
> +It uses dynamically loaded endpoint exporters which are basic python files that
> +must implement two functions:
> +
> +    def info() -> dict[MetricName, MetricInfo]:
> +        """
> +        Mapping of metric names to their description and type.
> +        """
> +
> +    def metrics(sock: TelemetrySocket) -> list[MetricValue]:
> +        """
> +        Request data from sock and return it as metric values. A metric value
> +        is a 3-tuple: (name: str, value: any, labels: dict). Each name must be
> +        present in info().
> +        """
> +
> +The sock argument passed to metrics() has a single method:
> +
> +    def cmd(self, uri, arg=None) -> dict | list:
> +        """
> +        Request JSON data to the telemetry socket and parse it to python
> +        values.
> +        """
> +
> +See existing endpoints for examples.
> +
> +The exporter supports multiple output formats:
> +
> +prometheus://ADDRESS:PORT
> +openmetrics://ADDRESS:PORT
> +  Expose the enabled endpoints via a local HTTP server listening on the
> +  specified address and port. GET requests on that server are served with
> +  text/plain responses in the prometheus/openmetrics format.
> +
> +  More details:
> +  https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format
> +
> +carbon://ADDRESS:PORT
> +graphite://ADDRESS:PORT
> +  Export all enabled endpoints to the specified TCP ADDRESS:PORT in the pickle
> +  carbon format.
> +
> +  More details:
> +  https://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-pickle-protocol
> +'''
> +
> +import argparse
> +import importlib.util
> +import json
> +import logging
> +import os
> +import pickle
> +import re
> +import socket
> +import struct
> +import sys
> +import time
> +import typing
> +from http import HTTPStatus, server
> +from urllib.parse import urlparse
> +
> +LOG = logging.getLogger(__name__)
> +# Use local endpoints path only when running from source
> +LOCAL = os.path.join(os.path.dirname(__file__), "telemetry-endpoints")
> +DEFAULT_LOAD_PATHS = []
> +if os.path.isdir(LOCAL):
> +    DEFAULT_LOAD_PATHS.append(LOCAL)
> +DEFAULT_LOAD_PATHS += [
> +    "/usr/local/share/dpdk/telemetry-endpoints",
> +    "/usr/share/dpdk/telemetry-endpoints",
> +]
> +DEFAULT_OUTPUT = "openmetrics://:9876"
> +
> +
> +def main():
> +    logging.basicConfig(
> +        stream=sys.stdout,
> +        level=logging.INFO,
> +        format="%(asctime)s %(levelname)s %(message)s",
> +        datefmt="%Y-%m-%d %H:%M:%S",
> +    )
> +    parser = argparse.ArgumentParser(
> +        description=__doc__,
> +        formatter_class=argparse.RawDescriptionHelpFormatter,
> +    )
> +    parser.add_argument(
> +        "-o",
> +        "--output",
> +        metavar="FORMAT://PARAMETERS",
> +        default=urlparse(DEFAULT_OUTPUT),
> +        type=urlparse,
> +        help=f"""
> +        Output format (default: "{DEFAULT_OUTPUT}"). Depending on the format,
> +        URL elements have different meanings. By default, the exporter starts a
> +        local HTTP server on port 9876 that serves requests in the
> +        prometheus/openmetrics plain text format.
> +        """,
> +    )
> +    parser.add_argument(
> +        "-p",
> +        "--load-path",
> +        dest="load_paths",
> +        type=lambda v: v.split(os.pathsep),
> +        default=DEFAULT_LOAD_PATHS,
> +        help=f"""
> +        The list of paths from which to disvover endpoints.
> +        (default: "{os.pathsep.join(DEFAULT_LOAD_PATHS)}").
> +        """,
> +    )
> +    parser.add_argument(
> +        "-e",
> +        "--endpoint",
> +        dest="endpoints",
> +        metavar="ENDPOINT",
> +        action="append",
> +        help="""
> +        Telemetry endpoint to export (by default, all discovered endpoints are
> +        enabled). This option can be specified more than once.
> +        """,
> +    )
> +    parser.add_argument(
> +        "-l",
> +        "--list",
> +        action="store_true",
> +        help="""
> +        Only list detected endpoints and exit.
> +        """,
> +    )
> +    parser.add_argument(
> +        "-s",
> +        "--socket-path",
> +        default="/run/dpdk/rte/dpdk_telemetry.v2",
> +        help="""
> +        The DPDK telemetry socket path (default: "%(default)s").
> +        """,
> +    )
> +    args = parser.parse_args()
> +    output = OUTPUT_FORMATS.get(args.output.scheme)
> +    if output is None:
> +        parser.error(f"unsupported output format: {args.output.scheme}://")
> +
> +    try:
> +        endpoints = load_endpoints(args.load_paths, args.endpoints)
> +        if args.list:
> +            return
> +    except Exception as e:
> +        parser.error(str(e))
> +
> +    output(args, endpoints)
> +
> +
> +class TelemetrySocket:
> +    """
> +    Abstraction of the DPDK telemetry socket.
> +    """
> +
> +    def __init__(self, path: str):
> +        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET)
> +        self.sock.connect(path)
> +        data = json.loads(self.sock.recv(1024).decode())
> +        self.max_output_len = data["max_output_len"]
> +
> +    def cmd(
> +        self, uri: str, arg: typing.Any = None
> +    ) -> typing.Optional[typing.Union[dict, list]]:
> +        """
> +        Request JSON data to the telemetry socket and parse it to python
> +        values.
> +        """
> +        if arg is not None:
> +            u = f"{uri},{arg}"
> +        else:
> +            u = uri
> +        self.sock.send(u.encode("utf-8"))
> +        data = self.sock.recv(self.max_output_len)
> +        return json.loads(data.decode("utf-8"))[uri]
> +
> +    def __enter__(self):
> +        return self
> +
> +    def __exit__(self, *args, **kwargs):
> +        self.sock.close()
> +
> +
> +MetricDescription = str
> +MetricType = str
> +MetricName = str
> +MetricLabels = typing.Dict[str, typing.Any]
> +MetricInfo = typing.Tuple[MetricDescription, MetricType]
> +MetricValue = typing.Tuple[MetricName, typing.Any, MetricLabels]
> +
> +
> +class TelemetryEndpoint:
> +    """
> +    Placeholder class only used for typing annotations.
> +    """
> +
> +    @staticmethod
> +    def info() -> typing.Dict[MetricName, MetricInfo]:
> +        """
> +        Mapping of metric names to their description and type.
> +        """
> +        raise NotImplementedError()
> +
> +    @staticmethod
> +    def metrics(sock: TelemetrySocket) -> typing.List[MetricValue]:
> +        """
> +        Request data from sock and return it as metric values. Each metric
> +        name must be present in info().
> +        """
> +        raise NotImplementedError()
> +
> +
> +def load_endpoints(
> +    paths: typing.List[str], names: typing.List[str]
> +) -> typing.List[TelemetryEndpoint]:
> +    """
> +    Load selected telemetry endpoints from the specified paths.
> +    """
> +
> +    endpoints = {}
> +    dwb = sys.dont_write_bytecode
> +    sys.dont_write_bytecode = True  # never generate .pyc files for endpoints
> +
> +    for p in paths:
> +        if not os.path.isdir(p):
> +            continue
> +        for fname in os.listdir(p):
> +            f = os.path.join(p, fname)
> +            if os.path.isdir(f):
> +                continue
> +            try:
> +                name, _ = os.path.splitext(fname)
> +                if names is not None and name not in names:
> +                    # not selected by user
> +                    continue
> +                if name in endpoints:
> +                    # endpoint with same name already loaded
> +                    continue
> +                spec = importlib.util.spec_from_file_location(name, f)
> +                module = importlib.util.module_from_spec(spec)
> +                spec.loader.exec_module(module)
> +                endpoints[name] = module
> +            except Exception:
> +                LOG.exception("parsing endpoint: %s", f)
> +
> +    if not endpoints:
> +        raise Exception("no telemetry endpoints detected/selected")
> +
> +    sys.dont_write_bytecode = dwb
> +
> +    modules = []
> +    info = {}
> +    for name, module in sorted(endpoints.items()):
> +        LOG.info("using endpoint: %s (from %s)", name, module.__file__)
> +        try:
> +            for metric, (description, type_) in module.info().items():
> +                info[(name, metric)] = (description, type_)
> +            modules.append(module)
> +        except Exception:
> +            LOG.exception("getting endpoint info: %s", name)
> +    return modules
> +
> +
> +def serve_openmetrics(
> +    args: argparse.Namespace, endpoints: typing.List[TelemetryEndpoint]
> +):
> +    """
> +    Start an HTTP server and serve requests in the openmetrics/prometheus
> +    format.
> +    """
> +    listen = (args.output.hostname or "", int(args.output.port or 80))
> +    with server.HTTPServer(listen, OpenmetricsHandler) as httpd:
> +        httpd.dpdk_socket_path = args.socket_path
> +        httpd.telemetry_endpoints = endpoints
> +        LOG.info("listening on port %s", httpd.server_port)
> +        try:
> +            httpd.serve_forever()
> +        except KeyboardInterrupt:
> +            LOG.info("shutting down")
> +
> +
> +class OpenmetricsHandler(server.BaseHTTPRequestHandler):
> +    """
> +    Basic HTTP handler that returns prometheus/openmetrics formatted responses.
> +    """
> +
> +    CONTENT_TYPE = "text/plain; version=0.0.4; charset=utf-8"
> +
> +    def escape(self, value: typing.Any) -> str:
> +        """
> +        Escape a metric label value.
> +        """
> +        value = str(value)
> +        value = value.replace('"', '\\"')
> +        value = value.replace("\\", "\\\\")
> +        return value.replace("\n", "\\n")
> +
> +    def do_GET(self):
> +        """
> +        Called uppon GET requests.
> +        """
> +        try:
> +            lines = []
> +            metrics_names = set()
> +            with TelemetrySocket(self.server.dpdk_socket_path) as sock:
> +                for e in self.server.telemetry_endpoints:
> +                    info = e.info()
> +                    metrics_lines = []
> +                    try:
> +                        metrics = e.metrics(sock)
> +                    except Exception:
> +                        LOG.exception("%s: metrics collection failed", e.__name__)
> +                        continue
> +                    for name, value, labels in metrics:
> +                        fullname = re.sub(r"\W", "_", f"dpdk_{e.__name__}_{name}")
> +                        labels = ", ".join(
> +                            f'{k}="{self.escape(v)}"' for k, v in labels.items()
> +                        )
> +                        if labels:
> +                            labels = f"{{{labels}}}"
> +                        metrics_lines.append(f"{fullname}{labels} {value}")
> +                        if fullname not in metrics_names:
> +                            metrics_names.add(fullname)
> +                            desc, metric_type = info[name]
> +                            lines += [
> +                                f"# HELP {fullname} {desc}",
> +                                f"# TYPE {fullname} {metric_type}",
> +                            ]
> +                    lines += metrics_lines
> +            if not lines:
> +                self.send_error(HTTPStatus.INTERNAL_SERVER_ERROR)
> +                LOG.error(
> +                    "%s %s: no metrics collected",
> +                    self.address_string(),
> +                    self.requestline,
> +                )
> +            body = "\n".join(lines).encode("utf-8") + b"\n"
> +            self.send_response(HTTPStatus.OK)
> +            self.send_header("Content-Type", self.CONTENT_TYPE)
> +            self.send_header("Content-Length", str(len(body)))
> +            self.end_headers()
> +            self.wfile.write(body)
> +            LOG.info("%s %s", self.address_string(), self.requestline)
> +
> +        except (FileNotFoundError, ConnectionRefusedError):
> +            self.send_error(HTTPStatus.SERVICE_UNAVAILABLE)
> +            LOG.exception(
> +                "%s %s: telemetry socket not available",
> +                self.address_string(),
> +                self.requestline,
> +            )
> +        except Exception:
> +            self.send_error(HTTPStatus.INTERNAL_SERVER_ERROR)
> +            LOG.exception("%s %s", self.address_string(), self.requestline)
> +
> +    def log_message(self, fmt, *args):
> +        pass  # disable built-in logger
> +
> +
> +def export_carbon(args: argparse.Namespace, endpoints: typing.List[TelemetryEndpoint]):
> +    """
> +    Collect all metrics and export them to a carbon server in the pickle format.
> +    """
> +    addr = (args.output.hostname or "", int(args.output.port or 80))
> +    with TelemetrySocket(args.socket_path) as dpdk:
> +        with socket.socket() as carbon:
> +            carbon.connect(addr)
> +            all_metrics = []
> +            for e in endpoints:
> +                try:
> +                    metrics = e.metrics(dpdk)
> +                except Exception:
> +                    LOG.exception("%s: metrics collection failed", e.__name__)
> +                    continue
> +                for name, value, labels in metrics:
> +                    fullname = re.sub(r"\W", ".", f"dpdk.{e.__name__}.{name}")
> +                    for key, val in labels.items():
> +                        val = str(val).replace(";", "")
> +                        fullname += f";{key}={val}"
> +                    all_metrics.append((fullname, (time.time(), value)))
> +            if not all_metrics:
> +                raise Exception("no metrics collected")
> +            payload = pickle.dumps(all_metrics, protocol=2)
> +            header = struct.pack("!L", len(payload))
> +            buf = header + payload
> +            carbon.sendall(buf)
> +
> +
> +OUTPUT_FORMATS = {
> +    "openmetrics": serve_openmetrics,
> +    "prometheus": serve_openmetrics,
> +    "carbon": export_carbon,
> +    "graphite": export_carbon,
> +}
> +
> +
> +if __name__ == "__main__":
> +    main()
> diff --git a/usertools/meson.build b/usertools/meson.build
> index 740b4832f36d..eb48e2f4403f 100644
> --- a/usertools/meson.build
> +++ b/usertools/meson.build
> @@ -11,5 +11,11 @@ install_data([
>              'dpdk-telemetry.py',
>              'dpdk-hugepages.py',
>              'dpdk-rss-flows.py',
> +            'dpdk-telemetry-exporter.py',
>          ],
>          install_dir: 'bin')
> +
> +install_subdir(
> +        'telemetry-endpoints',
> +        install_dir: 'share/dpdk',
> +        strip_directory: false)
> diff --git a/usertools/telemetry-endpoints/counters.py b/usertools/telemetry-endpoints/counters.py
> new file mode 100644
> index 000000000000..e17cffb43b2c
> --- /dev/null
> +++ b/usertools/telemetry-endpoints/counters.py
> @@ -0,0 +1,47 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2023 Robin Jarry
> +
> +RX_PACKETS = "rx_packets"
> +RX_BYTES = "rx_bytes"
> +RX_MISSED = "rx_missed"
> +RX_NOMBUF = "rx_nombuf"
> +RX_ERRORS = "rx_errors"
> +TX_PACKETS = "tx_packets"
> +TX_BYTES = "tx_bytes"
> +TX_ERRORS = "tx_errors"
> +
> +
> +def info() -> "dict[Name, tuple[Description, Type]]":
> +    return {
> +        RX_PACKETS: ("Number of successfully received packets.", "counter"),
> +        RX_BYTES: ("Number of successfully received bytes.", "counter"),
> +        RX_MISSED: (
> +            "Number of packets dropped by the HW because Rx queues are full.",
> +            "counter",
> +        ),
> +        RX_NOMBUF: ("Number of Rx mbuf allocation failures.", "counter"),
> +        RX_ERRORS: ("Number of erroneous received packets.", "counter"),
> +        TX_PACKETS: ("Number of successfully transmitted packets.", "counter"),
> +        TX_BYTES: ("Number of successfully transmitted bytes.", "counter"),
> +        TX_ERRORS: ("Number of packet transmission failures.", "counter"),
> +    }
> +
> +
> +def metrics(sock: "TelemetrySocket") -> "list[tuple[Name, Value, Labels]]":
> +    out = []
> +    for port_id in sock.cmd("/ethdev/list"):
> +        port = sock.cmd("/ethdev/info", port_id)
> +        stats = sock.cmd("/ethdev/stats", port_id)
> +        labels = {"port": port["name"]}
> +        out += [
> +            (RX_PACKETS, stats["ipackets"], labels),
> +            (RX_PACKETS, stats["ipackets"], labels),
> +            (RX_BYTES, stats["ibytes"], labels),
> +            (RX_MISSED, stats["imissed"], labels),
> +            (RX_NOMBUF, stats["rx_nombuf"], labels),
> +            (RX_ERRORS, stats["ierrors"], labels),
> +            (TX_PACKETS, stats["opackets"], labels),
> +            (TX_BYTES, stats["obytes"], labels),
> +            (TX_ERRORS, stats["oerrors"], labels),
> +        ]
> +    return out
> diff --git a/usertools/telemetry-endpoints/cpu.py b/usertools/telemetry-endpoints/cpu.py
> new file mode 100644
> index 000000000000..d38d8d6e2558
> --- /dev/null
> +++ b/usertools/telemetry-endpoints/cpu.py
> @@ -0,0 +1,29 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2023 Robin Jarry
> +
> +CPU_TOTAL = "total_cycles"
> +CPU_BUSY = "busy_cycles"
> +
> +
> +def info() -> "dict[Name, tuple[Description, Type]]":
> +    return {
> +        CPU_TOTAL: ("Total number of CPU cycles.", "counter"),
> +        CPU_BUSY: ("Number of busy CPU cycles.", "counter"),
> +    }
> +
> +
> +def metrics(sock: "TelemetrySocket") -> "list[tuple[Name, Value, Labels]]":
> +    out = []
> +    for lcore_id in sock.cmd("/eal/lcore/list"):
> +        lcore = sock.cmd("/eal/lcore/info", lcore_id)
> +        cpu = ",".join(str(c) for c in lcore.get("cpuset", []))
> +        total = lcore.get("total_cycles")
> +        busy = lcore.get("busy_cycles", 0)
> +        if not (cpu and total):
> +            continue
> +        labels = {"cpu": cpu, "numa": lcore.get("socket", 0)}
> +        out += [
> +            (CPU_TOTAL, total, labels),
> +            (CPU_BUSY, busy, labels),
> +        ]
> +    return out
> diff --git a/usertools/telemetry-endpoints/memory.py b/usertools/telemetry-endpoints/memory.py
> new file mode 100644
> index 000000000000..32cce1e59382
> --- /dev/null
> +++ b/usertools/telemetry-endpoints/memory.py
> @@ -0,0 +1,37 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2023 Robin Jarry
> +
> +MEM_TOTAL = "total_bytes"
> +MEM_USED = "used_bytes"
> +
> +
> +def info() -> "dict[Name, tuple[Description, Type]]":
> +    return {
> +        MEM_TOTAL: ("The total size of reserved memory in bytes.", "gauge"),
> +        MEM_USED: ("The currently used memory in bytes.", "gauge"),
> +    }
> +
> +
> +def metrics(sock: "TelemetrySocket") -> "list[tuple[Name, Value, Labels]]":
> +    zones = {}
> +    used = 0
> +    for zone in sock.cmd("/eal/memzone_list") or []:
> +        z = sock.cmd("/eal/memzone_info", zone)
> +        start = int(z["Hugepage_base"], 16)
> +        end = start + (z["Hugepage_size"] * z["Hugepage_used"])
> +        used += z["Length"]
> +        for s, e in list(zones.items()):
> +            if s < start < e < end:
> +                zones[s] = end
> +                break
> +            if start < s < end < e:
> +                del zones[s]
> +                zones[start] = e
> +                break
> +        else:
> +            zones[start] = end
> +
> +    return [
> +        (MEM_TOTAL, sum(end - start for (start, end) in zones.items()), {}),
> +        (MEM_USED, max(0, used), {}),
> +    ]
> -- 
> 2.44.0