Re: [PATCH 2/2] ethdev: fix race on ports for telemetry commands

[David Marchand <david.marchand@redhat.com>]

On Wed, Oct 2, 2024 at 6:27 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Wed, Oct 02, 2024 at 05:57:08PM +0200, David Marchand wrote:
> > While invoking telemetry commands (which may happen at any time,
> > out of control of the application), an application thread may
> > concurrently add/remove ports.
> > The telemetry callbacks may then access partially
> > initialised/uninitialised ethdev data.
> >
> > Reuse the ethdev lock that protects port allocation/destruction.
> >
> > Fixes: c190daedb9b1 ("ethdev: add telemetry callbacks")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
> >  lib/ethdev/rte_ethdev_telemetry.c | 93 +++++++++++++++++++------------
> >  1 file changed, 56 insertions(+), 37 deletions(-)
> >
> > diff --git a/lib/ethdev/rte_ethdev_telemetry.c b/lib/ethdev/rte_ethdev_telemetry.c
> > index 8031a58595..7f9c924209 100644
> > --- a/lib/ethdev/rte_ethdev_telemetry.c
> > +++ b/lib/ethdev/rte_ethdev_telemetry.c
> > @@ -6,6 +6,7 @@
> >  #include <stdlib.h>
> >
> >  #include <rte_kvargs.h>
> > +#include <rte_spinlock.h>
> >  #include <rte_telemetry.h>
> >
> >  #include "rte_ethdev.h"
> > @@ -1403,43 +1404,61 @@ eth_dev_handle_port_tm_node_caps(const char *cmd __rte_unused,
> >       return ret;
> >  }
> >
> > +#define ETHDEV_TELEMETRY_HANDLERS \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/list", eth_dev_handle_port_list, \
> > +             "Returns list of available ethdev ports. Takes no parameters") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/stats", eth_dev_handle_port_stats, \
> > +             "Returns the common stats for a port. Parameters: int port_id") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/xstats", eth_dev_handle_port_xstats, \
> > +             "Returns the extended stats for a port. Parameters: int port_id,hide_zero=true|false(Optional for indicates hide zero xstats)") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/dump_priv", eth_dev_handle_port_dump_priv, \
> > +             "Returns dump private information for a port. Parameters: int port_id") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/link_status", eth_dev_handle_port_link_status, \
> > +             "Returns the link status for a port. Parameters: int port_id") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/info", eth_dev_handle_port_info, \
> > +             "Returns the device info for a port. Parameters: int port_id") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/module_eeprom", eth_dev_handle_port_module_eeprom, \
> > +             "Returns module EEPROM info with SFF specs. Parameters: int port_id") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/macs", eth_dev_handle_port_macs, \
> > +             "Returns the MAC addresses for a port. Parameters: int port_id") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/flow_ctrl", eth_dev_handle_port_flow_ctrl, \
> > +             "Returns flow ctrl info for a port. Parameters: int port_id") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/rx_queue", eth_dev_handle_port_rxq, \
> > +             "Returns Rx queue info for a port. Parameters: int port_id, int queue_id (Optional if only one queue)") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/tx_queue", eth_dev_handle_port_txq, \
> > +             "Returns Tx queue info for a port. Parameters: int port_id, int queue_id (Optional if only one queue)") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/dcb", eth_dev_handle_port_dcb, \
> > +             "Returns DCB info for a port. Parameters: int port_id") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/rss_info", eth_dev_handle_port_rss_info, \
> > +             "Returns RSS info for a port. Parameters: int port_id") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/fec", eth_dev_handle_port_fec, \
> > +             "Returns FEC info for a port. Parameters: int port_id") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/vlan", eth_dev_handle_port_vlan, \
> > +             "Returns VLAN info for a port. Parameters: int port_id") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/tm_capability", eth_dev_handle_port_tm_caps, \
> > +             "Returns TM Capabilities info for a port. Parameters: int port_id") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/tm_level_capability", eth_dev_handle_port_tm_level_caps, \
> > +             "Returns TM Level Capabilities info for a port. Parameters: int port_id, int level_id (see tm_capability for the max)") \
> > +     ETHDEV_TELEMETRY_HANDLER("/ethdev/tm_node_capability", eth_dev_handle_port_tm_node_caps, \
> > +             "Returns TM Node Capabilities info for a port. Parameters: int port_id, int node_id (see tm_capability for the max)")
> > +
> > +#define ETHDEV_TELEMETRY_HANDLER(command, func, usage) \
> > +static int func ## _locked(const char *cmd __rte_unused, const char *params, \
> > +     struct rte_tel_data *d) \
> > +{ \
> > +     int ret; \
> > +     rte_spinlock_lock(rte_mcfg_ethdev_get_lock()); \
> > +     ret = func(cmd, params, d); \
> > +     rte_spinlock_unlock(rte_mcfg_ethdev_get_lock()); \
> > +     return ret; \
> > +}
> > +ETHDEV_TELEMETRY_HANDLERS
> > +#undef ETHDEV_TELEMETRY_HANDLER
> > +
>
> Not really a massive fan of such use of macros in the code, since I think
> it makes things obscure for the casual reader. However, I see why this
> approach has been taken. I think the macro code needs some documentation
> explaining why this was done this way.

I can add comments explaining how this is for protecting accesses to port.

>
> >  RTE_INIT(ethdev_init_telemetry)
> >  {
> > -     rte_telemetry_register_cmd("/ethdev/list", eth_dev_handle_port_list,
> > -                     "Returns list of available ethdev ports. Takes no parameters");
> > -     rte_telemetry_register_cmd("/ethdev/stats", eth_dev_handle_port_stats,
> > -                     "Returns the common stats for a port. Parameters: int port_id");
> > -     rte_telemetry_register_cmd("/ethdev/xstats", eth_dev_handle_port_xstats,
> > -                     "Returns the extended stats for a port. Parameters: int port_id,hide_zero=true|false(Optional for indicates hide zero xstats)");
> > -     rte_telemetry_register_cmd("/ethdev/dump_priv", eth_dev_handle_port_dump_priv,
> > -                     "Returns dump private information for a port. Parameters: int port_id");
> > -     rte_telemetry_register_cmd("/ethdev/link_status",
> > -                     eth_dev_handle_port_link_status,
> > -                     "Returns the link status for a port. Parameters: int port_id");
> > -     rte_telemetry_register_cmd("/ethdev/info", eth_dev_handle_port_info,
> > -                     "Returns the device info for a port. Parameters: int port_id");
> > -     rte_telemetry_register_cmd("/ethdev/module_eeprom", eth_dev_handle_port_module_eeprom,
> > -                     "Returns module EEPROM info with SFF specs. Parameters: int port_id");
> > -     rte_telemetry_register_cmd("/ethdev/macs", eth_dev_handle_port_macs,
> > -                     "Returns the MAC addresses for a port. Parameters: int port_id");
> > -     rte_telemetry_register_cmd("/ethdev/flow_ctrl", eth_dev_handle_port_flow_ctrl,
> > -                     "Returns flow ctrl info for a port. Parameters: int port_id");
> > -     rte_telemetry_register_cmd("/ethdev/rx_queue", eth_dev_handle_port_rxq,
> > -                     "Returns Rx queue info for a port. Parameters: int port_id, int queue_id (Optional if only one queue)");
> > -     rte_telemetry_register_cmd("/ethdev/tx_queue", eth_dev_handle_port_txq,
> > -                     "Returns Tx queue info for a port. Parameters: int port_id, int queue_id (Optional if only one queue)");
> > -     rte_telemetry_register_cmd("/ethdev/dcb", eth_dev_handle_port_dcb,
> > -                     "Returns DCB info for a port. Parameters: int port_id");
> > -     rte_telemetry_register_cmd("/ethdev/rss_info", eth_dev_handle_port_rss_info,
> > -                     "Returns RSS info for a port. Parameters: int port_id");
> > -     rte_telemetry_register_cmd("/ethdev/fec", eth_dev_handle_port_fec,
> > -                     "Returns FEC info for a port. Parameters: int port_id");
> > -     rte_telemetry_register_cmd("/ethdev/vlan", eth_dev_handle_port_vlan,
> > -                     "Returns VLAN info for a port. Parameters: int port_id");
> > -     rte_telemetry_register_cmd("/ethdev/tm_capability", eth_dev_handle_port_tm_caps,
> > -                     "Returns TM Capabilities info for a port. Parameters: int port_id");
> > -     rte_telemetry_register_cmd("/ethdev/tm_level_capability", eth_dev_handle_port_tm_level_caps,
> > -                     "Returns TM Level Capabilities info for a port. Parameters: int port_id, int level_id (see tm_capability for the max)");
> > -     rte_telemetry_register_cmd("/ethdev/tm_node_capability", eth_dev_handle_port_tm_node_caps,
> > -                     "Returns TM Node Capabilities info for a port. Parameters: int port_id, int node_id (see tm_capability for the max)");
> > +#define ETHDEV_TELEMETRY_HANDLER(command, func, usage) \
> > +     rte_telemetry_register_cmd(command, func ## _locked, usage);
> > +     ETHDEV_TELEMETRY_HANDLERS
> > +#undef ETHDEV_TELEMETRY_HANDLER
> >  }
>
> An alternative to this macro-fu, is to just define a single ethdev
> telemetry function, and within that, take the lock and then dispatch to the
> appropriate subfunction based upon the actual command coming in. The
> dispatch may be slightly slower due to the additional text matching (only
> from byte 8 onwards, so very short strings), but I think the code could be
> a simpler in C rather than in macros, and the perf impact for telemetry is
> likely to be negligible, compared to the overhead of the socket I/O etc.

Hopefully, dispatching performance is not important here.

(skipping the byte 8 stuff) is your proposal something like:

static int
one_callback_to_rule_them_all(const char *cmd, const char *params,
struct rte_tel_data *d)
{
       telemetry_cb cb = NULL;
       int ret = -EINVAL;

       if (strcmp(cmd, "/ethdev/list") == 0) { cb = eth_dev_handle_port_list; }
       else if (strcmp(cmd, "/ethdev/stats") == 0) { cb =
eth_dev_handle_port_stats; }
       else if (strcmp(cmd, "/ethdev/xstats") == 0) { cb =
eth_dev_handle_port_xstats; }
...
       if (cb != NULL) {
               rte_spinlock_lock(rte_mcfg_ethdev_get_lock());
               ret = cb(cmd, params, d);
               rte_spinlock_unlock(rte_mcfg_ethdev_get_lock());
       }

       return ret;
}

RTE_INIT(ethdev_init_telemetry)
{

       rte_telemetry_register_cmd("/ethdev/list", one_callback_to_rule_them_all,
                       "Returns list of available ethdev ports. Takes
no parameters");
       rte_telemetry_register_cmd("/ethdev/stats",
one_callback_to_rule_them_all,
                       "Returns the common stats for a port.
Parameters: int port_id");
       rte_telemetry_register_cmd("/ethdev/xstats",
one_callback_to_rule_them_all,
                       "Returns the extended stats for a port.
Parameters: int port_id,hide_zero=true|false(Optional for indicates
hide zero xstats)");
...

Which I find inelegant and not that great for maintenance: having the
same info (especially strings that are only evaluated at runtime) in
two locations is a call to inadvertence bugs.

The macros I propose are a way to avoid splitting the callback
function and the command names.
Then addition of a handler is tied with a single declaration.

+       ETHDEV_TELEMETRY_HANDLER("/ethdev/newstuff", eth_dev_new_handler, \
+               "New shiny command. Takes no parameters") \


-- 
David Marchand