From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 72E7043215; Fri, 27 Oct 2023 13:01:44 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2D42D42831; Fri, 27 Oct 2023 13:01:35 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by mails.dpdk.org (Postfix) with ESMTP id 2194240F16 for ; Fri, 27 Oct 2023 13:01:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698404494; x=1729940494; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=cKoyzD6rwHZe2+y4k2RcrY4M6hWlBHfTuV4CDrL3/i0=; b=aB3dH9cqvayT3D/K6O03pzmHXgCnucA9S19VeZ5uDXKquwAxjpnvEVPi cSZlL1b+Dbk5xmrSJ62pzwaQBV8FGz/fLs71Ibhp4BX2pRC8y5oMm8Nlx LqCyIXE1OMCXYAko5tcw9AOV0RLdMnP5YYtA61pTYG9gG3vhpudoIBXEg reHD/UH0dtGuFdQ3sMkjBJ4FySJnyq27eQys7oSHJ63Xl1KxkPy6banYJ Ocyfqf3klj+p9ML/BOt04V1CMoPDlinYj1NZ/UhQf+TjhJe6k812ZNMWk qB+DxKxtIZMA90CmRVkcGaK6WIu8fH1eYYY5pyRIFAqFEsOCk+cx42zMf A==; X-IronPort-AV: E=McAfee;i="6600,9927,10875"; a="546680" X-IronPort-AV: E=Sophos;i="6.03,256,1694761200"; d="scan'208";a="546680" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2023 04:01:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10875"; a="883157839" X-IronPort-AV: E=Sophos;i="6.03,256,1694761200"; d="scan'208";a="883157839" Received: from silpixa00401385.ir.intel.com ([10.237.214.154]) by orsmga004.jf.intel.com with ESMTP; 27 Oct 2023 04:01:30 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: david.marchand@redhat.com, rjarry@redhat.com, Bruce Richardson Subject: [PATCH v7 2/9] buildtools: script to generate cmdline boilerplate Date: Fri, 27 Oct 2023 12:01:10 +0100 Message-Id: <20231027110117.70995-3-bruce.richardson@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231027110117.70995-1-bruce.richardson@intel.com> References: <20230802170052.955323-1-bruce.richardson@intel.com> <20231027110117.70995-1-bruce.richardson@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Provide a "dpdk-cmdline-gen.py" script for application developers to quickly generate the boilerplate code necessary for using the cmdline library. Example of use: The script takes an input file with a list of commands the user wants in the app, where the parameter variables are tagged with the type. For example: $ cat commands.list list add x y echo message add socket path quit When run through the script as "./dpdk-cmdline-gen.py commands.list", the output will be the contents of a header file with all the boilerplate necessary for a commandline instance with those commands. If the flag --stubs is passed, an output header filename must also be passed, in which case both a header file with the definitions and a C file with function stubs in it is written to disk. The separation is so that the header file can be rewritten at any future point to add more commands, while the C file can be kept as-is and extended by the user with any additional functions needed. Signed-off-by: Bruce Richardson --- buildtools/dpdk-cmdline-gen.py | 195 ++++++++++++++++++++++++++++++ buildtools/meson.build | 7 ++ doc/guides/prog_guide/cmdline.rst | 131 +++++++++++++++++++- 3 files changed, 332 insertions(+), 1 deletion(-) create mode 100755 buildtools/dpdk-cmdline-gen.py diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py new file mode 100755 index 0000000000..c208121363 --- /dev/null +++ b/buildtools/dpdk-cmdline-gen.py @@ -0,0 +1,195 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2023 Intel Corporation +# +""" +Script to automatically generate boilerplate for using DPDK cmdline library. +""" + +import argparse +import sys + +PARSE_FN_PARAMS = "void *parsed_result, struct cmdline *cl, void *data" +PARSE_FN_BODY = """ + /* TODO: command action */ + RTE_SET_USED(parsed_result); + RTE_SET_USED(cl); + RTE_SET_USED(data); +""" +NUMERIC_TYPES = [ + "UINT8", + "UINT16", + "UINT32", + "UINT64", + "INT8", + "INT16", + "INT32", + "INT64", +] + + +def process_command(lineno, tokens, comment): + """Generate the structures and definitions for a single command.""" + out = [] + cfile_out = [] + + if tokens[0].startswith("<"): + raise ValueError(f"Error line {lineno + 1}: command must start with a literal string") + + name_tokens = [] + for t in tokens: + if t.startswith("<"): + break + name_tokens.append(t) + name = "_".join(name_tokens) + + result_struct = [] + initializers = [] + token_list = [] + for t in tokens: + if t.startswith("<"): + t_type, t_name = t[1:].split(">") + t_val = "NULL" + else: + t_type = "STRING" + t_name = t + t_val = f'"{t}"' + + if t_type == "STRING": + result_struct.append(f"\tcmdline_fixed_string_t {t_name};") + initializers.append( + f"static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\n" + f"\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});" + ) + elif t_type in NUMERIC_TYPES: + result_struct.append(f"\t{t_type.lower()}_t {t_name};") + initializers.append( + f"static cmdline_parse_token_num_t cmd_{name}_{t_name}_tok =\n" + f"\tTOKEN_NUM_INITIALIZER(struct cmd_{name}_result, {t_name}, RTE_{t_type});" + ) + elif t_type in ["IP", "IP_ADDR", "IPADDR"]: + result_struct.append(f"\tcmdline_ipaddr_t {t_name};") + initializers.append( + f"cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok =\n" + f"\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {t_name});" + ) + else: + raise TypeError(f"Error line {lineno + 1}: unknown token type '{t_type}'") + token_list.append(f"cmd_{name}_{t_name}_tok") + + out.append(f'/* Auto-generated handling for command "{" ".join(tokens)}" */') + # output function prototype + func_sig = f"void\ncmd_{name}_parsed({PARSE_FN_PARAMS})" + out.append(f"extern {func_sig};\n") + # output result data structure + out.append(f"struct cmd_{name}_result {{\n" + "\n".join(result_struct) + "\n};\n") + # output the initializer tokens + out.append("\n".join(initializers) + "\n") + # output the instance structure + inst_elems = "\n".join([f"\t\t(void *)&{t}," for t in token_list]) + out.append( + f"""\ +static cmdline_parse_inst_t cmd_{name} = {{ +\t.f = cmd_{name}_parsed, +\t.data = NULL, +\t.help_str = "{comment}", +\t.tokens = {{ +{inst_elems} +\t\tNULL, +\t}} +}}; +""" + ) + # output function template if C file being written + cfile_out.append(f"{func_sig}\n{{{PARSE_FN_BODY}}}\n") + + # return the instance structure name + return (f"cmd_{name}", out, cfile_out) + + +def process_commands(infile, hfile, cfile, ctxname): + """Generate boilerplate output for a list of commands from infile.""" + instances = [] + + hfile.write( + f"""\ +/* File autogenerated by {sys.argv[0]} */ +#ifndef GENERATED_COMMANDS_H +#define GENERATED_COMMANDS_H +#include +#include +#include +#include +#include + +""" + ) + + for lineno, line in enumerate(infile.readlines()): + if line.lstrip().startswith("#"): + continue + if "#" not in line: + line = line + "#" # ensure split always works, even if no help text + tokens, comment = line.split("#", 1) + cmd_inst, h_out, c_out = process_command(lineno, tokens.strip().split(), comment.strip()) + hfile.write("\n".join(h_out)) + if cfile: + cfile.write("\n".join(c_out)) + instances.append(cmd_inst) + + inst_join_str = ",\n\t&" + hfile.write( + f""" +static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{ +\t&{inst_join_str.join(instances)}, +\tNULL +}}; + +#endif /* GENERATED_COMMANDS_H */ +""" + ) + + +def main(): + """Application main entry point.""" + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument( + "--stubs", + action="store_true", + help="Produce C file with empty function stubs for each command", + ) + ap.add_argument( + "--output-file", + "-o", + default="-", + help="Output header filename [default to stdout]", + ) + ap.add_argument( + "--context-name", + default="ctx", + help="Name given to the cmdline context variable in the output header [default=ctx]", + ) + ap.add_argument("infile", type=argparse.FileType("r"), help="File with list of commands") + args = ap.parse_args() + + if not args.stubs: + if args.output_file == "-": + process_commands(args.infile, sys.stdout, None, args.context_name) + else: + with open(args.output_file, "w") as hfile: + process_commands(args.infile, hfile, None, args.context_name) + else: + if not args.output_file.endswith(".h"): + ap.error( + "-o/--output-file: specify an output filename ending with .h when creating stubs" + ) + + cfilename = args.output_file[:-2] + ".c" + with open(args.output_file, "w") as hfile: + with open(cfilename, "w") as cfile: + cfile.write(f'#include "{args.output_file}"\n\n') + process_commands(args.infile, hfile, cfile, args.context_name) + + +if __name__ == "__main__": + main() diff --git a/buildtools/meson.build b/buildtools/meson.build index 948ac17dd2..72447b60a0 100644 --- a/buildtools/meson.build +++ b/buildtools/meson.build @@ -19,6 +19,13 @@ get_cpu_count_cmd = py3 + files('get-cpu-count.py') get_numa_count_cmd = py3 + files('get-numa-count.py') get_test_suites_cmd = py3 + files('get-test-suites.py') has_hugepages_cmd = py3 + files('has-hugepages.py') +cmdline_gen_cmd = py3 + files('dpdk-cmdline-gen.py') + +# install any build tools that end-users might want also +install_data([ + 'dpdk-cmdline-gen.py', + ], + install_dir: 'bin') # select library and object file format pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()] diff --git a/doc/guides/prog_guide/cmdline.rst b/doc/guides/prog_guide/cmdline.rst index 40f49a30cc..0b96b770e2 100644 --- a/doc/guides/prog_guide/cmdline.rst +++ b/doc/guides/prog_guide/cmdline.rst @@ -44,7 +44,136 @@ Adding a command-line instance to an application involves a number of coding ste 6. Within your main application code, create a new command-line instance passing in the context. -The next few subsections will cover each of these steps in more detail, +Many of these steps can be automated using the script ``dpdk-cmdline-gen.py`` installed by DPDK, +and found in the ``buildtools`` folder in the source tree. +This section covers adding a command-line using this script to generate the boiler plate, +while the following section, +`Worked Example of Adding Command-line to an Application`_ covers the steps to do so manually. + +Creating a Command List File +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``dpdk-cmdline-gen.py`` script takes as input a list of commands to be used by the application. +While these can be piped to it via standard input, using a list file is probably best. + +The format of the list file must be: + +* Comment lines start with '#' as first non-whitespace character + +* One command per line + +* Variable fields are prefixed by the type-name in angle-brackets, for example: + + * ``message`` + + * ``port_id`` + + * ``src_ip`` + +* The help text for a command is given in the form of a comment on the same line as the command + +An example list file, with a variety of (unrelated) commands, is shown below:: + + # example list file + list # show all entries + add x y # add x and y + echo message # print message to screen + add socket path # add unix socket with the given path + quit # close the application + +Running the Generator Script +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To generate the necessary definitions for a command-line, run ``dpdk-cmdline-gen.py`` passing the list file as parameter. +The script will output the generated C code to standard output, +the contents of which are in the form of a C header file. +Optionally, an output filename may be specified via the ``-o/--output-file`` argument. + +The generated content includes: + +* The result structure definitions for each command + +* The token initializers for each structure field + +* An "extern" function prototype for the callback for each command + +* A parse context for each command, including the per-command comments as help string + +* A command-line context array definition, suitable for passing to ``cmdline_new`` + +If so desired, the script can also output function stubs for the callback functions for each command. +This behaviour is triggered by passing the ``--stubs`` flag to the script. +In this case, an output file must be provided with a filename ending in ".h", +and the callback stubs will be written to an equivalent ".c" file. + +.. note:: + + The stubs are written to a separate file, + to allow continuous use of the script to regenerate the command-line header, + without overwriting any code the user has added to the callback functions. + This makes it easy to incrementally add new commands to an existing application. + +Providing the Function Callbacks +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As discussed above, the script output is a header file, containing structure definitions, +but the callback functions themselves obviously have to be provided by the user. +These callback functions must be provided as non-static functions in a C file, +and named ``cmd__parsed``. +The function prototypes can be seen in the generated output header. + +The "cmdname" part of the function name is built up by combining the non-variable initial tokens in the command. +So, given the commands in our worked example below: ``quit`` and ``show port stats ``, +the callback functions would be: + +.. code:: c + + void + cmd_quit_parsed(void *parsed_result, struct cmdline *cl, void *data) + { + ... + } + + void + cmd_show_port_stats_parsed(void *parsed_result, struct cmdline *cl, void *data) + { + ... + } + +These functions must be provided by the developer, but, as stated above, +stub functions may be generated by the script automatically using the ``--stubs`` parameter. + +The same "cmdname" stem is used in the naming of the generated structures too. +To get at the results structure for each command above, +the ``parsed_result`` parameter should be cast to ``struct cmd_quit_result`` +or ``struct cmd_show_port_stats_result`` respectively. + +Integrating with the Application +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To integrate the script output with the application, +we must ``#include`` the generated header into our applications C file, +and then have the command-line created via either ``cmdline_new`` or ``cmdline_stdin_new``. +The first parameter to the function call should be the context array in the generated header file, +``ctx`` by default. (Modifiable via script parameter). + +The callback functions may be in this same file, or in a separate one - +they just need to be available to the linker at build-time. + +Limitations of the Script Approach +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The script approach works for most commands that a user may wish to add to an application. +However, it does not support the full range of functions possible with the DPDK command-line library. +For example, +it is not possible using the script to multiplex multiple commands into a single callback function. +To use this functionality, the user should follow the instructions in the next section +`Worked Example of Adding Command-line to an Application`_ to manually configure a command-line instance. + +Worked Example of Adding Command-line to an Application +------------------------------------------------------- + +The next few subsections will cover each of the steps listed in `Adding Command-line to an Application`_ in more detail, working through an example to add two commands to a command-line instance. Those two commands will be: -- 2.39.2