From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9C688431FC; Wed, 25 Oct 2023 15:04:16 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7DE26402D4; Wed, 25 Oct 2023 15:04:16 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 66382402D0 for ; Wed, 25 Oct 2023 15:04:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1698239054; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=83RFx0gZnVEnkbeGLdKg2oL71A4t7PEUcaRa7ID1MtI=; b=hN6HPEPFwP7nK65LTekpDaAtwLBB/pOvrcLS8/8q4fInz3N2aPDnNfBsG5zR7/pOKYs0MH ig3kVBHV+nhymDaXOS8ksHa7hk5en4GYWfTNfPAIwE8jPcAI5U0I05UDiI4JVi3Di/e+cQ MuSfmuE97emKsjPqKjXeipFT0Sk8bPA= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-91-Qr2fwCeGOb2hC3tMNGP9FQ-1; Wed, 25 Oct 2023 09:04:08 -0400 X-MC-Unique: Qr2fwCeGOb2hC3tMNGP9FQ-1 Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-4091c9bdb8eso3378555e9.1 for ; Wed, 25 Oct 2023 06:04:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698239046; x=1698843846; h=in-reply-to:references:content-transfer-encoding:mime-version:to :from:cc:subject:message-id:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hyg/MYMy2cX7K9rlje0bslJLVejv1Opv/jN+BnanHtw=; b=jGk40xkpVP2vQlvFV/3JhRPSCXNayBlinrQd4Y1zs9agJ153pqqx4sRfFDlbTKMpyR L6l7z6zHJCGt9lKVtTJclmG6PMgcmy9W54CWQrkVbguTXepVqTkUxgzt9L45i+EMKplM TstCoovN+8GH4V4NPuUwSrMZscEotePt+tFAU5zuX/XEOCSQqRBC87G+RWrGt2h6Ze8p KHcjI4/t0u3PUX3PINLVxjv/o3kvI+5OfWOlc1F7b4NtieOnJFQgcmJWaht6RVF7OF0k 08Ex2Mc3PlM8zYgA5aEMBjS4uLQLkg5SMfx8KR0uS/y6j+z/Fo0zJrcGz/eFAvXXXMRL okMw== X-Gm-Message-State: AOJu0YzQ5p7Iqzp2gtoC9qaH1H3tc8uWLdnWTMPqEl//Q+hX/lMvTUA5 68jMIqgA20TbzCJl8VHylrILF0gyY8wxZ7GiOdnEymZMF+LdiBJmqoYqHn4maMDcaFW4SFdb8Xm wfuH3XsJBcLY= X-Received: by 2002:a05:600c:4ec8:b0:402:e68f:8a4f with SMTP id g8-20020a05600c4ec800b00402e68f8a4fmr15238210wmq.0.1698239046627; Wed, 25 Oct 2023 06:04:06 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFD8q50lwhyOuuTlUqlHGqyBmpOHHgatnRnvLXy6sInCvuV3A97OQR65y5o3iBXgcTKzqghEQ== X-Received: by 2002:a05:600c:4ec8:b0:402:e68f:8a4f with SMTP id g8-20020a05600c4ec800b00402e68f8a4fmr15238167wmq.0.1698239046163; Wed, 25 Oct 2023 06:04:06 -0700 (PDT) Received: from localhost (2a01cb000f8b9700598da2e1679e8383.ipv6.abo.wanadoo.fr. [2a01:cb00:f8b:9700:598d:a2e1:679e:8383]) by smtp.gmail.com with ESMTPSA id m42-20020a05600c3b2a00b00407752bd834sm14889211wms.1.2023.10.25.06.04.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 25 Oct 2023 06:04:05 -0700 (PDT) Date: Wed, 25 Oct 2023 15:04:05 +0200 Message-Id: Subject: Re: [PATCH v5 2/9] buildtools: script to generate cmdline boilerplate Cc: From: "Robin Jarry" To: "Bruce Richardson" , Mime-Version: 1.0 X-Mailer: aerc/0.16.0-46-gd838396a542d References: <20230802170052.955323-1-bruce.richardson@intel.com> <20231017121318.146007-1-bruce.richardson@intel.com> <20231017121318.146007-3-bruce.richardson@intel.com> In-Reply-To: <20231017121318.146007-3-bruce.richardson@intel.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=UTF-8; format=Flowed Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Bruce Richardson, Oct 17, 2023 at 14:13: > Provide a "dpdk-cmdline-gen.py" script for application developers to > quickly generate the boilerplate code necessary for using the cmdline > library. > > Example of use: > The script takes an input file with a list of commands the user wants in > the app, where the parameter variables are tagged with the type. > For example: > > =09$ cat commands.list > =09list > =09add x y > =09echo message > =09add socket path > =09quit > > When run through the script as "./dpdk-cmdline-gen.py commands.list", > the output will be the contents of a header file with all the > boilerplate necessary for a commandline instance with those commands. > > If the flag --stubs is passed, an output header filename must also be > passed, in which case both a header file with the definitions and a C > file with function stubs in it is written to disk. The separation is so > that the header file can be rewritten at any future point to add more > commands, while the C file can be kept as-is and extended by the user > with any additional functions needed. > > Signed-off-by: Bruce Richardson > --- > buildtools/dpdk-cmdline-gen.py | 190 ++++++++++++++++++++++++++++++ > buildtools/meson.build | 7 ++ > doc/guides/prog_guide/cmdline.rst | 131 +++++++++++++++++++- > 3 files changed, 327 insertions(+), 1 deletion(-) > create mode 100755 buildtools/dpdk-cmdline-gen.py Hi Bruce, thanks for the respin! I have some small remarks inline. > diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen= .py > new file mode 100755 > index 0000000000..6cb7610de4 > --- /dev/null > +++ b/buildtools/dpdk-cmdline-gen.py > @@ -0,0 +1,190 @@ > +#!/usr/bin/env python3 > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright(c) 2023 Intel Corporation > +# > +""" > +Script to automatically generate boilerplate for using DPDK cmdline libr= ary. > +""" > + > +import argparse > +import sys > + > +PARSE_FN_PARAMS =3D "void *parsed_result, struct cmdline *cl, void *data= " > +PARSE_FN_BODY =3D """ > + /* TODO: command action */ > + RTE_SET_USED(parsed_result); > + RTE_SET_USED(cl); > + RTE_SET_USED(data); > +""" > +NUMERIC_TYPES =3D [ > + "UINT8", > + "UINT16", > + "UINT32", > + "UINT64", > + "INT8", > + "INT16", > + "INT32", > + "INT64", > +] > + > + > +def process_command(lineno, tokens, comment): > + """Generate the structures and definitions for a single command.""" > + out =3D [] > + cfile_out =3D [] > + > + if tokens[0].startswith("<"): > + raise ValueError(f"Error line {lineno + 1}: command must start w= ith a literal string") > + > + name_tokens =3D [] > + for t in tokens: > + if t.startswith("<"): > + break > + name_tokens.append(t) > + name =3D "_".join(name_tokens) > + > + result_struct =3D [] > + initializers =3D [] > + token_list =3D [] > + for t in tokens: > + if t.startswith("<"): > + t_type, t_name =3D t[1:].split(">") > + t_val =3D "NULL" > + else: > + t_type =3D "STRING" > + t_name =3D t > + t_val =3D f'"{t}"' > + > + if t_type =3D=3D "STRING": > + result_struct.append(f"\tcmdline_fixed_string_t {t_name};") > + initializers.append( > + f"static cmdline_parse_token_string_t cmd_{name}_{t_name= }_tok =3D\n" > + + f"\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result,= {t_name}, {t_val});" Since you are now using multiline strings in process_commands(), why not=20 use them everywhere? It would make the code more readable in my opinion and would avoid=20 inline f-string concatenation. > + ) > + elif t_type in NUMERIC_TYPES: > + result_struct.append(f"\t{t_type.lower()}_t {t_name};") > + initializers.append( > + f"static cmdline_parse_token_num_t cmd_{name}_{t_name}_t= ok =3D\n" > + + f"\tTOKEN_NUM_INITIALIZER(struct cmd_{name}_result, {t= _name}, RTE_{t_type});" > + ) > + elif t_type in ["IP", "IP_ADDR", "IPADDR"]: > + result_struct.append(f"\tcmdline_ipaddr_t {t_name};") > + initializers.append( > + f"cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok = =3D\n" > + + f"\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {= t_name});" > + ) > + else: > + raise TypeError(f"Error line {lineno + 1}: unknown token typ= e '{t_type}'") > + token_list.append(f"cmd_{name}_{t_name}_tok") > + > + out.append(f'/* Auto-generated handling for command "{" ".join(token= s)}" */') > + # output function prototype > + func_sig =3D f"void\ncmd_{name}_parsed({PARSE_FN_PARAMS})" > + out.append(f"extern {func_sig};\n") > + # output result data structure > + out.append(f"struct cmd_{name}_result {{\n" + "\n".join(result_struc= t) + "\n};\n") > + # output the initializer tokens > + out.append("\n".join(initializers) + "\n") > + # output the instance structure > + out.append( > + f"static cmdline_parse_inst_t cmd_{name} =3D {{\n" > + + f"\t.f =3D cmd_{name}_parsed,\n" > + + "\t.data =3D NULL,\n" > + + f'\t.help_str =3D "{comment}",\n' > + + "\t.tokens =3D {" Especially here :) > + ) > + for t in token_list: > + out.append(f"\t\t(void *)&{t},") > + out.append("\t\tNULL\n" + "\t}\n" + "};\n") > + # output function template if C file being written > + cfile_out.append(f"{func_sig}\n{{{PARSE_FN_BODY}}}\n") > + > + # return the instance structure name > + return (f"cmd_{name}", out, cfile_out) > + > + > +def process_commands(infile, hfile, cfile, ctxname): > + """Generate boilerplate output for a list of commands from infile.""= " > + instances =3D [] > + > + hfile.write( > + f"""/* File autogenerated by {sys.argv[0]} */ > +#ifndef GENERATED_COMMANDS_H > +#define GENERATED_COMMANDS_H > +#include > +#include > +#include > +#include > +#include > + > +""" > + ) > + > + for lineno, line in enumerate(infile.readlines()): > + if line.lstrip().startswith("#"): > + continue > + if "#" not in line: > + line =3D line + "#" # ensure split always works, even if no= help text > + tokens, comment =3D line.split("#", 1) > + cmd_inst, h_out, c_out =3D process_command(lineno, tokens.strip(= ).split(), comment.strip()) > + hfile.write("\n".join(h_out)) > + if cfile: > + cfile.write("\n".join(c_out)) > + instances.append(cmd_inst) > + > + inst_join_str =3D ",\n\t&" > + hfile.write( > + f""" > +static __rte_used cmdline_parse_ctx_t {ctxname}[] =3D {{ > +\t&{inst_join_str.join(instances)}, > +\tNULL > +}}; > + > +#endif /* GENERATED_COMMANDS_H */ > +""" By the way, you can put literal tabs in the multiline strings. That way=20 the indentation will look the same than in the generated C code. f""" static __rte_used cmdline_parse_ctx_t {ctxname}[] =3D {{ =09&{inst_join_str.join(instances)}, =09NULL }}; #endif /* GENERATED_COMMANDS_H */ """ > + ) > + > + > +def main(): > + """Application main entry point.""" > + ap =3D argparse.ArgumentParser(description=3D__doc__) > + ap.add_argument( > + "--stubs", > + action=3D"store_true", > + help=3D"Produce C file with empty function stubs for each comman= d", > + ) > + ap.add_argument( > + "--output-file", > + "-o", > + default=3D"-", > + help=3D"Output header filename [default to stdout]", > + ) > + ap.add_argument( > + "--context-name", > + default=3D"ctx", > + help=3D"Name given to the cmdline context variable in the output= header [default=3Dctx]", > + ) > + ap.add_argument("infile", type=3Dargparse.FileType("r"), help=3D"Fil= e with list of commands") > + args =3D ap.parse_args() > + > + if not args.stubs: > + if args.output_file =3D=3D "-": > + process_commands(args.infile, sys.stdout, None, args.context= _name) > + else: > + with open(args.output_file, "w") as hfile: > + process_commands(args.infile, hfile, None, args.context_= name) > + else: > + if not args.output_file.endswith(".h"): > + ap.error( > + "-o/--output-file: specify an output filename ending wit= h .h when creating stubs" > + ) > + > + cfilename =3D args.output_file[:-2] + ".c" > + with open(args.output_file, "w") as hfile: > + with open(cfilename, "w") as cfile: > + cfile.write(f'#include "{args.output_file}"\n\n') > + process_commands(args.infile, hfile, cfile, args.context= _name) > + > + > +if __name__ =3D=3D "__main__": > + main() --=20 Robin Jarry Principal Software Engineer Red Hat, Telco/NFV