Re: [PATCH v10 00/21] Simplify running with high-numbered CPUs

[Stephen Hemminger <stephen@networkplumber.org>]

On Wed,  8 Oct 2025 21:42:23 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:

> The ultimate of this patchset is to make it easier to run on systems
> with large numbers of cores, by simplifying the process of using core
> numbers >RTE_MAX_LCORE. The new EAL arg ``-remap-lcore-ids``, also
> shortened to ``-R``, is added to DPDK to support this.
> 
> However, in order to add this new flag easily, the first dozen or more
> patches rework the argument handling in EAL to simplify things, using
> the argparse library for argument handling.
> 
> When processing cmdline arguments in DPDK, we always do so with very
> little context. So, for example, when processing the "-l" flag, we have
> no idea whether there will be later a --proc-type=secondary flag. We
> have all sorts of post-arg-processing checks in place to try and catch
> these scenarios.
> 
> To improve this situation, this patchset tries to simplify the handling
> of argument processing, by explicitly doing an initial pass to collate
> all arguments into a structure. Thereafter, the actual arg parsing is
> done in a fixed order, meaning that e.g. when processing the
> --main-lcore flag, we have already processed the service core flags. We
> also can far quicker and easier check for conflicting options, since
> they can all be checked for NULL/non-NULL in the arg structure
> immediately after the struct has been populated.
> 
> An additional benefit of this work is that the argument parsing for EAL
> is much more centralised into common options and the options list file.
> This single list with ifdefs makes it clear to the viewer what options
> are common across OS's, vs what are unix-only or linux-only.
> 
> Once the cleanup and rework is done, adding the new options for
> remapping cores becomes a lot simpler, since we can very easily check
> for scenarios like multi-process and handle those appropriately.
> 
> 
> V10:
> * add patch 2 to fix func_reentrancy test bug reported mid-way through
>   the series.
> * added patches 7 & 8 to resolve arg parsing compat issues:
>    - support reordering args to have non-flag parameters at the end so
>      we can e.g. mix autotest names in with EAL args
>    - support having optional params with single-letter args without
>      having to use "=" syntax, i.e. -l4 rather than -l=4
> * fixed issues with the final patch when the autodetected range of cores
>   was >RTE_MAX_LCORE
> 
> V9: rebase to latest main. CI complains cannot apply v8 patches.
> 
> V8:
> * dropped the final two patches from the series, dropping the new -L
>   option in favour of the -R modifier.
> * reordered patch 11 to be with the other argparse patches (now patch 5)
> * added patch 12, which uses macros to initialize the args structure
>   from the arguments header file, avoiding potential issues when we add
>   new args.
> * simplified and consolidated lcore mask and core list parsing to always
>   work off cpusets rather than arrays of uint8
> * enhanced debug printouts to also work better with cpusets and handle
>   core values in those sets >= RTE_MAX_LCORE
> * for completeness, ensure the new -R option works for coremasks, and
>   for cases where no explicit core-list or coremask is specified.
> 
> V7:
> * expand the scope of the patchset beyond just cleanup to add in the
>   extra 3 patches for -L and --lcoreid-base option.
> 
> V6:
> * Rebase to apply cleanly on latest main
> 
> V5:
> * Resubmit of V4, which didn't get sent correctly, or picked up correctly
>   in patchwork.
> 
> V4:
> * Updated patch 5 to auto-generate the arg struct definition from the same
>   list of defines used to construct the argument list.
> 
> V3:
> * Added 3 new initial patches, one for minor build-system addition, and two
>   for functionality in argparse to allow the user-callback help function to
>   be maintained as we move to argparse.
> * Added doc updates in the first EAL patch adding the long options
> * Fixed ASAN issues by adding a patch to properly clean up EAL init - both
>   memory allocations and fixing the run-once flag
> * Put ifdefs around the linux-only or unix-only options in EAL patch 2
> * Updated args to handle numa-mem and numa-limit as equivalent socket-mem
>   and socket-limit
> 
> Bruce Richardson (21):
>   build: add define for the OS environment name
>   test/func_reentrancy: fix args to EAL init call
>   argparse: export function to print help text for object
>   argparse: allow user-override of help printing
>   argparse: add documentation on supported value types
>   argparse: add support for parsing core lists
>   argparse: allow optional flag reordering
>   argparse: support parameters to short options without "="
>   eal: add long options for each short option
>   eal: define the EAL parameters in argparse format
>   eal: gather EAL args before processing
>   eal: ensure proper cleanup on EAL init failure
>   eal: combine parameter validation checks
>   eal: simplify handling of conflicting cmdline options
>   eal: automatically init arg list options
>   eal: add internal fn for converting cpuset to string
>   eal: use common cpuset to string function
>   eal: introduce lcore remapping option for coremasks
>   eal: rework internal coremask parsing to use cpu sets
>   eal: allow lcore ID remapping with core lists
>   eal: allow lcore remapping with autodetected core affinity
> 
>  app/test/test_argparse.c                   |  695 ++++++++++
>  app/test/test_func_reentrancy.c            |    5 +-
>  config/meson.build                         |    1 +
>  doc/guides/contributing/design.rst         |    2 +-
>  doc/guides/linux_gsg/eal_args.include.rst  |   42 +-
>  doc/guides/prog_guide/argparse_lib.rst     |  138 +-
>  doc/guides/rel_notes/release_25_11.rst     |   18 +
>  drivers/event/dlb2/dlb2_priv.h             |    2 +-
>  drivers/event/dlb2/pf/base/dlb2_resource.c |   11 +-
>  lib/argparse/rte_argparse.c                |  181 ++-
>  lib/argparse/rte_argparse.h                |   29 +-
>  lib/eal/common/eal_common_lcore.c          |   13 +-
>  lib/eal/common/eal_common_memory.c         |    3 +-
>  lib/eal/common/eal_common_options.c        | 1439 +++++++++++---------
>  lib/eal/common/eal_common_thread.c         |   43 +-
>  lib/eal/common/eal_option_list.h           |   98 ++
>  lib/eal/common/eal_options.h               |  104 +-
>  lib/eal/common/eal_private.h               |   21 +
>  lib/eal/common/eal_thread.h                |   13 +-
>  lib/eal/freebsd/eal.c                      |  245 +---
>  lib/eal/include/rte_eal.h                  |   20 -
>  lib/eal/linux/eal.c                        |  470 +------
>  lib/eal/linux/eal_memory.c                 |    2 +-
>  lib/eal/meson.build                        |    2 +-
>  lib/eal/windows/eal.c                      |  156 +--
>  lib/meson.build                            |    1 +
>  26 files changed, 2148 insertions(+), 1606 deletions(-)
>  create mode 100644 lib/eal/common/eal_option_list.h
> 
> --
> 2.48.1
> 

There is one other thing about eal_init. The arguments in argv can not be const
right now, since there are cases where args are modified. Could we fix that.
Code like dpdk-dumpcap has to strdup() the args.