RE: [PATCH v2 0/3] allow easier use of high lcore-ids

["Morten Brørup" <mb@smartsharesystems.com>]

> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Monday, 7 April 2025 12.41
> 
> On Mon, Apr 07, 2025 at 12:15:13PM +0200, Morten Brørup wrote:
> > > From: Bruce Richardson [mailto:bruce.richardson@intel.com] Sent:
> > > Monday, 7 April 2025 11.49
> > >
> > > On Mon, Apr 07, 2025 at 09:04:05AM +0200, David Marchand wrote:
> > > > Hello Bruce,
> > > >
> > > > On Tue, Apr 1, 2025 at 4:08 PM Bruce Richardson
> > > > <bruce.richardson@intel.com> wrote:
> > > > >
> > > > > On Mon, Mar 24, 2025 at 05:30:26PM +0000, Bruce Richardson
> wrote:
> > > > > > Traditionally, DPDK has had a direct mapping of internal
> lcore-
> > > ids, to
> > > > > > the actual core numbers in use. With higher core count
> servers
> > > becoming
> > > > > > more prevalent the issue becomes one of increasing memory
> > > footprint when
> > > > > > using such a scheme, due to the need to have all arrays
> > > dimensioned for
> > > > > > all cores on the system, whether or not those cores are in
> use by
> > > the
> > > > > > app.
> > > > > >
> > > > > > Therefore, the decision was made in the past to not expand
> the
> > > > > > build-time RTE_MAX_LCORE value beyond 128. Instead, it was
> > > recommended
> > > > > > that users use the "--lcores" EAL parameter to take the high-
> > > numbered
> > > > > > cores they wish to use and map them to lcore-ids within the 0
> -
> > > 128
> > > > > > range. While this works, this is a little clunky as it means
> that
> > > > > > instead of just passing, for example, "-l 130-139", the user
> must
> > > > > > instead pass "--lcores 0@130,1@131,2@132,3@133,...."
> > > > > >
> > > > > > This patchset attempts to simplify the situation by adding a
> new
> > > flag to
> > > > > > do this mapping automatically. To use cores 130-139 and map
> them
> > > to ids
> > > > > > 0-9 internally, the EAL args now become: "-l 130-139 --map-
> lcore-
> > > ids",
> > > > > > or using the shorter "-M" version of the flag: "-Ml 130-139".
> > > > > >
> > > > > > Adding this new parameter required some rework of the
> existing
> > > arg
> > > > > > parsing code, because in current DPDK the args are parsed and
> > > checked in
> > > > > > the order they appear on the commandline. This means that
> using
> > > the
> > > > > > example above, the core parameter 130-139 will be rejected
> > > immediately
> > > > > > before the "map-lcore-ids" parameter is seen. To work around
> > > this, the
> > > > > > core (and service core) parameters are not parsed when seen,
> > > instead
> > > > > > they are only saved off and parsed after all arguments are
> > > parsed. The
> > > > > > "-l" and "-c" parameters are converted into "--lcores"
> arguments,
> > > so all
> > > > > > assigning of lcore ids is done there in all cases.
> > > > > >
> > > > > > RFC->v2: * converted printf to DEBUG log * added "-M" as
> shorter
> > > > > > version of flag * added documentation * renamed internal API
> that
> > > > > > was changed to avoid any potential
> > > hidden
> > > > > >   runtime issues.
> > > > > >
> > > > > > Bruce Richardson (3): eal: centralize core parameter parsing
> eal:
> > > > > > convert core masks and lists to core sets eal: allow
> automatic
> > > > > > mapping of high lcore ids
> > > > > >
> > > > > Ping for review.
> > > > >
> > > > > At a high level, does this feature seem useful to users?
> > > >
> > > > This seems useful, though I am not I would touch the existing
> > > options.
> > > > I would have gone with a simple -L option (taking the same kind
> of
> > > > input than -l but with new behavior), and not combine a flag with
> > > > existing options.
> > > >
> > >
> > > That would be an easier patchset to do up. However, it would then
> mean
> > > that we have no less than 4 different ways to specify the cores to
> use:
> > > "- c", "-l", "-L", "--lcores" - and therefore need 4 different sets
> of
> > > parsing options for them, and more checks to ensure we have only
> one of
> > > the 4 specified in any run. That's why I chose the modifier option,
> and
> > > to try and consolidate the core setup a bit.
> > >
> > > However, if having a completely new option is preferred, I am happy
> > > enough to do up a different patchset for that.
> > >
> > > > I scanned through the series, not much to say.  Maybe add a unit
> test
> > > > for new cmdline option.
> > > >
> > > Sure. Once it's decided what approach (if any) to take, I'll do up
> a
> > > new patchset, taking into account any relevant feedback on this
> set.
> > >
> > > /Bruce
> >
> > Changing the EAL parameter parser to a "two pass parser" makes sense.
> I
> > think checking for existence of more than one lcore specification
> options
> > should suffice; we don't need to accept multiple lcore specification
> > options and check for conflicts.
> >
> > When remapping, do we need to support gaps in the "lcore" (logical
> cores)
> > array, e.g. for secondary processes, or can it be continuous from 0
> to
> > the number of specified lcores?
> >
> > And are new EAL parameters for this really beneficial?  Doesn't e.g.
> "-l
> > 0-9@130-139,100@140" suffice?
> >
> Actually, I believe "0-9@130-139"[1]  is not the same as
> "0@130,1@131,2@132,...". The latter affinities one thread to one core
> ten
> times over, while the former affinitizes 10 threads to 10 cores -
> leaving
> those threads free to move about within the 10 cores specified.

Interesting. The documentation [GSG] isn't clear to me about this; a example there could help clarify.

[GSG]: https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#lcore-related-options

If users are manually passing lcore parameters to the EAL, then I see why some sort of remapping shorthand is useful.
IMO, if the mappings are relatively exotic, it should be acceptable requiring an external script to build a long list of mapping parameters and then invoke the application with those script-generated EAL parameters.
This would reduce the scope to support relatively simple, common mappings.

Could we expand the --lcores syntax to support common mappings?

E.g. "0-9@130+" to do what I thought.
The lack of "()" treats the entries individually (not as a group), and the "+" indicates auto-increment.

A more advanced example:
"0-9@(130-131)+", meaning lcore 0 gets cpus 130-131, lcore 1 gets cpus 132-133, etc.

> 
> Just to confirm, I tweaked our helloworld example to print the cpu
> affinity
> of each core when printing.
> 
> ./build/examples/dpdk-helloworld --no-pci --lcores '(0-3)@(30-33)'
> EAL: Detected CPU lcores: 96
> EAL: Detected NUMA nodes: 2
> EAL: Detected static linkage of DPDK
> EAL: Multi-process socket /run/user/11304126/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> EAL: VFIO support initialized
> hello from core 1, with thread affinity for cores: 30 31 32 33
> hello from core 3, with thread affinity for cores: 30 31 32 33
> hello from core 2, with thread affinity for cores: 30 31 32 33
> hello from core 0, with thread affinity for cores: 30 31 32 33
> 
> ./build/examples/dpdk-helloworld --no-pci --lcores
> '0@30,1@31,2@32,3@33'
> EAL: Detected CPU lcores: 96
> EAL: Detected NUMA nodes: 2
> EAL: Detected static linkage of DPDK
> EAL: Multi-process socket /run/user/11304126/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> EAL: VFIO support initialized
> hello from core 1, with thread affinity for cores: 31 hello from core
> 3, with thread affinity for cores:
> hello from core 2, with thread affinity for cores: 32
> hello from core 0, with thread affinity for cores: 30
> 33
> 
> Regards,
> /Bruce
> 
> [1] This actually needs to be "(0-9)@(130-139)", and with "--lcores",
> not
> just "-l", there are actually different flags with different behaviours