Re: [PATCH v2 0/3] allow easier use of high lcore-ids

[Bruce Richardson <bruce.richardson@intel.com>]

On Mon, Apr 07, 2025 at 01:32:59PM +0200, Morten Brørup wrote:
> > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Sent: Monday, 7 April 2025 12.41
> > 
> > On Mon, Apr 07, 2025 at 12:15:13PM +0200, Morten Brørup wrote:
> > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com] Sent:
> > > > Monday, 7 April 2025 11.49
> > > >
> > > > On Mon, Apr 07, 2025 at 09:04:05AM +0200, David Marchand wrote:
> > > > > Hello Bruce,
> > > > >
> > > > > On Tue, Apr 1, 2025 at 4:08 PM Bruce Richardson
> > > > > <bruce.richardson@intel.com> wrote:
> > > > > >
> > > > > > On Mon, Mar 24, 2025 at 05:30:26PM +0000, Bruce Richardson
> > wrote:
> > > > > > > Traditionally, DPDK has had a direct mapping of internal
> > lcore-
> > > > ids, to
> > > > > > > the actual core numbers in use. With higher core count
> > servers
> > > > becoming
> > > > > > > more prevalent the issue becomes one of increasing memory
> > > > footprint when
> > > > > > > using such a scheme, due to the need to have all arrays
> > > > dimensioned for
> > > > > > > all cores on the system, whether or not those cores are in
> > use by
> > > > the
> > > > > > > app.
> > > > > > >
> > > > > > > Therefore, the decision was made in the past to not expand
> > the
> > > > > > > build-time RTE_MAX_LCORE value beyond 128. Instead, it was
> > > > recommended
> > > > > > > that users use the "--lcores" EAL parameter to take the high-
> > > > numbered
> > > > > > > cores they wish to use and map them to lcore-ids within the 0
> > -
> > > > 128
> > > > > > > range. While this works, this is a little clunky as it means
> > that
> > > > > > > instead of just passing, for example, "-l 130-139", the user
> > must
> > > > > > > instead pass "--lcores 0@130,1@131,2@132,3@133,...."
> > > > > > >
> > > > > > > This patchset attempts to simplify the situation by adding a
> > new
> > > > flag to
> > > > > > > do this mapping automatically. To use cores 130-139 and map
> > them
> > > > to ids
> > > > > > > 0-9 internally, the EAL args now become: "-l 130-139 --map-
> > lcore-
> > > > ids",
> > > > > > > or using the shorter "-M" version of the flag: "-Ml 130-139".
> > > > > > >
> > > > > > > Adding this new parameter required some rework of the
> > existing
> > > > arg
> > > > > > > parsing code, because in current DPDK the args are parsed and
> > > > checked in
> > > > > > > the order they appear on the commandline. This means that
> > using
> > > > the
> > > > > > > example above, the core parameter 130-139 will be rejected
> > > > immediately
> > > > > > > before the "map-lcore-ids" parameter is seen. To work around
> > > > this, the
> > > > > > > core (and service core) parameters are not parsed when seen,
> > > > instead
> > > > > > > they are only saved off and parsed after all arguments are
> > > > parsed. The
> > > > > > > "-l" and "-c" parameters are converted into "--lcores"
> > arguments,
> > > > so all
> > > > > > > assigning of lcore ids is done there in all cases.
> > > > > > >
> > > > > > > RFC->v2: * converted printf to DEBUG log * added "-M" as
> > shorter
> > > > > > > version of flag * added documentation * renamed internal API
> > that
> > > > > > > was changed to avoid any potential
> > > > hidden
> > > > > > >   runtime issues.
> > > > > > >
> > > > > > > Bruce Richardson (3): eal: centralize core parameter parsing
> > eal:
> > > > > > > convert core masks and lists to core sets eal: allow
> > automatic
> > > > > > > mapping of high lcore ids
> > > > > > >
> > > > > > Ping for review.
> > > > > >
> > > > > > At a high level, does this feature seem useful to users?
> > > > >
> > > > > This seems useful, though I am not I would touch the existing
> > > > options.
> > > > > I would have gone with a simple -L option (taking the same kind
> > of
> > > > > input than -l but with new behavior), and not combine a flag with
> > > > > existing options.
> > > > >
> > > >
> > > > That would be an easier patchset to do up. However, it would then
> > mean
> > > > that we have no less than 4 different ways to specify the cores to
> > use:
> > > > "- c", "-l", "-L", "--lcores" - and therefore need 4 different sets
> > of
> > > > parsing options for them, and more checks to ensure we have only
> > one of
> > > > the 4 specified in any run. That's why I chose the modifier option,
> > and
> > > > to try and consolidate the core setup a bit.
> > > >
> > > > However, if having a completely new option is preferred, I am happy
> > > > enough to do up a different patchset for that.
> > > >
> > > > > I scanned through the series, not much to say.  Maybe add a unit
> > test
> > > > > for new cmdline option.
> > > > >
> > > > Sure. Once it's decided what approach (if any) to take, I'll do up
> > a
> > > > new patchset, taking into account any relevant feedback on this
> > set.
> > > >
> > > > /Bruce
> > >
> > > Changing the EAL parameter parser to a "two pass parser" makes sense.
> > I
> > > think checking for existence of more than one lcore specification
> > options
> > > should suffice; we don't need to accept multiple lcore specification
> > > options and check for conflicts.
> > >
> > > When remapping, do we need to support gaps in the "lcore" (logical
> > cores)
> > > array, e.g. for secondary processes, or can it be continuous from 0
> > to
> > > the number of specified lcores?
> > >
> > > And are new EAL parameters for this really beneficial?  Doesn't e.g.
> > "-l
> > > 0-9@130-139,100@140" suffice?
> > >
> > Actually, I believe "0-9@130-139"[1]  is not the same as
> > "0@130,1@131,2@132,...". The latter affinities one thread to one core
> > ten
> > times over, while the former affinitizes 10 threads to 10 cores -
> > leaving
> > those threads free to move about within the 10 cores specified.
> 
> Interesting. The documentation [GSG] isn't clear to me about this; a example there could help clarify.
> 
> [GSG]: https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#lcore-related-options
> 

Yep, agreed.

> If users are manually passing lcore parameters to the EAL, then I see why some sort of remapping shorthand is useful.
> IMO, if the mappings are relatively exotic, it should be acceptable requiring an external script to build a long list of mapping parameters and then invoke the application with those script-generated EAL parameters.
> This would reduce the scope to support relatively simple, common mappings.
> 
> Could we expand the --lcores syntax to support common mappings?
> 
> E.g. "0-9@130+" to do what I thought.
> The lack of "()" treats the entries individually (not as a group), and the "+" indicates auto-increment.
> 
> A more advanced example:
> "0-9@(130-131)+", meaning lcore 0 gets cpus 130-131, lcore 1 gets cpus 132-133, etc.
> 

My issues with the above syntax idea is:
* I think it's overly complicating the lcores parameter adding in the
  support for the "+" symbol - especially in the advanced case example you
  provide. I worry about code maintainability here.
* More significantly for me, I think it's also getting things backwards in
  that it is focusing more on the lcore ids visible to the app, rather than
  the physical cores to be used. For the example above of 0-9@130+, what I
  would expect is that the user is mainly thinking about the cores he wants
  to use 130-139, which are then to be mapped to lower ids. If we had the
  syntax reversed where the physical cores were first, I'd say it would
  make more sense, e.g. 130-139@0+
* finally, as I found out last month, there are systems on which the cores
  are spread across numa-nodes on odd/even boundaries, so to have an app
  running on socket 0, you need to give the core ids as a list i.e.
  0,2,4,6, and cannot use ranges. [This also reenforces the point above
  too, where again it's the internal ids we need to generalize, not the
  physical cpus]

My thinking on the matter, and I'm happy to be corrected if I'm wrong here,
is that end-users are unlikely to significantly care what the actual lcore
ids are internally in the program, so long as they work and are unique.
What does matter is the physical cpus on which the code is to run.
Therefore, my approach to this is to find the simplest possible solution
whereby the user can just provide a list of cores and tell EAL to just map
them to reasonable values. For the "reasonable" values, I would imagine
that for the vast majority of cases starting "0" is what is wanted. For
anything beyond that, we already have the existing --lcores syntax to be
used.

My 2c.

/Bruce