* [RFC PATCH 0/3] allow easier use of high lcore-ids
@ 2025-03-13 11:38 Bruce Richardson
2025-03-13 11:38 ` [RFC PATCH 1/3] eal: centralize core parameter parsing Bruce Richardson
` (4 more replies)
0 siblings, 5 replies; 31+ messages in thread
From: Bruce Richardson @ 2025-03-13 11:38 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson
Traditionally, DPDK has had a direct mapping of internal lcore-ids, to
the actual core numbers in use. With higher core count servers becoming
more prevalent the issue becomes one of increasing memory footprint when
using such a scheme, due to the need to have all arrays dimensioned for
all cores on the system, whether or not those cores are in use by the
app.
Therefore, the decision was made in the past to not expand the
build-time RTE_MAX_LCORE value beyond 128. Instead, it was recommended
that users use the "--lcores" EAL parameter to take the high-numbered
cores they wish to use and map them to lcore-ids within the 0 - 128
range. While this works, this is a little clunky as it means that
instead of just passing, for example, "-l 130-139", the user must
instead pass "--lcores 0@130,1@131,2@132,3@133,...."
This patchset attempts to simplify the situation by adding a new flag to
do this mapping automatically. To use cores 130-139 and map them to ids
0-9 internally, the EAL args now become: "-l 130-139 --map-lcore-ids".
Adding this new parameter required some rework of the existing arg
parsing code, because in current DPDK the args are parsed and checked in
the order they appear on the commandline. This means that using the
example above, the core parameter 130-139 will be rejected immediately
before the "map-lcore-ids" parameter is seen. To work around this, the
core (and service core) parameters are not parsed when seen, instead
they are only saved off and parsed after all arguments are parsed. The
"-l" and "-c" parameters are converted into "--lcores" arguments, so all
assigning of lcore ids is done there in all cases.
TODOs and requests-for-feedback:
- is there a suitable single-letter abbreviation we could use for this
mapping option. For example, if using "x" it would mean we could use
e.g. "-xl 130-139" for core options.
- still printfs in the code. This is to make it clearer for anyone
testing what is happening.
- doc updates - will be done if feedback is positive to move from RFC to
proper patch.
Bruce Richardson (3):
eal: centralize core parameter parsing
eal: convert core masks and lists to core sets
eal: allow automatic mapping of high lcore ids
drivers/event/dlb2/dlb2_priv.h | 2 -
drivers/event/dlb2/pf/base/dlb2_resource.c | 48 +--
lib/eal/common/eal_common_options.c | 338 +++++++--------------
lib/eal/common/eal_options.h | 2 +
lib/eal/include/rte_eal.h | 2 +-
5 files changed, 126 insertions(+), 266 deletions(-)
--
2.43.0
^ permalink raw reply [flat|nested] 31+ messages in thread
* [RFC PATCH 1/3] eal: centralize core parameter parsing
2025-03-13 11:38 [RFC PATCH 0/3] allow easier use of high lcore-ids Bruce Richardson
@ 2025-03-13 11:38 ` Bruce Richardson
2025-03-13 11:38 ` [RFC PATCH 2/3] eal: convert core masks and lists to core sets Bruce Richardson
` (3 subsequent siblings)
4 siblings, 0 replies; 31+ messages in thread
From: Bruce Richardson @ 2025-03-13 11:38 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson
Rather than parsing the lcore parameters as they are encountered, just
save off the lcore parameters and then parse them at the end. This
allows better knowledge of what parameters are available or not when
parsing.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
lib/eal/common/eal_common_options.c | 183 +++++++++++++---------------
1 file changed, 84 insertions(+), 99 deletions(-)
diff --git a/lib/eal/common/eal_common_options.c b/lib/eal/common/eal_common_options.c
index 79db9a47dd..55c49a923f 100644
--- a/lib/eal/common/eal_common_options.c
+++ b/lib/eal/common/eal_common_options.c
@@ -151,7 +151,13 @@ TAILQ_HEAD_INITIALIZER(devopt_list);
static int main_lcore_parsed;
static int mem_parsed;
-static int core_parsed;
+static struct lcore_options {
+ const char *core_mask_opt;
+ const char *core_list_opt;
+ const char *core_map_opt;
+ const char *service_mask_opt;
+ const char *service_list_opt;
+} lcore_options = {0};
/* Allow the application to print its usage message too if set */
static rte_usage_hook_t rte_application_usage_hook;
@@ -674,7 +680,7 @@ eal_parse_service_coremask(const char *coremask)
if (count == 0)
return -1;
- if (core_parsed && taken_lcore_count != count) {
+ if (taken_lcore_count != count) {
EAL_LOG(WARNING,
"Not all service cores are in the coremask. "
"Please ensure -c or -l includes service cores");
@@ -684,17 +690,6 @@ eal_parse_service_coremask(const char *coremask)
return 0;
}
-static int
-eal_service_cores_parsed(void)
-{
- int idx;
- for (idx = 0; idx < RTE_MAX_LCORE; idx++) {
- if (lcore_config[idx].core_role == ROLE_SERVICE)
- return 1;
- }
- return 0;
-}
-
static int
update_lcore_config(int *cores)
{
@@ -903,7 +898,7 @@ eal_parse_service_corelist(const char *corelist)
if (count == 0)
return -1;
- if (core_parsed && taken_lcore_count != count) {
+ if (taken_lcore_count != count) {
EAL_LOG(WARNING,
"Not all service cores were in the coremask. "
"Please ensure -c or -l includes service cores");
@@ -1673,83 +1668,20 @@ eal_parse_common_option(int opt, const char *optarg,
a_used = 1;
break;
/* coremask */
- case 'c': {
- int lcore_indexes[RTE_MAX_LCORE];
-
- if (eal_service_cores_parsed())
- EAL_LOG(WARNING,
- "Service cores parsed before dataplane cores. Please ensure -c is before -s or -S");
- if (rte_eal_parse_coremask(optarg, lcore_indexes) < 0) {
- EAL_LOG(ERR, "invalid coremask syntax");
- return -1;
- }
- if (update_lcore_config(lcore_indexes) < 0) {
- char *available = available_cores();
-
- EAL_LOG(ERR,
- "invalid coremask, please check specified cores are part of %s",
- available);
- free(available);
- return -1;
- }
-
- if (core_parsed) {
- EAL_LOG(ERR, "Option -c is ignored, because (%s) is set!",
- (core_parsed == LCORE_OPT_LST) ? "-l" :
- (core_parsed == LCORE_OPT_MAP) ? "--lcores" :
- "-c");
- return -1;
- }
-
- core_parsed = LCORE_OPT_MSK;
+ case 'c':
+ lcore_options.core_mask_opt = optarg;
break;
- }
/* corelist */
- case 'l': {
- int lcore_indexes[RTE_MAX_LCORE];
-
- if (eal_service_cores_parsed())
- EAL_LOG(WARNING,
- "Service cores parsed before dataplane cores. Please ensure -l is before -s or -S");
-
- if (eal_parse_corelist(optarg, lcore_indexes) < 0) {
- EAL_LOG(ERR, "invalid core list syntax");
- return -1;
- }
- if (update_lcore_config(lcore_indexes) < 0) {
- char *available = available_cores();
-
- EAL_LOG(ERR,
- "invalid core list, please check specified cores are part of %s",
- available);
- free(available);
- return -1;
- }
-
- if (core_parsed) {
- EAL_LOG(ERR, "Option -l is ignored, because (%s) is set!",
- (core_parsed == LCORE_OPT_MSK) ? "-c" :
- (core_parsed == LCORE_OPT_MAP) ? "--lcores" :
- "-l");
- return -1;
- }
-
- core_parsed = LCORE_OPT_LST;
+ case 'l':
+ lcore_options.core_list_opt = optarg;
break;
- }
/* service coremask */
case 's':
- if (eal_parse_service_coremask(optarg) < 0) {
- EAL_LOG(ERR, "invalid service coremask");
- return -1;
- }
+ lcore_options.service_mask_opt = optarg;
break;
/* service corelist */
case 'S':
- if (eal_parse_service_corelist(optarg) < 0) {
- EAL_LOG(ERR, "invalid service core list");
- return -1;
- }
+ lcore_options.service_list_opt = optarg;
break;
/* size of memory */
case 'm':
@@ -1918,21 +1850,7 @@ eal_parse_common_option(int opt, const char *optarg,
#endif /* !RTE_EXEC_ENV_WINDOWS */
case OPT_LCORES_NUM:
- if (eal_parse_lcores(optarg) < 0) {
- EAL_LOG(ERR, "invalid parameter for --"
- OPT_LCORES);
- return -1;
- }
-
- if (core_parsed) {
- EAL_LOG(ERR, "Option --lcores is ignored, because (%s) is set!",
- (core_parsed == LCORE_OPT_LST) ? "-l" :
- (core_parsed == LCORE_OPT_MSK) ? "-c" :
- "--lcores");
- return -1;
- }
-
- core_parsed = LCORE_OPT_MAP;
+ lcore_options.core_map_opt = optarg;
break;
case OPT_LEGACY_MEM_NUM:
conf->legacy_mem = 1;
@@ -1973,6 +1891,7 @@ eal_parse_common_option(int opt, const char *optarg,
}
+
return 0;
ba_conflict:
@@ -2046,8 +1965,74 @@ eal_adjust_config(struct internal_config *internal_cfg)
struct internal_config *internal_conf =
eal_get_internal_configuration();
- if (!core_parsed)
+ /* handle all the various core options, ignoring order of them.
+ * First check that multiple lcore options (coremask, corelist, coremap) are
+ * not used together. Then check that the service options (coremask, corelist)
+ * are not both set. In both cases error out if multiple options are set.
+ */
+ if ((lcore_options.core_mask_opt && lcore_options.core_list_opt) ||
+ (lcore_options.core_mask_opt && lcore_options.core_map_opt) ||
+ (lcore_options.core_list_opt && lcore_options.core_map_opt)) {
+ EAL_LOG(ERR, "Only one of -c, -l and --"OPT_LCORES" may be given at a time");
+ return -1;
+ }
+ if ((lcore_options.service_mask_opt && lcore_options.service_list_opt)) {
+ EAL_LOG(ERR, "Only one of -s and -S may be given at a time");
+ return -1;
+ }
+
+ if (lcore_options.core_mask_opt) {
+ int lcore_indexes[RTE_MAX_LCORE];
+
+ if (rte_eal_parse_coremask(lcore_options.core_mask_opt, lcore_indexes) < 0) {
+ EAL_LOG(ERR, "invalid coremask syntax");
+ return -1;
+ }
+ if (update_lcore_config(lcore_indexes) < 0) {
+ char *available = available_cores();
+
+ EAL_LOG(ERR,
+ "invalid coremask, please check specified cores are part of %s",
+ available);
+ free(available);
+ return -1;
+ }
+ } else if (lcore_options.core_list_opt) {
+ int lcore_indexes[RTE_MAX_LCORE];
+
+ if (eal_parse_corelist(lcore_options.core_list_opt, lcore_indexes) < 0) {
+ EAL_LOG(ERR, "invalid core list syntax");
+ return -1;
+ }
+ if (update_lcore_config(lcore_indexes) < 0) {
+ char *available = available_cores();
+
+ EAL_LOG(ERR,
+ "invalid core list, please check specified cores are part of %s",
+ available);
+ free(available);
+ return -1;
+ }
+ } else if (lcore_options.core_map_opt) {
+ if (eal_parse_lcores(lcore_options.core_map_opt) < 0) {
+ EAL_LOG(ERR, "invalid parameter for --" OPT_LCORES);
+ return -1;
+ }
+ } else {
eal_auto_detect_cores(cfg);
+ }
+
+ if (lcore_options.service_mask_opt) {
+ if (eal_parse_service_coremask(lcore_options.service_mask_opt) < 0) {
+ EAL_LOG(ERR, "invalid service coremask");
+ return -1;
+ }
+ } else if (lcore_options.service_list_opt) {
+ if (eal_parse_service_corelist(lcore_options.service_list_opt) < 0) {
+ EAL_LOG(ERR, "invalid service core list");
+ return -1;
+ }
+ }
if (internal_conf->process_type == RTE_PROC_AUTO)
internal_conf->process_type = eal_proc_type_detect();
--
2.43.0
^ permalink raw reply [flat|nested] 31+ messages in thread
* [RFC PATCH 2/3] eal: convert core masks and lists to core sets
2025-03-13 11:38 [RFC PATCH 0/3] allow easier use of high lcore-ids Bruce Richardson
2025-03-13 11:38 ` [RFC PATCH 1/3] eal: centralize core parameter parsing Bruce Richardson
@ 2025-03-13 11:38 ` Bruce Richardson
2025-03-13 11:38 ` [RFC PATCH 3/3] eal: allow automatic mapping of high lcore ids Bruce Richardson
` (2 subsequent siblings)
4 siblings, 0 replies; 31+ messages in thread
From: Bruce Richardson @ 2025-03-13 11:38 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson
Rather than directly parsing and working with the core mask and core
list parameters, convert them into core maps, and centralise all core
parsing there. This allows future work to adjust the mappings of cores
when generating that mapping parameter.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
NOTE: this does change the behaviour of the internal API rte_eal_parse
core_mask, used by dlb driver. However, since this is an internal API
changing behaviour should be allowed AFAIK.
---
drivers/event/dlb2/dlb2_priv.h | 2 -
drivers/event/dlb2/pf/base/dlb2_resource.c | 48 ++---
lib/eal/common/eal_common_options.c | 193 ++++-----------------
3 files changed, 53 insertions(+), 190 deletions(-)
diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
index 52da31ed31..63c0bea155 100644
--- a/drivers/event/dlb2/dlb2_priv.h
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -737,8 +737,6 @@ void dlb2_event_build_hcws(struct dlb2_port *qm_port,
uint8_t *sched_type,
uint8_t *queue_id);
-/* Extern functions */
-extern int rte_eal_parse_coremask(const char *coremask, int *cores);
/* Extern globals */
extern struct process_local_port_data dlb2_port[][DLB2_NUM_PORT_TYPES];
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 3004902118..c4bc248afc 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -922,49 +922,31 @@ dlb2_resource_probe(struct dlb2_hw *hw, const void *probe_args)
{
const struct dlb2_devargs *args = (const struct dlb2_devargs *)probe_args;
const char *mask = args ? args->producer_coremask : NULL;
- int cpu = 0, cnt = 0, cores[RTE_MAX_LCORE], i;
+ int cpu = 0;
if (args) {
mask = (const char *)args->producer_coremask;
}
- if (mask && rte_eal_parse_coremask(mask, cores)) {
- DLB2_HW_ERR(hw, ": Invalid producer coremask=%s\n", mask);
- return -1;
- }
-
+ cpu = rte_get_next_lcore(-1, 1, 0);
hw->num_prod_cores = 0;
- for (i = 0; i < RTE_MAX_LCORE; i++) {
- bool is_pcore = (mask && cores[i] != -1);
-
- if (rte_lcore_is_enabled(i)) {
- if (is_pcore) {
- /*
- * Populate the producer cores from parsed
- * coremask
- */
- hw->prod_core_list[cores[i]] = i;
- hw->num_prod_cores++;
-
- } else if ((++cnt == DLB2_EAL_PROBE_CORE ||
- rte_lcore_count() < DLB2_EAL_PROBE_CORE)) {
- /*
- * If no producer coremask is provided, use the
- * second EAL core to probe
- */
- cpu = i;
- break;
- }
- } else if (is_pcore) {
- DLB2_HW_ERR(hw, "Producer coremask(%s) must be a subset of EAL coremask\n",
- mask);
+ if (mask) {
+ int n = rte_eal_parse_coremask(mask, hw->prod_core_list);
+ if (n <= 0) {
+ DLB2_HW_ERR(hw, ": Invalid producer coremask=%s\n", mask);
return -1;
}
+ hw->num_prod_cores = n;
+ cpu = hw->prod_core_list[0];
+ for (u8 i = 0; i < hw->num_prod_cores; i++) {
+ if (!rte_lcore_is_enabled(hw->prod_core_list[i])) {
+ DLB2_HW_ERR(hw, "Producer coremask(%s) must be a subset of EAL coremask\n",
+ mask);
+ return -1;
+ }
+ }
}
- /* Use the first core in producer coremask to probe */
- if (hw->num_prod_cores)
- cpu = hw->prod_core_list[0];
dlb2_get_pp_allocation(hw, cpu, DLB2_LDB_PORT);
dlb2_get_pp_allocation(hw, cpu, DLB2_DIR_PORT);
diff --git a/lib/eal/common/eal_common_options.c b/lib/eal/common/eal_common_options.c
index 55c49a923f..807ba760ce 100644
--- a/lib/eal/common/eal_common_options.c
+++ b/lib/eal/common/eal_common_options.c
@@ -690,33 +690,6 @@ eal_parse_service_coremask(const char *coremask)
return 0;
}
-static int
-update_lcore_config(int *cores)
-{
- struct rte_config *cfg = rte_eal_get_configuration();
- unsigned int count = 0;
- unsigned int i;
- int ret = 0;
-
- for (i = 0; i < RTE_MAX_LCORE; i++) {
- if (cores[i] != -1) {
- if (eal_cpu_detected(i) == 0) {
- EAL_LOG(ERR, "lcore %u unavailable", i);
- ret = -1;
- continue;
- }
- cfg->lcore_role[i] = ROLE_RTE;
- count++;
- } else {
- cfg->lcore_role[i] = ROLE_OFF;
- }
- lcore_config[i].core_index = cores[i];
- }
- if (!ret)
- cfg->lcore_count = count;
- return ret;
-}
-
static int
check_core_list(int *lcores, unsigned int count)
{
@@ -759,7 +732,6 @@ int
rte_eal_parse_coremask(const char *coremask, int *cores)
{
const char *coremask_orig = coremask;
- int lcores[RTE_MAX_LCORE];
unsigned int count = 0;
int i, j, idx;
int val;
@@ -803,30 +775,19 @@ rte_eal_parse_coremask(const char *coremask, int *cores)
RTE_MAX_LCORE);
return -1;
}
- lcores[count++] = idx;
+ cores[count++] = idx;
}
}
}
if (count == 0) {
- EAL_LOG(ERR, "No lcores in coremask: [%s]",
- coremask_orig);
+ EAL_LOG(ERR, "No lcores in coremask: [%s]", coremask_orig);
return -1;
}
- if (check_core_list(lcores, count))
+ if (check_core_list(cores, count) != 0)
return -1;
- /*
- * Now that we've got a list of cores no longer than RTE_MAX_LCORE,
- * and no lcore in that list is greater than RTE_MAX_LCORE, populate
- * the cores array.
- */
- do {
- count--;
- cores[lcores[count]] = count;
- } while (count != 0);
-
- return 0;
+ return count;
}
static int
@@ -911,7 +872,6 @@ static int
eal_parse_corelist(const char *corelist, int *cores)
{
unsigned int count = 0, i;
- int lcores[RTE_MAX_LCORE];
char *end = NULL;
int min, max;
int idx;
@@ -949,7 +909,7 @@ eal_parse_corelist(const char *corelist, int *cores)
/* Check if this idx is already present */
for (i = 0; i < count; i++) {
- if (lcores[i] == idx)
+ if (cores[i] == idx)
dup = true;
}
if (dup)
@@ -959,7 +919,7 @@ eal_parse_corelist(const char *corelist, int *cores)
RTE_MAX_LCORE);
return -1;
}
- lcores[count++] = idx;
+ cores[count++] = idx;
}
min = -1;
} else
@@ -967,23 +927,15 @@ eal_parse_corelist(const char *corelist, int *cores)
corelist = end + 1;
} while (*end != '\0');
- if (count == 0)
+ if (count == 0) {
+ EAL_LOG(ERR, "No lcores in corelist");
return -1;
+ }
- if (check_core_list(lcores, count))
+ if (check_core_list(cores, count))
return -1;
- /*
- * Now that we've got a list of cores no longer than RTE_MAX_LCORE,
- * and no lcore in that list is greater than RTE_MAX_LCORE, populate
- * the cores array.
- */
- do {
- count--;
- cores[lcores[count]] = count;
- } while (count != 0);
-
- return 0;
+ return count;
}
/* Changes the lcore id of the main thread */
@@ -1500,75 +1452,6 @@ eal_parse_base_virtaddr(const char *arg)
return 0;
}
-/* caller is responsible for freeing the returned string */
-static char *
-available_cores(void)
-{
- char *str = NULL;
- int previous;
- int sequence;
- char *tmp;
- int idx;
-
- /* find the first available cpu */
- for (idx = 0; idx < RTE_MAX_LCORE; idx++) {
- if (eal_cpu_detected(idx) == 0)
- continue;
- break;
- }
- if (idx >= RTE_MAX_LCORE)
- return NULL;
-
- /* first sequence */
- if (asprintf(&str, "%d", idx) < 0)
- return NULL;
- previous = idx;
- sequence = 0;
-
- for (idx++ ; idx < RTE_MAX_LCORE; idx++) {
- if (eal_cpu_detected(idx) == 0)
- continue;
-
- if (idx == previous + 1) {
- previous = idx;
- sequence = 1;
- continue;
- }
-
- /* finish current sequence */
- if (sequence) {
- if (asprintf(&tmp, "%s-%d", str, previous) < 0) {
- free(str);
- return NULL;
- }
- free(str);
- str = tmp;
- }
-
- /* new sequence */
- if (asprintf(&tmp, "%s,%d", str, idx) < 0) {
- free(str);
- return NULL;
- }
- free(str);
- str = tmp;
- previous = idx;
- sequence = 0;
- }
-
- /* finish last sequence */
- if (sequence) {
- if (asprintf(&tmp, "%s-%d", str, previous) < 0) {
- free(str);
- return NULL;
- }
- free(str);
- str = tmp;
- }
-
- return str;
-}
-
#define HUGE_UNLINK_NEVER "never"
static int
@@ -1981,43 +1864,43 @@ eal_adjust_config(struct internal_config *internal_cfg)
return -1;
}
- if (lcore_options.core_mask_opt) {
- int lcore_indexes[RTE_MAX_LCORE];
-
- if (rte_eal_parse_coremask(lcore_options.core_mask_opt, lcore_indexes) < 0) {
- EAL_LOG(ERR, "invalid coremask syntax");
- return -1;
- }
- if (update_lcore_config(lcore_indexes) < 0) {
- char *available = available_cores();
- EAL_LOG(ERR,
- "invalid coremask, please check specified cores are part of %s",
- available);
- free(available);
- return -1;
- }
- } else if (lcore_options.core_list_opt) {
+ if (lcore_options.core_mask_opt || lcore_options.core_list_opt) {
int lcore_indexes[RTE_MAX_LCORE];
+ int nb_indexes = lcore_options.core_list_opt ?
+ eal_parse_corelist(lcore_options.core_list_opt, lcore_indexes) :
+ rte_eal_parse_coremask(lcore_options.core_mask_opt, lcore_indexes);
- if (eal_parse_corelist(lcore_options.core_list_opt, lcore_indexes) < 0) {
- EAL_LOG(ERR, "invalid core list syntax");
+ if (nb_indexes < 0)
return -1;
- }
- if (update_lcore_config(lcore_indexes) < 0) {
- char *available = available_cores();
- EAL_LOG(ERR,
- "invalid core list, please check specified cores are part of %s",
- available);
- free(available);
- return -1;
+ char *core_map_opt = malloc(RTE_MAX_LCORE * 10);
+ size_t core_map_len = 0;
+ for (i = 0; i < nb_indexes; i++) {
+ if (!eal_cpu_detected(lcore_indexes[i])) {
+ EAL_LOG(ERR, "core %d not present", lcore_indexes[i]);
+ return -1;
+ }
+ int n = snprintf(core_map_opt + core_map_len,
+ (RTE_MAX_LCORE * 10) - core_map_len,
+ "%s%d", i == 0 ? "" : ",", lcore_indexes[i]);
+ if (n < 0 || (size_t)n >= (RTE_MAX_LCORE * 10) - core_map_len) {
+ EAL_LOG(ERR, "core map string too long");
+ return -1;
+ }
+ core_map_len += n;
}
- } else if (lcore_options.core_map_opt) {
+ lcore_options.core_map_opt = core_map_opt;
+ printf("Generated core map: %s\n", lcore_options.core_map_opt);
+ }
+
+ if (lcore_options.core_map_opt) {
if (eal_parse_lcores(lcore_options.core_map_opt) < 0) {
EAL_LOG(ERR, "invalid parameter for --" OPT_LCORES);
return -1;
}
+ if (lcore_options.core_mask_opt || lcore_options.core_list_opt)
+ free(RTE_CAST_PTR(void *,lcore_options.core_map_opt));
} else {
eal_auto_detect_cores(cfg);
}
--
2.43.0
^ permalink raw reply [flat|nested] 31+ messages in thread
* [RFC PATCH 3/3] eal: allow automatic mapping of high lcore ids
2025-03-13 11:38 [RFC PATCH 0/3] allow easier use of high lcore-ids Bruce Richardson
2025-03-13 11:38 ` [RFC PATCH 1/3] eal: centralize core parameter parsing Bruce Richardson
2025-03-13 11:38 ` [RFC PATCH 2/3] eal: convert core masks and lists to core sets Bruce Richardson
@ 2025-03-13 11:38 ` Bruce Richardson
2025-03-24 17:30 ` [PATCH v2 0/3] allow easier use of high lcore-ids Bruce Richardson
2025-05-02 15:11 ` [PATCH v3 0/4] rework and expand EAL lcore options Bruce Richardson
4 siblings, 0 replies; 31+ messages in thread
From: Bruce Richardson @ 2025-03-13 11:38 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson
To use cores with IDs greater than RTE_MAX_LCORE the user must provide a
mapping of those higher core numbers to ids within the 0 - RTE_MAX_LCORE
range. This can be awkward to do manually when more than a few lcores
are involved, so instead we can provide an extra option to EAL to do
this manually.
This patch therefore introduces the "map-lcore-ids" flag, which
automatically maps all provided lcore numbers to start at zero.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
drivers/event/dlb2/pf/base/dlb2_resource.c | 2 +-
lib/eal/common/eal_common_options.c | 32 ++++++++++++++--------
lib/eal/common/eal_options.h | 2 ++
lib/eal/include/rte_eal.h | 2 +-
4 files changed, 25 insertions(+), 13 deletions(-)
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index c4bc248afc..541a16db24 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -931,7 +931,7 @@ dlb2_resource_probe(struct dlb2_hw *hw, const void *probe_args)
cpu = rte_get_next_lcore(-1, 1, 0);
hw->num_prod_cores = 0;
if (mask) {
- int n = rte_eal_parse_coremask(mask, hw->prod_core_list);
+ int n = rte_eal_parse_coremask(mask, hw->prod_core_list, true);
if (n <= 0) {
DLB2_HW_ERR(hw, ": Invalid producer coremask=%s\n", mask);
return -1;
diff --git a/lib/eal/common/eal_common_options.c b/lib/eal/common/eal_common_options.c
index 807ba760ce..107b9eaad5 100644
--- a/lib/eal/common/eal_common_options.c
+++ b/lib/eal/common/eal_common_options.c
@@ -106,6 +106,7 @@ eal_long_options[] = {
{OPT_NO_TELEMETRY, 0, NULL, OPT_NO_TELEMETRY_NUM },
{OPT_FORCE_MAX_SIMD_BITWIDTH, 1, NULL, OPT_FORCE_MAX_SIMD_BITWIDTH_NUM},
{OPT_HUGE_WORKER_STACK, 2, NULL, OPT_HUGE_WORKER_STACK_NUM },
+ {OPT_MAP_LCORE_IDS, 0, NULL, OPT_MAP_LCORE_IDS_NUM },
{0, 0, NULL, 0 }
};
@@ -152,6 +153,7 @@ TAILQ_HEAD_INITIALIZER(devopt_list);
static int main_lcore_parsed;
static int mem_parsed;
static struct lcore_options {
+ bool map_lcore_ids;
const char *core_mask_opt;
const char *core_list_opt;
const char *core_map_opt;
@@ -723,13 +725,14 @@ check_core_list(int *lcores, unsigned int count)
if (len > 0)
lcorestr[len - 1] = 0;
EAL_LOG(ERR, "To use high physical core ids, "
- "please use --lcores to map them to lcore ids below RTE_MAX_LCORE, "
+ "please use --map-lcore-ids flag to map core ids automatically, "
+ "or use --lcores to map them manually to lcore ids below RTE_MAX_LCORE, "
"e.g. --lcores %s", lcorestr);
return -1;
}
int
-rte_eal_parse_coremask(const char *coremask, int *cores)
+rte_eal_parse_coremask(const char *coremask, int *cores, bool no_check)
{
const char *coremask_orig = coremask;
unsigned int count = 0;
@@ -784,7 +787,7 @@ rte_eal_parse_coremask(const char *coremask, int *cores)
return -1;
}
- if (check_core_list(cores, count) != 0)
+ if (!no_check && check_core_list(cores, count) != 0)
return -1;
return count;
@@ -869,7 +872,7 @@ eal_parse_service_corelist(const char *corelist)
}
static int
-eal_parse_corelist(const char *corelist, int *cores)
+eal_parse_corelist(const char *corelist, int *cores, bool no_check)
{
unsigned int count = 0, i;
char *end = NULL;
@@ -932,7 +935,7 @@ eal_parse_corelist(const char *corelist, int *cores)
return -1;
}
- if (check_core_list(cores, count))
+ if (!no_check && check_core_list(cores, count))
return -1;
return count;
@@ -1767,6 +1770,9 @@ eal_parse_common_option(int opt, const char *optarg,
return -1;
}
break;
+ case OPT_MAP_LCORE_IDS_NUM:
+ lcore_options.map_lcore_ids = true;
+ break;
/* don't know what to do, leave this to caller */
default:
@@ -1866,10 +1872,11 @@ eal_adjust_config(struct internal_config *internal_cfg)
if (lcore_options.core_mask_opt || lcore_options.core_list_opt) {
- int lcore_indexes[RTE_MAX_LCORE];
+ bool map_ids = lcore_options.map_lcore_ids;
+ int idxs[RTE_MAX_LCORE];
int nb_indexes = lcore_options.core_list_opt ?
- eal_parse_corelist(lcore_options.core_list_opt, lcore_indexes) :
- rte_eal_parse_coremask(lcore_options.core_mask_opt, lcore_indexes);
+ eal_parse_corelist(lcore_options.core_list_opt, idxs, map_ids) :
+ rte_eal_parse_coremask(lcore_options.core_mask_opt, idxs, map_ids);
if (nb_indexes < 0)
return -1;
@@ -1877,13 +1884,16 @@ eal_adjust_config(struct internal_config *internal_cfg)
char *core_map_opt = malloc(RTE_MAX_LCORE * 10);
size_t core_map_len = 0;
for (i = 0; i < nb_indexes; i++) {
- if (!eal_cpu_detected(lcore_indexes[i])) {
- EAL_LOG(ERR, "core %d not present", lcore_indexes[i]);
+ if (!eal_cpu_detected(idxs[i])) {
+ EAL_LOG(ERR, "core %d not present", idxs[i]);
return -1;
}
int n = snprintf(core_map_opt + core_map_len,
(RTE_MAX_LCORE * 10) - core_map_len,
- "%s%d", i == 0 ? "" : ",", lcore_indexes[i]);
+ "%s%d@%d",
+ i == 0 ? "" : ",",
+ map_ids ? i: idxs[i],
+ idxs[i]);
if (n < 0 || (size_t)n >= (RTE_MAX_LCORE * 10) - core_map_len) {
EAL_LOG(ERR, "core map string too long");
return -1;
diff --git a/lib/eal/common/eal_options.h b/lib/eal/common/eal_options.h
index 95fb4f6108..384c2d016e 100644
--- a/lib/eal/common/eal_options.h
+++ b/lib/eal/common/eal_options.h
@@ -93,6 +93,8 @@ enum {
OPT_FORCE_MAX_SIMD_BITWIDTH_NUM,
#define OPT_HUGE_WORKER_STACK "huge-worker-stack"
OPT_HUGE_WORKER_STACK_NUM,
+#define OPT_MAP_LCORE_IDS "map-lcore-ids"
+ OPT_MAP_LCORE_IDS_NUM,
OPT_LONG_MAX_NUM
};
diff --git a/lib/eal/include/rte_eal.h b/lib/eal/include/rte_eal.h
index c826e143f1..a53fb29c45 100644
--- a/lib/eal/include/rte_eal.h
+++ b/lib/eal/include/rte_eal.h
@@ -508,7 +508,7 @@ rte_eal_get_runtime_dir(void);
*/
__rte_internal
int
-rte_eal_parse_coremask(const char *coremask, int *cores);
+rte_eal_parse_coremask(const char *coremask, int *cores, bool nocheck);
#ifdef __cplusplus
}
--
2.43.0
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 0/3] allow easier use of high lcore-ids
2025-03-13 11:38 [RFC PATCH 0/3] allow easier use of high lcore-ids Bruce Richardson
` (2 preceding siblings ...)
2025-03-13 11:38 ` [RFC PATCH 3/3] eal: allow automatic mapping of high lcore ids Bruce Richardson
@ 2025-03-24 17:30 ` Bruce Richardson
2025-03-24 17:30 ` [PATCH v2 1/3] eal: centralize core parameter parsing Bruce Richardson
` (3 more replies)
2025-05-02 15:11 ` [PATCH v3 0/4] rework and expand EAL lcore options Bruce Richardson
4 siblings, 4 replies; 31+ messages in thread
From: Bruce Richardson @ 2025-03-24 17:30 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson
Traditionally, DPDK has had a direct mapping of internal lcore-ids, to
the actual core numbers in use. With higher core count servers becoming
more prevalent the issue becomes one of increasing memory footprint when
using such a scheme, due to the need to have all arrays dimensioned for
all cores on the system, whether or not those cores are in use by the
app.
Therefore, the decision was made in the past to not expand the
build-time RTE_MAX_LCORE value beyond 128. Instead, it was recommended
that users use the "--lcores" EAL parameter to take the high-numbered
cores they wish to use and map them to lcore-ids within the 0 - 128
range. While this works, this is a little clunky as it means that
instead of just passing, for example, "-l 130-139", the user must
instead pass "--lcores 0@130,1@131,2@132,3@133,...."
This patchset attempts to simplify the situation by adding a new flag to
do this mapping automatically. To use cores 130-139 and map them to ids
0-9 internally, the EAL args now become: "-l 130-139 --map-lcore-ids",
or using the shorter "-M" version of the flag: "-Ml 130-139".
Adding this new parameter required some rework of the existing arg
parsing code, because in current DPDK the args are parsed and checked in
the order they appear on the commandline. This means that using the
example above, the core parameter 130-139 will be rejected immediately
before the "map-lcore-ids" parameter is seen. To work around this, the
core (and service core) parameters are not parsed when seen, instead
they are only saved off and parsed after all arguments are parsed. The
"-l" and "-c" parameters are converted into "--lcores" arguments, so all
assigning of lcore ids is done there in all cases.
RFC->v2:
* converted printf to DEBUG log
* added "-M" as shorter version of flag
* added documentation
* renamed internal API that was changed to avoid any potential hidden
runtime issues.
Bruce Richardson (3):
eal: centralize core parameter parsing
eal: convert core masks and lists to core sets
eal: allow automatic mapping of high lcore ids
doc/guides/linux_gsg/eal_args.include.rst | 26 +-
drivers/event/dlb2/dlb2_priv.h | 2 -
drivers/event/dlb2/pf/base/dlb2_resource.c | 48 +--
lib/eal/common/eal_common_options.c | 340 +++++++--------------
lib/eal/common/eal_options.h | 3 +
lib/eal/include/rte_eal.h | 14 +-
lib/eal/version.map | 2 +-
7 files changed, 161 insertions(+), 274 deletions(-)
--
2.45.2
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 1/3] eal: centralize core parameter parsing
2025-03-24 17:30 ` [PATCH v2 0/3] allow easier use of high lcore-ids Bruce Richardson
@ 2025-03-24 17:30 ` Bruce Richardson
2025-04-07 6:58 ` David Marchand
2025-03-24 17:30 ` [PATCH v2 2/3] eal: convert core masks and lists to core sets Bruce Richardson
` (2 subsequent siblings)
3 siblings, 1 reply; 31+ messages in thread
From: Bruce Richardson @ 2025-03-24 17:30 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, Tyler Retzlaff
Rather than parsing the lcore parameters as they are encountered, just
save off the lcore parameters and then parse them at the end. This
allows better knowledge of what parameters are available or not when
parsing.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
lib/eal/common/eal_common_options.c | 183 +++++++++++++---------------
1 file changed, 84 insertions(+), 99 deletions(-)
diff --git a/lib/eal/common/eal_common_options.c b/lib/eal/common/eal_common_options.c
index 79db9a47dd..55c49a923f 100644
--- a/lib/eal/common/eal_common_options.c
+++ b/lib/eal/common/eal_common_options.c
@@ -151,7 +151,13 @@ TAILQ_HEAD_INITIALIZER(devopt_list);
static int main_lcore_parsed;
static int mem_parsed;
-static int core_parsed;
+static struct lcore_options {
+ const char *core_mask_opt;
+ const char *core_list_opt;
+ const char *core_map_opt;
+ const char *service_mask_opt;
+ const char *service_list_opt;
+} lcore_options = {0};
/* Allow the application to print its usage message too if set */
static rte_usage_hook_t rte_application_usage_hook;
@@ -674,7 +680,7 @@ eal_parse_service_coremask(const char *coremask)
if (count == 0)
return -1;
- if (core_parsed && taken_lcore_count != count) {
+ if (taken_lcore_count != count) {
EAL_LOG(WARNING,
"Not all service cores are in the coremask. "
"Please ensure -c or -l includes service cores");
@@ -684,17 +690,6 @@ eal_parse_service_coremask(const char *coremask)
return 0;
}
-static int
-eal_service_cores_parsed(void)
-{
- int idx;
- for (idx = 0; idx < RTE_MAX_LCORE; idx++) {
- if (lcore_config[idx].core_role == ROLE_SERVICE)
- return 1;
- }
- return 0;
-}
-
static int
update_lcore_config(int *cores)
{
@@ -903,7 +898,7 @@ eal_parse_service_corelist(const char *corelist)
if (count == 0)
return -1;
- if (core_parsed && taken_lcore_count != count) {
+ if (taken_lcore_count != count) {
EAL_LOG(WARNING,
"Not all service cores were in the coremask. "
"Please ensure -c or -l includes service cores");
@@ -1673,83 +1668,20 @@ eal_parse_common_option(int opt, const char *optarg,
a_used = 1;
break;
/* coremask */
- case 'c': {
- int lcore_indexes[RTE_MAX_LCORE];
-
- if (eal_service_cores_parsed())
- EAL_LOG(WARNING,
- "Service cores parsed before dataplane cores. Please ensure -c is before -s or -S");
- if (rte_eal_parse_coremask(optarg, lcore_indexes) < 0) {
- EAL_LOG(ERR, "invalid coremask syntax");
- return -1;
- }
- if (update_lcore_config(lcore_indexes) < 0) {
- char *available = available_cores();
-
- EAL_LOG(ERR,
- "invalid coremask, please check specified cores are part of %s",
- available);
- free(available);
- return -1;
- }
-
- if (core_parsed) {
- EAL_LOG(ERR, "Option -c is ignored, because (%s) is set!",
- (core_parsed == LCORE_OPT_LST) ? "-l" :
- (core_parsed == LCORE_OPT_MAP) ? "--lcores" :
- "-c");
- return -1;
- }
-
- core_parsed = LCORE_OPT_MSK;
+ case 'c':
+ lcore_options.core_mask_opt = optarg;
break;
- }
/* corelist */
- case 'l': {
- int lcore_indexes[RTE_MAX_LCORE];
-
- if (eal_service_cores_parsed())
- EAL_LOG(WARNING,
- "Service cores parsed before dataplane cores. Please ensure -l is before -s or -S");
-
- if (eal_parse_corelist(optarg, lcore_indexes) < 0) {
- EAL_LOG(ERR, "invalid core list syntax");
- return -1;
- }
- if (update_lcore_config(lcore_indexes) < 0) {
- char *available = available_cores();
-
- EAL_LOG(ERR,
- "invalid core list, please check specified cores are part of %s",
- available);
- free(available);
- return -1;
- }
-
- if (core_parsed) {
- EAL_LOG(ERR, "Option -l is ignored, because (%s) is set!",
- (core_parsed == LCORE_OPT_MSK) ? "-c" :
- (core_parsed == LCORE_OPT_MAP) ? "--lcores" :
- "-l");
- return -1;
- }
-
- core_parsed = LCORE_OPT_LST;
+ case 'l':
+ lcore_options.core_list_opt = optarg;
break;
- }
/* service coremask */
case 's':
- if (eal_parse_service_coremask(optarg) < 0) {
- EAL_LOG(ERR, "invalid service coremask");
- return -1;
- }
+ lcore_options.service_mask_opt = optarg;
break;
/* service corelist */
case 'S':
- if (eal_parse_service_corelist(optarg) < 0) {
- EAL_LOG(ERR, "invalid service core list");
- return -1;
- }
+ lcore_options.service_list_opt = optarg;
break;
/* size of memory */
case 'm':
@@ -1918,21 +1850,7 @@ eal_parse_common_option(int opt, const char *optarg,
#endif /* !RTE_EXEC_ENV_WINDOWS */
case OPT_LCORES_NUM:
- if (eal_parse_lcores(optarg) < 0) {
- EAL_LOG(ERR, "invalid parameter for --"
- OPT_LCORES);
- return -1;
- }
-
- if (core_parsed) {
- EAL_LOG(ERR, "Option --lcores is ignored, because (%s) is set!",
- (core_parsed == LCORE_OPT_LST) ? "-l" :
- (core_parsed == LCORE_OPT_MSK) ? "-c" :
- "--lcores");
- return -1;
- }
-
- core_parsed = LCORE_OPT_MAP;
+ lcore_options.core_map_opt = optarg;
break;
case OPT_LEGACY_MEM_NUM:
conf->legacy_mem = 1;
@@ -1973,6 +1891,7 @@ eal_parse_common_option(int opt, const char *optarg,
}
+
return 0;
ba_conflict:
@@ -2046,8 +1965,74 @@ eal_adjust_config(struct internal_config *internal_cfg)
struct internal_config *internal_conf =
eal_get_internal_configuration();
- if (!core_parsed)
+ /* handle all the various core options, ignoring order of them.
+ * First check that multiple lcore options (coremask, corelist, coremap) are
+ * not used together. Then check that the service options (coremask, corelist)
+ * are not both set. In both cases error out if multiple options are set.
+ */
+ if ((lcore_options.core_mask_opt && lcore_options.core_list_opt) ||
+ (lcore_options.core_mask_opt && lcore_options.core_map_opt) ||
+ (lcore_options.core_list_opt && lcore_options.core_map_opt)) {
+ EAL_LOG(ERR, "Only one of -c, -l and --"OPT_LCORES" may be given at a time");
+ return -1;
+ }
+ if ((lcore_options.service_mask_opt && lcore_options.service_list_opt)) {
+ EAL_LOG(ERR, "Only one of -s and -S may be given at a time");
+ return -1;
+ }
+
+ if (lcore_options.core_mask_opt) {
+ int lcore_indexes[RTE_MAX_LCORE];
+
+ if (rte_eal_parse_coremask(lcore_options.core_mask_opt, lcore_indexes) < 0) {
+ EAL_LOG(ERR, "invalid coremask syntax");
+ return -1;
+ }
+ if (update_lcore_config(lcore_indexes) < 0) {
+ char *available = available_cores();
+
+ EAL_LOG(ERR,
+ "invalid coremask, please check specified cores are part of %s",
+ available);
+ free(available);
+ return -1;
+ }
+ } else if (lcore_options.core_list_opt) {
+ int lcore_indexes[RTE_MAX_LCORE];
+
+ if (eal_parse_corelist(lcore_options.core_list_opt, lcore_indexes) < 0) {
+ EAL_LOG(ERR, "invalid core list syntax");
+ return -1;
+ }
+ if (update_lcore_config(lcore_indexes) < 0) {
+ char *available = available_cores();
+
+ EAL_LOG(ERR,
+ "invalid core list, please check specified cores are part of %s",
+ available);
+ free(available);
+ return -1;
+ }
+ } else if (lcore_options.core_map_opt) {
+ if (eal_parse_lcores(lcore_options.core_map_opt) < 0) {
+ EAL_LOG(ERR, "invalid parameter for --" OPT_LCORES);
+ return -1;
+ }
+ } else {
eal_auto_detect_cores(cfg);
+ }
+
+ if (lcore_options.service_mask_opt) {
+ if (eal_parse_service_coremask(lcore_options.service_mask_opt) < 0) {
+ EAL_LOG(ERR, "invalid service coremask");
+ return -1;
+ }
+ } else if (lcore_options.service_list_opt) {
+ if (eal_parse_service_corelist(lcore_options.service_list_opt) < 0) {
+ EAL_LOG(ERR, "invalid service core list");
+ return -1;
+ }
+ }
if (internal_conf->process_type == RTE_PROC_AUTO)
internal_conf->process_type = eal_proc_type_detect();
--
2.45.2
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 2/3] eal: convert core masks and lists to core sets
2025-03-24 17:30 ` [PATCH v2 0/3] allow easier use of high lcore-ids Bruce Richardson
2025-03-24 17:30 ` [PATCH v2 1/3] eal: centralize core parameter parsing Bruce Richardson
@ 2025-03-24 17:30 ` Bruce Richardson
2025-04-07 6:59 ` David Marchand
2025-03-24 17:30 ` [PATCH v2 3/3] eal: allow automatic mapping of high lcore ids Bruce Richardson
2025-04-01 14:06 ` [PATCH v2 0/3] allow easier use of high lcore-ids Bruce Richardson
3 siblings, 1 reply; 31+ messages in thread
From: Bruce Richardson @ 2025-03-24 17:30 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, Pravin Pathak, Tyler Retzlaff
Rather than directly parsing and working with the core mask and core
list parameters, convert them into core maps, and centralise all core
parsing there. This allows future work to adjust the mappings of cores
when generating that mapping parameter.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
NOTE: this patch changes the behaviour of the exported, but internal API
rte_eal_parse_core_mask, used by dlb driver. Since this is an internal
API changing behaviour is allowed. However, to prevent issues - in case
any external app is using the API - we rename the function to
rte_eal_expand_coremask.
---
drivers/event/dlb2/dlb2_priv.h | 2 -
drivers/event/dlb2/pf/base/dlb2_resource.c | 48 ++---
lib/eal/common/eal_common_options.c | 195 +++++----------------
lib/eal/include/rte_eal.h | 12 +-
lib/eal/version.map | 2 +-
5 files changed, 60 insertions(+), 199 deletions(-)
diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
index 52da31ed31..63c0bea155 100644
--- a/drivers/event/dlb2/dlb2_priv.h
+++ b/drivers/event/dlb2/dlb2_priv.h
@@ -737,8 +737,6 @@ void dlb2_event_build_hcws(struct dlb2_port *qm_port,
uint8_t *sched_type,
uint8_t *queue_id);
-/* Extern functions */
-extern int rte_eal_parse_coremask(const char *coremask, int *cores);
/* Extern globals */
extern struct process_local_port_data dlb2_port[][DLB2_NUM_PORT_TYPES];
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 3004902118..9b087b03b5 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -922,49 +922,31 @@ dlb2_resource_probe(struct dlb2_hw *hw, const void *probe_args)
{
const struct dlb2_devargs *args = (const struct dlb2_devargs *)probe_args;
const char *mask = args ? args->producer_coremask : NULL;
- int cpu = 0, cnt = 0, cores[RTE_MAX_LCORE], i;
+ int cpu = 0;
if (args) {
mask = (const char *)args->producer_coremask;
}
- if (mask && rte_eal_parse_coremask(mask, cores)) {
- DLB2_HW_ERR(hw, ": Invalid producer coremask=%s\n", mask);
- return -1;
- }
-
+ cpu = rte_get_next_lcore(-1, 1, 0);
hw->num_prod_cores = 0;
- for (i = 0; i < RTE_MAX_LCORE; i++) {
- bool is_pcore = (mask && cores[i] != -1);
-
- if (rte_lcore_is_enabled(i)) {
- if (is_pcore) {
- /*
- * Populate the producer cores from parsed
- * coremask
- */
- hw->prod_core_list[cores[i]] = i;
- hw->num_prod_cores++;
-
- } else if ((++cnt == DLB2_EAL_PROBE_CORE ||
- rte_lcore_count() < DLB2_EAL_PROBE_CORE)) {
- /*
- * If no producer coremask is provided, use the
- * second EAL core to probe
- */
- cpu = i;
- break;
- }
- } else if (is_pcore) {
- DLB2_HW_ERR(hw, "Producer coremask(%s) must be a subset of EAL coremask\n",
- mask);
+ if (mask) {
+ int n = rte_eal_expand_coremask(mask, hw->prod_core_list);
+ if (n <= 0) {
+ DLB2_HW_ERR(hw, ": Invalid producer coremask=%s\n", mask);
return -1;
}
+ hw->num_prod_cores = n;
+ cpu = hw->prod_core_list[0];
+ for (u8 i = 0; i < hw->num_prod_cores; i++) {
+ if (!rte_lcore_is_enabled(hw->prod_core_list[i])) {
+ DLB2_HW_ERR(hw, "Producer coremask(%s) must be a subset of EAL coremask\n",
+ mask);
+ return -1;
+ }
+ }
}
- /* Use the first core in producer coremask to probe */
- if (hw->num_prod_cores)
- cpu = hw->prod_core_list[0];
dlb2_get_pp_allocation(hw, cpu, DLB2_LDB_PORT);
dlb2_get_pp_allocation(hw, cpu, DLB2_DIR_PORT);
diff --git a/lib/eal/common/eal_common_options.c b/lib/eal/common/eal_common_options.c
index 55c49a923f..22ea6b24e4 100644
--- a/lib/eal/common/eal_common_options.c
+++ b/lib/eal/common/eal_common_options.c
@@ -690,33 +690,6 @@ eal_parse_service_coremask(const char *coremask)
return 0;
}
-static int
-update_lcore_config(int *cores)
-{
- struct rte_config *cfg = rte_eal_get_configuration();
- unsigned int count = 0;
- unsigned int i;
- int ret = 0;
-
- for (i = 0; i < RTE_MAX_LCORE; i++) {
- if (cores[i] != -1) {
- if (eal_cpu_detected(i) == 0) {
- EAL_LOG(ERR, "lcore %u unavailable", i);
- ret = -1;
- continue;
- }
- cfg->lcore_role[i] = ROLE_RTE;
- count++;
- } else {
- cfg->lcore_role[i] = ROLE_OFF;
- }
- lcore_config[i].core_index = cores[i];
- }
- if (!ret)
- cfg->lcore_count = count;
- return ret;
-}
-
static int
check_core_list(int *lcores, unsigned int count)
{
@@ -756,10 +729,9 @@ check_core_list(int *lcores, unsigned int count)
}
int
-rte_eal_parse_coremask(const char *coremask, int *cores)
+rte_eal_expand_coremask(const char *coremask, int *cores)
{
const char *coremask_orig = coremask;
- int lcores[RTE_MAX_LCORE];
unsigned int count = 0;
int i, j, idx;
int val;
@@ -803,30 +775,19 @@ rte_eal_parse_coremask(const char *coremask, int *cores)
RTE_MAX_LCORE);
return -1;
}
- lcores[count++] = idx;
+ cores[count++] = idx;
}
}
}
if (count == 0) {
- EAL_LOG(ERR, "No lcores in coremask: [%s]",
- coremask_orig);
+ EAL_LOG(ERR, "No lcores in coremask: [%s]", coremask_orig);
return -1;
}
- if (check_core_list(lcores, count))
+ if (check_core_list(cores, count) != 0)
return -1;
- /*
- * Now that we've got a list of cores no longer than RTE_MAX_LCORE,
- * and no lcore in that list is greater than RTE_MAX_LCORE, populate
- * the cores array.
- */
- do {
- count--;
- cores[lcores[count]] = count;
- } while (count != 0);
-
- return 0;
+ return count;
}
static int
@@ -911,7 +872,6 @@ static int
eal_parse_corelist(const char *corelist, int *cores)
{
unsigned int count = 0, i;
- int lcores[RTE_MAX_LCORE];
char *end = NULL;
int min, max;
int idx;
@@ -949,7 +909,7 @@ eal_parse_corelist(const char *corelist, int *cores)
/* Check if this idx is already present */
for (i = 0; i < count; i++) {
- if (lcores[i] == idx)
+ if (cores[i] == idx)
dup = true;
}
if (dup)
@@ -959,7 +919,7 @@ eal_parse_corelist(const char *corelist, int *cores)
RTE_MAX_LCORE);
return -1;
}
- lcores[count++] = idx;
+ cores[count++] = idx;
}
min = -1;
} else
@@ -967,23 +927,15 @@ eal_parse_corelist(const char *corelist, int *cores)
corelist = end + 1;
} while (*end != '\0');
- if (count == 0)
+ if (count == 0) {
+ EAL_LOG(ERR, "No lcores in corelist");
return -1;
+ }
- if (check_core_list(lcores, count))
+ if (check_core_list(cores, count))
return -1;
- /*
- * Now that we've got a list of cores no longer than RTE_MAX_LCORE,
- * and no lcore in that list is greater than RTE_MAX_LCORE, populate
- * the cores array.
- */
- do {
- count--;
- cores[lcores[count]] = count;
- } while (count != 0);
-
- return 0;
+ return count;
}
/* Changes the lcore id of the main thread */
@@ -1500,75 +1452,6 @@ eal_parse_base_virtaddr(const char *arg)
return 0;
}
-/* caller is responsible for freeing the returned string */
-static char *
-available_cores(void)
-{
- char *str = NULL;
- int previous;
- int sequence;
- char *tmp;
- int idx;
-
- /* find the first available cpu */
- for (idx = 0; idx < RTE_MAX_LCORE; idx++) {
- if (eal_cpu_detected(idx) == 0)
- continue;
- break;
- }
- if (idx >= RTE_MAX_LCORE)
- return NULL;
-
- /* first sequence */
- if (asprintf(&str, "%d", idx) < 0)
- return NULL;
- previous = idx;
- sequence = 0;
-
- for (idx++ ; idx < RTE_MAX_LCORE; idx++) {
- if (eal_cpu_detected(idx) == 0)
- continue;
-
- if (idx == previous + 1) {
- previous = idx;
- sequence = 1;
- continue;
- }
-
- /* finish current sequence */
- if (sequence) {
- if (asprintf(&tmp, "%s-%d", str, previous) < 0) {
- free(str);
- return NULL;
- }
- free(str);
- str = tmp;
- }
-
- /* new sequence */
- if (asprintf(&tmp, "%s,%d", str, idx) < 0) {
- free(str);
- return NULL;
- }
- free(str);
- str = tmp;
- previous = idx;
- sequence = 0;
- }
-
- /* finish last sequence */
- if (sequence) {
- if (asprintf(&tmp, "%s-%d", str, previous) < 0) {
- free(str);
- return NULL;
- }
- free(str);
- str = tmp;
- }
-
- return str;
-}
-
#define HUGE_UNLINK_NEVER "never"
static int
@@ -1981,43 +1864,43 @@ eal_adjust_config(struct internal_config *internal_cfg)
return -1;
}
- if (lcore_options.core_mask_opt) {
- int lcore_indexes[RTE_MAX_LCORE];
-
- if (rte_eal_parse_coremask(lcore_options.core_mask_opt, lcore_indexes) < 0) {
- EAL_LOG(ERR, "invalid coremask syntax");
- return -1;
- }
- if (update_lcore_config(lcore_indexes) < 0) {
- char *available = available_cores();
- EAL_LOG(ERR,
- "invalid coremask, please check specified cores are part of %s",
- available);
- free(available);
- return -1;
- }
- } else if (lcore_options.core_list_opt) {
+ if (lcore_options.core_mask_opt || lcore_options.core_list_opt) {
int lcore_indexes[RTE_MAX_LCORE];
+ int nb_indexes = lcore_options.core_list_opt ?
+ eal_parse_corelist(lcore_options.core_list_opt, lcore_indexes) :
+ rte_eal_expand_coremask(lcore_options.core_mask_opt, lcore_indexes);
- if (eal_parse_corelist(lcore_options.core_list_opt, lcore_indexes) < 0) {
- EAL_LOG(ERR, "invalid core list syntax");
+ if (nb_indexes < 0)
return -1;
- }
- if (update_lcore_config(lcore_indexes) < 0) {
- char *available = available_cores();
- EAL_LOG(ERR,
- "invalid core list, please check specified cores are part of %s",
- available);
- free(available);
- return -1;
+ char *core_map_opt = malloc(RTE_MAX_LCORE * 10);
+ size_t core_map_len = 0;
+ for (i = 0; i < nb_indexes; i++) {
+ if (!eal_cpu_detected(lcore_indexes[i])) {
+ EAL_LOG(ERR, "core %d not present", lcore_indexes[i]);
+ return -1;
+ }
+ int n = snprintf(core_map_opt + core_map_len,
+ (RTE_MAX_LCORE * 10) - core_map_len,
+ "%s%d", i == 0 ? "" : ",", lcore_indexes[i]);
+ if (n < 0 || (size_t)n >= (RTE_MAX_LCORE * 10) - core_map_len) {
+ EAL_LOG(ERR, "core map string too long");
+ return -1;
+ }
+ core_map_len += n;
}
- } else if (lcore_options.core_map_opt) {
+ lcore_options.core_map_opt = core_map_opt;
+ EAL_LOG(DEBUG, "Generated core map: '%s'", lcore_options.core_map_opt);
+ }
+
+ if (lcore_options.core_map_opt) {
if (eal_parse_lcores(lcore_options.core_map_opt) < 0) {
EAL_LOG(ERR, "invalid parameter for --" OPT_LCORES);
return -1;
}
+ if (lcore_options.core_mask_opt || lcore_options.core_list_opt)
+ free(RTE_CAST_PTR(void *, lcore_options.core_map_opt));
} else {
eal_auto_detect_cores(cfg);
}
diff --git a/lib/eal/include/rte_eal.h b/lib/eal/include/rte_eal.h
index c826e143f1..013075487c 100644
--- a/lib/eal/include/rte_eal.h
+++ b/lib/eal/include/rte_eal.h
@@ -493,22 +493,20 @@ rte_eal_get_runtime_dir(void);
/**
* Convert a string describing a mask of core ids into an array of core ids.
*
- * On success, the passed array is filled with the orders of the core ids
- * present in the mask (-1 indicating that a core id is absent).
- * For example, passing a 0xa coremask results in cores[1] = 0, cores[3] = 1,
- * and the rest of the array is set to -1.
+ * On success, the passed array is filled with the core ids present in the mask.
*
* @param coremask
* A string describing a mask of core ids.
* @param cores
- * An array where to store the core ids orders.
+ * The output array to store the core ids.
* This array must be at least RTE_MAX_LCORE large.
* @return
- * 0 on success, -1 if the string content was invalid.
+ * The number of cores in the coremask, and in the returned "cores" array,
+ * -1 if the string content was invalid.
*/
__rte_internal
int
-rte_eal_parse_coremask(const char *coremask, int *cores);
+rte_eal_expand_coremask(const char *coremask, int *cores);
#ifdef __cplusplus
}
diff --git a/lib/eal/version.map b/lib/eal/version.map
index a20c713eb1..b51051ee38 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -405,8 +405,8 @@ INTERNAL {
rte_bus_register;
rte_bus_unregister;
+ rte_eal_expand_coremask;
rte_eal_get_baseaddr;
- rte_eal_parse_coremask;
rte_firmware_read;
rte_intr_allow_others;
rte_intr_cap_multiple;
--
2.45.2
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 3/3] eal: allow automatic mapping of high lcore ids
2025-03-24 17:30 ` [PATCH v2 0/3] allow easier use of high lcore-ids Bruce Richardson
2025-03-24 17:30 ` [PATCH v2 1/3] eal: centralize core parameter parsing Bruce Richardson
2025-03-24 17:30 ` [PATCH v2 2/3] eal: convert core masks and lists to core sets Bruce Richardson
@ 2025-03-24 17:30 ` Bruce Richardson
2025-04-01 14:06 ` [PATCH v2 0/3] allow easier use of high lcore-ids Bruce Richardson
3 siblings, 0 replies; 31+ messages in thread
From: Bruce Richardson @ 2025-03-24 17:30 UTC (permalink / raw)
To: dev; +Cc: Bruce Richardson, Pravin Pathak, Tyler Retzlaff
To use cores with IDs greater than RTE_MAX_LCORE the user must provide a
mapping of those higher core numbers to ids within the 0 - RTE_MAX_LCORE
range. This can be awkward to do manually when more than a few lcores
are involved, so instead we can provide an extra option to EAL to do
this manually.
This patch therefore introduces the "map-lcore-ids" flag, which
automatically maps all provided lcore numbers to start at zero.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
doc/guides/linux_gsg/eal_args.include.rst | 26 ++++++++++++++++-
drivers/event/dlb2/pf/base/dlb2_resource.c | 2 +-
lib/eal/common/eal_common_options.c | 34 +++++++++++++++-------
lib/eal/common/eal_options.h | 3 ++
lib/eal/include/rte_eal.h | 4 ++-
5 files changed, 55 insertions(+), 14 deletions(-)
diff --git a/doc/guides/linux_gsg/eal_args.include.rst b/doc/guides/linux_gsg/eal_args.include.rst
index 9cfbf7de84..1d9525f73c 100644
--- a/doc/guides/linux_gsg/eal_args.include.rst
+++ b/doc/guides/linux_gsg/eal_args.include.rst
@@ -13,7 +13,7 @@ Lcore-related options
List of cores to run on
The argument format is ``<c1>[-c2][,c3[-c4],...]``
- where ``c1``, ``c2``, etc are core indexes between 0 and 128.
+ where ``c1``, ``c2``, etc are core indexes between 0 and RTE_MAX_LCORE (default 128).
* ``--lcores <core map>``
@@ -29,10 +29,34 @@ Lcore-related options
The grouping ``()`` can be omitted for single element group.
The ``@`` can be omitted if cpus and lcores have the same value.
+.. Note::
+ When using ``--lcores`` parameter to map lcore ids to cpus,
+ the lcore ids must be in the range 0 to ``RTE_MAX_LCORE`` (default 128),
+ but the cpu ids need not be.
+
+ This allows an application to use cpus with ids greater than ``RTE_MAX_LCORE``,
+ so long as the total number of lcores is less than or equal to ``RTE_MAX_LCORE``.
+
.. Note::
At a given instance only one core option ``--lcores``, ``-l`` or ``-c`` can
be used.
+* ``--map-lcore-ids``, ``-M``
+
+ Automatically map the cpus given in a coremask using ``-c`` parameter,
+ or in a core list using ``-l`` parameter, to internal lcore-ids starting at 0.
+ (If ``--lcores`` parameter is provided, this parameter is ignored.)
+
+ This can provide a shorter way, compared to using ``--lcores`` parameter,
+ to use cpus with ids greater than ``RTE_MAX_LCORE`` in an application.
+
+ For example, ``-c 0x3000 --map-lcore-ids`` will cause cpus 12 and 13 to be used,
+ with application lcore ids of 0 and 1 respectively.
+ Similarly, ``-Ml 94,95,31-33`` is equivalent to ``--lcores=0@94,1@95,2@31,3@32,4@33``,
+ where the application lcore ids are 0-4,
+ mapped to the physical cpus 94, 95, 31, 32 and 33 respectively.
+
+
* ``--main-lcore <core ID>``
Core ID that is used as main.
diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
index 9b087b03b5..d38aa5115e 100644
--- a/drivers/event/dlb2/pf/base/dlb2_resource.c
+++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
@@ -931,7 +931,7 @@ dlb2_resource_probe(struct dlb2_hw *hw, const void *probe_args)
cpu = rte_get_next_lcore(-1, 1, 0);
hw->num_prod_cores = 0;
if (mask) {
- int n = rte_eal_expand_coremask(mask, hw->prod_core_list);
+ int n = rte_eal_expand_coremask(mask, hw->prod_core_list, true);
if (n <= 0) {
DLB2_HW_ERR(hw, ": Invalid producer coremask=%s\n", mask);
return -1;
diff --git a/lib/eal/common/eal_common_options.c b/lib/eal/common/eal_common_options.c
index 22ea6b24e4..25061ffb9b 100644
--- a/lib/eal/common/eal_common_options.c
+++ b/lib/eal/common/eal_common_options.c
@@ -56,6 +56,7 @@ eal_short_options[] =
"d:" /* driver */
"h" /* help */
"l:" /* corelist */
+ "M" /* map lcore ids to zero*/
"S:" /* service corelist */
"m:" /* memory size */
"n:" /* memory channels */
@@ -106,6 +107,7 @@ eal_long_options[] = {
{OPT_NO_TELEMETRY, 0, NULL, OPT_NO_TELEMETRY_NUM },
{OPT_FORCE_MAX_SIMD_BITWIDTH, 1, NULL, OPT_FORCE_MAX_SIMD_BITWIDTH_NUM},
{OPT_HUGE_WORKER_STACK, 2, NULL, OPT_HUGE_WORKER_STACK_NUM },
+ {OPT_MAP_LCORE_IDS, 0, NULL, OPT_MAP_LCORE_IDS_NUM },
{0, 0, NULL, 0 }
};
@@ -152,6 +154,7 @@ TAILQ_HEAD_INITIALIZER(devopt_list);
static int main_lcore_parsed;
static int mem_parsed;
static struct lcore_options {
+ bool map_lcore_ids;
const char *core_mask_opt;
const char *core_list_opt;
const char *core_map_opt;
@@ -723,13 +726,14 @@ check_core_list(int *lcores, unsigned int count)
if (len > 0)
lcorestr[len - 1] = 0;
EAL_LOG(ERR, "To use high physical core ids, "
- "please use --lcores to map them to lcore ids below RTE_MAX_LCORE, "
+ "please use --map-lcore-ids/-M flag to map core ids automatically, "
+ "or use --lcores to map them manually to lcore ids below RTE_MAX_LCORE, "
"e.g. --lcores %s", lcorestr);
return -1;
}
int
-rte_eal_expand_coremask(const char *coremask, int *cores)
+rte_eal_expand_coremask(const char *coremask, int *cores, bool no_check)
{
const char *coremask_orig = coremask;
unsigned int count = 0;
@@ -784,7 +788,7 @@ rte_eal_expand_coremask(const char *coremask, int *cores)
return -1;
}
- if (check_core_list(cores, count) != 0)
+ if (!no_check && check_core_list(cores, count) != 0)
return -1;
return count;
@@ -869,7 +873,7 @@ eal_parse_service_corelist(const char *corelist)
}
static int
-eal_parse_corelist(const char *corelist, int *cores)
+eal_parse_corelist(const char *corelist, int *cores, bool no_check)
{
unsigned int count = 0, i;
char *end = NULL;
@@ -932,7 +936,7 @@ eal_parse_corelist(const char *corelist, int *cores)
return -1;
}
- if (check_core_list(cores, count))
+ if (!no_check && check_core_list(cores, count))
return -1;
return count;
@@ -1558,6 +1562,10 @@ eal_parse_common_option(int opt, const char *optarg,
case 'l':
lcore_options.core_list_opt = optarg;
break;
+ /* map lcore ids */
+ case 'M':
+ lcore_options.map_lcore_ids = true;
+ break;
/* service coremask */
case 's':
lcore_options.service_mask_opt = optarg;
@@ -1866,10 +1874,11 @@ eal_adjust_config(struct internal_config *internal_cfg)
if (lcore_options.core_mask_opt || lcore_options.core_list_opt) {
- int lcore_indexes[RTE_MAX_LCORE];
+ bool map_ids = lcore_options.map_lcore_ids;
+ int idxs[RTE_MAX_LCORE];
int nb_indexes = lcore_options.core_list_opt ?
- eal_parse_corelist(lcore_options.core_list_opt, lcore_indexes) :
- rte_eal_expand_coremask(lcore_options.core_mask_opt, lcore_indexes);
+ eal_parse_corelist(lcore_options.core_list_opt, idxs, map_ids) :
+ rte_eal_expand_coremask(lcore_options.core_mask_opt, idxs, map_ids);
if (nb_indexes < 0)
return -1;
@@ -1877,13 +1886,16 @@ eal_adjust_config(struct internal_config *internal_cfg)
char *core_map_opt = malloc(RTE_MAX_LCORE * 10);
size_t core_map_len = 0;
for (i = 0; i < nb_indexes; i++) {
- if (!eal_cpu_detected(lcore_indexes[i])) {
- EAL_LOG(ERR, "core %d not present", lcore_indexes[i]);
+ if (!eal_cpu_detected(idxs[i])) {
+ EAL_LOG(ERR, "core %d not present", idxs[i]);
return -1;
}
int n = snprintf(core_map_opt + core_map_len,
(RTE_MAX_LCORE * 10) - core_map_len,
- "%s%d", i == 0 ? "" : ",", lcore_indexes[i]);
+ "%s%d@%d",
+ i == 0 ? "" : ",",
+ map_ids ? i : idxs[i],
+ idxs[i]);
if (n < 0 || (size_t)n >= (RTE_MAX_LCORE * 10) - core_map_len) {
EAL_LOG(ERR, "core map string too long");
return -1;
diff --git a/lib/eal/common/eal_options.h b/lib/eal/common/eal_options.h
index 95fb4f6108..1344bce9ba 100644
--- a/lib/eal/common/eal_options.h
+++ b/lib/eal/common/eal_options.h
@@ -17,6 +17,9 @@ enum {
OPT_DEV_ALLOW_NUM = 'a',
#define OPT_DEV_BLOCK "block"
OPT_DEV_BLOCK_NUM = 'b',
+#define OPT_MAP_LCORE_IDS "map-lcore-ids"
+ OPT_MAP_LCORE_IDS_NUM = 'M',
+
/* first long only option value must be >= 256, so that we won't
* conflict with short options */
diff --git a/lib/eal/include/rte_eal.h b/lib/eal/include/rte_eal.h
index 013075487c..2f7a7dcfbe 100644
--- a/lib/eal/include/rte_eal.h
+++ b/lib/eal/include/rte_eal.h
@@ -500,13 +500,15 @@ rte_eal_get_runtime_dir(void);
* @param cores
* The output array to store the core ids.
* This array must be at least RTE_MAX_LCORE large.
+ * @param nocheck
+ * If true, skip checking that the core ids are between 0..RTE_MAX_LCORE
* @return
* The number of cores in the coremask, and in the returned "cores" array,
* -1 if the string content was invalid.
*/
__rte_internal
int
-rte_eal_expand_coremask(const char *coremask, int *cores);
+rte_eal_expand_coremask(const char *coremask, int *cores, bool nocheck);
#ifdef __cplusplus
}
--
2.45.2
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v2 0/3] allow easier use of high lcore-ids
2025-03-24 17:30 ` [PATCH v2 0/3] allow easier use of high lcore-ids Bruce Richardson
` (2 preceding siblings ...)
2025-03-24 17:30 ` [PATCH v2 3/3] eal: allow automatic mapping of high lcore ids Bruce Richardson
@ 2025-04-01 14:06 ` Bruce Richardson
2025-04-07 7:04 ` David Marchand
3 siblings, 1 reply; 31+ messages in thread
From: Bruce Richardson @ 2025-04-01 14:06 UTC (permalink / raw)
To: dev
On Mon, Mar 24, 2025 at 05:30:26PM +0000, Bruce Richardson wrote:
> Traditionally, DPDK has had a direct mapping of internal lcore-ids, to
> the actual core numbers in use. With higher core count servers becoming
> more prevalent the issue becomes one of increasing memory footprint when
> using such a scheme, due to the need to have all arrays dimensioned for
> all cores on the system, whether or not those cores are in use by the
> app.
>
> Therefore, the decision was made in the past to not expand the
> build-time RTE_MAX_LCORE value beyond 128. Instead, it was recommended
> that users use the "--lcores" EAL parameter to take the high-numbered
> cores they wish to use and map them to lcore-ids within the 0 - 128
> range. While this works, this is a little clunky as it means that
> instead of just passing, for example, "-l 130-139", the user must
> instead pass "--lcores 0@130,1@131,2@132,3@133,...."
>
> This patchset attempts to simplify the situation by adding a new flag to
> do this mapping automatically. To use cores 130-139 and map them to ids
> 0-9 internally, the EAL args now become: "-l 130-139 --map-lcore-ids",
> or using the shorter "-M" version of the flag: "-Ml 130-139".
>
> Adding this new parameter required some rework of the existing arg
> parsing code, because in current DPDK the args are parsed and checked in
> the order they appear on the commandline. This means that using the
> example above, the core parameter 130-139 will be rejected immediately
> before the "map-lcore-ids" parameter is seen. To work around this, the
> core (and service core) parameters are not parsed when seen, instead
> they are only saved off and parsed after all arguments are parsed. The
> "-l" and "-c" parameters are converted into "--lcores" arguments, so all
> assigning of lcore ids is done there in all cases.
>
> RFC->v2:
> * converted printf to DEBUG log
> * added "-M" as shorter version of flag
> * added documentation
> * renamed internal API that was changed to avoid any potential hidden
> runtime issues.
>
> Bruce Richardson (3):
> eal: centralize core parameter parsing
> eal: convert core masks and lists to core sets
> eal: allow automatic mapping of high lcore ids
>
Ping for review.
At a high level, does this feature seem useful to users?
/Bruce
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v2 1/3] eal: centralize core parameter parsing
2025-03-24 17:30 ` [PATCH v2 1/3] eal: centralize core parameter parsing Bruce Richardson
@ 2025-04-07 6:58 ` David Marchand
0 siblings, 0 replies; 31+ messages in thread
From: David Marchand @ 2025-04-07 6:58 UTC (permalink / raw)
To: Bruce Richardson; +Cc: dev, Tyler Retzlaff
On Mon, Mar 24, 2025 at 6:31 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> Rather than parsing the lcore parameters as they are encountered, just
> save off the lcore parameters and then parse them at the end. This
> allows better knowledge of what parameters are available or not when
> parsing.
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
> lib/eal/common/eal_common_options.c | 183 +++++++++++++---------------
> 1 file changed, 84 insertions(+), 99 deletions(-)
>
> diff --git a/lib/eal/common/eal_common_options.c b/lib/eal/common/eal_common_options.c
> index 79db9a47dd..55c49a923f 100644
> --- a/lib/eal/common/eal_common_options.c
> +++ b/lib/eal/common/eal_common_options.c
> @@ -151,7 +151,13 @@ TAILQ_HEAD_INITIALIZER(devopt_list);
>
> static int main_lcore_parsed;
> static int mem_parsed;
> -static int core_parsed;
> +static struct lcore_options {
> + const char *core_mask_opt;
> + const char *core_list_opt;
> + const char *core_map_opt;
> + const char *service_mask_opt;
> + const char *service_list_opt;
> +} lcore_options = {0};
eal_parse_main_lcore() still checks if selected main lcore is a
service lcore. I think this check is useless now.
>
> /* Allow the application to print its usage message too if set */
> static rte_usage_hook_t rte_application_usage_hook;
> @@ -674,7 +680,7 @@ eal_parse_service_coremask(const char *coremask)
> if (count == 0)
> return -1;
>
> - if (core_parsed && taken_lcore_count != count) {
> + if (taken_lcore_count != count) {
> EAL_LOG(WARNING,
> "Not all service cores are in the coremask. "
> "Please ensure -c or -l includes service cores");
> @@ -684,17 +690,6 @@ eal_parse_service_coremask(const char *coremask)
> return 0;
> }
>
> -static int
> -eal_service_cores_parsed(void)
> -{
> - int idx;
> - for (idx = 0; idx < RTE_MAX_LCORE; idx++) {
> - if (lcore_config[idx].core_role == ROLE_SERVICE)
> - return 1;
> - }
> - return 0;
> -}
> -
> static int
> update_lcore_config(int *cores)
> {
> @@ -903,7 +898,7 @@ eal_parse_service_corelist(const char *corelist)
> if (count == 0)
> return -1;
>
> - if (core_parsed && taken_lcore_count != count) {
> + if (taken_lcore_count != count) {
> EAL_LOG(WARNING,
> "Not all service cores were in the coremask. "
> "Please ensure -c or -l includes service cores");
> @@ -1673,83 +1668,20 @@ eal_parse_common_option(int opt, const char *optarg,
> a_used = 1;
> break;
> /* coremask */
> - case 'c': {
> - int lcore_indexes[RTE_MAX_LCORE];
> -
> - if (eal_service_cores_parsed())
> - EAL_LOG(WARNING,
> - "Service cores parsed before dataplane cores. Please ensure -c is before -s or -S");
> - if (rte_eal_parse_coremask(optarg, lcore_indexes) < 0) {
> - EAL_LOG(ERR, "invalid coremask syntax");
> - return -1;
> - }
> - if (update_lcore_config(lcore_indexes) < 0) {
> - char *available = available_cores();
> -
> - EAL_LOG(ERR,
> - "invalid coremask, please check specified cores are part of %s",
> - available);
> - free(available);
> - return -1;
> - }
> -
> - if (core_parsed) {
> - EAL_LOG(ERR, "Option -c is ignored, because (%s) is set!",
> - (core_parsed == LCORE_OPT_LST) ? "-l" :
> - (core_parsed == LCORE_OPT_MAP) ? "--lcores" :
> - "-c");
> - return -1;
> - }
> -
> - core_parsed = LCORE_OPT_MSK;
> + case 'c':
> + lcore_options.core_mask_opt = optarg;
> break;
> - }
> /* corelist */
> - case 'l': {
> - int lcore_indexes[RTE_MAX_LCORE];
> -
> - if (eal_service_cores_parsed())
> - EAL_LOG(WARNING,
> - "Service cores parsed before dataplane cores. Please ensure -l is before -s or -S");
> -
> - if (eal_parse_corelist(optarg, lcore_indexes) < 0) {
> - EAL_LOG(ERR, "invalid core list syntax");
> - return -1;
> - }
> - if (update_lcore_config(lcore_indexes) < 0) {
> - char *available = available_cores();
> -
> - EAL_LOG(ERR,
> - "invalid core list, please check specified cores are part of %s",
> - available);
> - free(available);
> - return -1;
> - }
> -
> - if (core_parsed) {
> - EAL_LOG(ERR, "Option -l is ignored, because (%s) is set!",
> - (core_parsed == LCORE_OPT_MSK) ? "-c" :
> - (core_parsed == LCORE_OPT_MAP) ? "--lcores" :
> - "-l");
> - return -1;
> - }
> -
> - core_parsed = LCORE_OPT_LST;
> + case 'l':
> + lcore_options.core_list_opt = optarg;
> break;
> - }
> /* service coremask */
> case 's':
> - if (eal_parse_service_coremask(optarg) < 0) {
> - EAL_LOG(ERR, "invalid service coremask");
> - return -1;
> - }
> + lcore_options.service_mask_opt = optarg;
> break;
> /* service corelist */
> case 'S':
> - if (eal_parse_service_corelist(optarg) < 0) {
> - EAL_LOG(ERR, "invalid service core list");
> - return -1;
> - }
> + lcore_options.service_list_opt = optarg;
> break;
> /* size of memory */
> case 'm':
> @@ -1918,21 +1850,7 @@ eal_parse_common_option(int opt, const char *optarg,
> #endif /* !RTE_EXEC_ENV_WINDOWS */
>
> case OPT_LCORES_NUM:
> - if (eal_parse_lcores(optarg) < 0) {
> - EAL_LOG(ERR, "invalid parameter for --"
> - OPT_LCORES);
> - return -1;
> - }
> -
> - if (core_parsed) {
> - EAL_LOG(ERR, "Option --lcores is ignored, because (%s) is set!",
> - (core_parsed == LCORE_OPT_LST) ? "-l" :
> - (core_parsed == LCORE_OPT_MSK) ? "-c" :
> - "--lcores");
> - return -1;
> - }
> -
> - core_parsed = LCORE_OPT_MAP;
> + lcore_options.core_map_opt = optarg;
> break;
> case OPT_LEGACY_MEM_NUM:
> conf->legacy_mem = 1;
> @@ -1973,6 +1891,7 @@ eal_parse_common_option(int opt, const char *optarg,
>
> }
>
> +
Unneeded empty line.
> return 0;
>
> ba_conflict:
> @@ -2046,8 +1965,74 @@ eal_adjust_config(struct internal_config *internal_cfg)
> struct internal_config *internal_conf =
> eal_get_internal_configuration();
>
> - if (!core_parsed)
> + /* handle all the various core options, ignoring order of them.
Handle
> + * First check that multiple lcore options (coremask, corelist, coremap) are
> + * not used together. Then check that the service options (coremask, corelist)
> + * are not both set. In both cases error out if multiple options are set.
> + */
> + if ((lcore_options.core_mask_opt && lcore_options.core_list_opt) ||
> + (lcore_options.core_mask_opt && lcore_options.core_map_opt) ||
> + (lcore_options.core_list_opt && lcore_options.core_map_opt)) {
> + EAL_LOG(ERR, "Only one of -c, -l and --"OPT_LCORES" may be given at a time");
> + return -1;
> + }
> + if ((lcore_options.service_mask_opt && lcore_options.service_list_opt)) {
> + EAL_LOG(ERR, "Only one of -s and -S may be given at a time");
> + return -1;
> + }
> +
> + if (lcore_options.core_mask_opt) {
> + int lcore_indexes[RTE_MAX_LCORE];
> +
> + if (rte_eal_parse_coremask(lcore_options.core_mask_opt, lcore_indexes) < 0) {
> + EAL_LOG(ERR, "invalid coremask syntax");
> + return -1;
> + }
> + if (update_lcore_config(lcore_indexes) < 0) {
> + char *available = available_cores();
> +
> + EAL_LOG(ERR,
> + "invalid coremask, please check specified cores are part of %s",
> + available);
> + free(available);
> + return -1;
> + }
> + } else if (lcore_options.core_list_opt) {
> + int lcore_indexes[RTE_MAX_LCORE];
> +
> + if (eal_parse_corelist(lcore_options.core_list_opt, lcore_indexes) < 0) {
> + EAL_LOG(ERR, "invalid core list syntax");
> + return -1;
> + }
> + if (update_lcore_config(lcore_indexes) < 0) {
> + char *available = available_cores();
> +
> + EAL_LOG(ERR,
> + "invalid core list, please check specified cores are part of %s",
> + available);
> + free(available);
> + return -1;
> + }
> + } else if (lcore_options.core_map_opt) {
> + if (eal_parse_lcores(lcore_options.core_map_opt) < 0) {
> + EAL_LOG(ERR, "invalid parameter for --" OPT_LCORES);
> + return -1;
> + }
> + } else {
> eal_auto_detect_cores(cfg);
> + }
> +
> + if (lcore_options.service_mask_opt) {
> + if (eal_parse_service_coremask(lcore_options.service_mask_opt) < 0) {
> + EAL_LOG(ERR, "invalid service coremask");
> + return -1;
> + }
> + } else if (lcore_options.service_list_opt) {
> + if (eal_parse_service_corelist(lcore_options.service_list_opt) < 0) {
> + EAL_LOG(ERR, "invalid service core list");
> + return -1;
> + }
> + }
>
> if (internal_conf->process_type == RTE_PROC_AUTO)
> internal_conf->process_type = eal_proc_type_detect();
> --
> 2.45.2
>
--
David Marchand
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v2 2/3] eal: convert core masks and lists to core sets
2025-03-24 17:30 ` [PATCH v2 2/3] eal: convert core masks and lists to core sets Bruce Richardson
@ 2025-04-07 6:59 ` David Marchand
0 siblings, 0 replies; 31+ messages in thread
From: David Marchand @ 2025-04-07 6:59 UTC (permalink / raw)
To: Bruce Richardson; +Cc: dev, Pravin Pathak, Tyler Retzlaff
On Mon, Mar 24, 2025 at 6:31 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> Rather than directly parsing and working with the core mask and core
> list parameters, convert them into core maps, and centralise all core
> parsing there. This allows future work to adjust the mappings of cores
> when generating that mapping parameter.
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
>
> NOTE: this patch changes the behaviour of the exported, but internal API
> rte_eal_parse_core_mask, used by dlb driver. Since this is an internal
> API changing behaviour is allowed. However, to prevent issues - in case
> any external app is using the API - we rename the function to
> rte_eal_expand_coremask.
An external app is not supposed to call this but I agree a build error
is more explicit.
> ---
> drivers/event/dlb2/dlb2_priv.h | 2 -
> drivers/event/dlb2/pf/base/dlb2_resource.c | 48 ++---
> lib/eal/common/eal_common_options.c | 195 +++++----------------
> lib/eal/include/rte_eal.h | 12 +-
> lib/eal/version.map | 2 +-
> 5 files changed, 60 insertions(+), 199 deletions(-)
>
> diff --git a/drivers/event/dlb2/dlb2_priv.h b/drivers/event/dlb2/dlb2_priv.h
> index 52da31ed31..63c0bea155 100644
> --- a/drivers/event/dlb2/dlb2_priv.h
> +++ b/drivers/event/dlb2/dlb2_priv.h
> @@ -737,8 +737,6 @@ void dlb2_event_build_hcws(struct dlb2_port *qm_port,
> uint8_t *sched_type,
> uint8_t *queue_id);
>
> -/* Extern functions */
> -extern int rte_eal_parse_coremask(const char *coremask, int *cores);
Ugly...
>
> /* Extern globals */
> extern struct process_local_port_data dlb2_port[][DLB2_NUM_PORT_TYPES];
> diff --git a/drivers/event/dlb2/pf/base/dlb2_resource.c b/drivers/event/dlb2/pf/base/dlb2_resource.c
> index 3004902118..9b087b03b5 100644
> --- a/drivers/event/dlb2/pf/base/dlb2_resource.c
> +++ b/drivers/event/dlb2/pf/base/dlb2_resource.c
> @@ -922,49 +922,31 @@ dlb2_resource_probe(struct dlb2_hw *hw, const void *probe_args)
> {
> const struct dlb2_devargs *args = (const struct dlb2_devargs *)probe_args;
> const char *mask = args ? args->producer_coremask : NULL;
> - int cpu = 0, cnt = 0, cores[RTE_MAX_LCORE], i;
> + int cpu = 0;
>
> if (args) {
> mask = (const char *)args->producer_coremask;
> }
I have some trouble with this code:
const char *mask = args ? args->producer_coremask : NULL;
...
if (args) {
mask = (const char *)args->producer_coremask;
}
It seems redundant...
>
> - if (mask && rte_eal_parse_coremask(mask, cores)) {
> - DLB2_HW_ERR(hw, ": Invalid producer coremask=%s\n", mask);
> - return -1;
> - }
> -
> + cpu = rte_get_next_lcore(-1, 1, 0);
> hw->num_prod_cores = 0;
> - for (i = 0; i < RTE_MAX_LCORE; i++) {
> - bool is_pcore = (mask && cores[i] != -1);
> -
> - if (rte_lcore_is_enabled(i)) {
> - if (is_pcore) {
> - /*
> - * Populate the producer cores from parsed
> - * coremask
> - */
> - hw->prod_core_list[cores[i]] = i;
> - hw->num_prod_cores++;
> -
> - } else if ((++cnt == DLB2_EAL_PROBE_CORE ||
> - rte_lcore_count() < DLB2_EAL_PROBE_CORE)) {
> - /*
> - * If no producer coremask is provided, use the
> - * second EAL core to probe
> - */
> - cpu = i;
> - break;
> - }
> - } else if (is_pcore) {
> - DLB2_HW_ERR(hw, "Producer coremask(%s) must be a subset of EAL coremask\n",
> - mask);
> + if (mask) {
> + int n = rte_eal_expand_coremask(mask, hw->prod_core_list);
> + if (n <= 0) {
> + DLB2_HW_ERR(hw, ": Invalid producer coremask=%s\n", mask);
> return -1;
> }
> + hw->num_prod_cores = n;
> + cpu = hw->prod_core_list[0];
>
> + for (u8 i = 0; i < hw->num_prod_cores; i++) {
> + if (!rte_lcore_is_enabled(hw->prod_core_list[i])) {
> + DLB2_HW_ERR(hw, "Producer coremask(%s) must be a subset of EAL coremask\n",
> + mask);
> + return -1;
> + }
> + }
> }
> - /* Use the first core in producer coremask to probe */
> - if (hw->num_prod_cores)
> - cpu = hw->prod_core_list[0];
>
> dlb2_get_pp_allocation(hw, cpu, DLB2_LDB_PORT);
> dlb2_get_pp_allocation(hw, cpu, DLB2_DIR_PORT);
> diff --git a/lib/eal/common/eal_common_options.c b/lib/eal/common/eal_common_options.c
> index 55c49a923f..22ea6b24e4 100644
> --- a/lib/eal/common/eal_common_options.c
> +++ b/lib/eal/common/eal_common_options.c
> @@ -690,33 +690,6 @@ eal_parse_service_coremask(const char *coremask)
> return 0;
> }
>
> -static int
> -update_lcore_config(int *cores)
> -{
> - struct rte_config *cfg = rte_eal_get_configuration();
> - unsigned int count = 0;
> - unsigned int i;
> - int ret = 0;
> -
> - for (i = 0; i < RTE_MAX_LCORE; i++) {
> - if (cores[i] != -1) {
> - if (eal_cpu_detected(i) == 0) {
> - EAL_LOG(ERR, "lcore %u unavailable", i);
> - ret = -1;
> - continue;
> - }
> - cfg->lcore_role[i] = ROLE_RTE;
> - count++;
> - } else {
> - cfg->lcore_role[i] = ROLE_OFF;
> - }
> - lcore_config[i].core_index = cores[i];
> - }
> - if (!ret)
> - cfg->lcore_count = count;
> - return ret;
> -}
> -
> static int
> check_core_list(int *lcores, unsigned int count)
> {
> @@ -756,10 +729,9 @@ check_core_list(int *lcores, unsigned int count)
> }
>
> int
> -rte_eal_parse_coremask(const char *coremask, int *cores)
> +rte_eal_expand_coremask(const char *coremask, int *cores)
> {
> const char *coremask_orig = coremask;
> - int lcores[RTE_MAX_LCORE];
> unsigned int count = 0;
> int i, j, idx;
> int val;
> @@ -803,30 +775,19 @@ rte_eal_parse_coremask(const char *coremask, int *cores)
> RTE_MAX_LCORE);
> return -1;
> }
> - lcores[count++] = idx;
> + cores[count++] = idx;
> }
> }
> }
> if (count == 0) {
> - EAL_LOG(ERR, "No lcores in coremask: [%s]",
> - coremask_orig);
> + EAL_LOG(ERR, "No lcores in coremask: [%s]", coremask_orig);
> return -1;
> }
>
> - if (check_core_list(lcores, count))
> + if (check_core_list(cores, count) != 0)
> return -1;
>
> - /*
> - * Now that we've got a list of cores no longer than RTE_MAX_LCORE,
> - * and no lcore in that list is greater than RTE_MAX_LCORE, populate
> - * the cores array.
> - */
> - do {
> - count--;
> - cores[lcores[count]] = count;
> - } while (count != 0);
> -
> - return 0;
> + return count;
> }
>
> static int
> @@ -911,7 +872,6 @@ static int
> eal_parse_corelist(const char *corelist, int *cores)
> {
> unsigned int count = 0, i;
> - int lcores[RTE_MAX_LCORE];
> char *end = NULL;
> int min, max;
> int idx;
> @@ -949,7 +909,7 @@ eal_parse_corelist(const char *corelist, int *cores)
>
> /* Check if this idx is already present */
> for (i = 0; i < count; i++) {
> - if (lcores[i] == idx)
> + if (cores[i] == idx)
> dup = true;
> }
> if (dup)
> @@ -959,7 +919,7 @@ eal_parse_corelist(const char *corelist, int *cores)
> RTE_MAX_LCORE);
> return -1;
> }
> - lcores[count++] = idx;
> + cores[count++] = idx;
> }
> min = -1;
> } else
> @@ -967,23 +927,15 @@ eal_parse_corelist(const char *corelist, int *cores)
> corelist = end + 1;
> } while (*end != '\0');
>
> - if (count == 0)
> + if (count == 0) {
> + EAL_LOG(ERR, "No lcores in corelist");
> return -1;
> + }
>
> - if (check_core_list(lcores, count))
> + if (check_core_list(cores, count))
> return -1;
>
> - /*
> - * Now that we've got a list of cores no longer than RTE_MAX_LCORE,
> - * and no lcore in that list is greater than RTE_MAX_LCORE, populate
> - * the cores array.
> - */
> - do {
> - count--;
> - cores[lcores[count]] = count;
> - } while (count != 0);
> -
> - return 0;
> + return count;
> }
>
> /* Changes the lcore id of the main thread */
> @@ -1500,75 +1452,6 @@ eal_parse_base_virtaddr(const char *arg)
> return 0;
> }
>
> -/* caller is responsible for freeing the returned string */
> -static char *
> -available_cores(void)
> -{
> - char *str = NULL;
> - int previous;
> - int sequence;
> - char *tmp;
> - int idx;
> -
> - /* find the first available cpu */
> - for (idx = 0; idx < RTE_MAX_LCORE; idx++) {
> - if (eal_cpu_detected(idx) == 0)
> - continue;
> - break;
> - }
> - if (idx >= RTE_MAX_LCORE)
> - return NULL;
> -
> - /* first sequence */
> - if (asprintf(&str, "%d", idx) < 0)
> - return NULL;
> - previous = idx;
> - sequence = 0;
> -
> - for (idx++ ; idx < RTE_MAX_LCORE; idx++) {
> - if (eal_cpu_detected(idx) == 0)
> - continue;
> -
> - if (idx == previous + 1) {
> - previous = idx;
> - sequence = 1;
> - continue;
> - }
> -
> - /* finish current sequence */
> - if (sequence) {
> - if (asprintf(&tmp, "%s-%d", str, previous) < 0) {
> - free(str);
> - return NULL;
> - }
> - free(str);
> - str = tmp;
> - }
> -
> - /* new sequence */
> - if (asprintf(&tmp, "%s,%d", str, idx) < 0) {
> - free(str);
> - return NULL;
> - }
> - free(str);
> - str = tmp;
> - previous = idx;
> - sequence = 0;
> - }
> -
> - /* finish last sequence */
> - if (sequence) {
> - if (asprintf(&tmp, "%s-%d", str, previous) < 0) {
> - free(str);
> - return NULL;
> - }
> - free(str);
> - str = tmp;
> - }
> -
> - return str;
> -}
> -
Not sure why you remove this helper, it was supposed to help users
with better logs.
> #define HUGE_UNLINK_NEVER "never"
>
> static int
> @@ -1981,43 +1864,43 @@ eal_adjust_config(struct internal_config *internal_cfg)
> return -1;
> }
>
> - if (lcore_options.core_mask_opt) {
> - int lcore_indexes[RTE_MAX_LCORE];
> -
> - if (rte_eal_parse_coremask(lcore_options.core_mask_opt, lcore_indexes) < 0) {
> - EAL_LOG(ERR, "invalid coremask syntax");
> - return -1;
> - }
> - if (update_lcore_config(lcore_indexes) < 0) {
> - char *available = available_cores();
>
> - EAL_LOG(ERR,
> - "invalid coremask, please check specified cores are part of %s",
> - available);
> - free(available);
> - return -1;
> - }
> - } else if (lcore_options.core_list_opt) {
> + if (lcore_options.core_mask_opt || lcore_options.core_list_opt) {
> int lcore_indexes[RTE_MAX_LCORE];
> + int nb_indexes = lcore_options.core_list_opt ?
> + eal_parse_corelist(lcore_options.core_list_opt, lcore_indexes) :
> + rte_eal_expand_coremask(lcore_options.core_mask_opt, lcore_indexes);
>
> - if (eal_parse_corelist(lcore_options.core_list_opt, lcore_indexes) < 0) {
> - EAL_LOG(ERR, "invalid core list syntax");
> + if (nb_indexes < 0)
> return -1;
> - }
> - if (update_lcore_config(lcore_indexes) < 0) {
> - char *available = available_cores();
>
> - EAL_LOG(ERR,
> - "invalid core list, please check specified cores are part of %s",
> - available);
> - free(available);
> - return -1;
> + char *core_map_opt = malloc(RTE_MAX_LCORE * 10);
> + size_t core_map_len = 0;
> + for (i = 0; i < nb_indexes; i++) {
> + if (!eal_cpu_detected(lcore_indexes[i])) {
> + EAL_LOG(ERR, "core %d not present", lcore_indexes[i]);
core_map_opt is leaked (idem in next return).
> + return -1;
I would not return here, but instead provide a full report of which
cores are invalid before returning an error.
> + }
> + int n = snprintf(core_map_opt + core_map_len,
> + (RTE_MAX_LCORE * 10) - core_map_len,
> + "%s%d", i == 0 ? "" : ",", lcore_indexes[i]);
> + if (n < 0 || (size_t)n >= (RTE_MAX_LCORE * 10) - core_map_len) {
> + EAL_LOG(ERR, "core map string too long");
> + return -1;
Or use asprintf.
> + }
> + core_map_len += n;
> }
> - } else if (lcore_options.core_map_opt) {
> + lcore_options.core_map_opt = core_map_opt;
> + EAL_LOG(DEBUG, "Generated core map: '%s'", lcore_options.core_map_opt);
> + }
> +
> + if (lcore_options.core_map_opt) {
> if (eal_parse_lcores(lcore_options.core_map_opt) < 0) {
> EAL_LOG(ERR, "invalid parameter for --" OPT_LCORES);
> return -1;
> }
> + if (lcore_options.core_mask_opt || lcore_options.core_list_opt)
> + free(RTE_CAST_PTR(void *, lcore_options.core_map_opt));
> } else {
> eal_auto_detect_cores(cfg);
> }
> diff --git a/lib/eal/include/rte_eal.h b/lib/eal/include/rte_eal.h
> index c826e143f1..013075487c 100644
> --- a/lib/eal/include/rte_eal.h
> +++ b/lib/eal/include/rte_eal.h
> @@ -493,22 +493,20 @@ rte_eal_get_runtime_dir(void);
> /**
> * Convert a string describing a mask of core ids into an array of core ids.
> *
> - * On success, the passed array is filled with the orders of the core ids
> - * present in the mask (-1 indicating that a core id is absent).
> - * For example, passing a 0xa coremask results in cores[1] = 0, cores[3] = 1,
> - * and the rest of the array is set to -1.
> + * On success, the passed array is filled with the core ids present in the mask.
> *
> * @param coremask
> * A string describing a mask of core ids.
> * @param cores
> - * An array where to store the core ids orders.
> + * The output array to store the core ids.
> * This array must be at least RTE_MAX_LCORE large.
> * @return
> - * 0 on success, -1 if the string content was invalid.
> + * The number of cores in the coremask, and in the returned "cores" array,
> + * -1 if the string content was invalid.
> */
> __rte_internal
> int
> -rte_eal_parse_coremask(const char *coremask, int *cores);
> +rte_eal_expand_coremask(const char *coremask, int *cores);
>
> #ifdef __cplusplus
> }
> diff --git a/lib/eal/version.map b/lib/eal/version.map
> index a20c713eb1..b51051ee38 100644
> --- a/lib/eal/version.map
> +++ b/lib/eal/version.map
> @@ -405,8 +405,8 @@ INTERNAL {
>
> rte_bus_register;
> rte_bus_unregister;
> + rte_eal_expand_coremask;
> rte_eal_get_baseaddr;
> - rte_eal_parse_coremask;
> rte_firmware_read;
> rte_intr_allow_others;
> rte_intr_cap_multiple;
> --
> 2.45.2
>
--
David Marchand
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v2 0/3] allow easier use of high lcore-ids
2025-04-01 14:06 ` [PATCH v2 0/3] allow easier use of high lcore-ids Bruce Richardson
@ 2025-04-07 7:04 ` David Marchand
2025-04-07 9:48 ` Bruce Richardson
0 siblings, 1 reply; 31+ messages in thread
From: David Marchand @ 2025-04-07 7:04 UTC (permalink / raw)
To: Bruce Richardson; +Cc: dev
Hello Bruce,
On Tue, Apr 1, 2025 at 4:08 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Mon, Mar 24, 2025 at 05:30:26PM +0000, Bruce Richardson wrote:
> > Traditionally, DPDK has had a direct mapping of internal lcore-ids, to
> > the actual core numbers in use. With higher core count servers becoming
> > more prevalent the issue becomes one of increasing memory footprint when
> > using such a scheme, due to the need to have all arrays dimensioned for
> > all cores on the system, whether or not those cores are in use by the
> > app.
> >
> > Therefore, the decision was made in the past to not expand the
> > build-time RTE_MAX_LCORE value beyond 128. Instead, it was recommended
> > that users use the "--lcores" EAL parameter to take the high-numbered
> > cores they wish to use and map them to lcore-ids within the 0 - 128
> > range. While this works, this is a little clunky as it means that
> > instead of just passing, for example, "-l 130-139", the user must
> > instead pass "--lcores 0@130,1@131,2@132,3@133,...."
> >
> > This patchset attempts to simplify the situation by adding a new flag to
> > do this mapping automatically. To use cores 130-139 and map them to ids
> > 0-9 internally, the EAL args now become: "-l 130-139 --map-lcore-ids",
> > or using the shorter "-M" version of the flag: "-Ml 130-139".
> >
> > Adding this new parameter required some rework of the existing arg
> > parsing code, because in current DPDK the args are parsed and checked in
> > the order they appear on the commandline. This means that using the
> > example above, the core parameter 130-139 will be rejected immediately
> > before the "map-lcore-ids" parameter is seen. To work around this, the
> > core (and service core) parameters are not parsed when seen, instead
> > they are only saved off and parsed after all arguments are parsed. The
> > "-l" and "-c" parameters are converted into "--lcores" arguments, so all
> > assigning of lcore ids is done there in all cases.
> >
> > RFC->v2:
> > * converted printf to DEBUG log
> > * added "-M" as shorter version of flag
> > * added documentation
> > * renamed internal API that was changed to avoid any potential hidden
> > runtime issues.
> >
> > Bruce Richardson (3):
> > eal: centralize core parameter parsing
> > eal: convert core masks and lists to core sets
> > eal: allow automatic mapping of high lcore ids
> >
> Ping for review.
>
> At a high level, does this feature seem useful to users?
This seems useful, though I am not I would touch the existing options.
I would have gone with a simple -L option (taking the same kind of
input than -l but with new behavior), and not combine a flag with
existing options.
I scanned through the series, not much to say.
Maybe add a unit test for new cmdline option.
--
David Marchand
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v2 0/3] allow easier use of high lcore-ids
2025-04-07 7:04 ` David Marchand
@ 2025-04-07 9:48 ` Bruce Richardson
2025-04-07 10:15 ` Morten Brørup
0 siblings, 1 reply; 31+ messages in thread
From: Bruce Richardson @ 2025-04-07 9:48 UTC (permalink / raw)
To: David Marchand; +Cc: dev
On Mon, Apr 07, 2025 at 09:04:05AM +0200, David Marchand wrote:
> Hello Bruce,
>
> On Tue, Apr 1, 2025 at 4:08 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > On Mon, Mar 24, 2025 at 05:30:26PM +0000, Bruce Richardson wrote:
> > > Traditionally, DPDK has had a direct mapping of internal lcore-ids, to
> > > the actual core numbers in use. With higher core count servers becoming
> > > more prevalent the issue becomes one of increasing memory footprint when
> > > using such a scheme, due to the need to have all arrays dimensioned for
> > > all cores on the system, whether or not those cores are in use by the
> > > app.
> > >
> > > Therefore, the decision was made in the past to not expand the
> > > build-time RTE_MAX_LCORE value beyond 128. Instead, it was recommended
> > > that users use the "--lcores" EAL parameter to take the high-numbered
> > > cores they wish to use and map them to lcore-ids within the 0 - 128
> > > range. While this works, this is a little clunky as it means that
> > > instead of just passing, for example, "-l 130-139", the user must
> > > instead pass "--lcores 0@130,1@131,2@132,3@133,...."
> > >
> > > This patchset attempts to simplify the situation by adding a new flag to
> > > do this mapping automatically. To use cores 130-139 and map them to ids
> > > 0-9 internally, the EAL args now become: "-l 130-139 --map-lcore-ids",
> > > or using the shorter "-M" version of the flag: "-Ml 130-139".
> > >
> > > Adding this new parameter required some rework of the existing arg
> > > parsing code, because in current DPDK the args are parsed and checked in
> > > the order they appear on the commandline. This means that using the
> > > example above, the core parameter 130-139 will be rejected immediately
> > > before the "map-lcore-ids" parameter is seen. To work around this, the
> > > core (and service core) parameters are not parsed when seen, instead
> > > they are only saved off and parsed after all arguments are parsed. The
> > > "-l" and "-c" parameters are converted into "--lcores" arguments, so all
> > > assigning of lcore ids is done there in all cases.
> > >
> > > RFC->v2:
> > > * converted printf to DEBUG log
> > > * added "-M" as shorter version of flag
> > > * added documentation
> > > * renamed internal API that was changed to avoid any potential hidden
> > > runtime issues.
> > >
> > > Bruce Richardson (3):
> > > eal: centralize core parameter parsing
> > > eal: convert core masks and lists to core sets
> > > eal: allow automatic mapping of high lcore ids
> > >
> > Ping for review.
> >
> > At a high level, does this feature seem useful to users?
>
> This seems useful, though I am not I would touch the existing options.
> I would have gone with a simple -L option (taking the same kind of
> input than -l but with new behavior), and not combine a flag with
> existing options.
>
That would be an easier patchset to do up. However, it would then mean that
we have no less than 4 different ways to specify the cores to use: "-c",
"-l", "-L", "--lcores" - and therefore need 4 different sets of parsing
options for them, and more checks to ensure we have only one of the 4
specified in any run. That's why I chose the modifier option, and to try
and consolidate the core setup a bit.
However, if having a completely new option is preferred, I am happy enough
to do up a different patchset for that.
> I scanned through the series, not much to say.
> Maybe add a unit test for new cmdline option.
>
Sure. Once it's decided what approach (if any) to take, I'll do up a new
patchset, taking into account any relevant feedback on this set.
/Bruce
^ permalink raw reply [flat|nested] 31+ messages in thread
* RE: [PATCH v2 0/3] allow easier use of high lcore-ids
2025-04-07 9:48 ` Bruce Richardson
@ 2025-04-07 10:15 ` Morten Brørup
2025-04-07 10:40 ` Bruce Richardson
2025-04-07 15:14 ` Stephen Hemminger
0 siblings, 2 replies; 31+ messages in thread
From: Morten Brørup @ 2025-04-07 10:15 UTC (permalink / raw)
To: Bruce Richardson, David Marchand; +Cc: dev
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Monday, 7 April 2025 11.49
>
> On Mon, Apr 07, 2025 at 09:04:05AM +0200, David Marchand wrote:
> > Hello Bruce,
> >
> > On Tue, Apr 1, 2025 at 4:08 PM Bruce Richardson
> > <bruce.richardson@intel.com> wrote:
> > >
> > > On Mon, Mar 24, 2025 at 05:30:26PM +0000, Bruce Richardson wrote:
> > > > Traditionally, DPDK has had a direct mapping of internal lcore-
> ids, to
> > > > the actual core numbers in use. With higher core count servers
> becoming
> > > > more prevalent the issue becomes one of increasing memory
> footprint when
> > > > using such a scheme, due to the need to have all arrays
> dimensioned for
> > > > all cores on the system, whether or not those cores are in use by
> the
> > > > app.
> > > >
> > > > Therefore, the decision was made in the past to not expand the
> > > > build-time RTE_MAX_LCORE value beyond 128. Instead, it was
> recommended
> > > > that users use the "--lcores" EAL parameter to take the high-
> numbered
> > > > cores they wish to use and map them to lcore-ids within the 0 -
> 128
> > > > range. While this works, this is a little clunky as it means that
> > > > instead of just passing, for example, "-l 130-139", the user must
> > > > instead pass "--lcores 0@130,1@131,2@132,3@133,...."
> > > >
> > > > This patchset attempts to simplify the situation by adding a new
> flag to
> > > > do this mapping automatically. To use cores 130-139 and map them
> to ids
> > > > 0-9 internally, the EAL args now become: "-l 130-139 --map-lcore-
> ids",
> > > > or using the shorter "-M" version of the flag: "-Ml 130-139".
> > > >
> > > > Adding this new parameter required some rework of the existing
> arg
> > > > parsing code, because in current DPDK the args are parsed and
> checked in
> > > > the order they appear on the commandline. This means that using
> the
> > > > example above, the core parameter 130-139 will be rejected
> immediately
> > > > before the "map-lcore-ids" parameter is seen. To work around
> this, the
> > > > core (and service core) parameters are not parsed when seen,
> instead
> > > > they are only saved off and parsed after all arguments are
> parsed. The
> > > > "-l" and "-c" parameters are converted into "--lcores" arguments,
> so all
> > > > assigning of lcore ids is done there in all cases.
> > > >
> > > > RFC->v2:
> > > > * converted printf to DEBUG log
> > > > * added "-M" as shorter version of flag
> > > > * added documentation
> > > > * renamed internal API that was changed to avoid any potential
> hidden
> > > > runtime issues.
> > > >
> > > > Bruce Richardson (3):
> > > > eal: centralize core parameter parsing
> > > > eal: convert core masks and lists to core sets
> > > > eal: allow automatic mapping of high lcore ids
> > > >
> > > Ping for review.
> > >
> > > At a high level, does this feature seem useful to users?
> >
> > This seems useful, though I am not I would touch the existing
> options.
> > I would have gone with a simple -L option (taking the same kind of
> > input than -l but with new behavior), and not combine a flag with
> > existing options.
> >
>
> That would be an easier patchset to do up. However, it would then mean
> that
> we have no less than 4 different ways to specify the cores to use: "-
> c",
> "-l", "-L", "--lcores" - and therefore need 4 different sets of parsing
> options for them, and more checks to ensure we have only one of the 4
> specified in any run. That's why I chose the modifier option, and to
> try
> and consolidate the core setup a bit.
>
> However, if having a completely new option is preferred, I am happy
> enough
> to do up a different patchset for that.
>
> > I scanned through the series, not much to say.
> > Maybe add a unit test for new cmdline option.
> >
> Sure. Once it's decided what approach (if any) to take, I'll do up a
> new
> patchset, taking into account any relevant feedback on this set.
>
> /Bruce
Changing the EAL parameter parser to a "two pass parser" makes sense.
I think checking for existence of more than one lcore specification options should suffice; we don't need to accept multiple lcore specification options and check for conflicts.
When remapping, do we need to support gaps in the "lcore" (logical cores) array, e.g. for secondary processes, or can it be continuous from 0 to the number of specified lcores?
And are new EAL parameters for this really beneficial?
Doesn't e.g. "-l 0-9@130-139,100@140" suffice?
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v2 0/3] allow easier use of high lcore-ids
2025-04-07 10:15 ` Morten Brørup
@ 2025-04-07 10:40 ` Bruce Richardson
2025-04-07 11:32 ` Morten Brørup
2025-04-07 15:14 ` Stephen Hemminger
1 sibling, 1 reply; 31+ messages in thread
From: Bruce Richardson @ 2025-04-07 10:40 UTC (permalink / raw)
To: Morten Brørup; +Cc: David Marchand, dev
On Mon, Apr 07, 2025 at 12:15:13PM +0200, Morten Brørup wrote:
> > From: Bruce Richardson [mailto:bruce.richardson@intel.com] Sent:
> > Monday, 7 April 2025 11.49
> >
> > On Mon, Apr 07, 2025 at 09:04:05AM +0200, David Marchand wrote:
> > > Hello Bruce,
> > >
> > > On Tue, Apr 1, 2025 at 4:08 PM Bruce Richardson
> > > <bruce.richardson@intel.com> wrote:
> > > >
> > > > On Mon, Mar 24, 2025 at 05:30:26PM +0000, Bruce Richardson wrote:
> > > > > Traditionally, DPDK has had a direct mapping of internal lcore-
> > ids, to
> > > > > the actual core numbers in use. With higher core count servers
> > becoming
> > > > > more prevalent the issue becomes one of increasing memory
> > footprint when
> > > > > using such a scheme, due to the need to have all arrays
> > dimensioned for
> > > > > all cores on the system, whether or not those cores are in use by
> > the
> > > > > app.
> > > > >
> > > > > Therefore, the decision was made in the past to not expand the
> > > > > build-time RTE_MAX_LCORE value beyond 128. Instead, it was
> > recommended
> > > > > that users use the "--lcores" EAL parameter to take the high-
> > numbered
> > > > > cores they wish to use and map them to lcore-ids within the 0 -
> > 128
> > > > > range. While this works, this is a little clunky as it means that
> > > > > instead of just passing, for example, "-l 130-139", the user must
> > > > > instead pass "--lcores 0@130,1@131,2@132,3@133,...."
> > > > >
> > > > > This patchset attempts to simplify the situation by adding a new
> > flag to
> > > > > do this mapping automatically. To use cores 130-139 and map them
> > to ids
> > > > > 0-9 internally, the EAL args now become: "-l 130-139 --map-lcore-
> > ids",
> > > > > or using the shorter "-M" version of the flag: "-Ml 130-139".
> > > > >
> > > > > Adding this new parameter required some rework of the existing
> > arg
> > > > > parsing code, because in current DPDK the args are parsed and
> > checked in
> > > > > the order they appear on the commandline. This means that using
> > the
> > > > > example above, the core parameter 130-139 will be rejected
> > immediately
> > > > > before the "map-lcore-ids" parameter is seen. To work around
> > this, the
> > > > > core (and service core) parameters are not parsed when seen,
> > instead
> > > > > they are only saved off and parsed after all arguments are
> > parsed. The
> > > > > "-l" and "-c" parameters are converted into "--lcores" arguments,
> > so all
> > > > > assigning of lcore ids is done there in all cases.
> > > > >
> > > > > RFC->v2: * converted printf to DEBUG log * added "-M" as shorter
> > > > > version of flag * added documentation * renamed internal API that
> > > > > was changed to avoid any potential
> > hidden
> > > > > runtime issues.
> > > > >
> > > > > Bruce Richardson (3): eal: centralize core parameter parsing eal:
> > > > > convert core masks and lists to core sets eal: allow automatic
> > > > > mapping of high lcore ids
> > > > >
> > > > Ping for review.
> > > >
> > > > At a high level, does this feature seem useful to users?
> > >
> > > This seems useful, though I am not I would touch the existing
> > options.
> > > I would have gone with a simple -L option (taking the same kind of
> > > input than -l but with new behavior), and not combine a flag with
> > > existing options.
> > >
> >
> > That would be an easier patchset to do up. However, it would then mean
> > that we have no less than 4 different ways to specify the cores to use:
> > "- c", "-l", "-L", "--lcores" - and therefore need 4 different sets of
> > parsing options for them, and more checks to ensure we have only one of
> > the 4 specified in any run. That's why I chose the modifier option, and
> > to try and consolidate the core setup a bit.
> >
> > However, if having a completely new option is preferred, I am happy
> > enough to do up a different patchset for that.
> >
> > > I scanned through the series, not much to say. Maybe add a unit test
> > > for new cmdline option.
> > >
> > Sure. Once it's decided what approach (if any) to take, I'll do up a
> > new patchset, taking into account any relevant feedback on this set.
> >
> > /Bruce
>
> Changing the EAL parameter parser to a "two pass parser" makes sense. I
> think checking for existence of more than one lcore specification options
> should suffice; we don't need to accept multiple lcore specification
> options and check for conflicts.
>
> When remapping, do we need to support gaps in the "lcore" (logical cores)
> array, e.g. for secondary processes, or can it be continuous from 0 to
> the number of specified lcores?
>
> And are new EAL parameters for this really beneficial? Doesn't e.g. "-l
> 0-9@130-139,100@140" suffice?
>
Actually, I believe "0-9@130-139"[1] is not the same as
"0@130,1@131,2@132,...". The latter affinities one thread to one core ten
times over, while the former affinitizes 10 threads to 10 cores - leaving
those threads free to move about within the 10 cores specified.
Just to confirm, I tweaked our helloworld example to print the cpu affinity
of each core when printing.
./build/examples/dpdk-helloworld --no-pci --lcores '(0-3)@(30-33)'
EAL: Detected CPU lcores: 96
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /run/user/11304126/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
hello from core 1, with thread affinity for cores: 30 31 32 33
hello from core 3, with thread affinity for cores: 30 31 32 33
hello from core 2, with thread affinity for cores: 30 31 32 33
hello from core 0, with thread affinity for cores: 30 31 32 33
./build/examples/dpdk-helloworld --no-pci --lcores '0@30,1@31,2@32,3@33'
EAL: Detected CPU lcores: 96
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /run/user/11304126/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
hello from core 1, with thread affinity for cores: 31 hello from core 3, with thread affinity for cores:
hello from core 2, with thread affinity for cores: 32
hello from core 0, with thread affinity for cores: 30
33
Regards,
/Bruce
[1] This actually needs to be "(0-9)@(130-139)", and with "--lcores", not
just "-l", there are actually different flags with different behaviours
^ permalink raw reply [flat|nested] 31+ messages in thread
* RE: [PATCH v2 0/3] allow easier use of high lcore-ids
2025-04-07 10:40 ` Bruce Richardson
@ 2025-04-07 11:32 ` Morten Brørup
2025-04-07 11:56 ` Bruce Richardson
0 siblings, 1 reply; 31+ messages in thread
From: Morten Brørup @ 2025-04-07 11:32 UTC (permalink / raw)
To: Bruce Richardson; +Cc: David Marchand, dev
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Monday, 7 April 2025 12.41
>
> On Mon, Apr 07, 2025 at 12:15:13PM +0200, Morten Brørup wrote:
> > > From: Bruce Richardson [mailto:bruce.richardson@intel.com] Sent:
> > > Monday, 7 April 2025 11.49
> > >
> > > On Mon, Apr 07, 2025 at 09:04:05AM +0200, David Marchand wrote:
> > > > Hello Bruce,
> > > >
> > > > On Tue, Apr 1, 2025 at 4:08 PM Bruce Richardson
> > > > <bruce.richardson@intel.com> wrote:
> > > > >
> > > > > On Mon, Mar 24, 2025 at 05:30:26PM +0000, Bruce Richardson
> wrote:
> > > > > > Traditionally, DPDK has had a direct mapping of internal
> lcore-
> > > ids, to
> > > > > > the actual core numbers in use. With higher core count
> servers
> > > becoming
> > > > > > more prevalent the issue becomes one of increasing memory
> > > footprint when
> > > > > > using such a scheme, due to the need to have all arrays
> > > dimensioned for
> > > > > > all cores on the system, whether or not those cores are in
> use by
> > > the
> > > > > > app.
> > > > > >
> > > > > > Therefore, the decision was made in the past to not expand
> the
> > > > > > build-time RTE_MAX_LCORE value beyond 128. Instead, it was
> > > recommended
> > > > > > that users use the "--lcores" EAL parameter to take the high-
> > > numbered
> > > > > > cores they wish to use and map them to lcore-ids within the 0
> -
> > > 128
> > > > > > range. While this works, this is a little clunky as it means
> that
> > > > > > instead of just passing, for example, "-l 130-139", the user
> must
> > > > > > instead pass "--lcores 0@130,1@131,2@132,3@133,...."
> > > > > >
> > > > > > This patchset attempts to simplify the situation by adding a
> new
> > > flag to
> > > > > > do this mapping automatically. To use cores 130-139 and map
> them
> > > to ids
> > > > > > 0-9 internally, the EAL args now become: "-l 130-139 --map-
> lcore-
> > > ids",
> > > > > > or using the shorter "-M" version of the flag: "-Ml 130-139".
> > > > > >
> > > > > > Adding this new parameter required some rework of the
> existing
> > > arg
> > > > > > parsing code, because in current DPDK the args are parsed and
> > > checked in
> > > > > > the order they appear on the commandline. This means that
> using
> > > the
> > > > > > example above, the core parameter 130-139 will be rejected
> > > immediately
> > > > > > before the "map-lcore-ids" parameter is seen. To work around
> > > this, the
> > > > > > core (and service core) parameters are not parsed when seen,
> > > instead
> > > > > > they are only saved off and parsed after all arguments are
> > > parsed. The
> > > > > > "-l" and "-c" parameters are converted into "--lcores"
> arguments,
> > > so all
> > > > > > assigning of lcore ids is done there in all cases.
> > > > > >
> > > > > > RFC->v2: * converted printf to DEBUG log * added "-M" as
> shorter
> > > > > > version of flag * added documentation * renamed internal API
> that
> > > > > > was changed to avoid any potential
> > > hidden
> > > > > > runtime issues.
> > > > > >
> > > > > > Bruce Richardson (3): eal: centralize core parameter parsing
> eal:
> > > > > > convert core masks and lists to core sets eal: allow
> automatic
> > > > > > mapping of high lcore ids
> > > > > >
> > > > > Ping for review.
> > > > >
> > > > > At a high level, does this feature seem useful to users?
> > > >
> > > > This seems useful, though I am not I would touch the existing
> > > options.
> > > > I would have gone with a simple -L option (taking the same kind
> of
> > > > input than -l but with new behavior), and not combine a flag with
> > > > existing options.
> > > >
> > >
> > > That would be an easier patchset to do up. However, it would then
> mean
> > > that we have no less than 4 different ways to specify the cores to
> use:
> > > "- c", "-l", "-L", "--lcores" - and therefore need 4 different sets
> of
> > > parsing options for them, and more checks to ensure we have only
> one of
> > > the 4 specified in any run. That's why I chose the modifier option,
> and
> > > to try and consolidate the core setup a bit.
> > >
> > > However, if having a completely new option is preferred, I am happy
> > > enough to do up a different patchset for that.
> > >
> > > > I scanned through the series, not much to say. Maybe add a unit
> test
> > > > for new cmdline option.
> > > >
> > > Sure. Once it's decided what approach (if any) to take, I'll do up
> a
> > > new patchset, taking into account any relevant feedback on this
> set.
> > >
> > > /Bruce
> >
> > Changing the EAL parameter parser to a "two pass parser" makes sense.
> I
> > think checking for existence of more than one lcore specification
> options
> > should suffice; we don't need to accept multiple lcore specification
> > options and check for conflicts.
> >
> > When remapping, do we need to support gaps in the "lcore" (logical
> cores)
> > array, e.g. for secondary processes, or can it be continuous from 0
> to
> > the number of specified lcores?
> >
> > And are new EAL parameters for this really beneficial? Doesn't e.g.
> "-l
> > 0-9@130-139,100@140" suffice?
> >
> Actually, I believe "0-9@130-139"[1] is not the same as
> "0@130,1@131,2@132,...". The latter affinities one thread to one core
> ten
> times over, while the former affinitizes 10 threads to 10 cores -
> leaving
> those threads free to move about within the 10 cores specified.
Interesting. The documentation [GSG] isn't clear to me about this; a example there could help clarify.
[GSG]: https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#lcore-related-options
If users are manually passing lcore parameters to the EAL, then I see why some sort of remapping shorthand is useful.
IMO, if the mappings are relatively exotic, it should be acceptable requiring an external script to build a long list of mapping parameters and then invoke the application with those script-generated EAL parameters.
This would reduce the scope to support relatively simple, common mappings.
Could we expand the --lcores syntax to support common mappings?
E.g. "0-9@130+" to do what I thought.
The lack of "()" treats the entries individually (not as a group), and the "+" indicates auto-increment.
A more advanced example:
"0-9@(130-131)+", meaning lcore 0 gets cpus 130-131, lcore 1 gets cpus 132-133, etc.
>
> Just to confirm, I tweaked our helloworld example to print the cpu
> affinity
> of each core when printing.
>
> ./build/examples/dpdk-helloworld --no-pci --lcores '(0-3)@(30-33)'
> EAL: Detected CPU lcores: 96
> EAL: Detected NUMA nodes: 2
> EAL: Detected static linkage of DPDK
> EAL: Multi-process socket /run/user/11304126/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> EAL: VFIO support initialized
> hello from core 1, with thread affinity for cores: 30 31 32 33
> hello from core 3, with thread affinity for cores: 30 31 32 33
> hello from core 2, with thread affinity for cores: 30 31 32 33
> hello from core 0, with thread affinity for cores: 30 31 32 33
>
> ./build/examples/dpdk-helloworld --no-pci --lcores
> '0@30,1@31,2@32,3@33'
> EAL: Detected CPU lcores: 96
> EAL: Detected NUMA nodes: 2
> EAL: Detected static linkage of DPDK
> EAL: Multi-process socket /run/user/11304126/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> EAL: VFIO support initialized
> hello from core 1, with thread affinity for cores: 31 hello from core
> 3, with thread affinity for cores:
> hello from core 2, with thread affinity for cores: 32
> hello from core 0, with thread affinity for cores: 30
> 33
>
> Regards,
> /Bruce
>
> [1] This actually needs to be "(0-9)@(130-139)", and with "--lcores",
> not
> just "-l", there are actually different flags with different behaviours
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v2 0/3] allow easier use of high lcore-ids
2025-04-07 11:32 ` Morten Brørup
@ 2025-04-07 11:56 ` Bruce Richardson
2025-04-07 12:25 ` Morten Brørup
0 siblings, 1 reply; 31+ messages in thread
From: Bruce Richardson @ 2025-04-07 11:56 UTC (permalink / raw)
To: Morten Brørup; +Cc: David Marchand, dev
On Mon, Apr 07, 2025 at 01:32:59PM +0200, Morten Brørup wrote:
> > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Sent: Monday, 7 April 2025 12.41
> >
> > On Mon, Apr 07, 2025 at 12:15:13PM +0200, Morten Brørup wrote:
> > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com] Sent:
> > > > Monday, 7 April 2025 11.49
> > > >
> > > > On Mon, Apr 07, 2025 at 09:04:05AM +0200, David Marchand wrote:
> > > > > Hello Bruce,
> > > > >
> > > > > On Tue, Apr 1, 2025 at 4:08 PM Bruce Richardson
> > > > > <bruce.richardson@intel.com> wrote:
> > > > > >
> > > > > > On Mon, Mar 24, 2025 at 05:30:26PM +0000, Bruce Richardson
> > wrote:
> > > > > > > Traditionally, DPDK has had a direct mapping of internal
> > lcore-
> > > > ids, to
> > > > > > > the actual core numbers in use. With higher core count
> > servers
> > > > becoming
> > > > > > > more prevalent the issue becomes one of increasing memory
> > > > footprint when
> > > > > > > using such a scheme, due to the need to have all arrays
> > > > dimensioned for
> > > > > > > all cores on the system, whether or not those cores are in
> > use by
> > > > the
> > > > > > > app.
> > > > > > >
> > > > > > > Therefore, the decision was made in the past to not expand
> > the
> > > > > > > build-time RTE_MAX_LCORE value beyond 128. Instead, it was
> > > > recommended
> > > > > > > that users use the "--lcores" EAL parameter to take the high-
> > > > numbered
> > > > > > > cores they wish to use and map them to lcore-ids within the 0
> > -
> > > > 128
> > > > > > > range. While this works, this is a little clunky as it means
> > that
> > > > > > > instead of just passing, for example, "-l 130-139", the user
> > must
> > > > > > > instead pass "--lcores 0@130,1@131,2@132,3@133,...."
> > > > > > >
> > > > > > > This patchset attempts to simplify the situation by adding a
> > new
> > > > flag to
> > > > > > > do this mapping automatically. To use cores 130-139 and map
> > them
> > > > to ids
> > > > > > > 0-9 internally, the EAL args now become: "-l 130-139 --map-
> > lcore-
> > > > ids",
> > > > > > > or using the shorter "-M" version of the flag: "-Ml 130-139".
> > > > > > >
> > > > > > > Adding this new parameter required some rework of the
> > existing
> > > > arg
> > > > > > > parsing code, because in current DPDK the args are parsed and
> > > > checked in
> > > > > > > the order they appear on the commandline. This means that
> > using
> > > > the
> > > > > > > example above, the core parameter 130-139 will be rejected
> > > > immediately
> > > > > > > before the "map-lcore-ids" parameter is seen. To work around
> > > > this, the
> > > > > > > core (and service core) parameters are not parsed when seen,
> > > > instead
> > > > > > > they are only saved off and parsed after all arguments are
> > > > parsed. The
> > > > > > > "-l" and "-c" parameters are converted into "--lcores"
> > arguments,
> > > > so all
> > > > > > > assigning of lcore ids is done there in all cases.
> > > > > > >
> > > > > > > RFC->v2: * converted printf to DEBUG log * added "-M" as
> > shorter
> > > > > > > version of flag * added documentation * renamed internal API
> > that
> > > > > > > was changed to avoid any potential
> > > > hidden
> > > > > > > runtime issues.
> > > > > > >
> > > > > > > Bruce Richardson (3): eal: centralize core parameter parsing
> > eal:
> > > > > > > convert core masks and lists to core sets eal: allow
> > automatic
> > > > > > > mapping of high lcore ids
> > > > > > >
> > > > > > Ping for review.
> > > > > >
> > > > > > At a high level, does this feature seem useful to users?
> > > > >
> > > > > This seems useful, though I am not I would touch the existing
> > > > options.
> > > > > I would have gone with a simple -L option (taking the same kind
> > of
> > > > > input than -l but with new behavior), and not combine a flag with
> > > > > existing options.
> > > > >
> > > >
> > > > That would be an easier patchset to do up. However, it would then
> > mean
> > > > that we have no less than 4 different ways to specify the cores to
> > use:
> > > > "- c", "-l", "-L", "--lcores" - and therefore need 4 different sets
> > of
> > > > parsing options for them, and more checks to ensure we have only
> > one of
> > > > the 4 specified in any run. That's why I chose the modifier option,
> > and
> > > > to try and consolidate the core setup a bit.
> > > >
> > > > However, if having a completely new option is preferred, I am happy
> > > > enough to do up a different patchset for that.
> > > >
> > > > > I scanned through the series, not much to say. Maybe add a unit
> > test
> > > > > for new cmdline option.
> > > > >
> > > > Sure. Once it's decided what approach (if any) to take, I'll do up
> > a
> > > > new patchset, taking into account any relevant feedback on this
> > set.
> > > >
> > > > /Bruce
> > >
> > > Changing the EAL parameter parser to a "two pass parser" makes sense.
> > I
> > > think checking for existence of more than one lcore specification
> > options
> > > should suffice; we don't need to accept multiple lcore specification
> > > options and check for conflicts.
> > >
> > > When remapping, do we need to support gaps in the "lcore" (logical
> > cores)
> > > array, e.g. for secondary processes, or can it be continuous from 0
> > to
> > > the number of specified lcores?
> > >
> > > And are new EAL parameters for this really beneficial? Doesn't e.g.
> > "-l
> > > 0-9@130-139,100@140" suffice?
> > >
> > Actually, I believe "0-9@130-139"[1] is not the same as
> > "0@130,1@131,2@132,...". The latter affinities one thread to one core
> > ten
> > times over, while the former affinitizes 10 threads to 10 cores -
> > leaving
> > those threads free to move about within the 10 cores specified.
>
> Interesting. The documentation [GSG] isn't clear to me about this; a example there could help clarify.
>
> [GSG]: https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#lcore-related-options
>
Yep, agreed.
> If users are manually passing lcore parameters to the EAL, then I see why some sort of remapping shorthand is useful.
> IMO, if the mappings are relatively exotic, it should be acceptable requiring an external script to build a long list of mapping parameters and then invoke the application with those script-generated EAL parameters.
> This would reduce the scope to support relatively simple, common mappings.
>
> Could we expand the --lcores syntax to support common mappings?
>
> E.g. "0-9@130+" to do what I thought.
> The lack of "()" treats the entries individually (not as a group), and the "+" indicates auto-increment.
>
> A more advanced example:
> "0-9@(130-131)+", meaning lcore 0 gets cpus 130-131, lcore 1 gets cpus 132-133, etc.
>
My issues with the above syntax idea is:
* I think it's overly complicating the lcores parameter adding in the
support for the "+" symbol - especially in the advanced case example you
provide. I worry about code maintainability here.
* More significantly for me, I think it's also getting things backwards in
that it is focusing more on the lcore ids visible to the app, rather than
the physical cores to be used. For the example above of 0-9@130+, what I
would expect is that the user is mainly thinking about the cores he wants
to use 130-139, which are then to be mapped to lower ids. If we had the
syntax reversed where the physical cores were first, I'd say it would
make more sense, e.g. 130-139@0+
* finally, as I found out last month, there are systems on which the cores
are spread across numa-nodes on odd/even boundaries, so to have an app
running on socket 0, you need to give the core ids as a list i.e.
0,2,4,6, and cannot use ranges. [This also reenforces the point above
too, where again it's the internal ids we need to generalize, not the
physical cpus]
My thinking on the matter, and I'm happy to be corrected if I'm wrong here,
is that end-users are unlikely to significantly care what the actual lcore
ids are internally in the program, so long as they work and are unique.
What does matter is the physical cpus on which the code is to run.
Therefore, my approach to this is to find the simplest possible solution
whereby the user can just provide a list of cores and tell EAL to just map
them to reasonable values. For the "reasonable" values, I would imagine
that for the vast majority of cases starting "0" is what is wanted. For
anything beyond that, we already have the existing --lcores syntax to be
used.
My 2c.
/Bruce
^ permalink raw reply [flat|nested] 31+ messages in thread
* RE: [PATCH v2 0/3] allow easier use of high lcore-ids
2025-04-07 11:56 ` Bruce Richardson
@ 2025-04-07 12:25 ` Morten Brørup
2025-04-07 12:41 ` Bruce Richardson
0 siblings, 1 reply; 31+ messages in thread
From: Morten Brørup @ 2025-04-07 12:25 UTC (permalink / raw)
To: Bruce Richardson; +Cc: David Marchand, dev
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Monday, 7 April 2025 13.56
>
> On Mon, Apr 07, 2025 at 01:32:59PM +0200, Morten Brørup wrote:
> > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > Sent: Monday, 7 April 2025 12.41
> > >
> > > On Mon, Apr 07, 2025 at 12:15:13PM +0200, Morten Brørup wrote:
> > > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent:
> > > > > Monday, 7 April 2025 11.49
> > > > >
> > > > > On Mon, Apr 07, 2025 at 09:04:05AM +0200, David Marchand wrote:
> > > > > > Hello Bruce,
> > > > > >
> > > > > > On Tue, Apr 1, 2025 at 4:08 PM Bruce Richardson
> > > > > > <bruce.richardson@intel.com> wrote:
> > > > > > >
> > > > > > > On Mon, Mar 24, 2025 at 05:30:26PM +0000, Bruce Richardson
> > > wrote:
> > > > > > > > Traditionally, DPDK has had a direct mapping of internal
> > > lcore-
> > > > > ids, to
> > > > > > > > the actual core numbers in use. With higher core count
> > > servers
> > > > > becoming
> > > > > > > > more prevalent the issue becomes one of increasing memory
> > > > > footprint when
> > > > > > > > using such a scheme, due to the need to have all arrays
> > > > > dimensioned for
> > > > > > > > all cores on the system, whether or not those cores are
> in
> > > use by
> > > > > the
> > > > > > > > app.
> > > > > > > >
> > > > > > > > Therefore, the decision was made in the past to not
> expand
> > > the
> > > > > > > > build-time RTE_MAX_LCORE value beyond 128. Instead, it
> was
> > > > > recommended
> > > > > > > > that users use the "--lcores" EAL parameter to take the
> high-
> > > > > numbered
> > > > > > > > cores they wish to use and map them to lcore-ids within
> the 0
> > > -
> > > > > 128
> > > > > > > > range. While this works, this is a little clunky as it
> means
> > > that
> > > > > > > > instead of just passing, for example, "-l 130-139", the
> user
> > > must
> > > > > > > > instead pass "--lcores 0@130,1@131,2@132,3@133,...."
> > > > > > > >
> > > > > > > > This patchset attempts to simplify the situation by
> adding a
> > > new
> > > > > flag to
> > > > > > > > do this mapping automatically. To use cores 130-139 and
> map
> > > them
> > > > > to ids
> > > > > > > > 0-9 internally, the EAL args now become: "-l 130-139 --
> map-
> > > lcore-
> > > > > ids",
> > > > > > > > or using the shorter "-M" version of the flag: "-Ml 130-
> 139".
> > > > > > > >
> > > > > > > > Adding this new parameter required some rework of the
> > > existing
> > > > > arg
> > > > > > > > parsing code, because in current DPDK the args are parsed
> and
> > > > > checked in
> > > > > > > > the order they appear on the commandline. This means that
> > > using
> > > > > the
> > > > > > > > example above, the core parameter 130-139 will be
> rejected
> > > > > immediately
> > > > > > > > before the "map-lcore-ids" parameter is seen. To work
> around
> > > > > this, the
> > > > > > > > core (and service core) parameters are not parsed when
> seen,
> > > > > instead
> > > > > > > > they are only saved off and parsed after all arguments
> are
> > > > > parsed. The
> > > > > > > > "-l" and "-c" parameters are converted into "--lcores"
> > > arguments,
> > > > > so all
> > > > > > > > assigning of lcore ids is done there in all cases.
> > > > > > > >
> > > > > > > > RFC->v2: * converted printf to DEBUG log * added "-M" as
> > > shorter
> > > > > > > > version of flag * added documentation * renamed internal
> API
> > > that
> > > > > > > > was changed to avoid any potential
> > > > > hidden
> > > > > > > > runtime issues.
> > > > > > > >
> > > > > > > > Bruce Richardson (3): eal: centralize core parameter
> parsing
> > > eal:
> > > > > > > > convert core masks and lists to core sets eal: allow
> > > automatic
> > > > > > > > mapping of high lcore ids
> > > > > > > >
> > > > > > > Ping for review.
> > > > > > >
> > > > > > > At a high level, does this feature seem useful to users?
> > > > > >
> > > > > > This seems useful, though I am not I would touch the existing
> > > > > options.
> > > > > > I would have gone with a simple -L option (taking the same
> kind
> > > of
> > > > > > input than -l but with new behavior), and not combine a flag
> with
> > > > > > existing options.
> > > > > >
> > > > >
> > > > > That would be an easier patchset to do up. However, it would
> then
> > > mean
> > > > > that we have no less than 4 different ways to specify the cores
> to
> > > use:
> > > > > "- c", "-l", "-L", "--lcores" - and therefore need 4 different
> sets
> > > of
> > > > > parsing options for them, and more checks to ensure we have
> only
> > > one of
> > > > > the 4 specified in any run. That's why I chose the modifier
> option,
> > > and
> > > > > to try and consolidate the core setup a bit.
> > > > >
> > > > > However, if having a completely new option is preferred, I am
> happy
> > > > > enough to do up a different patchset for that.
> > > > >
> > > > > > I scanned through the series, not much to say. Maybe add a
> unit
> > > test
> > > > > > for new cmdline option.
> > > > > >
> > > > > Sure. Once it's decided what approach (if any) to take, I'll do
> up
> > > a
> > > > > new patchset, taking into account any relevant feedback on this
> > > set.
> > > > >
> > > > > /Bruce
> > > >
> > > > Changing the EAL parameter parser to a "two pass parser" makes
> sense.
> > > I
> > > > think checking for existence of more than one lcore specification
> > > options
> > > > should suffice; we don't need to accept multiple lcore
> specification
> > > > options and check for conflicts.
> > > >
> > > > When remapping, do we need to support gaps in the "lcore"
> (logical
> > > cores)
> > > > array, e.g. for secondary processes, or can it be continuous from
> 0
> > > to
> > > > the number of specified lcores?
> > > >
> > > > And are new EAL parameters for this really beneficial? Doesn't
> e.g.
> > > "-l
> > > > 0-9@130-139,100@140" suffice?
> > > >
> > > Actually, I believe "0-9@130-139"[1] is not the same as
> > > "0@130,1@131,2@132,...". The latter affinities one thread to one
> core
> > > ten
> > > times over, while the former affinitizes 10 threads to 10 cores -
> > > leaving
> > > those threads free to move about within the 10 cores specified.
> >
> > Interesting. The documentation [GSG] isn't clear to me about this; a
> example there could help clarify.
> >
> > [GSG]:
> https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#lcore-
> related-options
> >
>
> Yep, agreed.
>
> > If users are manually passing lcore parameters to the EAL, then I see
> why some sort of remapping shorthand is useful.
> > IMO, if the mappings are relatively exotic, it should be acceptable
> requiring an external script to build a long list of mapping parameters
> and then invoke the application with those script-generated EAL
> parameters.
> > This would reduce the scope to support relatively simple, common
> mappings.
> >
> > Could we expand the --lcores syntax to support common mappings?
> >
> > E.g. "0-9@130+" to do what I thought.
> > The lack of "()" treats the entries individually (not as a group),
> and the "+" indicates auto-increment.
> >
> > A more advanced example:
> > "0-9@(130-131)+", meaning lcore 0 gets cpus 130-131, lcore 1 gets
> cpus 132-133, etc.
> >
>
> My issues with the above syntax idea is:
> * I think it's overly complicating the lcores parameter adding in the
> support for the "+" symbol - especially in the advanced case example
> you
> provide. I worry about code maintainability here.
> * More significantly for me, I think it's also getting things backwards
> in
> that it is focusing more on the lcore ids visible to the app, rather
> than
> the physical cores to be used. For the example above of 0-9@130+,
> what I
> would expect is that the user is mainly thinking about the cores he
> wants
> to use 130-139, which are then to be mapped to lower ids. If we had
> the
> syntax reversed where the physical cores were first, I'd say it would
> make more sense, e.g. 130-139@0+
I 100 % agree on the syntax being backwards.
A good reason for introducing a new parameter, rather than expanding on "--lcores".
We should consider deprecating the old (backwards) syntax, so users don’t get confused about one EAL parameter being "backwards" of the other.
> * finally, as I found out last month, there are systems on which the
> cores
> are spread across numa-nodes on odd/even boundaries, so to have an
> app
> running on socket 0, you need to give the core ids as a list i.e.
> 0,2,4,6, and cannot use ranges. [This also reenforces the point above
> too, where again it's the internal ids we need to generalize, not the
> physical cpus]
For this, we could use "/2" (like in subnets), or "+2" as increment parameter.
>
> My thinking on the matter, and I'm happy to be corrected if I'm wrong
> here,
> is that end-users are unlikely to significantly care what the actual
> lcore
> ids are internally in the program, so long as they work and are unique.
Generally yes.
Note that we should allow the same physical CPU core to be assigned to multiple lcores...
If a CPU core is shared between multiple worker lcores, then it could be problematic, but with a mix of lcore roles, this might be handy for some applications. E.g. a virtual machine with only one CPU core could use that single CPU core as both main and worker lcore, or as main and service lcore.
Sharing an lcore between threads requires the developer to take special care of e.g. spinlocks, but the EAL parameter parser should not prohibit it. It might log a notice, though.
> What does matter is the physical cpus on which the code is to run.
> Therefore, my approach to this is to find the simplest possible
> solution
> whereby the user can just provide a list of cores and tell EAL to just
> map
> them to reasonable values. For the "reasonable" values, I would imagine
> that for the vast majority of cases starting "0" is what is wanted. For
> anything beyond that, we already have the existing --lcores syntax to
> be
> used.
Agree with all of the above. :-)
>
> My 2c.
>
> /Bruce
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v2 0/3] allow easier use of high lcore-ids
2025-04-07 12:25 ` Morten Brørup
@ 2025-04-07 12:41 ` Bruce Richardson
2025-04-07 13:18 ` Morten Brørup
0 siblings, 1 reply; 31+ messages in thread
From: Bruce Richardson @ 2025-04-07 12:41 UTC (permalink / raw)
To: Morten Brørup; +Cc: David Marchand, dev
On Mon, Apr 07, 2025 at 02:25:46PM +0200, Morten Brørup wrote:
> > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Sent: Monday, 7 April 2025 13.56
> >
> > On Mon, Apr 07, 2025 at 01:32:59PM +0200, Morten Brørup wrote:
> > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > > Sent: Monday, 7 April 2025 12.41
> > > >
> > > > On Mon, Apr 07, 2025 at 12:15:13PM +0200, Morten Brørup wrote:
> > > > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Sent:
> > > > > > Monday, 7 April 2025 11.49
> > > > > >
> > > > > > On Mon, Apr 07, 2025 at 09:04:05AM +0200, David Marchand wrote:
> > > > > > > Hello Bruce,
> > > > > > >
> > > > > > > On Tue, Apr 1, 2025 at 4:08 PM Bruce Richardson
> > > > > > > <bruce.richardson@intel.com> wrote:
> > > > > > > >
> > > > > > > > On Mon, Mar 24, 2025 at 05:30:26PM +0000, Bruce Richardson
> > > > wrote:
> > > > > > > > > Traditionally, DPDK has had a direct mapping of internal
> > > > lcore-
> > > > > > ids, to
> > > > > > > > > the actual core numbers in use. With higher core count
> > > > servers
> > > > > > becoming
> > > > > > > > > more prevalent the issue becomes one of increasing memory
> > > > > > footprint when
> > > > > > > > > using such a scheme, due to the need to have all arrays
> > > > > > dimensioned for
> > > > > > > > > all cores on the system, whether or not those cores are
> > in
> > > > use by
> > > > > > the
> > > > > > > > > app.
> > > > > > > > >
> > > > > > > > > Therefore, the decision was made in the past to not
> > expand
> > > > the
> > > > > > > > > build-time RTE_MAX_LCORE value beyond 128. Instead, it
> > was
> > > > > > recommended
> > > > > > > > > that users use the "--lcores" EAL parameter to take the
> > high-
> > > > > > numbered
> > > > > > > > > cores they wish to use and map them to lcore-ids within
> > the 0
> > > > -
> > > > > > 128
> > > > > > > > > range. While this works, this is a little clunky as it
> > means
> > > > that
> > > > > > > > > instead of just passing, for example, "-l 130-139", the
> > user
> > > > must
> > > > > > > > > instead pass "--lcores 0@130,1@131,2@132,3@133,...."
> > > > > > > > >
> > > > > > > > > This patchset attempts to simplify the situation by
> > adding a
> > > > new
> > > > > > flag to
> > > > > > > > > do this mapping automatically. To use cores 130-139 and
> > map
> > > > them
> > > > > > to ids
> > > > > > > > > 0-9 internally, the EAL args now become: "-l 130-139 --
> > map-
> > > > lcore-
> > > > > > ids",
> > > > > > > > > or using the shorter "-M" version of the flag: "-Ml 130-
> > 139".
> > > > > > > > >
> > > > > > > > > Adding this new parameter required some rework of the
> > > > existing
> > > > > > arg
> > > > > > > > > parsing code, because in current DPDK the args are parsed
> > and
> > > > > > checked in
> > > > > > > > > the order they appear on the commandline. This means that
> > > > using
> > > > > > the
> > > > > > > > > example above, the core parameter 130-139 will be
> > rejected
> > > > > > immediately
> > > > > > > > > before the "map-lcore-ids" parameter is seen. To work
> > around
> > > > > > this, the
> > > > > > > > > core (and service core) parameters are not parsed when
> > seen,
> > > > > > instead
> > > > > > > > > they are only saved off and parsed after all arguments
> > are
> > > > > > parsed. The
> > > > > > > > > "-l" and "-c" parameters are converted into "--lcores"
> > > > arguments,
> > > > > > so all
> > > > > > > > > assigning of lcore ids is done there in all cases.
> > > > > > > > >
> > > > > > > > > RFC->v2: * converted printf to DEBUG log * added "-M" as
> > > > shorter
> > > > > > > > > version of flag * added documentation * renamed internal
> > API
> > > > that
> > > > > > > > > was changed to avoid any potential
> > > > > > hidden
> > > > > > > > > runtime issues.
> > > > > > > > >
> > > > > > > > > Bruce Richardson (3): eal: centralize core parameter
> > parsing
> > > > eal:
> > > > > > > > > convert core masks and lists to core sets eal: allow
> > > > automatic
> > > > > > > > > mapping of high lcore ids
> > > > > > > > >
> > > > > > > > Ping for review.
> > > > > > > >
> > > > > > > > At a high level, does this feature seem useful to users?
> > > > > > >
> > > > > > > This seems useful, though I am not I would touch the existing
> > > > > > options.
> > > > > > > I would have gone with a simple -L option (taking the same
> > kind
> > > > of
> > > > > > > input than -l but with new behavior), and not combine a flag
> > with
> > > > > > > existing options.
> > > > > > >
> > > > > >
> > > > > > That would be an easier patchset to do up. However, it would
> > then
> > > > mean
> > > > > > that we have no less than 4 different ways to specify the cores
> > to
> > > > use:
> > > > > > "- c", "-l", "-L", "--lcores" - and therefore need 4 different
> > sets
> > > > of
> > > > > > parsing options for them, and more checks to ensure we have
> > only
> > > > one of
> > > > > > the 4 specified in any run. That's why I chose the modifier
> > option,
> > > > and
> > > > > > to try and consolidate the core setup a bit.
> > > > > >
> > > > > > However, if having a completely new option is preferred, I am
> > happy
> > > > > > enough to do up a different patchset for that.
> > > > > >
> > > > > > > I scanned through the series, not much to say. Maybe add a
> > unit
> > > > test
> > > > > > > for new cmdline option.
> > > > > > >
> > > > > > Sure. Once it's decided what approach (if any) to take, I'll do
> > up
> > > > a
> > > > > > new patchset, taking into account any relevant feedback on this
> > > > set.
> > > > > >
> > > > > > /Bruce
> > > > >
> > > > > Changing the EAL parameter parser to a "two pass parser" makes
> > sense.
> > > > I
> > > > > think checking for existence of more than one lcore specification
> > > > options
> > > > > should suffice; we don't need to accept multiple lcore
> > specification
> > > > > options and check for conflicts.
> > > > >
> > > > > When remapping, do we need to support gaps in the "lcore"
> > (logical
> > > > cores)
> > > > > array, e.g. for secondary processes, or can it be continuous from
> > 0
> > > > to
> > > > > the number of specified lcores?
> > > > >
> > > > > And are new EAL parameters for this really beneficial? Doesn't
> > e.g.
> > > > "-l
> > > > > 0-9@130-139,100@140" suffice?
> > > > >
> > > > Actually, I believe "0-9@130-139"[1] is not the same as
> > > > "0@130,1@131,2@132,...". The latter affinities one thread to one
> > core
> > > > ten
> > > > times over, while the former affinitizes 10 threads to 10 cores -
> > > > leaving
> > > > those threads free to move about within the 10 cores specified.
> > >
> > > Interesting. The documentation [GSG] isn't clear to me about this; a
> > example there could help clarify.
> > >
> > > [GSG]:
> > https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#lcore-
> > related-options
> > >
> >
> > Yep, agreed.
> >
> > > If users are manually passing lcore parameters to the EAL, then I see
> > why some sort of remapping shorthand is useful.
> > > IMO, if the mappings are relatively exotic, it should be acceptable
> > requiring an external script to build a long list of mapping parameters
> > and then invoke the application with those script-generated EAL
> > parameters.
> > > This would reduce the scope to support relatively simple, common
> > mappings.
> > >
> > > Could we expand the --lcores syntax to support common mappings?
> > >
> > > E.g. "0-9@130+" to do what I thought.
> > > The lack of "()" treats the entries individually (not as a group),
> > and the "+" indicates auto-increment.
> > >
> > > A more advanced example:
> > > "0-9@(130-131)+", meaning lcore 0 gets cpus 130-131, lcore 1 gets
> > cpus 132-133, etc.
> > >
> >
> > My issues with the above syntax idea is:
> > * I think it's overly complicating the lcores parameter adding in the
> > support for the "+" symbol - especially in the advanced case example
> > you
> > provide. I worry about code maintainability here.
> > * More significantly for me, I think it's also getting things backwards
> > in
> > that it is focusing more on the lcore ids visible to the app, rather
> > than
> > the physical cores to be used. For the example above of 0-9@130+,
> > what I
> > would expect is that the user is mainly thinking about the cores he
> > wants
> > to use 130-139, which are then to be mapped to lower ids. If we had
> > the
> > syntax reversed where the physical cores were first, I'd say it would
> > make more sense, e.g. 130-139@0+
>
> I 100 % agree on the syntax being backwards.
> A good reason for introducing a new parameter, rather than expanding on "--lcores".
>
Yes and no.
I agree with not expanding on --lcores, but I also don't think any new
parameter added should attempt to reproduce all of what lcores does. I
would leave --lcores as-is, as the power-tool for lcore config e.g. what
you talk about below for mapping multiple lcores to the same physical cpu.
> We should consider deprecating the old (backwards) syntax, so users don’t get confused about one EAL parameter being "backwards" of the other.
>
I disagree with this. I would be ok with deprecating the old "-c" coremask
syntax - I think the time is long past when we should be dealing with
masks. However, removing "-l" and "--lcores" flag is, to me anyway, too big
and jarring a change for end users for us to consider.
> > * finally, as I found out last month, there are systems on which the
> > cores
> > are spread across numa-nodes on odd/even boundaries, so to have an
> > app
> > running on socket 0, you need to give the core ids as a list i.e.
> > 0,2,4,6, and cannot use ranges. [This also reenforces the point above
> > too, where again it's the internal ids we need to generalize, not the
> > physical cpus]
>
> For this, we could use "/2" (like in subnets), or "+2" as increment parameter.
>
> >
> > My thinking on the matter, and I'm happy to be corrected if I'm wrong
> > here,
> > is that end-users are unlikely to significantly care what the actual
> > lcore
> > ids are internally in the program, so long as they work and are unique.
>
> Generally yes.
> Note that we should allow the same physical CPU core to be assigned to multiple lcores...
> If a CPU core is shared between multiple worker lcores, then it could be problematic, but with a mix of lcore roles, this might be handy for some applications. E.g. a virtual machine with only one CPU core could use that single CPU core as both main and worker lcore, or as main and service lcore.
> Sharing an lcore between threads requires the developer to take special care of e.g. spinlocks, but the EAL parameter parser should not prohibit it. It might log a notice, though.
>
This is already taken care of via --lcores, so I don't see the need to
reimplement it using a new flag also. Any new flags we add should be kept
deliberately simple.
> > What does matter is the physical cpus on which the code is to run.
> > Therefore, my approach to this is to find the simplest possible
> > solution
> > whereby the user can just provide a list of cores and tell EAL to just
> > map
> > them to reasonable values. For the "reasonable" values, I would imagine
> > that for the vast majority of cases starting "0" is what is wanted. For
> > anything beyond that, we already have the existing --lcores syntax to
> > be
> > used.
>
> Agree with all of the above. :-)
>
Great. That still leaves us with the problem of what exactly to do as the
best solution. Here are some alternatives that I see:
1. Add a modifier flag for -l and -c parameters to auto-remap the lcore ids
to zero, so user is just specifying physical CPU id's.
2. Add a new flag for specifying physical cpu ids, which auto-remaps the
cores specified to zero.
2a. To simplify our code and user experience we could at the same time:
* deprecate "-c" flag for coremasks
* make "-l" and "--lcores" the same flag just in long and short
versions. This should not break anything as "-l" parameter is
just as subset of what "--lcores" provides.
* that would leave us with effectively two core flag paths:
- -l/--lcores, behaviour as now, full explicit control
- -L/--lcores-remapped, takes simplified core list (only "," and
"-" supported as with "-l" now), and maps them to zero-based.
Third options? Any other feedback?
/Bruce
^ permalink raw reply [flat|nested] 31+ messages in thread
* RE: [PATCH v2 0/3] allow easier use of high lcore-ids
2025-04-07 12:41 ` Bruce Richardson
@ 2025-04-07 13:18 ` Morten Brørup
2025-04-07 13:24 ` Bruce Richardson
0 siblings, 1 reply; 31+ messages in thread
From: Morten Brørup @ 2025-04-07 13:18 UTC (permalink / raw)
To: Bruce Richardson; +Cc: David Marchand, dev
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Monday, 7 April 2025 14.42
>
> On Mon, Apr 07, 2025 at 02:25:46PM +0200, Morten Brørup wrote:
> > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > Sent: Monday, 7 April 2025 13.56
> > >
> > > On Mon, Apr 07, 2025 at 01:32:59PM +0200, Morten Brørup wrote:
> > > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > > > Sent: Monday, 7 April 2025 12.41
> > > > >
> > > > > On Mon, Apr 07, 2025 at 12:15:13PM +0200, Morten Brørup wrote:
> > > > > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > Sent:
> > > > > > > Monday, 7 April 2025 11.49
> > > > > > >
> > > > > > > On Mon, Apr 07, 2025 at 09:04:05AM +0200, David Marchand
> wrote:
> > > > > > > > Hello Bruce,
> > > > > > > >
> > > > > > > > On Tue, Apr 1, 2025 at 4:08 PM Bruce Richardson
> > > > > > > > <bruce.richardson@intel.com> wrote:
> > > > > > > > >
> > > > > > > > > On Mon, Mar 24, 2025 at 05:30:26PM +0000, Bruce
> Richardson
> > > > > wrote:
> > > > > > > > > > Traditionally, DPDK has had a direct mapping of
> internal
> > > > > lcore-
> > > > > > > ids, to
> > > > > > > > > > the actual core numbers in use. With higher core
> count
> > > > > servers
> > > > > > > becoming
> > > > > > > > > > more prevalent the issue becomes one of increasing
> memory
> > > > > > > footprint when
> > > > > > > > > > using such a scheme, due to the need to have all
> arrays
> > > > > > > dimensioned for
> > > > > > > > > > all cores on the system, whether or not those cores
> are
> > > in
> > > > > use by
> > > > > > > the
> > > > > > > > > > app.
> > > > > > > > > >
> > > > > > > > > > Therefore, the decision was made in the past to not
> > > expand
> > > > > the
> > > > > > > > > > build-time RTE_MAX_LCORE value beyond 128. Instead,
> it
> > > was
> > > > > > > recommended
> > > > > > > > > > that users use the "--lcores" EAL parameter to take
> the
> > > high-
> > > > > > > numbered
> > > > > > > > > > cores they wish to use and map them to lcore-ids
> within
> > > the 0
> > > > > -
> > > > > > > 128
> > > > > > > > > > range. While this works, this is a little clunky as
> it
> > > means
> > > > > that
> > > > > > > > > > instead of just passing, for example, "-l 130-139",
> the
> > > user
> > > > > must
> > > > > > > > > > instead pass "--lcores 0@130,1@131,2@132,3@133,...."
> > > > > > > > > >
> > > > > > > > > > This patchset attempts to simplify the situation by
> > > adding a
> > > > > new
> > > > > > > flag to
> > > > > > > > > > do this mapping automatically. To use cores 130-139
> and
> > > map
> > > > > them
> > > > > > > to ids
> > > > > > > > > > 0-9 internally, the EAL args now become: "-l 130-139
> --
> > > map-
> > > > > lcore-
> > > > > > > ids",
> > > > > > > > > > or using the shorter "-M" version of the flag: "-Ml
> 130-
> > > 139".
> > > > > > > > > >
> > > > > > > > > > Adding this new parameter required some rework of the
> > > > > existing
> > > > > > > arg
> > > > > > > > > > parsing code, because in current DPDK the args are
> parsed
> > > and
> > > > > > > checked in
> > > > > > > > > > the order they appear on the commandline. This means
> that
> > > > > using
> > > > > > > the
> > > > > > > > > > example above, the core parameter 130-139 will be
> > > rejected
> > > > > > > immediately
> > > > > > > > > > before the "map-lcore-ids" parameter is seen. To work
> > > around
> > > > > > > this, the
> > > > > > > > > > core (and service core) parameters are not parsed
> when
> > > seen,
> > > > > > > instead
> > > > > > > > > > they are only saved off and parsed after all
> arguments
> > > are
> > > > > > > parsed. The
> > > > > > > > > > "-l" and "-c" parameters are converted into "--
> lcores"
> > > > > arguments,
> > > > > > > so all
> > > > > > > > > > assigning of lcore ids is done there in all cases.
> > > > > > > > > >
> > > > > > > > > > RFC->v2: * converted printf to DEBUG log * added "-M"
> as
> > > > > shorter
> > > > > > > > > > version of flag * added documentation * renamed
> internal
> > > API
> > > > > that
> > > > > > > > > > was changed to avoid any potential
> > > > > > > hidden
> > > > > > > > > > runtime issues.
> > > > > > > > > >
> > > > > > > > > > Bruce Richardson (3): eal: centralize core parameter
> > > parsing
> > > > > eal:
> > > > > > > > > > convert core masks and lists to core sets eal: allow
> > > > > automatic
> > > > > > > > > > mapping of high lcore ids
> > > > > > > > > >
> > > > > > > > > Ping for review.
> > > > > > > > >
> > > > > > > > > At a high level, does this feature seem useful to
> users?
> > > > > > > >
> > > > > > > > This seems useful, though I am not I would touch the
> existing
> > > > > > > options.
> > > > > > > > I would have gone with a simple -L option (taking the
> same
> > > kind
> > > > > of
> > > > > > > > input than -l but with new behavior), and not combine a
> flag
> > > with
> > > > > > > > existing options.
> > > > > > > >
> > > > > > >
> > > > > > > That would be an easier patchset to do up. However, it
> would
> > > then
> > > > > mean
> > > > > > > that we have no less than 4 different ways to specify the
> cores
> > > to
> > > > > use:
> > > > > > > "- c", "-l", "-L", "--lcores" - and therefore need 4
> different
> > > sets
> > > > > of
> > > > > > > parsing options for them, and more checks to ensure we have
> > > only
> > > > > one of
> > > > > > > the 4 specified in any run. That's why I chose the modifier
> > > option,
> > > > > and
> > > > > > > to try and consolidate the core setup a bit.
> > > > > > >
> > > > > > > However, if having a completely new option is preferred, I
> am
> > > happy
> > > > > > > enough to do up a different patchset for that.
> > > > > > >
> > > > > > > > I scanned through the series, not much to say. Maybe add
> a
> > > unit
> > > > > test
> > > > > > > > for new cmdline option.
> > > > > > > >
> > > > > > > Sure. Once it's decided what approach (if any) to take,
> I'll do
> > > up
> > > > > a
> > > > > > > new patchset, taking into account any relevant feedback on
> this
> > > > > set.
> > > > > > >
> > > > > > > /Bruce
> > > > > >
> > > > > > Changing the EAL parameter parser to a "two pass parser"
> makes
> > > sense.
> > > > > I
> > > > > > think checking for existence of more than one lcore
> specification
> > > > > options
> > > > > > should suffice; we don't need to accept multiple lcore
> > > specification
> > > > > > options and check for conflicts.
> > > > > >
> > > > > > When remapping, do we need to support gaps in the "lcore"
> > > (logical
> > > > > cores)
> > > > > > array, e.g. for secondary processes, or can it be continuous
> from
> > > 0
> > > > > to
> > > > > > the number of specified lcores?
> > > > > >
> > > > > > And are new EAL parameters for this really beneficial?
> Doesn't
> > > e.g.
> > > > > "-l
> > > > > > 0-9@130-139,100@140" suffice?
> > > > > >
> > > > > Actually, I believe "0-9@130-139"[1] is not the same as
> > > > > "0@130,1@131,2@132,...". The latter affinities one thread to
> one
> > > core
> > > > > ten
> > > > > times over, while the former affinitizes 10 threads to 10 cores
> -
> > > > > leaving
> > > > > those threads free to move about within the 10 cores specified.
> > > >
> > > > Interesting. The documentation [GSG] isn't clear to me about
> this; a
> > > example there could help clarify.
> > > >
> > > > [GSG]:
> > >
> https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#lcore-
> > > related-options
> > > >
> > >
> > > Yep, agreed.
> > >
> > > > If users are manually passing lcore parameters to the EAL, then I
> see
> > > why some sort of remapping shorthand is useful.
> > > > IMO, if the mappings are relatively exotic, it should be
> acceptable
> > > requiring an external script to build a long list of mapping
> parameters
> > > and then invoke the application with those script-generated EAL
> > > parameters.
> > > > This would reduce the scope to support relatively simple, common
> > > mappings.
> > > >
> > > > Could we expand the --lcores syntax to support common mappings?
> > > >
> > > > E.g. "0-9@130+" to do what I thought.
> > > > The lack of "()" treats the entries individually (not as a
> group),
> > > and the "+" indicates auto-increment.
> > > >
> > > > A more advanced example:
> > > > "0-9@(130-131)+", meaning lcore 0 gets cpus 130-131, lcore 1 gets
> > > cpus 132-133, etc.
> > > >
> > >
> > > My issues with the above syntax idea is:
> > > * I think it's overly complicating the lcores parameter adding in
> the
> > > support for the "+" symbol - especially in the advanced case
> example
> > > you
> > > provide. I worry about code maintainability here.
> > > * More significantly for me, I think it's also getting things
> backwards
> > > in
> > > that it is focusing more on the lcore ids visible to the app,
> rather
> > > than
> > > the physical cores to be used. For the example above of 0-9@130+,
> > > what I
> > > would expect is that the user is mainly thinking about the cores
> he
> > > wants
> > > to use 130-139, which are then to be mapped to lower ids. If we
> had
> > > the
> > > syntax reversed where the physical cores were first, I'd say it
> would
> > > make more sense, e.g. 130-139@0+
> >
> > I 100 % agree on the syntax being backwards.
> > A good reason for introducing a new parameter, rather than expanding
> on "--lcores".
> >
>
> Yes and no.
> I agree with not expanding on --lcores, but I also don't think any new
> parameter added should attempt to reproduce all of what lcores does. I
> would leave --lcores as-is, as the power-tool for lcore config e.g.
> what
> you talk about below for mapping multiple lcores to the same physical
> cpu.
>
> > We should consider deprecating the old (backwards) syntax, so users
> don’t get confused about one EAL parameter being "backwards" of the
> other.
> >
>
> I disagree with this. I would be ok with deprecating the old "-c"
> coremask
> syntax - I think the time is long past when we should be dealing with
> masks. However, removing "-l" and "--lcores" flag is, to me anyway, too
> big
> and jarring a change for end users for us to consider.
>
> > > * finally, as I found out last month, there are systems on which
> the
> > > cores
> > > are spread across numa-nodes on odd/even boundaries, so to have
> an
> > > app
> > > running on socket 0, you need to give the core ids as a list i.e.
> > > 0,2,4,6, and cannot use ranges. [This also reenforces the point
> above
> > > too, where again it's the internal ids we need to generalize, not
> the
> > > physical cpus]
> >
> > For this, we could use "/2" (like in subnets), or "+2" as increment
> parameter.
> >
> > >
> > > My thinking on the matter, and I'm happy to be corrected if I'm
> wrong
> > > here,
> > > is that end-users are unlikely to significantly care what the
> actual
> > > lcore
> > > ids are internally in the program, so long as they work and are
> unique.
> >
> > Generally yes.
> > Note that we should allow the same physical CPU core to be assigned
> to multiple lcores...
> > If a CPU core is shared between multiple worker lcores, then it could
> be problematic, but with a mix of lcore roles, this might be handy for
> some applications. E.g. a virtual machine with only one CPU core could
> use that single CPU core as both main and worker lcore, or as main and
> service lcore.
> > Sharing an lcore between threads requires the developer to take
> special care of e.g. spinlocks, but the EAL parameter parser should not
> prohibit it. It might log a notice, though.
> >
>
> This is already taken care of via --lcores, so I don't see the need to
> reimplement it using a new flag also. Any new flags we add should be
> kept
> deliberately simple.
>
> > > What does matter is the physical cpus on which the code is to run.
> > > Therefore, my approach to this is to find the simplest possible
> > > solution
> > > whereby the user can just provide a list of cores and tell EAL to
> just
> > > map
> > > them to reasonable values. For the "reasonable" values, I would
> imagine
> > > that for the vast majority of cases starting "0" is what is wanted.
> For
> > > anything beyond that, we already have the existing --lcores syntax
> to
> > > be
> > > used.
> >
> > Agree with all of the above. :-)
> >
>
> Great. That still leaves us with the problem of what exactly to do as
> the
> best solution. Here are some alternatives that I see:
>
> 1. Add a modifier flag for -l and -c parameters to auto-remap the lcore
> ids
> to zero, so user is just specifying physical CPU id's.
> 2. Add a new flag for specifying physical cpu ids, which auto-remaps
> the
> cores specified to zero.
> 2a. To simplify our code and user experience we could at the same
> time:
> * deprecate "-c" flag for coremasks
> * make "-l" and "--lcores" the same flag just in long and short
> versions. This should not break anything as "-l" parameter is
> just as subset of what "--lcores" provides.
> * that would leave us with effectively two core flag paths:
> - -l/--lcores, behaviour as now, full explicit control
> - -L/--lcores-remapped, takes simplified core list (only ","
> and
> "-" supported as with "-l" now), and maps them to zero-
> based.
+1 for 2a.
>
> Third options? Any other feedback?
If the internal lcore ids are effectively hidden by passing physical CPU ids, any related options should take physical CPU ids too. I.e. the options for choosing main lcore and service lcores also need variants with other names.
The service cores are passed as a core mask; this should be deprecated (like "-c"). And superseded by a parameter taking a core list.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v2 0/3] allow easier use of high lcore-ids
2025-04-07 13:18 ` Morten Brørup
@ 2025-04-07 13:24 ` Bruce Richardson
0 siblings, 0 replies; 31+ messages in thread
From: Bruce Richardson @ 2025-04-07 13:24 UTC (permalink / raw)
To: Morten Brørup; +Cc: David Marchand, dev
On Mon, Apr 07, 2025 at 03:18:36PM +0200, Morten Brørup wrote:
> > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Sent: Monday, 7 April 2025 14.42
> >
> > On Mon, Apr 07, 2025 at 02:25:46PM +0200, Morten Brørup wrote:
> > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > > Sent: Monday, 7 April 2025 13.56
> > > >
> > > > On Mon, Apr 07, 2025 at 01:32:59PM +0200, Morten Brørup wrote:
> > > > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > > > > Sent: Monday, 7 April 2025 12.41
> > > > > >
> > > > > > On Mon, Apr 07, 2025 at 12:15:13PM +0200, Morten Brørup wrote:
> > > > > > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > > Sent:
> > > > > > > > Monday, 7 April 2025 11.49
> > > > > > > >
> > > > > > > > On Mon, Apr 07, 2025 at 09:04:05AM +0200, David Marchand
> > wrote:
> > > > > > > > > Hello Bruce,
> > > > > > > > >
> > > > > > > > > On Tue, Apr 1, 2025 at 4:08 PM Bruce Richardson
> > > > > > > > > <bruce.richardson@intel.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 24, 2025 at 05:30:26PM +0000, Bruce
> > Richardson
> > > > > > wrote:
> > > > > > > > > > > Traditionally, DPDK has had a direct mapping of
> > internal
> > > > > > lcore-
> > > > > > > > ids, to
> > > > > > > > > > > the actual core numbers in use. With higher core
> > count
> > > > > > servers
> > > > > > > > becoming
> > > > > > > > > > > more prevalent the issue becomes one of increasing
> > memory
> > > > > > > > footprint when
> > > > > > > > > > > using such a scheme, due to the need to have all
> > arrays
> > > > > > > > dimensioned for
> > > > > > > > > > > all cores on the system, whether or not those cores
> > are
> > > > in
> > > > > > use by
> > > > > > > > the
> > > > > > > > > > > app.
> > > > > > > > > > >
> > > > > > > > > > > Therefore, the decision was made in the past to not
> > > > expand
> > > > > > the
> > > > > > > > > > > build-time RTE_MAX_LCORE value beyond 128. Instead,
> > it
> > > > was
> > > > > > > > recommended
> > > > > > > > > > > that users use the "--lcores" EAL parameter to take
> > the
> > > > high-
> > > > > > > > numbered
> > > > > > > > > > > cores they wish to use and map them to lcore-ids
> > within
> > > > the 0
> > > > > > -
> > > > > > > > 128
> > > > > > > > > > > range. While this works, this is a little clunky as
> > it
> > > > means
> > > > > > that
> > > > > > > > > > > instead of just passing, for example, "-l 130-139",
> > the
> > > > user
> > > > > > must
> > > > > > > > > > > instead pass "--lcores 0@130,1@131,2@132,3@133,...."
> > > > > > > > > > >
> > > > > > > > > > > This patchset attempts to simplify the situation by
> > > > adding a
> > > > > > new
> > > > > > > > flag to
> > > > > > > > > > > do this mapping automatically. To use cores 130-139
> > and
> > > > map
> > > > > > them
> > > > > > > > to ids
> > > > > > > > > > > 0-9 internally, the EAL args now become: "-l 130-139
> > --
> > > > map-
> > > > > > lcore-
> > > > > > > > ids",
> > > > > > > > > > > or using the shorter "-M" version of the flag: "-Ml
> > 130-
> > > > 139".
> > > > > > > > > > >
> > > > > > > > > > > Adding this new parameter required some rework of the
> > > > > > existing
> > > > > > > > arg
> > > > > > > > > > > parsing code, because in current DPDK the args are
> > parsed
> > > > and
> > > > > > > > checked in
> > > > > > > > > > > the order they appear on the commandline. This means
> > that
> > > > > > using
> > > > > > > > the
> > > > > > > > > > > example above, the core parameter 130-139 will be
> > > > rejected
> > > > > > > > immediately
> > > > > > > > > > > before the "map-lcore-ids" parameter is seen. To work
> > > > around
> > > > > > > > this, the
> > > > > > > > > > > core (and service core) parameters are not parsed
> > when
> > > > seen,
> > > > > > > > instead
> > > > > > > > > > > they are only saved off and parsed after all
> > arguments
> > > > are
> > > > > > > > parsed. The
> > > > > > > > > > > "-l" and "-c" parameters are converted into "--
> > lcores"
> > > > > > arguments,
> > > > > > > > so all
> > > > > > > > > > > assigning of lcore ids is done there in all cases.
> > > > > > > > > > >
> > > > > > > > > > > RFC->v2: * converted printf to DEBUG log * added "-M"
> > as
> > > > > > shorter
> > > > > > > > > > > version of flag * added documentation * renamed
> > internal
> > > > API
> > > > > > that
> > > > > > > > > > > was changed to avoid any potential
> > > > > > > > hidden
> > > > > > > > > > > runtime issues.
> > > > > > > > > > >
> > > > > > > > > > > Bruce Richardson (3): eal: centralize core parameter
> > > > parsing
> > > > > > eal:
> > > > > > > > > > > convert core masks and lists to core sets eal: allow
> > > > > > automatic
> > > > > > > > > > > mapping of high lcore ids
> > > > > > > > > > >
> > > > > > > > > > Ping for review.
> > > > > > > > > >
> > > > > > > > > > At a high level, does this feature seem useful to
> > users?
> > > > > > > > >
> > > > > > > > > This seems useful, though I am not I would touch the
> > existing
> > > > > > > > options.
> > > > > > > > > I would have gone with a simple -L option (taking the
> > same
> > > > kind
> > > > > > of
> > > > > > > > > input than -l but with new behavior), and not combine a
> > flag
> > > > with
> > > > > > > > > existing options.
> > > > > > > > >
> > > > > > > >
> > > > > > > > That would be an easier patchset to do up. However, it
> > would
> > > > then
> > > > > > mean
> > > > > > > > that we have no less than 4 different ways to specify the
> > cores
> > > > to
> > > > > > use:
> > > > > > > > "- c", "-l", "-L", "--lcores" - and therefore need 4
> > different
> > > > sets
> > > > > > of
> > > > > > > > parsing options for them, and more checks to ensure we have
> > > > only
> > > > > > one of
> > > > > > > > the 4 specified in any run. That's why I chose the modifier
> > > > option,
> > > > > > and
> > > > > > > > to try and consolidate the core setup a bit.
> > > > > > > >
> > > > > > > > However, if having a completely new option is preferred, I
> > am
> > > > happy
> > > > > > > > enough to do up a different patchset for that.
> > > > > > > >
> > > > > > > > > I scanned through the series, not much to say. Maybe add
> > a
> > > > unit
> > > > > > test
> > > > > > > > > for new cmdline option.
> > > > > > > > >
> > > > > > > > Sure. Once it's decided what approach (if any) to take,
> > I'll do
> > > > up
> > > > > > a
> > > > > > > > new patchset, taking into account any relevant feedback on
> > this
> > > > > > set.
> > > > > > > >
> > > > > > > > /Bruce
> > > > > > >
> > > > > > > Changing the EAL parameter parser to a "two pass parser"
> > makes
> > > > sense.
> > > > > > I
> > > > > > > think checking for existence of more than one lcore
> > specification
> > > > > > options
> > > > > > > should suffice; we don't need to accept multiple lcore
> > > > specification
> > > > > > > options and check for conflicts.
> > > > > > >
> > > > > > > When remapping, do we need to support gaps in the "lcore"
> > > > (logical
> > > > > > cores)
> > > > > > > array, e.g. for secondary processes, or can it be continuous
> > from
> > > > 0
> > > > > > to
> > > > > > > the number of specified lcores?
> > > > > > >
> > > > > > > And are new EAL parameters for this really beneficial?
> > Doesn't
> > > > e.g.
> > > > > > "-l
> > > > > > > 0-9@130-139,100@140" suffice?
> > > > > > >
> > > > > > Actually, I believe "0-9@130-139"[1] is not the same as
> > > > > > "0@130,1@131,2@132,...". The latter affinities one thread to
> > one
> > > > core
> > > > > > ten
> > > > > > times over, while the former affinitizes 10 threads to 10 cores
> > -
> > > > > > leaving
> > > > > > those threads free to move about within the 10 cores specified.
> > > > >
> > > > > Interesting. The documentation [GSG] isn't clear to me about
> > this; a
> > > > example there could help clarify.
> > > > >
> > > > > [GSG]:
> > > >
> > https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#lcore-
> > > > related-options
> > > > >
> > > >
> > > > Yep, agreed.
> > > >
> > > > > If users are manually passing lcore parameters to the EAL, then I
> > see
> > > > why some sort of remapping shorthand is useful.
> > > > > IMO, if the mappings are relatively exotic, it should be
> > acceptable
> > > > requiring an external script to build a long list of mapping
> > parameters
> > > > and then invoke the application with those script-generated EAL
> > > > parameters.
> > > > > This would reduce the scope to support relatively simple, common
> > > > mappings.
> > > > >
> > > > > Could we expand the --lcores syntax to support common mappings?
> > > > >
> > > > > E.g. "0-9@130+" to do what I thought.
> > > > > The lack of "()" treats the entries individually (not as a
> > group),
> > > > and the "+" indicates auto-increment.
> > > > >
> > > > > A more advanced example:
> > > > > "0-9@(130-131)+", meaning lcore 0 gets cpus 130-131, lcore 1 gets
> > > > cpus 132-133, etc.
> > > > >
> > > >
> > > > My issues with the above syntax idea is:
> > > > * I think it's overly complicating the lcores parameter adding in
> > the
> > > > support for the "+" symbol - especially in the advanced case
> > example
> > > > you
> > > > provide. I worry about code maintainability here.
> > > > * More significantly for me, I think it's also getting things
> > backwards
> > > > in
> > > > that it is focusing more on the lcore ids visible to the app,
> > rather
> > > > than
> > > > the physical cores to be used. For the example above of 0-9@130+,
> > > > what I
> > > > would expect is that the user is mainly thinking about the cores
> > he
> > > > wants
> > > > to use 130-139, which are then to be mapped to lower ids. If we
> > had
> > > > the
> > > > syntax reversed where the physical cores were first, I'd say it
> > would
> > > > make more sense, e.g. 130-139@0+
> > >
> > > I 100 % agree on the syntax being backwards.
> > > A good reason for introducing a new parameter, rather than expanding
> > on "--lcores".
> > >
> >
> > Yes and no.
> > I agree with not expanding on --lcores, but I also don't think any new
> > parameter added should attempt to reproduce all of what lcores does. I
> > would leave --lcores as-is, as the power-tool for lcore config e.g.
> > what
> > you talk about below for mapping multiple lcores to the same physical
> > cpu.
> >
> > > We should consider deprecating the old (backwards) syntax, so users
> > don’t get confused about one EAL parameter being "backwards" of the
> > other.
> > >
> >
> > I disagree with this. I would be ok with deprecating the old "-c"
> > coremask
> > syntax - I think the time is long past when we should be dealing with
> > masks. However, removing "-l" and "--lcores" flag is, to me anyway, too
> > big
> > and jarring a change for end users for us to consider.
> >
> > > > * finally, as I found out last month, there are systems on which
> > the
> > > > cores
> > > > are spread across numa-nodes on odd/even boundaries, so to have
> > an
> > > > app
> > > > running on socket 0, you need to give the core ids as a list i.e.
> > > > 0,2,4,6, and cannot use ranges. [This also reenforces the point
> > above
> > > > too, where again it's the internal ids we need to generalize, not
> > the
> > > > physical cpus]
> > >
> > > For this, we could use "/2" (like in subnets), or "+2" as increment
> > parameter.
> > >
> > > >
> > > > My thinking on the matter, and I'm happy to be corrected if I'm
> > wrong
> > > > here,
> > > > is that end-users are unlikely to significantly care what the
> > actual
> > > > lcore
> > > > ids are internally in the program, so long as they work and are
> > unique.
> > >
> > > Generally yes.
> > > Note that we should allow the same physical CPU core to be assigned
> > to multiple lcores...
> > > If a CPU core is shared between multiple worker lcores, then it could
> > be problematic, but with a mix of lcore roles, this might be handy for
> > some applications. E.g. a virtual machine with only one CPU core could
> > use that single CPU core as both main and worker lcore, or as main and
> > service lcore.
> > > Sharing an lcore between threads requires the developer to take
> > special care of e.g. spinlocks, but the EAL parameter parser should not
> > prohibit it. It might log a notice, though.
> > >
> >
> > This is already taken care of via --lcores, so I don't see the need to
> > reimplement it using a new flag also. Any new flags we add should be
> > kept
> > deliberately simple.
> >
> > > > What does matter is the physical cpus on which the code is to run.
> > > > Therefore, my approach to this is to find the simplest possible
> > > > solution
> > > > whereby the user can just provide a list of cores and tell EAL to
> > just
> > > > map
> > > > them to reasonable values. For the "reasonable" values, I would
> > imagine
> > > > that for the vast majority of cases starting "0" is what is wanted.
> > For
> > > > anything beyond that, we already have the existing --lcores syntax
> > to
> > > > be
> > > > used.
> > >
> > > Agree with all of the above. :-)
> > >
> >
> > Great. That still leaves us with the problem of what exactly to do as
> > the
> > best solution. Here are some alternatives that I see:
> >
> > 1. Add a modifier flag for -l and -c parameters to auto-remap the lcore
> > ids
> > to zero, so user is just specifying physical CPU id's.
> > 2. Add a new flag for specifying physical cpu ids, which auto-remaps
> > the
> > cores specified to zero.
> > 2a. To simplify our code and user experience we could at the same
> > time:
> > * deprecate "-c" flag for coremasks
> > * make "-l" and "--lcores" the same flag just in long and short
> > versions. This should not break anything as "-l" parameter is
> > just as subset of what "--lcores" provides.
> > * that would leave us with effectively two core flag paths:
> > - -l/--lcores, behaviour as now, full explicit control
> > - -L/--lcores-remapped, takes simplified core list (only ","
> > and
> > "-" supported as with "-l" now), and maps them to zero-
> > based.
>
> +1 for 2a.
>
> >
> > Third options? Any other feedback?
>
> If the internal lcore ids are effectively hidden by passing physical CPU ids, any related options should take physical CPU ids too. I.e. the options for choosing main lcore and service lcores also need variants with other names.
Yes, I suppose so. That may be tricky, but we'll have to see what we can
do.
> The service cores are passed as a core mask; this should be deprecated (like "-c"). And superseded by a parameter taking a core list.
>
Yes to deprecate -s. I believe -S already exists to take a service core
list, so that can be marked off the list.
/Bruce
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v2 0/3] allow easier use of high lcore-ids
2025-04-07 10:15 ` Morten Brørup
2025-04-07 10:40 ` Bruce Richardson
@ 2025-04-07 15:14 ` Stephen Hemminger
2025-04-07 15:38 ` Bruce Richardson
1 sibling, 1 reply; 31+ messages in thread
From: Stephen Hemminger @ 2025-04-07 15:14 UTC (permalink / raw)
To: Morten Brørup; +Cc: Bruce Richardson, David Marchand, dev
On Mon, 7 Apr 2025 12:15:13 +0200
Morten Brørup <mb@smartsharesystems.com> wrote:
> > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Sent: Monday, 7 April 2025 11.49
> >
> > On Mon, Apr 07, 2025 at 09:04:05AM +0200, David Marchand wrote:
> > > Hello Bruce,
> > >
> > > On Tue, Apr 1, 2025 at 4:08 PM Bruce Richardson
> > > <bruce.richardson@intel.com> wrote:
> > > >
> > > > On Mon, Mar 24, 2025 at 05:30:26PM +0000, Bruce Richardson wrote:
> > > > > Traditionally, DPDK has had a direct mapping of internal lcore-
> > ids, to
> > > > > the actual core numbers in use. With higher core count servers
> > becoming
> > > > > more prevalent the issue becomes one of increasing memory
> > footprint when
> > > > > using such a scheme, due to the need to have all arrays
> > dimensioned for
> > > > > all cores on the system, whether or not those cores are in use by
> > the
> > > > > app.
> > > > >
> > > > > Therefore, the decision was made in the past to not expand the
> > > > > build-time RTE_MAX_LCORE value beyond 128. Instead, it was
> > recommended
> > > > > that users use the "--lcores" EAL parameter to take the high-
> > numbered
> > > > > cores they wish to use and map them to lcore-ids within the 0 -
> > 128
> > > > > range. While this works, this is a little clunky as it means that
> > > > > instead of just passing, for example, "-l 130-139", the user must
> > > > > instead pass "--lcores 0@130,1@131,2@132,3@133,...."
> > > > >
> > > > > This patchset attempts to simplify the situation by adding a new
> > flag to
> > > > > do this mapping automatically. To use cores 130-139 and map them
> > to ids
> > > > > 0-9 internally, the EAL args now become: "-l 130-139 --map-lcore-
> > ids",
> > > > > or using the shorter "-M" version of the flag: "-Ml 130-139".
> > > > >
> > > > > Adding this new parameter required some rework of the existing
> > arg
> > > > > parsing code, because in current DPDK the args are parsed and
> > checked in
> > > > > the order they appear on the commandline. This means that using
> > the
> > > > > example above, the core parameter 130-139 will be rejected
> > immediately
> > > > > before the "map-lcore-ids" parameter is seen. To work around
> > this, the
> > > > > core (and service core) parameters are not parsed when seen,
> > instead
> > > > > they are only saved off and parsed after all arguments are
> > parsed. The
> > > > > "-l" and "-c" parameters are converted into "--lcores" arguments,
> > so all
> > > > > assigning of lcore ids is done there in all cases.
> > > > >
> > > > > RFC->v2:
> > > > > * converted printf to DEBUG log
> > > > > * added "-M" as shorter version of flag
> > > > > * added documentation
> > > > > * renamed internal API that was changed to avoid any potential
> > hidden
> > > > > runtime issues.
> > > > >
> > > > > Bruce Richardson (3):
> > > > > eal: centralize core parameter parsing
> > > > > eal: convert core masks and lists to core sets
> > > > > eal: allow automatic mapping of high lcore ids
> > > > >
> > > > Ping for review.
> > > >
> > > > At a high level, does this feature seem useful to users?
> > >
> > > This seems useful, though I am not I would touch the existing
> > options.
> > > I would have gone with a simple -L option (taking the same kind of
> > > input than -l but with new behavior), and not combine a flag with
> > > existing options.
> > >
> >
> > That would be an easier patchset to do up. However, it would then mean
> > that
> > we have no less than 4 different ways to specify the cores to use: "-
> > c",
> > "-l", "-L", "--lcores" - and therefore need 4 different sets of parsing
> > options for them, and more checks to ensure we have only one of the 4
> > specified in any run. That's why I chose the modifier option, and to
> > try
> > and consolidate the core setup a bit.
> >
> > However, if having a completely new option is preferred, I am happy
> > enough
> > to do up a different patchset for that.
> >
> > > I scanned through the series, not much to say.
> > > Maybe add a unit test for new cmdline option.
> > >
> > Sure. Once it's decided what approach (if any) to take, I'll do up a
> > new
> > patchset, taking into account any relevant feedback on this set.
> >
> > /Bruce
>
> Changing the EAL parameter parser to a "two pass parser" makes sense.
> I think checking for existence of more than one lcore specification options should suffice; we don't need to accept multiple lcore specification options and check for conflicts.
There already is a first pass to catch log parameters, could the offset arg be handled there?
> When remapping, do we need to support gaps in the "lcore" (logical cores) array, e.g. for secondary processes, or can it be continuous from 0 to the number of specified lcores?
>
> And are new EAL parameters for this really beneficial?
> Doesn't e.g. "-l 0-9@130-139,100@140" suffice?
>
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v2 0/3] allow easier use of high lcore-ids
2025-04-07 15:14 ` Stephen Hemminger
@ 2025-04-07 15:38 ` Bruce Richardson
0 siblings, 0 replies; 31+ messages in thread
From: Bruce Richardson @ 2025-04-07 15:38 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: Morten Brørup, David Marchand, dev
On Mon, Apr 07, 2025 at 08:14:50AM -0700, Stephen Hemminger wrote:
> On Mon, 7 Apr 2025 12:15:13 +0200
> Morten Brørup <mb@smartsharesystems.com> wrote:
>
> > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > Sent: Monday, 7 April 2025 11.49
> > >
> > > On Mon, Apr 07, 2025 at 09:04:05AM +0200, David Marchand wrote:
> > > > Hello Bruce,
> > > >
> > > > On Tue, Apr 1, 2025 at 4:08 PM Bruce Richardson
> > > > <bruce.richardson@intel.com> wrote:
> > > > >
> > > > > On Mon, Mar 24, 2025 at 05:30:26PM +0000, Bruce Richardson wrote:
> > > > > > Traditionally, DPDK has had a direct mapping of internal lcore-
> > > ids, to
> > > > > > the actual core numbers in use. With higher core count servers
> > > becoming
> > > > > > more prevalent the issue becomes one of increasing memory
> > > footprint when
> > > > > > using such a scheme, due to the need to have all arrays
> > > dimensioned for
> > > > > > all cores on the system, whether or not those cores are in use by
> > > the
> > > > > > app.
> > > > > >
> > > > > > Therefore, the decision was made in the past to not expand the
> > > > > > build-time RTE_MAX_LCORE value beyond 128. Instead, it was
> > > recommended
> > > > > > that users use the "--lcores" EAL parameter to take the high-
> > > numbered
> > > > > > cores they wish to use and map them to lcore-ids within the 0 -
> > > 128
> > > > > > range. While this works, this is a little clunky as it means that
> > > > > > instead of just passing, for example, "-l 130-139", the user must
> > > > > > instead pass "--lcores 0@130,1@131,2@132,3@133,...."
> > > > > >
> > > > > > This patchset attempts to simplify the situation by adding a new
> > > flag to
> > > > > > do this mapping automatically. To use cores 130-139 and map them
> > > to ids
> > > > > > 0-9 internally, the EAL args now become: "-l 130-139 --map-lcore-
> > > ids",
> > > > > > or using the shorter "-M" version of the flag: "-Ml 130-139".
> > > > > >
> > > > > > Adding this new parameter required some rework of the existing
> > > arg
> > > > > > parsing code, because in current DPDK the args are parsed and
> > > checked in
> > > > > > the order they appear on the commandline. This means that using
> > > the
> > > > > > example above, the core parameter 130-139 will be rejected
> > > immediately
> > > > > > before the "map-lcore-ids" parameter is seen. To work around
> > > this, the
> > > > > > core (and service core) parameters are not parsed when seen,
> > > instead
> > > > > > they are only saved off and parsed after all arguments are
> > > parsed. The
> > > > > > "-l" and "-c" parameters are converted into "--lcores" arguments,
> > > so all
> > > > > > assigning of lcore ids is done there in all cases.
> > > > > >
> > > > > > RFC->v2:
> > > > > > * converted printf to DEBUG log
> > > > > > * added "-M" as shorter version of flag
> > > > > > * added documentation
> > > > > > * renamed internal API that was changed to avoid any potential
> > > hidden
> > > > > > runtime issues.
> > > > > >
> > > > > > Bruce Richardson (3):
> > > > > > eal: centralize core parameter parsing
> > > > > > eal: convert core masks and lists to core sets
> > > > > > eal: allow automatic mapping of high lcore ids
> > > > > >
> > > > > Ping for review.
> > > > >
> > > > > At a high level, does this feature seem useful to users?
> > > >
> > > > This seems useful, though I am not I would touch the existing
> > > options.
> > > > I would have gone with a simple -L option (taking the same kind of
> > > > input than -l but with new behavior), and not combine a flag with
> > > > existing options.
> > > >
> > >
> > > That would be an easier patchset to do up. However, it would then mean
> > > that
> > > we have no less than 4 different ways to specify the cores to use: "-
> > > c",
> > > "-l", "-L", "--lcores" - and therefore need 4 different sets of parsing
> > > options for them, and more checks to ensure we have only one of the 4
> > > specified in any run. That's why I chose the modifier option, and to
> > > try
> > > and consolidate the core setup a bit.
> > >
> > > However, if having a completely new option is preferred, I am happy
> > > enough
> > > to do up a different patchset for that.
> > >
> > > > I scanned through the series, not much to say.
> > > > Maybe add a unit test for new cmdline option.
> > > >
> > > Sure. Once it's decided what approach (if any) to take, I'll do up a
> > > new
> > > patchset, taking into account any relevant feedback on this set.
> > >
> > > /Bruce
> >
> > Changing the EAL parameter parser to a "two pass parser" makes sense.
> > I think checking for existence of more than one lcore specification options should suffice; we don't need to accept multiple lcore specification options and check for conflicts.
>
> There already is a first pass to catch log parameters, could the offset arg be handled there?
>
It could, but I'd rather not get into further handling of args in a
two-pass setup. If we go that way, we might be better to do a completely
"delayed-parsing" setup, where we use getopt to put all arguments into a
structure with named pointers for each arg type. Thereafter we do the
actual processing of args from the structure itself, allowing us to do all
arg processing in a fixed/known order. Unfortunately, that would be a
significant change in how things are done.
Also, from the discussion on this thread, there seems to be some support
for having a completely new cmdline arg that takes the core list and always
remaps them, rather than using a modifier to existing args. Your opinions
on the relative benefits/drawbacks of the two approaches are welcome! :-)
/Bruce
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v3 0/4] rework and expand EAL lcore options
2025-03-13 11:38 [RFC PATCH 0/3] allow easier use of high lcore-ids Bruce Richardson
` (3 preceding siblings ...)
2025-03-24 17:30 ` [PATCH v2 0/3] allow easier use of high lcore-ids Bruce Richardson
@ 2025-05-02 15:11 ` Bruce Richardson
2025-05-02 15:11 ` [PATCH v3 1/4] eal: deprecate old coremask-based EAL parameters Bruce Richardson
` (3 more replies)
4 siblings, 4 replies; 31+ messages in thread
From: Bruce Richardson @ 2025-05-02 15:11 UTC (permalink / raw)
To: dev; +Cc: david.marchand, mb, stephen, Bruce Richardson
Following discussion earlier on this thread on the previous RFCs and
patches submitted [1], this series is a slightly different take on
solving the same fundamental problem - making it easy to use CPU cores
with ids >= RTE_MAX_LCORE, without having massive amounts of code
complexity internally. As previously discussed[2] this patchset
implements proposal "2a" and does the following:
* marks the old coremask parameter (-c) and service core mask (-s) as
deprecated, so they can be dropped in future. Everything should now be
done using core lists where possible.
* consolidates the, previously separate but similar, "-l" and "--lcores"
flags into just short and long versions of the same thing. This does
not affect the "--lcores" flag at all, but expands the scope of what
was possible with "-l".
* a new flag, with long/short versions, is added as "-L" to allow the
user to just specify a range or set of physical CPU core ids to be
used by the app, and have them automatically mapped internally to a
set of lcore ids starting at 0. This provides the "easy solution" to
the original problem stated above.
For completeness, since it's in the same area, the previously separate
documentation patch, adding examples for the "--lcores" parameter, is
rolled into this series as patch 3.
[1] https://patches.dpdk.org/project/dpdk/list/?series=34911&state=*
[2] https://inbox.dpdk.org/dev/Z_PIBLQIT3eu6Ab3@bricha3-mobl1.ger.corp.intel.com/#t
Bruce Richardson (4):
eal: deprecate old coremask-based EAL parameters
eal: merge corelist and core mapping options
doc: provide examples of using lcores EAL parameter
eal: add option for auto-mapping cpu ids to low lcore ids
doc/guides/eventdevs/dlb2.rst | 6 +-
doc/guides/faq/faq.rst | 8 +-
doc/guides/linux_gsg/build_sample_apps.rst | 7 +-
doc/guides/linux_gsg/eal_args.include.rst | 61 ++++--
.../prog_guide/env_abstraction_layer.rst | 3 +-
doc/guides/prog_guide/meson_ut.rst | 2 +-
doc/guides/prog_guide/multi_proc_support.rst | 2 +-
doc/guides/prog_guide/service_cores.rst | 8 +-
doc/guides/rel_notes/deprecation.rst | 10 +
doc/guides/sample_app_ug/ip_frag.rst | 7 +-
doc/guides/sample_app_ug/ip_reassembly.rst | 7 +-
doc/guides/sample_app_ug/multi_process.rst | 14 +-
doc/guides/sample_app_ug/qos_scheduler.rst | 2 +-
doc/guides/sample_app_ug/test_pipeline.rst | 2 +-
doc/guides/tools/testbbdev.rst | 2 +-
lib/eal/common/eal_common_options.c | 202 +++++++-----------
lib/eal/common/eal_options.h | 6 +-
lib/eal/windows/include/sched.h | 1 +
18 files changed, 168 insertions(+), 182 deletions(-)
--
2.45.2
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v3 1/4] eal: deprecate old coremask-based EAL parameters
2025-05-02 15:11 ` [PATCH v3 0/4] rework and expand EAL lcore options Bruce Richardson
@ 2025-05-02 15:11 ` Bruce Richardson
2025-05-02 15:51 ` Stephen Hemminger
2025-05-02 15:11 ` [PATCH v3 2/4] eal: merge corelist and core mapping options Bruce Richardson
` (2 subsequent siblings)
3 siblings, 1 reply; 31+ messages in thread
From: Bruce Richardson @ 2025-05-02 15:11 UTC (permalink / raw)
To: dev; +Cc: david.marchand, mb, stephen, Bruce Richardson
As the number of cores/cpus on platforms has increased over the years,
the use of coremasks rather than core-lists for identifying DPDK cores
has become more and more unwieldy. At this point, let's deprecate the
coremask-based EAL parameters for future removal, and point users to the
core-list based versions instead.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
doc/guides/eventdevs/dlb2.rst | 6 +++---
doc/guides/faq/faq.rst | 8 +++-----
doc/guides/linux_gsg/build_sample_apps.rst | 7 +++----
doc/guides/linux_gsg/eal_args.include.rst | 8 ++------
doc/guides/prog_guide/meson_ut.rst | 2 +-
doc/guides/prog_guide/multi_proc_support.rst | 2 +-
doc/guides/prog_guide/service_cores.rst | 8 ++++----
doc/guides/rel_notes/deprecation.rst | 10 ++++++++++
doc/guides/sample_app_ug/ip_frag.rst | 7 +------
doc/guides/sample_app_ug/ip_reassembly.rst | 7 +------
doc/guides/sample_app_ug/multi_process.rst | 14 +++++---------
doc/guides/sample_app_ug/qos_scheduler.rst | 2 +-
doc/guides/sample_app_ug/test_pipeline.rst | 2 +-
doc/guides/tools/testbbdev.rst | 2 +-
lib/eal/common/eal_common_options.c | 6 ++++++
15 files changed, 43 insertions(+), 48 deletions(-)
diff --git a/doc/guides/eventdevs/dlb2.rst b/doc/guides/eventdevs/dlb2.rst
index 2532d92888..d1b736830d 100644
--- a/doc/guides/eventdevs/dlb2.rst
+++ b/doc/guides/eventdevs/dlb2.rst
@@ -408,9 +408,9 @@ the DLB device locally available on the same tile along with other
resources. To allocate optimal resources, probing is done for each
producer port (PP) for a given CPU and the best performing ports are
allocated to producers. The cpu used for probing is either the first
-core of producer coremask (if present) or the second core of EAL
-coremask. This will be extended later to probe for all CPUs in the
-producer coremask or EAL coremask. Producer coremask can be passed
+core of producer coremask DLB2 device parameter (if present) or the second core of EAL
+core list. This will be extended later to probe for all CPUs in the
+producer coremask or EAL core list. Producer coremask can be passed
along with the BDF of the DLB devices.
.. code-block:: console
diff --git a/doc/guides/faq/faq.rst b/doc/guides/faq/faq.rst
index 297cb5119e..ddcb018b3a 100644
--- a/doc/guides/faq/faq.rst
+++ b/doc/guides/faq/faq.rst
@@ -47,11 +47,9 @@ therefore all the hugepages are allocated on the wrong socket.
To avoid this scenario, either lower the amount of hugepage memory available to 1 GB size (or less), or run the application with taskset
affinitizing the application to a would-be main core.
-For example, if your EAL coremask is 0xff0, the main core will usually be the first core in the coremask (0x10); this is what you have to supply to taskset::
+For example, if your EAL core list is '4-11', the main core will usually be the first core in the list (core 4); this is what you have to supply to taskset::
- taskset 0x10 ./l2fwd -l 4-11 -n 2
-
-.. Note: Instead of '-c 0xff0' use the '-l 4-11' as a cleaner way to define lcores.
+ taskset -c 4 ./l2fwd -l 4-11 -n 2
In this way, the hugepages have a greater chance of being allocated to the correct socket.
Additionally, a ``--socket-mem`` option could be used to ensure the availability of memory for each socket, so that if hugepages were allocated on
@@ -167,7 +165,7 @@ Can I split packet RX to use DPDK and have an application's higher order functio
----------------------------------------------------------------------------------------------------------------
The DPDK's lcore threads are Linux pthreads bound onto specific cores. Configure the DPDK to do work on the same
-cores and run the application's other work on other cores using the DPDK's "coremask" setting to specify which
+cores and run the application's other work on other cores using the DPDK's "core list" (-l/--lcores) setting to specify which
cores it should launch itself on.
diff --git a/doc/guides/linux_gsg/build_sample_apps.rst b/doc/guides/linux_gsg/build_sample_apps.rst
index 433839ecc7..c568549f26 100644
--- a/doc/guides/linux_gsg/build_sample_apps.rst
+++ b/doc/guides/linux_gsg/build_sample_apps.rst
@@ -126,10 +126,9 @@ and that cores 0-3 are present and are to be used for running the application)::
Logical Core Use by Applications
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The coremask (-c 0x0f) or corelist (-l 0-3) parameter is always mandatory for DPDK applications.
-Each bit of the mask corresponds to the equivalent logical core number as reported by Linux. The preferred corelist option is a cleaner method to define cores to be used.
+The corelist (-l/--lcores 0-3) parameter is always mandatory for DPDK applications.
Since these logical core numbers, and their mapping to specific cores on specific NUMA sockets, can vary from platform to platform,
-it is recommended that the core layout for each platform be considered when choosing the coremask/corelist to use in each case.
+it is recommended that the core layout for each platform be considered when choosing the corelist to use in each case.
On initialization of the EAL layer by a DPDK application, the logical cores to be used and their socket location are displayed.
This information can also be determined for all cores on the system by examining the ``/proc/cpuinfo`` file, for example, by running cat ``/proc/cpuinfo``.
@@ -151,7 +150,7 @@ This can be useful when using other processors to understand the mapping of the
.. warning::
- The logical core layout can change between different board layouts and should be checked before selecting an application coremask/corelist.
+ The logical core layout can change between different board layouts and should be checked before selecting an application corelist.
Hugepage Memory Use by Applications
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/doc/guides/linux_gsg/eal_args.include.rst b/doc/guides/linux_gsg/eal_args.include.rst
index 9cfbf7de84..7ffd2e2535 100644
--- a/doc/guides/linux_gsg/eal_args.include.rst
+++ b/doc/guides/linux_gsg/eal_args.include.rst
@@ -4,10 +4,6 @@
Lcore-related options
~~~~~~~~~~~~~~~~~~~~~
-* ``-c <core mask>``
-
- Set the hexadecimal bitmask of the cores to run on.
-
* ``-l <core list>``
List of cores to run on
@@ -37,9 +33,9 @@ Lcore-related options
Core ID that is used as main.
-* ``-s <service core mask>``
+* ``-S <service core list>``
- Hexadecimal bitmask of cores to be used as service cores.
+ List of cores to be used as service cores.
Device-related options
~~~~~~~~~~~~~~~~~~~~~~
diff --git a/doc/guides/prog_guide/meson_ut.rst b/doc/guides/prog_guide/meson_ut.rst
index 78cf3f845c..9bc52a30fc 100644
--- a/doc/guides/prog_guide/meson_ut.rst
+++ b/doc/guides/prog_guide/meson_ut.rst
@@ -60,7 +60,7 @@ Arguments of ``test()`` that can be provided in meson.build are as below:
Note: the content of meson ``--test-args`` option and the content of ``args``
are appended when invoking the DPDK test binary.
-Because of this, it is recommended not to set any default coremask or memory
+Because of this, it is recommended not to set any default corelist or memory
configuration in per test ``args`` and rather let users select what best fits
their environment. If a test can't run, then it should be skipped, as described
below.
diff --git a/doc/guides/prog_guide/multi_proc_support.rst b/doc/guides/prog_guide/multi_proc_support.rst
index 0c57145470..dd57b6f8c1 100644
--- a/doc/guides/prog_guide/multi_proc_support.rst
+++ b/doc/guides/prog_guide/multi_proc_support.rst
@@ -166,7 +166,7 @@ Some of these are documented below:
so it is recommended that it be disabled only when absolutely necessary,
and only when the implications of this change have been understood.
-* All DPDK processes running as a single application and using shared memory must have distinct coremask/corelist arguments.
+* All DPDK processes running as a single application and using shared memory must have distinct corelist arguments.
It is not possible to have a primary and secondary instance, or two secondary instances,
using any of the same logical cores.
Attempting to do so can cause corruption of memory pool caches, among other issues.
diff --git a/doc/guides/prog_guide/service_cores.rst b/doc/guides/prog_guide/service_cores.rst
index d4e6c3d6e6..5284eeb96a 100644
--- a/doc/guides/prog_guide/service_cores.rst
+++ b/doc/guides/prog_guide/service_cores.rst
@@ -26,10 +26,10 @@ Service Core Initialization
~~~~~~~~~~~~~~~~~~~~~~~~~~~
There are two methods to having service cores in a DPDK application, either by
-using the service coremask, or by dynamically adding cores using the API.
-The simpler of the two is to pass the `-s` coremask argument to EAL, which will
-take any cores available in the main DPDK coremask, and if the bits are also set
-in the service coremask the cores become service-cores instead of DPDK
+using the service corelist, or by dynamically adding cores using the API.
+The simpler of the two is to pass the `-S` corelist argument to EAL, which will
+take any cores available in the main DPDK corelist, and if also set
+in the service corelist the cores become service-cores instead of DPDK
application lcores.
Enabling Services on Cores
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 36489f6e68..2ea898ff8a 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -17,6 +17,16 @@ Other API and ABI deprecation notices are to be posted below.
Deprecation Notices
-------------------
+* EAL: The ``-c <coremask>`` commandline parameter is deprecated
+ and will be removed in a future release.
+ Use the ``-l <corelist>`` or ``--lcores=<corelist>`` parameters instead
+ to specify the cores to be used when running a DPDK application.
+
+* EAL: The ``-s <service-coremask>`` commandline parameter is deprecated
+ and will be removed in a future release.
+ Use the ``-S <service-corelist>`` parameter instead
+ to specify the cores to be used for background services in DPDK.
+
* build: The ``enable_kmods`` option is deprecated and will be removed in a future release.
Setting/clearing the option has no impact on the build.
Instead, kernel modules will be always built for OS's where out-of-tree kernel modules
diff --git a/doc/guides/sample_app_ug/ip_frag.rst b/doc/guides/sample_app_ug/ip_frag.rst
index 4b2071cbae..3f0b5ce5e7 100644
--- a/doc/guides/sample_app_ug/ip_frag.rst
+++ b/doc/guides/sample_app_ug/ip_frag.rst
@@ -67,12 +67,7 @@ To run the example in linux environment with 2 lcores (2,4) over 2 ports(0,2) wi
.. code-block:: console
./<build_dir>/examples/dpdk-ip_fragmentation -l 2,4 -n 3 -- -p 5
- EAL: coremask set to 14
- EAL: Detected lcore 0 on socket 0
- EAL: Detected lcore 1 on socket 1
- EAL: Detected lcore 2 on socket 0
- EAL: Detected lcore 3 on socket 1
- EAL: Detected lcore 4 on socket 0
+ EAL: Detected CPU lcores: ...
...
Initializing port 0 on lcore 2... Address:00:1B:21:76:FA:2C, rxq=0 txq=2,0 txq=4,1
diff --git a/doc/guides/sample_app_ug/ip_reassembly.rst b/doc/guides/sample_app_ug/ip_reassembly.rst
index b8800dd9e7..e55ee7eb79 100644
--- a/doc/guides/sample_app_ug/ip_reassembly.rst
+++ b/doc/guides/sample_app_ug/ip_reassembly.rst
@@ -63,12 +63,7 @@ with 1 Rx queue per lcore:
.. code-block:: console
./<build_dir>/examples/dpdk-ip_reassembly -l 2,4 -n 3 -- -p 5
- EAL: coremask set to 14
- EAL: Detected lcore 0 on socket 0
- EAL: Detected lcore 1 on socket 1
- EAL: Detected lcore 2 on socket 0
- EAL: Detected lcore 3 on socket 1
- EAL: Detected lcore 4 on socket 0
+ EAL: Detected CPU lcores: ...
...
Initializing port 0 on lcore 2... Address:00:1B:21:76:FA:2C, rxq=0 txq=2,0 txq=4,1
diff --git a/doc/guides/sample_app_ug/multi_process.rst b/doc/guides/sample_app_ug/multi_process.rst
index e2b1b16c84..29bca806e1 100644
--- a/doc/guides/sample_app_ug/multi_process.rst
+++ b/doc/guides/sample_app_ug/multi_process.rst
@@ -36,7 +36,7 @@ Running the Application
^^^^^^^^^^^^^^^^^^^^^^^
To run the application, start ``simple_mp`` binary in one terminal,
-passing at least two cores in the coremask/corelist:
+passing at least two cores in the corelist:
.. code-block:: console
@@ -50,11 +50,7 @@ The process should start successfully and display a command prompt as follows:
.. code-block:: console
$ ./<build_dir>/examples/dpdk-simple_mp -l 0-1 -n 4 --proc-type=primary
- EAL: coremask set to 3
- EAL: Detected lcore 0 on socket 0
- EAL: Detected lcore 1 on socket 0
- EAL: Detected lcore 2 on socket 0
- EAL: Detected lcore 3 on socket 0
+ EAL: Detected CPU lcores: ...
...
EAL: Requesting 2 pages of size 1073741824
@@ -72,7 +68,7 @@ The process should start successfully and display a command prompt as follows:
simple_mp >
To run the secondary process to communicate with the primary process,
-again run the same binary setting at least two cores in the coremask/corelist:
+again run the same binary setting at least two cores in the corelist:
.. code-block:: console
@@ -237,8 +233,8 @@ In addition to the EAL parameters, the application-specific parameters are:
.. note::
In the server process, has a single thread using the lowest numbered lcore
- in the coremask/corelist, performs all packet I/O.
- If coremask/corelist parameter specifies with more than a single lcore bit set,
+ in the corelist, performs all packet I/O.
+ If corelist parameter specifies with more than a single lcore,
an additional lcore will be used for a thread to print packet count periodically.
The server application stores configuration data in shared memory,
diff --git a/doc/guides/sample_app_ug/qos_scheduler.rst b/doc/guides/sample_app_ug/qos_scheduler.rst
index 9936b99172..36ada4902c 100644
--- a/doc/guides/sample_app_ug/qos_scheduler.rst
+++ b/doc/guides/sample_app_ug/qos_scheduler.rst
@@ -194,7 +194,7 @@ Another example with 2 packet flow configurations using different ports but shar
Note that independent cores for the packet flow configurations for each of the RX, WT and TX thread are also supported,
providing flexibility to balance the work.
-The EAL coremask/corelist is constrained to contain the default main core 1 and the RX, WT and TX cores only.
+The EAL corelist is constrained to contain the default main core 1 and the RX, WT and TX cores only.
Explanation
-----------
diff --git a/doc/guides/sample_app_ug/test_pipeline.rst b/doc/guides/sample_app_ug/test_pipeline.rst
index d57d08fb2c..818be93cd6 100644
--- a/doc/guides/sample_app_ug/test_pipeline.rst
+++ b/doc/guides/sample_app_ug/test_pipeline.rst
@@ -47,7 +47,7 @@ The application execution command line is:
./dpdk-test-pipeline [EAL options] -- -p PORTMASK --TABLE_TYPE
-The -c or -l EAL CPU coremask/corelist option has to contain exactly 3 CPU cores.
+The ``-l/--lcores`` EAL CPU corelist option has to contain exactly 3 CPU cores.
The first CPU core in the core mask is assigned for core A, the second for core B and the third for core C.
The PORTMASK parameter must contain 2 or 4 ports.
diff --git a/doc/guides/tools/testbbdev.rst b/doc/guides/tools/testbbdev.rst
index ddb8d787be..8677cd2c43 100644
--- a/doc/guides/tools/testbbdev.rst
+++ b/doc/guides/tools/testbbdev.rst
@@ -78,7 +78,7 @@ The following are the command-line options:
``-l NUM_LCORES, --num_lcores NUM_LCORES``
Specifies number of lcores to run. If not specified num_lcores is set
- according to value from RTE configuration (EAL coremask)
+ according to value from RTE configuration (EAL corelist)
``-b BURST_SIZE [BURST_SIZE ...], --burst-size BURST_SIZE [BURST_SIZE ...]``
Specifies operations enqueue/dequeue burst size. If not specified burst_size is
diff --git a/lib/eal/common/eal_common_options.c b/lib/eal/common/eal_common_options.c
index b6fff7ec05..19c5997c7c 100644
--- a/lib/eal/common/eal_common_options.c
+++ b/lib/eal/common/eal_common_options.c
@@ -614,6 +614,9 @@ eal_parse_service_coremask(const char *coremask)
int val;
uint32_t taken_lcore_count = 0;
+ EAL_LOG(WARNING, "'-s <service-coremask>' is deprecated, and will be removed in a future release.");
+ EAL_LOG(WARNING, "\tUse '-S <service-corelist>' option instead.");
+
if (coremask == NULL)
return -1;
/* Remove all blank characters ahead and after .
@@ -777,6 +780,9 @@ rte_eal_parse_coremask(const char *coremask, int *cores)
cores[idx] = -1;
idx = 0;
+ EAL_LOG(WARNING, "'-c <coremask>' option is deprecated, and will be removed in a future release");
+ EAL_LOG(WARNING, "\tUse '-l <corelist>' or '--lcores=<corelist>' option instead");
+
/* Remove all blank characters ahead and after .
* Remove 0x/0X if exists.
*/
--
2.45.2
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v3 2/4] eal: merge corelist and core mapping options
2025-05-02 15:11 ` [PATCH v3 0/4] rework and expand EAL lcore options Bruce Richardson
2025-05-02 15:11 ` [PATCH v3 1/4] eal: deprecate old coremask-based EAL parameters Bruce Richardson
@ 2025-05-02 15:11 ` Bruce Richardson
2025-05-02 15:11 ` [PATCH v3 3/4] doc: provide examples of using lcores EAL parameter Bruce Richardson
2025-05-02 15:11 ` [PATCH v3 4/4] eal: add option for auto-mapping cpu ids to low lcore ids Bruce Richardson
3 siblings, 0 replies; 31+ messages in thread
From: Bruce Richardson @ 2025-05-02 15:11 UTC (permalink / raw)
To: dev; +Cc: david.marchand, mb, stephen, Bruce Richardson
The "-l" EAL parameter supported a subset of the features that were
provided by the "--lcores" long option. There is no need to have two
different options with different behaviour in this case, so we can just
eliminate the special-case handling for "-l", and have it as a shortened
form of "--lcores".
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
doc/guides/linux_gsg/eal_args.include.rst | 10 +-
.../prog_guide/env_abstraction_layer.rst | 3 +-
lib/eal/common/eal_common_options.c | 135 ++----------------
lib/eal/common/eal_options.h | 4 +-
4 files changed, 21 insertions(+), 131 deletions(-)
diff --git a/doc/guides/linux_gsg/eal_args.include.rst b/doc/guides/linux_gsg/eal_args.include.rst
index 7ffd2e2535..01fe6a3006 100644
--- a/doc/guides/linux_gsg/eal_args.include.rst
+++ b/doc/guides/linux_gsg/eal_args.include.rst
@@ -4,16 +4,14 @@
Lcore-related options
~~~~~~~~~~~~~~~~~~~~~
-* ``-l <core list>``
+* ``-l/--lcores <core list>``
List of cores to run on
- The argument format is ``<c1>[-c2][,c3[-c4],...]``
- where ``c1``, ``c2``, etc are core indexes between 0 and 128.
+ Simplest argument format is ``<c1>[-c2][,c3[-c4],...]``
+ where ``c1``, ``c2``, etc are core indexes between 0 and ``RTE_MAX_LCORE`` (default 128).
-* ``--lcores <core map>``
-
- Map lcore set to physical cpu set
+ This argument can also be used to map lcore set to physical cpu set
The argument format is::
diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 04214a05b2..fca9cc0afa 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -729,7 +729,7 @@ As EAL pthreads usually bind 1:1 to the physical CPU, the *_lcore_id* is typical
When using multiple pthreads, however, the binding is no longer always 1:1 between an EAL pthread and a specified physical CPU.
The EAL pthread may have affinity to a CPU set, and as such the *_lcore_id* will not be the same as the CPU ID.
-For this reason, there is an EAL long option '--lcores' defined to assign the CPU affinity of lcores.
+For this reason, there is an EAL option ``--lcores`` (or just ``-l``) defined to assign the CPU affinity of lcores.
For a specified lcore ID or ID group, the option allows setting the CPU set for that EAL pthread.
The format pattern:
@@ -753,7 +753,6 @@ If a '\@cpu_set' value is not supplied, the value of 'cpu_set' will default to t
lcore 8 runs on cpuset 0x100 (cpu 8).
Using this option, for each given lcore ID, the associated CPUs can be assigned.
-It's also compatible with the pattern of corelist('-l') option.
non-EAL pthread support
~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/lib/eal/common/eal_common_options.c b/lib/eal/common/eal_common_options.c
index 19c5997c7c..ed514ec1d1 100644
--- a/lib/eal/common/eal_common_options.c
+++ b/lib/eal/common/eal_common_options.c
@@ -46,7 +46,6 @@
#define BITS_PER_HEX 4
#define LCORE_OPT_LST 1
#define LCORE_OPT_MSK 2
-#define LCORE_OPT_MAP 3
const char
eal_short_options[] =
@@ -921,85 +920,6 @@ eal_parse_service_corelist(const char *corelist)
return 0;
}
-static int
-eal_parse_corelist(const char *corelist, int *cores)
-{
- unsigned int count = 0, i;
- int lcores[RTE_MAX_LCORE];
- char *end = NULL;
- int min, max;
- int idx;
-
- for (idx = 0; idx < RTE_MAX_LCORE; idx++)
- cores[idx] = -1;
-
- /* Remove all blank characters ahead */
- while (isblank(*corelist))
- corelist++;
-
- /* Get list of cores */
- min = -1;
- do {
- while (isblank(*corelist))
- corelist++;
- if (*corelist == '\0')
- return -1;
- errno = 0;
- idx = strtol(corelist, &end, 10);
- if (errno || end == NULL)
- return -1;
- if (idx < 0)
- return -1;
- while (isblank(*end))
- end++;
- if (*end == '-') {
- min = idx;
- } else if ((*end == ',') || (*end == '\0')) {
- max = idx;
- if (min == -1)
- min = idx;
- for (idx = min; idx <= max; idx++) {
- bool dup = false;
-
- /* Check if this idx is already present */
- for (i = 0; i < count; i++) {
- if (lcores[i] == idx)
- dup = true;
- }
- if (dup)
- continue;
- if (count >= RTE_MAX_LCORE) {
- EAL_LOG(ERR, "Too many lcores provided. Cannot exceed RTE_MAX_LCORE (%d)",
- RTE_MAX_LCORE);
- return -1;
- }
- lcores[count++] = idx;
- }
- min = -1;
- } else
- return -1;
- corelist = end + 1;
- } while (*end != '\0');
-
- if (count == 0)
- return -1;
-
- if (check_core_list(lcores, count))
- return -1;
-
- /*
- * Now that we've got a list of cores no longer than RTE_MAX_LCORE,
- * and no lcore in that list is greater than RTE_MAX_LCORE, populate
- * the cores array.
- */
- do {
- count--;
- cores[lcores[count]] = count;
- } while (count != 0);
-
- return 0;
-}
-
/* Changes the lcore id of the main thread */
static int
eal_parse_main_lcore(const char *arg)
@@ -1703,10 +1623,10 @@ eal_parse_common_option(int opt, const char *optarg,
}
if (core_parsed) {
- EAL_LOG(ERR, "Option -c is ignored, because (%s) is set!",
- (core_parsed == LCORE_OPT_LST) ? "-l" :
- (core_parsed == LCORE_OPT_MAP) ? "--lcores" :
- "-c");
+ if (core_parsed == LCORE_OPT_MSK)
+ EAL_LOG(ERR, "Option '-c' passed multiple times to EAL");
+ else
+ EAL_LOG(ERR, "Option -c is ignored, because option -l/--lcores used");
return -1;
}
@@ -1715,31 +1635,20 @@ eal_parse_common_option(int opt, const char *optarg,
}
/* corelist */
case 'l': {
- int lcore_indexes[RTE_MAX_LCORE];
-
if (eal_service_cores_parsed())
EAL_LOG(WARNING,
"Service cores parsed before dataplane cores. Please ensure -l is before -s or -S");
- if (eal_parse_corelist(optarg, lcore_indexes) < 0) {
- EAL_LOG(ERR, "invalid core list syntax");
- return -1;
- }
- if (update_lcore_config(lcore_indexes) < 0) {
- char *available = available_cores();
-
- EAL_LOG(ERR,
- "invalid core list, please check specified cores are part of %s",
- available);
- free(available);
+ if (eal_parse_lcores(optarg) < 0) {
+ EAL_LOG(ERR, "invalid parameter for -l/--" OPT_LCORES);
return -1;
}
if (core_parsed) {
- EAL_LOG(ERR, "Option -l is ignored, because (%s) is set!",
- (core_parsed == LCORE_OPT_MSK) ? "-c" :
- (core_parsed == LCORE_OPT_MAP) ? "--lcores" :
- "-l");
+ if (core_parsed == LCORE_OPT_LST)
+ EAL_LOG(ERR, "Core list option passed multiple times to EAL");
+ else
+ EAL_LOG(ERR, "Option '-l/--lcores' is ignored, because coremask option used");
return -1;
}
@@ -1926,23 +1835,6 @@ eal_parse_common_option(int opt, const char *optarg,
}
#endif /* !RTE_EXEC_ENV_WINDOWS */
- case OPT_LCORES_NUM:
- if (eal_parse_lcores(optarg) < 0) {
- EAL_LOG(ERR, "invalid parameter for --"
- OPT_LCORES);
- return -1;
- }
-
- if (core_parsed) {
- EAL_LOG(ERR, "Option --lcores is ignored, because (%s) is set!",
- (core_parsed == LCORE_OPT_LST) ? "-l" :
- (core_parsed == LCORE_OPT_MSK) ? "-c" :
- "--lcores");
- return -1;
- }
-
- core_parsed = LCORE_OPT_MAP;
- break;
case OPT_LEGACY_MEM_NUM:
conf->legacy_mem = 1;
break;
@@ -2214,10 +2106,11 @@ eal_common_usage(void)
printf("[options]\n\n"
"EAL common options:\n"
" -c COREMASK Hexadecimal bitmask of cores to run on\n"
- " -l CORELIST List of cores to run on\n"
- " The argument format is <c1>[-c2][,c3[-c4],...]\n"
+ " -l, --"OPT_LCORES" CORELIST\n"
+ " List of cores to run on\n"
+ " The basic argument format is <c1>[-c2][,c3[-c4],...]\n"
" where c1, c2, etc are core indexes between 0 and %d\n"
- " --"OPT_LCORES" COREMAP Map lcore set to physical cpu set\n"
+ " Can also be used to map lcore set to physical cpu set\n"
" The argument format is\n"
" '<lcores[@cpus]>[<,lcores[@cpus]>...]'\n"
" lcores and cpus list are grouped by '(' and ')'\n"
diff --git a/lib/eal/common/eal_options.h b/lib/eal/common/eal_options.h
index 95fb4f6108..7b84b7d778 100644
--- a/lib/eal/common/eal_options.h
+++ b/lib/eal/common/eal_options.h
@@ -17,6 +17,8 @@ enum {
OPT_DEV_ALLOW_NUM = 'a',
#define OPT_DEV_BLOCK "block"
OPT_DEV_BLOCK_NUM = 'b',
+#define OPT_LCORES "lcores"
+ OPT_LCORES_NUM = 'l',
/* first long only option value must be >= 256, so that we won't
* conflict with short options */
@@ -31,8 +33,6 @@ enum {
OPT_HUGE_DIR_NUM,
#define OPT_HUGE_UNLINK "huge-unlink"
OPT_HUGE_UNLINK_NUM,
-#define OPT_LCORES "lcores"
- OPT_LCORES_NUM,
#define OPT_LOG_COLOR "log-color"
OPT_LOG_COLOR_NUM,
#define OPT_LOG_LEVEL "log-level"
--
2.45.2
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v3 3/4] doc: provide examples of using lcores EAL parameter
2025-05-02 15:11 ` [PATCH v3 0/4] rework and expand EAL lcore options Bruce Richardson
2025-05-02 15:11 ` [PATCH v3 1/4] eal: deprecate old coremask-based EAL parameters Bruce Richardson
2025-05-02 15:11 ` [PATCH v3 2/4] eal: merge corelist and core mapping options Bruce Richardson
@ 2025-05-02 15:11 ` Bruce Richardson
2025-05-02 15:11 ` [PATCH v3 4/4] eal: add option for auto-mapping cpu ids to low lcore ids Bruce Richardson
3 siblings, 0 replies; 31+ messages in thread
From: Bruce Richardson @ 2025-05-02 15:11 UTC (permalink / raw)
To: dev; +Cc: david.marchand, mb, stephen, Bruce Richardson
The "--lcores" EAL parameter has a very powerful syntax that can be used
to provide precise control over lcore mappings. The docs however, only
provided a minimal description of what it can do. Augment the docs by
providing some examples of use of the option, and what the resulting
core mappings would be.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
---
doc/guides/linux_gsg/eal_args.include.rst | 27 +++++++++++++++++++++++
1 file changed, 27 insertions(+)
diff --git a/doc/guides/linux_gsg/eal_args.include.rst b/doc/guides/linux_gsg/eal_args.include.rst
index 01fe6a3006..d530215784 100644
--- a/doc/guides/linux_gsg/eal_args.include.rst
+++ b/doc/guides/linux_gsg/eal_args.include.rst
@@ -23,6 +23,33 @@ Lcore-related options
The grouping ``()`` can be omitted for single element group.
The ``@`` can be omitted if cpus and lcores have the same value.
+ Examples:
+
+ ``--lcores=1-3``: Run threads on physical CPUs 1, 2 and 3,
+ with each thread having the same lcore id as the physical CPU id.
+
+ ``--lcores=1@(1,2)``: Run a single thread with lcore id 1,
+ but with that thread bound to both physical CPUs 1 and 2,
+ so it can run on either, as determined by the operating system.
+
+ ``--lcores='1@31,2@32,3@33'``: Run threads having internal lcore ids of 1, 2 and 3,
+ but with the threads being bound to physical CPUs 31, 32 and 33 respectively.
+
+ ``--lcores='(1-3)@(31-33)'``: Run three threads with lcore ids 1, 2 and 3.
+ Unlike the previous example above,
+ each of these threads is not bound to one specific physical CPU,
+ but rather, all three threads are instead bound to the three physical CPUs 31, 32 and 33.
+ This means that each of the three threads can move between the physical CPUs 31-33,
+ as decided by the OS as the application runs.
+
+ ``--lcores=(1-3)@20``: Run three threads, with lcore ids 1, 2 and 3,
+ where all three threads are bound to (can only run on) physical CPU 20.
+
+.. Note::
+ Binding multiple DPDK lcores to a single physical CPU can cause problems with poor performance
+ or deadlock when using DPDK rings or memory pools or spinlocks.
+ Such a configuration should only be used with care.
+
.. Note::
At a given instance only one core option ``--lcores``, ``-l`` or ``-c`` can
be used.
--
2.45.2
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v3 4/4] eal: add option for auto-mapping cpu ids to low lcore ids
2025-05-02 15:11 ` [PATCH v3 0/4] rework and expand EAL lcore options Bruce Richardson
` (2 preceding siblings ...)
2025-05-02 15:11 ` [PATCH v3 3/4] doc: provide examples of using lcores EAL parameter Bruce Richardson
@ 2025-05-02 15:11 ` Bruce Richardson
3 siblings, 0 replies; 31+ messages in thread
From: Bruce Richardson @ 2025-05-02 15:11 UTC (permalink / raw)
To: dev; +Cc: david.marchand, mb, stephen, Bruce Richardson
When attempting to run DPDK with cpu ids > RTE_MAX_LCORE, it can be
awkward to build up a command-line parameter specifying the cpu to lcore
mappings core-by-core. To simplify things for the common case, add in
the -L, --lcores-automap option, to take a list of cpu ids, which may or
may not be >= RTE_MAX_LCORE, and map them to lcore ids starting at zero.
For example, running lcores_autotest with dpdk-test on cores 140-143
with a build where RTE_MAX_LCORE is 128, is now as simple as:
$ ./dpdk-test -L 140-143 -- lcores_autotest
...
EAL threads count: 4, RTE_MAX_LCORE=128
lcore 0, socket 1, role RTE, cpuset 140
lcore 1, socket 1, role RTE, cpuset 141
lcore 2, socket 1, role RTE, cpuset 142
lcore 3, socket 1, role RTE, cpuset 143
...
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
doc/guides/linux_gsg/eal_args.include.rst | 16 +++++-
lib/eal/common/eal_common_options.c | 69 +++++++++++++++++++++--
lib/eal/common/eal_options.h | 2 +
lib/eal/windows/include/sched.h | 1 +
4 files changed, 81 insertions(+), 7 deletions(-)
diff --git a/doc/guides/linux_gsg/eal_args.include.rst b/doc/guides/linux_gsg/eal_args.include.rst
index d530215784..d8257fe8fc 100644
--- a/doc/guides/linux_gsg/eal_args.include.rst
+++ b/doc/guides/linux_gsg/eal_args.include.rst
@@ -34,6 +34,7 @@ Lcore-related options
``--lcores='1@31,2@32,3@33'``: Run threads having internal lcore ids of 1, 2 and 3,
but with the threads being bound to physical CPUs 31, 32 and 33 respectively.
+ [See also the ``-L/--lcores-automap`` option below.]
``--lcores='(1-3)@(31-33)'``: Run three threads with lcore ids 1, 2 and 3.
Unlike the previous example above,
@@ -50,9 +51,20 @@ Lcore-related options
or deadlock when using DPDK rings or memory pools or spinlocks.
Such a configuration should only be used with care.
+* ``-L/--lcores-automap <core list>``
+
+ List of CPUs to run on, using lcore ids starting at 0.
+
+ Argument format is as <c1>[-c2][,c3[-c4],...]
+
+ One lcore thread will be spawned per cpu id specified,
+ and pinned to the specified cpu id, but with the lowest available lcore id.
+
+ This provides a convenient way to use CPU cores with ids greater than RTE_MAX_LCORE value.
+
.. Note::
- At a given instance only one core option ``--lcores``, ``-l`` or ``-c`` can
- be used.
+ At a given instance only one core option ``--lcores``, ``-l``,
+ ``--lcores-automap``, ``-L`` or ``-c`` can be used.
* ``--main-lcore <core ID>``
diff --git a/lib/eal/common/eal_common_options.c b/lib/eal/common/eal_common_options.c
index ed514ec1d1..c77654c365 100644
--- a/lib/eal/common/eal_common_options.c
+++ b/lib/eal/common/eal_common_options.c
@@ -56,6 +56,7 @@ eal_short_options[] =
"d:" /* driver */
"h" /* help */
"l:" /* corelist */
+ "L:" /* corelist with auto lcore id remapping */
"S:" /* service corelist */
"m:" /* memory size */
"n:" /* memory channels */
@@ -1223,6 +1224,52 @@ eal_parse_lcores(const char *lcores)
return ret;
}
+static int
+eal_parse_remapped_lcores(const char *optarg)
+{
+ struct rte_config *cfg = rte_eal_get_configuration();
+ const size_t optarg_len = strlen(optarg);
+ rte_cpuset_t cpuset;
+
+ /* Reset lcore config */
+ for (unsigned int idx = 0; idx < RTE_MAX_LCORE; idx++) {
+ cfg->lcore_role[idx] = ROLE_OFF;
+ lcore_config[idx].core_index = -1;
+ CPU_ZERO(&lcore_config[idx].cpuset);
+ }
+
+ /* the eal_parse_set API only handles "," within (), so wrap string */
+ char *tmp_optarg = malloc(optarg_len + 3);
+ if (tmp_optarg == NULL) {
+ EAL_LOG(ERR, "Error with malloc for temporary optarg string");
+ return -1;
+ }
+ snprintf(tmp_optarg, optarg_len + 3, "(%s)", optarg);
+
+ /* parse wrapped string */
+ int parsed_chars = eal_parse_set(tmp_optarg, &cpuset);
+ free(tmp_optarg);
+ if (parsed_chars == -1 || CPU_COUNT(&cpuset) == 0) {
+ EAL_LOG(ERR, "Invalid corelist for remapping: %s", optarg);
+ return -1;
+ }
+
+ unsigned int lcore_id = 0;
+ unsigned int cpu_id = 0;
+ while (CPU_COUNT(&cpuset) > 0) {
+ if (CPU_ISSET(cpu_id, &cpuset)) {
+ cfg->lcore_role[lcore_id] = ROLE_RTE;
+ lcore_config[lcore_id].core_index = lcore_id;
+ CPU_SET(cpu_id, &lcore_config[lcore_id].cpuset);
+ CPU_CLR(cpu_id, &cpuset);
+ lcore_id++;
+ }
+ cpu_id++;
+ }
+
+ return 0;
+}
+
static void
eal_log_usage(void)
{
@@ -1634,21 +1681,25 @@ eal_parse_common_option(int opt, const char *optarg,
break;
}
/* corelist */
+ case 'L':
case 'l': {
if (eal_service_cores_parsed())
EAL_LOG(WARNING,
"Service cores parsed before dataplane cores. Please ensure -l is before -s or -S");
- if (eal_parse_lcores(optarg) < 0) {
- EAL_LOG(ERR, "invalid parameter for -l/--" OPT_LCORES);
+ int retval = opt == 'l' ?
+ eal_parse_lcores(optarg) :
+ eal_parse_remapped_lcores(optarg);
+ if (retval < 0) {
+ EAL_LOG(ERR, "invalid parameter for lcore list option: '%s'", optarg);
return -1;
}
if (core_parsed) {
if (core_parsed == LCORE_OPT_LST)
- EAL_LOG(ERR, "Core list option passed multiple times to EAL");
+ EAL_LOG(ERR, "Multiple core list options passed to EAL");
else
- EAL_LOG(ERR, "Option '-l/--lcores' is ignored, because coremask option used");
+ EAL_LOG(ERR, "Core list option is ignored, because coremask option used");
return -1;
}
@@ -2118,6 +2169,14 @@ eal_common_usage(void)
" ',' is used for single number separator.\n"
" '( )' can be omitted for single element group,\n"
" '@' can be omitted if cpus and lcores have the same value\n"
+ " -L, --"OPT_LCORES_AUTOMAP" CORELIST\n"
+ " List of CPUs to run on, using lcore ids starting at 0.\n"
+ " Argument format is as <c1>[-c2][,c3[-c4],...]\n"
+ " One lcore thread will be spawned per cpu id specified,\n"
+ " and pinned to the specified cpu id, but with the lowest\n"
+ " available lcore id.\n"
+ " This provides a convenient way to use CPU cores with ids\n"
+ " greater than RTE_MAX_LCORE value (%d)\n"
" -s SERVICE COREMASK Hexadecimal bitmask of cores to be used as service cores\n"
" --"OPT_MAIN_LCORE" ID Core ID that is used as main\n"
" --"OPT_MBUF_POOL_OPS_NAME" Pool ops name for mbuf to use\n"
@@ -2191,5 +2250,5 @@ eal_common_usage(void)
" --"OPT_NO_PCI" Disable PCI\n"
" --"OPT_NO_HPET" Disable HPET\n"
" --"OPT_NO_SHCONF" No shared config (mmap'd files)\n"
- "\n", RTE_MAX_LCORE);
+ "\n", RTE_MAX_LCORE, RTE_MAX_LCORE);
}
diff --git a/lib/eal/common/eal_options.h b/lib/eal/common/eal_options.h
index 7b84b7d778..cdea876635 100644
--- a/lib/eal/common/eal_options.h
+++ b/lib/eal/common/eal_options.h
@@ -19,6 +19,8 @@ enum {
OPT_DEV_BLOCK_NUM = 'b',
#define OPT_LCORES "lcores"
OPT_LCORES_NUM = 'l',
+#define OPT_LCORES_AUTOMAP "lcores-automap"
+ OPT_LCORES_AUTOMAP_NUM = 'L',
/* first long only option value must be >= 256, so that we won't
* conflict with short options */
diff --git a/lib/eal/windows/include/sched.h b/lib/eal/windows/include/sched.h
index 912fed12c2..04ee046bb7 100644
--- a/lib/eal/windows/include/sched.h
+++ b/lib/eal/windows/include/sched.h
@@ -31,6 +31,7 @@ typedef struct _rte_cpuset_s {
#define RTE_HAS_CPUSET
#define CPU_SET(b, s) ((s)->_bits[_WHICH_SET(b)] |= (1LL << _WHICH_BIT(b)))
+#define CPU_CLR(b, s) ((s)->_bits[_WHICH_SET(b)] &= ~(1LL << _WHICH_BIT(b)))
#define CPU_ZERO(s) \
do { \
--
2.45.2
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v3 1/4] eal: deprecate old coremask-based EAL parameters
2025-05-02 15:11 ` [PATCH v3 1/4] eal: deprecate old coremask-based EAL parameters Bruce Richardson
@ 2025-05-02 15:51 ` Stephen Hemminger
2025-05-02 16:00 ` Bruce Richardson
0 siblings, 1 reply; 31+ messages in thread
From: Stephen Hemminger @ 2025-05-02 15:51 UTC (permalink / raw)
To: Bruce Richardson; +Cc: dev, david.marchand, mb
On Fri, 2 May 2025 16:11:31 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:
> As the number of cores/cpus on platforms has increased over the years,
> the use of coremasks rather than core-lists for identifying DPDK cores
> has become more and more unwieldy. At this point, let's deprecate the
> coremask-based EAL parameters for future removal, and point users to the
> core-list based versions instead.
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Agree.
There are some more syntax options on Linux kernel command line:
https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/kernel-parameters.rst#cpu-lists
cpu lists
Some kernel parameters take a list of CPUs as a value, e.g. isolcpus, nohz_full, irqaffinity, rcu_nocbs. The format of this list is:
<cpu number>,...,<cpu number>
or
<cpu number>-<cpu number> (must be a positive range in ascending order)
or a mixture
<cpu number>,...,<cpu number>-<cpu number>
Note that for the special case of a range one can split the range into equal sized groups and for each group use some amount from the beginning of that group:
<cpu number>-<cpu number>:<used size>/<group size>
For example one can add to the command line following parameter:
isolcpus=1,2,10-20,100-2000:2/25
where the final item represents CPUs 100,101,125,126,150,151,...
The value "N" can be used to represent the numerically last CPU on the system, i.e "foo_cpus=16-N" would be equivalent to "16-31" on a 32 core system.
Keep in mind that "N" is dynamic, so if system changes cause the bitmap width to change, such as less cores in the CPU list, then N and any ranges using N will also change. Use the same on a small 4 core system, and "16-N" becomes "16-3" and now the same boot input will be flagged as invalid (start > end).
The special case-tolerant group name "all" has a meaning of selecting all CPUs, so that "nohz_full=all" is the equivalent of "nohz_full=0-N".
The semantics of "N" and "all" is supported on a level of bitmaps and holds for all users of bitmap_parselist().
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v3 1/4] eal: deprecate old coremask-based EAL parameters
2025-05-02 15:51 ` Stephen Hemminger
@ 2025-05-02 16:00 ` Bruce Richardson
2025-05-03 6:09 ` Morten Brørup
0 siblings, 1 reply; 31+ messages in thread
From: Bruce Richardson @ 2025-05-02 16:00 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev, david.marchand, mb
On Fri, May 02, 2025 at 08:51:22AM -0700, Stephen Hemminger wrote:
> On Fri, 2 May 2025 16:11:31 +0100
> Bruce Richardson <bruce.richardson@intel.com> wrote:
>
> > As the number of cores/cpus on platforms has increased over the years,
> > the use of coremasks rather than core-lists for identifying DPDK cores
> > has become more and more unwieldy. At this point, let's deprecate the
> > coremask-based EAL parameters for future removal, and point users to the
> > core-list based versions instead.
> >
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
>
> Agree.
>
> There are some more syntax options on Linux kernel command line:
> https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/kernel-parameters.rst#cpu-lists
>
> cpu lists
> Some kernel parameters take a list of CPUs as a value, e.g. isolcpus, nohz_full, irqaffinity, rcu_nocbs. The format of this list is:
>
> <cpu number>,...,<cpu number>
> or
>
> <cpu number>-<cpu number> (must be a positive range in ascending order)
> or a mixture
>
> <cpu number>,...,<cpu number>-<cpu number>
>
This we already support.
> Note that for the special case of a range one can split the range into equal sized groups and for each group use some amount from the beginning of that group:
>
> <cpu number>-<cpu number>:<used size>/<group size>
> For example one can add to the command line following parameter:
>
> isolcpus=1,2,10-20,100-2000:2/25
> where the final item represents CPUs 100,101,125,126,150,151,...
>
> The value "N" can be used to represent the numerically last CPU on the system, i.e "foo_cpus=16-N" would be equivalent to "16-31" on a 32 core system.
>
> Keep in mind that "N" is dynamic, so if system changes cause the bitmap width to change, such as less cores in the CPU list, then N and any ranges using N will also change. Use the same on a small 4 core system, and "16-N" becomes "16-3" and now the same boot input will be flagged as invalid (start > end).
>
> The special case-tolerant group name "all" has a meaning of selecting all CPUs, so that "nohz_full=all" is the equivalent of "nohz_full=0-N".
>
> The semantics of "N" and "all" is supported on a level of bitmaps and holds for all users of bitmap_parselist().
>
Those would be good/interesting extensions to have, if someone wants to add
them to EAL. Hopefully after this patchset, adding them should be easier as
I think we should only have one place in EAL where we parse groups like that:
"eal_parse_set()" function.
/Bruce
^ permalink raw reply [flat|nested] 31+ messages in thread
* RE: [PATCH v3 1/4] eal: deprecate old coremask-based EAL parameters
2025-05-02 16:00 ` Bruce Richardson
@ 2025-05-03 6:09 ` Morten Brørup
0 siblings, 0 replies; 31+ messages in thread
From: Morten Brørup @ 2025-05-03 6:09 UTC (permalink / raw)
To: Bruce Richardson, Stephen Hemminger; +Cc: dev, david.marchand
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Friday, 2 May 2025 18.01
>
> On Fri, May 02, 2025 at 08:51:22AM -0700, Stephen Hemminger wrote:
> > On Fri, 2 May 2025 16:11:31 +0100
> > Bruce Richardson <bruce.richardson@intel.com> wrote:
> >
> > > As the number of cores/cpus on platforms has increased over the
> years,
> > > the use of coremasks rather than core-lists for identifying DPDK
> cores
> > > has become more and more unwieldy. At this point, let's deprecate
> the
> > > coremask-based EAL parameters for future removal, and point users
> to the
> > > core-list based versions instead.
> > >
> > > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> >
> > Agree.
> >
> > There are some more syntax options on Linux kernel command line:
> > https://github.com/torvalds/linux/blob/master/Documentation/admin-
> guide/kernel-parameters.rst#cpu-lists
> >
> > cpu lists
> > Some kernel parameters take a list of CPUs as a value, e.g. isolcpus,
> nohz_full, irqaffinity, rcu_nocbs. The format of this list is:
> >
> > <cpu number>,...,<cpu number>
> > or
> >
> > <cpu number>-<cpu number> (must be a positive range in ascending
> order)
> > or a mixture
> >
> > <cpu number>,...,<cpu number>-<cpu number>
> >
>
> This we already support.
>
> > Note that for the special case of a range one can split the range
> into equal sized groups and for each group use some amount from the
> beginning of that group:
> >
> > <cpu number>-<cpu number>:<used size>/<group size>
> > For example one can add to the command line following parameter:
> >
> > isolcpus=1,2,10-20,100-2000:2/25
> > where the final item represents CPUs 100,101,125,126,150,151,...
> >
> > The value "N" can be used to represent the numerically last CPU on
> the system, i.e "foo_cpus=16-N" would be equivalent to "16-31" on a 32
> core system.
> >
> > Keep in mind that "N" is dynamic, so if system changes cause the
> bitmap width to change, such as less cores in the CPU list, then N and
> any ranges using N will also change. Use the same on a small 4 core
> system, and "16-N" becomes "16-3" and now the same boot input will be
> flagged as invalid (start > end).
> >
> > The special case-tolerant group name "all" has a meaning of selecting
> all CPUs, so that "nohz_full=all" is the equivalent of "nohz_full=0-N".
> >
> > The semantics of "N" and "all" is supported on a level of bitmaps and
> holds for all users of bitmap_parselist().
> >
>
> Those would be good/interesting extensions to have, if someone wants to
> add
> them to EAL. Hopefully after this patchset, adding them should be
> easier as
> I think we should only have one place in EAL where we parse groups like
> that:
> "eal_parse_set()" function.
Yes, such a syntax parser can be added later.
In the Kernel, it's a generic parser for fixed length bit arrays.
I could imagine using it for other purposes than CPUs, e.g. VLANs or ports.
The "N" feature is *must have* in the Kernel, where it's used for Kernel boot parameters, set before any script to determine the number of CPUs on the system can be run.
That's not the case with EAL parameters, where "N" and "all" only would be *nice to have*.
^ permalink raw reply [flat|nested] 31+ messages in thread
end of thread, other threads:[~2025-05-03 6:09 UTC | newest]
Thread overview: 31+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-03-13 11:38 [RFC PATCH 0/3] allow easier use of high lcore-ids Bruce Richardson
2025-03-13 11:38 ` [RFC PATCH 1/3] eal: centralize core parameter parsing Bruce Richardson
2025-03-13 11:38 ` [RFC PATCH 2/3] eal: convert core masks and lists to core sets Bruce Richardson
2025-03-13 11:38 ` [RFC PATCH 3/3] eal: allow automatic mapping of high lcore ids Bruce Richardson
2025-03-24 17:30 ` [PATCH v2 0/3] allow easier use of high lcore-ids Bruce Richardson
2025-03-24 17:30 ` [PATCH v2 1/3] eal: centralize core parameter parsing Bruce Richardson
2025-04-07 6:58 ` David Marchand
2025-03-24 17:30 ` [PATCH v2 2/3] eal: convert core masks and lists to core sets Bruce Richardson
2025-04-07 6:59 ` David Marchand
2025-03-24 17:30 ` [PATCH v2 3/3] eal: allow automatic mapping of high lcore ids Bruce Richardson
2025-04-01 14:06 ` [PATCH v2 0/3] allow easier use of high lcore-ids Bruce Richardson
2025-04-07 7:04 ` David Marchand
2025-04-07 9:48 ` Bruce Richardson
2025-04-07 10:15 ` Morten Brørup
2025-04-07 10:40 ` Bruce Richardson
2025-04-07 11:32 ` Morten Brørup
2025-04-07 11:56 ` Bruce Richardson
2025-04-07 12:25 ` Morten Brørup
2025-04-07 12:41 ` Bruce Richardson
2025-04-07 13:18 ` Morten Brørup
2025-04-07 13:24 ` Bruce Richardson
2025-04-07 15:14 ` Stephen Hemminger
2025-04-07 15:38 ` Bruce Richardson
2025-05-02 15:11 ` [PATCH v3 0/4] rework and expand EAL lcore options Bruce Richardson
2025-05-02 15:11 ` [PATCH v3 1/4] eal: deprecate old coremask-based EAL parameters Bruce Richardson
2025-05-02 15:51 ` Stephen Hemminger
2025-05-02 16:00 ` Bruce Richardson
2025-05-03 6:09 ` Morten Brørup
2025-05-02 15:11 ` [PATCH v3 2/4] eal: merge corelist and core mapping options Bruce Richardson
2025-05-02 15:11 ` [PATCH v3 3/4] doc: provide examples of using lcores EAL parameter Bruce Richardson
2025-05-02 15:11 ` [PATCH v3 4/4] eal: add option for auto-mapping cpu ids to low lcore ids Bruce Richardson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).