[dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[Bruce Richardson]

Allow the user to specify that they don't want any core pinning from DPDK
by passing in the coremask of 0.
---
 lib/librte_eal/common/eal_common_options.c | 18 +++++++++++++++---
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/freebsd/eal.c               |  5 +++--
 lib/librte_eal/linux/eal.c                 |  5 +++--
 4 files changed, 22 insertions(+), 7 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index 622c7bc429..94029bf7f1 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -1522,9 +1522,21 @@ eal_parse_common_option(int opt, const char *optarg,
 		if (eal_service_cores_parsed())
 			RTE_LOG(WARNING, EAL,
 				"Service cores parsed before dataplane cores. Please ensure -c is before -s or -S\n");
-		if (eal_parse_coremask(optarg, lcore_indexes) < 0) {
-			RTE_LOG(ERR, EAL, "invalid coremask syntax\n");
-			return -1;
+
+		if (strcmp(optarg, "0") == 0 || strcmp(optarg, "0x0") == 0) {
+			/* if -c 0 passed, don't affinitize anything, so set
+			 * up a single core 0 as active, but mark it not to have
+			 * pthread_setaffinity called on it.
+			 */
+			memset(lcore_indexes, -1, sizeof(lcore_indexes));
+			conf->no_main_affinity = 1;
+			lcore_indexes[0] = 0;
+			RTE_CPU_FILL(&lcore_config[0].cpuset);
+		} else {
+			if (eal_parse_coremask(optarg, lcore_indexes) < 0) {
+				RTE_LOG(ERR, EAL, "invalid coremask syntax\n");
+				return -1;
+			}
 		}
 		if (update_lcore_config(lcore_indexes) < 0) {
 			char *available = available_cores();
diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
index 51dbe86e2b..db46c49b84 100644
--- a/lib/librte_eal/common/eal_internal_cfg.h
+++ b/lib/librte_eal/common/eal_internal_cfg.h
@@ -50,6 +50,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned no_main_affinity; /**< disable main lcore CPU pinning */
 	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
 										* instead of native TSC */
 	volatile unsigned no_shconf;      /**< true if there is no shared config */
diff --git a/lib/librte_eal/freebsd/eal.c b/lib/librte_eal/freebsd/eal.c
index 51478358c7..a30a6e54d4 100644
--- a/lib/librte_eal/freebsd/eal.c
+++ b/lib/librte_eal/freebsd/eal.c
@@ -850,8 +850,9 @@ rte_eal_init(int argc, char **argv)
 
 	eal_check_mem_on_local_socket();
 
-	if (pthread_setaffinity_np(pthread_self(), sizeof(rte_cpuset_t),
-			&lcore_config[config->main_lcore].cpuset) != 0) {
+	if (!internal_conf->no_main_affinity &&
+			pthread_setaffinity_np(pthread_self(), sizeof(rte_cpuset_t),
+				&lcore_config[config->main_lcore].cpuset) != 0) {
 		rte_eal_init_alert("Cannot set affinity");
 		rte_errno = EINVAL;
 		return -1;
diff --git a/lib/librte_eal/linux/eal.c b/lib/librte_eal/linux/eal.c
index 32b48c3de9..e3390766ca 100644
--- a/lib/librte_eal/linux/eal.c
+++ b/lib/librte_eal/linux/eal.c
@@ -1214,8 +1214,9 @@ rte_eal_init(int argc, char **argv)
 
 	eal_check_mem_on_local_socket();
 
-	if (pthread_setaffinity_np(pthread_self(), sizeof(rte_cpuset_t),
-			&lcore_config[config->main_lcore].cpuset) != 0) {
+	if (!internal_conf->no_main_affinity &&
+			pthread_setaffinity_np(pthread_self(), sizeof(rte_cpuset_t),
+				&lcore_config[config->main_lcore].cpuset) != 0) {
 		rte_eal_init_alert("Cannot set affinity");
 		rte_errno = EINVAL;
 		return -1;
-- 
2.27.0

Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[Bruce Richardson]

On Tue, Feb 16, 2021 at 09:43:00AM +0000, Bruce Richardson wrote:
> Allow the user to specify that they don't want any core pinning from DPDK
> by passing in the coremask of 0.
> ---

Apologies, missed my signoff, will add in V2, but will wait for feedback on
this V1 first.

/Bruce

Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[Burakov, Anatoly]

On 16-Feb-21 9:43 AM, Bruce Richardson wrote:
> Allow the user to specify that they don't want any core pinning from DPDK
> by passing in the coremask of 0.
> ---

I haven't checked what happens yet, but down the line we also set 
affinity for service cores as well as interrupt thread. what would be 
the semantics of those in this particular case? do we want the same 
ability for service cores (i.e. pick a non-affinitized core)? And where 
does interrupt thread affinitize in this case (presumably, nowhere too)?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[Bruce Richardson]

On Tue, Feb 16, 2021 at 10:36:13AM +0000, Burakov, Anatoly wrote:
> On 16-Feb-21 9:43 AM, Bruce Richardson wrote:
> > Allow the user to specify that they don't want any core pinning from DPDK
> > by passing in the coremask of 0.
> > ---
> 
> I haven't checked what happens yet, but down the line we also set affinity
> for service cores as well as interrupt thread. what would be the semantics
> of those in this particular case? do we want the same ability for service
> cores (i.e. pick a non-affinitized core)? And where does interrupt thread
> affinitize in this case (presumably, nowhere too)?
> 
I have not checked the service core setup, because a) I forgot about them
and b) I'm not sure how their affinity rules work with respect to the main
lcore mask. On the other hand I did check out that the lcore mask for all
non-pinned threads, or control threads, is the full set of bits as
expected.

/Bruce

Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[Burakov, Anatoly]

On 16-Feb-21 10:46 AM, Bruce Richardson wrote:
> On Tue, Feb 16, 2021 at 10:36:13AM +0000, Burakov, Anatoly wrote:
>> On 16-Feb-21 9:43 AM, Bruce Richardson wrote:
>>> Allow the user to specify that they don't want any core pinning from DPDK
>>> by passing in the coremask of 0.
>>> ---
>>
>> I haven't checked what happens yet, but down the line we also set affinity
>> for service cores as well as interrupt thread. what would be the semantics
>> of those in this particular case? do we want the same ability for service
>> cores (i.e. pick a non-affinitized core)? And where does interrupt thread
>> affinitize in this case (presumably, nowhere too)?
>>
> I have not checked the service core setup, because a) I forgot about them
> and b) I'm not sure how their affinity rules work with respect to the main
> lcore mask. On the other hand I did check out that the lcore mask for all
> non-pinned threads, or control threads, is the full set of bits as
> expected.
> 
> /Bruce
> 

+Harry,

I believe service core mask must not overlap with lcore masks, so 
presumably using 0 as lcore mask would make it so that any service core 
mask will be valid (which is presumably what we want?). Should service 
cores also have a "just pick a core" parameter?

I'm assuming this use-case is explicitly avoiding the CPU/memory/NIA 
NUMA affinity question, so i'm not bringing it up :)

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[Van Haaren, Harry]

> -----Original Message-----
> From: Burakov, Anatoly <anatoly.burakov@intel.com>
> Sent: Tuesday, February 16, 2021 10:53 AM
> To: Richardson, Bruce <bruce.richardson@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
> affinitization
> 
> On 16-Feb-21 10:46 AM, Bruce Richardson wrote:
> > On Tue, Feb 16, 2021 at 10:36:13AM +0000, Burakov, Anatoly wrote:
> >> On 16-Feb-21 9:43 AM, Bruce Richardson wrote:
> >>> Allow the user to specify that they don't want any core pinning from DPDK
> >>> by passing in the coremask of 0.
> >>> ---
> >>
> >> I haven't checked what happens yet, but down the line we also set affinity
> >> for service cores as well as interrupt thread. what would be the semantics
> >> of those in this particular case? do we want the same ability for service
> >> cores (i.e. pick a non-affinitized core)? And where does interrupt thread
> >> affinitize in this case (presumably, nowhere too)?
> >>
> > I have not checked the service core setup, because a) I forgot about them
> > and b) I'm not sure how their affinity rules work with respect to the main
> > lcore mask. On the other hand I did check out that the lcore mask for all
> > non-pinned threads, or control threads, is the full set of bits as
> > expected.
> >
> > /Bruce
> >
> 
> +Harry,
> 
> I believe service core mask must not overlap with lcore masks, so
> presumably using 0 as lcore mask would make it so that any service core
> mask will be valid (which is presumably what we want?). 

Services cores -S list or -s <mask> *must* overlap with the RTE lcores, EAL
then"steals" the service cores from the application lcores, code that implements here:
http://git.dpdk.org/dpdk-stable/tree/lib/librte_eal/common/eal_common_options.c?h=20.11#n657

> Should service cores also have a "just pick a core" parameter?

I'm not sure, depends on what the bigger goal is here.
Assuming we're enabling this for ROLE_RTE threads, then
it would seem to me that ROLE_SERVICE and control threads
would require similar treatment?


> I'm assuming this use-case is explicitly avoiding the CPU/memory/NIA
> NUMA affinity question, so i'm not bringing it up :)
>
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[Bruce Richardson]

On Tue, Feb 16, 2021 at 05:22:25PM +0000, Van Haaren, Harry wrote:
> > -----Original Message-----
> > From: Burakov, Anatoly <anatoly.burakov@intel.com>
> > Sent: Tuesday, February 16, 2021 10:53 AM
> > To: Richardson, Bruce <bruce.richardson@intel.com>; Van Haaren, Harry
> > <harry.van.haaren@intel.com>
> > Cc: dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
> > affinitization
> >
> > On 16-Feb-21 10:46 AM, Bruce Richardson wrote:
> > > On Tue, Feb 16, 2021 at 10:36:13AM +0000, Burakov, Anatoly wrote:
> > >> On 16-Feb-21 9:43 AM, Bruce Richardson wrote:
> > >>> Allow the user to specify that they don't want any core pinning from DPDK
> > >>> by passing in the coremask of 0.
> > >>> ---
> > >>
> > >> I haven't checked what happens yet, but down the line we also set affinity
> > >> for service cores as well as interrupt thread. what would be the semantics
> > >> of those in this particular case? do we want the same ability for service
> > >> cores (i.e. pick a non-affinitized core)? And where does interrupt thread
> > >> affinitize in this case (presumably, nowhere too)?
> > >>
> > > I have not checked the service core setup, because a) I forgot about them
> > > and b) I'm not sure how their affinity rules work with respect to the main
> > > lcore mask. On the other hand I did check out that the lcore mask for all
> > > non-pinned threads, or control threads, is the full set of bits as
> > > expected.
> > >
> > > /Bruce
> > >
> >
> > +Harry,
> >
> > I believe service core mask must not overlap with lcore masks, so
> > presumably using 0 as lcore mask would make it so that any service core
> > mask will be valid (which is presumably what we want?).
> 
> Services cores -S list or -s <mask> *must* overlap with the RTE lcores, EAL
> then"steals" the service cores from the application lcores, code that implements here:
> http://git.dpdk.org/dpdk-stable/tree/lib/librte_eal/common/eal_common_options.c?h=20.11#n657
> 
> > Should service cores also have a "just pick a core" parameter?
> 
> I'm not sure, depends on what the bigger goal is here.
> Assuming we're enabling this for ROLE_RTE threads, then
> it would seem to me that ROLE_SERVICE and control threads
> would require similar treatment?
> 
Control threads are affinitised to all cores not in the coremask, which
means in this case that they can run anywhere on the system the OS chooses.
In case of service cores, it would seem that using service cores with an
empty coremask is just not compatible. I would assume that this
incompatibility already exists when one has a coremask with only one core
already in it.

/Bruce

Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[Van Haaren, Harry]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Tuesday, February 16, 2021 5:31 PM
> To: Van Haaren, Harry <harry.van.haaren@intel.com>
> Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
> affinitization
> 
> On Tue, Feb 16, 2021 at 05:22:25PM +0000, Van Haaren, Harry wrote:
> > > -----Original Message-----
> > > From: Burakov, Anatoly <anatoly.burakov@intel.com>
> > > Sent: Tuesday, February 16, 2021 10:53 AM
> > > To: Richardson, Bruce <bruce.richardson@intel.com>; Van Haaren, Harry
> > > <harry.van.haaren@intel.com>
> > > Cc: dev@dpdk.org
> > > Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
> > > affinitization
> > >
> > > On 16-Feb-21 10:46 AM, Bruce Richardson wrote:
> > > > On Tue, Feb 16, 2021 at 10:36:13AM +0000, Burakov, Anatoly wrote:
> > > >> On 16-Feb-21 9:43 AM, Bruce Richardson wrote:
> > > >>> Allow the user to specify that they don't want any core pinning from
> DPDK
> > > >>> by passing in the coremask of 0.
> > > >>> ---
> > > >>
> > > >> I haven't checked what happens yet, but down the line we also set affinity
> > > >> for service cores as well as interrupt thread. what would be the semantics
> > > >> of those in this particular case? do we want the same ability for service
> > > >> cores (i.e. pick a non-affinitized core)? And where does interrupt thread
> > > >> affinitize in this case (presumably, nowhere too)?
> > > >>
> > > > I have not checked the service core setup, because a) I forgot about them
> > > > and b) I'm not sure how their affinity rules work with respect to the main
> > > > lcore mask. On the other hand I did check out that the lcore mask for all
> > > > non-pinned threads, or control threads, is the full set of bits as
> > > > expected.
> > > >
> > > > /Bruce
> > > >
> > >
> > > +Harry,
> > >
> > > I believe service core mask must not overlap with lcore masks, so
> > > presumably using 0 as lcore mask would make it so that any service core
> > > mask will be valid (which is presumably what we want?).
> >
> > Services cores -S list or -s <mask> *must* overlap with the RTE lcores, EAL
> > then"steals" the service cores from the application lcores, code that
> implements here:
> > http://git.dpdk.org/dpdk-
> stable/tree/lib/librte_eal/common/eal_common_options.c?h=20.11#n657
> >
> > > Should service cores also have a "just pick a core" parameter?
> >
> > I'm not sure, depends on what the bigger goal is here.
> > Assuming we're enabling this for ROLE_RTE threads, then
> > it would seem to me that ROLE_SERVICE and control threads
> > would require similar treatment?
> >
> Control threads are affinitised to all cores not in the coremask, which
> means in this case that they can run anywhere on the system the OS chooses.

Ah ok, fair enough yes.

> In case of service cores, it would seem that using service cores with an
> empty coremask is just not compatible. I would assume that this
> incompatibility already exists when one has a coremask with only one core
> already in it.

Yes, correct, it would leave zero lcores for ROLE_RTE, meaning no lcores for the application.
A possible solution would be to special case a zero service core mask and apply the same
treatment as ROLE_RTE coremask?

Others likely have better ideas - I don't have time to follow DPDK threading/pinning topic
closely at the moment.

Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[Burakov, Anatoly]

On 16-Feb-21 5:44 PM, Van Haaren, Harry wrote:
>> -----Original Message-----
>> From: Bruce Richardson <bruce.richardson@intel.com>
>> Sent: Tuesday, February 16, 2021 5:31 PM
>> To: Van Haaren, Harry <harry.van.haaren@intel.com>
>> Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; dev@dpdk.org
>> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
>> affinitization
>>
>> On Tue, Feb 16, 2021 at 05:22:25PM +0000, Van Haaren, Harry wrote:
>>>> -----Original Message-----
>>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>>>> Sent: Tuesday, February 16, 2021 10:53 AM
>>>> To: Richardson, Bruce <bruce.richardson@intel.com>; Van Haaren, Harry
>>>> <harry.van.haaren@intel.com>
>>>> Cc: dev@dpdk.org
>>>> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
>>>> affinitization
>>>>
>>>> On 16-Feb-21 10:46 AM, Bruce Richardson wrote:
>>>>> On Tue, Feb 16, 2021 at 10:36:13AM +0000, Burakov, Anatoly wrote:
>>>>>> On 16-Feb-21 9:43 AM, Bruce Richardson wrote:
>>>>>>> Allow the user to specify that they don't want any core pinning from
>> DPDK
>>>>>>> by passing in the coremask of 0.
>>>>>>> ---
>>>>>>
>>>>>> I haven't checked what happens yet, but down the line we also set affinity
>>>>>> for service cores as well as interrupt thread. what would be the semantics
>>>>>> of those in this particular case? do we want the same ability for service
>>>>>> cores (i.e. pick a non-affinitized core)? And where does interrupt thread
>>>>>> affinitize in this case (presumably, nowhere too)?
>>>>>>
>>>>> I have not checked the service core setup, because a) I forgot about them
>>>>> and b) I'm not sure how their affinity rules work with respect to the main
>>>>> lcore mask. On the other hand I did check out that the lcore mask for all
>>>>> non-pinned threads, or control threads, is the full set of bits as
>>>>> expected.
>>>>>
>>>>> /Bruce
>>>>>
>>>>
>>>> +Harry,
>>>>
>>>> I believe service core mask must not overlap with lcore masks, so
>>>> presumably using 0 as lcore mask would make it so that any service core
>>>> mask will be valid (which is presumably what we want?).
>>>
>>> Services cores -S list or -s <mask> *must* overlap with the RTE lcores, EAL
>>> then"steals" the service cores from the application lcores, code that
>> implements here:
>>> http://git.dpdk.org/dpdk-
>> stable/tree/lib/librte_eal/common/eal_common_options.c?h=20.11#n657
>>>
>>>> Should service cores also have a "just pick a core" parameter?
>>>
>>> I'm not sure, depends on what the bigger goal is here.
>>> Assuming we're enabling this for ROLE_RTE threads, then
>>> it would seem to me that ROLE_SERVICE and control threads
>>> would require similar treatment?
>>>
>> Control threads are affinitised to all cores not in the coremask, which
>> means in this case that they can run anywhere on the system the OS chooses.
> 
> Ah ok, fair enough yes.
> 
>> In case of service cores, it would seem that using service cores with an
>> empty coremask is just not compatible. I would assume that this
>> incompatibility already exists when one has a coremask with only one core
>> already in it.
> 
> Yes, correct, it would leave zero lcores for ROLE_RTE, meaning no lcores for the application.
> A possible solution would be to special case a zero service core mask and apply the same
> treatment as ROLE_RTE coremask?
> 
> Others likely have better ideas - I don't have time to follow DPDK threading/pinning topic
> closely at the moment.
> 

I don't think it's a good idea to disallow service cores functionality 
in this case, but i don't have a way to solve this, other than 
implementing similar 0x0 coremask for service cores and assume it always 
means "one core affinitized to wherever the OS feels like it". After 
all, with lcore mask 0x0 we assume user wants one single core only, so 
following that, one single service core is a valid extrapolation IMO.

Perhaps specifying the number of l/s cores when using 0x0 would be 
interesting, but IMO unless there's ask for it, i'd rather not 
overcomplicate things and go with similar semantics for service cores, 
and just allow a 0x0 coremask that means only one unaffinitized service 
core will be created.

Thoughts?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[Van Haaren, Harry]

> -----Original Message-----
> From: Burakov, Anatoly <anatoly.burakov@intel.com>
> Sent: Wednesday, February 17, 2021 12:09 PM
> To: Van Haaren, Harry <harry.van.haaren@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
> affinitization
> 
> On 16-Feb-21 5:44 PM, Van Haaren, Harry wrote:
> >> -----Original Message-----
> >> From: Bruce Richardson <bruce.richardson@intel.com>
> >> Sent: Tuesday, February 16, 2021 5:31 PM
> >> To: Van Haaren, Harry <harry.van.haaren@intel.com>
> >> Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; dev@dpdk.org
> >> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
> >> affinitization
> >>
> >> On Tue, Feb 16, 2021 at 05:22:25PM +0000, Van Haaren, Harry wrote:
> >>>> -----Original Message-----
> >>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
> >>>> Sent: Tuesday, February 16, 2021 10:53 AM
> >>>> To: Richardson, Bruce <bruce.richardson@intel.com>; Van Haaren, Harry
> >>>> <harry.van.haaren@intel.com>
> >>>> Cc: dev@dpdk.org
> >>>> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
> >>>> affinitization
> >>>>
> >>>> On 16-Feb-21 10:46 AM, Bruce Richardson wrote:
> >>>>> On Tue, Feb 16, 2021 at 10:36:13AM +0000, Burakov, Anatoly wrote:
> >>>>>> On 16-Feb-21 9:43 AM, Bruce Richardson wrote:
> >>>>>>> Allow the user to specify that they don't want any core pinning from
> >> DPDK
> >>>>>>> by passing in the coremask of 0.
> >>>>>>> ---
> >>>>>>
> >>>>>> I haven't checked what happens yet, but down the line we also set
> affinity
> >>>>>> for service cores as well as interrupt thread. what would be the
> semantics
> >>>>>> of those in this particular case? do we want the same ability for service
> >>>>>> cores (i.e. pick a non-affinitized core)? And where does interrupt thread
> >>>>>> affinitize in this case (presumably, nowhere too)?
> >>>>>>
> >>>>> I have not checked the service core setup, because a) I forgot about them
> >>>>> and b) I'm not sure how their affinity rules work with respect to the main
> >>>>> lcore mask. On the other hand I did check out that the lcore mask for all
> >>>>> non-pinned threads, or control threads, is the full set of bits as
> >>>>> expected.
> >>>>>
> >>>>> /Bruce
> >>>>>
> >>>>
> >>>> +Harry,
> >>>>
> >>>> I believe service core mask must not overlap with lcore masks, so
> >>>> presumably using 0 as lcore mask would make it so that any service core
> >>>> mask will be valid (which is presumably what we want?).
> >>>
> >>> Services cores -S list or -s <mask> *must* overlap with the RTE lcores, EAL
> >>> then"steals" the service cores from the application lcores, code that
> >> implements here:
> >>> http://git.dpdk.org/dpdk-
> >> stable/tree/lib/librte_eal/common/eal_common_options.c?h=20.11#n657
> >>>
> >>>> Should service cores also have a "just pick a core" parameter?
> >>>
> >>> I'm not sure, depends on what the bigger goal is here.
> >>> Assuming we're enabling this for ROLE_RTE threads, then
> >>> it would seem to me that ROLE_SERVICE and control threads
> >>> would require similar treatment?
> >>>
> >> Control threads are affinitised to all cores not in the coremask, which
> >> means in this case that they can run anywhere on the system the OS chooses.
> >
> > Ah ok, fair enough yes.
> >
> >> In case of service cores, it would seem that using service cores with an
> >> empty coremask is just not compatible. I would assume that this
> >> incompatibility already exists when one has a coremask with only one core
> >> already in it.
> >
> > Yes, correct, it would leave zero lcores for ROLE_RTE, meaning no lcores for the
> application.
> > A possible solution would be to special case a zero service core mask and apply
> the same
> > treatment as ROLE_RTE coremask?
> >
> > Others likely have better ideas - I don't have time to follow DPDK
> threading/pinning topic
> > closely at the moment.
> >
> 
> I don't think it's a good idea to disallow service cores functionality
> in this case, but i don't have a way to solve this, other than
> implementing similar 0x0 coremask for service cores and assume it always
> means "one core affinitized to wherever the OS feels like it". After
> all, with lcore mask 0x0 we assume user wants one single core only, so
> following that, one single service core is a valid extrapolation IMO.

OK with me - seems reasonable.

> Perhaps specifying the number of l/s cores when using 0x0 would be
> interesting, but IMO unless there's ask for it, i'd rather not
> overcomplicate things and go with similar semantics for service cores,
> and just allow a 0x0 coremask that means only one unaffinitized service
> core will be created.
> 
> Thoughts?

Agree with keeping-it-simple if possible, and agree that unaffinitized with
a single service-core with a 0x0 mask makes sense.

Most important to me is to maintain backward compatibility with existing
usage of -S and -s, but this shouldn't break anything? (Famous last words..)

> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[Bruce Richardson]

On Wed, Feb 17, 2021 at 12:14:36PM +0000, Van Haaren, Harry wrote:
> > -----Original Message-----
> > From: Burakov, Anatoly <anatoly.burakov@intel.com>
> > Sent: Wednesday, February 17, 2021 12:09 PM
> > To: Van Haaren, Harry <harry.van.haaren@intel.com>; Richardson, Bruce
> > <bruce.richardson@intel.com>
> > Cc: dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
> > affinitization
> >
> > On 16-Feb-21 5:44 PM, Van Haaren, Harry wrote:
> > >> -----Original Message-----
> > >> From: Bruce Richardson <bruce.richardson@intel.com>
> > >> Sent: Tuesday, February 16, 2021 5:31 PM
> > >> To: Van Haaren, Harry <harry.van.haaren@intel.com>
> > >> Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; dev@dpdk.org
> > >> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
> > >> affinitization
> > >>
> > >> On Tue, Feb 16, 2021 at 05:22:25PM +0000, Van Haaren, Harry wrote:
> > >>>> -----Original Message-----
> > >>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
> > >>>> Sent: Tuesday, February 16, 2021 10:53 AM
> > >>>> To: Richardson, Bruce <bruce.richardson@intel.com>; Van Haaren, Harry
> > >>>> <harry.van.haaren@intel.com>
> > >>>> Cc: dev@dpdk.org
> > >>>> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
> > >>>> affinitization
> > >>>>
> > >>>> On 16-Feb-21 10:46 AM, Bruce Richardson wrote:
> > >>>>> On Tue, Feb 16, 2021 at 10:36:13AM +0000, Burakov, Anatoly wrote:
> > >>>>>> On 16-Feb-21 9:43 AM, Bruce Richardson wrote:
> > >>>>>>> Allow the user to specify that they don't want any core pinning from
> > >> DPDK
> > >>>>>>> by passing in the coremask of 0.
> > >>>>>>> ---
> > >>>>>>
> > >>>>>> I haven't checked what happens yet, but down the line we also set
> > affinity
> > >>>>>> for service cores as well as interrupt thread. what would be the
> > semantics
> > >>>>>> of those in this particular case? do we want the same ability for service
> > >>>>>> cores (i.e. pick a non-affinitized core)? And where does interrupt thread
> > >>>>>> affinitize in this case (presumably, nowhere too)?
> > >>>>>>
> > >>>>> I have not checked the service core setup, because a) I forgot about them
> > >>>>> and b) I'm not sure how their affinity rules work with respect to the main
> > >>>>> lcore mask. On the other hand I did check out that the lcore mask for all
> > >>>>> non-pinned threads, or control threads, is the full set of bits as
> > >>>>> expected.
> > >>>>>
> > >>>>> /Bruce
> > >>>>>
> > >>>>
> > >>>> +Harry,
> > >>>>
> > >>>> I believe service core mask must not overlap with lcore masks, so
> > >>>> presumably using 0 as lcore mask would make it so that any service core
> > >>>> mask will be valid (which is presumably what we want?).
> > >>>
> > >>> Services cores -S list or -s <mask> *must* overlap with the RTE lcores, EAL
> > >>> then"steals" the service cores from the application lcores, code that
> > >> implements here:
> > >>> http://git.dpdk.org/dpdk-
> > >> stable/tree/lib/librte_eal/common/eal_common_options.c?h=20.11#n657
> > >>>
> > >>>> Should service cores also have a "just pick a core" parameter?
> > >>>
> > >>> I'm not sure, depends on what the bigger goal is here.
> > >>> Assuming we're enabling this for ROLE_RTE threads, then
> > >>> it would seem to me that ROLE_SERVICE and control threads
> > >>> would require similar treatment?
> > >>>
> > >> Control threads are affinitised to all cores not in the coremask, which
> > >> means in this case that they can run anywhere on the system the OS chooses.
> > >
> > > Ah ok, fair enough yes.
> > >
> > >> In case of service cores, it would seem that using service cores with an
> > >> empty coremask is just not compatible. I would assume that this
> > >> incompatibility already exists when one has a coremask with only one core
> > >> already in it.
> > >
> > > Yes, correct, it would leave zero lcores for ROLE_RTE, meaning no lcores for the
> > application.
> > > A possible solution would be to special case a zero service core mask and apply
> > the same
> > > treatment as ROLE_RTE coremask?
> > >
> > > Others likely have better ideas - I don't have time to follow DPDK
> > threading/pinning topic
> > > closely at the moment.
> > >
> >
> > I don't think it's a good idea to disallow service cores functionality
> > in this case, but i don't have a way to solve this, other than
> > implementing similar 0x0 coremask for service cores and assume it always
> > means "one core affinitized to wherever the OS feels like it". After
> > all, with lcore mask 0x0 we assume user wants one single core only, so
> > following that, one single service core is a valid extrapolation IMO.
> 
> OK with me - seems reasonable.
> 
> > Perhaps specifying the number of l/s cores when using 0x0 would be
> > interesting, but IMO unless there's ask for it, i'd rather not
> > overcomplicate things and go with similar semantics for service cores,
> > and just allow a 0x0 coremask that means only one unaffinitized service
> > core will be created.
> >
> > Thoughts?
> 
> Agree with keeping-it-simple if possible, and agree that unaffinitized with
> a single service-core with a 0x0 mask makes sense.
> 
> Most important to me is to maintain backward compatibility with existing
> usage of -S and -s, but this shouldn't break anything? (Famous last words..)
> 

Not sure I entirely follow all of this. Is the suggestion just to extend -s
processing to allow "0" as coremask too? That would be independent then of
any core masks passed in for -c/-l options, right? As well as working well
with this patch, it would also solve the issue of using single core with a
coremask of e.g. 0x1 too, I think.

Is my understanding correct?

/Bruce

Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[Burakov, Anatoly]

On 17-Feb-21 1:26 PM, Bruce Richardson wrote:
> On Wed, Feb 17, 2021 at 12:14:36PM +0000, Van Haaren, Harry wrote:
>>> -----Original Message-----
>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>>> Sent: Wednesday, February 17, 2021 12:09 PM
>>> To: Van Haaren, Harry <harry.van.haaren@intel.com>; Richardson, Bruce
>>> <bruce.richardson@intel.com>
>>> Cc: dev@dpdk.org
>>> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
>>> affinitization
>>>
>>> On 16-Feb-21 5:44 PM, Van Haaren, Harry wrote:
>>>>> -----Original Message-----
>>>>> From: Bruce Richardson <bruce.richardson@intel.com>
>>>>> Sent: Tuesday, February 16, 2021 5:31 PM
>>>>> To: Van Haaren, Harry <harry.van.haaren@intel.com>
>>>>> Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; dev@dpdk.org
>>>>> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
>>>>> affinitization
>>>>>
>>>>> On Tue, Feb 16, 2021 at 05:22:25PM +0000, Van Haaren, Harry wrote:
>>>>>>> -----Original Message-----
>>>>>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>>>>>>> Sent: Tuesday, February 16, 2021 10:53 AM
>>>>>>> To: Richardson, Bruce <bruce.richardson@intel.com>; Van Haaren, Harry
>>>>>>> <harry.van.haaren@intel.com>
>>>>>>> Cc: dev@dpdk.org
>>>>>>> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
>>>>>>> affinitization
>>>>>>>
>>>>>>> On 16-Feb-21 10:46 AM, Bruce Richardson wrote:
>>>>>>>> On Tue, Feb 16, 2021 at 10:36:13AM +0000, Burakov, Anatoly wrote:
>>>>>>>>> On 16-Feb-21 9:43 AM, Bruce Richardson wrote:
>>>>>>>>>> Allow the user to specify that they don't want any core pinning from
>>>>> DPDK
>>>>>>>>>> by passing in the coremask of 0.
>>>>>>>>>> ---
>>>>>>>>>
>>>>>>>>> I haven't checked what happens yet, but down the line we also set
>>> affinity
>>>>>>>>> for service cores as well as interrupt thread. what would be the
>>> semantics
>>>>>>>>> of those in this particular case? do we want the same ability for service
>>>>>>>>> cores (i.e. pick a non-affinitized core)? And where does interrupt thread
>>>>>>>>> affinitize in this case (presumably, nowhere too)?
>>>>>>>>>
>>>>>>>> I have not checked the service core setup, because a) I forgot about them
>>>>>>>> and b) I'm not sure how their affinity rules work with respect to the main
>>>>>>>> lcore mask. On the other hand I did check out that the lcore mask for all
>>>>>>>> non-pinned threads, or control threads, is the full set of bits as
>>>>>>>> expected.
>>>>>>>>
>>>>>>>> /Bruce
>>>>>>>>
>>>>>>>
>>>>>>> +Harry,
>>>>>>>
>>>>>>> I believe service core mask must not overlap with lcore masks, so
>>>>>>> presumably using 0 as lcore mask would make it so that any service core
>>>>>>> mask will be valid (which is presumably what we want?).
>>>>>>
>>>>>> Services cores -S list or -s <mask> *must* overlap with the RTE lcores, EAL
>>>>>> then"steals" the service cores from the application lcores, code that
>>>>> implements here:
>>>>>> http://git.dpdk.org/dpdk-
>>>>> stable/tree/lib/librte_eal/common/eal_common_options.c?h=20.11#n657
>>>>>>
>>>>>>> Should service cores also have a "just pick a core" parameter?
>>>>>>
>>>>>> I'm not sure, depends on what the bigger goal is here.
>>>>>> Assuming we're enabling this for ROLE_RTE threads, then
>>>>>> it would seem to me that ROLE_SERVICE and control threads
>>>>>> would require similar treatment?
>>>>>>
>>>>> Control threads are affinitised to all cores not in the coremask, which
>>>>> means in this case that they can run anywhere on the system the OS chooses.
>>>>
>>>> Ah ok, fair enough yes.
>>>>
>>>>> In case of service cores, it would seem that using service cores with an
>>>>> empty coremask is just not compatible. I would assume that this
>>>>> incompatibility already exists when one has a coremask with only one core
>>>>> already in it.
>>>>
>>>> Yes, correct, it would leave zero lcores for ROLE_RTE, meaning no lcores for the
>>> application.
>>>> A possible solution would be to special case a zero service core mask and apply
>>> the same
>>>> treatment as ROLE_RTE coremask?
>>>>
>>>> Others likely have better ideas - I don't have time to follow DPDK
>>> threading/pinning topic
>>>> closely at the moment.
>>>>
>>>
>>> I don't think it's a good idea to disallow service cores functionality
>>> in this case, but i don't have a way to solve this, other than
>>> implementing similar 0x0 coremask for service cores and assume it always
>>> means "one core affinitized to wherever the OS feels like it". After
>>> all, with lcore mask 0x0 we assume user wants one single core only, so
>>> following that, one single service core is a valid extrapolation IMO.
>>
>> OK with me - seems reasonable.
>>
>>> Perhaps specifying the number of l/s cores when using 0x0 would be
>>> interesting, but IMO unless there's ask for it, i'd rather not
>>> overcomplicate things and go with similar semantics for service cores,
>>> and just allow a 0x0 coremask that means only one unaffinitized service
>>> core will be created.
>>>
>>> Thoughts?
>>
>> Agree with keeping-it-simple if possible, and agree that unaffinitized with
>> a single service-core with a 0x0 mask makes sense.
>>
>> Most important to me is to maintain backward compatibility with existing
>> usage of -S and -s, but this shouldn't break anything? (Famous last words..)
>>
> 
> Not sure I entirely follow all of this. Is the suggestion just to extend -s
> processing to allow "0" as coremask too? That would be independent then of
> any core masks passed in for -c/-l options, right? As well as working well
> with this patch, it would also solve the issue of using single core with a
> coremask of e.g. 0x1 too, I think.
> 
> Is my understanding correct?
> 
> /Bruce
> 

Yes, that's exactly what i meant :) Sorry for being long-winded and unclear.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[David Marchand]

On Tue, Feb 16, 2021 at 10:52 AM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Tue, Feb 16, 2021 at 09:43:00AM +0000, Bruce Richardson wrote:
> > Allow the user to specify that they don't want any core pinning from DPDK
> > by passing in the coremask of 0.
> > ---
>
> Apologies, missed my signoff, will add in V2, but will wait for feedback on
> this V1 first.

I did not see a v2, is this still for 21.05?

What are the usecases for this feature?
I could think of using this in combination with external threads in
applications like OVS... ?


-- 
David Marchand

Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[Stephen Hemminger]

On Wed, 14 Apr 2021 18:15:34 +0200
David Marchand <david.marchand@redhat.com> wrote:

> On Tue, Feb 16, 2021 at 10:52 AM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > On Tue, Feb 16, 2021 at 09:43:00AM +0000, Bruce Richardson wrote:  
> > > Allow the user to specify that they don't want any core pinning from DPDK
> > > by passing in the coremask of 0.
> > > ---  
> >
> > Apologies, missed my signoff, will add in V2, but will wait for feedback on
> > this V1 first.  
> 
> I did not see a v2, is this still for 21.05?
> 
> What are the usecases for this feature?
> I could think of using this in combination with external threads in
> applications like OVS... ?
> 
> 

It would cause lots of DPDK applications to discover the problems with
preemption and priority inversion with DPDK spinlocks...

Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[Bruce Richardson]

On Wed, Apr 14, 2021 at 06:15:34PM +0200, David Marchand wrote:
> On Tue, Feb 16, 2021 at 10:52 AM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > On Tue, Feb 16, 2021 at 09:43:00AM +0000, Bruce Richardson wrote:
> > > Allow the user to specify that they don't want any core pinning from
> > > DPDK by passing in the coremask of 0.  ---
> >
> > Apologies, missed my signoff, will add in V2, but will wait for
> > feedback on this V1 first.
> 
> I did not see a v2, is this still for 21.05?
> 
> What are the usecases for this feature?  I could think of using this in
> combination with external threads in applications like OVS... ?
> 
There was some good feedback and asks for extension of this to cover other
core masks like for service cores. Unfortunately, as things stand, I'm
unlikely to get enough time to work on this within 21.05 timeframe. :-(
If there is still interest, perhaps someone else can pick it up to do a V2,
otherwise I'll hopefully pick it up again in a few weeks for 21.08.

/Bruce

Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

[Bruce Richardson]

On Wed, Apr 14, 2021 at 09:29:38AM -0700, Stephen Hemminger wrote:
> On Wed, 14 Apr 2021 18:15:34 +0200
> David Marchand <david.marchand@redhat.com> wrote:
> 
> > On Tue, Feb 16, 2021 at 10:52 AM Bruce Richardson
> > <bruce.richardson@intel.com> wrote:
> > >
> > > On Tue, Feb 16, 2021 at 09:43:00AM +0000, Bruce Richardson wrote:  
> > > > Allow the user to specify that they don't want any core pinning from DPDK
> > > > by passing in the coremask of 0.
> > > > ---  
> > >
> > > Apologies, missed my signoff, will add in V2, but will wait for feedback on
> > > this V1 first.  
> > 
> > I did not see a v2, is this still for 21.05?
> > 
> > What are the usecases for this feature?
> > I could think of using this in combination with external threads in
> > applications like OVS... ?
> > 
> > 
> 
> It would cause lots of DPDK applications to discover the problems with
> preemption and priority inversion with DPDK spinlocks...

I'd actually be surprised. I would expect most apps that want this sort of
thing are already just using a single master core (perhaps with faked EAL
args) and adding their own threads on top of that. All this patch would do
is remove the requirement for that master lcore.

/Bruce