Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization

["Burakov, Anatoly" <anatoly.burakov@intel.com>]

On 17-Feb-21 1:26 PM, Bruce Richardson wrote:
> On Wed, Feb 17, 2021 at 12:14:36PM +0000, Van Haaren, Harry wrote:
>>> -----Original Message-----
>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>>> Sent: Wednesday, February 17, 2021 12:09 PM
>>> To: Van Haaren, Harry <harry.van.haaren@intel.com>; Richardson, Bruce
>>> <bruce.richardson@intel.com>
>>> Cc: dev@dpdk.org
>>> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
>>> affinitization
>>>
>>> On 16-Feb-21 5:44 PM, Van Haaren, Harry wrote:
>>>>> -----Original Message-----
>>>>> From: Bruce Richardson <bruce.richardson@intel.com>
>>>>> Sent: Tuesday, February 16, 2021 5:31 PM
>>>>> To: Van Haaren, Harry <harry.van.haaren@intel.com>
>>>>> Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; dev@dpdk.org
>>>>> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
>>>>> affinitization
>>>>>
>>>>> On Tue, Feb 16, 2021 at 05:22:25PM +0000, Van Haaren, Harry wrote:
>>>>>>> -----Original Message-----
>>>>>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>>>>>>> Sent: Tuesday, February 16, 2021 10:53 AM
>>>>>>> To: Richardson, Bruce <bruce.richardson@intel.com>; Van Haaren, Harry
>>>>>>> <harry.van.haaren@intel.com>
>>>>>>> Cc: dev@dpdk.org
>>>>>>> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
>>>>>>> affinitization
>>>>>>>
>>>>>>> On 16-Feb-21 10:46 AM, Bruce Richardson wrote:
>>>>>>>> On Tue, Feb 16, 2021 at 10:36:13AM +0000, Burakov, Anatoly wrote:
>>>>>>>>> On 16-Feb-21 9:43 AM, Bruce Richardson wrote:
>>>>>>>>>> Allow the user to specify that they don't want any core pinning from
>>>>> DPDK
>>>>>>>>>> by passing in the coremask of 0.
>>>>>>>>>> ---
>>>>>>>>>
>>>>>>>>> I haven't checked what happens yet, but down the line we also set
>>> affinity
>>>>>>>>> for service cores as well as interrupt thread. what would be the
>>> semantics
>>>>>>>>> of those in this particular case? do we want the same ability for service
>>>>>>>>> cores (i.e. pick a non-affinitized core)? And where does interrupt thread
>>>>>>>>> affinitize in this case (presumably, nowhere too)?
>>>>>>>>>
>>>>>>>> I have not checked the service core setup, because a) I forgot about them
>>>>>>>> and b) I'm not sure how their affinity rules work with respect to the main
>>>>>>>> lcore mask. On the other hand I did check out that the lcore mask for all
>>>>>>>> non-pinned threads, or control threads, is the full set of bits as
>>>>>>>> expected.
>>>>>>>>
>>>>>>>> /Bruce
>>>>>>>>
>>>>>>>
>>>>>>> +Harry,
>>>>>>>
>>>>>>> I believe service core mask must not overlap with lcore masks, so
>>>>>>> presumably using 0 as lcore mask would make it so that any service core
>>>>>>> mask will be valid (which is presumably what we want?).
>>>>>>
>>>>>> Services cores -S list or -s <mask> *must* overlap with the RTE lcores, EAL
>>>>>> then"steals" the service cores from the application lcores, code that
>>>>> implements here:
>>>>>> http://git.dpdk.org/dpdk-
>>>>> stable/tree/lib/librte_eal/common/eal_common_options.c?h=20.11#n657
>>>>>>
>>>>>>> Should service cores also have a "just pick a core" parameter?
>>>>>>
>>>>>> I'm not sure, depends on what the bigger goal is here.
>>>>>> Assuming we're enabling this for ROLE_RTE threads, then
>>>>>> it would seem to me that ROLE_SERVICE and control threads
>>>>>> would require similar treatment?
>>>>>>
>>>>> Control threads are affinitised to all cores not in the coremask, which
>>>>> means in this case that they can run anywhere on the system the OS chooses.
>>>>
>>>> Ah ok, fair enough yes.
>>>>
>>>>> In case of service cores, it would seem that using service cores with an
>>>>> empty coremask is just not compatible. I would assume that this
>>>>> incompatibility already exists when one has a coremask with only one core
>>>>> already in it.
>>>>
>>>> Yes, correct, it would leave zero lcores for ROLE_RTE, meaning no lcores for the
>>> application.
>>>> A possible solution would be to special case a zero service core mask and apply
>>> the same
>>>> treatment as ROLE_RTE coremask?
>>>>
>>>> Others likely have better ideas - I don't have time to follow DPDK
>>> threading/pinning topic
>>>> closely at the moment.
>>>>
>>>
>>> I don't think it's a good idea to disallow service cores functionality
>>> in this case, but i don't have a way to solve this, other than
>>> implementing similar 0x0 coremask for service cores and assume it always
>>> means "one core affinitized to wherever the OS feels like it". After
>>> all, with lcore mask 0x0 we assume user wants one single core only, so
>>> following that, one single service core is a valid extrapolation IMO.
>>
>> OK with me - seems reasonable.
>>
>>> Perhaps specifying the number of l/s cores when using 0x0 would be
>>> interesting, but IMO unless there's ask for it, i'd rather not
>>> overcomplicate things and go with similar semantics for service cores,
>>> and just allow a 0x0 coremask that means only one unaffinitized service
>>> core will be created.
>>>
>>> Thoughts?
>>
>> Agree with keeping-it-simple if possible, and agree that unaffinitized with
>> a single service-core with a 0x0 mask makes sense.
>>
>> Most important to me is to maintain backward compatibility with existing
>> usage of -S and -s, but this shouldn't break anything? (Famous last words..)
>>
> 
> Not sure I entirely follow all of this. Is the suggestion just to extend -s
> processing to allow "0" as coremask too? That would be independent then of
> any core masks passed in for -c/-l options, right? As well as working well
> with this patch, it would also solve the issue of using single core with a
> coremask of e.g. 0x1 too, I think.
> 
> Is my understanding correct?
> 
> /Bruce
> 

Yes, that's exactly what i meant :) Sorry for being long-winded and unclear.

-- 
Thanks,
Anatoly