DPDK patches and discussions
 help / color / mirror / Atom feed
From: Bruce Richardson <bruce.richardson@intel.com>
To: "Van Haaren, Harry" <harry.van.haaren@intel.com>
Cc: "Burakov, Anatoly" <anatoly.burakov@intel.com>,
	"dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization
Date: Wed, 17 Feb 2021 13:26:49 +0000	[thread overview]
Message-ID: <20210217132649.GA1171@bricha3-MOBL.ger.corp.intel.com> (raw)
In-Reply-To: <BYAPR11MB3143642FE8C78E3DD66E2259D7869@BYAPR11MB3143.namprd11.prod.outlook.com>

On Wed, Feb 17, 2021 at 12:14:36PM +0000, Van Haaren, Harry wrote:
> > -----Original Message-----
> > From: Burakov, Anatoly <anatoly.burakov@intel.com>
> > Sent: Wednesday, February 17, 2021 12:09 PM
> > To: Van Haaren, Harry <harry.van.haaren@intel.com>; Richardson, Bruce
> > <bruce.richardson@intel.com>
> > Cc: dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
> > affinitization
> >
> > On 16-Feb-21 5:44 PM, Van Haaren, Harry wrote:
> > >> -----Original Message-----
> > >> From: Bruce Richardson <bruce.richardson@intel.com>
> > >> Sent: Tuesday, February 16, 2021 5:31 PM
> > >> To: Van Haaren, Harry <harry.van.haaren@intel.com>
> > >> Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; dev@dpdk.org
> > >> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
> > >> affinitization
> > >>
> > >> On Tue, Feb 16, 2021 at 05:22:25PM +0000, Van Haaren, Harry wrote:
> > >>>> -----Original Message-----
> > >>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
> > >>>> Sent: Tuesday, February 16, 2021 10:53 AM
> > >>>> To: Richardson, Bruce <bruce.richardson@intel.com>; Van Haaren, Harry
> > >>>> <harry.van.haaren@intel.com>
> > >>>> Cc: dev@dpdk.org
> > >>>> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-
> > >>>> affinitization
> > >>>>
> > >>>> On 16-Feb-21 10:46 AM, Bruce Richardson wrote:
> > >>>>> On Tue, Feb 16, 2021 at 10:36:13AM +0000, Burakov, Anatoly wrote:
> > >>>>>> On 16-Feb-21 9:43 AM, Bruce Richardson wrote:
> > >>>>>>> Allow the user to specify that they don't want any core pinning from
> > >> DPDK
> > >>>>>>> by passing in the coremask of 0.
> > >>>>>>> ---
> > >>>>>>
> > >>>>>> I haven't checked what happens yet, but down the line we also set
> > affinity
> > >>>>>> for service cores as well as interrupt thread. what would be the
> > semantics
> > >>>>>> of those in this particular case? do we want the same ability for service
> > >>>>>> cores (i.e. pick a non-affinitized core)? And where does interrupt thread
> > >>>>>> affinitize in this case (presumably, nowhere too)?
> > >>>>>>
> > >>>>> I have not checked the service core setup, because a) I forgot about them
> > >>>>> and b) I'm not sure how their affinity rules work with respect to the main
> > >>>>> lcore mask. On the other hand I did check out that the lcore mask for all
> > >>>>> non-pinned threads, or control threads, is the full set of bits as
> > >>>>> expected.
> > >>>>>
> > >>>>> /Bruce
> > >>>>>
> > >>>>
> > >>>> +Harry,
> > >>>>
> > >>>> I believe service core mask must not overlap with lcore masks, so
> > >>>> presumably using 0 as lcore mask would make it so that any service core
> > >>>> mask will be valid (which is presumably what we want?).
> > >>>
> > >>> Services cores -S list or -s <mask> *must* overlap with the RTE lcores, EAL
> > >>> then"steals" the service cores from the application lcores, code that
> > >> implements here:
> > >>> http://git.dpdk.org/dpdk-
> > >> stable/tree/lib/librte_eal/common/eal_common_options.c?h=20.11#n657
> > >>>
> > >>>> Should service cores also have a "just pick a core" parameter?
> > >>>
> > >>> I'm not sure, depends on what the bigger goal is here.
> > >>> Assuming we're enabling this for ROLE_RTE threads, then
> > >>> it would seem to me that ROLE_SERVICE and control threads
> > >>> would require similar treatment?
> > >>>
> > >> Control threads are affinitised to all cores not in the coremask, which
> > >> means in this case that they can run anywhere on the system the OS chooses.
> > >
> > > Ah ok, fair enough yes.
> > >
> > >> In case of service cores, it would seem that using service cores with an
> > >> empty coremask is just not compatible. I would assume that this
> > >> incompatibility already exists when one has a coremask with only one core
> > >> already in it.
> > >
> > > Yes, correct, it would leave zero lcores for ROLE_RTE, meaning no lcores for the
> > application.
> > > A possible solution would be to special case a zero service core mask and apply
> > the same
> > > treatment as ROLE_RTE coremask?
> > >
> > > Others likely have better ideas - I don't have time to follow DPDK
> > threading/pinning topic
> > > closely at the moment.
> > >
> >
> > I don't think it's a good idea to disallow service cores functionality
> > in this case, but i don't have a way to solve this, other than
> > implementing similar 0x0 coremask for service cores and assume it always
> > means "one core affinitized to wherever the OS feels like it". After
> > all, with lcore mask 0x0 we assume user wants one single core only, so
> > following that, one single service core is a valid extrapolation IMO.
> 
> OK with me - seems reasonable.
> 
> > Perhaps specifying the number of l/s cores when using 0x0 would be
> > interesting, but IMO unless there's ask for it, i'd rather not
> > overcomplicate things and go with similar semantics for service cores,
> > and just allow a 0x0 coremask that means only one unaffinitized service
> > core will be created.
> >
> > Thoughts?
> 
> Agree with keeping-it-simple if possible, and agree that unaffinitized with
> a single service-core with a 0x0 mask makes sense.
> 
> Most important to me is to maintain backward compatibility with existing
> usage of -S and -s, but this shouldn't break anything? (Famous last words..)
> 

Not sure I entirely follow all of this. Is the suggestion just to extend -s
processing to allow "0" as coremask too? That would be independent then of
any core masks passed in for -c/-l options, right? As well as working well
with this patch, it would also solve the issue of using single core with a
coremask of e.g. 0x1 too, I think.

Is my understanding correct?

/Bruce

  reply	other threads:[~2021-02-17 13:27 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-16  9:43 Bruce Richardson
2021-02-16  9:51 ` Bruce Richardson
2021-04-14 16:15   ` David Marchand
2021-04-14 16:29     ` Stephen Hemminger
2021-04-14 17:02       ` Bruce Richardson
2021-04-14 16:55     ` Bruce Richardson
2021-02-16 10:36 ` Burakov, Anatoly
2021-02-16 10:46   ` Bruce Richardson
2021-02-16 10:52     ` Burakov, Anatoly
2021-02-16 17:22       ` Van Haaren, Harry
2021-02-16 17:30         ` Bruce Richardson
2021-02-16 17:44           ` Van Haaren, Harry
2021-02-17 12:09             ` Burakov, Anatoly
2021-02-17 12:14               ` Van Haaren, Harry
2021-02-17 13:26                 ` Bruce Richardson [this message]
2021-02-17 13:37                   ` Burakov, Anatoly

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210217132649.GA1171@bricha3-MOBL.ger.corp.intel.com \
    --to=bruce.richardson@intel.com \
    --cc=anatoly.burakov@intel.com \
    --cc=dev@dpdk.org \
    --cc=harry.van.haaren@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).