[dpdk-dev] [RFCv2] service core concept

[dpdk-dev] [RFCv2] service core concept

[Van Haaren, Harry]

> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Van Haaren, Harry
> Sent: Thursday, May 25, 2017 2:28 PM
> Subject: Re: [dpdk-dev] [RFC] service core concept header implementation
> 
> Thanks everybody for the input on the RFC - appreciated! From an application point-of-view, I
> see merit in Konstantin's suggestions for the API, over the RFC I sent previously. I will re-
> work the API taking inspiration from both APIs and send an RFCv2, you'll be on CC :)


Hi All,

This email is the v2 RFC for Service Cores, in response to the v1 sent previously[1].
The service-cores API has been updated, and various concepts have changed.

The API has been redesigned, with Konstantin's API suggestions as a base, and the
other comments taken into account. In my opinion this v2 API is more application-centric,
and enables the common application tasks much better. Such tasks are for example start/stop
of a service, and add/remove of a service core.

In particular this version of the API enables applications that are not aware of services to
benefit from the services concept, as EAL args can be used to setup services and service cores.
With this design, switching to/from SW/HW PMD is transparent to the application. An example
use-case is the Eventdev HW PMD to Eventdev SW PMD that requires a service core. 

I have noted the implementation comments that were raised on the v1. For v2, I think our time
is better spent looking at the API design, and I will handle implementation feedback in the
follow-up patchset to v2 RFC.

Below a summary of what we are trying to achieve, and the current API design.
Have a good weekend! Cheers, -Harry


Summary of goals (summarized from a previous email)
1. Allowing libraries and drivers to register the fact that they require background processing
2. Providing support for easily multiplexing these independent functions from different libs onto a different core
3. Providing support for the application to configure the running of these background services on specific cores
4. Once configured, hiding these services and the cores they run on from the rest of the application,
   so that the rest of the app logic does not need to change depending on whether service cores are in use or not


=== RFC v2 API ===

There are 3 parts to the API; they separate the concerns of each "user" of the API:
	- Service Registration
	- Service Config
	- ServiceCore Config

Service Registration: 
A service requires a core. It only knows about its NUMA node, and that it requires CPU time.
Registration of a service requires only that information.

Service Config:
An application may configure what service runs where using the Service Config APIs.
EAL is capable of performing this during rte_eal_init() if requested by passing a
--service-cores argument. The application (mostly) calls these functions at initialization
time, with start() and stop() being available to dynamically switch on/off a service if required.

ServiceCore Config
An application can start/stop or add/remove service-lcores using the ServiceCore Config, allowing
dynamically scaling the number of used lcores by services. Lcores used as service-cores are removed
from the application coremask, and are not available to remote_launch() as they are already in use.


Service Registration:
int32_t rte_service_register(const struct rte_service_spec *spec);
int32_t rte_service_unregister(struct rte_service_spec *service);

Service Configuration:
uint32_t rte_service_get_count(void);
struct rte_service_spec *rte_service_get_by_id(uint32_t id);
const char *rte_service_get_name(const struct rte_service_spec *service);
int32_t rte_service_set_coremask(struct rte_service_spec *service, const rte_cpuset_t *coremask);
int32_t rte_service_start(struct rte_service_spec *service); /* runtime function */
int32_t rte_service_stop(struct rte_service_spec *service);  /* runtime function */

ServiceCore Configuration:
int32_t rte_service_cores_start(void);
int32_t rte_service_cores_stop(void);
int32_t rte_service_cores_add(const rte_cpuset_t *coremask);
int32_t rte_service_cores_del(const rte_cpuset_t *coremask);


I am working on a patchset - but for now I would appreciate general design feedback,
particularly if a specific use-case is not handled.

Re: [dpdk-dev] [RFCv2] service core concept

[Ananyev, Konstantin]

Hi Harry,

> >
> > Thanks everybody for the input on the RFC - appreciated! From an application point-of-view, I
> > see merit in Konstantin's suggestions for the API, over the RFC I sent previously. I will re-
> > work the API taking inspiration from both APIs and send an RFCv2, you'll be on CC :)
> 
> 
> Hi All,
> 
> This email is the v2 RFC for Service Cores, in response to the v1 sent previously[1].
> The service-cores API has been updated, and various concepts have changed.
> 
> The API has been redesigned, with Konstantin's API suggestions as a base, and the
> other comments taken into account. In my opinion this v2 API is more application-centric,
> and enables the common application tasks much better. Such tasks are for example start/stop
> of a service, and add/remove of a service core.
> 
> In particular this version of the API enables applications that are not aware of services to
> benefit from the services concept, as EAL args can be used to setup services and service cores.
> With this design, switching to/from SW/HW PMD is transparent to the application. An example
> use-case is the Eventdev HW PMD to Eventdev SW PMD that requires a service core.
> 
> I have noted the implementation comments that were raised on the v1. For v2, I think our time
> is better spent looking at the API design, and I will handle implementation feedback in the
> follow-up patchset to v2 RFC.
> 
> Below a summary of what we are trying to achieve, and the current API design.
> Have a good weekend! Cheers, -Harry

Looks good to me in general.
The only comment I have - do we really need to put it into rte_eal_init()
and a new EAL command-line parameter for it? 
Might be better to leave it to the particular app to decide.
Konstantin

> 
> 
> Summary of goals (summarized from a previous email)
> 1. Allowing libraries and drivers to register the fact that they require background processing
> 2. Providing support for easily multiplexing these independent functions from different libs onto a different core
> 3. Providing support for the application to configure the running of these background services on specific cores
> 4. Once configured, hiding these services and the cores they run on from the rest of the application,
>    so that the rest of the app logic does not need to change depending on whether service cores are in use or not
> 
> 
> === RFC v2 API ===
> 
> There are 3 parts to the API; they separate the concerns of each "user" of the API:
> 	- Service Registration
> 	- Service Config
> 	- ServiceCore Config
> 
> Service Registration:
> A service requires a core. It only knows about its NUMA node, and that it requires CPU time.
> Registration of a service requires only that information.
> 
> Service Config:
> An application may configure what service runs where using the Service Config APIs.
> EAL is capable of performing this during rte_eal_init() if requested by passing a
> --service-cores argument. The application (mostly) calls these functions at initialization
> time, with start() and stop() being available to dynamically switch on/off a service if required.
> 
> ServiceCore Config
> An application can start/stop or add/remove service-lcores using the ServiceCore Config, allowing
> dynamically scaling the number of used lcores by services. Lcores used as service-cores are removed
> from the application coremask, and are not available to remote_launch() as they are already in use.
> 
> 
> Service Registration:
> int32_t rte_service_register(const struct rte_service_spec *spec);
> int32_t rte_service_unregister(struct rte_service_spec *service);
> 
> Service Configuration:
> uint32_t rte_service_get_count(void);
> struct rte_service_spec *rte_service_get_by_id(uint32_t id);
> const char *rte_service_get_name(const struct rte_service_spec *service);
> int32_t rte_service_set_coremask(struct rte_service_spec *service, const rte_cpuset_t *coremask);
> int32_t rte_service_start(struct rte_service_spec *service); /* runtime function */
> int32_t rte_service_stop(struct rte_service_spec *service);  /* runtime function */
> 
> ServiceCore Configuration:
> int32_t rte_service_cores_start(void);
> int32_t rte_service_cores_stop(void);
> int32_t rte_service_cores_add(const rte_cpuset_t *coremask);
> int32_t rte_service_cores_del(const rte_cpuset_t *coremask);
> 
> 
> I am working on a patchset - but for now I would appreciate general design feedback,
> particularly if a specific use-case is not handled.

Re: [dpdk-dev] [RFCv2] service core concept

[Van Haaren, Harry]

> -----Original Message-----
> From: Ananyev, Konstantin
> Sent: Saturday, June 3, 2017 11:23 AM
> To: Van Haaren, Harry <harry.van.haaren@intel.com>; dev@dpdk.org
> Cc: Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob <jerin.jacob@caviumnetworks.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; Wiles, Keith <keith.wiles@intel.com>
> Subject: RE: [dpdk-dev] [RFCv2] service core concept

<snip>

> > In particular this version of the API enables applications that are not aware of services to
> > benefit from the services concept, as EAL args can be used to setup services and service
> cores.
> > With this design, switching to/from SW/HW PMD is transparent to the application. An example
> > use-case is the Eventdev HW PMD to Eventdev SW PMD that requires a service core.
> >
> > I have noted the implementation comments that were raised on the v1. For v2, I think our
> time
> > is better spent looking at the API design, and I will handle implementation feedback in the
> > follow-up patchset to v2 RFC.
> >
> > Below a summary of what we are trying to achieve, and the current API design.
> > Have a good weekend! Cheers, -Harry
> 
>
> Looks good to me in general.
> The only comment I have - do we really need to put it into rte_eal_init()
> and a new EAL command-line parameter for it?
> Might be better to leave it to the particular app to decide.


There are a number of options here, each with its own merit:

A) Services/cores config in EAL
Benefit is that service functionality can be transparent to the application. Negative is that the complexity is in EAL.

B) Application configures services/cores
Benefit is no added EAL complexity. Negative is that application code has to configure cores (duplicated per application).


To answer this question, I think we need to estimate how many applications would benefit from EAL integration and balance that against the "complexity cost" of doing so. I do like the simplicity of option (B), however if there is significant value in total transparency to the application I think (A) is the better choice.


Input on A) or B) welcomed! -Harry

Re: [dpdk-dev] [RFCv2] service core concept

[Ananyev, Konstantin]

> -----Original Message-----
> From: Van Haaren, Harry
> Sent: Tuesday, June 6, 2017 11:26 AM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
> Cc: Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob <jerin.jacob@caviumnetworks.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Wiles, Keith <keith.wiles@intel.com>
> Subject: RE: [dpdk-dev] [RFCv2] service core concept
> 
> > -----Original Message-----
> > From: Ananyev, Konstantin
> > Sent: Saturday, June 3, 2017 11:23 AM
> > To: Van Haaren, Harry <harry.van.haaren@intel.com>; dev@dpdk.org
> > Cc: Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob <jerin.jacob@caviumnetworks.com>;
> > Richardson, Bruce <bruce.richardson@intel.com>; Wiles, Keith <keith.wiles@intel.com>
> > Subject: RE: [dpdk-dev] [RFCv2] service core concept
> 
> <snip>
> 
> > > In particular this version of the API enables applications that are not aware of services to
> > > benefit from the services concept, as EAL args can be used to setup services and service
> > cores.
> > > With this design, switching to/from SW/HW PMD is transparent to the application. An example
> > > use-case is the Eventdev HW PMD to Eventdev SW PMD that requires a service core.
> > >
> > > I have noted the implementation comments that were raised on the v1. For v2, I think our
> > time
> > > is better spent looking at the API design, and I will handle implementation feedback in the
> > > follow-up patchset to v2 RFC.
> > >
> > > Below a summary of what we are trying to achieve, and the current API design.
> > > Have a good weekend! Cheers, -Harry
> >
> >
> > Looks good to me in general.
> > The only comment I have - do we really need to put it into rte_eal_init()
> > and a new EAL command-line parameter for it?
> > Might be better to leave it to the particular app to decide.
> 
> 
> There are a number of options here, each with its own merit:
> 
> A) Services/cores config in EAL
> Benefit is that service functionality can be transparent to the application. Negative is that the complexity is in EAL.

It is not only complexity of EAL, as I understand, it would mean that
EAL will have a dependency on this new library.
Konstantin

> 
> B) Application configures services/cores
> Benefit is no added EAL complexity. Negative is that application code has to configure cores (duplicated per application).
> 
> 
> To answer this question, I think we need to estimate how many applications would benefit from EAL integration and balance that against
> the "complexity cost" of doing so. I do like the simplicity of option (B), however if there is significant value in total transparency to the
> application I think (A) is the better choice.
> 
> 
> Input on A) or B) welcomed! -Harry

Re: [dpdk-dev] [RFCv2] service core concept

[Bruce Richardson]

On Tue, Jun 06, 2017 at 11:25:57AM +0100, Van Haaren, Harry wrote:
> > -----Original Message-----
> > From: Ananyev, Konstantin
> > Sent: Saturday, June 3, 2017 11:23 AM
> > To: Van Haaren, Harry <harry.van.haaren@intel.com>; dev@dpdk.org
> > Cc: Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob <jerin.jacob@caviumnetworks.com>;
> > Richardson, Bruce <bruce.richardson@intel.com>; Wiles, Keith <keith.wiles@intel.com>
> > Subject: RE: [dpdk-dev] [RFCv2] service core concept
> 
> <snip>
> 
> > > In particular this version of the API enables applications that are not aware of services to
> > > benefit from the services concept, as EAL args can be used to setup services and service
> > cores.
> > > With this design, switching to/from SW/HW PMD is transparent to the application. An example
> > > use-case is the Eventdev HW PMD to Eventdev SW PMD that requires a service core.
> > >
> > > I have noted the implementation comments that were raised on the v1. For v2, I think our
> > time
> > > is better spent looking at the API design, and I will handle implementation feedback in the
> > > follow-up patchset to v2 RFC.
> > >
> > > Below a summary of what we are trying to achieve, and the current API design.
> > > Have a good weekend! Cheers, -Harry
> > 
> >
> > Looks good to me in general.
> > The only comment I have - do we really need to put it into rte_eal_init()
> > and a new EAL command-line parameter for it?
> > Might be better to leave it to the particular app to decide.
> 
> 
> There are a number of options here, each with its own merit:
> 
> A) Services/cores config in EAL
> Benefit is that service functionality can be transparent to the application. Negative is that the complexity is in EAL.
> 
> B) Application configures services/cores
> Benefit is no added EAL complexity. Negative is that application code has to configure cores (duplicated per application).
> 
> 
> To answer this question, I think we need to estimate how many applications would benefit from EAL integration and balance that against the "complexity cost" of doing so. I do like the simplicity of option (B), however if there is significant value in total transparency to the application I think (A) is the better choice.
> 
> 
> Input on A) or B) welcomed! -Harry

I'm definitely in favour of having it in EAL. The whole reason for doing
this work is to make it easy for applications to dedicate cores to
background tasks - including applications written before this
functionality was added. By merging this into EAL, we can have
transparency in the app, as we can have the service cores completely in
the background, and the app can call rte_eal_mp_remote_launch() exactly
as before, without unexpected failures. If we move this externally, the
app needs to be reworked to take account of that fact, and call new,
service-core aware, launch functions instead.

Regards,
/Bruce

Re: [dpdk-dev] [RFCv2] service core concept

[Ananyev, Konstantin]

> -----Original Message-----
> From: Richardson, Bruce
> Sent: Tuesday, June 6, 2017 3:54 PM
> To: Van Haaren, Harry <harry.van.haaren@intel.com>
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org; Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob
> <jerin.jacob@caviumnetworks.com>; Wiles, Keith <keith.wiles@intel.com>
> Subject: Re: [dpdk-dev] [RFCv2] service core concept
> 
> On Tue, Jun 06, 2017 at 11:25:57AM +0100, Van Haaren, Harry wrote:
> > > -----Original Message-----
> > > From: Ananyev, Konstantin
> > > Sent: Saturday, June 3, 2017 11:23 AM
> > > To: Van Haaren, Harry <harry.van.haaren@intel.com>; dev@dpdk.org
> > > Cc: Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob <jerin.jacob@caviumnetworks.com>;
> > > Richardson, Bruce <bruce.richardson@intel.com>; Wiles, Keith <keith.wiles@intel.com>
> > > Subject: RE: [dpdk-dev] [RFCv2] service core concept
> >
> > <snip>
> >
> > > > In particular this version of the API enables applications that are not aware of services to
> > > > benefit from the services concept, as EAL args can be used to setup services and service
> > > cores.
> > > > With this design, switching to/from SW/HW PMD is transparent to the application. An example
> > > > use-case is the Eventdev HW PMD to Eventdev SW PMD that requires a service core.
> > > >
> > > > I have noted the implementation comments that were raised on the v1. For v2, I think our
> > > time
> > > > is better spent looking at the API design, and I will handle implementation feedback in the
> > > > follow-up patchset to v2 RFC.
> > > >
> > > > Below a summary of what we are trying to achieve, and the current API design.
> > > > Have a good weekend! Cheers, -Harry
> > >
> > >
> > > Looks good to me in general.
> > > The only comment I have - do we really need to put it into rte_eal_init()
> > > and a new EAL command-line parameter for it?
> > > Might be better to leave it to the particular app to decide.
> >
> >
> > There are a number of options here, each with its own merit:
> >
> > A) Services/cores config in EAL
> > Benefit is that service functionality can be transparent to the application. Negative is that the complexity is in EAL.
> >
> > B) Application configures services/cores
> > Benefit is no added EAL complexity. Negative is that application code has to configure cores (duplicated per application).
> >
> >
> > To answer this question, I think we need to estimate how many applications would benefit from EAL integration and balance that against
> the "complexity cost" of doing so. I do like the simplicity of option (B), however if there is significant value in total transparency to the
> application I think (A) is the better choice.
> >
> >
> > Input on A) or B) welcomed! -Harry
> 
> I'm definitely in favour of having it in EAL. The whole reason for doing
> this work is to make it easy for applications to dedicate cores to
> background tasks - including applications written before this
> functionality was added. By merging this into EAL, we can have
> transparency in the app, as we can have the service cores completely in
> the background, and the app can call rte_eal_mp_remote_launch() exactly
> as before, without unexpected failures. If we move this externally, the
> app needs to be reworked to take account of that fact, and call new,
> service-core aware, launch functions instead.

Not sure I understood you here:
If the app don' plan to use any cores for services, it for sure will be able to call
rte_eal_mp_remote_launch() as before (no services running case).
>From other side, if the app would like to use services - it would need to specify
which service it wants to run, and for each service provide a coremask, even if
EAL already allocates service cores for it.
Or are you talking about the when EAL allocates service cores, and then
PMDs themselves (or EAL again) register their services on that cores?
That's probably possible, but how PMD would know which service core(s) it allowed to use?
Another EAL cmd-line parameter(s) or extension of existing '-w/--vdev' or something else?
Things might get over-complicated here - in theory there could be multiple PMDs,
each of them can have more than one service, running on multiple sets of cores, etc.
Konstantin

 

Re: [dpdk-dev] [RFCv2] service core concept

[Van Haaren, Harry]

> From: Ananyev, Konstantin
> Sent: Tuesday, June 6, 2017 4:29 PM
> Subject: RE: [dpdk-dev] [RFCv2] service core concept
> 
> 
> > From: Richardson, Bruce
> > Sent: Tuesday, June 6, 2017 3:54 PM
> >
> > On Tue, Jun 06, 2017 at 11:25:57AM +0100, Van Haaren, Harry wrote:
> > > > From: Ananyev, Konstantin
> > > > Sent: Saturday, June 3, 2017 11:23 AM
> > >
> > > <snip>

<snip other discussion>

> > >
> > > There are a number of options here, each with its own merit:
> > >
> > > A) Services/cores config in EAL
> > > Benefit is that service functionality can be transparent to the application. Negative is
> that the complexity is in EAL.
> > >
> > > B) Application configures services/cores
> > > Benefit is no added EAL complexity. Negative is that application code has to configure
> cores (duplicated per application).
> > >
> > >
> > > To answer this question, I think we need to estimate how many applications would benefit
> from EAL integration and balance that against
> > the "complexity cost" of doing so. I do like the simplicity of option (B), however if there
> is significant value in total transparency to the
> > application I think (A) is the better choice.
> > >
> > >
> > > Input on A) or B) welcomed! -Harry
> >
> > I'm definitely in favour of having it in EAL. The whole reason for doing
> > this work is to make it easy for applications to dedicate cores to
> > background tasks - including applications written before this
> > functionality was added. By merging this into EAL, we can have
> > transparency in the app, as we can have the service cores completely in
> > the background, and the app can call rte_eal_mp_remote_launch() exactly
> > as before, without unexpected failures. If we move this externally, the
> > app needs to be reworked to take account of that fact, and call new,
> > service-core aware, launch functions instead.
> 
> Not sure I understood you here:
> If the app don' plan to use any cores for services, it for sure will be able to call
> rte_eal_mp_remote_launch() as before (no services running case).

Correct - EAL behavior remains unchanged if --service-cores=0xf is not passed


> From other side, if the app would like to use services - it would need to specify
> which service it wants to run, and for each service provide a coremask, even if
> EAL already allocates service cores for it.

See next paragraph


> Or are you talking about the when EAL allocates service cores, and then
> PMDs themselves (or EAL again) register their services on that cores?

EAL could provide sane default behavior. For example, round-robin services over available service-cores. Multithread-capable services can be registered on all service cores. Its not a perfect solution for all service-to-core mapping problems, but I'd guess about 80% of cases would be covered: using a single service with a single service core dedicated to it :)


> That's probably possible, but how PMD would know which service core(s) it allowed to use?

The PMD shouldn't be deciding - EAL for basic sanity config, or Application for advanced usage.


> Things might get over-complicated here - in theory there could be multiple PMDs,
> each of them can have more than one service, running on multiple sets of cores, etc.

True - the NxM service:core mapping possibility can be huge - the API allows the application the flexibility if that flexibility is really required. If the flexibility is not required, the round-robin 1:1 service:core EAL scheme should cover it?

-Harry

Re: [dpdk-dev] [RFCv2] service core concept

[Ananyev, Konstantin]

> -----Original Message-----
> From: Van Haaren, Harry
> Sent: Tuesday, June 6, 2017 4:41 PM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>
> Cc: dev@dpdk.org; Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob <jerin.jacob@caviumnetworks.com>; Wiles, Keith
> <keith.wiles@intel.com>
> Subject: RE: [dpdk-dev] [RFCv2] service core concept
> 
> > From: Ananyev, Konstantin
> > Sent: Tuesday, June 6, 2017 4:29 PM
> > Subject: RE: [dpdk-dev] [RFCv2] service core concept
> >
> >
> > > From: Richardson, Bruce
> > > Sent: Tuesday, June 6, 2017 3:54 PM
> > >
> > > On Tue, Jun 06, 2017 at 11:25:57AM +0100, Van Haaren, Harry wrote:
> > > > > From: Ananyev, Konstantin
> > > > > Sent: Saturday, June 3, 2017 11:23 AM
> > > >
> > > > <snip>
> 
> <snip other discussion>
> 
> > > >
> > > > There are a number of options here, each with its own merit:
> > > >
> > > > A) Services/cores config in EAL
> > > > Benefit is that service functionality can be transparent to the application. Negative is
> > that the complexity is in EAL.
> > > >
> > > > B) Application configures services/cores
> > > > Benefit is no added EAL complexity. Negative is that application code has to configure
> > cores (duplicated per application).
> > > >
> > > >
> > > > To answer this question, I think we need to estimate how many applications would benefit
> > from EAL integration and balance that against
> > > the "complexity cost" of doing so. I do like the simplicity of option (B), however if there
> > is significant value in total transparency to the
> > > application I think (A) is the better choice.
> > > >
> > > >
> > > > Input on A) or B) welcomed! -Harry
> > >
> > > I'm definitely in favour of having it in EAL. The whole reason for doing
> > > this work is to make it easy for applications to dedicate cores to
> > > background tasks - including applications written before this
> > > functionality was added. By merging this into EAL, we can have
> > > transparency in the app, as we can have the service cores completely in
> > > the background, and the app can call rte_eal_mp_remote_launch() exactly
> > > as before, without unexpected failures. If we move this externally, the
> > > app needs to be reworked to take account of that fact, and call new,
> > > service-core aware, launch functions instead.
> >
> > Not sure I understood you here:
> > If the app don' plan to use any cores for services, it for sure will be able to call
> > rte_eal_mp_remote_launch() as before (no services running case).
> 
> Correct - EAL behavior remains unchanged if --service-cores=0xf is not passed
> 
> 
> > From other side, if the app would like to use services - it would need to specify
> > which service it wants to run, and for each service provide a coremask, even if
> > EAL already allocates service cores for it.
> 
> See next paragraph
> 
> 
> > Or are you talking about the when EAL allocates service cores, and then
> > PMDs themselves (or EAL again) register their services on that cores?
> 
> EAL could provide sane default behavior. For example, round-robin services over available service-cores. Multithread-capable services can
> be registered on all service cores. Its not a perfect solution for all service-to-core mapping problems, but I'd guess about 80% of cases
> would be covered: using a single service with a single service core dedicated to it :)
> 
> 
> > That's probably possible, but how PMD would know which service core(s) it allowed to use?
> 
> The PMD shouldn't be deciding - EAL for basic sanity config, or Application for advanced usage.
> 
> 
> > Things might get over-complicated here - in theory there could be multiple PMDs,
> > each of them can have more than one service, running on multiple sets of cores, etc.
> 
> True - the NxM service:core mapping possibility can be huge - the API allows the application the flexibility if that flexibility is really required.
> If the flexibility is not required, the round-robin 1:1 service:core EAL scheme should cover it?

Ok, so if I understand you right: by default EAL will allow each PMD to register it's services on all available service cores?
Konstantin 

Re: [dpdk-dev] [RFCv2] service core concept

[Van Haaren, Harry]

> -----Original Message-----
> From: Ananyev, Konstantin
> Sent: Wednesday, June 7, 2017 10:51 AM
> To: Van Haaren, Harry <harry.van.haaren@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>
> Cc: dev@dpdk.org; Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob
> <jerin.jacob@caviumnetworks.com>; Wiles, Keith <keith.wiles@intel.com>
> Subject: RE: [dpdk-dev] [RFCv2] service core concept
> 
> 
> 
> > -----Original Message-----
> > From: Van Haaren, Harry
> > Sent: Tuesday, June 6, 2017 4:41 PM
> > To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>
> > Cc: dev@dpdk.org; Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob
> <jerin.jacob@caviumnetworks.com>; Wiles, Keith
> > <keith.wiles@intel.com>
> > Subject: RE: [dpdk-dev] [RFCv2] service core concept
> >
> > > From: Ananyev, Konstantin
> > > Sent: Tuesday, June 6, 2017 4:29 PM
> > > Subject: RE: [dpdk-dev] [RFCv2] service core concept
> > >
> > >
> > > > From: Richardson, Bruce
> > > > Sent: Tuesday, June 6, 2017 3:54 PM
> > > >
> > > > On Tue, Jun 06, 2017 at 11:25:57AM +0100, Van Haaren, Harry wrote:
> > > > > > From: Ananyev, Konstantin
> > > > > > Sent: Saturday, June 3, 2017 11:23 AM
> > > > >
> > > > > <snip>
> >
> > <snip other discussion>
> >
> > > > >
> > > > > There are a number of options here, each with its own merit:
> > > > >
> > > > > A) Services/cores config in EAL
> > > > > Benefit is that service functionality can be transparent to the application. Negative
> is
> > > that the complexity is in EAL.
> > > > >
> > > > > B) Application configures services/cores
> > > > > Benefit is no added EAL complexity. Negative is that application code has to configure
> > > cores (duplicated per application).
> > > > >
> > > > >
> > > > > To answer this question, I think we need to estimate how many applications would
> benefit
> > > from EAL integration and balance that against
> > > > the "complexity cost" of doing so. I do like the simplicity of option (B), however if
> there
> > > is significant value in total transparency to the
> > > > application I think (A) is the better choice.
> > > > >
> > > > >
> > > > > Input on A) or B) welcomed! -Harry
> > > >
> > > > I'm definitely in favour of having it in EAL. The whole reason for doing
> > > > this work is to make it easy for applications to dedicate cores to
> > > > background tasks - including applications written before this
> > > > functionality was added. By merging this into EAL, we can have
> > > > transparency in the app, as we can have the service cores completely in
> > > > the background, and the app can call rte_eal_mp_remote_launch() exactly
> > > > as before, without unexpected failures. If we move this externally, the
> > > > app needs to be reworked to take account of that fact, and call new,
> > > > service-core aware, launch functions instead.
> > >
> > > Not sure I understood you here:
> > > If the app don' plan to use any cores for services, it for sure will be able to call
> > > rte_eal_mp_remote_launch() as before (no services running case).
> >
> > Correct - EAL behavior remains unchanged if --service-cores=0xf is not passed
> >
> >
> > > From other side, if the app would like to use services - it would need to specify
> > > which service it wants to run, and for each service provide a coremask, even if
> > > EAL already allocates service cores for it.
> >
> > See next paragraph
> >
> >
> > > Or are you talking about the when EAL allocates service cores, and then
> > > PMDs themselves (or EAL again) register their services on that cores?
> >
> > EAL could provide sane default behavior. For example, round-robin services over available
> service-cores. Multithread-capable services can
> > be registered on all service cores. Its not a perfect solution for all service-to-core
> mapping problems, but I'd guess about 80% of cases
> > would be covered: using a single service with a single service core dedicated to it :)
> >
> >
> > > That's probably possible, but how PMD would know which service core(s) it allowed to use?
> >
> > The PMD shouldn't be deciding - EAL for basic sanity config, or Application for advanced
> usage.
> >
> >
> > > Things might get over-complicated here - in theory there could be multiple PMDs,
> > > each of them can have more than one service, running on multiple sets of cores, etc.
> >
> > True - the NxM service:core mapping possibility can be huge - the API allows the application
> the flexibility if that flexibility is really required.
> > If the flexibility is not required, the round-robin 1:1 service:core EAL scheme should cover
> it?
> 
> Ok, so if I understand you right: by default EAL will allow each PMD to register it's services
> on all available service cores?

Close, but I don't see the PMD being involved in core mapping. I think of it like this:

1) A PMD registers its service (unaware of number of service cores available)
2) EAL provides default core to service mappings
2.1) Application configures using API for advanced uses (optional)


Worked examples of EAL default core mapping with two services registered:
- Eventdev SW PMD
- Ethdev to eventdev RX

Example A) With cores >= services, the services get one core assigned each:
./dpdk-app --service-cores=0x3
eventdev_sw0            : lcore 0
ethdev_to_eventdev_rx0  : lcore 1

Example B) With more services than cores, the services share the available cores: 
./dpdk-app --service-cores=0x1
eventdev_sw0            : lcore 0
ethdev_to_eventdev_rx0  : lcore 0


The EAL core-mapping logic round-robins services onto cores. If there are more cores than services, they are not used (and a warning print given). If there are more services than cores, they are wrapped back to the first core, and services share the core (example B).

Keep in mind this is just for simple use-cases. For complex services to cores mappings the application has the service API to configure it precisely as it wishes.

Re: [dpdk-dev] [RFCv2] service core concept

[Ananyev, Konstantin]

> -----Original Message-----
> From: Van Haaren, Harry
> Sent: Wednesday, June 7, 2017 11:30 AM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>
> Cc: dev@dpdk.org; Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob <jerin.jacob@caviumnetworks.com>; Wiles, Keith
> <keith.wiles@intel.com>
> Subject: RE: [dpdk-dev] [RFCv2] service core concept
> 
> 
> 
> > -----Original Message-----
> > From: Ananyev, Konstantin
> > Sent: Wednesday, June 7, 2017 10:51 AM
> > To: Van Haaren, Harry <harry.van.haaren@intel.com>; Richardson, Bruce
> > <bruce.richardson@intel.com>
> > Cc: dev@dpdk.org; Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob
> > <jerin.jacob@caviumnetworks.com>; Wiles, Keith <keith.wiles@intel.com>
> > Subject: RE: [dpdk-dev] [RFCv2] service core concept
> >
> >
> >
> > > -----Original Message-----
> > > From: Van Haaren, Harry
> > > Sent: Tuesday, June 6, 2017 4:41 PM
> > > To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> > <bruce.richardson@intel.com>
> > > Cc: dev@dpdk.org; Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob
> > <jerin.jacob@caviumnetworks.com>; Wiles, Keith
> > > <keith.wiles@intel.com>
> > > Subject: RE: [dpdk-dev] [RFCv2] service core concept
> > >
> > > > From: Ananyev, Konstantin
> > > > Sent: Tuesday, June 6, 2017 4:29 PM
> > > > Subject: RE: [dpdk-dev] [RFCv2] service core concept
> > > >
> > > >
> > > > > From: Richardson, Bruce
> > > > > Sent: Tuesday, June 6, 2017 3:54 PM
> > > > >
> > > > > On Tue, Jun 06, 2017 at 11:25:57AM +0100, Van Haaren, Harry wrote:
> > > > > > > From: Ananyev, Konstantin
> > > > > > > Sent: Saturday, June 3, 2017 11:23 AM
> > > > > >
> > > > > > <snip>
> > >
> > > <snip other discussion>
> > >
> > > > > >
> > > > > > There are a number of options here, each with its own merit:
> > > > > >
> > > > > > A) Services/cores config in EAL
> > > > > > Benefit is that service functionality can be transparent to the application. Negative
> > is
> > > > that the complexity is in EAL.
> > > > > >
> > > > > > B) Application configures services/cores
> > > > > > Benefit is no added EAL complexity. Negative is that application code has to configure
> > > > cores (duplicated per application).
> > > > > >
> > > > > >
> > > > > > To answer this question, I think we need to estimate how many applications would
> > benefit
> > > > from EAL integration and balance that against
> > > > > the "complexity cost" of doing so. I do like the simplicity of option (B), however if
> > there
> > > > is significant value in total transparency to the
> > > > > application I think (A) is the better choice.
> > > > > >
> > > > > >
> > > > > > Input on A) or B) welcomed! -Harry
> > > > >
> > > > > I'm definitely in favour of having it in EAL. The whole reason for doing
> > > > > this work is to make it easy for applications to dedicate cores to
> > > > > background tasks - including applications written before this
> > > > > functionality was added. By merging this into EAL, we can have
> > > > > transparency in the app, as we can have the service cores completely in
> > > > > the background, and the app can call rte_eal_mp_remote_launch() exactly
> > > > > as before, without unexpected failures. If we move this externally, the
> > > > > app needs to be reworked to take account of that fact, and call new,
> > > > > service-core aware, launch functions instead.
> > > >
> > > > Not sure I understood you here:
> > > > If the app don' plan to use any cores for services, it for sure will be able to call
> > > > rte_eal_mp_remote_launch() as before (no services running case).
> > >
> > > Correct - EAL behavior remains unchanged if --service-cores=0xf is not passed
> > >
> > >
> > > > From other side, if the app would like to use services - it would need to specify
> > > > which service it wants to run, and for each service provide a coremask, even if
> > > > EAL already allocates service cores for it.
> > >
> > > See next paragraph
> > >
> > >
> > > > Or are you talking about the when EAL allocates service cores, and then
> > > > PMDs themselves (or EAL again) register their services on that cores?
> > >
> > > EAL could provide sane default behavior. For example, round-robin services over available
> > service-cores. Multithread-capable services can
> > > be registered on all service cores. Its not a perfect solution for all service-to-core
> > mapping problems, but I'd guess about 80% of cases
> > > would be covered: using a single service with a single service core dedicated to it :)
> > >
> > >
> > > > That's probably possible, but how PMD would know which service core(s) it allowed to use?
> > >
> > > The PMD shouldn't be deciding - EAL for basic sanity config, or Application for advanced
> > usage.
> > >
> > >
> > > > Things might get over-complicated here - in theory there could be multiple PMDs,
> > > > each of them can have more than one service, running on multiple sets of cores, etc.
> > >
> > > True - the NxM service:core mapping possibility can be huge - the API allows the application
> > the flexibility if that flexibility is really required.
> > > If the flexibility is not required, the round-robin 1:1 service:core EAL scheme should cover
> > it?
> >
> > Ok, so if I understand you right: by default EAL will allow each PMD to register it's services
> > on all available service cores?
> 
> Close, but I don't see the PMD being involved in core mapping. I think of it like this:
> 
> 1) A PMD registers its service (unaware of number of service cores available)
> 2) EAL provides default core to service mappings
> 2.1) Application configures using API for advanced uses (optional)

Ok, thanks for explanation.
I am still not quite happy with the fact that EAL will have a dependency on service lib,
but I see your point and indeed it might be usefull for many cases.
So wouldn't object here.
Konstantin

> 
> 
> Worked examples of EAL default core mapping with two services registered:
> - Eventdev SW PMD
> - Ethdev to eventdev RX
> 
> Example A) With cores >= services, the services get one core assigned each:
> ./dpdk-app --service-cores=0x3
> eventdev_sw0            : lcore 0
> ethdev_to_eventdev_rx0  : lcore 1
> 
> Example B) With more services than cores, the services share the available cores:
> ./dpdk-app --service-cores=0x1
> eventdev_sw0            : lcore 0
> ethdev_to_eventdev_rx0  : lcore 0
> 
> 
> The EAL core-mapping logic round-robins services onto cores. If there are more cores than services, they are not used (and a warning print
> given). If there are more services than cores, they are wrapped back to the first core, and services share the core (example B).
> 
> Keep in mind this is just for simple use-cases. For complex services to cores mappings the application has the service API to configure it
> precisely as it wishes.

Re: [dpdk-dev] [RFCv2] service core concept

[Jerin Jacob]

-----Original Message-----
> Date: Fri, 2 Jun 2017 16:09:13 +0000
> From: "Van Haaren, Harry" <harry.van.haaren@intel.com>
> To: "dev@dpdk.org" <dev@dpdk.org>
> CC: Thomas Monjalon <thomas@monjalon.net>, Jerin Jacob
>  <jerin.jacob@caviumnetworks.com>, "Richardson, Bruce"
>  <bruce.richardson@intel.com>, "Ananyev, Konstantin"
>  <konstantin.ananyev@intel.com>, "Wiles, Keith" <keith.wiles@intel.com>
> Subject: [dpdk-dev] [RFCv2] service core concept
> 
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Van Haaren, Harry
> > Sent: Thursday, May 25, 2017 2:28 PM
> > Subject: Re: [dpdk-dev] [RFC] service core concept header implementation
> > 
> > Thanks everybody for the input on the RFC - appreciated! From an application point-of-view, I
> > see merit in Konstantin's suggestions for the API, over the RFC I sent previously. I will re-
> > work the API taking inspiration from both APIs and send an RFCv2, you'll be on CC :)
> 
> 
> Hi All,
> 
> This email is the v2 RFC for Service Cores, in response to the v1 sent previously[1].
> The service-cores API has been updated, and various concepts have changed.
> 
> The API has been redesigned, with Konstantin's API suggestions as a base, and the
> other comments taken into account. In my opinion this v2 API is more application-centric,
> and enables the common application tasks much better. Such tasks are for example start/stop
> of a service, and add/remove of a service core.
> 
> In particular this version of the API enables applications that are not aware of services to
> benefit from the services concept, as EAL args can be used to setup services and service cores.
> With this design, switching to/from SW/HW PMD is transparent to the application. An example
> use-case is the Eventdev HW PMD to Eventdev SW PMD that requires a service core. 
> 
> I have noted the implementation comments that were raised on the v1. For v2, I think our time
> is better spent looking at the API design, and I will handle implementation feedback in the
> follow-up patchset to v2 RFC.
> 
> Below a summary of what we are trying to achieve, and the current API design.
> Have a good weekend! Cheers, -Harry
> 
> 
> Summary of goals (summarized from a previous email)
> 1. Allowing libraries and drivers to register the fact that they require background processing
> 2. Providing support for easily multiplexing these independent functions from different libs onto a different core
> 3. Providing support for the application to configure the running of these background services on specific cores
> 4. Once configured, hiding these services and the cores they run on from the rest of the application,
>    so that the rest of the app logic does not need to change depending on whether service cores are in use or not
> 
> 
> === RFC v2 API ===
> 
> There are 3 parts to the API; they separate the concerns of each "user" of the API:
> 	- Service Registration
> 	- Service Config
> 	- ServiceCore Config
> 
> Service Registration: 
> A service requires a core. It only knows about its NUMA node, and that it requires CPU time.
> Registration of a service requires only that information.
> 
> Service Config:
> An application may configure what service runs where using the Service Config APIs.
> EAL is capable of performing this during rte_eal_init() if requested by passing a
> --service-cores argument. The application (mostly) calls these functions at initialization
> time, with start() and stop() being available to dynamically switch on/off a service if required.
> 
> ServiceCore Config
> An application can start/stop or add/remove service-lcores using the ServiceCore Config, allowing
> dynamically scaling the number of used lcores by services. Lcores used as service-cores are removed
> from the application coremask, and are not available to remote_launch() as they are already in use.
> 
> 
> Service Registration:
> int32_t rte_service_register(const struct rte_service_spec *spec);
> int32_t rte_service_unregister(struct rte_service_spec *service);
> 
> Service Configuration:
> uint32_t rte_service_get_count(void);
> struct rte_service_spec *rte_service_get_by_id(uint32_t id);
> const char *rte_service_get_name(const struct rte_service_spec *service);
> int32_t rte_service_set_coremask(struct rte_service_spec *service, const rte_cpuset_t *coremask);
> int32_t rte_service_start(struct rte_service_spec *service); /* runtime function */
> int32_t rte_service_stop(struct rte_service_spec *service);  /* runtime function */

Looks good to me in general.

How about an API to query the service function running status?
bool rte_service_is_running(struct rte_service_spec *service); or something similar.


> 
> ServiceCore Configuration:
> int32_t rte_service_cores_start(void);
> int32_t rte_service_cores_stop(void);
> int32_t rte_service_cores_add(const rte_cpuset_t *coremask);
> int32_t rte_service_cores_del(const rte_cpuset_t *coremask);
> 
> 
> I am working on a patchset - but for now I would appreciate general design feedback,
> particularly if a specific use-case is not handled.

Re: [dpdk-dev] [RFCv2] service core concept

[Van Haaren, Harry]

> -----Original Message-----
> From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> Sent: Monday, June 5, 2017 8:23 AM
> To: Van Haaren, Harry <harry.van.haaren@intel.com>
> Cc: dev@dpdk.org; Thomas Monjalon <thomas@monjalon.net>; Richardson, Bruce
> <bruce.richardson@intel.com>; Ananyev, Konstantin <konstantin.ananyev@intel.com>; Wiles, Keith
> <keith.wiles@intel.com>
> Subject: Re: [dpdk-dev] [RFCv2] service core concept

<snip API proposal>


> Looks good to me in general.
> 
> How about an API to query the service function running status?
> bool rte_service_is_running(struct rte_service_spec *service); or something similar.


Good idea - noted and will implement for patchset.