From: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>
To: "Richardson, Bruce" <bruce.richardson@intel.com>
Cc: "Van Haaren, Harry" <harry.van.haaren@intel.com>,
"dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [RFC] service core concept header implementation
Date: Wed, 17 May 2017 13:47:15 +0000 [thread overview]
Message-ID: <2601191342CEEE43887BDE71AB9772583FAF76D7@IRSMSX109.ger.corp.intel.com> (raw)
In-Reply-To: <20170517125806.GA14804@bricha3-MOBL3.ger.corp.intel.com>
Hi Bruce,
> >
>
> Hi Konstantin,
>
> Thanks for the feedback, the specific detail of which I'll perhaps leave
> for Harry to reply to or include in a later version of this patchset.
> However, from a higher level, I think the big difference in what we
> envisage compared to what you suggest in your comments is that these are
> not services set up by the application. If the app wants to run
> something it uses remote_launch as now. This is instead for other
> components to request threads - or a share of a thread - for their own
> use, since they cannot call remote_launch directly, as the app owns the
> threads, not the individual libraries.
Ok, thanks for clarification about who supposed to be the main consumer for that.
That makes sense.
Though I am not sure why the API suggested wouldn't fit your purposes.
Though I think both PMD/core libraries and app layer might be interested in that: i.e.
app might also have some background processing tasks, that might need to be run
on several cores (or core slices).
Konstantin
> See also comments made in reply to Thomas mail.
>
> Hope this helps clarify things a little.
>
> /Bruce
>
> > >
> > > This patch adds a service header to DPDK EAL. This header is
> > > an RFC for a mechanism to allow DPDK components request a
> > > callback function to be invoked.
> > >
> > > The application can set the number of service cores, and
> > > a coremask for each particular services. The implementation
> > > of this functionality in rte_services.c (not completed) would
> > > use atomics to ensure that a service can only be polled by a
> > > single lcore at a time.
> > >
> > > Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>
> > > ---
> > > lib/librte_eal/common/include/rte_service.h | 211 ++++++++++++++++++++++++++++
> > > 1 file changed, 211 insertions(+)
> > > create mode 100644 lib/librte_eal/common/include/rte_service.h
> > >
> >
> > ...
> >
> > > +
> > > +/**
> > > + * Signature of callback back function to run a service.
> > > + */
> > > +typedef void (*rte_eal_service_func)(void *args);
> > > +
> > > +struct rte_service_config {
> > > + /* name of the service */
> > > + char name[RTE_SERVICE_NAMESIZE];
> > > + /* cores that run this service */
> > > + uint64_t coremask;
> >
> > Why uint64_t?
> > Even now by default we support more than 64 cores.
> > We have rte_cpuset_t for such things.
> >
> > Ok, the first main question:
> > Is that possible to register multiple services with intersecting coremasks or not?
> > If not, then I suppose there is no practical difference from what we have right now
> > with eal_remote_launch() .
> > If yes, then several questions arise:
> > a) How the service scheduling function will going to switch from one service to another
> > on particular core?
> > As we don't have interrupt framework for that, I suppose the only choice is to rely
> > on service voluntary fiving up cpu. Is that so?
> > b) Would it be possible to specify percentage of core cycles that service can consume?
> > I suppose the answer is 'no', at least for current iteration.
> >
> > > + /* when set, multiple lcores can run this service simultaneously without
> > > + * the need for software atomics to ensure that two cores do not
> > > + * attempt to run the service at the same time.
> > > + */
> > > + uint8_t multithread_capable;
> >
> > Ok, and what would happen if this flag is not set, and user specified more
> > then one cpu in the coremask?
> >
> > > +};
> > > +
> > > +/** @internal - this will not be visible to application, defined in a seperate
> > > + * rte_eal_service_impl.h header.
> >
> > Why not?
> > If the application registers the service, it for sure needs to know what exactly that service
> > would do (cb func and data).
> >
> > > The application does not need to be exposed
> > > + * to the actual function pointers - so hide them. */
> > > +struct rte_service {
> > > + /* callback to be called to run the service */
> > > + rte_eal_service_func cb;
> > > + /* args to the callback function */
> > > + void *cb_args;
> >
> > If user plans to run that service on multiple cores, then he might
> > need a separate instance of cb_args for each core.
> >
> > > + /* configuration of the service */
> > > + struct rte_service_config config;
> > > +};
> > > +
> > > +/**
> > > + * @internal - Only DPDK internal components request "service" cores.
> > > + *
> > > + * Registers a service in EAL.
> > > + *
> > > + * Registered services' configurations are exposed to an application using
> > > + * *rte_eal_service_get_all*. These services have names which can be understood
> > > + * by the application, and the application logic can decide how to allocate
> > > + * cores to run the various services.
> > > + *
> > > + * This function is expected to be called by a DPDK component to indicate that
> > > + * it require a CPU to run a specific function in order for it to perform its
> > > + * processing. An example of such a component is the eventdev software PMD.
> > > + *
> > > + * The config struct should be filled in as appropriate by the PMD. For example
> > > + * the name field should include some level of detail (e.g. "ethdev_p1_rx_q3"
> > > + * might mean RX from an ethdev from port 1, on queue 3).
> > > + *
> > > + * @param service
> > > + * The service structure to be registered with EAL.
> > > + *
> > > + * @return
> > > + * On success, zero
> > > + * On failure, a negative error
> > > + */
> > > +int rte_eal_service_register(const struct rte_service *service);
> > > +
> > > +/**
> > > + * Get the count of services registered in the EAL.
> > > + *
> > > + * @return the number of services registered with EAL.
> > > + */
> > > +uint32_t rte_eal_service_get_count();
> > > +
> > > +/**
> > > + * Writes all registered services to the application supplied array.
> > > + *
> > > + * This function can be used by the application to understand if and what
> > > + * services require running. Each service provides a config struct exposing
> > > + * attributes of the service, which can be used by the application to decide on
> > > + * its strategy of running services on cores.
> > > + *
> > > + * @param service_config
> > > + * An array of service config structures to be filled in
> > > + *
> > > + * @param n
> > > + * The size of the service config array
> > > + *
> > > + * @return 0 on successful providing all service configs
> > > + * -ENOSPC if array passed in is too small
> > > + */
> > > +int rte_eal_service_get_all(const struct rte_service_config *service_config,
> > > + uint32_t n);
> >
> > I'd suggest to follow snprintf() notation here: always return number of all existing services.
> > Then you wouldn't need rte_eal_service_get_count() at all.
> >
> > > +
> > > +/**
> > > + * Use *lcore_id* as a service core.
> > > + *
> > > + * EAL will internally remove *lcore_id* from the application accessible
> > > + * core list. As a result, the application will never have its code run on
> > > + * the service core, making the service cores totally transparent to an app.
> > > + *
> > > + * This function should be called before *rte_eal_service_set_coremask* so that
> > > + * when setting the core mask the value can be error checked to ensure that EAL
> > > + * has an lcore backing the coremask specified.
> > > + */
> > > +int rte_eal_service_use_lcore(uint16_t lcore_id);
> > > +
> > > +/**
> > > + * Set a coremask for a service.
> > > + *
> > > + * A configuration for a service indicates which cores run the service, and
> > > + * what other parameters are required to correclty handle the underlying device.
> > > + *
> > > + * Examples of advanced usage might be a HW device that supports multiple lcores
> > > + * transmitting packets on the same queue - the hardware provides a mechanism
> > > + * for multithreaded enqueue. In this case, the atomic_
> > > + *
> > > + * @return 0 Success
> > > + * -ENODEV Device with *name* does not exist
> > > + * -ENOTSUP Configuration is un-supported
> > > + */
> > > +int rte_eal_service_set_coremask(const char *name, uint64_t coremask);
> >
> > Not sure why do we need that function?
> > We do know a service coremask after register() is executed.
> > Would it be possible to add/remove cores from the service on the fly?
> > I.E without stopping the actual service scheduling function first?
> >
> > >6. The application now calls remote_launch() as usual, and the application
> > > + * can perform its processing as required without manually handling the
> > > + * partitioning of lcore resources for DPDK functionality.
> >
> > Ok, if the user has to do remote_launch() manually for each service core anyway...
> > Again it means user can't start/stop individual services, it has to stop the whole service
> > core first(and all services it is running) to stop a particular service.
> > Sounds not very convenient for the user from my perspective.
> > Wonder can we have an API like that:
> >
> > /*
> > * reserves all cores in coremask as service cores.
> > * in fact it would mean eal_remote_launch(service_main_func) on each of them.
> > */
> > int rte_eal_service_start(const rte_cpuset_t *coremask);
> >
> > /*
> > * stops all services and returns service lcores back to the EAL.
> > */
> > int rte_eal_service_stop((const rte_cpuset_t *coremask);
> >
> > struct rte_service_config {name; cb_func; flags; ...};
> >
> > struct rte_service *rte_service_register(struct rte_service_config *cfg);
> > rte_service_unregister(struct rte_service *srv);
> >
> > /*
> > * adds/deletes lcore from the list of service cpus to run this service on.
> > * An open question - should we allow to do add/del lcore while service is running or not.
> > * From user perspective, it would be much more convenient to allow.
> > */
> > int rte_service_add_lcore(struct rte_service *, uint32_t lcore_id, void *lcore_cb_arg);
> > int rte_service_del_lcore(struct rte_service *, uint32_t lcore_id);
> >
> > /*
> > * starts/stops the service.
> > */
> > int rte_service_start(struct rte_service *srv, const rte_cpuset_t *coremask);
> > int rte_service_stop(struct rte_service *srv, const rte_cpuset_t *coremask);
> >
> >
> >
> >
> >
> >
> >
> >
next prev parent reply other threads:[~2017-05-17 13:49 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-05-03 11:29 [dpdk-dev] [RFC] Service Cores concept Harry van Haaren
2017-05-03 11:29 ` [dpdk-dev] [RFC] service core concept header implementation Harry van Haaren
2017-05-04 6:35 ` Jerin Jacob
2017-05-04 12:01 ` Jerin Jacob
2017-05-05 15:48 ` Van Haaren, Harry
2017-05-06 14:26 ` Jerin Jacob
2017-05-17 12:47 ` Ananyev, Konstantin
2017-05-17 12:58 ` Bruce Richardson
2017-05-17 13:47 ` Ananyev, Konstantin [this message]
2017-05-25 13:27 ` Van Haaren, Harry
2017-05-16 22:11 ` [dpdk-dev] [RFC] Service Cores concept Thomas Monjalon
2017-05-16 22:48 ` Wiles, Keith
2017-05-17 10:32 ` Bruce Richardson
2017-05-17 11:28 ` Thomas Monjalon
2017-05-17 12:36 ` Bruce Richardson
2017-05-17 14:51 ` [dpdk-dev] Discuss plugin threading model for DPDK Wiles, Keith
2017-05-17 15:46 ` Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2601191342CEEE43887BDE71AB9772583FAF76D7@IRSMSX109.ger.corp.intel.com \
--to=konstantin.ananyev@intel.com \
--cc=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=harry.van.haaren@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).