Re: drivers use of service cores

[Stephen Hemminger <stephen@networkplumber.org>]

On Fri, 15 Sep 2023 15:54:02 +0200
Morten Brørup <mb@smartsharesystems.com> wrote:

> > For context, Thomas and I (and a few others) had a brief discussion
> > about this topic
> > at userspace in Dublin earlier this week.  I have a bit of better
> > understanding of the
> > problem-space, and we made some progress in technical solutions too.
> >   
> > > I think we can improve the developer experience for using service  
> > cores  
> > > from a driver, like finding or allocating a service core.
> > > We may take some code and ideas from sfc and nfp drivers,
> > > like in these functions:
> > > 	nfp_map_service()
> > > 	sfc_mae_counter_service_register()
> > > 	sfc_get_service_lcore()
> > >
> > > If it is not possible to use a service core, we could default to using  
> > a control thread.  
> > > So the driver would never fail because of a thread initialization.  
> > 
> > There was input from a few people that "hidden threads" that their DPDK
> > application
> > doesn't know about can cause issues (e.g. a driver creating a thread
> > "behind the application's back").
> > I think Thomas suggested a callback function the application could hook-
> > into, to either accept/decline
> > the drivers "request" to create a thread.
> > 
> > The default could be "accept" if the application doesn't hook the
> > callback, allowing drivers to default to
> > achieving work, and allowing power-users to manually handle specific
> > threading-requirements. I have
> > not strong preference here, just writing down the discussions and
> > feedback from Userspace.
> >   
> > > What do you think about proposing such a high level API
> > > in order to get more drivers using it?  

> > I believe service-cores was required to transparently enable certain
> > use-cases of HW-acceleration,
> > Initially Eventdev/SW PMD, but it is of course possible for other
> > components in DPDK to use it.
> > 
> > I do recall some folks had concerns over "scope creep" when initially
> > discussing service-cores upstreaming, and perhaps they're right.
> > I'm not sure how much more functionality is desired here, vs better
> > usability of the service-cores APIs. Perhaps a POC patch of the
> > NFP, SFC, etc use-cases would help drive towards a code-level
> > discussion?  
> 
> Since the discussion in Dublin, I have given this some thoughts. Here's what I think...
> 
> My key objection is this: CPU cycles and CPU cores might be scarce resources. And even though a driver has some "important" work to do, the application might have some other work to do, which is more important and perhaps also timing sensitive. It is a core design principle of DPDK that the application is in full control! Drivers should not have the ability to take away resources from the application; resources should be explicitly allocated and given to the driver by the application.


In the embedded world, it is common that cores and other resources are scarce.
At my previous role, the number of available cores and memory was controlled by cgroups
and the actual allocation decisions were done high up in the project management
team. I doubt this is a unique case.

Also, when a network virtual appliance is run in a VM. The number of cores
in the VM impacts the cost. You pay a lot more for more cores.
Bottom line, cores are not free.

Many applications are not setup to use service lcores. And some are
not even setup to use timers.