Re: [dpdk-dev] [dpdk-techboard] RFC: DPDK drivers for DPDK bus types

[Bruce Richardson <bruce.richardson@intel.com>]

On Thu, Mar 11, 2021 at 04:44:49PM +0100, Thomas Monjalon wrote:
> 11/03/2021 16:19, Bruce Richardson:
> > Hi all,
> > 
> > looking for input here into the area of bus-type drivers and interaction
> > with other drivers in DPDK.
> > 
> > By way of context, I'm looking at extending the vdev support in the
> > "raw/ioat" driver (file raw/ioat/idxd_vdev.c) to make it more user
> > friendly.  These devices are accessed by DPDK via nodes in /dev and
> > paths in /sys, with the vdev parameters being passed identifying the
> > particular devices to use. However, the presence of these devices can be
> > detected at runtime by a scan of /dev and /sys, and so it's easy enough
> > to implement a custom bus-type driver in DPDK to detect these, rather
> > than having the user pass in vdev parameters (which can get awkward to
> > use as the number of devices increases).
> 
> I agree. vdev bus should be used for creating device from the void.
> If the device has its roots in the system (HW or SW), there should be
> a bus for that.
> 
> 
> > However, looking through a few other drivers in the "bus" directory, it
> > appears that scanning system paths, i.e. /sys, is fairly common, so I'm
> > wondering if it's possible to have some sharing of functionality here.
> > Unfortunately, the use of /sys in each of these drivers I've looked at
> > seem sufficiently different to me that I've not immediately seen a
> > common level of abstraction we can use. Therefore I'm looking for
> > suggestions here that those in the community might have.
> 
> Not sure it's worth looking for such sharing between bus.
> 
> 
> > On a related note, I'm also concerned about the need for a single device
> > type, e.g. one used by DPDK and shared with the kernel, to require two
> > separate drivers to work together to support it - a bus driver for
> > scanning and a type-specific driver for the actual functional
> > implementation. Can we not find a way to reduce the number of drivers
> > needing to be supported?
> 
> The bus driver is managing the device life.
> The device driver implements a functional class.
> I don't see what to save.
> Maybe you are biased because the rawdev class is fake.
> 

The fact of the rawdev class being fake is irrelevant to this, the same
would apply to any device class which as you say - "has it's roots in the
system". However, for some of these devices, the roots in the system may
well be unique, and therefore - irrespective of device class - one needs
two separate drivers to work with such a device, a bus driver to do system
scanning, and a device driver to manage the instances.

> 
> > Following on from this, and if we can't find a good way of doing a
> > generic driver for scanning /sys nodes, I wonder if there is value in
> > providing a "generic" bus implementation in DPDK, as a catch-all for
> > device drivers which need their own custom probing, but that do not
> > neatly fall into the other types. The way this might work is to have the
> > scanning and probing of devices left entirely to the device driver
> > implementation itself. For example, rather than creating an idxd
> > bus driver, it would be easier and more self-contained to have the
> > rawdev driver itself able to perform scanning and probing - keeping the
> > code all in one place. All the bus driver would have to do is maintain a
> > list of drivers and found devices reported by the individual driver
> > after they have done their own probing.
> 
> So you mean there is a single user of the bus,
> so the implementation could be moved into the device driver,
> relying on a fake bus?
> 

Exactly. If the bus exists for only one device instance - and will only
ever exist for one device - why not allow the driver itself to have an
extra function called "scan", rather than having to create a custom bus
driver just to do that scanning.

> 
> > Other possible candidates to think about that might be able to use
> > their own probing from a generic bus might be, e.g. af_xdp driver, or a
> > TAP or memif driver.
> 
> These devices don't exist naturally in the system, so I think they should
> be vdev.
> 
AF_XDP devices certainly exist in the system, and serves at least as a good
theoretical model of exactly what I am describing. Drivers for TAP or memif
could be written to scan for and attach to already-existing instances on
the system too. 

For the AF_XDP case, suppose that the kernel was extended to allow for
certain queues to be marked or named, one could then create an AF_XDP bus
driver to look for named queues on each NIC interface. Similarly a TAP bus
device could scan for TAP instances previously created and configured and
attach to them. The key thing is that in each case, the scanning done is
unique and applies to only one net driver - nothing rawdev related about
this! Therefore, while I don't think we would ever really implement bus
drivers for these specific cases, they are examples which I think shows the
value in having a bus type to allow generic scanning, rather than forcing a
new bus driver to be created for each individual type of scanning
necessary.

Regards,
/Bruce