Re: [dpdk-dev] kernel binding of devices + hotplug

[Matan Azrad <matan@mellanox.com>]

Hi Bruce

From: Bruce Richardson, Monday, April 16, 2018 11:32 AM
> On Sat, Apr 14, 2018 at 08:10:28PM +0000, Matan Azrad wrote:
> > Hi all
> >
> > From: Burakov, Anatoly, Friday, April 13, 2018 8:41 PM
> > > To: Bruce Richardson <bruce.richardson@intel.com>; Thomas Monjalon
> > > <thomas@monjalon.net>
> > > Cc: dev@dpdk.org; pmatilai@redhat.com; david.marchand@6wind.com;
> > > jia.guo@intel.com; Matan Azrad <matan@mellanox.com>;
> > > konstantin.ananyev@intel.com; stephen@networkplumber.org;
> > > fbl@redhat.com
> > > Subject: Re: kernel binding of devices + hotplug
> > >
> > > On 13-Apr-18 5:40 PM, Bruce Richardson wrote:
> > > > On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote:
> > > >> It's time to think (again) how we bind devices with kernel modules.
> > > >> We need to decide how we want to manage hotplugged devices with
> > > DPDK.
> > > >>
> > > >> A bit of history first.
> > > >> There was some code in DPDK for bind/unbind, but it has been
> > > >> removed in DPDK 1.7 -
> > > >>
> > >
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdp
> > > d
> > > >>
> > >
> k.org%2Fcommit%2F5d8751b83&data=02%7C01%7Cmatan%40mellanox.com
> > > %7C6ea5
> > > >>
> > >
> 5ce994ff4bb0d65208d5a165b417%7Ca652971c7d2e4d9ba6a4d149256f461b%7
> > > C0%7
> > > >>
> > >
> C0%7C636592380565078675&sdata=uLRDAk65hYtJYxjIvY20de377yayCN5DrjCZ
> > > x8H
> > > >> p61o%3D&reserved=0 Copy of the commit message (in 2014):
> > > >> "
> > > >> 	The bind/unbind operations should not be handled by the eal.
> > > >> 	These operations should be either done outside of dpdk or
> > > >> 	inside the PMDs themselves as these are their problems.
> > > >> "
> > > >>
> > > >> The question raised at this time (4 years ago) is still under discussion.
> > > >> Should we manage binding inside or outside DPDK?
> > > >> Should it be controlled in the application or in the OS base?
> > > >>
> > > >> As you know, we use dpdk-devbind.py.
> > > >> This tool lacks two major features:
> > > >> 	- persistent configuration
> > > >> 	- hotplug
> > > >>
> > > >> If we consider that the DPDK applications should be able to apply
> > > >> its own policy to choose the devices to bind, then we need to
> > > >> implement binding in the PMD (with EAL helpers).
> > > >>
> > > >> On the other hand, if we consider that it is the system
> > > >> responsibility, then we could choose systemd/udev and driverctl.
> > > >>
> > > >> The debate is launched!
> > > >>
> > > >
> > > > Allow me to nail my colours to the mast early! :-)
> > > >
> > > > I believe it's system not application responsibility.
> > > > I also believe I have previously explained my reasons for that
> > > > choice in some of the previous email threads.
> > >
> > > For what it's worth, I tend to agree, if only because writing code
> > > for what is essentially a bunch of read/write/filesystem enumeration
> > > in C is extremely fiddly and error prone :) IMO things like this are
> > > better handled either by scripts, or by tools whose sole purpose is doing
> exactly that (or both).
> > >
> > > I like having scripts like devbind in DPDK because we can tailor
> > > them to our use cases better, and having them is amenable to
> > > automation, but while I wouldn't be opposed to removing them
> > > altogether in favor of some external tool
> > > (systemd/udev/driverctl/whatever), in my humble opinion moving them
> back into EAL or even PMD's would be a mistake.
> > >
> >
> > Since the application runs in the system by a command of the system user I
> think the responsibility is for the user.
> > The DPDK user forwards the control of some devices to the DPDK
> > application using the EAL whitelist\blacklist mode to specify the
> > devices, Any DPDK PMD should know which binding it needs to
>> probe\control the device and can apply it, So, if the user asks to control on a
>> device by DPDK application it makes sense that the application will do the
>> correct binding to the device since the user wants to use it(no need to ask
>> more operation of pre binding from the user).
> 
> Completely agree that it is ultimately up to the user. However, what I don't
> want to see is the case where the user always has to specify a big long list of
> device whitelist and blacklist options to each run of an application. Instead, if
> device management is done at the system level via udev (for example)
> configured via devicectl, then the application commandline can be vastly
> simplified.

Actually you say that the whitelist\blacklist mechanism is not good enough and the binding workarounds it.
The user need to specify somehow the devices it want to run, 
I think that specifying the device you want by -w option (no need to specify what you don't want in -w case) is really simpler  and more descriptive than binding each device you want by prior process to its correct driver.
 
> It also allows better usability across systems, since the same
> commandline can be used on multiple systems with different hardware, with
> the actual device management rules having been already configured at
> system install/setup time in udev.

But the user still needs to configure the udev per device for each system, I think that command line is better.

> > Regarding the conflict of system rules for a device, it is again the user
> responsibility, whatever we will decide for the binding procedure of DPDK
> application the user needs to take it into account and to solve such like
> conflicts.
> > One option is to remove any binding rules of a DPDK device in the DPDK
> application initialization and adjust the new rules by the PMDs, then any
> conflict should not disturb the user.
> 
> If the device management is only managed in one place, i.e. not in DPDK,
> then there is no conflict to manage.

I can't agree with this statement,
The essence of DPDK is to give a good alternative to managing network devices,
DPDK actually takes a lot of management area to manage by itself to do the user life better :)

Moreover,
Instead of bind script usage and dpdk running , just run dpdk (do all the job in one place).

> > In current hot-plug case the application will need to do a lot of work to
> bind\remap devices in plug-in\plug-out events while the PMD could have all
> the knowledge to do it.
> 
> At the cost of duplicating a lot of code between PMDs.

Why a lot of code? 1 helper in EAL to be used for each relevant PMD.
Each PMD just call to the EAL helper with the bind driver type.
Really simple and immediate.

I think it is better than duplication of user operations.

> > One more issue with the script is that the user should do different bind per
> device, in case of PMD responsibility the user can forget it:
> > Think about that, any time the user wants to switch\add new supported nic
> it should update the script usage and to do per nic operation contrary to the
> DPDK principles.
> >
> 
> The udev rules syntax should provide adequate capabilities here for us to
> match the correct binding behaviour. No need to have it in DPDK too.

We can use it from DPDK.
 
> /Bruce