Re: [dpdk-dev] kernel binding of devices + hotplug

[Matan Azrad <matan@mellanox.com>]

Hi Guo

From: Guo, Jia, Monday, April 16, 2018 12:26 PM
> To: Matan Azrad <matan@mellanox.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>; Bruce Richardson
> <bruce.richardson@intel.com>; Thomas Monjalon <thomas@monjalon.net>
> Cc: dev@dpdk.org; pmatilai@redhat.com; david.marchand@6wind.com;
> konstantin.ananyev@intel.com; stephen@networkplumber.org;
> fbl@redhat.com
> Subject: Re: kernel binding of devices + hotplug
> 
> hi, all
> 
> 
> On 4/15/2018 4:10 AM, Matan Azrad wrote:
> > Hi all
> >
> > From: Burakov, Anatoly, Friday, April 13, 2018 8:41 PM
> >> To: Bruce Richardson <bruce.richardson@intel.com>; Thomas Monjalon
> >> <thomas@monjalon.net>
> >> Cc: dev@dpdk.org; pmatilai@redhat.com; david.marchand@6wind.com;
> >> jia.guo@intel.com; Matan Azrad <matan@mellanox.com>;
> >> konstantin.ananyev@intel.com; stephen@networkplumber.org;
> >> fbl@redhat.com
> >> Subject: Re: kernel binding of devices + hotplug
> >>
> >> On 13-Apr-18 5:40 PM, Bruce Richardson wrote:
> >>> On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote:
> >>>> It's time to think (again) how we bind devices with kernel modules.
> >>>> We need to decide how we want to manage hotplugged devices with
> >> DPDK.
> >>>> A bit of history first.
> >>>> There was some code in DPDK for bind/unbind, but it has been
> >>>> removed in DPDK 1.7 -
> >>>>
> >>
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdpd
> >>
> k.org%2Fcommit%2F5d8751b83&data=02%7C01%7Cmatan%40mellanox.com
> >> %7C6ea5
> >>
> 5ce994ff4bb0d65208d5a165b417%7Ca652971c7d2e4d9ba6a4d149256f461b%7
> >> C0%7
> >>
> C0%7C636592380565078675&sdata=uLRDAk65hYtJYxjIvY20de377yayCN5DrjCZ
> >> x8H
> >>>> p61o%3D&reserved=0 Copy of the commit message (in 2014):
> >>>> "
> >>>> 	The bind/unbind operations should not be handled by the eal.
> >>>> 	These operations should be either done outside of dpdk or
> >>>> 	inside the PMDs themselves as these are their problems.
> >>>> "
> >>>>
> >>>> The question raised at this time (4 years ago) is still under discussion.
> >>>> Should we manage binding inside or outside DPDK?
> >>>> Should it be controlled in the application or in the OS base?
> >>>>
> >>>> As you know, we use dpdk-devbind.py.
> >>>> This tool lacks two major features:
> >>>> 	- persistent configuration
> >>>> 	- hotplug
> >>>>
> >>>> If we consider that the DPDK applications should be able to apply
> >>>> its own policy to choose the devices to bind, then we need to
> >>>> implement binding in the PMD (with EAL helpers).
> >>>>
> >>>> On the other hand, if we consider that it is the system
> >>>> responsibility, then we could choose systemd/udev and driverctl.
> >>>>
> >>>> The debate is launched!
> >>>>
> >>> Allow me to nail my colours to the mast early! :-)
> >>>
> >>> I believe it's system not application responsibility.
> >>> I also believe I have previously explained my reasons for that
> >>> choice in some of the previous email threads.
> >> For what it's worth, I tend to agree, if only because writing code
> >> for what is essentially a bunch of read/write/filesystem enumeration
> >> in C is extremely fiddly and error prone :) IMO things like this are
> >> better handled either by scripts, or by tools whose sole purpose is doing
> exactly that (or both).
> >>
> >> I like having scripts like devbind in DPDK because we can tailor them
> >> to our use cases better, and having them is amenable to automation,
> >> but while I wouldn't be opposed to removing them altogether in favor
> >> of some external tool (systemd/udev/driverctl/whatever), in my humble
> >> opinion moving them back into EAL or even PMD's would be a mistake.
> >>
> > Since the application runs in the system by a command of the system user I
> think the responsibility is for the user.
> > The DPDK user forwards the control of some devices to the DPDK
> > application using the EAL whitelist\blacklist mode to specify the
> > devices, Any DPDK PMD should know which binding it needs to
> probe\control the device and can apply it, So, if the user asks to control on a
> device by DPDK application it makes sense that the application will do the
> correct binding to the device since the user wants to use it(no need to ask
> more operation of pre binding from the user).
> >
> > Regarding the conflict of system rules for a device, it is again the user
> responsibility, whatever we will decide for the binding procedure of DPDK
> application the user needs to take it into account and to solve such like
> conflicts.
> > One option is to remove any binding rules of a DPDK device in the DPDK
> application initialization and adjust the new rules by the PMDs, then any
> conflict should not disturb the user.
> >
> > In current hot-plug case the application will need to do a lot of work to
> bind\remap devices in plug-in\plug-out events while the PMD could have all
> the knowledge to do it.
> >
> > One more issue with the script is that the user should do different bind per
> device, in case of PMD responsibility the user can forget it:
> > Think about that, any time the user wants to switch\add new supported nic
> it should update the script usage and to do per nic operation contrary to the
> DPDK principles.
> >
> > Matan.
> >> Thanks,
> >> Anatoly
> when device appear whenever dpdk is runnning or not, the device will
> default bind to Kernel driver,

Yes

> user or say system admin could use the script or
> tools to rebind a specific driver which according their request so i think user
> space tools provide functional and user have the binding responsibility rather
> than the app or PMD.

The app\PMD is running by the user, so it is OK(the user just need to know what the app does).

> i don't understand why over ride to other driver is the
> scope of an specific PMD.

Because the PMD can know what binding it needs for using the device.
All the options can be handled by EAL  - the PMD just need to say what is the driver to bind.

> And if there is conflict by rules , user could over
> ride it and take the control.

We can define that all the control of an rte_device is for DPDK.
 
> Per dpdk hotplug, the purpose is for the app failsafe and VM live migration,
> driverctl is focus driver control and udev is focus on device hotplug but no
> userspace failure handle , there are still not an complete hotplug solution for
> dpdk app,  even the script have binding, it look like as a minimal "system
> admin tools" fit for dpdk, but not persistent and not hotplug functional.

I suggested full solution without app involvement:
Plug-out detected by EAL->PMD notification->remap by the PMD->app notification->just stop to use the device.
Plug-in detected by EAL->whitelist\blacklist match->probe->bind by PMD -> app notification->start to use the device.

>  so if
> we aim to provider a total hotplug service at dpdk framework, offload these
> works from application and user that is need. In normal user take the driver
> control by tools/script(both ok in my opinion) optional at initial , while in
> hotplug case, failure occur on runtime, dpdk take the control of driver from
> user. compare with app and PMD, do you agree dpdk best to take the role on
> runtime?

Yes, but by the PMD, not by the app\user.
So, EAL will probe the device if it was matched with the whitelist\blacklist rules and the PMD will do the correct bind in the probe function.
The application just should get notification of the probing and to start to manage the device.

> if only user could take the role, the SRIOV live migration of dpdk
> would hard to be make sense.

Sorry, don't understand you here.

> i agree let app don't care about any bind/failure handle, in current hotplug
> solution failure handle embedded in dpdk but binding is exposed to app, if
> got agreement to dpdk binding , i could let dpdk self do it.
> if vote result is not let dpdk charge it, i will temporary remove the binding
> function to shrink the hotplug scope. postpone  it to further failsafe and live
> migrate topic.
> 
> basically speaking, for binding, script vs tools vs PMD vs dpdk, i vote dpdk , at
> lest run time case.

As I said above - I vote for both EAL\PMDs.