[dpdk-dev] kernel binding of devices + hotplug

[dpdk-dev] kernel binding of devices + hotplug

[Thomas Monjalon]

It's time to think (again) how we bind devices with kernel modules.
We need to decide how we want to manage hotplugged devices with DPDK.

A bit of history first.
There was some code in DPDK for bind/unbind, but it has been removed
in DPDK 1.7 - http://dpdk.org/commit/5d8751b83
Copy of the commit message (in 2014):
"
	The bind/unbind operations should not be handled by the eal.
	These operations should be either done outside of dpdk or
	inside the PMDs themselves as these are their problems.
"

The question raised at this time (4 years ago) is still under discussion.
Should we manage binding inside or outside DPDK?
Should it be controlled in the application or in the OS base?

As you know, we use dpdk-devbind.py.
This tool lacks two major features:
	- persistent configuration
	- hotplug

If we consider that the DPDK applications should be able to apply its own
policy to choose the devices to bind, then we need to implement binding
in the PMD (with EAL helpers).

On the other hand, if we consider that it is the system responsibility,
then we could choose systemd/udev and driverctl.

The debate is launched!

Please find more details in the references below.

Announce of driverctl:
	http://dpdk.org/ml/archives/dev/2015-December/029500.html
Repository of driverctl:
	https://gitlab.com/driverctl/driverctl

Discussion about binding script and driverctl:
	http://dpdk.org/ml/archives/dev/2018-April/095687.html

Patch to implement binding in DPDK (for hotplug):
	http://dpdk.org/ml/archives/dev/2018-April/095714.html

Discussion in the same hotplug series:
	http://dpdk.org/ml/archives/dev/2018-April/097058.html

Re: [dpdk-dev] kernel binding of devices + hotplug

[Bruce Richardson]

On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote:
> It's time to think (again) how we bind devices with kernel modules.
> We need to decide how we want to manage hotplugged devices with DPDK.
> 
> A bit of history first.
> There was some code in DPDK for bind/unbind, but it has been removed
> in DPDK 1.7 - http://dpdk.org/commit/5d8751b83
> Copy of the commit message (in 2014):
> "
> 	The bind/unbind operations should not be handled by the eal.
> 	These operations should be either done outside of dpdk or
> 	inside the PMDs themselves as these are their problems.
> "
> 
> The question raised at this time (4 years ago) is still under discussion.
> Should we manage binding inside or outside DPDK?
> Should it be controlled in the application or in the OS base?
> 
> As you know, we use dpdk-devbind.py.
> This tool lacks two major features:
> 	- persistent configuration
> 	- hotplug
> 
> If we consider that the DPDK applications should be able to apply its own
> policy to choose the devices to bind, then we need to implement binding
> in the PMD (with EAL helpers).
> 
> On the other hand, if we consider that it is the system responsibility,
> then we could choose systemd/udev and driverctl.
> 
> The debate is launched!
> 

Allow me to nail my colours to the mast early! :-)

I believe it's system not application responsibility.
I also believe I have previously explained my reasons for that choice in
some of the previous email threads.

/Bruce

Re: [dpdk-dev] kernel binding of devices + hotplug

[Burakov, Anatoly]

On 13-Apr-18 5:40 PM, Bruce Richardson wrote:
> On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote:
>> It's time to think (again) how we bind devices with kernel modules.
>> We need to decide how we want to manage hotplugged devices with DPDK.
>>
>> A bit of history first.
>> There was some code in DPDK for bind/unbind, but it has been removed
>> in DPDK 1.7 - http://dpdk.org/commit/5d8751b83
>> Copy of the commit message (in 2014):
>> "
>> 	The bind/unbind operations should not be handled by the eal.
>> 	These operations should be either done outside of dpdk or
>> 	inside the PMDs themselves as these are their problems.
>> "
>>
>> The question raised at this time (4 years ago) is still under discussion.
>> Should we manage binding inside or outside DPDK?
>> Should it be controlled in the application or in the OS base?
>>
>> As you know, we use dpdk-devbind.py.
>> This tool lacks two major features:
>> 	- persistent configuration
>> 	- hotplug
>>
>> If we consider that the DPDK applications should be able to apply its own
>> policy to choose the devices to bind, then we need to implement binding
>> in the PMD (with EAL helpers).
>>
>> On the other hand, if we consider that it is the system responsibility,
>> then we could choose systemd/udev and driverctl.
>>
>> The debate is launched!
>>
> 
> Allow me to nail my colours to the mast early! :-)
> 
> I believe it's system not application responsibility.
> I also believe I have previously explained my reasons for that choice in
> some of the previous email threads.

For what it's worth, I tend to agree, if only because writing code for 
what is essentially a bunch of read/write/filesystem enumeration in C is 
extremely fiddly and error prone :) IMO things like this are better 
handled either by scripts, or by tools whose sole purpose is doing 
exactly that (or both).

I like having scripts like devbind in DPDK because we can tailor them to 
our use cases better, and having them is amenable to automation, but 
while I wouldn't be opposed to removing them altogether in favor of some 
external tool (systemd/udev/driverctl/whatever), in my humble opinion 
moving them back into EAL or even PMD's would be a mistake.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] kernel binding of devices + hotplug

[Matan Azrad]

Hi all

From: Burakov, Anatoly, Friday, April 13, 2018 8:41 PM
> To: Bruce Richardson <bruce.richardson@intel.com>; Thomas Monjalon
> <thomas@monjalon.net>
> Cc: dev@dpdk.org; pmatilai@redhat.com; david.marchand@6wind.com;
> jia.guo@intel.com; Matan Azrad <matan@mellanox.com>;
> konstantin.ananyev@intel.com; stephen@networkplumber.org;
> fbl@redhat.com
> Subject: Re: kernel binding of devices + hotplug
> 
> On 13-Apr-18 5:40 PM, Bruce Richardson wrote:
> > On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote:
> >> It's time to think (again) how we bind devices with kernel modules.
> >> We need to decide how we want to manage hotplugged devices with
> DPDK.
> >>
> >> A bit of history first.
> >> There was some code in DPDK for bind/unbind, but it has been removed
> >> in DPDK 1.7 -
> >>
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdpd
> >>
> k.org%2Fcommit%2F5d8751b83&data=02%7C01%7Cmatan%40mellanox.com
> %7C6ea5
> >>
> 5ce994ff4bb0d65208d5a165b417%7Ca652971c7d2e4d9ba6a4d149256f461b%7
> C0%7
> >>
> C0%7C636592380565078675&sdata=uLRDAk65hYtJYxjIvY20de377yayCN5DrjCZ
> x8H
> >> p61o%3D&reserved=0 Copy of the commit message (in 2014):
> >> "
> >> 	The bind/unbind operations should not be handled by the eal.
> >> 	These operations should be either done outside of dpdk or
> >> 	inside the PMDs themselves as these are their problems.
> >> "
> >>
> >> The question raised at this time (4 years ago) is still under discussion.
> >> Should we manage binding inside or outside DPDK?
> >> Should it be controlled in the application or in the OS base?
> >>
> >> As you know, we use dpdk-devbind.py.
> >> This tool lacks two major features:
> >> 	- persistent configuration
> >> 	- hotplug
> >>
> >> If we consider that the DPDK applications should be able to apply its
> >> own policy to choose the devices to bind, then we need to implement
> >> binding in the PMD (with EAL helpers).
> >>
> >> On the other hand, if we consider that it is the system
> >> responsibility, then we could choose systemd/udev and driverctl.
> >>
> >> The debate is launched!
> >>
> >
> > Allow me to nail my colours to the mast early! :-)
> >
> > I believe it's system not application responsibility.
> > I also believe I have previously explained my reasons for that choice
> > in some of the previous email threads.
> 
> For what it's worth, I tend to agree, if only because writing code for what is
> essentially a bunch of read/write/filesystem enumeration in C is extremely
> fiddly and error prone :) IMO things like this are better handled either by
> scripts, or by tools whose sole purpose is doing exactly that (or both).
> 
> I like having scripts like devbind in DPDK because we can tailor them to our
> use cases better, and having them is amenable to automation, but while I
> wouldn't be opposed to removing them altogether in favor of some external
> tool (systemd/udev/driverctl/whatever), in my humble opinion moving them
> back into EAL or even PMD's would be a mistake.
> 

Since the application runs in the system by a command of the system user I think the responsibility is for the user.
The DPDK user forwards the control of some devices to the DPDK application using the EAL whitelist\blacklist mode to specify the devices,
Any DPDK PMD should know which binding it needs to probe\control the device and can apply it,
So, if the user asks to control on a device by DPDK application it makes sense that the application will do the correct binding to the device since the user wants to use it(no need to ask more operation of pre binding from the user).

Regarding the conflict of system rules for a device, it is again the user responsibility, whatever we will decide for the binding procedure of DPDK application the user needs to take it into account and to solve such like conflicts.
One option is to remove any binding rules of a DPDK device in the DPDK application initialization and adjust the new rules by the PMDs, then any conflict should not disturb the user.

In current hot-plug case the application will need to do a lot of work to bind\remap devices in plug-in\plug-out events while the PMD could have all the knowledge to do it. 

One more issue with the script is that the user should do different bind per device, in case of PMD responsibility the user can forget it:
Think about that, any time the user wants to switch\add new supported nic it should update the script usage and to do per nic operation contrary to the DPDK principles.

Matan.

> Thanks,
> Anatoly

Re: [dpdk-dev] kernel binding of devices + hotplug

[Stephen Hemminger]

My vote is to work with udev and not try to replace it.

Driverctl works well. Just not for bifurcated driver

On Fri, Apr 13, 2018, 9:31 AM Thomas Monjalon <thomas@monjalon.net> wrote:

> It's time to think (again) how we bind devices with kernel modules.
> We need to decide how we want to manage hotplugged devices with DPDK.
>
> A bit of history first.
> There was some code in DPDK for bind/unbind, but it has been removed
> in DPDK 1.7 - http://dpdk.org/commit/5d8751b83
> Copy of the commit message (in 2014):
> "
>         The bind/unbind operations should not be handled by the eal.
>         These operations should be either done outside of dpdk or
>         inside the PMDs themselves as these are their problems.
> "
>
> The question raised at this time (4 years ago) is still under discussion.
> Should we manage binding inside or outside DPDK?
> Should it be controlled in the application or in the OS base?
>
> As you know, we use dpdk-devbind.py.
> This tool lacks two major features:
>         - persistent configuration
>         - hotplug
>
> If we consider that the DPDK applications should be able to apply its own
> policy to choose the devices to bind, then we need to implement binding
> in the PMD (with EAL helpers).
>
> On the other hand, if we consider that it is the system responsibility,
> then we could choose systemd/udev and driverctl.
>
> The debate is launched!
>
> Please find more details in the references below.
>
> Announce of driverctl:
>         http://dpdk.org/ml/archives/dev/2015-December/029500.html
> Repository of driverctl:
>         https://gitlab.com/driverctl/driverctl
>
> Discussion about binding script and driverctl:
>         http://dpdk.org/ml/archives/dev/2018-April/095687.html
>
> Patch to implement binding in DPDK (for hotplug):
>         http://dpdk.org/ml/archives/dev/2018-April/095714.html
>
> Discussion in the same hotplug series:
>         http://dpdk.org/ml/archives/dev/2018-April/097058.html
>
>
>
>

Re: [dpdk-dev] kernel binding of devices + hotplug

[Wiles, Keith]

On Apr 13, 2018, at 11:40 AM, Bruce Richardson <bruce.richardson@intel.com<mailto:bruce.richardson@intel.com>> wrote:

On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote:
It's time to think (again) how we bind devices with kernel modules.
We need to decide how we want to manage hotplugged devices with DPDK.

A bit of history first.
There was some code in DPDK for bind/unbind, but it has been removed
in DPDK 1.7 - http://dpdk.org/commit/5d8751b83
Copy of the commit message (in 2014):
"
The bind/unbind operations should not be handled by the eal.
These operations should be either done outside of dpdk or
inside the PMDs themselves as these are their problems.
"

The question raised at this time (4 years ago) is still under discussion.
Should we manage binding inside or outside DPDK?
Should it be controlled in the application or in the OS base?

As you know, we use dpdk-devbind.py.
This tool lacks two major features:
- persistent configuration
- hotplug

If we consider that the DPDK applications should be able to apply its own
policy to choose the devices to bind, then we need to implement binding
in the PMD (with EAL helpers).

On the other hand, if we consider that it is the system responsibility,
then we could choose systemd/udev and driverctl.

The debate is launched!


Allow me to nail my colours to the mast early! :-)

I believe it's system not application responsibility.
I also believe I have previously explained my reasons for that choice in
some of the previous email threads.

I agree the system should be handling hot-plug and device bindings as the sys-admin or orchestration layer would be handling what device appears/disappears plus which are accessible by an application. Now an application would need to know when a device appears/disappears, which should be handled by the current hot-plug support (I assume).


/Bruce

Regards,
Keith

Re: [dpdk-dev] kernel binding of devices + hotplug

[Bruce Richardson]

On Sat, Apr 14, 2018 at 08:10:28PM +0000, Matan Azrad wrote:
> Hi all
> 
> From: Burakov, Anatoly, Friday, April 13, 2018 8:41 PM
> > To: Bruce Richardson <bruce.richardson@intel.com>; Thomas Monjalon
> > <thomas@monjalon.net>
> > Cc: dev@dpdk.org; pmatilai@redhat.com; david.marchand@6wind.com;
> > jia.guo@intel.com; Matan Azrad <matan@mellanox.com>;
> > konstantin.ananyev@intel.com; stephen@networkplumber.org;
> > fbl@redhat.com
> > Subject: Re: kernel binding of devices + hotplug
> > 
> > On 13-Apr-18 5:40 PM, Bruce Richardson wrote:
> > > On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote:
> > >> It's time to think (again) how we bind devices with kernel modules.
> > >> We need to decide how we want to manage hotplugged devices with
> > DPDK.
> > >>
> > >> A bit of history first.
> > >> There was some code in DPDK for bind/unbind, but it has been removed
> > >> in DPDK 1.7 -
> > >>
> > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdpd
> > >>
> > k.org%2Fcommit%2F5d8751b83&data=02%7C01%7Cmatan%40mellanox.com
> > %7C6ea5
> > >>
> > 5ce994ff4bb0d65208d5a165b417%7Ca652971c7d2e4d9ba6a4d149256f461b%7
> > C0%7
> > >>
> > C0%7C636592380565078675&sdata=uLRDAk65hYtJYxjIvY20de377yayCN5DrjCZ
> > x8H
> > >> p61o%3D&reserved=0 Copy of the commit message (in 2014):
> > >> "
> > >> 	The bind/unbind operations should not be handled by the eal.
> > >> 	These operations should be either done outside of dpdk or
> > >> 	inside the PMDs themselves as these are their problems.
> > >> "
> > >>
> > >> The question raised at this time (4 years ago) is still under discussion.
> > >> Should we manage binding inside or outside DPDK?
> > >> Should it be controlled in the application or in the OS base?
> > >>
> > >> As you know, we use dpdk-devbind.py.
> > >> This tool lacks two major features:
> > >> 	- persistent configuration
> > >> 	- hotplug
> > >>
> > >> If we consider that the DPDK applications should be able to apply its
> > >> own policy to choose the devices to bind, then we need to implement
> > >> binding in the PMD (with EAL helpers).
> > >>
> > >> On the other hand, if we consider that it is the system
> > >> responsibility, then we could choose systemd/udev and driverctl.
> > >>
> > >> The debate is launched!
> > >>
> > >
> > > Allow me to nail my colours to the mast early! :-)
> > >
> > > I believe it's system not application responsibility.
> > > I also believe I have previously explained my reasons for that choice
> > > in some of the previous email threads.
> > 
> > For what it's worth, I tend to agree, if only because writing code for what is
> > essentially a bunch of read/write/filesystem enumeration in C is extremely
> > fiddly and error prone :) IMO things like this are better handled either by
> > scripts, or by tools whose sole purpose is doing exactly that (or both).
> > 
> > I like having scripts like devbind in DPDK because we can tailor them to our
> > use cases better, and having them is amenable to automation, but while I
> > wouldn't be opposed to removing them altogether in favor of some external
> > tool (systemd/udev/driverctl/whatever), in my humble opinion moving them
> > back into EAL or even PMD's would be a mistake.
> > 
> 
> Since the application runs in the system by a command of the system user I think the responsibility is for the user.
> The DPDK user forwards the control of some devices to the DPDK application using the EAL whitelist\blacklist mode to specify the devices,
> Any DPDK PMD should know which binding it needs to probe\control the device and can apply it,
> So, if the user asks to control on a device by DPDK application it makes sense that the application will do the correct binding to the device since the user wants to use it(no need to ask more operation of pre binding from the user).

Completely agree that it is ultimately up to the user. However, what I
don't want to see is the case where the user always has to specify a big
long list of device whitelist and blacklist options to each run of an
application. Instead, if device management is done at the system level via
udev (for example) configured via devicectl, then the application
commandline can be vastly simplified. It also allows better usability
across systems, since the same commandline can be used on multiple systems
with different hardware, with the actual device management rules having
been already configured at system install/setup time in udev.

> 
> Regarding the conflict of system rules for a device, it is again the user responsibility, whatever we will decide for the binding procedure of DPDK application the user needs to take it into account and to solve such like conflicts.
> One option is to remove any binding rules of a DPDK device in the DPDK application initialization and adjust the new rules by the PMDs, then any conflict should not disturb the user.

If the device management is only managed in one place, i.e. not in DPDK,
then there is no conflict to manage.

> 
> In current hot-plug case the application will need to do a lot of work to bind\remap devices in plug-in\plug-out events while the PMD could have all the knowledge to do it. 

At the cost of duplicating a lot of code between PMDs.

> 
> One more issue with the script is that the user should do different bind per device, in case of PMD responsibility the user can forget it:
> Think about that, any time the user wants to switch\add new supported nic it should update the script usage and to do per nic operation contrary to the DPDK principles.
> 

The udev rules syntax should provide adequate capabilities here for us to
match the correct binding behaviour. No need to have it in DPDK too.

/Bruce

Re: [dpdk-dev] kernel binding of devices + hotplug

[Guo, Jia]

hi, all


On 4/15/2018 4:10 AM, Matan Azrad wrote:
> Hi all
>
> From: Burakov, Anatoly, Friday, April 13, 2018 8:41 PM
>> To: Bruce Richardson <bruce.richardson@intel.com>; Thomas Monjalon
>> <thomas@monjalon.net>
>> Cc: dev@dpdk.org; pmatilai@redhat.com; david.marchand@6wind.com;
>> jia.guo@intel.com; Matan Azrad <matan@mellanox.com>;
>> konstantin.ananyev@intel.com; stephen@networkplumber.org;
>> fbl@redhat.com
>> Subject: Re: kernel binding of devices + hotplug
>>
>> On 13-Apr-18 5:40 PM, Bruce Richardson wrote:
>>> On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote:
>>>> It's time to think (again) how we bind devices with kernel modules.
>>>> We need to decide how we want to manage hotplugged devices with
>> DPDK.
>>>> A bit of history first.
>>>> There was some code in DPDK for bind/unbind, but it has been removed
>>>> in DPDK 1.7 -
>>>>
>> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdpd
>> k.org%2Fcommit%2F5d8751b83&data=02%7C01%7Cmatan%40mellanox.com
>> %7C6ea5
>> 5ce994ff4bb0d65208d5a165b417%7Ca652971c7d2e4d9ba6a4d149256f461b%7
>> C0%7
>> C0%7C636592380565078675&sdata=uLRDAk65hYtJYxjIvY20de377yayCN5DrjCZ
>> x8H
>>>> p61o%3D&reserved=0 Copy of the commit message (in 2014):
>>>> "
>>>> 	The bind/unbind operations should not be handled by the eal.
>>>> 	These operations should be either done outside of dpdk or
>>>> 	inside the PMDs themselves as these are their problems.
>>>> "
>>>>
>>>> The question raised at this time (4 years ago) is still under discussion.
>>>> Should we manage binding inside or outside DPDK?
>>>> Should it be controlled in the application or in the OS base?
>>>>
>>>> As you know, we use dpdk-devbind.py.
>>>> This tool lacks two major features:
>>>> 	- persistent configuration
>>>> 	- hotplug
>>>>
>>>> If we consider that the DPDK applications should be able to apply its
>>>> own policy to choose the devices to bind, then we need to implement
>>>> binding in the PMD (with EAL helpers).
>>>>
>>>> On the other hand, if we consider that it is the system
>>>> responsibility, then we could choose systemd/udev and driverctl.
>>>>
>>>> The debate is launched!
>>>>
>>> Allow me to nail my colours to the mast early! :-)
>>>
>>> I believe it's system not application responsibility.
>>> I also believe I have previously explained my reasons for that choice
>>> in some of the previous email threads.
>> For what it's worth, I tend to agree, if only because writing code for what is
>> essentially a bunch of read/write/filesystem enumeration in C is extremely
>> fiddly and error prone :) IMO things like this are better handled either by
>> scripts, or by tools whose sole purpose is doing exactly that (or both).
>>
>> I like having scripts like devbind in DPDK because we can tailor them to our
>> use cases better, and having them is amenable to automation, but while I
>> wouldn't be opposed to removing them altogether in favor of some external
>> tool (systemd/udev/driverctl/whatever), in my humble opinion moving them
>> back into EAL or even PMD's would be a mistake.
>>
> Since the application runs in the system by a command of the system user I think the responsibility is for the user.
> The DPDK user forwards the control of some devices to the DPDK application using the EAL whitelist\blacklist mode to specify the devices,
> Any DPDK PMD should know which binding it needs to probe\control the device and can apply it,
> So, if the user asks to control on a device by DPDK application it makes sense that the application will do the correct binding to the device since the user wants to use it(no need to ask more operation of pre binding from the user).
>
> Regarding the conflict of system rules for a device, it is again the user responsibility, whatever we will decide for the binding procedure of DPDK application the user needs to take it into account and to solve such like conflicts.
> One option is to remove any binding rules of a DPDK device in the DPDK application initialization and adjust the new rules by the PMDs, then any conflict should not disturb the user.
>
> In current hot-plug case the application will need to do a lot of work to bind\remap devices in plug-in\plug-out events while the PMD could have all the knowledge to do it.
>
> One more issue with the script is that the user should do different bind per device, in case of PMD responsibility the user can forget it:
> Think about that, any time the user wants to switch\add new supported nic it should update the script usage and to do per nic operation contrary to the DPDK principles.
>
> Matan.
>> Thanks,
>> Anatoly
when device appear whenever dpdk is runnning or not, the device will 
default bind to Kernel driver, user or say system admin could use the 
script or tools to rebind a specific driver which according their request
so i think user space tools provide functional and user have the binding 
responsibility rather than the app or PMD. i don't understand why over 
ride to other driver is the scope of an specific PMD. And if there is 
conflict by rules , user could over ride it and take the control.

Per dpdk hotplug, the purpose is for the app failsafe and VM live 
migration,  driverctl is focus driver control and udev is focus on 
device hotplug but no userspace failure handle , there are still not an 
complete hotplug solution for dpdk app,  even the script have binding, 
it look like as a minimal "system admin tools" fit for dpdk, but not 
persistent and not hotplug functional.  so if we aim to provider a total 
hotplug service at dpdk framework, offload these works from application 
and user that is need. In normal user take the driver control by 
tools/script(both ok in my opinion) optional at initial , while in 
hotplug case, failure occur on runtime, dpdk take the control of driver 
from user. compare with app and PMD, do you agree dpdk best to take the 
role on runtime? if only user could take the role, the SRIOV live 
migration of dpdk would hard to be make sense.

i agree let app don't care about any bind/failure handle, in current 
hotplug solution failure handle embedded in dpdk but binding is exposed 
to app, if got agreement to dpdk binding , i could let dpdk self do it. 
if vote result is not let dpdk charge it, i will temporary remove the 
binding function to shrink the hotplug scope. postpone  it to further 
failsafe and live migrate topic.

basically speaking, for binding, script vs tools vs PMD vs dpdk, i vote 
dpdk , at lest run time case.

Re: [dpdk-dev] kernel binding of devices + hotplug

[Matan Azrad]

Hi Guo

From: Guo, Jia, Monday, April 16, 2018 12:26 PM
> To: Matan Azrad <matan@mellanox.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>; Bruce Richardson
> <bruce.richardson@intel.com>; Thomas Monjalon <thomas@monjalon.net>
> Cc: dev@dpdk.org; pmatilai@redhat.com; david.marchand@6wind.com;
> konstantin.ananyev@intel.com; stephen@networkplumber.org;
> fbl@redhat.com
> Subject: Re: kernel binding of devices + hotplug
> 
> hi, all
> 
> 
> On 4/15/2018 4:10 AM, Matan Azrad wrote:
> > Hi all
> >
> > From: Burakov, Anatoly, Friday, April 13, 2018 8:41 PM
> >> To: Bruce Richardson <bruce.richardson@intel.com>; Thomas Monjalon
> >> <thomas@monjalon.net>
> >> Cc: dev@dpdk.org; pmatilai@redhat.com; david.marchand@6wind.com;
> >> jia.guo@intel.com; Matan Azrad <matan@mellanox.com>;
> >> konstantin.ananyev@intel.com; stephen@networkplumber.org;
> >> fbl@redhat.com
> >> Subject: Re: kernel binding of devices + hotplug
> >>
> >> On 13-Apr-18 5:40 PM, Bruce Richardson wrote:
> >>> On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote:
> >>>> It's time to think (again) how we bind devices with kernel modules.
> >>>> We need to decide how we want to manage hotplugged devices with
> >> DPDK.
> >>>> A bit of history first.
> >>>> There was some code in DPDK for bind/unbind, but it has been
> >>>> removed in DPDK 1.7 -
> >>>>
> >>
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdpd
> >>
> k.org%2Fcommit%2F5d8751b83&data=02%7C01%7Cmatan%40mellanox.com
> >> %7C6ea5
> >>
> 5ce994ff4bb0d65208d5a165b417%7Ca652971c7d2e4d9ba6a4d149256f461b%7
> >> C0%7
> >>
> C0%7C636592380565078675&sdata=uLRDAk65hYtJYxjIvY20de377yayCN5DrjCZ
> >> x8H
> >>>> p61o%3D&reserved=0 Copy of the commit message (in 2014):
> >>>> "
> >>>> 	The bind/unbind operations should not be handled by the eal.
> >>>> 	These operations should be either done outside of dpdk or
> >>>> 	inside the PMDs themselves as these are their problems.
> >>>> "
> >>>>
> >>>> The question raised at this time (4 years ago) is still under discussion.
> >>>> Should we manage binding inside or outside DPDK?
> >>>> Should it be controlled in the application or in the OS base?
> >>>>
> >>>> As you know, we use dpdk-devbind.py.
> >>>> This tool lacks two major features:
> >>>> 	- persistent configuration
> >>>> 	- hotplug
> >>>>
> >>>> If we consider that the DPDK applications should be able to apply
> >>>> its own policy to choose the devices to bind, then we need to
> >>>> implement binding in the PMD (with EAL helpers).
> >>>>
> >>>> On the other hand, if we consider that it is the system
> >>>> responsibility, then we could choose systemd/udev and driverctl.
> >>>>
> >>>> The debate is launched!
> >>>>
> >>> Allow me to nail my colours to the mast early! :-)
> >>>
> >>> I believe it's system not application responsibility.
> >>> I also believe I have previously explained my reasons for that
> >>> choice in some of the previous email threads.
> >> For what it's worth, I tend to agree, if only because writing code
> >> for what is essentially a bunch of read/write/filesystem enumeration
> >> in C is extremely fiddly and error prone :) IMO things like this are
> >> better handled either by scripts, or by tools whose sole purpose is doing
> exactly that (or both).
> >>
> >> I like having scripts like devbind in DPDK because we can tailor them
> >> to our use cases better, and having them is amenable to automation,
> >> but while I wouldn't be opposed to removing them altogether in favor
> >> of some external tool (systemd/udev/driverctl/whatever), in my humble
> >> opinion moving them back into EAL or even PMD's would be a mistake.
> >>
> > Since the application runs in the system by a command of the system user I
> think the responsibility is for the user.
> > The DPDK user forwards the control of some devices to the DPDK
> > application using the EAL whitelist\blacklist mode to specify the
> > devices, Any DPDK PMD should know which binding it needs to
> probe\control the device and can apply it, So, if the user asks to control on a
> device by DPDK application it makes sense that the application will do the
> correct binding to the device since the user wants to use it(no need to ask
> more operation of pre binding from the user).
> >
> > Regarding the conflict of system rules for a device, it is again the user
> responsibility, whatever we will decide for the binding procedure of DPDK
> application the user needs to take it into account and to solve such like
> conflicts.
> > One option is to remove any binding rules of a DPDK device in the DPDK
> application initialization and adjust the new rules by the PMDs, then any
> conflict should not disturb the user.
> >
> > In current hot-plug case the application will need to do a lot of work to
> bind\remap devices in plug-in\plug-out events while the PMD could have all
> the knowledge to do it.
> >
> > One more issue with the script is that the user should do different bind per
> device, in case of PMD responsibility the user can forget it:
> > Think about that, any time the user wants to switch\add new supported nic
> it should update the script usage and to do per nic operation contrary to the
> DPDK principles.
> >
> > Matan.
> >> Thanks,
> >> Anatoly
> when device appear whenever dpdk is runnning or not, the device will
> default bind to Kernel driver,

Yes

> user or say system admin could use the script or
> tools to rebind a specific driver which according their request so i think user
> space tools provide functional and user have the binding responsibility rather
> than the app or PMD.

The app\PMD is running by the user, so it is OK(the user just need to know what the app does).

> i don't understand why over ride to other driver is the
> scope of an specific PMD.

Because the PMD can know what binding it needs for using the device.
All the options can be handled by EAL  - the PMD just need to say what is the driver to bind.

> And if there is conflict by rules , user could over
> ride it and take the control.

We can define that all the control of an rte_device is for DPDK.
 
> Per dpdk hotplug, the purpose is for the app failsafe and VM live migration,
> driverctl is focus driver control and udev is focus on device hotplug but no
> userspace failure handle , there are still not an complete hotplug solution for
> dpdk app,  even the script have binding, it look like as a minimal "system
> admin tools" fit for dpdk, but not persistent and not hotplug functional.

I suggested full solution without app involvement:
Plug-out detected by EAL->PMD notification->remap by the PMD->app notification->just stop to use the device.
Plug-in detected by EAL->whitelist\blacklist match->probe->bind by PMD -> app notification->start to use the device.

>  so if
> we aim to provider a total hotplug service at dpdk framework, offload these
> works from application and user that is need. In normal user take the driver
> control by tools/script(both ok in my opinion) optional at initial , while in
> hotplug case, failure occur on runtime, dpdk take the control of driver from
> user. compare with app and PMD, do you agree dpdk best to take the role on
> runtime?

Yes, but by the PMD, not by the app\user.
So, EAL will probe the device if it was matched with the whitelist\blacklist rules and the PMD will do the correct bind in the probe function.
The application just should get notification of the probing and to start to manage the device.

> if only user could take the role, the SRIOV live migration of dpdk
> would hard to be make sense.

Sorry, don't understand you here.

> i agree let app don't care about any bind/failure handle, in current hotplug
> solution failure handle embedded in dpdk but binding is exposed to app, if
> got agreement to dpdk binding , i could let dpdk self do it.
> if vote result is not let dpdk charge it, i will temporary remove the binding
> function to shrink the hotplug scope. postpone  it to further failsafe and live
> migrate topic.
> 
> basically speaking, for binding, script vs tools vs PMD vs dpdk, i vote dpdk , at
> lest run time case.

As I said above - I vote for both EAL\PMDs.

Re: [dpdk-dev] kernel binding of devices + hotplug

[Matan Azrad]

Hi Bruce

From: Bruce Richardson, Monday, April 16, 2018 11:32 AM
> On Sat, Apr 14, 2018 at 08:10:28PM +0000, Matan Azrad wrote:
> > Hi all
> >
> > From: Burakov, Anatoly, Friday, April 13, 2018 8:41 PM
> > > To: Bruce Richardson <bruce.richardson@intel.com>; Thomas Monjalon
> > > <thomas@monjalon.net>
> > > Cc: dev@dpdk.org; pmatilai@redhat.com; david.marchand@6wind.com;
> > > jia.guo@intel.com; Matan Azrad <matan@mellanox.com>;
> > > konstantin.ananyev@intel.com; stephen@networkplumber.org;
> > > fbl@redhat.com
> > > Subject: Re: kernel binding of devices + hotplug
> > >
> > > On 13-Apr-18 5:40 PM, Bruce Richardson wrote:
> > > > On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote:
> > > >> It's time to think (again) how we bind devices with kernel modules.
> > > >> We need to decide how we want to manage hotplugged devices with
> > > DPDK.
> > > >>
> > > >> A bit of history first.
> > > >> There was some code in DPDK for bind/unbind, but it has been
> > > >> removed in DPDK 1.7 -
> > > >>
> > >
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdp
> > > d
> > > >>
> > >
> k.org%2Fcommit%2F5d8751b83&data=02%7C01%7Cmatan%40mellanox.com
> > > %7C6ea5
> > > >>
> > >
> 5ce994ff4bb0d65208d5a165b417%7Ca652971c7d2e4d9ba6a4d149256f461b%7
> > > C0%7
> > > >>
> > >
> C0%7C636592380565078675&sdata=uLRDAk65hYtJYxjIvY20de377yayCN5DrjCZ
> > > x8H
> > > >> p61o%3D&reserved=0 Copy of the commit message (in 2014):
> > > >> "
> > > >> 	The bind/unbind operations should not be handled by the eal.
> > > >> 	These operations should be either done outside of dpdk or
> > > >> 	inside the PMDs themselves as these are their problems.
> > > >> "
> > > >>
> > > >> The question raised at this time (4 years ago) is still under discussion.
> > > >> Should we manage binding inside or outside DPDK?
> > > >> Should it be controlled in the application or in the OS base?
> > > >>
> > > >> As you know, we use dpdk-devbind.py.
> > > >> This tool lacks two major features:
> > > >> 	- persistent configuration
> > > >> 	- hotplug
> > > >>
> > > >> If we consider that the DPDK applications should be able to apply
> > > >> its own policy to choose the devices to bind, then we need to
> > > >> implement binding in the PMD (with EAL helpers).
> > > >>
> > > >> On the other hand, if we consider that it is the system
> > > >> responsibility, then we could choose systemd/udev and driverctl.
> > > >>
> > > >> The debate is launched!
> > > >>
> > > >
> > > > Allow me to nail my colours to the mast early! :-)
> > > >
> > > > I believe it's system not application responsibility.
> > > > I also believe I have previously explained my reasons for that
> > > > choice in some of the previous email threads.
> > >
> > > For what it's worth, I tend to agree, if only because writing code
> > > for what is essentially a bunch of read/write/filesystem enumeration
> > > in C is extremely fiddly and error prone :) IMO things like this are
> > > better handled either by scripts, or by tools whose sole purpose is doing
> exactly that (or both).
> > >
> > > I like having scripts like devbind in DPDK because we can tailor
> > > them to our use cases better, and having them is amenable to
> > > automation, but while I wouldn't be opposed to removing them
> > > altogether in favor of some external tool
> > > (systemd/udev/driverctl/whatever), in my humble opinion moving them
> back into EAL or even PMD's would be a mistake.
> > >
> >
> > Since the application runs in the system by a command of the system user I
> think the responsibility is for the user.
> > The DPDK user forwards the control of some devices to the DPDK
> > application using the EAL whitelist\blacklist mode to specify the
> > devices, Any DPDK PMD should know which binding it needs to
>> probe\control the device and can apply it, So, if the user asks to control on a
>> device by DPDK application it makes sense that the application will do the
>> correct binding to the device since the user wants to use it(no need to ask
>> more operation of pre binding from the user).
> 
> Completely agree that it is ultimately up to the user. However, what I don't
> want to see is the case where the user always has to specify a big long list of
> device whitelist and blacklist options to each run of an application. Instead, if
> device management is done at the system level via udev (for example)
> configured via devicectl, then the application commandline can be vastly
> simplified.

Actually you say that the whitelist\blacklist mechanism is not good enough and the binding workarounds it.
The user need to specify somehow the devices it want to run, 
I think that specifying the device you want by -w option (no need to specify what you don't want in -w case) is really simpler  and more descriptive than binding each device you want by prior process to its correct driver.
 
> It also allows better usability across systems, since the same
> commandline can be used on multiple systems with different hardware, with
> the actual device management rules having been already configured at
> system install/setup time in udev.

But the user still needs to configure the udev per device for each system, I think that command line is better.

> > Regarding the conflict of system rules for a device, it is again the user
> responsibility, whatever we will decide for the binding procedure of DPDK
> application the user needs to take it into account and to solve such like
> conflicts.
> > One option is to remove any binding rules of a DPDK device in the DPDK
> application initialization and adjust the new rules by the PMDs, then any
> conflict should not disturb the user.
> 
> If the device management is only managed in one place, i.e. not in DPDK,
> then there is no conflict to manage.

I can't agree with this statement,
The essence of DPDK is to give a good alternative to managing network devices,
DPDK actually takes a lot of management area to manage by itself to do the user life better :)

Moreover,
Instead of bind script usage and dpdk running , just run dpdk (do all the job in one place).

> > In current hot-plug case the application will need to do a lot of work to
> bind\remap devices in plug-in\plug-out events while the PMD could have all
> the knowledge to do it.
> 
> At the cost of duplicating a lot of code between PMDs.

Why a lot of code? 1 helper in EAL to be used for each relevant PMD.
Each PMD just call to the EAL helper with the bind driver type.
Really simple and immediate.

I think it is better than duplication of user operations.

> > One more issue with the script is that the user should do different bind per
> device, in case of PMD responsibility the user can forget it:
> > Think about that, any time the user wants to switch\add new supported nic
> it should update the script usage and to do per nic operation contrary to the
> DPDK principles.
> >
> 
> The udev rules syntax should provide adequate capabilities here for us to
> match the correct binding behaviour. No need to have it in DPDK too.

We can use it from DPDK.
 
> /Bruce

Re: [dpdk-dev] kernel binding of devices + hotplug

[Stephen Hemminger]

On Mon, 16 Apr 2018 16:11:12 +0000
Matan Azrad <matan@mellanox.com> wrote:

> > If the device management is only managed in one place, i.e. not in DPDK,
> > then there is no conflict to manage.  
> 
> I can't agree with this statement,
> The essence of DPDK is to give a good alternative to managing network devices,
> DPDK actually takes a lot of management area to manage by itself to do the user life better :)

More is not better! DPDK is poorly integrated into Linux overall system. Doing
more in DPDK makes this worse not better.

Buried under this discussion is the fact that the Mellanox bifurcated driver
behaves completely differently from every other driver. This makes coming to
a common solution much harder. The bifurcated model has advantages and disadvantages,
in this case it is a disadvantage since it is not easy to manage usage when
it is a shared resource.

Re: [dpdk-dev] kernel binding of devices + hotplug

[Matan Azrad]

Hi Stephen

From: Stephen Hemminger, Monday, April 16, 2018 7:57 PM
> On Mon, 16 Apr 2018 16:11:12 +0000
> Matan Azrad <matan@mellanox.com> wrote:
> 
> > > If the device management is only managed in one place, i.e. not in
> > > DPDK, then there is no conflict to manage.
> >
> > I can't agree with this statement,
> > The essence of DPDK is to give a good alternative to managing network
> > devices, DPDK actually takes a lot of management area to manage by
> > itself to do the user life better :)
> 
> More is not better! DPDK is poorly integrated into Linux overall system. Doing
> more in DPDK makes this worse not better.

In this case I think that yes, more is better.
Please explain why do you think that in this case it is worse.
 
> Buried under this discussion is the fact that the Mellanox bifurcated driver
> behaves completely differently from every other driver. This makes coming
> to a common solution much harder. The bifurcated model has advantages
> and disadvantages, in this case it is a disadvantage since it is not easy to
> manage usage when it is a shared resource.

Sorry, what is your point?

Re: [dpdk-dev] kernel binding of devices + hotplug

[Stephen Hemminger]

On Mon, 16 Apr 2018 17:10:09 +0000
Matan Azrad <matan@mellanox.com> wrote:

> Hi Stephen
> 
> From: Stephen Hemminger, Monday, April 16, 2018 7:57 PM
> > On Mon, 16 Apr 2018 16:11:12 +0000
> > Matan Azrad <matan@mellanox.com> wrote:
> >   
> > > > If the device management is only managed in one place, i.e. not in
> > > > DPDK, then there is no conflict to manage.  
> > >
> > > I can't agree with this statement,
> > > The essence of DPDK is to give a good alternative to managing network
> > > devices, DPDK actually takes a lot of management area to manage by
> > > itself to do the user life better :)  
> > 
> > More is not better! DPDK is poorly integrated into Linux overall system. Doing
> > more in DPDK makes this worse not better.  
> 
> In this case I think that yes, more is better.
> Please explain why do you think that in this case it is worse.

DPDK should work with udev, not try and replace functionality that is
already done by udev and systemd. Having a parallel and different model
makes life harder for users.

>  
> > Buried under this discussion is the fact that the Mellanox bifurcated driver
> > behaves completely differently from every other driver. This makes coming
> > to a common solution much harder. The bifurcated model has advantages
> > and disadvantages, in this case it is a disadvantage since it is not easy to
> > manage usage when it is a shared resource.  
> 
> Sorry, what is your point?

The bifurcated model does not play well with driverctl. It just works differently
than other drivers. The bifurcated model works better for simple case of shared
device on bare metal; but it is harder for the transparent bonding model
used on Azure. The eth device is not really available for direct use in kernel;
and there is discussion in netdev about hiding it as well which will break
more things here.

Re: [dpdk-dev] kernel binding of devices + hotplug

[Matan Azrad]

Hi Stephen

From: Stephen Hemminger, Monday, April 16, 2018 8:19 PM
> On Mon, 16 Apr 2018 17:10:09 +0000
> Matan Azrad <matan@mellanox.com> wrote:
> 
> > Hi Stephen
> >
> > From: Stephen Hemminger, Monday, April 16, 2018 7:57 PM
> > > On Mon, 16 Apr 2018 16:11:12 +0000
> > > Matan Azrad <matan@mellanox.com> wrote:
> > >
> > > > > If the device management is only managed in one place, i.e. not
> > > > > in DPDK, then there is no conflict to manage.
> > > >
> > > > I can't agree with this statement, The essence of DPDK is to give
> > > > a good alternative to managing network devices, DPDK actually
> > > > takes a lot of management area to manage by itself to do the user
> > > > life better :)
> > >
> > > More is not better! DPDK is poorly integrated into Linux overall
> > > system. Doing more in DPDK makes this worse not better.
> >
> > In this case I think that yes, more is better.
> > Please explain why do you think that in this case it is worse.
> 
> DPDK should work with udev, not try and replace functionality that is already
> done by udev and systemd. Having a parallel and different model makes life
> harder for users.

This is the same model of the user:
The user knows what DPDK does regarding to binding so he just takes it into account. The same as he knows that the device is used by the DPDK and take it into account.


> 
> >
> > > Buried under this discussion is the fact that the Mellanox
> > > bifurcated driver behaves completely differently from every other
> > > driver. This makes coming to a common solution much harder. The
> > > bifurcated model has advantages and disadvantages, in this case it
> > > is a disadvantage since it is not easy to manage usage when it is a shared
> resource.
> >
> > Sorry, what is your point?
> 
> The bifurcated model does not play well with driverctl. It just works
> differently than other drivers. The bifurcated model works better for simple
> case of shared device on bare metal; but it is harder for the transparent
> bonding model used on Azure. The eth device is not really available for direct
> use in kernel; and there is discussion in netdev about hiding it as well which
> will break more things here.

And? How it is relevant to the binding discussion? 

Re: [dpdk-dev] kernel binding of devices + hotplug

[Thomas Monjalon]

16/04/2018 18:57, Stephen Hemminger:
> Buried under this discussion is the fact that the Mellanox bifurcated driver
> behaves completely differently from every other driver. This makes coming to
> a common solution much harder. The bifurcated model has advantages and disadvantages,
> in this case it is a disadvantage since it is not easy to manage usage when
> it is a shared resource.

The bifurcated model can apply to more cases than Mellanox.
For instance, the AF_XDP PMD will be bifurcated.
In the case of AF_XDP, you don't need to bind a kernel module to the device.

We have a good idea of how to program the right binding inside DPDK:
depending on how the device is configured by the application or from
the command line options, we can guess which binding is expected.
We need to compare it with how we could integrate the same
"transparent binding" with udev/driverctl.

The system tools are nice for persistent and global configuration.
The other requirement is to be able to run a one-shot DPDK application
without changing the persistent system configuration.
We may need to hook udev/driverctl live to make it happen.

Re: [dpdk-dev] kernel binding of devices + hotplug

[Ananyev, Konstantin]

Hi everyone,
Few comments from me below.
In summary I am also in favor to keep binding decision at system level (outside DPDK).
Konstantin

> -----Original Message-----
> From: Matan Azrad [mailto:matan@mellanox.com]
> Sent: Monday, April 16, 2018 5:11 PM
> To: Richardson, Bruce <bruce.richardson@intel.com>
> Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; Thomas Monjalon <thomas@monjalon.net>; dev@dpdk.org; pmatilai@redhat.com;
> david.marchand@6wind.com; Guo, Jia <jia.guo@intel.com>; Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> stephen@networkplumber.org; fbl@redhat.com
> Subject: RE: kernel binding of devices + hotplug
> 
> Hi Bruce
> 
> From: Bruce Richardson, Monday, April 16, 2018 11:32 AM
> > On Sat, Apr 14, 2018 at 08:10:28PM +0000, Matan Azrad wrote:
> > > Hi all
> > >
> > > From: Burakov, Anatoly, Friday, April 13, 2018 8:41 PM
> > > > To: Bruce Richardson <bruce.richardson@intel.com>; Thomas Monjalon
> > > > <thomas@monjalon.net>
> > > > Cc: dev@dpdk.org; pmatilai@redhat.com; david.marchand@6wind.com;
> > > > jia.guo@intel.com; Matan Azrad <matan@mellanox.com>;
> > > > konstantin.ananyev@intel.com; stephen@networkplumber.org;
> > > > fbl@redhat.com
> > > > Subject: Re: kernel binding of devices + hotplug
> > > >
> > > > On 13-Apr-18 5:40 PM, Bruce Richardson wrote:
> > > > > On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote:
> > > > >> It's time to think (again) how we bind devices with kernel modules.
> > > > >> We need to decide how we want to manage hotplugged devices with
> > > > DPDK.
> > > > >>
> > > > >> A bit of history first.
> > > > >> There was some code in DPDK for bind/unbind, but it has been
> > > > >> removed in DPDK 1.7 -
> > > > >>
> > > >
> > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdp
> > > > d
> > > > >>
> > > >
> > k.org%2Fcommit%2F5d8751b83&data=02%7C01%7Cmatan%40mellanox.com
> > > > %7C6ea5
> > > > >>
> > > >
> > 5ce994ff4bb0d65208d5a165b417%7Ca652971c7d2e4d9ba6a4d149256f461b%7
> > > > C0%7
> > > > >>
> > > >
> > C0%7C636592380565078675&sdata=uLRDAk65hYtJYxjIvY20de377yayCN5DrjCZ
> > > > x8H
> > > > >> p61o%3D&reserved=0 Copy of the commit message (in 2014):
> > > > >> "
> > > > >> 	The bind/unbind operations should not be handled by the eal.
> > > > >> 	These operations should be either done outside of dpdk or
> > > > >> 	inside the PMDs themselves as these are their problems.
> > > > >> "
> > > > >>
> > > > >> The question raised at this time (4 years ago) is still under discussion.
> > > > >> Should we manage binding inside or outside DPDK?
> > > > >> Should it be controlled in the application or in the OS base?
> > > > >>
> > > > >> As you know, we use dpdk-devbind.py.
> > > > >> This tool lacks two major features:
> > > > >> 	- persistent configuration
> > > > >> 	- hotplug
> > > > >>
> > > > >> If we consider that the DPDK applications should be able to apply
> > > > >> its own policy to choose the devices to bind, then we need to
> > > > >> implement binding in the PMD (with EAL helpers).
> > > > >>
> > > > >> On the other hand, if we consider that it is the system
> > > > >> responsibility, then we could choose systemd/udev and driverctl.
> > > > >>
> > > > >> The debate is launched!
> > > > >>
> > > > >
> > > > > Allow me to nail my colours to the mast early! :-)
> > > > >
> > > > > I believe it's system not application responsibility.
> > > > > I also believe I have previously explained my reasons for that
> > > > > choice in some of the previous email threads.
> > > >
> > > > For what it's worth, I tend to agree, if only because writing code
> > > > for what is essentially a bunch of read/write/filesystem enumeration
> > > > in C is extremely fiddly and error prone :) IMO things like this are
> > > > better handled either by scripts, or by tools whose sole purpose is doing
> > exactly that (or both).
> > > >
> > > > I like having scripts like devbind in DPDK because we can tailor
> > > > them to our use cases better, and having them is amenable to
> > > > automation, but while I wouldn't be opposed to removing them
> > > > altogether in favor of some external tool
> > > > (systemd/udev/driverctl/whatever), in my humble opinion moving them
> > back into EAL or even PMD's would be a mistake.
> > > >
> > >
> > > Since the application runs in the system by a command of the system user I
> > think the responsibility is for the user.
> > > The DPDK user forwards the control of some devices to the DPDK
> > > application using the EAL whitelist\blacklist mode to specify the
> > > devices, Any DPDK PMD should know which binding it needs to
> >> probe\control the device and can apply it, So, if the user asks to control on a
> >> device by DPDK application it makes sense that the application will do the
> >> correct binding to the device since the user wants to use it(no need to ask
> >> more operation of pre binding from the user).
> >
> > Completely agree that it is ultimately up to the user. However, what I don't
> > want to see is the case where the user always has to specify a big long list of
> > device whitelist and blacklist options to each run of an application. Instead, if
> > device management is done at the system level via udev (for example)
> > configured via devicectl, then the application commandline can be vastly
> > simplified.
> 
> Actually you say that the whitelist\blacklist mechanism is not good enough and the binding workarounds it.
> The user need to specify somehow the devices it want to run,
> I think that specifying the device you want by -w option (no need to specify what you don't want in -w case) is really simpler  and more
> descriptive than binding each device you want by prior process to its correct driver.

But what if user changes his mind and decides to give particular device back to the kernel?
Should he restart the dpdk application? In some cases it might be not desirable.

> 
> > It also allows better usability across systems, since the same
> > commandline can be used on multiple systems with different hardware, with
> > the actual device management rules having been already configured at
> > system install/setup time in udev.
> 
> But the user still needs to configure the udev per device for each system, I think that command line is better.
> 
> > > Regarding the conflict of system rules for a device, it is again the user
> > responsibility, whatever we will decide for the binding procedure of DPDK
> > application the user needs to take it into account and to solve such like
> > conflicts.
> > > One option is to remove any binding rules of a DPDK device in the DPDK
> > application initialization and adjust the new rules by the PMDs, then any
> > conflict should not disturb the user.
> >
> > If the device management is only managed in one place, i.e. not in DPDK,
> > then there is no conflict to manage.
> 
> I can't agree with this statement,
> The essence of DPDK is to give a good alternative to managing network devices,
> DPDK actually takes a lot of management area to manage by itself to do the user life better :)

>From my point - it is not about managing particular device.
It is about making decision who (kenel/dpdk) will manage that device.
>From usability perspective it seems to me that better to keep it in one place for all devices.
So udev (or any other sysadmin tool) seems like a right choice here. 

> 
> Moreover,
> Instead of bind script usage and dpdk running , just run dpdk (do all the job in one place).
> 
> > > In current hot-plug case the application will need to do a lot of work to
> > bind\remap devices in plug-in\plug-out events while the PMD could have all
> > the knowledge to do it.
> >
> > At the cost of duplicating a lot of code between PMDs.
> 
> Why a lot of code? 1 helper in EAL to be used for each relevant PMD.
> Each PMD just call to the EAL helper with the bind driver type.
> Really simple and immediate.
> 
> I think it is better than duplication of user operations.
> 
> > > One more issue with the script is that the user should do different bind per
> > device, in case of PMD responsibility the user can forget it:
> > > Think about that, any time the user wants to switch\add new supported nic
> > it should update the script usage and to do per nic operation contrary to the
> > DPDK principles.
> > >
> >
> > The udev rules syntax should provide adequate capabilities here for us to
> > match the correct binding behaviour. No need to have it in DPDK too.
> 
> We can use it from DPDK.
> 
> > /Bruce

Re: [dpdk-dev] kernel binding of devices + hotplug

[Matan Azrad]

Hi Konstantin

From: Ananyev, Konstantin, Tuesday, April 17, 2018 12:23 PM
> To: Matan Azrad <matan@mellanox.com>; Richardson, Bruce
> <bruce.richardson@intel.com>
> Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; Thomas Monjalon
> <thomas@monjalon.net>; dev@dpdk.org; pmatilai@redhat.com;
> david.marchand@6wind.com; Guo, Jia <jia.guo@intel.com>;
> stephen@networkplumber.org; fbl@redhat.com
> Subject: RE: kernel binding of devices + hotplug
> 
> 
> Hi everyone,
> Few comments from me below.
> In summary I am also in favor to keep binding decision at system level
> (outside DPDK).
> Konstantin
> 
> > -----Original Message-----
> > From: Matan Azrad [mailto:matan@mellanox.com]
> > Sent: Monday, April 16, 2018 5:11 PM
> > To: Richardson, Bruce <bruce.richardson@intel.com>
> > Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; Thomas Monjalon
> > <thomas@monjalon.net>; dev@dpdk.org; pmatilai@redhat.com;
> > david.marchand@6wind.com; Guo, Jia <jia.guo@intel.com>; Ananyev,
> > Konstantin <konstantin.ananyev@intel.com>;
> stephen@networkplumber.org;
> > fbl@redhat.com
> > Subject: RE: kernel binding of devices + hotplug
> >
> > Hi Bruce
> >
> > From: Bruce Richardson, Monday, April 16, 2018 11:32 AM
> > > On Sat, Apr 14, 2018 at 08:10:28PM +0000, Matan Azrad wrote:
> > > > Hi all
> > > >
> > > > From: Burakov, Anatoly, Friday, April 13, 2018 8:41 PM
> > > > > To: Bruce Richardson <bruce.richardson@intel.com>; Thomas
> > > > > Monjalon <thomas@monjalon.net>
> > > > > Cc: dev@dpdk.org; pmatilai@redhat.com;
> david.marchand@6wind.com;
> > > > > jia.guo@intel.com; Matan Azrad <matan@mellanox.com>;
> > > > > konstantin.ananyev@intel.com; stephen@networkplumber.org;
> > > > > fbl@redhat.com
> > > > > Subject: Re: kernel binding of devices + hotplug
> > > > >
> > > > > On 13-Apr-18 5:40 PM, Bruce Richardson wrote:
> > > > > > On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote:
> > > > > >> It's time to think (again) how we bind devices with kernel modules.
> > > > > >> We need to decide how we want to manage hotplugged devices
> > > > > >> with
> > > > > DPDK.
> > > > > >>
> > > > > >> A bit of history first.
> > > > > >> There was some code in DPDK for bind/unbind, but it has been
> > > > > >> removed in DPDK 1.7 -
> > > > > >>
> > > > >
> > >
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdp
> > > > > d
> > > > > >>
> > > > >
> > >
> k.org%2Fcommit%2F5d8751b83&data=02%7C01%7Cmatan%40mellanox.com
> > > > > %7C6ea5
> > > > > >>
> > > > >
> > >
> 5ce994ff4bb0d65208d5a165b417%7Ca652971c7d2e4d9ba6a4d149256f461b%7
> > > > > C0%7
> > > > > >>
> > > > >
> > >
> C0%7C636592380565078675&sdata=uLRDAk65hYtJYxjIvY20de377yayCN5DrjCZ
> > > > > x8H
> > > > > >> p61o%3D&reserved=0 Copy of the commit message (in 2014):
> > > > > >> "
> > > > > >> 	The bind/unbind operations should not be handled by the
> eal.
> > > > > >> 	These operations should be either done outside of dpdk or
> > > > > >> 	inside the PMDs themselves as these are their problems.
> > > > > >> "
> > > > > >>
> > > > > >> The question raised at this time (4 years ago) is still under
> discussion.
> > > > > >> Should we manage binding inside or outside DPDK?
> > > > > >> Should it be controlled in the application or in the OS base?
> > > > > >>
> > > > > >> As you know, we use dpdk-devbind.py.
> > > > > >> This tool lacks two major features:
> > > > > >> 	- persistent configuration
> > > > > >> 	- hotplug
> > > > > >>
> > > > > >> If we consider that the DPDK applications should be able to
> > > > > >> apply its own policy to choose the devices to bind, then we
> > > > > >> need to implement binding in the PMD (with EAL helpers).
> > > > > >>
> > > > > >> On the other hand, if we consider that it is the system
> > > > > >> responsibility, then we could choose systemd/udev and driverctl.
> > > > > >>
> > > > > >> The debate is launched!
> > > > > >>
> > > > > >
> > > > > > Allow me to nail my colours to the mast early! :-)
> > > > > >
> > > > > > I believe it's system not application responsibility.
> > > > > > I also believe I have previously explained my reasons for that
> > > > > > choice in some of the previous email threads.
> > > > >
> > > > > For what it's worth, I tend to agree, if only because writing
> > > > > code for what is essentially a bunch of read/write/filesystem
> > > > > enumeration in C is extremely fiddly and error prone :) IMO
> > > > > things like this are better handled either by scripts, or by
> > > > > tools whose sole purpose is doing
> > > exactly that (or both).
> > > > >
> > > > > I like having scripts like devbind in DPDK because we can tailor
> > > > > them to our use cases better, and having them is amenable to
> > > > > automation, but while I wouldn't be opposed to removing them
> > > > > altogether in favor of some external tool
> > > > > (systemd/udev/driverctl/whatever), in my humble opinion moving
> > > > > them
> > > back into EAL or even PMD's would be a mistake.
> > > > >
> > > >
> > > > Since the application runs in the system by a command of the
> > > > system user I
> > > think the responsibility is for the user.
> > > > The DPDK user forwards the control of some devices to the DPDK
> > > > application using the EAL whitelist\blacklist mode to specify the
> > > > devices, Any DPDK PMD should know which binding it needs to
> > >> probe\control the device and can apply it, So, if the user asks to
> > >> control on a device by DPDK application it makes sense that the
> > >> application will do the correct binding to the device since the
> > >> user wants to use it(no need to ask more operation of pre binding from
> the user).
> > >
> > > Completely agree that it is ultimately up to the user. However, what
> > > I don't want to see is the case where the user always has to specify
> > > a big long list of device whitelist and blacklist options to each
> > > run of an application. Instead, if device management is done at the
> > > system level via udev (for example) configured via devicectl, then
> > > the application commandline can be vastly simplified.
> >
> > Actually you say that the whitelist\blacklist mechanism is not good enough
> and the binding workarounds it.
> > The user need to specify somehow the devices it want to run, I think
> > that specifying the device you want by -w option (no need to specify
> > what you don't want in -w case) is really simpler  and more descriptive than
> binding each device you want by prior process to its correct driver.
> 
> But what if user changes his mind and decides to give particular device back
> to the kernel?
> Should he restart the dpdk application? In some cases it might be not
> desirable.

And what is the behavior now in this case?
Looks like if we want to solve it we need to add mechanism to stop these particular devices DPDK management in any case.
 
> > > It also allows better usability across systems, since the same
> > > commandline can be used on multiple systems with different hardware,
> > > with the actual device management rules having been already
> > > configured at system install/setup time in udev.
> >
> > But the user still needs to configure the udev per device for each system, I
> think that command line is better.
> >
> > > > Regarding the conflict of system rules for a device, it is again
> > > > the user
> > > responsibility, whatever we will decide for the binding procedure of
> > > DPDK application the user needs to take it into account and to solve
> > > such like conflicts.
> > > > One option is to remove any binding rules of a DPDK device in the
> > > > DPDK
> > > application initialization and adjust the new rules by the PMDs,
> > > then any conflict should not disturb the user.
> > >
> > > If the device management is only managed in one place, i.e. not in
> > > DPDK, then there is no conflict to manage.
> >
> > I can't agree with this statement,
> > The essence of DPDK is to give a good alternative to managing network
> > devices, DPDK actually takes a lot of management area to manage by
> > itself to do the user life better :)
> 
> From my point - it is not about managing particular device.
> It is about making decision who (kenel/dpdk) will manage that device.

Doesn't the w\b mechanism comes to solve it?

> From usability perspective it seems to me that better to keep it in one place
> for all devices.
> So udev (or any other sysadmin tool) seems like a right choice here.

Please explain why?
And if you are talking about 1 place:
Why not to do the binding in the same place where we define which device to run?
Why not to ask from the user to specify the device and that's it?

> > Moreover,
> > Instead of bind script usage and dpdk running , just run dpdk (do all the job
> in one place).
> >
> > > > In current hot-plug case the application will need to do a lot of
> > > > work to
> > > bind\remap devices in plug-in\plug-out events while the PMD could
> > > have all the knowledge to do it.
> > >
> > > At the cost of duplicating a lot of code between PMDs.
> >
> > Why a lot of code? 1 helper in EAL to be used for each relevant PMD.
> > Each PMD just call to the EAL helper with the bind driver type.
> > Really simple and immediate.
> >
> > I think it is better than duplication of user operations.
> >

And what's about hot-plug? Who should do the bind in plug-in event?
 
> > > > One more issue with the script is that the user should do
> > > > different bind per
> > > device, in case of PMD responsibility the user can forget it:
> > > > Think about that, any time the user wants to switch\add new
> > > > supported nic
> > > it should update the script usage and to do per nic operation
> > > contrary to the DPDK principles.
> > > >
> > >
> > > The udev rules syntax should provide adequate capabilities here for
> > > us to match the correct binding behaviour. No need to have it in DPDK
> too.
> >
> > We can use it from DPDK.
> >
> > > /Bruce

Re: [dpdk-dev] kernel binding of devices + hotplug

[Ananyev, Konstantin]

Hi Matan,

> 
> 
> Hi Konstantin
> 
> From: Ananyev, Konstantin, Tuesday, April 17, 2018 12:23 PM
> > To: Matan Azrad <matan@mellanox.com>; Richardson, Bruce
> > <bruce.richardson@intel.com>
> > Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; Thomas Monjalon
> > <thomas@monjalon.net>; dev@dpdk.org; pmatilai@redhat.com;
> > david.marchand@6wind.com; Guo, Jia <jia.guo@intel.com>;
> > stephen@networkplumber.org; fbl@redhat.com
> > Subject: RE: kernel binding of devices + hotplug
> >
> >
> > Hi everyone,
> > Few comments from me below.
> > In summary I am also in favor to keep binding decision at system level
> > (outside DPDK).
> > Konstantin
> >
> > > -----Original Message-----
> > > From: Matan Azrad [mailto:matan@mellanox.com]
> > > Sent: Monday, April 16, 2018 5:11 PM
> > > To: Richardson, Bruce <bruce.richardson@intel.com>
> > > Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; Thomas Monjalon
> > > <thomas@monjalon.net>; dev@dpdk.org; pmatilai@redhat.com;
> > > david.marchand@6wind.com; Guo, Jia <jia.guo@intel.com>; Ananyev,
> > > Konstantin <konstantin.ananyev@intel.com>;
> > stephen@networkplumber.org;
> > > fbl@redhat.com
> > > Subject: RE: kernel binding of devices + hotplug
> > >
> > > Hi Bruce
> > >
> > > From: Bruce Richardson, Monday, April 16, 2018 11:32 AM
> > > > On Sat, Apr 14, 2018 at 08:10:28PM +0000, Matan Azrad wrote:
> > > > > Hi all
> > > > >
> > > > > From: Burakov, Anatoly, Friday, April 13, 2018 8:41 PM
> > > > > > To: Bruce Richardson <bruce.richardson@intel.com>; Thomas
> > > > > > Monjalon <thomas@monjalon.net>
> > > > > > Cc: dev@dpdk.org; pmatilai@redhat.com;
> > david.marchand@6wind.com;
> > > > > > jia.guo@intel.com; Matan Azrad <matan@mellanox.com>;
> > > > > > konstantin.ananyev@intel.com; stephen@networkplumber.org;
> > > > > > fbl@redhat.com
> > > > > > Subject: Re: kernel binding of devices + hotplug
> > > > > >
> > > > > > On 13-Apr-18 5:40 PM, Bruce Richardson wrote:
> > > > > > > On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote:
> > > > > > >> It's time to think (again) how we bind devices with kernel modules.
> > > > > > >> We need to decide how we want to manage hotplugged devices
> > > > > > >> with
> > > > > > DPDK.
> > > > > > >>
> > > > > > >> A bit of history first.
> > > > > > >> There was some code in DPDK for bind/unbind, but it has been
> > > > > > >> removed in DPDK 1.7 -
> > > > > > >>
> > > > > >
> > > >
> > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdp
> > > > > > d
> > > > > > >>
> > > > > >
> > > >
> > k.org%2Fcommit%2F5d8751b83&data=02%7C01%7Cmatan%40mellanox.com
> > > > > > %7C6ea5
> > > > > > >>
> > > > > >
> > > >
> > 5ce994ff4bb0d65208d5a165b417%7Ca652971c7d2e4d9ba6a4d149256f461b%7
> > > > > > C0%7
> > > > > > >>
> > > > > >
> > > >
> > C0%7C636592380565078675&sdata=uLRDAk65hYtJYxjIvY20de377yayCN5DrjCZ
> > > > > > x8H
> > > > > > >> p61o%3D&reserved=0 Copy of the commit message (in 2014):
> > > > > > >> "
> > > > > > >> 	The bind/unbind operations should not be handled by the
> > eal.
> > > > > > >> 	These operations should be either done outside of dpdk or
> > > > > > >> 	inside the PMDs themselves as these are their problems.
> > > > > > >> "
> > > > > > >>
> > > > > > >> The question raised at this time (4 years ago) is still under
> > discussion.
> > > > > > >> Should we manage binding inside or outside DPDK?
> > > > > > >> Should it be controlled in the application or in the OS base?
> > > > > > >>
> > > > > > >> As you know, we use dpdk-devbind.py.
> > > > > > >> This tool lacks two major features:
> > > > > > >> 	- persistent configuration
> > > > > > >> 	- hotplug
> > > > > > >>
> > > > > > >> If we consider that the DPDK applications should be able to
> > > > > > >> apply its own policy to choose the devices to bind, then we
> > > > > > >> need to implement binding in the PMD (with EAL helpers).
> > > > > > >>
> > > > > > >> On the other hand, if we consider that it is the system
> > > > > > >> responsibility, then we could choose systemd/udev and driverctl.
> > > > > > >>
> > > > > > >> The debate is launched!
> > > > > > >>
> > > > > > >
> > > > > > > Allow me to nail my colours to the mast early! :-)
> > > > > > >
> > > > > > > I believe it's system not application responsibility.
> > > > > > > I also believe I have previously explained my reasons for that
> > > > > > > choice in some of the previous email threads.
> > > > > >
> > > > > > For what it's worth, I tend to agree, if only because writing
> > > > > > code for what is essentially a bunch of read/write/filesystem
> > > > > > enumeration in C is extremely fiddly and error prone :) IMO
> > > > > > things like this are better handled either by scripts, or by
> > > > > > tools whose sole purpose is doing
> > > > exactly that (or both).
> > > > > >
> > > > > > I like having scripts like devbind in DPDK because we can tailor
> > > > > > them to our use cases better, and having them is amenable to
> > > > > > automation, but while I wouldn't be opposed to removing them
> > > > > > altogether in favor of some external tool
> > > > > > (systemd/udev/driverctl/whatever), in my humble opinion moving
> > > > > > them
> > > > back into EAL or even PMD's would be a mistake.
> > > > > >
> > > > >
> > > > > Since the application runs in the system by a command of the
> > > > > system user I
> > > > think the responsibility is for the user.
> > > > > The DPDK user forwards the control of some devices to the DPDK
> > > > > application using the EAL whitelist\blacklist mode to specify the
> > > > > devices, Any DPDK PMD should know which binding it needs to
> > > >> probe\control the device and can apply it, So, if the user asks to
> > > >> control on a device by DPDK application it makes sense that the
> > > >> application will do the correct binding to the device since the
> > > >> user wants to use it(no need to ask more operation of pre binding from
> > the user).
> > > >
> > > > Completely agree that it is ultimately up to the user. However, what
> > > > I don't want to see is the case where the user always has to specify
> > > > a big long list of device whitelist and blacklist options to each
> > > > run of an application. Instead, if device management is done at the
> > > > system level via udev (for example) configured via devicectl, then
> > > > the application commandline can be vastly simplified.
> > >
> > > Actually you say that the whitelist\blacklist mechanism is not good enough
> > and the binding workarounds it.
> > > The user need to specify somehow the devices it want to run, I think
> > > that specifying the device you want by -w option (no need to specify
> > > what you don't want in -w case) is really simpler  and more descriptive than
> > binding each device you want by prior process to its correct driver.
> >
> > But what if user changes his mind and decides to give particular device back
> > to the kernel?
> > Should he restart the dpdk application? In some cases it might be not
> > desirable.
> 
> And what is the behavior now in this case?

Now we don't have hotplug support at all, so nothing to worry about, I guess :)

> Looks like if we want to solve it we need to add mechanism to stop these particular devices DPDK management in any case.

I am not talking about managing a device itself (start/stop/attach/detach).
Let say you start dpdk with '-w dev -w dev2'.
dev1 is active, traffic is going through it, etc., while dev2 is unplugged.
Now user is about to plug dev2, though just before that he changed his mind and decided to give
that device to kernel (or different dpdk app), ideally without re-starting the app.
With just command-line support there is no option to do that.
Same for opposite case - you started dpdk app with '-w dev1' and then decided that you 
also want that app to manage hot-pluggable dev2 too. 

> 
> > > > It also allows better usability across systems, since the same
> > > > commandline can be used on multiple systems with different hardware,
> > > > with the actual device management rules having been already
> > > > configured at system install/setup time in udev.
> > >
> > > But the user still needs to configure the udev per device for each system, I
> > think that command line is better.
> > >
> > > > > Regarding the conflict of system rules for a device, it is again
> > > > > the user
> > > > responsibility, whatever we will decide for the binding procedure of
> > > > DPDK application the user needs to take it into account and to solve
> > > > such like conflicts.
> > > > > One option is to remove any binding rules of a DPDK device in the
> > > > > DPDK
> > > > application initialization and adjust the new rules by the PMDs,
> > > > then any conflict should not disturb the user.
> > > >
> > > > If the device management is only managed in one place, i.e. not in
> > > > DPDK, then there is no conflict to manage.
> > >
> > > I can't agree with this statement,
> > > The essence of DPDK is to give a good alternative to managing network
> > > devices, DPDK actually takes a lot of management area to manage by
> > > itself to do the user life better :)
> >
> > From my point - it is not about managing particular device.
> > It is about making decision who (kenel/dpdk) will manage that device.
> 
> Doesn't the w\b mechanism comes to solve it?

It does till some extent, but I don't think it is the best possible way.

> 
> > From usability perspective it seems to me that better to keep it in one place
> > for all devices.
> > So udev (or any other sysadmin tool) seems like a right choice here.
> 
> Please explain why?
> And if you are talking about 1 place:
> Why not to do the binding in the same place where we define which device to run?

Not all devices are always managed by dpdk apps.
Some of them will still be managed by kernel.
Again there could be several independent dpdk apps running on the same system.
Obviously dpdk app itself can't be a centric place to handle all that varieties.

Again - if there exists a system tool that provides that functionality - why not to use it?

> Why not to ask from the user to specify the device and that's it?
> 
> > > Moreover,
> > > Instead of bind script usage and dpdk running , just run dpdk (do all the job
> > in one place).
> > >
> > > > > In current hot-plug case the application will need to do a lot of
> > > > > work to
> > > > bind\remap devices in plug-in\plug-out events while the PMD could
> > > > have all the knowledge to do it.
> > > >
> > > > At the cost of duplicating a lot of code between PMDs.
> > >
> > > Why a lot of code? 1 helper in EAL to be used for each relevant PMD.
> > > Each PMD just call to the EAL helper with the bind driver type.
> > > Really simple and immediate.
> > >
> > > I think it is better than duplication of user operations.
> > >
> 
> And what's about hot-plug? Who should do the bind in plug-in event?
> 
> > > > > One more issue with the script is that the user should do
> > > > > different bind per
> > > > device, in case of PMD responsibility the user can forget it:
> > > > > Think about that, any time the user wants to switch\add new
> > > > > supported nic
> > > > it should update the script usage and to do per nic operation
> > > > contrary to the DPDK principles.
> > > > >
> > > >
> > > > The udev rules syntax should provide adequate capabilities here for
> > > > us to match the correct binding behaviour. No need to have it in DPDK
> > too.
> > >
> > > We can use it from DPDK.
> > >
> > > > /Bruce

Re: [dpdk-dev] kernel binding of devices + hotplug

[Flavio Leitner]

On Sun, Apr 15, 2018 at 01:48:36AM +0000, Stephen Hemminger wrote:
> My vote is to work with udev and not try to replace it.
> 
> Driverctl works well. Just not for bifurcated driver

I second that.  We also have other system configs to care about like
kernel parameters and hugepage configuration which I think follow the
same idea that they are system wide configs and should not be managed
by DPDK itself.

fbl


> 
> On Fri, Apr 13, 2018, 9:31 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> 
> > It's time to think (again) how we bind devices with kernel modules.
> > We need to decide how we want to manage hotplugged devices with DPDK.
> >
> > A bit of history first.
> > There was some code in DPDK for bind/unbind, but it has been removed
> > in DPDK 1.7 - http://dpdk.org/commit/5d8751b83
> > Copy of the commit message (in 2014):
> > "
> >         The bind/unbind operations should not be handled by the eal.
> >         These operations should be either done outside of dpdk or
> >         inside the PMDs themselves as these are their problems.
> > "
> >
> > The question raised at this time (4 years ago) is still under discussion.
> > Should we manage binding inside or outside DPDK?
> > Should it be controlled in the application or in the OS base?
> >
> > As you know, we use dpdk-devbind.py.
> > This tool lacks two major features:
> >         - persistent configuration
> >         - hotplug
> >
> > If we consider that the DPDK applications should be able to apply its own
> > policy to choose the devices to bind, then we need to implement binding
> > in the PMD (with EAL helpers).
> >
> > On the other hand, if we consider that it is the system responsibility,
> > then we could choose systemd/udev and driverctl.
> >
> > The debate is launched!
> >
> > Please find more details in the references below.
> >
> > Announce of driverctl:
> >         http://dpdk.org/ml/archives/dev/2015-December/029500.html
> > Repository of driverctl:
> >         https://gitlab.com/driverctl/driverctl
> >
> > Discussion about binding script and driverctl:
> >         http://dpdk.org/ml/archives/dev/2018-April/095687.html
> >
> > Patch to implement binding in DPDK (for hotplug):
> >         http://dpdk.org/ml/archives/dev/2018-April/095714.html
> >
> > Discussion in the same hotplug series:
> >         http://dpdk.org/ml/archives/dev/2018-April/097058.html
> >
> >
> >
> >

-- 
Flavio

Re: [dpdk-dev] kernel binding of devices + hotplug

[Stephen Hemminger]

On Wed, 18 Apr 2018 11:11:01 -0300
Flavio Leitner <fbl@redhat.com> wrote:

> On Sun, Apr 15, 2018 at 01:48:36AM +0000, Stephen Hemminger wrote:
> > My vote is to work with udev and not try to replace it.
> > 
> > Driverctl works well. Just not for bifurcated driver  
> 
> I second that.  We also have other system configs to care about like
> kernel parameters and hugepage configuration which I think follow the
> same idea that they are system wide configs and should not be managed
> by DPDK itself.

Maybe teach driverctl (and udev) to handle bifurcated drivers.
Unfortunately, vendors are very fractured on how network devices are managed.

Re: [dpdk-dev] kernel binding of devices + hotplug

[Flavio Leitner]

On Wed, Apr 18, 2018 at 11:17:47AM -0700, Stephen Hemminger wrote:
> On Wed, 18 Apr 2018 11:11:01 -0300
> Flavio Leitner <fbl@redhat.com> wrote:
> 
> > On Sun, Apr 15, 2018 at 01:48:36AM +0000, Stephen Hemminger wrote:
> > > My vote is to work with udev and not try to replace it.
> > > 
> > > Driverctl works well. Just not for bifurcated driver  
> > 
> > I second that.  We also have other system configs to care about like
> > kernel parameters and hugepage configuration which I think follow the
> > same idea that they are system wide configs and should not be managed
> > by DPDK itself.
> 
> Maybe teach driverctl (and udev) to handle bifurcated drivers.

I don't know the challenges to tech driverctl to handle bifurcated
drivers but I would agree that it should be our first place to look at.

> Unfortunately, vendors are very fractured on how network devices are managed.

You mean distros? hw vendors? all vendors? :)

Perhaps if community focus on something, then they might follow at some
point.

-- 
Flavio

Re: [dpdk-dev] kernel binding of devices + hotplug

[Alejandro Lucero]

I do not completely understand the discussion, but I think the disagreement
is due to how some devices interact with DPDK, at least Mellanox ones. I'm
saying that because we have a DPDK app which starts with no device at all
(--no-pci) and it relies on device plugging attach/detach for configuring
and removing ports once devices are bound to VFIO or UIO drivers. Maybe I'm
wrong, but I think because Mellanox cards do not use VFIO or UIO drivers
but some specific bound using verbs inside the PMD, leaving all this
binding to the system does not fit them.

If that is the case, although I agree with leaving the device binding to
the system, I think it would be fair to contemplate a dual approach for
legacy reasons, or to leave time for implementing a pseudo system driver
which Mellanox can use for having same functionality.

On Wed, Apr 18, 2018 at 7:54 PM, Flavio Leitner <fbl@redhat.com> wrote:

> On Wed, Apr 18, 2018 at 11:17:47AM -0700, Stephen Hemminger wrote:
> > On Wed, 18 Apr 2018 11:11:01 -0300
> > Flavio Leitner <fbl@redhat.com> wrote:
> >
> > > On Sun, Apr 15, 2018 at 01:48:36AM +0000, Stephen Hemminger wrote:
> > > > My vote is to work with udev and not try to replace it.
> > > >
> > > > Driverctl works well. Just not for bifurcated driver
> > >
> > > I second that.  We also have other system configs to care about like
> > > kernel parameters and hugepage configuration which I think follow the
> > > same idea that they are system wide configs and should not be managed
> > > by DPDK itself.
> >
> > Maybe teach driverctl (and udev) to handle bifurcated drivers.
>
> I don't know the challenges to tech driverctl to handle bifurcated
> drivers but I would agree that it should be our first place to look at.
>
> > Unfortunately, vendors are very fractured on how network devices are
> managed.
>
> You mean distros? hw vendors? all vendors? :)
>
> Perhaps if community focus on something, then they might follow at some
> point.
>
> --
> Flavio
>

Re: [dpdk-dev] kernel binding of devices + hotplug

[Thomas Monjalon]

19/04/2018 08:04, Alejandro Lucero:
> I do not completely understand the discussion, but I think the disagreement
> is due to how some devices interact with DPDK, at least Mellanox ones. I'm
> saying that because we have a DPDK app which starts with no device at all
> (--no-pci) and it relies on device plugging attach/detach for configuring
> and removing ports once devices are bound to VFIO or UIO drivers. Maybe I'm
> wrong, but I think because Mellanox cards do not use VFIO or UIO drivers
> but some specific bound using verbs inside the PMD, leaving all this
> binding to the system does not fit them.

Mellanox uses a bifurcated model for any use.
Others could use a bifurcated model thanks to AF_XDP.
That's why it is more correct to compare "bifurcated model" vs "UIO/VFIO".

> If that is the case, although I agree with leaving the device binding to
> the system, I think it would be fair to contemplate a dual approach for
> legacy reasons, or to leave time for implementing a pseudo system driver
> which Mellanox can use for having same functionality.

I summarize the comparison:
- On one hand, we can configure all the devices only once in DPDK,
but it gives super-powers to the DPDK application.
- On the other hand, we can do a part of the setup at system level
(some kernel binding or flow bifurcation), and we do another part
of the setup in DPDK, splitting/duplicating the setup info in two places.


> On Wed, Apr 18, 2018 at 7:54 PM, Flavio Leitner <fbl@redhat.com> wrote:
> > On Wed, Apr 18, 2018 at 11:17:47AM -0700, Stephen Hemminger wrote:
> > > On Wed, 18 Apr 2018 11:11:01 -0300
> > > Flavio Leitner <fbl@redhat.com> wrote:
> > > > On Sun, Apr 15, 2018 at 01:48:36AM +0000, Stephen Hemminger wrote:
> > > > > My vote is to work with udev and not try to replace it.
> > > > >
> > > > > Driverctl works well. Just not for bifurcated driver
> > > >
> > > > I second that.  We also have other system configs to care about like
> > > > kernel parameters and hugepage configuration which I think follow the
> > > > same idea that they are system wide configs and should not be managed
> > > > by DPDK itself.
> > >
> > > Maybe teach driverctl (and udev) to handle bifurcated drivers.
> >
> > I don't know the challenges to tech driverctl to handle bifurcated
> > drivers but I would agree that it should be our first place to look at.
> >
> > > Unfortunately, vendors are very fractured on how network devices are
> > managed.
> >
> > You mean distros? hw vendors? all vendors? :)
> >
> > Perhaps if community focus on something, then they might follow at some
> > point.
> >
> > --
> > Flavio
> >
> 

Re: [dpdk-dev] kernel binding of devices + hotplug

[Bruce Richardson]

On Thu, Apr 19, 2018 at 10:24:24AM +0200, Thomas Monjalon wrote:
> 19/04/2018 08:04, Alejandro Lucero:
> > I do not completely understand the discussion, but I think the
> > disagreement is due to how some devices interact with DPDK, at least
> > Mellanox ones. I'm saying that because we have a DPDK app which starts
> > with no device at all (--no-pci) and it relies on device plugging
> > attach/detach for configuring and removing ports once devices are bound
> > to VFIO or UIO drivers. Maybe I'm wrong, but I think because Mellanox
> > cards do not use VFIO or UIO drivers but some specific bound using
> > verbs inside the PMD, leaving all this binding to the system does not
> > fit them.
> 
> Mellanox uses a bifurcated model for any use.  Others could use a
> bifurcated model thanks to AF_XDP.  That's why it is more correct to
> compare "bifurcated model" vs "UIO/VFIO".
> 
> > If that is the case, although I agree with leaving the device binding
> > to the system, I think it would be fair to contemplate a dual approach
> > for legacy reasons, or to leave time for implementing a pseudo system
> > driver which Mellanox can use for having same functionality.
> 
> I summarize the comparison: - On one hand, we can configure all the
> devices only once in DPDK, but it gives super-powers to the DPDK
> application.  - On the other hand, we can do a part of the setup at
> system level (some kernel binding or flow bifurcation), and we do another
> part of the setup in DPDK, splitting/duplicating the setup info in two
> places.
> 
> 
I disagree with this final assessment. If there is duplication, it would
appear more in the former case than in the latter, as the logic for
determining driver binding and ownership would have to be duplicated in
both the kernel and DPDK.  Also, there are plenty of instances where the
kernel is going to use the devices without DPDK, so you can't reduce
duplication by putting functionality into DPDK - the kernel is not going to
remove functionality just because DPDK offers it! Therefore, if minimizing
duplication is important to you, the solution to chose is to use as much
from the kernel (and associated utilities like udev) as possible and not
implement it in DPDK.

Re: [dpdk-dev] kernel binding of devices + hotplug

[Thomas Monjalon]

19/04/2018 10:40, Bruce Richardson:
> On Thu, Apr 19, 2018 at 10:24:24AM +0200, Thomas Monjalon wrote:
> > 19/04/2018 08:04, Alejandro Lucero:
> > > I do not completely understand the discussion, but I think the
> > > disagreement is due to how some devices interact with DPDK, at least
> > > Mellanox ones. I'm saying that because we have a DPDK app which starts
> > > with no device at all (--no-pci) and it relies on device plugging
> > > attach/detach for configuring and removing ports once devices are bound
> > > to VFIO or UIO drivers. Maybe I'm wrong, but I think because Mellanox
> > > cards do not use VFIO or UIO drivers but some specific bound using
> > > verbs inside the PMD, leaving all this binding to the system does not
> > > fit them.
> > 
> > Mellanox uses a bifurcated model for any use.  Others could use a
> > bifurcated model thanks to AF_XDP.  That's why it is more correct to
> > compare "bifurcated model" vs "UIO/VFIO".
> > 
> > > If that is the case, although I agree with leaving the device binding
> > > to the system, I think it would be fair to contemplate a dual approach
> > > for legacy reasons, or to leave time for implementing a pseudo system
> > > driver which Mellanox can use for having same functionality.
> > 
> > I summarize the comparison: - On one hand, we can configure all the
> > devices only once in DPDK, but it gives super-powers to the DPDK
> > application.  - On the other hand, we can do a part of the setup at
> > system level (some kernel binding or flow bifurcation), and we do another
> > part of the setup in DPDK, splitting/duplicating the setup info in two
> > places.
> > 
> I disagree with this final assessment. If there is duplication, it would
> appear more in the former case than in the latter, as the logic for
> determining driver binding and ownership would have to be duplicated in
> both the kernel and DPDK.  Also, there are plenty of instances where the
> kernel is going to use the devices without DPDK, so you can't reduce
> duplication by putting functionality into DPDK - the kernel is not going to
> remove functionality just because DPDK offers it! Therefore, if minimizing
> duplication is important to you, the solution to chose is to use as much
> from the kernel (and associated utilities like udev) as possible and not
> implement it in DPDK.

In any case, we need to choose and configure DPDK devices in DPDK
application, right?
If implementing all DPDK setup in DPDK, the only thing to do in system
config, is to NOT use the device (set IP, etc).
So preparing the devices for DPDK (or other userpace app) as part of the
system setup is an addition.

Note: I don't know which decision is better, I am just trying to think
about what the solutions are.

Re: [dpdk-dev] kernel binding of devices + hotplug

[Matan Azrad]

Hi Konstantin

From: Ananyev, Konstantin, Tuesday, April 17, 2018 2:00 PM
> Hi Matan,
> > Hi Konstantin
> > From: Ananyev, Konstantin, Tuesday, April 17, 2018 12:23 PM

> > > > Actually you say that the whitelist\blacklist mechanism is not
> > > > good enough
> > > and the binding workarounds it.
> > > > The user need to specify somehow the devices it want to run, I
> > > > think that specifying the device you want by -w option (no need to
> > > > specify what you don't want in -w case) is really simpler  and
> > > > more descriptive than
> > > binding each device you want by prior process to its correct driver.
> > >
> > > But what if user changes his mind and decides to give particular
> > > device back to the kernel?
> > > Should he restart the dpdk application? In some cases it might be
> > > not desirable.
> >
> > And what is the behavior now in this case?
> 
> Now we don't have hotplug support at all, so nothing to worry about, I guess
> :)

Looks like we are going to support it soon (Guo series have started it) :)
We have already fail-safe driver to support hot-plug, and a real use case to manage Azure dpdk system.
I think we must take it into account when we discuss on binding. 

> > Looks like if we want to solve it we need to add mechanism to stop these
> particular devices DPDK management in any case.
> 
> I am not talking about managing a device itself (start/stop/attach/detach).
> Let say you start dpdk with '-w dev -w dev2'.
> dev1 is active, traffic is going through it, etc., while dev2 is unplugged.
> Now user is about to plug dev2, though just before that he changed his mind
> and decided to give that device to kernel (or different dpdk app), ideally
> without re-starting the app.
> With just command-line support there is no option to do that.
> Same for opposite case - you started dpdk app with '-w dev1' and then
> decided that you also want that app to manage hot-pluggable dev2 too.

And what if the user wants to switch the management from a dpdk application to other user-space\dpdk application?
Here the binding method doesn't work.
Looks like you are suggesting a new feature regardless binding. 

> > > > > It also allows better usability across systems, since the same
> > > > > commandline can be used on multiple systems with different
> > > > > hardware, with the actual device management rules having been
> > > > > already configured at system install/setup time in udev.
> > > >
> > > > But the user still needs to configure the udev per device for each
> > > > system, I
> > > think that command line is better.
> > > >
> > > > > > Regarding the conflict of system rules for a device, it is
> > > > > > again the user
> > > > > responsibility, whatever we will decide for the binding
> > > > > procedure of DPDK application the user needs to take it into
> > > > > account and to solve such like conflicts.
> > > > > > One option is to remove any binding rules of a DPDK device in
> > > > > > the DPDK
> > > > > application initialization and adjust the new rules by the PMDs,
> > > > > then any conflict should not disturb the user.
> > > > >
> > > > > If the device management is only managed in one place, i.e. not
> > > > > in DPDK, then there is no conflict to manage.
> > > >
> > > > I can't agree with this statement, The essence of DPDK is to give
> > > > a good alternative to managing network devices, DPDK actually
> > > > takes a lot of management area to manage by itself to do the user
> > > > life better :)
> > >
> > > From my point - it is not about managing particular device.
> > > It is about making decision who (kenel/dpdk) will manage that device.
> >
> > Doesn't the w\b mechanism comes to solve it?
> 
> It does till some extent, but I don't think it is the best possible way.

Why? pros and cons against alternative ways...

> >
> > > From usability perspective it seems to me that better to keep it in
> > > one place for all devices.
> > > So udev (or any other sysadmin tool) seems like a right choice here.
> >
> > Please explain why?
> > And if you are talking about 1 place:
> > Why not to do the binding in the same place where we define which device
> to run?
> 
> Not all devices are always managed by dpdk apps.
> Some of them will still be managed by kernel.
> Again there could be several independent dpdk apps running on the same
> system.
> Obviously dpdk app itself can't be a centric place to handle all that varieties.

You can use whitelist way to specify the devices for each app and that's it.

The binding to UIO driver cannot choose the specific user-space\dpdk application for the device management.

> Again - if there exists a system tool that provides that functionality - why not
> to use it?

Use it by dpdk or any other user-space application needs to manage devices.