[dpdk-users] [announce] driverctl: utility for persistent alternative driver binding

[dpdk-users] [announce] driverctl: utility for persistent alternative driver binding

[Panu Matilainen]

Hi all,

While this is not directly related to DPDK or OVS, it is potentially 
useful for users of both, so excuse me for cross-posting.

Quoting from the project README (for the full text see
http://laiskiainen.org/git/?p=driverctl.git;a=blob_plain;f=README)

 > driverctl is a tool for manipulating and inspecting the system
 > device driver choices.
 >
 > Devices are normally assigned to their sole designated kernel driver
 > by default. However in some situations it may be desireable to
 > override that default, for example to try an older driver to
 > work around a regression in a driver or to try an experimental
 > alternative driver. Another common use-case is pass-through
 > drivers and driver stubs to allow userspace to drive the device,
 > such as in case of virtualization.
 >
 > driverctl integrates with udev to support overriding
 > driver selection for both cold- and hotplugged devices from the
 > moment of discovery, but can also change already assigned drivers,
 > assuming they are not in use by the system. The driver overrides
 > created by driverctl are persistent across system reboots
 > by default.
 >
 > Usage
 > -----
 >
 > Find devices currently driven by ixgbe driver:
 >
 > # driverctl -v list-devices | grep ixgbe
 > 0000:01:00.0 ixgbe (Ethernet 10G 4P X520/I350 rNDC)
 > 0000:01:00.1 ixgbe (Ethernet 10G 4P X520/I350 rNDC)
 >
 > Change them to use the vfio-pci driver:
 > # driverctl set-override 0000:01:00.0 vfio-pci
 > # driverctl set-override 0000:01:00.1 vfio-pci
 >
 > Find devices with driver overrides:
 > # driverctl -v list-devices|grep \\*
 > 0000:01:00.0 vfio-pci [*] (Ethernet 10G 4P X520/I350 rNDC)
 > 0000:01:00.1 vfio-pci [*] (Ethernet 10G 4P X520/I350 rNDC)
 >
 > Remove the override from slot 0000:01:00.1:
 > # driverctl unset-override 0000:01:00.1

DPDK of course has its own dpdk_nic_bind(.py) tool for this purpose, the 
main differences to driverctl are:
- driverctl bindings are persistent across system boots
- driverctl bindings take place immediately on cold- and hotplug
- driverctl is a generic tool not limited to network adapters
- dpdk_nic_bind being a special purpose tool has many more
   sanity checks for its supported use-cases
- dpdk_nic_bind supports binding multiple NICs at once

The project currently lives at
     http://laiskiainen.org/git/?p=driverctl.git

Feedback, patches etc are welcome.

	- Panu -

Re: [dpdk-users] [ovs-discuss] [announce] driverctl: utility for persistent alternative driver binding

[Gray, Mark D]

I welcome this initiative, one question below:

> -----Original Message-----
> From: discuss [mailto:discuss-bounces@openvswitch.org] On Behalf Of Panu
> Matilainen
> Sent: Friday, December 4, 2015 10:54 AM
> To: dev@dpdk.org; users@dpdk.org; dev@openvswitch.org;
> discuss@openvswitch.org
> Subject: [ovs-discuss] [announce] driverctl: utility for persistent alternative
> driver binding
> 
> Hi all,
> 
> While this is not directly related to DPDK or OVS, it is potentially
> useful for users of both, so excuse me for cross-posting.
> 
> Quoting from the project README (for the full text see
> http://laiskiainen.org/git/?p=driverctl.git;a=blob_plain;f=README)
> 
>  > driverctl is a tool for manipulating and inspecting the system
>  > device driver choices.
>  >
>  > Devices are normally assigned to their sole designated kernel driver
>  > by default. However in some situations it may be desireable to
>  > override that default, for example to try an older driver to
>  > work around a regression in a driver or to try an experimental
>  > alternative driver. Another common use-case is pass-through
>  > drivers and driver stubs to allow userspace to drive the device,
>  > such as in case of virtualization.
>  >
>  > driverctl integrates with udev to support overriding
>  > driver selection for both cold- and hotplugged devices from the
>  > moment of discovery, but can also change already assigned drivers,
>  > assuming they are not in use by the system. The driver overrides
>  > created by driverctl are persistent across system reboots
>  > by default.
>  >
>  > Usage
>  > -----
>  >
>  > Find devices currently driven by ixgbe driver:
>  >
>  > # driverctl -v list-devices | grep ixgbe
>  > 0000:01:00.0 ixgbe (Ethernet 10G 4P X520/I350 rNDC)
>  > 0000:01:00.1 ixgbe (Ethernet 10G 4P X520/I350 rNDC)
>  >
>  > Change them to use the vfio-pci driver:
>  > # driverctl set-override 0000:01:00.0 vfio-pci
>  > # driverctl set-override 0000:01:00.1 vfio-pci
>  >
>  > Find devices with driver overrides:
>  > # driverctl -v list-devices|grep \\*
>  > 0000:01:00.0 vfio-pci [*] (Ethernet 10G 4P X520/I350 rNDC)
>  > 0000:01:00.1 vfio-pci [*] (Ethernet 10G 4P X520/I350 rNDC)
>  >
>  > Remove the override from slot 0000:01:00.1:
>  > # driverctl unset-override 0000:01:00.1
> 
> DPDK of course has its own dpdk_nic_bind(.py) tool for this purpose, the
> main differences to driverctl are:
> - driverctl bindings are persistent across system boots

 [Gray, Mark D] This is great!

Will this integrate with, for example in Red Hat-based systems, /etc/sysconfig/network-scripts/ifcfg-X? In DPDK, could we then
potentially reference devices by that (arbitrary) name?

> - driverctl bindings take place immediately on cold- and hotplug
> - driverctl is a generic tool not limited to network adapters
> - dpdk_nic_bind being a special purpose tool has many more
>    sanity checks for its supported use-cases
> - dpdk_nic_bind supports binding multiple NICs at once
> 
> The project currently lives at
>      http://laiskiainen.org/git/?p=driverctl.git
> 
> Feedback, patches etc are welcome.
> 
> 	- Panu -
> _______________________________________________
> discuss mailing list
> discuss@openvswitch.org
> http://openvswitch.org/mailman/listinfo/discuss

Re: [dpdk-users] [ovs-discuss] [announce] driverctl: utility for persistent alternative driver binding

[Panu Matilainen]

On 12/04/2015 05:44 PM, Gray, Mark D wrote:
> I welcome this initiative, one question below:
>
>> -----Original Message-----
>> From: discuss [mailto:discuss-bounces@openvswitch.org] On Behalf Of Panu
>> Matilainen
>> Sent: Friday, December 4, 2015 10:54 AM
>> To: dev@dpdk.org; users@dpdk.org; dev@openvswitch.org;
>> discuss@openvswitch.org
>> Subject: [ovs-discuss] [announce] driverctl: utility for persistent alternative
>> driver binding
>>
>> Hi all,
>>
>> While this is not directly related to DPDK or OVS, it is potentially
>> useful for users of both, so excuse me for cross-posting.
>>
>> Quoting from the project README (for the full text see
>> http://laiskiainen.org/git/?p=driverctl.git;a=blob_plain;f=README)
>>
>>   > driverctl is a tool for manipulating and inspecting the system
>>   > device driver choices.
>>   >
>>   > Devices are normally assigned to their sole designated kernel driver
>>   > by default. However in some situations it may be desireable to
>>   > override that default, for example to try an older driver to
>>   > work around a regression in a driver or to try an experimental
>>   > alternative driver. Another common use-case is pass-through
>>   > drivers and driver stubs to allow userspace to drive the device,
>>   > such as in case of virtualization.
>>   >
>>   > driverctl integrates with udev to support overriding
>>   > driver selection for both cold- and hotplugged devices from the
>>   > moment of discovery, but can also change already assigned drivers,
>>   > assuming they are not in use by the system. The driver overrides
>>   > created by driverctl are persistent across system reboots
>>   > by default.
>>   >
>>   > Usage
>>   > -----
>>   >
>>   > Find devices currently driven by ixgbe driver:
>>   >
>>   > # driverctl -v list-devices | grep ixgbe
>>   > 0000:01:00.0 ixgbe (Ethernet 10G 4P X520/I350 rNDC)
>>   > 0000:01:00.1 ixgbe (Ethernet 10G 4P X520/I350 rNDC)
>>   >
>>   > Change them to use the vfio-pci driver:
>>   > # driverctl set-override 0000:01:00.0 vfio-pci
>>   > # driverctl set-override 0000:01:00.1 vfio-pci
>>   >
>>   > Find devices with driver overrides:
>>   > # driverctl -v list-devices|grep \\*
>>   > 0000:01:00.0 vfio-pci [*] (Ethernet 10G 4P X520/I350 rNDC)
>>   > 0000:01:00.1 vfio-pci [*] (Ethernet 10G 4P X520/I350 rNDC)
>>   >
>>   > Remove the override from slot 0000:01:00.1:
>>   > # driverctl unset-override 0000:01:00.1
>>
>> DPDK of course has its own dpdk_nic_bind(.py) tool for this purpose, the
>> main differences to driverctl are:
>> - driverctl bindings are persistent across system boots
>
>   [Gray, Mark D] This is great!
>
> Will this integrate with, for example in Red Hat-based systems, /etc/sysconfig/network-scripts/ifcfg-X? In DPDK, could we then
> potentially reference devices by that (arbitrary) name?

driverctl is not specific to NICs so network-scripts integration is out 
of scope.

That aside, maybe I'm missing something but I'm not sure what there is 
to integrate with since DPDK ports are ultimately application specific. 
For OVS I've sent a patch to support managing OVS DPDK ports via 
network-scripts: 
http://openvswitch.org/pipermail/dev/2015-December/062850.html

	. Panu -

Re: [dpdk-users] [ovs-discuss] [announce] driverctl: utility for persistent alternative driver binding

[Gray, Mark D]

> -----Original Message-----
> From: Panu Matilainen [mailto:pmatilai@redhat.com]
> Sent: Tuesday, December 8, 2015 12:05 PM
> To: Gray, Mark D; dev@dpdk.org; users@dpdk.org; dev@openvswitch.org;
> discuss@openvswitch.org
> Subject: Re: [ovs-discuss] [announce] driverctl: utility for persistent
> alternative driver binding
> 
> On 12/04/2015 05:44 PM, Gray, Mark D wrote:
> > I welcome this initiative, one question below:
> >
> >> -----Original Message-----
> >> From: discuss [mailto:discuss-bounces@openvswitch.org] On Behalf Of
> >> Panu Matilainen
> >> Sent: Friday, December 4, 2015 10:54 AM
> >> To: dev@dpdk.org; users@dpdk.org; dev@openvswitch.org;
> >> discuss@openvswitch.org
> >> Subject: [ovs-discuss] [announce] driverctl: utility for persistent
> >> alternative driver binding
> >>
> >> Hi all,
> >>
> >> While this is not directly related to DPDK or OVS, it is potentially
> >> useful for users of both, so excuse me for cross-posting.
> >>
> >> Quoting from the project README (for the full text see
> >> http://laiskiainen.org/git/?p=driverctl.git;a=blob_plain;f=README)
> >>
> >>   > driverctl is a tool for manipulating and inspecting the system
> >>   > device driver choices.
> >>   >
> >>   > Devices are normally assigned to their sole designated kernel driver
> >>   > by default. However in some situations it may be desireable to
> >>   > override that default, for example to try an older driver to
> >>   > work around a regression in a driver or to try an experimental
> >>   > alternative driver. Another common use-case is pass-through
> >>   > drivers and driver stubs to allow userspace to drive the device,
> >>   > such as in case of virtualization.
> >>   >
> >>   > driverctl integrates with udev to support overriding
> >>   > driver selection for both cold- and hotplugged devices from the
> >>   > moment of discovery, but can also change already assigned drivers,
> >>   > assuming they are not in use by the system. The driver overrides
> >>   > created by driverctl are persistent across system reboots
> >>   > by default.
> >>   >
> >>   > Usage
> >>   > -----
> >>   >
> >>   > Find devices currently driven by ixgbe driver:
> >>   >
> >>   > # driverctl -v list-devices | grep ixgbe
> >>   > 0000:01:00.0 ixgbe (Ethernet 10G 4P X520/I350 rNDC)
> >>   > 0000:01:00.1 ixgbe (Ethernet 10G 4P X520/I350 rNDC)
> >>   >
> >>   > Change them to use the vfio-pci driver:
> >>   > # driverctl set-override 0000:01:00.0 vfio-pci
> >>   > # driverctl set-override 0000:01:00.1 vfio-pci
> >>   >
> >>   > Find devices with driver overrides:
> >>   > # driverctl -v list-devices|grep \\*
> >>   > 0000:01:00.0 vfio-pci [*] (Ethernet 10G 4P X520/I350 rNDC)
> >>   > 0000:01:00.1 vfio-pci [*] (Ethernet 10G 4P X520/I350 rNDC)
> >>   >
> >>   > Remove the override from slot 0000:01:00.1:
> >>   > # driverctl unset-override 0000:01:00.1
> >>
> >> DPDK of course has its own dpdk_nic_bind(.py) tool for this purpose,
> >> the main differences to driverctl are:
> >> - driverctl bindings are persistent across system boots
> >
> >   [Gray, Mark D] This is great!
> >
> > Will this integrate with, for example in Red Hat-based systems,
> > /etc/sysconfig/network-scripts/ifcfg-X? In DPDK, could we then potentially
> reference devices by that (arbitrary) name?
> 
> driverctl is not specific to NICs so network-scripts integration is out of scope.
> 

I forgot that point. It makes sense.

> That aside, maybe I'm missing something but I'm not sure what there is to
> integrate with since DPDK ports are ultimately application specific.
> For OVS I've sent a patch to support managing OVS DPDK ports via
> network-scripts:
> http://openvswitch.org/pipermail/dev/2015-December/062850.html

Thanks, I wasn’t familiar with this work but this is a good approach.

> 
> 	. Panu -
> 
What I am thinking would be an ideal end state for OVS would be the possibility to add a port using ovs-vsctl add-port referencing the device name

ovs-vsctl add-port p1p2

If dpdk was enabled, then OVS could query something(?) to determine which physical interface this name actually referred to and then from that figure out what dpdk port number it referred to.