DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] Permanently binding NIC ports with DPDK drivers
@ 2015-11-11 16:13 Montorsi, Francesco
  2015-11-11 16:28 ` Bruce Richardson
                   ` (2 more replies)
  0 siblings, 3 replies; 14+ messages in thread
From: Montorsi, Francesco @ 2015-11-11 16:13 UTC (permalink / raw)
  To: dev

Hi,
Is there a way to permanently (i.e., have the configuration automatically applied after reboot) bind a NIC port to DPDK?

In case there's none, I'm thinking to save in my software a list of the NIC ports chosen by the user for use with DPDK and then, upon software startup to just do
    for (int i=0; i < ...; i++)
     system("dpdk_nic_bind.py --bind=igb_uio " + PCI_device_chosen[i]);
Do you see any problem with that? 

Thanks!
Francesco Montorsi

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [dpdk-dev] Permanently binding NIC ports with DPDK drivers
  2015-11-11 16:13 [dpdk-dev] Permanently binding NIC ports with DPDK drivers Montorsi, Francesco
@ 2015-11-11 16:28 ` Bruce Richardson
  2015-11-11 16:59   ` Montorsi, Francesco
  2015-11-12 12:57   ` Panu Matilainen
  2015-11-11 19:01 ` Matthew Hall
  2015-11-12  0:14 ` Mcnamara, John
  2 siblings, 2 replies; 14+ messages in thread
From: Bruce Richardson @ 2015-11-11 16:28 UTC (permalink / raw)
  To: Montorsi, Francesco; +Cc: dev

On Wed, Nov 11, 2015 at 04:13:01PM +0000, Montorsi, Francesco wrote:
> Hi,
> Is there a way to permanently (i.e., have the configuration automatically applied after reboot) bind a NIC port to DPDK?
> 
> In case there's none, I'm thinking to save in my software a list of the NIC ports chosen by the user for use with DPDK and then, upon software startup to just do
>     for (int i=0; i < ...; i++)
>      system("dpdk_nic_bind.py --bind=igb_uio " + PCI_device_chosen[i]);
> Do you see any problem with that? 
> 
> Thanks!
> Francesco Montorsi
> 

Hi Francesco,

I'm not aware of any way to make the bindings permanent across reboots. What you
have suggested will work, but there are probably better ways to do the same thing.
For example, a couple of lines in an rc.local script can reapply the bindings at
boot for you. I'm sure others can suggest other ways of having the same effect,
for example, there may be a way to automatically do this using udev or systemd
or some such package.

/Bruce

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [dpdk-dev] Permanently binding NIC ports with DPDK drivers
  2015-11-11 16:28 ` Bruce Richardson
@ 2015-11-11 16:59   ` Montorsi, Francesco
  2015-11-11 17:29     ` Stephen Hemminger
  2015-11-12 12:57   ` Panu Matilainen
  1 sibling, 1 reply; 14+ messages in thread
From: Montorsi, Francesco @ 2015-11-11 16:59 UTC (permalink / raw)
  To: Bruce Richardson; +Cc: dev

Hi Bruce,

> -----Original Message-----
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> I'm not aware of any way to make the bindings permanent across reboots.
> What you have suggested will work, but there are probably better ways to
> do the same thing.

I agree... let's see if somebody else has suggestions :)

In any case my idea is to make my software as much independent as possible from troubles with future HW and future DPDK versions. A way to do that would be to leave all the bind steps and intelligence inside the dpdk_nic_bind.py script and just use that (since it will be probably always up to date and correct). My only concern is that (reading the python code) dpdk_nic_bind.py script does not return with an error code != 0 if something bad happens during binding... maybe it may be worth doing such a small change...

Just my 2 cents,
Francesco

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [dpdk-dev] Permanently binding NIC ports with DPDK drivers
  2015-11-11 16:59   ` Montorsi, Francesco
@ 2015-11-11 17:29     ` Stephen Hemminger
  0 siblings, 0 replies; 14+ messages in thread
From: Stephen Hemminger @ 2015-11-11 17:29 UTC (permalink / raw)
  To: Montorsi, Francesco; +Cc: dev

On Wed, 11 Nov 2015 16:59:14 +0000
"Montorsi, Francesco" <fmontorsi@empirix.com> wrote:

> Hi Bruce,
> 
> > -----Original Message-----
> > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > I'm not aware of any way to make the bindings permanent across reboots.
> > What you have suggested will work, but there are probably better ways to
> > do the same thing.
> 
> I agree... let's see if somebody else has suggestions :)
> 
> In any case my idea is to make my software as much independent as possible from troubles with future HW and future DPDK versions. A way to do that would be to leave all the bind steps and intelligence inside the dpdk_nic_bind.py script and just use that (since it will be probably always up to date and correct). My only concern is that (reading the python code) dpdk_nic_bind.py script does not return with an error code != 0 if something bad happens during binding... maybe it may be worth doing such a small change...
> 
> Just my 2 cents,
> Francesco
> 
> 
> 
> 

I would recommend using PCI id's and not depending in anyway on port.
If you want, I can submit a patch to add derive the systemd/udev compatiable
name from existing DPDK port.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [dpdk-dev] Permanently binding NIC ports with DPDK drivers
  2015-11-11 16:13 [dpdk-dev] Permanently binding NIC ports with DPDK drivers Montorsi, Francesco
  2015-11-11 16:28 ` Bruce Richardson
@ 2015-11-11 19:01 ` Matthew Hall
  2015-11-12  0:14 ` Mcnamara, John
  2 siblings, 0 replies; 14+ messages in thread
From: Matthew Hall @ 2015-11-11 19:01 UTC (permalink / raw)
  To: Montorsi, Francesco; +Cc: dev

In my development environment I set up an at-boot provisioning script that 
does it. I recommend using scripts and not shelling out from C code. ;)

On Wed, Nov 11, 2015 at 04:13:01PM +0000, Montorsi, Francesco wrote:
> Hi,
> Is there a way to permanently (i.e., have the configuration automatically applied after reboot) bind a NIC port to DPDK?
> 
> In case there's none, I'm thinking to save in my software a list of the NIC ports chosen by the user for use with DPDK and then, upon software startup to just do
>     for (int i=0; i < ...; i++)
>      system("dpdk_nic_bind.py --bind=igb_uio " + PCI_device_chosen[i]);
> Do you see any problem with that? 
> 
> Thanks!
> Francesco Montorsi
> 

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [dpdk-dev] Permanently binding NIC ports with DPDK drivers
  2015-11-11 16:13 [dpdk-dev] Permanently binding NIC ports with DPDK drivers Montorsi, Francesco
  2015-11-11 16:28 ` Bruce Richardson
  2015-11-11 19:01 ` Matthew Hall
@ 2015-11-12  0:14 ` Mcnamara, John
  2015-11-13 11:48   ` Montorsi, Francesco
  2 siblings, 1 reply; 14+ messages in thread
From: Mcnamara, John @ 2015-11-12  0:14 UTC (permalink / raw)
  To: Montorsi, Francesco, dev

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Montorsi, Francesco
> Sent: Wednesday, November 11, 2015 4:13 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] Permanently binding NIC ports with DPDK drivers
> 
> Hi,
> Is there a way to permanently (i.e., have the configuration automatically
> applied after reboot) bind a NIC port to DPDK?


Hi,

The Ubuntu dpdk package for 15.10 contains system scripts with functions for reserving hugepages and binding interfaces on bootup:


    /etc/dpdk/dpdk.conf
    /etc/dpdk/interfaces
    /etc/init.d/dpdk
    /lib/dpdk/dpdk-init
    /lib/systemd/system/dpdk.service
    /sbin/dpdk_nic_bind
    /usr/bin/testpmd
    /usr/share/doc/dpdk/README.Debian
    /usr/share/doc/dpdk/changelog.Debian.gz
    /usr/share/doc/dpdk/copyright
    /usr/share/dpdk/tools/cpu_layout.py
    /usr/share/dpdk/tools/dpdk_nic_bind.py
    /usr/share/dpdk/tools/setup.sh
    /usr/share/python/runtime.d/dpdk.rtupdate

    http://packages.ubuntu.com/wily/amd64/dpdk/filelist

If you have the latest version of Ubuntu you can check that out or else download and extract the files from the .deb to see how they do it.

John.
-- 

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [dpdk-dev] Permanently binding NIC ports with DPDK drivers
  2015-11-11 16:28 ` Bruce Richardson
  2015-11-11 16:59   ` Montorsi, Francesco
@ 2015-11-12 12:57   ` Panu Matilainen
  2015-11-13 11:44     ` Montorsi, Francesco
  2015-11-13 14:42     ` Aaron Conole
  1 sibling, 2 replies; 14+ messages in thread
From: Panu Matilainen @ 2015-11-12 12:57 UTC (permalink / raw)
  To: Bruce Richardson, Montorsi, Francesco; +Cc: dev

On 11/11/2015 06:28 PM, Bruce Richardson wrote:
> On Wed, Nov 11, 2015 at 04:13:01PM +0000, Montorsi, Francesco wrote:
>> Hi,
>> Is there a way to permanently (i.e., have the configuration automatically applied after reboot) bind a NIC port to DPDK?
>>
>> In case there's none, I'm thinking to save in my software a list of the NIC ports chosen by the user for use with DPDK and then, upon software startup to just do
>>      for (int i=0; i < ...; i++)
>>       system("dpdk_nic_bind.py --bind=igb_uio " + PCI_device_chosen[i]);
>> Do you see any problem with that?
>>
>> Thanks!
>> Francesco Montorsi
>>
>
> Hi Francesco,
>
> I'm not aware of any way to make the bindings permanent across reboots. What you
> have suggested will work, but there are probably better ways to do the same thing.
> For example, a couple of lines in an rc.local script can reapply the bindings at
> boot for you. I'm sure others can suggest other ways of having the same effect,
> for example, there may be a way to automatically do this using udev or systemd
> or some such package.

I've been looking into this recently, here's what I have so far:
http://laiskiainen.org/git/?p=driverctl.git

For the impatient, "make rpm" should produce something usable for recent 
Fedora/RHEL systems, usage looks somewhat like this:

Find devices currently driven by ixgbe driver:
# driverctl -v list-devices | grep ixgbe
0000:01:00.0 ixgbe (Ethernet 10G 4P X520/I350 rNDC)
0000:01:00.1 ixgbe (Ethernet 10G 4P X520/I350 rNDC)

Change them to use the vfio-pci driver permanently:
# driverctl set-override 0000:01:00.0 vfio-pci
# driverctl set-override 0000:01:00.1 vfio-pci

Find devices with driver overrides:
[root@wsfd-netdev32 ~]# driverctl -v list-devices|grep \*
0000:01:00.0 vfio-pci [*] (Ethernet 10G 4P X520/I350 rNDC)
0000:01:00.1 vfio-pci [*] (Ethernet 10G 4P X520/I350 rNDC)

Remove the permanent driver override for device 0000:01:00.1:
# driverctl unset-override 0000:01:00.1

In addition it has udev rules to export vfio and uio devices on systemd 
level, eg the above looks like this with normal drivers:

# systemctl |grep 0000:01:00
sys-devices-pci0000:00-0000:00:03.0-0000:01:00.0-net-em1.device 
                  loaded active plugged   Ethernet 10G 4P X520/I350 rNDC
sys-devices-pci0000:00-0000:00:03.0-0000:01:00.1-net-em2.device 
                  loaded active plugged   Ethernet 10G 4P X520/I350 rNDC

When changed to vfio, with upstream systemd/udev rules they would just 
disappear entirely, but with the driverctl rules they become:

# systemctl |grep 0000:01:00
sys-devices-pci0000:00-0000:00:03.0-0000:01:00.0-vfio.device 
                  loaded active plugged 
/sys/devices/pci0000:00/0000:00:03.0/0000:01:00.0/vfio
sys-devices-pci0000:00-0000:00:03.0-0000:01:00.1-vfio.device 
                  loaded active plugged 
/sys/devices/pci0000:00/0000:00:03.0/0000:01:00.1/vfio

	- Panu -

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [dpdk-dev] Permanently binding NIC ports with DPDK drivers
  2015-11-12 12:57   ` Panu Matilainen
@ 2015-11-13 11:44     ` Montorsi, Francesco
  2015-11-13 12:17       ` Panu Matilainen
  2015-11-13 14:42     ` Aaron Conole
  1 sibling, 1 reply; 14+ messages in thread
From: Montorsi, Francesco @ 2015-11-13 11:44 UTC (permalink / raw)
  To: Panu Matilainen, Bruce Richardson; +Cc: dev

Hi Panu,

> -----Original Message-----
> From: Panu Matilainen [mailto:pmatilai@redhat.com]
> I've been looking into this recently, here's what I have so far:
> http://laiskiainen.org/git/?p=driverctl.git
> 
Thanks I tried the script (I just had to change /bin/sh into /bin/bash on first line) and it works fine on my Ubuntu 14.04.
Unfortunately I'm forced to run my sw on SLES 11.3 (which btw uses kernel 3.0.76 which I don't know if is compatible with VFIO) and there the output of "list-devices" is plain empty. I will try to look at what's happening...

Thanks,
Francesco


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [dpdk-dev] Permanently binding NIC ports with DPDK drivers
  2015-11-12  0:14 ` Mcnamara, John
@ 2015-11-13 11:48   ` Montorsi, Francesco
  2015-11-13 12:24     ` Panu Matilainen
  0 siblings, 1 reply; 14+ messages in thread
From: Montorsi, Francesco @ 2015-11-13 11:48 UTC (permalink / raw)
  To: Mcnamara, John, dev

Hi John,

> -----Original Message-----
> From: Mcnamara, John [mailto:john.mcnamara@intel.com]
> 
> The Ubuntu dpdk package for 15.10 contains system scripts with functions for
> reserving hugepages and binding interfaces on bootup:
> 
> 
>     /etc/dpdk/dpdk.conf
>     /etc/dpdk/interfaces
>     /etc/init.d/dpdk
>     /lib/dpdk/dpdk-init
>     /lib/systemd/system/dpdk.service
>     /sbin/dpdk_nic_bind
>     /usr/bin/testpmd
>     /usr/share/doc/dpdk/README.Debian
>     /usr/share/doc/dpdk/changelog.Debian.gz
>     /usr/share/doc/dpdk/copyright
>     /usr/share/dpdk/tools/cpu_layout.py
>     /usr/share/dpdk/tools/dpdk_nic_bind.py
>     /usr/share/dpdk/tools/setup.sh
>     /usr/share/python/runtime.d/dpdk.rtupdate
> 
>     http://packages.ubuntu.com/wily/amd64/dpdk/filelist
> 
> If you have the latest version of Ubuntu you can check that out or else
> download and extract the files from the .deb to see how they do it.
> 

This certainly looks very useful. I inspected the package and the files you mentioned and indeed it looks like a good way to go, specially if Ubuntu distribution is moving in that direction (hopefully other distros will follow too).

Thanks a lot!

Francesco

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [dpdk-dev] Permanently binding NIC ports with DPDK drivers
  2015-11-13 11:44     ` Montorsi, Francesco
@ 2015-11-13 12:17       ` Panu Matilainen
  0 siblings, 0 replies; 14+ messages in thread
From: Panu Matilainen @ 2015-11-13 12:17 UTC (permalink / raw)
  To: Montorsi, Francesco, Bruce Richardson; +Cc: dev

On 11/13/2015 01:44 PM, Montorsi, Francesco wrote:
> Hi Panu,
>
>> -----Original Message-----
>> From: Panu Matilainen [mailto:pmatilai@redhat.com]
>> I've been looking into this recently, here's what I have so far:
>> http://laiskiainen.org/git/?p=driverctl.git
>>
> Thanks I tried the script (I just had to change /bin/sh into
> /bin/bash on first line) and it works fine on my Ubuntu 14.04.

Oops, fixed now. Thanks for pointing that out, the line between sh and 
bash is (too) blurry on Fedora & friends.

> Unfortunately I'm forced to run my sw on SLES 11.3 (which btw uses
> kernel 3.0.76 which I don't know if is compatible with VFIO) and there
> the output of "list-devices" is plain empty. I will try to look at
> what's happening...

list-devices currently only lists devices which support driver 
overriding[*], which is a relatively new kernel feature and would be 
missing from 3.0.76:
https://www.redhat.com/archives/libvir-list/2014-April/msg00222.html

It might well be possible to fall back to the "old style" bind system if 
the kernel doesn't support driver_override. I've no immediate plans to 
look into that but lets just say patches would at least be considered :)

	- Panu -

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [dpdk-dev] Permanently binding NIC ports with DPDK drivers
  2015-11-13 11:48   ` Montorsi, Francesco
@ 2015-11-13 12:24     ` Panu Matilainen
  0 siblings, 0 replies; 14+ messages in thread
From: Panu Matilainen @ 2015-11-13 12:24 UTC (permalink / raw)
  To: Montorsi, Francesco, Mcnamara, John, dev

On 11/13/2015 01:48 PM, Montorsi, Francesco wrote:
> Hi John,
>
>> -----Original Message-----
>> From: Mcnamara, John [mailto:john.mcnamara@intel.com]
>>
>> The Ubuntu dpdk package for 15.10 contains system scripts with functions for
>> reserving hugepages and binding interfaces on bootup:
>>
>>
>>      /etc/dpdk/dpdk.conf
>>      /etc/dpdk/interfaces
>>      /etc/init.d/dpdk
>>      /lib/dpdk/dpdk-init
>>      /lib/systemd/system/dpdk.service
>>      /sbin/dpdk_nic_bind
>>      /usr/bin/testpmd
>>      /usr/share/doc/dpdk/README.Debian
>>      /usr/share/doc/dpdk/changelog.Debian.gz
>>      /usr/share/doc/dpdk/copyright
>>      /usr/share/dpdk/tools/cpu_layout.py
>>      /usr/share/dpdk/tools/dpdk_nic_bind.py
>>      /usr/share/dpdk/tools/setup.sh
>>      /usr/share/python/runtime.d/dpdk.rtupdate
>>
>>      http://packages.ubuntu.com/wily/amd64/dpdk/filelist
>>
>> If you have the latest version of Ubuntu you can check that out or else
>> download and extract the files from the .deb to see how they do it.
>>
>
> This certainly looks very useful. I inspected the package and the
> files you mentioned and indeed it looks like a good way to go, specially
> if Ubuntu distribution is moving in that direction (hopefully other
> distros will follow too).
>
> Thanks a lot!

The downside of the Ubuntu-approach is that the setup is fairly static 
and does not support automatically binding hotplugged devices, AFAICS.
Whether that matters to your use-case is of course en entirely different 
question.

	- Panu -

> Francesco
>
>

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [dpdk-dev] Permanently binding NIC ports with DPDK drivers
  2015-11-12 12:57   ` Panu Matilainen
  2015-11-13 11:44     ` Montorsi, Francesco
@ 2015-11-13 14:42     ` Aaron Conole
  1 sibling, 0 replies; 14+ messages in thread
From: Aaron Conole @ 2015-11-13 14:42 UTC (permalink / raw)
  To: Panu Matilainen; +Cc: dev

Panu Matilainen <pmatilai@redhat.com> writes:
> On 11/11/2015 06:28 PM, Bruce Richardson wrote:
>> On Wed, Nov 11, 2015 at 04:13:01PM +0000, Montorsi, Francesco wrote:
>>> Hi,
>>> Is there a way to permanently (i.e., have the configuration automatically applied after reboot) bind a NIC port to DPDK?
>>>
>>> In case there's none, I'm thinking to save in my software a list of the NIC ports chosen by the user for use with DPDK and then, upon software startup to just do
>>>      for (int i=0; i < ...; i++)
>>>       system("dpdk_nic_bind.py --bind=igb_uio " + PCI_device_chosen[i]);
>>> Do you see any problem with that?
>>>
>>> Thanks!
>>> Francesco Montorsi
>>>
>>
>> Hi Francesco,
>>
>> I'm not aware of any way to make the bindings permanent across
>> reboots. What you
>> have suggested will work, but there are probably better ways to do
>> the same thing.
>> For example, a couple of lines in an rc.local script can reapply the
>> bindings at
>> boot for you. I'm sure others can suggest other ways of having the
>> same effect,
>> for example, there may be a way to automatically do this using udev or systemd
>> or some such package.
>
> I've been looking into this recently, here's what I have so far:
> http://laiskiainen.org/git/?p=driverctl.git
>

I've been using this recently, on my systems.

+1 for driverctl :-)

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [dpdk-dev] Permanently binding NIC ports with DPDK drivers
  2016-08-25  4:57 Keren Hochman
@ 2016-08-25 11:01 ` Ferruh Yigit
  0 siblings, 0 replies; 14+ messages in thread
From: Ferruh Yigit @ 2016-08-25 11:01 UTC (permalink / raw)
  To: Keren Hochman, dev; +Cc: Panu Matilainen

On 8/25/2016 5:57 AM, Keren Hochman wrote:
> Hi,
> I there a way to permanently bind a nic port when using DPDK drier with
> kernel < 3.6 ? (In this kernel VFIO driver is not supported)?
> Thanks, Keren
> 

There was a tool from Panu for this purpose:
http://dpdk.org/ml/archives/dev/2015-December/029500.html

Regards,
ferruh

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [dpdk-dev] Permanently binding NIC ports with DPDK drivers
@ 2016-08-25  4:57 Keren Hochman
  2016-08-25 11:01 ` Ferruh Yigit
  0 siblings, 1 reply; 14+ messages in thread
From: Keren Hochman @ 2016-08-25  4:57 UTC (permalink / raw)
  To: dev

Hi,
I there a way to permanently bind a nic port when using DPDK drier with
kernel < 3.6 ? (In this kernel VFIO driver is not supported)?
Thanks, Keren

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2016-08-25 11:01 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-11-11 16:13 [dpdk-dev] Permanently binding NIC ports with DPDK drivers Montorsi, Francesco
2015-11-11 16:28 ` Bruce Richardson
2015-11-11 16:59   ` Montorsi, Francesco
2015-11-11 17:29     ` Stephen Hemminger
2015-11-12 12:57   ` Panu Matilainen
2015-11-13 11:44     ` Montorsi, Francesco
2015-11-13 12:17       ` Panu Matilainen
2015-11-13 14:42     ` Aaron Conole
2015-11-11 19:01 ` Matthew Hall
2015-11-12  0:14 ` Mcnamara, John
2015-11-13 11:48   ` Montorsi, Francesco
2015-11-13 12:24     ` Panu Matilainen
2016-08-25  4:57 Keren Hochman
2016-08-25 11:01 ` Ferruh Yigit

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).