Running DPDK in a LXC-based Container ================================================================================ Author: Krishnamurthy Jambur Note: This README assumes you have the a LXC based container running on the IA platform. This was tested on Windriver OVP Linux 6.0 In this README we are will go through some of the unique steps involved in attaching a NIC to the LXC-based Container and the running a sample DPDP application in the container. 1. Attaching NIC device to the container ---------------------------------------- Let's assume for this example that you would like to have a management interface and then separate dedicated 10G NIC to run DPDK. With this requirement your LXC config file should have one networking sections one for veth. For the 10G NIC we need to use uio device file system veth uses a paired set of network interfaces, one inside the container specified by the option lxc.network.name, and one in the host specified by the option lxc.network.pair. The latter is attached automatically to a bridge in the host if the name of the bridge is specified by the option lxc.network.link. # networking-veth lxc.network.type = veth lxc.network.link = virbr0 lxc.network.veth.pair = vc0 lxc.network.name = eth0 lxc.network.hwaddr = ac:de:48:34:01:01 lxc.network.flags = up lxc.network.ipv4 = 192.168.122.12/24 lxc.network.ipv4.gateway = 192.168.122.1 once you have this your LXC container will have an IP using bridge. Next step is to install DPDK igb_uio on the host and then bind it to the the 10G Niantic NIC ports as usual. Once installed do ls -l /dev/uio* on the host. These are the uioX interface corresponding to the the NIC ports to which DPDK igb_uio is binded. DPDK application will use these device files. In order to provide access to these device files inside the LXC container not down the major number [e.g. in this case 249] and add this config to the LXC config file lxc.cgroup.devices.allow = c 249:* rwm once you start the container then you need to create these uio0, uio1,... devices nodes inside the container file system using mknod command as usual with the same major and minor numbers. mknod /dev/uio3 c 249 0 mknod /dev/uio3 c 249 3 mknod /dev/uio2 c 249 2 mknod /dev/uio1 c 249 1 2. Host allocated Huge page access inside container ---------------------------------------------------- For this set it is assumes that you would have pre-allocated hugepages and mounted on the on the host. Once you have this set-up then you need to provide access to hugetlbfs file system to the container using the LXC config file. lxc.mount.entry = /mnt/huge mnt/huge none bind,create=dir 0 0 once you have this DPDK application will be able to use the host mounted huge pages in the container. Next step is run the DPDK application as usual in the LXC Container Regards, Krishna -------------------------------------------------------------- Intel Shannon Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 Business address: Dromore House, East Park, Shannon, Co. Clare This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Here is the docker step, it is quite straightforward. Basically they are the same as LXC.
UIO:
docker run -i -t --device=/dev/uio0:/dev/uio0 --device=/dev/uio1:/dev/uio1 -v /mnt:/mnt fedora20-dpdk-2 /bin/bash
VFIO:
docker run -i -t --privileged --device=/dev/vfio/vfio:/dev/vfio/vfio --device=/dev/vfio/22:/dev/vfio/22 -v /mnt:/mnt -v /var/run:/var/run fedora20-dpdk-2 /bin/bash
There are open/ioctl calls on vfio device files, and I haven't figured out exactly the capability we need to give to docker for each sys call, otherwise we could remove --privilege option.
MP process:
With /var/run or /var/run/.rte_* mounted to docker, we could run multiple process between dockers.
(We could generate the dpdk cfg files in /var/run/dpdk, then we could mount whole directories to docker without specify each file).
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Murthy, Krishna J
> Sent: Wednesday, October 08, 2014 6:44 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] [README]:Running DPDK in a LXC-based Container
>
> Running DPDK in a LXC-based Container
> =================================================================
> ===============
> Author: Krishnamurthy Jambur
>
> Note: This README assumes you have the a LXC based container running on
> the IA platform. This was tested on Windriver OVP Linux 6.0
>
> In this README we are will go through some of the unique steps involved
> in attaching a NIC to the LXC-based Container and the running a sample
> DPDP application in the container.
>
> 1. Attaching NIC device to the container
> ----------------------------------------
> Let's assume for this example that you would like to have a management
> interface
> and then separate dedicated 10G NIC to run DPDK. With this requirement your
> LXC
> config file should have one networking sections one for veth. For the 10G NIC
> we
> need to use uio device file system
>
> veth uses a paired set of network interfaces, one inside the container specified
> by the option lxc.network.name, and one in the host specified by the option
> lxc.network.pair. The latter is attached automatically to a bridge in the host
> if the name of the bridge is specified by the option lxc.network.link.
>
> # networking-veth
> lxc.network.type = veth
> lxc.network.link = virbr0
> lxc.network.veth.pair = vc0
> lxc.network.name = eth0
> lxc.network.hwaddr = ac:de:48:34:01:01
> lxc.network.flags = up
> lxc.network.ipv4 = 192.168.122.12/24
> lxc.network.ipv4.gateway = 192.168.122.1
>
> once you have this your LXC container will have an IP using bridge. Next step
> is to install DPDK igb_uio on the host and then bind it to the the 10G Niantic
> NIC ports as usual. Once installed do ls -l /dev/uio* on the host. These are the
> uioX interface corresponding to the the NIC ports to which DPDK igb_uio is
> binded. DPDK application will use these device files. In order to provide access
> to these device files inside the LXC container not down the major number [e.g.
> in this case 249] and add this config to the LXC config file
>
> lxc.cgroup.devices.allow = c 249:* rwm
>
> once you start the container then you need to create these uio0, uio1,...
> devices nodes inside the container file system using mknod command as usual
> with the same major and minor numbers.
>
> mknod /dev/uio3 c 249 0
> mknod /dev/uio3 c 249 3
> mknod /dev/uio2 c 249 2
> mknod /dev/uio1 c 249 1
>
> 2. Host allocated Huge page access inside container
> ----------------------------------------------------
> For this set it is assumes that you would have pre-allocated hugepages and
> mounted on the on the host. Once you have this set-up then you need to provide
> access to hugetlbfs file system to the container using the LXC config file.
>
> lxc.mount.entry = /mnt/huge mnt/huge none bind,create=dir 0 0
>
> once you have this DPDK application will be able to use the host mounted huge
> pages in the container.
>
> Next step is run the DPDK application as usual in the LXC Container
>
> Regards,
> Krishna
>
> --------------------------------------------------------------
> Intel Shannon Limited
> Registered in Ireland
> Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
> Registered Number: 308263
> Business address: Dromore House, East Park, Shannon, Co. Clare
>
> This e-mail and any attachments may contain confidential material for the sole
> use of the intended recipient(s). Any review or distribution by others is strictly
> prohibited. If you are not the intended recipient, please contact the sender and
> delete all copies.
>
Hi All, I am trying to run DPDK application on Ubuntu based Conatiner, I had followed the steps as described by "Krishnamurthy Jambur" in his earlier post lxc.cgroup.devices.allow = c 249:* rwm mknod /dev/uio0 c 249 0 lxc.mount.entry = /mnt/huge mnt/huge none bind,create=dir 0 0 But after starting the application inside container I am getting the below error EAL: Master lcore 4 is ready (tid=7491b900;cpuset=[4]) PMD: ENICPMD trace: rte_enic_pmd_init EAL: PCI device 0000:02:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:02:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:02:00.2 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Cannot open /sys/class/uio/uio0/device/config: Permission denied EAL: Error - exiting with code: 1 Cause: Requested device 0000:02:00.2 cannot be used Any pointers to resolve this error? -- Thanks & Regards, Mohit Prakash Saxena Citrix R & D India Pvt. Ltd. Mb: +91 90368 98569 <#UNIQUE_ID_SafeHtmlFilter_> Email: mohit.saxena@citrix.com <mohitprakash.saxena@citrix.com>
Hi Krishna, Off-late I have been investigating DPDK on Linux containers.. I came across this thread and found it very helpful.. But I still find some issues when I try to run the same DPDK application simultaneously on more than one LXC containers.. Even when following the example provided in this thread. I am unable to run the same application simultaneously in both host and more than one LXC containers.. Is there any way to achieve the same.. The basic issue I see in the DPDK design that kit assumes that it will use /mnt/huge/ file-path before using it.. making it unavailable to any other process.. Can this be changed to use the sub-directory /mnt/huge/$(uname -n)/<PID>. That way it cannot stop other process running on host/lxc from using the hugepages.. I have donot have much knowledge about how the hugepage filesystem works.. so pardon my ignorance... :) Looking forward to some help on this. Regards, -Pushpasis
Did you try with the --file-prefix testpmd option?
Just use different file-prefix for each DPDK app running.
On Thu, Feb 11, 2016 at 5:01 AM, Pushpasis Sarkar <pushpasis@gmail.com>
wrote:
> Hi Krishna,
>
> Off-late I have been investigating DPDK on Linux containers.. I came across
> this thread and found it very helpful.. But I still find some issues when I
> try to run the same DPDK application simultaneously on more than one LXC
> containers..
>
> Even when following the example provided in this thread. I am unable to run
> the same application simultaneously in both host and more than one LXC
> containers..
> Is there any way to achieve the same..
>
> The basic issue I see in the DPDK design that kit assumes that it will use
> /mnt/huge/ file-path before using it.. making it unavailable to any other
> process.. Can this be changed to use the sub-directory /mnt/huge/$(uname
> -n)/<PID>. That way it cannot stop other process running on host/lxc from
> using the hugepages.. I have donot have much knowledge about how the
> hugepage filesystem works.. so pardon my ignorance... :)
>
> Looking forward to some help on this.
>
> Regards,
> -Pushpasis
>
Hi Alejandro,
Thanks for the response. I tried the --file-prefix option.. But its the
same issue.. :( Looks like the --file-prefix option only determines the
name of the pages to be created under /mnt/huge/.
I will try the --huge-dir option as well and see if I can do anything with
it or not...
Thanks and Regards,
Pushpasis
On Thu, Feb 11, 2016 at 5:34 PM, Alejandro Lucero <
alejandro.lucero@netronome.com> wrote:
> [image: Boxbe] <https://www.boxbe.com/overview> Alejandro Lucero (
> alejandro.lucero@netronome.com) is not on your Guest List
> <https://www.boxbe.com/approved-list?tc_serial=24340454778&tc_rand=1758595516&utm_source=stf&utm_medium=email&utm_campaign=ANNO_MWTP&utm_content=001&token=d5AB60jxM3mp6UGnoaT1GyqDwTwwrUUOmC2KRJv2v5G6Yxmpi6xLug7B0Xnp73E2&key=8KrFUmOVgyyU7e09U%2F3XGcUfMiAzeGV6WqRa35oU89g%3D>
> | Approve sender
> <https://www.boxbe.com/anno?tc_serial=24340454778&tc_rand=1758595516&utm_source=stf&utm_medium=email&utm_campaign=ANNO_MWTP&utm_content=001&token=d5AB60jxM3mp6UGnoaT1GyqDwTwwrUUOmC2KRJv2v5G6Yxmpi6xLug7B0Xnp73E2&key=8KrFUmOVgyyU7e09U%2F3XGcUfMiAzeGV6WqRa35oU89g%3D>
> | Approve domain
> <https://www.boxbe.com/anno?tc_serial=24340454778&tc_rand=1758595516&utm_source=stf&utm_medium=email&utm_campaign=ANNO_MWTP&utm_content=001&dom&token=d5AB60jxM3mp6UGnoaT1GyqDwTwwrUUOmC2KRJv2v5G6Yxmpi6xLug7B0Xnp73E2&key=8KrFUmOVgyyU7e09U%2F3XGcUfMiAzeGV6WqRa35oU89g%3D>
>
> Did you try with the --file-prefix testpmd option?
>
> Just use different file-prefix for each DPDK app running.
>
> On Thu, Feb 11, 2016 at 5:01 AM, Pushpasis Sarkar <pushpasis@gmail.com>
> wrote:
>
>> Hi Krishna,
>>
>> Off-late I have been investigating DPDK on Linux containers.. I came
>> across
>> this thread and found it very helpful.. But I still find some issues when
>> I
>> try to run the same DPDK application simultaneously on more than one LXC
>> containers..
>>
>> Even when following the example provided in this thread. I am unable to
>> run
>> the same application simultaneously in both host and more than one LXC
>> containers..
>> Is there any way to achieve the same..
>>
>> The basic issue I see in the DPDK design that kit assumes that it will use
>> /mnt/huge/ file-path before using it.. making it unavailable to any other
>> process.. Can this be changed to use the sub-directory /mnt/huge/$(uname
>> -n)/<PID>. That way it cannot stop other process running on host/lxc from
>> using the hugepages.. I have donot have much knowledge about how the
>> hugepage filesystem works.. so pardon my ignorance... :)
>>
>> Looking forward to some help on this.
>>
>> Regards,
>> -Pushpasis
>>
>
>
>
On Thu, Feb 11, 2016 at 06:26:10PM +0530, Pushpasis Sarkar wrote: > Hi Alejandro, > > Thanks for the response. I tried the --file-prefix option.. But its the > same issue.. :( Looks like the --file-prefix option only determines the > name of the pages to be created under /mnt/huge/. > > I will try the --huge-dir option as well and see if I can do anything with > it or not... DPDK scans the mounted filesystems to determine the directory where hugetlbfs is mounted, and uses that directory. If you have multiple hugetlbfs mount points then you can use --huge-dir to select just one of them. However, that's probably not what you want, unless you have multiple hugepage sizes (e.g. 1G and 2M) and want one app to use one size, and the other app to use the other size. If you use --file-prefix to specify different prefixes for the different apps, you almost certainly also need to use "-m" or "--socket-mem" flags to limit the amount of memory each process uses. Without that option the first app to run will use all the hugepage memory, leaving none for the other one. Please see also the section "Running Multiple Independent DPDK Applications" section of the Programmer's Guide in the DPDK documentation: http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html#running-multiple-independent-dpdk-applications Regards. /Bruce > > Thanks and Regards, > Pushpasis > > On Thu, Feb 11, 2016 at 5:34 PM, Alejandro Lucero < > alejandro.lucero@netronome.com> wrote: > > > [image: Boxbe] <https://www.boxbe.com/overview> Alejandro Lucero ( > > alejandro.lucero@netronome.com) is not on your Guest List > > <https://www.boxbe.com/approved-list?tc_serial=24340454778&tc_rand=1758595516&utm_source=stf&utm_medium=email&utm_campaign=ANNO_MWTP&utm_content=001&token=d5AB60jxM3mp6UGnoaT1GyqDwTwwrUUOmC2KRJv2v5G6Yxmpi6xLug7B0Xnp73E2&key=8KrFUmOVgyyU7e09U%2F3XGcUfMiAzeGV6WqRa35oU89g%3D> > > | Approve sender > > <https://www.boxbe.com/anno?tc_serial=24340454778&tc_rand=1758595516&utm_source=stf&utm_medium=email&utm_campaign=ANNO_MWTP&utm_content=001&token=d5AB60jxM3mp6UGnoaT1GyqDwTwwrUUOmC2KRJv2v5G6Yxmpi6xLug7B0Xnp73E2&key=8KrFUmOVgyyU7e09U%2F3XGcUfMiAzeGV6WqRa35oU89g%3D> > > | Approve domain > > <https://www.boxbe.com/anno?tc_serial=24340454778&tc_rand=1758595516&utm_source=stf&utm_medium=email&utm_campaign=ANNO_MWTP&utm_content=001&dom&token=d5AB60jxM3mp6UGnoaT1GyqDwTwwrUUOmC2KRJv2v5G6Yxmpi6xLug7B0Xnp73E2&key=8KrFUmOVgyyU7e09U%2F3XGcUfMiAzeGV6WqRa35oU89g%3D> > > > > Did you try with the --file-prefix testpmd option? > > > > Just use different file-prefix for each DPDK app running. > > > > On Thu, Feb 11, 2016 at 5:01 AM, Pushpasis Sarkar <pushpasis@gmail.com> > > wrote: > > > >> Hi Krishna, > >> > >> Off-late I have been investigating DPDK on Linux containers.. I came > >> across > >> this thread and found it very helpful.. But I still find some issues when > >> I > >> try to run the same DPDK application simultaneously on more than one LXC > >> containers.. > >> > >> Even when following the example provided in this thread. I am unable to > >> run > >> the same application simultaneously in both host and more than one LXC > >> containers.. > >> Is there any way to achieve the same.. > >> > >> The basic issue I see in the DPDK design that kit assumes that it will use > >> /mnt/huge/ file-path before using it.. making it unavailable to any other > >> process.. Can this be changed to use the sub-directory /mnt/huge/$(uname > >> -n)/<PID>. That way it cannot stop other process running on host/lxc from > >> using the hugepages.. I have donot have much knowledge about how the > >> hugepage filesystem works.. so pardon my ignorance... :) > >> > >> Looking forward to some help on this. > >> > >> Regards, > >> -Pushpasis > >> > > > > > >
Hi Bruce,
Thanks a lot for pointing it out.. Yes this is exactly what I was facing..
If I have got it right the solution to my problem should be --file-prefix
and -m (--socket-mem) options.
Best Regards,
-Pushpasis
On Thu, Feb 11, 2016 at 6:38 PM, Bruce Richardson <
bruce.richardson@intel.com> wrote:
> [image: Boxbe] <https://www.boxbe.com/overview> Bruce Richardson (
> bruce.richardson@intel.com) is not on your Guest List
> <https://www.boxbe.com/approved-list?tc_serial=24341194751&tc_rand=193289084&utm_source=stf&utm_medium=email&utm_campaign=ANNO_MWTP&utm_content=001&token=pogEWyVcHKeztbfIMXyh7LB6YVKH6X6ZIhzzgOYsWCOwbGS2EFRXhhm5SuzLCNG5&key=zMUKncrojx%2Fo8PerE9lyAO6NLk3c7xLzHKAzeRsfbxk%3D>
> | Approve sender
> <https://www.boxbe.com/anno?tc_serial=24341194751&tc_rand=193289084&utm_source=stf&utm_medium=email&utm_campaign=ANNO_MWTP&utm_content=001&token=pogEWyVcHKeztbfIMXyh7LB6YVKH6X6ZIhzzgOYsWCOwbGS2EFRXhhm5SuzLCNG5&key=zMUKncrojx%2Fo8PerE9lyAO6NLk3c7xLzHKAzeRsfbxk%3D>
> | Approve domain
> <https://www.boxbe.com/anno?tc_serial=24341194751&tc_rand=193289084&utm_source=stf&utm_medium=email&utm_campaign=ANNO_MWTP&utm_content=001&dom&token=pogEWyVcHKeztbfIMXyh7LB6YVKH6X6ZIhzzgOYsWCOwbGS2EFRXhhm5SuzLCNG5&key=zMUKncrojx%2Fo8PerE9lyAO6NLk3c7xLzHKAzeRsfbxk%3D>
>
> On Thu, Feb 11, 2016 at 06:26:10PM +0530, Pushpasis Sarkar wrote:
> > Hi Alejandro,
> >
> > Thanks for the response. I tried the --file-prefix option.. But its the
> > same issue.. :( Looks like the --file-prefix option only determines the
> > name of the pages to be created under /mnt/huge/.
> >
> > I will try the --huge-dir option as well and see if I can do anything
> with
> > it or not...
>
> DPDK scans the mounted filesystems to determine the directory where
> hugetlbfs
> is mounted, and uses that directory. If you have multiple hugetlbfs mount
> points
> then you can use --huge-dir to select just one of them. However, that's
> probably
> not what you want, unless you have multiple hugepage sizes (e.g. 1G and
> 2M) and
> want one app to use one size, and the other app to use the other size.
>
> If you use --file-prefix to specify different prefixes for the different
> apps,
> you almost certainly also need to use "-m" or "--socket-mem" flags to
> limit the
> amount of memory each process uses. Without that option the first app to
> run
> will use all the hugepage memory, leaving none for the other one.
>
> Please see also the section "Running Multiple Independent DPDK
> Applications"
> section of the Programmer's Guide in the DPDK documentation:
>
>
> http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html#running-multiple-independent-dpdk-applications
>
> Regards.
> /Bruce
>
> >
> > Thanks and Regards,
> > Pushpasis
> >
> > On Thu, Feb 11, 2016 at 5:34 PM, Alejandro Lucero <
> > alejandro.lucero@netronome.com> wrote:
> >
> > > [image: Boxbe] <https://www.boxbe.com/overview> Alejandro Lucero (
> > > alejandro.lucero@netronome.com) is not on your Guest List
> > > <
> https://www.boxbe.com/approved-list?tc_serial=24340454778&tc_rand=1758595516&utm_source=stf&utm_medium=email&utm_campaign=ANNO_MWTP&utm_content=001&token=d5AB60jxM3mp6UGnoaT1GyqDwTwwrUUOmC2KRJv2v5G6Yxmpi6xLug7B0Xnp73E2&key=8KrFUmOVgyyU7e09U%2F3XGcUfMiAzeGV6WqRa35oU89g%3D
> >
> > > | Approve sender
> > > <
> https://www.boxbe.com/anno?tc_serial=24340454778&tc_rand=1758595516&utm_source=stf&utm_medium=email&utm_campaign=ANNO_MWTP&utm_content=001&token=d5AB60jxM3mp6UGnoaT1GyqDwTwwrUUOmC2KRJv2v5G6Yxmpi6xLug7B0Xnp73E2&key=8KrFUmOVgyyU7e09U%2F3XGcUfMiAzeGV6WqRa35oU89g%3D
> >
> > > | Approve domain
> > > <
> https://www.boxbe.com/anno?tc_serial=24340454778&tc_rand=1758595516&utm_source=stf&utm_medium=email&utm_campaign=ANNO_MWTP&utm_content=001&dom&token=d5AB60jxM3mp6UGnoaT1GyqDwTwwrUUOmC2KRJv2v5G6Yxmpi6xLug7B0Xnp73E2&key=8KrFUmOVgyyU7e09U%2F3XGcUfMiAzeGV6WqRa35oU89g%3D
> >
> > >
> > > Did you try with the --file-prefix testpmd option?
> > >
> > > Just use different file-prefix for each DPDK app running.
> > >
> > > On Thu, Feb 11, 2016 at 5:01 AM, Pushpasis Sarkar <pushpasis@gmail.com
> >
> > > wrote:
> > >
> > >> Hi Krishna,
> > >>
> > >> Off-late I have been investigating DPDK on Linux containers.. I came
> > >> across
> > >> this thread and found it very helpful.. But I still find some issues
> when
> > >> I
> > >> try to run the same DPDK application simultaneously on more than one
> LXC
> > >> containers..
> > >>
> > >> Even when following the example provided in this thread. I am unable
> to
> > >> run
> > >> the same application simultaneously in both host and more than one LXC
> > >> containers..
> > >> Is there any way to achieve the same..
> > >>
> > >> The basic issue I see in the DPDK design that kit assumes that it
> will use
> > >> /mnt/huge/ file-path before using it.. making it unavailable to any
> other
> > >> process.. Can this be changed to use the sub-directory
> /mnt/huge/$(uname
> > >> -n)/<PID>. That way it cannot stop other process running on host/lxc
> from
> > >> using the hugepages.. I have donot have much knowledge about how the
> > >> hugepage filesystem works.. so pardon my ignorance... :)
> > >>
> > >> Looking forward to some help on this.
> > >>
> > >> Regards,
> > >> -Pushpasis
> > >>
> > >
> > >
> > >
>
>