[dpdk-users] Mellanox 100G NIC, VF SR-IOV, docker container, EAL could not detect the device

[dpdk-users] Mellanox 100G NIC, VF SR-IOV, docker container, EAL could not detect the device

[Wu, Xiaoban]

Dear DPDK users,


I am trying to use the VF of the Mellanox 100G NIC enabled by SR-IOV. The following is what I have done.


     1. Add "intel_iommu=on iommu=pt" to kernel command line option, update-grub, and reboot

     2. Install the MLNX-OFED, reboot

     3. In default the card is in infiband mode, so I switched it to ethernet mode (in order to run DPDK application) and reboot.

     4. mst start

         mlxconfig -d /dev/mst/mt4115_pciconf0 q #query
         mlxconfig -d /dev/mst/mt4115_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=1
         reboot
     5. echo 1 > /sys/bus/pci/devices/0000\:83\:00.0/mlx5_num_vfs
     6. modprobe vfio-pci
     7. dpdk-devbind.py --status
     8. dpdk-devbind.py -b vfio-pci 0000:83:00.1
     9. ls -al /dev/vfio
     10. docker run -it --privileged --device=/dev/vfio/54:/dev/vfio/54 --device=/dev/vfio/vfio:/dev/vfio/vfio -v /mnt/huge/:/dev/hugepages/ -v /var/run:/var/run  $IMAGEID bash
     11. $dpdkapp -c 0x01 --socket-mem=128,128 --file-prefix="docker1"

However, in the EAL part, it does not list any usable devices
EAL: Detected 12 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
PMD: bnxt_rte_pmd_init() called for (null)
Error, rte_eth_dev_configure() returns negative!

Can anybody please point out any possible solution? Looking forward to your reply. Thanks very much for your help.

Best wishes,
Xiaoban

Re: [dpdk-users] Mellanox 100G NIC, VF SR-IOV, docker container, EAL could not detect the device

[Adrien Mazarguil]

On Tue, May 23, 2017 at 05:39:36AM +0000, Wu, Xiaoban wrote:
> Dear DPDK users,
> 
> 
> I am trying to use the VF of the Mellanox 100G NIC enabled by SR-IOV. The following is what I have done.
> 
> 
>      1. Add "intel_iommu=on iommu=pt" to kernel command line option, update-grub, and reboot
> 
>      2. Install the MLNX-OFED, reboot
> 
>      3. In default the card is in infiband mode, so I switched it to ethernet mode (in order to run DPDK application) and reboot.
> 
>      4. mst start
> 
>          mlxconfig -d /dev/mst/mt4115_pciconf0 q #query
>          mlxconfig -d /dev/mst/mt4115_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=1
>          reboot
>      5. echo 1 > /sys/bus/pci/devices/0000\:83\:00.0/mlx5_num_vfs
>      6. modprobe vfio-pci
>      7. dpdk-devbind.py --status
>      8. dpdk-devbind.py -b vfio-pci 0000:83:00.1
>      9. ls -al /dev/vfio
>      10. docker run -it --privileged --device=/dev/vfio/54:/dev/vfio/54 --device=/dev/vfio/vfio:/dev/vfio/vfio -v /mnt/huge/:/dev/hugepages/ -v /var/run:/var/run  $IMAGEID bash
>      11. $dpdkapp -c 0x01 --socket-mem=128,128 --file-prefix="docker1"
> 
> However, in the EAL part, it does not list any usable devices
> EAL: Detected 12 lcore(s)
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> PMD: bnxt_rte_pmd_init() called for (null)
> Error, rte_eth_dev_configure() returns negative!
> 
> Can anybody please point out any possible solution? Looking forward to your reply. Thanks very much for your help.

Seems like the issue is not related to your mlx5 device. From the above log
it appears that you also have a bnxt device on that system which DPDK
detects and attempts to use as it is running in blacklist mode. Perhaps that
device was not configured properly.

Try to white-list the devices you want to use by explicitly providing their
PCI bus addresses through -w arguments instead.

-- 
Adrien Mazarguil
6WIND

Re: [dpdk-users] Mellanox 100G NIC, VF SR-IOV, docker container, EAL could not detect the device

[Wu, Xiaoban]

Hi Adrien,


Thanks very much for your reply.


However, after the -w option is used, the EAL still could not find the VF devices.

$dpdk-app -c 0x01 --socket-mem=128,128 --file-prefix="docker1" -w 0000:83:00.1


EAL: Detected 12 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
PMD: bnxt_rte_pmd_init() called for (null)
Error, rte_eth_dev_configure() returns negative!

Best wishes,
Xiaoban



________________________________
From: Adrien Mazarguil <adrien.mazarguil@6wind.com>
Sent: Tuesday, May 23, 2017 4:12 AM
To: Wu, Xiaoban
Cc: users@dpdk.org
Subject: Re: [dpdk-users] Mellanox 100G NIC, VF SR-IOV, docker container, EAL could not detect the device

On Tue, May 23, 2017 at 05:39:36AM +0000, Wu, Xiaoban wrote:
> Dear DPDK users,
>
>
> I am trying to use the VF of the Mellanox 100G NIC enabled by SR-IOV. The following is what I have done.
>
>
>      1. Add "intel_iommu=on iommu=pt" to kernel command line option, update-grub, and reboot
>
>      2. Install the MLNX-OFED, reboot
>
>      3. In default the card is in infiband mode, so I switched it to ethernet mode (in order to run DPDK application) and reboot.
>
>      4. mst start
>
>          mlxconfig -d /dev/mst/mt4115_pciconf0 q #query
>          mlxconfig -d /dev/mst/mt4115_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=1
>          reboot
>      5. echo 1 > /sys/bus/pci/devices/0000\:83\:00.0/mlx5_num_vfs
>      6. modprobe vfio-pci
>      7. dpdk-devbind.py --status
>      8. dpdk-devbind.py -b vfio-pci 0000:83:00.1
>      9. ls -al /dev/vfio
>      10. docker run -it --privileged --device=/dev/vfio/54:/dev/vfio/54 --device=/dev/vfio/vfio:/dev/vfio/vfio -v /mnt/huge/:/dev/hugepages/ -v /var/run:/var/run  $IMAGEID bash
>      11. $dpdkapp -c 0x01 --socket-mem=128,128 --file-prefix="docker1"
>
> However, in the EAL part, it does not list any usable devices
> EAL: Detected 12 lcore(s)
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> PMD: bnxt_rte_pmd_init() called for (null)
> Error, rte_eth_dev_configure() returns negative!
>
> Can anybody please point out any possible solution? Looking forward to your reply. Thanks very much for your help.

Seems like the issue is not related to your mlx5 device. From the above log
it appears that you also have a bnxt device on that system which DPDK
detects and attempts to use as it is running in blacklist mode. Perhaps that
device was not configured properly.

Try to white-list the devices you want to use by explicitly providing their
PCI bus addresses through -w arguments instead.

--
Adrien Mazarguil
6WIND

Re: [dpdk-users] Mellanox 100G NIC, VF SR-IOV, docker container, EAL could not detect the device

[Adrien Mazarguil]

On Wed, May 24, 2017 at 01:01:58AM +0000, Wu, Xiaoban wrote:
> Hi Adrien,
> 
> 
> Thanks very much for your reply.
> 
> 
> However, after the -w option is used, the EAL still could not find the VF devices.
> 
> $dpdk-app -c 0x01 --socket-mem=128,128 --file-prefix="docker1" -w 0000:83:00.1
> 
> 
> EAL: Detected 12 lcore(s)
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> PMD: bnxt_rte_pmd_init() called for (null)
> Error, rte_eth_dev_configure() returns negative!

I didn't take Docker into account. Since there is no VM involved, you must
*not* bind the devices to vfio-pci (do not run dpdk-devbind.py, no need to
even modprobe vfio-pci).

With mlx4/mlx5, doing so is only necessary when you intent to assign devices
to a VM, otherwise the spawned VFs must remain managed by their original
kernel drivers. You likely need to transfer the associated netdevices to
your Docker instance for this to work though. You'll likely need to somehow
transfer the related IB instances as well.

At least that's the theory, I've never personally tried to use Docker with
mlx4/mlx5 VFs.

-- 
Adrien Mazarguil
6WIND