Re: [dpdk-users] Correct setup of sfc

[Filip Janiszewski <contact@filipjaniszewski.com>]

Hi,

Apparently there was an additional option to click in the BIOS, now I'm
able to bind to vfio-pci and IOMMU is working correctly:

.
root usertools : ./dpdk-devbind.py --status-dev net

Network devices using DPDK-compatible driver
============================================
0000:65:00.0 'SFC9140 10/40G Ethernet Controller 0923' drv=vfio-pci
unused=sfc
0000:65:00.1 'SFC9140 10/40G Ethernet Controller 0923' drv=vfio-pci
unused=sfc

Network devices using kernel driver
===================================
0000:00:1f.6 'Ethernet Connection (2) I219-V 15b8' if=enp0s31f6
drv=e1000e unused=vfio-pci *Active*
0000:b3:00.0 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f0
drv=mlx5_core unused=vfio-pci
0000:b3:00.1 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f1
drv=mlx5_core unused=vfio-pci

Other Network devices
=====================
<none>

.

also:

.
root build : ll /sys/class/iommu/
total 0
lrwxrwxrwx. 1 root root 0 Jun 15  2018 dmar0 ->
../../devices/virtual/iommu/dmar0
lrwxrwxrwx. 1 root root 0 Jun 15  2018 dmar1 ->
../../devices/virtual/iommu/dmar1
lrwxrwxrwx. 1 root root 0 Jun 15  2018 dmar2 ->
../../devices/virtual/iommu/dmar2
lrwxrwxrwx. 1 root root 0 Jun 15  2018 dmar3 ->
../../devices/virtual/iommu/dmar3

.

Thanks everybody for the support.


Il 15/06/18 19:02, Filip Janiszewski ha scritto:
> Adding here also Rami Rosen to continue just one thread.
> 
> First of all thanks for replying, now here's the current status:
> 
> It seems that there might be some problems with IOMMU, according to
> dmesg logs:
> 
> .
> root build : dmesg | grep IOMMU
> [    0.000000] DMAR: IOMMU enabled
> [ 1179.652950] vboxpci: IOMMU not found (not registered)
> .
> 
> In particular the second line is a clear warning that IOMMU is not
> enabled, will try to work this out through the bios options. Also, as
> Andrew suggested I've had a look at /sys/class/iommu and it's empty:
> 
> .
> root build : ll /sys/class/iommu/
> total 0
> .
> 
> Anyway the output of --status-dev net is:
> 
> .
> root usertools : ./dpdk-devbind.py --status-dev net
> 
> Network devices using DPDK-compatible driver
> ============================================
> <none>
> 
> 
> 
> 
> 
> Network devices using kernel driver
> 
> 
> ===================================
> 
> 
> 0000:00:1f.6 'Ethernet Connection (2) I219-V 15b8' if=enp0s31f6
> drv=e1000e unused= *Active*
> 
> 0000:65:00.0 'SFC9140 10/40G Ethernet Controller 0923' if=enp101s0f0
> drv=sfc unused=
> 
> 0000:65:00.1 'SFC9140 10/40G Ethernet Controller 0923' if=enp101s0f1
> drv=sfc unused=
> 
> 0000:b3:00.0 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f0
> drv=mlx5_core unused=
> 
> 0000:b3:00.1 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f1
> drv=mlx5_core unused=
> 
> 
> 
> 
> Other Network devices
> 
> 
> =====================
> 
> 
> <none>
> .
> 
> I've been able to use the Solarflar device by binding it to the
> uio_pci_generic driver:
> 
> .
> root usertools : ./dpdk-devbind.py --status-dev net
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:65:00.0 'SFC9140 10/40G Ethernet Controller 0923'
> drv=uio_pci_generic unused=sfc
> 0000:65:00.1 'SFC9140 10/40G Ethernet Controller 0923'
> drv=uio_pci_generic unused=sfc
> 
> Network devices using kernel driver
> ===================================
> 0000:00:1f.6 'Ethernet Connection (2) I219-V 15b8' if=enp0s31f6
> drv=e1000e unused=uio_pci_generic *Active*
> 0000:b3:00.0 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f0
> drv=mlx5_core unused=uio_pci_generic
> 0000:b3:00.1 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f1
> drv=mlx5_core unused=uio_pci_generic
> .
> 
> and the card is properly recognized:
> 
> .
> EAL: PCI device 0000:00:1f.6 on NUMA socket 0
> EAL:   probe driver: 8086:15b8 net_e1000_em
> EAL: PCI device 0000:65:00.0 on NUMA socket 0
> EAL:   probe driver: 1924:923 net_sfc_efx
> PMD: sfc_efx 0000:65:00.0 #0: running FW variant is ultra-low-latency
> PMD: sfc_efx 0000:65:00.0 #0: use ef10 Rx datapath
> PMD: sfc_efx 0000:65:00.0 #0: use ef10 Tx datapath
> EAL: PCI device 0000:65:00.1 on NUMA socket 0
> EAL:   probe driver: 1924:923 net_sfc_efx
> PMD: sfc_efx 0000:65:00.1 #1: running FW variant is ultra-low-latency
> PMD: sfc_efx 0000:65:00.1 #1: use ef10 Rx datapath
> PMD: sfc_efx 0000:65:00.1 #1: use ef10 Tx datapath
> .
> 
> On the other hand, attempting to bind to vfio-pci as Rami suggested is
> not working:
> 
> .
> root usertools : ./dpdk-devbind.py --bind=vfio-pci 0000:65:00.0 0000:65:00.1
> Error: bind failed for 0000:65:00.0 - Cannot bind to driver vfio-pci
> Error: bind failed for 0000:65:00.1 - Cannot bind to driver vfio-pci
> .
> 
> And as requested, those are the dmesg logs:
> 
> .
> [ 2146.354434] VFIO - User Level meta-driver version: 0.3
> [ 2159.418937] vfio-pci: probe of 0000:65:00.0 failed with error -22
> [ 2159.434537] vfio-pci: probe of 0000:65:00.1 failed with error -22
> [ 2199.773476] vfio-pci: probe of 0000:65:00.0 failed with error -22
> [ 2199.788973] vfio-pci: probe of 0000:65:00.1 failed with error -22
> .
> 
> Which as far as I understand are referring to IOMMU not being enable on
> the machine, I guess I'll have to find some specific option for that as
> the BIOS suggest that "Intel Virtualization" is enabled, but that might
> refer to what we need here.
> 
> Filip
> 
> Il 15/06/18 11:44, Andrew Rybchenko ha scritto:
>> On 06/13/2018 10:14 PM, Filip Janiszewski wrote:
>>> Hi Andrew,
>>>
>>>
>>>> PCI devices of Solarflare NIC should be bound to vfio,
>>>> uio-pci-generic or
>>>> igb_uio (part of DPDK) module. In the case of Solarflare NICs, Linux
>>>> driver is
>>>> not required and not used in DPDK.
>>>>
>>>> So, you should load one of above modules (depending on your server
>>>> IOMMU configuration), push already created interfaces down and rebind
>>>> Solarflare PCI functions to the driver, something like:
>>>>
>>>> modprobe vfio-pci
>>>> ip link set enp101s0f0 down
>>>> ip link set enp101s0f1 down
>>>> dpdk-devbind.py --bind=vfio-pci 0000:65:00.0 0000:65:00.1
>>>>
>>>> The above assumes that dpdk-devbind.py script is in PATH.
>>>> And start DPDK as you do before.
>>>>
>>> For some reason vfio-pci is refusing to bind the device:
>>>
>>> .
>>> root usertools : ./dpdk-devbind.py --bind=vfio_pci 0000:65:00.0
>>> 0000:65:00.1
>>> Error: bind failed for 0000:65:00.0 - Cannot open
>>> /sys/bus/pci/drivers/vfio_pci/bind
>>> Error: bind failed for 0000:65:00.1 - Cannot open
>>> /sys/bus/pci/drivers/vfio_pci/bind
>>> .
>>>
>>> also:
>>>
>>> .
>>> root usertools : echo 0000:65:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
>>> bash: echo: write error: No such device
>>> .
>>>
>>> In the kernel command line I've included 'iommu=pt intel_iommu=on', but
>>> still not working. Well I guess this is not a DPDK issue anymore.
>>
>> What does the following command shows?
>> ./dpdk-devbind.py --status-dev net
>>
>> Is IOMMU (VT-d) enabled in BIOS/UEFI? You can check for files/symlinks
>> in /sys/class/iommu.
>>
>> As an experiment I'd try uio-pci-generic as well.
>> Also exact error logs from dmesg could be useful to understand what's
>> going on.
>>
>> Andrew.
> 

-- 
BR, Filip
+48 666 369 823