Re: [dpdk-users] Correct setup of sfc

[Filip Janiszewski <contact@filipjaniszewski.com>]

Adding here also Rami Rosen to continue just one thread.

First of all thanks for replying, now here's the current status:

It seems that there might be some problems with IOMMU, according to
dmesg logs:

.
root build : dmesg | grep IOMMU
[    0.000000] DMAR: IOMMU enabled
[ 1179.652950] vboxpci: IOMMU not found (not registered)
.

In particular the second line is a clear warning that IOMMU is not
enabled, will try to work this out through the bios options. Also, as
Andrew suggested I've had a look at /sys/class/iommu and it's empty:

.
root build : ll /sys/class/iommu/
total 0
.

Anyway the output of --status-dev net is:

.
root usertools : ./dpdk-devbind.py --status-dev net

Network devices using DPDK-compatible driver
============================================
<none>





Network devices using kernel driver


===================================


0000:00:1f.6 'Ethernet Connection (2) I219-V 15b8' if=enp0s31f6
drv=e1000e unused= *Active*

0000:65:00.0 'SFC9140 10/40G Ethernet Controller 0923' if=enp101s0f0
drv=sfc unused=

0000:65:00.1 'SFC9140 10/40G Ethernet Controller 0923' if=enp101s0f1
drv=sfc unused=

0000:b3:00.0 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f0
drv=mlx5_core unused=

0000:b3:00.1 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f1
drv=mlx5_core unused=




Other Network devices


=====================


<none>
.

I've been able to use the Solarflar device by binding it to the
uio_pci_generic driver:

.
root usertools : ./dpdk-devbind.py --status-dev net

Network devices using DPDK-compatible driver
============================================
0000:65:00.0 'SFC9140 10/40G Ethernet Controller 0923'
drv=uio_pci_generic unused=sfc
0000:65:00.1 'SFC9140 10/40G Ethernet Controller 0923'
drv=uio_pci_generic unused=sfc

Network devices using kernel driver
===================================
0000:00:1f.6 'Ethernet Connection (2) I219-V 15b8' if=enp0s31f6
drv=e1000e unused=uio_pci_generic *Active*
0000:b3:00.0 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f0
drv=mlx5_core unused=uio_pci_generic
0000:b3:00.1 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f1
drv=mlx5_core unused=uio_pci_generic
.

and the card is properly recognized:

.
EAL: PCI device 0000:00:1f.6 on NUMA socket 0
EAL:   probe driver: 8086:15b8 net_e1000_em
EAL: PCI device 0000:65:00.0 on NUMA socket 0
EAL:   probe driver: 1924:923 net_sfc_efx
PMD: sfc_efx 0000:65:00.0 #0: running FW variant is ultra-low-latency
PMD: sfc_efx 0000:65:00.0 #0: use ef10 Rx datapath
PMD: sfc_efx 0000:65:00.0 #0: use ef10 Tx datapath
EAL: PCI device 0000:65:00.1 on NUMA socket 0
EAL:   probe driver: 1924:923 net_sfc_efx
PMD: sfc_efx 0000:65:00.1 #1: running FW variant is ultra-low-latency
PMD: sfc_efx 0000:65:00.1 #1: use ef10 Rx datapath
PMD: sfc_efx 0000:65:00.1 #1: use ef10 Tx datapath
.

On the other hand, attempting to bind to vfio-pci as Rami suggested is
not working:

.
root usertools : ./dpdk-devbind.py --bind=vfio-pci 0000:65:00.0 0000:65:00.1
Error: bind failed for 0000:65:00.0 - Cannot bind to driver vfio-pci
Error: bind failed for 0000:65:00.1 - Cannot bind to driver vfio-pci
.

And as requested, those are the dmesg logs:

.
[ 2146.354434] VFIO - User Level meta-driver version: 0.3
[ 2159.418937] vfio-pci: probe of 0000:65:00.0 failed with error -22
[ 2159.434537] vfio-pci: probe of 0000:65:00.1 failed with error -22
[ 2199.773476] vfio-pci: probe of 0000:65:00.0 failed with error -22
[ 2199.788973] vfio-pci: probe of 0000:65:00.1 failed with error -22
.

Which as far as I understand are referring to IOMMU not being enable on
the machine, I guess I'll have to find some specific option for that as
the BIOS suggest that "Intel Virtualization" is enabled, but that might
refer to what we need here.

Filip

Il 15/06/18 11:44, Andrew Rybchenko ha scritto:
> On 06/13/2018 10:14 PM, Filip Janiszewski wrote:
>> Hi Andrew,
>>
>>
>>> PCI devices of Solarflare NIC should be bound to vfio,
>>> uio-pci-generic or
>>> igb_uio (part of DPDK) module. In the case of Solarflare NICs, Linux
>>> driver is
>>> not required and not used in DPDK.
>>>
>>> So, you should load one of above modules (depending on your server
>>> IOMMU configuration), push already created interfaces down and rebind
>>> Solarflare PCI functions to the driver, something like:
>>>
>>> modprobe vfio-pci
>>> ip link set enp101s0f0 down
>>> ip link set enp101s0f1 down
>>> dpdk-devbind.py --bind=vfio-pci 0000:65:00.0 0000:65:00.1
>>>
>>> The above assumes that dpdk-devbind.py script is in PATH.
>>> And start DPDK as you do before.
>>>
>> For some reason vfio-pci is refusing to bind the device:
>>
>> .
>> root usertools : ./dpdk-devbind.py --bind=vfio_pci 0000:65:00.0
>> 0000:65:00.1
>> Error: bind failed for 0000:65:00.0 - Cannot open
>> /sys/bus/pci/drivers/vfio_pci/bind
>> Error: bind failed for 0000:65:00.1 - Cannot open
>> /sys/bus/pci/drivers/vfio_pci/bind
>> .
>>
>> also:
>>
>> .
>> root usertools : echo 0000:65:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
>> bash: echo: write error: No such device
>> .
>>
>> In the kernel command line I've included 'iommu=pt intel_iommu=on', but
>> still not working. Well I guess this is not a DPDK issue anymore.
> 
> What does the following command shows?
> ./dpdk-devbind.py --status-dev net
> 
> Is IOMMU (VT-d) enabled in BIOS/UEFI? You can check for files/symlinks
> in /sys/class/iommu.
> 
> As an experiment I'd try uio-pci-generic as well.
> Also exact error logs from dmesg could be useful to understand what's
> going on.
> 
> Andrew.

-- 
BR, Filip
+48 666 369 823