[dpdk-users] VPP / DPDK issue with failsafe adapters

[dpdk-users] VPP / DPDK issue with failsafe adapters

[Chris King]

Hello,

I have been reaching out for support for a VPP / DPDK issue on the fd.io
forums here:
https://lists.fd.io/g/vpp-dev/topic/interface_activation_problem/69987045

Since my problem may be DPDK related (unsure) or at the boundary between
VPP and DPDK, I thought I would reach out here as well.

Here is the latest context and any insights would be appreciated:

Scenario: I am having difficulty getting my VPP interfaces activating
properly. I am trying to use DPDK-enabled netvsc devices with VPP in Ubuntu
18.04 in Azure.

Results: I have had a fair bit of success initializing DPDK directly on
this same box on the same adapters. Both of the following testpmd commands
work for me and the interfaces are attached to properly:
sudo ./testpmd -l 0-3 -n 4 --vdev='net_failsafe0,dev(0002:00:02.0)'
--vdev='net_failsafe1,dev(0003:00:02.0)' -b 0001:00:02.0 -- -i
sudo ./testpmd -l 0-3 -n 4 --vdev='net_vdev_netvsc0,iface=eth1'
--vdev='net_vdev_netvsc1,iface=eth2' -b 0001:00:02.0 -- -i

Now, my DPDK devices are configured like this in the VPP config:
dev 0002:00:02.0
  vdev net_vdev_netvsc0,iface=eth1

dev 0003:00:02.0
  vdev net_vdev_netvsc1,iface=eth2

And when I try to start VPP I see the following EAL parameters and failure
to attach to the devices:
2020/01/23 19:02:45:346 warn       dpdk       EAL init args: -c 2 -n 4
--in-memory --vdev net_vdev_netvsc0,iface=eth1 --vdev
net_vdev_netvsc1,iface=eth2 --file-prefix vpp -w 0002:00:02.0 -w
0003:00:02.0 --master-lcore 1
2020/01/23 19:02:45:570 warn       dpdk       unsupported rx offloads
requested on port 0: jumbo-frame
2020/01/23 19:02:45:572 warn       dpdk       unsupported rx offloads
requested on port 2: jumbo-frame
2020/01/23 19:02:45:574 notice     dpdk       EAL: Detected 4 lcore(s)
2020/01/23 19:02:45:574 notice     dpdk       EAL: Detected 1 NUMA nodes
2020/01/23 19:02:45:574 notice     dpdk       EAL: No available hugepages
reported in hugepages-1048576kB
2020/01/23 19:02:45:574 notice     dpdk       EAL: No free hugepages
reported in hugepages-1048576kB
2020/01/23 19:02:45:574 notice     dpdk       EAL: No available hugepages
reported in hugepages-1048576kB
2020/01/23 19:02:45:574 notice     dpdk       EAL: Probing VFIO support...
2020/01/23 19:02:45:574 notice     dpdk       EAL: WARNING! Base virtual
address hint (0xa80001000 != 0x7f0ac0000000) not respected!
2020/01/23 19:02:45:574 notice     dpdk       EAL:    This may cause issues
with mapping memory into secondary processes
2020/01/23 19:02:45:574 notice     dpdk       EAL: WARNING! Base virtual
address hint (0xc00002000 != 0x7f0280000000) not respected!
2020/01/23 19:02:45:574 notice     dpdk       EAL:    This may cause issues
with mapping memory into secondary processes
2020/01/23 19:02:45:574 notice     dpdk       EAL: WARNING! Base virtual
address hint (0xd80003000 != 0x7efa40000000) not respected!
2020/01/23 19:02:45:574 notice     dpdk       EAL:    This may cause issues
with mapping memory into secondary processes
2020/01/23 19:02:45:574 notice     dpdk       EAL: WARNING! Base virtual
address hint (0xdc0a64000 != 0x7ef63fe00000) not respected!
2020/01/23 19:02:45:574 notice     dpdk       EAL:    This may cause issues
with mapping memory into secondary processes
2020/01/23 19:02:45:574 notice     dpdk       EAL: WARNING! Base virtual
address hint (0xdc16c5000 != 0x7ef23fc00000) not respected!
2020/01/23 19:02:45:574 notice     dpdk       EAL:    This may cause issues
with mapping memory into secondary processes
2020/01/23 19:02:45:574 notice     dpdk       EAL: WARNING! Base virtual
address hint (0xdc2326000 != 0x7eee3fa00000) not respected!
2020/01/23 19:02:45:574 notice     dpdk       EAL:    This may cause issues
with mapping memory into secondary processes
2020/01/23 19:02:45:574 notice     dpdk       EAL: WARNING! Base virtual
address hint (0xdc2f87000 != 0x7eea3f800000) not respected!
2020/01/23 19:02:45:574 notice     dpdk       EAL:    This may cause issues
with mapping memory into secondary processes
2020/01/23 19:02:45:574 notice     dpdk       EAL: WARNING: cpu flags
constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
2020/01/23 19:02:45:574 notice     dpdk       net_vdev_netvsc: probably
using routed NetVSC interface "eth1" (index 3)
2020/01/23 19:02:45:574 notice     dpdk       EAL: Driver cannot attach the
device (0002:00:02.0)
2020/01/23 19:02:45:574 notice     dpdk       EAL: Failed to attach device
on primary process
2020/01/23 19:02:45:574 notice     dpdk       net_failsafe: sub_device 0
probe failed (No such file or directory)
2020/01/23 19:02:45:574 notice     dpdk       tun_alloc(): Rx trigger
disabled: Device or resource busy
2020/01/23 19:02:45:574 notice     dpdk       net_vdev_netvsc: probably
using routed NetVSC interface "eth2" (index 4)
2020/01/23 19:02:45:574 notice     dpdk       EAL: Driver cannot attach the
device (0003:00:02.0)
2020/01/23 19:02:45:574 notice     dpdk       EAL: Failed to attach device
on primary process
2020/01/23 19:02:45:574 notice     dpdk       net_failsafe: sub_device 0
probe failed (File exists)
2020/01/23 19:02:45:574 notice     dpdk       tun_alloc(): Rx trigger
disabled: Device or resource busy
2020/01/23 19:02:45:574 notice     dpdk       EAL:   VFIO support not
initialized
2020/01/23 19:02:45:574 notice     dpdk       EAL: Couldn't map new region
for DMA
2020/01/23 19:02:45:574 notice     dpdk       tun_alloc(): Rx trigger
disabled: Device or resource busy
2020/01/23 19:02:45:574 notice     dpdk       tun_alloc(): Rx trigger
disabled: Device or resource busy
2020/01/23 19:02:47:533 notice     dpdk       EAL: Driver cannot attach the
device (0002:00:02.0)
2020/01/23 19:02:47:533 notice     dpdk       EAL: Failed to attach device
on primary process
2020/01/23 19:02:47:533 notice     dpdk       net_failsafe: sub_device 0
probe failed (No such file or directory)

Does anyone know if there something I can do in the VPP startup.conf to get
DPDK devices to initialize correctly?

Thanks,

Chris

Re: [dpdk-users] VPP / DPDK issue with failsafe adapters

[Stephen Hemminger]

On Thu, 23 Jan 2020 14:25:06 -0500
Chris King <chrisking64@gmail.com> wrote:

> Hello,
> 
> I have been reaching out for support for a VPP / DPDK issue on the fd.io
> forums here:
> https://lists.fd.io/g/vpp-dev/topic/interface_activation_problem/69987045
> 
> Since my problem may be DPDK related (unsure) or at the boundary between
> VPP and DPDK, I thought I would reach out here as well.
> 
> Here is the latest context and any insights would be appreciated:
> 
> Scenario: I am having difficulty getting my VPP interfaces activating
> properly. I am trying to use DPDK-enabled netvsc devices with VPP in Ubuntu
> 18.04 in Azure.
>

Recent versions of VPP decided to do a native Mellanox driver and no longer
use the DPDK Mellanox driver. Since the native Mellanox driver is not visible
to DPDK, the failsafe (and netvsc PMD) can not manage the devices.

I got VPP working with failsafe and native netvsc PMD on earlier release of
VPP. But the VPP developers decided that a native Mellanox driver would avoid
the extra overhead of going from VPP to DPDK. In the process, they broke Azure.

This is a VPP problem (they broke it, they need to fix it). There really isn't
anything in DPDK that can be done to help. If VPP insists on doing their own
Mellanox native driver, then they need to do their own native VMBus driver.