DPDK Netvsc - RNDIS reports VF but device not found, retrying

DPDK Netvsc - RNDIS reports VF but device not found, retrying

[Oleksandr Nahnybida]

[-- Attachment #1: Type: text/plain, Size: 2573 bytes --]

Hello,

I am trying to set up dpdk with netvsc as master pmd on Azure following
https://learn.microsoft.com/en-us/azure/virtual-network/setup-dpdk?tabs=ubuntu
and
https://doc.dpdk.org/guides-22.11/nics/netvsc.html

but I have the following error messages

EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: 8565:00:02.0 (socket 0)
mlx5_net: DV flow is not supported.
mlx5_common: Failed to allocate DevX UAR (BF/NC)
mlx5_common: Failed to allocate UAR.
mlx5_net: Failed to prepare Tx DevX UAR.
mlx5_net: probe of PCI device 8565:00:02.0 aborted after encountering an
error: Operation not permitted
mlx5_common: Failed to load driver mlx5_eth
EAL: Requested device 8565:00:02.0 cannot be used
EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: d377:00:02.0 (socket 0)
mlx5_net: DV flow is not supported.
mlx5_common: Failed to allocate DevX UAR (BF/NC)
mlx5_common: Failed to allocate UAR.
mlx5_net: Failed to prepare Tx DevX UAR.
mlx5_net: probe of PCI device d377:00:02.0 aborted after encountering an
error: Operation not permitted
mlx5_common: Failed to load driver mlx5_eth
EAL: Requested device d377:00:02.0 cannot be used
EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: f97e:00:02.0 (socket 0)
mlx5_net: DV flow is not supported.
mlx5_common: Failed to allocate DevX UAR (BF/NC)
mlx5_common: Failed to allocate UAR.
mlx5_net: Failed to prepare Tx DevX UAR.
mlx5_net: probe of PCI device f97e:00:02.0 aborted after encountering an
error: Operation not permitted
mlx5_common: Failed to load driver mlx5_eth
EAL: Requested device f97e:00:02.0 cannot be used
EAL: Bus (pci) probe failed.
hn_vf_attach(): Couldn't find port for VF
hn_vf_add(): RNDIS reports VF but device not found, retrying
hn_vf_attach(): Couldn't find port for VF
hn_vf_add(): RNDIS reports VF but device not found, retrying
and so on in a loop

Also, with debug logs I see

mlx5_common: DevX read access NIC register=0X9055 failed errno=22 status=0
syndrome=0
mlx5_common: DevX read access NIC register=0X9055 failed errno=22 status=0
syndrome=0
mlx5_common: DevX read access NIC register=0X9055 failed errno=22 status=0
syndrome=0


The DPDK version is 22.11, running on
Linux dpdk0 5.15.0-1053-azure #61~20.04.1-Ubuntu
VM type D8ls v5, accelerated networking is on for two ports

driverctl -b vmbus list-devices

000d3a1c-e0df-000d-3a1c-e0df000d3a1c hv_netvsc
000d3a1e-4573-000d-3a1e-4573000d3a1e uio_hv_generic [*]
000d3a1e-47da-000d-3a1e-47da000d3a1e uio_hv_generic [*]


Any, ideas what I might be doing wrong? I see the same behavior with
testpmd and my app.

[-- Attachment #2: Type: text/html, Size: 3541 bytes --]

Re: DPDK Netvsc - RNDIS reports VF but device not found, retrying

[Stephen Hemminger]

On Tue, 23 Jan 2024 14:34:12 +0200
Oleksandr Nahnybida <oleksandrn@interfacemasters.com> wrote:

> Hello,
> 
> I am trying to set up dpdk with netvsc as master pmd on Azure following
> https://learn.microsoft.com/en-us/azure/virtual-network/setup-dpdk?tabs=ubuntu
> and
> https://doc.dpdk.org/guides-22.11/nics/netvsc.html
> 
> but I have the following error messages
> 
> EAL: VFIO support initialized
> EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: 8565:00:02.0 (socket 0)
> mlx5_net: DV flow is not supported.
> mlx5_common: Failed to allocate DevX UAR (BF/NC)
> mlx5_common: Failed to allocate UAR.
> mlx5_net: Failed to prepare Tx DevX UAR.
> mlx5_net: probe of PCI device 8565:00:02.0 aborted after encountering an
> error: Operation not permitted
> mlx5_common: Failed to load driver mlx5_eth
> EAL: Requested device 8565:00:02.0 cannot be used
> EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: d377:00:02.0 (socket 0)
> mlx5_net: DV flow is not supported.
> mlx5_common: Failed to allocate DevX UAR (BF/NC)
> mlx5_common: Failed to allocate UAR.
> mlx5_net: Failed to prepare Tx DevX UAR.
> mlx5_net: probe of PCI device d377:00:02.0 aborted after encountering an
> error: Operation not permitted
> mlx5_common: Failed to load driver mlx5_eth
> EAL: Requested device d377:00:02.0 cannot be used
> EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: f97e:00:02.0 (socket 0)
> mlx5_net: DV flow is not supported.
> mlx5_common: Failed to allocate DevX UAR (BF/NC)
> mlx5_common: Failed to allocate UAR.
> mlx5_net: Failed to prepare Tx DevX UAR.
> mlx5_net: probe of PCI device f97e:00:02.0 aborted after encountering an
> error: Operation not permitted
> mlx5_common: Failed to load driver mlx5_eth
> EAL: Requested device f97e:00:02.0 cannot be used
> EAL: Bus (pci) probe failed.
> hn_vf_attach(): Couldn't find port for VF
> hn_vf_add(): RNDIS reports VF but device not found, retrying
> hn_vf_attach(): Couldn't find port for VF
> hn_vf_add(): RNDIS reports VF but device not found, retrying
> and so on in a loop
> 
> Also, with debug logs I see
> 
> mlx5_common: DevX read access NIC register=0X9055 failed errno=22 status=0
> syndrome=0
> mlx5_common: DevX read access NIC register=0X9055 failed errno=22 status=0
> syndrome=0
> mlx5_common: DevX read access NIC register=0X9055 failed errno=22 status=0
> syndrome=0
> 
> 
> The DPDK version is 22.11, running on
> Linux dpdk0 5.15.0-1053-azure #61~20.04.1-Ubuntu
> VM type D8ls v5, accelerated networking is on for two ports
> 
> driverctl -b vmbus list-devices
> 
> 000d3a1c-e0df-000d-3a1c-e0df000d3a1c hv_netvsc
> 000d3a1e-4573-000d-3a1e-4573000d3a1e uio_hv_generic [*]
> 000d3a1e-47da-000d-3a1e-47da000d3a1e uio_hv_generic [*]
> 
> 
> Any, ideas what I might be doing wrong? I see the same behavior with
> testpmd and my app.

Looks like a mlx5 driver issue, which kernel driver are you using?
The netvsc code sees a VF but can't use it because of the mlx5 errors.

Re: DPDK Netvsc - RNDIS reports VF but device not found, retrying

[Oleksandr Nahnybida]

[-- Attachment #1: Type: text/plain, Size: 3720 bytes --]

azureuser@dpdk0:~$ ethtool -i enP47056s2
driver: mlx5_core
version: 5.15.0-1053-azure
firmware-version: 16.30.1284 (MSF0000000012)
expansion-rom-version:
bus-info: b7d0:00:02.0

I also installed rdma-core/libibverbs1 (dpdk was compiled
with  ibverbs_link=shared)

azureuser@dpdk0:~$ apt-cache policy rdma-core
rdma-core:
  Installed: 28.0-1ubuntu1
azureuser@dpdk0:~$ apt-cache policy libibverbs1
libibverbs1:
  Installed: 28.0-1ubuntu1

On Tue, Jan 23, 2024 at 6:32 PM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Tue, 23 Jan 2024 14:34:12 +0200
> Oleksandr Nahnybida <oleksandrn@interfacemasters.com> wrote:
>
> > Hello,
> >
> > I am trying to set up dpdk with netvsc as master pmd on Azure following
> >
> https://learn.microsoft.com/en-us/azure/virtual-network/setup-dpdk?tabs=ubuntu
> > and
> > https://doc.dpdk.org/guides-22.11/nics/netvsc.html
> >
> > but I have the following error messages
> >
> > EAL: VFIO support initialized
> > EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: 8565:00:02.0 (socket
> 0)
> > mlx5_net: DV flow is not supported.
> > mlx5_common: Failed to allocate DevX UAR (BF/NC)
> > mlx5_common: Failed to allocate UAR.
> > mlx5_net: Failed to prepare Tx DevX UAR.
> > mlx5_net: probe of PCI device 8565:00:02.0 aborted after encountering an
> > error: Operation not permitted
> > mlx5_common: Failed to load driver mlx5_eth
> > EAL: Requested device 8565:00:02.0 cannot be used
> > EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: d377:00:02.0 (socket
> 0)
> > mlx5_net: DV flow is not supported.
> > mlx5_common: Failed to allocate DevX UAR (BF/NC)
> > mlx5_common: Failed to allocate UAR.
> > mlx5_net: Failed to prepare Tx DevX UAR.
> > mlx5_net: probe of PCI device d377:00:02.0 aborted after encountering an
> > error: Operation not permitted
> > mlx5_common: Failed to load driver mlx5_eth
> > EAL: Requested device d377:00:02.0 cannot be used
> > EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: f97e:00:02.0 (socket
> 0)
> > mlx5_net: DV flow is not supported.
> > mlx5_common: Failed to allocate DevX UAR (BF/NC)
> > mlx5_common: Failed to allocate UAR.
> > mlx5_net: Failed to prepare Tx DevX UAR.
> > mlx5_net: probe of PCI device f97e:00:02.0 aborted after encountering an
> > error: Operation not permitted
> > mlx5_common: Failed to load driver mlx5_eth
> > EAL: Requested device f97e:00:02.0 cannot be used
> > EAL: Bus (pci) probe failed.
> > hn_vf_attach(): Couldn't find port for VF
> > hn_vf_add(): RNDIS reports VF but device not found, retrying
> > hn_vf_attach(): Couldn't find port for VF
> > hn_vf_add(): RNDIS reports VF but device not found, retrying
> > and so on in a loop
> >
> > Also, with debug logs I see
> >
> > mlx5_common: DevX read access NIC register=0X9055 failed errno=22
> status=0
> > syndrome=0
> > mlx5_common: DevX read access NIC register=0X9055 failed errno=22
> status=0
> > syndrome=0
> > mlx5_common: DevX read access NIC register=0X9055 failed errno=22
> status=0
> > syndrome=0
> >
> >
> > The DPDK version is 22.11, running on
> > Linux dpdk0 5.15.0-1053-azure #61~20.04.1-Ubuntu
> > VM type D8ls v5, accelerated networking is on for two ports
> >
> > driverctl -b vmbus list-devices
> >
> > 000d3a1c-e0df-000d-3a1c-e0df000d3a1c hv_netvsc
> > 000d3a1e-4573-000d-3a1e-4573000d3a1e uio_hv_generic [*]
> > 000d3a1e-47da-000d-3a1e-47da000d3a1e uio_hv_generic [*]
> >
> >
> > Any, ideas what I might be doing wrong? I see the same behavior with
> > testpmd and my app.
>
> Looks like a mlx5 driver issue, which kernel driver are you using?
> The netvsc code sees a VF but can't use it because of the mlx5 errors.
>

[-- Attachment #2: Type: text/html, Size: 4682 bytes --]

Re: DPDK Netvsc - RNDIS reports VF but device not found, retrying

[Stephen Hemminger]

On Tue, 23 Jan 2024 19:51:33 +0200
Oleksandr Nahnybida <oleksandrn@interfacemasters.com> wrote:

> azureuser@dpdk0:~$ ethtool -i enP47056s2
> driver: mlx5_core
> version: 5.15.0-1053-azure
> firmware-version: 16.30.1284 (MSF0000000012)
> expansion-rom-version:
> bus-info: b7d0:00:02.0
> 
> I also installed rdma-core/libibverbs1 (dpdk was compiled
> with  ibverbs_link=shared)
> 
> azureuser@dpdk0:~$ apt-cache policy rdma-core
> rdma-core:
>   Installed: 28.0-1ubuntu1
> azureuser@dpdk0:~$ apt-cache policy libibverbs1
> libibverbs1:
>   Installed: 28.0-1ubuntu1

Sorry, my expertise on all the variations of mlx5 driver stuff is
limited, and no longer have free access to Azure.

Re: DPDK Netvsc - RNDIS reports VF but device not found, retrying

[Oleksandr Nahnybida]

[-- Attachment #1: Type: text/plain, Size: 1035 bytes --]

No worries,

maybe you have a combination of VM type/kernel version/mlx5(or mlx4)
drivers that worked for you on Azure, just want to make dpdk work on Azure
in any capacity to begin with.

On Tue, Jan 23, 2024 at 8:13 PM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Tue, 23 Jan 2024 19:51:33 +0200
> Oleksandr Nahnybida <oleksandrn@interfacemasters.com> wrote:
>
> > azureuser@dpdk0:~$ ethtool -i enP47056s2
> > driver: mlx5_core
> > version: 5.15.0-1053-azure
> > firmware-version: 16.30.1284 (MSF0000000012)
> > expansion-rom-version:
> > bus-info: b7d0:00:02.0
> >
> > I also installed rdma-core/libibverbs1 (dpdk was compiled
> > with  ibverbs_link=shared)
> >
> > azureuser@dpdk0:~$ apt-cache policy rdma-core
> > rdma-core:
> >   Installed: 28.0-1ubuntu1
> > azureuser@dpdk0:~$ apt-cache policy libibverbs1
> > libibverbs1:
> >   Installed: 28.0-1ubuntu1
>
> Sorry, my expertise on all the variations of mlx5 driver stuff is
> limited, and no longer have free access to Azure.
>

[-- Attachment #2: Type: text/html, Size: 1490 bytes --]

Re: DPDK Netvsc - RNDIS reports VF but device not found, retrying

[Stephen Hemminger]

On Tue, 23 Jan 2024 21:06:22 +0200
Oleksandr Nahnybida <oleksandrn@interfacemasters.com> wrote:

> > >
> > > I also installed rdma-core/libibverbs1 (dpdk was compiled
> > > with  ibverbs_link=shared)

Try without ibverbs_link shared? Maybe 

Mlx5 maintainers are:

M: Dariusz Sosnowski <dsosnowski@nvidia.com>
M: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
M: Ori Kam <orika@nvidia.com>
M: Suanming Mou <suanmingm@nvidia.com>
M: Matan Azrad <matan@nvidia.com>

Re: DPDK Netvsc - RNDIS reports VF but device not found, retrying

[Oleksandr Nahnybida]

[-- Attachment #1: Type: text/plain, Size: 720 bytes --]

Update:
The issue disappeared when moved to the 6.2.0-1018-azure kernel and newer
rdma/ibverbs versions (28->39).

On Wed, Jan 24, 2024 at 3:33 AM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Tue, 23 Jan 2024 21:06:22 +0200
> Oleksandr Nahnybida <oleksandrn@interfacemasters.com> wrote:
>
> > > >
> > > > I also installed rdma-core/libibverbs1 (dpdk was compiled
> > > > with  ibverbs_link=shared)
>
> Try without ibverbs_link shared? Maybe
>
> Mlx5 maintainers are:
>
> M: Dariusz Sosnowski <dsosnowski@nvidia.com>
> M: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> M: Ori Kam <orika@nvidia.com>
> M: Suanming Mou <suanmingm@nvidia.com>
> M: Matan Azrad <matan@nvidia.com>
>

[-- Attachment #2: Type: text/html, Size: 1477 bytes --]