[dpdk-users] Cannot run DPDK applications using Mellanox NIC under Hyper-V

[dpdk-users] Cannot run DPDK applications using Mellanox NIC under Hyper-V

[Liwu Liu]

Hi Team,
    We used to have similar problem for Ubuntu 18.04 hypervisor and resolved by set log_num_mgm_entry_size=-1. (Refer to https://mails.dpdk.org/archives/users/2018-November/003647.html
)

  Somehow for Windows Servers with MLNX VPI, I do not know where to set such and DPDK over MLX4 on linux vm has same failure of attaching flow.

    [  374.568992] <mlx4_ib> __mlx4_ib_create_flow: mcg table is full. Fail to register network rule. size = 64 (out of memory error code)

    Would like to get your help on this. It seems to be the case that the PF interface is not configured to trust VF interfaces to be able to add new flow rules.

   Many thanks for help,

Liwu




***  Please note that this message and any attachments may contain confidential and proprietary material and information and are intended only for the use of the intended recipient(s). If you are not the intended recipient, you are hereby notified that any review, use, disclosure, dissemination, distribution or copying of this message and any attachments is strictly prohibited. If you have received this email in error, please immediately notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed. Please also note that any views, opinions, conclusions or commitments expressed in this message are those of the individual sender and do not necessarily reflect the views of Fortinet, Inc., its affiliates, and emails are not binding on Fortinet and only a writing manually signed by Fortinet's General Counsel can be a binding commitment of Fortinet to Fortinet's customers or partners. Thank you. ***

Re: [dpdk-users] Cannot run DPDK applications using Mellanox NIC under Hyper-V

[Stephen Hemminger]

On Fri, 4 Jan 2019 20:06:48 +0000
Liwu Liu <liwuliu@fortinet.com> wrote:

> Hi Team,
>     We used to have similar problem for Ubuntu 18.04 hypervisor and resolved by set log_num_mgm_entry_size=-1. (Refer to https://mails.dpdk.org/archives/users/2018-November/003647.html
> )
> 
>   Somehow for Windows Servers with MLNX VPI, I do not know where to set such and DPDK over MLX4 on linux vm has same failure of attaching flow.
> 
>     [  374.568992] <mlx4_ib> __mlx4_ib_create_flow: mcg table is full. Fail to register network rule. size = 64 (out of memory error code)
> 
>     Would like to get your help on this. It seems to be the case that the PF interface is not configured to trust VF interfaces to be able to add new flow rules.
> 
>    Many thanks for help,
> 
> Liwu
> 
> 
> 
> 
> ***  Please note that this message and any attachments may contain confidential and proprietary material and information and are intended only for the use of the intended recipient(s). If you are not the intended recipient, you are hereby notified that any review, use, disclosure, dissemination, distribution or copying of this message and any attachments is strictly prohibited. If you have received this email in error, please immediately notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed. Please also note that any views, opinions, conclusions or commitments expressed in this message are those of the individual sender and do not necessarily reflect the views of Fortinet, Inc., its affiliates, and emails are not binding on Fortinet and only a writing manually signed by Fortinet's General Counsel can be a binding commitment of Fortinet to Fortinet's customers or partners. Thank you. *** 
> 

How are using the Mellanox device with Windows Server? PCI passthrough or SR-IOV?

Re: [dpdk-users] Cannot run DPDK applications using Mellanox NIC under Hyper-V

[Liwu Liu]

It is SR-IOV. 
BR,
Liwu

-----Original Message-----
From: Stephen Hemminger <stephen@networkplumber.org> 
Sent: Friday, January 4, 2019 4:07 PM
To: Liwu Liu <liwuliu@fortinet.com>
Cc: users@dpdk.org
Subject: Re: [dpdk-users] Cannot run DPDK applications using Mellanox NIC under Hyper-V

On Fri, 4 Jan 2019 20:06:48 +0000
Liwu Liu <liwuliu@fortinet.com> wrote:

> Hi Team,
>     We used to have similar problem for Ubuntu 18.04 hypervisor and resolved by set log_num_mgm_entry_size=-1. (Refer to https://mails.dpdk.org/archives/users/2018-November/003647.html
> )
> 
>   Somehow for Windows Servers with MLNX VPI, I do not know where to set such and DPDK over MLX4 on linux vm has same failure of attaching flow.
> 
>     [  374.568992] <mlx4_ib> __mlx4_ib_create_flow: mcg table is full. Fail to register network rule. size = 64 (out of memory error code)
> 
>     Would like to get your help on this. It seems to be the case that the PF interface is not configured to trust VF interfaces to be able to add new flow rules.
> 
>    Many thanks for help,
> 
> Liwu
> 
> 
> 
> 
> ***  Please note that this message and any attachments may contain confidential and proprietary material and information and are intended only for the use of the intended recipient(s). If you are not the intended recipient, you are hereby notified that any review, use, disclosure, dissemination, distribution or copying of this message and any attachments is strictly prohibited. If you have received this email in error, please immediately notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed. Please also note that any views, opinions, conclusions or commitments expressed in this message are those of the individual sender and do not necessarily reflect the views of Fortinet, Inc., its affiliates, and emails are not binding on Fortinet and only a writing manually signed by Fortinet's General Counsel can be a binding commitment of Fortinet to Fortinet's customers or partners. Thank you. *** 
> 

How are using the Mellanox device with Windows Server? PCI passthrough or SR-IOV?

Re: [dpdk-users] Cannot run DPDK applications using Mellanox NIC under Hyper-V

[Liwu Liu]

Hi Stephen,

  Yeah we are seeing all NETVSC, failsafe PMD, TAP, MLX4 VF. We are expecting bifurcation mode where default traffic goes with MLX4_EN slave to NETVSC master to enter Linux net stack and certain flow-defined traffic shall go into DPDK RX queue rather than MLX4_EN net-dev queue. Unfortunately as we cannot define flow into VF NIC, we cannot re-direct traffic into DPDK user-land. 

   In fact I am looking for a way to steer all RX traffic from MLX4_EN slave device into DPDK RX ring. 

  Hypervisor: Windows server + MLNX VPI 5.5; VM: Linux dpdk-18.11 mlnx-ofed-kernel-4.4 rdma-core-43mlnx1 

  Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

Best,

Liwu

-----Original Message-----
From: Stephen Hemminger <stephen@networkplumber.org> 
Sent: Friday, January 4, 2019 4:24 PM
To: Liwu Liu <liwuliu@fortinet.com>
Subject: Re: [dpdk-users] Cannot run DPDK applications using Mellanox NIC under Hyper-V

On Sat, 5 Jan 2019 00:10:43 +0000
Liwu Liu <liwuliu@fortinet.com> wrote:

> Hi Stephen,
> 
>    It is SR-IOV
> 
>   Thanks,
> 
> Liwu
> 
> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Friday, January 4, 2019 4:07 PM
> To: Liwu Liu <liwuliu@fortinet.com>
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] Cannot run DPDK applications using Mellanox 
> NIC under Hyper-V
> 
> On Fri, 4 Jan 2019 20:06:48 +0000
> Liwu Liu <liwuliu@fortinet.com> wrote:
> 
> > Hi Team,
> >     We used to have similar problem for Ubuntu 18.04 hypervisor and 
> > resolved by set log_num_mgm_entry_size=-1. (Refer to 
> > https://mails.dpdk.org/archives/users/2018-November/003647.html
> > )
> > 
> >   Somehow for Windows Servers with MLNX VPI, I do not know where to set such and DPDK over MLX4 on linux vm has same failure of attaching flow.
> > 
> >     [  374.568992] <mlx4_ib> __mlx4_ib_create_flow: mcg table is 
> > full. Fail to register network rule. size = 64 (out of memory error 
> > code)
> > 
> >     Would like to get your help on this. It seems to be the case that the PF interface is not configured to trust VF interfaces to be able to add new flow rules.
> > 
> >    Many thanks for help,
> > 
> > Liwu
> > 
> > 
> > 
> > 
> > ***  Please note that this message and any attachments may contain 
> > confidential and proprietary material and information and are 
> > intended only for the use of the intended recipient(s). If you are 
> > not the intended recipient, you are hereby notified that any review, 
> > use, disclosure, dissemination, distribution or copying of this 
> > message and any attachments is strictly prohibited. If you have 
> > received this email in error, please immediately notify the sender 
> > and destroy this e-mail and any attachments and all copies, whether 
> > electronic or printed. Please also note that any views, opinions, 
> > conclusions or commitments expressed in this message are those of 
> > the individual sender and do not necessarily reflect the views of 
> > Fortinet, Inc., its affiliates, and emails are not binding on 
> > Fortinet and only a writing manually signed by Fortinet's General 
> > Counsel can be a binding commitment of Fortinet to Fortinet's 
> > customers or partners. Thank you. ***
> >   
> 
> How are using the Mellanox device with Windows Server? PCI passthrough or SR-IOV?

If using SR-IOV then you can't use the Mellanox device directly. You have to use the synthetic device. If you use vdev_netvsc pseudo-device, then it sets up a failsafe PMD, TAP and the VF (Mellanox) PMD for you.

The experimental way is to use the netvsc PMD which will manage VF if available.

Re: [dpdk-users] Cannot run DPDK applications using Mellanox NIC under Hyper-V

[Stephen Hemminger]

On Sat, 5 Jan 2019 00:35:13 +0000
Liwu Liu <liwuliu@fortinet.com> wrote:

> Hi Stephen,
> 
>   Yeah we are seeing all NETVSC, failsafe PMD, TAP, MLX4 VF. We are expecting bifurcation mode where default traffic goes with MLX4_EN slave to NETVSC master to enter Linux net stack and certain flow-defined traffic shall go into DPDK RX queue rather than MLX4_EN net-dev queue. Unfortunately as we cannot define flow into VF NIC, we cannot re-direct traffic into DPDK user-land. 

This is a little more complex.
You need to use rte_flow on the failsafe PMD and that is supposed to get propogated to TAP and MLX VF.
The TAP PMD programs BPF in kernel, and the MLX VF PMD programs the VF flow tables.

Haven't tried it myself.

Re: [dpdk-users] Cannot run DPDK applications using Mellanox NIC under Hyper-V

[Liwu Liu]

Updates: many thanks for MLNX engineering team, doing the following can resolve this issue:
1-	On power shell: "Set-VMNetworkAdapter -VMName <VM_NAME> -Name "Network Adapter" -IovQueuePairsRequested 64"
2-	In the registry editor, make sure that you increase the ExtraVFsQuotas percentage to be 1000 to use full resources. 
HKEY_LOCAL_MACHINE \SYSTEM\CurrentControlSet\Control\Class\{4d36e97d-e325-11ce-bfc1- 08002be10318}\<nn>\Parameters
3-	BlockFlowsForPerformance registry key:
HKEY_LOCAL_MACHINE \SYSTEM\CurrentControlSet\Control\Class\{4d36e97d-e325-11ce-bfc1- 08002be10318}\<nn>\Parameters
When the BlockFlowsForPerformance registry is set to 1, by default only L4 rules are allowed to be configured with rte_flow API. 
When the BlockFlowsForPerformance is set to 0, then L3 and L4 rules are allowed.

-----Original Message-----
From: Stephen Hemminger <stephen@networkplumber.org> 
Sent: Friday, January 4, 2019 5:08 PM
To: Liwu Liu <liwuliu@fortinet.com>
Cc: users@dpdk.org
Subject: Re: [dpdk-users] Cannot run DPDK applications using Mellanox NIC under Hyper-V

On Sat, 5 Jan 2019 00:35:13 +0000
Liwu Liu <liwuliu@fortinet.com> wrote:

> Hi Stephen,
> 
>   Yeah we are seeing all NETVSC, failsafe PMD, TAP, MLX4 VF. We are expecting bifurcation mode where default traffic goes with MLX4_EN slave to NETVSC master to enter Linux net stack and certain flow-defined traffic shall go into DPDK RX queue rather than MLX4_EN net-dev queue. Unfortunately as we cannot define flow into VF NIC, we cannot re-direct traffic into DPDK user-land. 

This is a little more complex.
You need to use rte_flow on the failsafe PMD and that is supposed to get propogated to TAP and MLX VF.
The TAP PMD programs BPF in kernel, and the MLX VF PMD programs the VF flow tables.

Haven't tried it myself.

Re: [dpdk-users] Cannot run DPDK applications using Mellanox NIC under Hyper-V

[Liwu Liu]

Hi Olga,

   We target at Azure though we use a local Windows Server for better debugging purpose.
   I tested on Ubuntu 18@Azure with SRIOV. It seems to be the case doing "ethtool -U" to attach flow also fails due to same reasone (error 12 out of memory).
   Many thanks,
Liwu

-----Original Message-----
From: Olga Shern <olgas@mellanox.com> 
Sent: Saturday, January 5, 2019 1:46 AM
To: Liwu Liu <liwuliu@fortinet.com>; users@dpdk.org
Subject: RE: [dpdk-users] Cannot run DPDK applications using Mellanox NIC under Hyper-V

HI Liwu, 

Are you running your application on Azure VMs?

Thanks,
Olga


-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Liwu Liu
Sent: Saturday, January 5, 2019 12:29 AM
To: users@dpdk.org
Subject: [dpdk-users] Cannot run DPDK applications using Mellanox NIC under Hyper-V

Hi Team,

    We used to have similar problem for Ubuntu 18.04 hypervisor and resolved by set log_num_mgm_entry_size=-1. (Refer to https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmails.dpdk.org%2Farchives%2Fusers%2F2018-November%2F003647.html&amp;data=02%7C01%7Colgas%40mellanox.com%7C0f967c6706714e2e992008d67294013a%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C1%7C636822377359958550&amp;sdata=AsxbxqWhKsGgMAMulz92p6SXzfPR%2FCxebrJ88fMO%2B5k%3D&amp;reserved=0
)

  Somehow for Windows Servers with MLNX VPI, I do not know where to set such and DPDK over MLX4 on linux vm has same failure of attaching flow.

    [  374.568992] <mlx4_ib> __mlx4_ib_create_flow: mcg table is full. Fail to register network rule. size = 64 (out of memory error code)

    Would like to get your help on this. It seems to be the case that the PF interface is not configured to trust VF interfaces to be able to add new flow rules.

   Many thanks for help,

Liwu




***  Please note that this message and any attachments may contain confidential and proprietary material and information and are intended only for the use of the intended recipient(s). If you are not the intended recipient, you are hereby notified that any review, use, disclosure, dissemination, distribution or copying of this message and any attachments is strictly prohibited. If you have received this email in error, please immediately notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed. Please also note that any views, opinions, conclusions or commitments expressed in this message are those of the individual sender and do not necessarily reflect the views of Fortinet, Inc., its affiliates, and emails are not binding on Fortinet and only a writing manually signed by Fortinet's General Counsel can be a binding commitment of Fortinet to Fortinet's customers or partners. Thank you. *** 

Re: [dpdk-users] Cannot run DPDK applications using Mellanox NIC under Hyper-V

[Olga Shern]

HI Liwu, 

Are you running your application on Azure VMs?

Thanks,
Olga


-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Liwu Liu
Sent: Saturday, January 5, 2019 12:29 AM
To: users@dpdk.org
Subject: [dpdk-users] Cannot run DPDK applications using Mellanox NIC under Hyper-V

Hi Team,

    We used to have similar problem for Ubuntu 18.04 hypervisor and resolved by set log_num_mgm_entry_size=-1. (Refer to https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmails.dpdk.org%2Farchives%2Fusers%2F2018-November%2F003647.html&data=02%7C01%7Colgas%40mellanox.com%7C0f967c6706714e2e992008d67294013a%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C1%7C636822377359958550&sdata=AsxbxqWhKsGgMAMulz92p6SXzfPR%2FCxebrJ88fMO%2B5k%3D&reserved=0
)

  Somehow for Windows Servers with MLNX VPI, I do not know where to set such and DPDK over MLX4 on linux vm has same failure of attaching flow.

    [  374.568992] <mlx4_ib> __mlx4_ib_create_flow: mcg table is full. Fail to register network rule. size = 64 (out of memory error code)

    Would like to get your help on this. It seems to be the case that the PF interface is not configured to trust VF interfaces to be able to add new flow rules.

   Many thanks for help,

Liwu




***  Please note that this message and any attachments may contain confidential and proprietary material and information and are intended only for the use of the intended recipient(s). If you are not the intended recipient, you are hereby notified that any review, use, disclosure, dissemination, distribution or copying of this message and any attachments is strictly prohibited. If you have received this email in error, please immediately notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed. Please also note that any views, opinions, conclusions or commitments expressed in this message are those of the individual sender and do not necessarily reflect the views of Fortinet, Inc., its affiliates, and emails are not binding on Fortinet and only a writing manually signed by Fortinet's General Counsel can be a binding commitment of Fortinet to Fortinet's customers or partners. Thank you. *** 

[dpdk-users] Cannot run DPDK applications using Mellanox NIC under Hyper-V

[Liwu Liu]

Hi Team,

    We used to have similar problem for Ubuntu 18.04 hypervisor and resolved by set log_num_mgm_entry_size=-1. (Refer to https://mails.dpdk.org/archives/users/2018-November/003647.html
)

  Somehow for Windows Servers with MLNX VPI, I do not know where to set such and DPDK over MLX4 on linux vm has same failure of attaching flow.

    [  374.568992] <mlx4_ib> __mlx4_ib_create_flow: mcg table is full. Fail to register network rule. size = 64 (out of memory error code)

    Would like to get your help on this. It seems to be the case that the PF interface is not configured to trust VF interfaces to be able to add new flow rules.

   Many thanks for help,

Liwu




***  Please note that this message and any attachments may contain confidential and proprietary material and information and are intended only for the use of the intended recipient(s). If you are not the intended recipient, you are hereby notified that any review, use, disclosure, dissemination, distribution or copying of this message and any attachments is strictly prohibited. If you have received this email in error, please immediately notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed. Please also note that any views, opinions, conclusions or commitments expressed in this message are those of the individual sender and do not necessarily reflect the views of Fortinet, Inc., its affiliates, and emails are not binding on Fortinet and only a writing manually signed by Fortinet's General Counsel can be a binding commitment of Fortinet to Fortinet's customers or partners. Thank you. ***