DPDK usage discussions
 help / color / mirror / Atom feed
From: Junzhi Gong <jug257@g.harvard.edu>
To: users@dpdk.org
Subject: [dpdk-users]  Cannot run DPDK applications using Mellanox NIC
Date: Thu, 1 Nov 2018 12:45:06 -0400	[thread overview]
Message-ID: <EF9100A2-068F-46E8-8DD5-63154C8AE6C1@g.harvard.edu> (raw)
In-Reply-To: <C921FE77-E35A-4FF1-A15E-99191AB1E972@g.harvard.edu>

Hi everyone,

I am sorry that I did not give many examples and details last time.

The Mellanox NIC is 

Device #1:
----------

  Device Type:      ConnectX3Pro
  Part Number:      MCX313A-BCC_Ax
  Description:      ConnectX-3 Pro EN network interface card; 40GigE; single-port QSFP; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1060111023
  PCI Device Name:  /dev/mst/mt4103_pci_cr0
  Port1 MAC:        ec0d9a07dc90
  Port2 MAC:        ec0d9a07dc91
  Versions:         Current        Available     
     FW             2.42.5000      N/A           
     PXE            3.4.0752       N/A           

The Mellanox OFED version is 4.4. I set 12 VFs on the host, and I can look at them like this:

$ lspci | grep Mellanox
04:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
04:00.1 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.2 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.3 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.4 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.5 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.6 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.7 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:01.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:01.1 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:01.2 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:01.3 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:01.4 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

All port is set to Ethernet Adapter:

$ connectx_port_config -s
--------------------------------
Port configuration for PCI device: 0000:04:00.0 is:
eth
--------------------------------
--------------------------------
Port configuration for PCI device: 0000:04:00.1 is:
eth
--------------------------------
--------------------------------
Port configuration for PCI device: 0000:04:00.2 is:
eth
--------------------------------
--------------------------------
Port configuration for PCI device: 0000:04:00.3 is:
eth
--------------------------------
--------------------------------
Port configuration for PCI device: 0000:04:00.4 is:
eth
--------------------------------
--------------------------------
Port configuration for PCI device: 0000:04:00.5 is:
eth
--------------------------------
--------------------------------
Port configuration for PCI device: 0000:04:00.6 is:
eth
--------------------------------
--------------------------------
Port configuration for PCI device: 0000:04:00.7 is:
eth
--------------------------------
--------------------------------
Port configuration for PCI device: 0000:04:01.0 is:
eth
--------------------------------
--------------------------------
Port configuration for PCI device: 0000:04:01.1 is:
eth
--------------------------------
--------------------------------
Port configuration for PCI device: 0000:04:01.2 is:
eth
--------------------------------
--------------------------------
Port configuration for PCI device: 0000:04:01.3 is:
eth
--------------------------------
--------------------------------
Port configuration for PCI device: 0000:04:01.4 is:
eth
--------------------------------

In file /etc/modprobe.d/mlx4_core.conf:

options mlx4_core log_num_mgm_entry_size=-7 num_vfs=12 port_type_array=2 probe_vf=12

My DPDK version is 18.08, and in file /config/common_base, I set:

CONFIG_RTE_LIBRTE_MLX4_PMD=y

After doing modprobe -a ib_uverbs mlx4_en mlx4_core mlx4_ib

I run several DPDK applications, and I got the same error message as follows.

Thank you!

Junzhi


> On Oct 26, 2018, at 4:27 PM, Junzhi Gong <jug257@g.harvard.edu> wrote:
> 
> Hi everyone,
> 
> Currently I have some trouble running DPDK applications using Mellanox NIC. All DPDK applications running with the Mellanox NIC (using SR-IOV VFs) will crash with the following error message:
> 
> PMD: net_mlx4: 0x559fa4a25b00: cannot attach flow rules (code 12, "Cannot allocate memory"), flow error type 2, cause 0x7f0ca7e47a80, message: flow rule rejected by device
> EAL: Error - exiting with code: 1
>   Cause: rte_eth_dev_start:err=-12, port=0
> 
> An example running click has the following error messages:
> 
> $ click --dpdk -l 1 -n 4 -w 0000:04:00.1 —- xxx.click
> 
> EAL: Detected 12 lcore(s)
> EAL: Detected 1 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> EAL: PCI device 0000:04:00.1 on NUMA socket 0
> EAL:   probe driver: 15b3:1004 net_mlx4
> PMD: net_mlx4: PCI information matches, using device "mlx4_1" (VF: true)
> PMD: net_mlx4: 1 port(s) detected
> PMD: net_mlx4: port 1 MAC address is 6a:f5:c3:1e:09:eb
> Initializing DPDK
> PMD: net_mlx4: 0x559471c5b700: cannot attach flow rules (code 12, "Cannot allocate memory"), flow error type 2, cause 0x7f620462c940, message: flow rule rejected by device
> mazu-nat.click:68: While initializing ‘FromDPDKDevice@3 :: FromDPDKDevice’:
>   Cannot start DPDK port 0: error -12
> 
> 0000:04:00.1 is the PCI address of the VF of the Mellanox NIC. I have tried several different DPDK applications (all applications have been re-compiled using the new configured DPDK), and I have also tried reboot the OS, but the problem still exists. 
> 
> I am using Mellanox NIC and its SR-IOV functions ('MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]' in details). The OS is Ubuntu 18.04. I have installed the Mellanox OFED (version 4.4, only the newest version is compatible to Ubuntu 18.04) and configure it by instructions in DPDK doc page (https://doc.dpdk.org/guides/nics/mlx4.html <https://doc.dpdk.org/guides/nics/mlx4.html>). The DPDK version is 18.08. I do not bind the Mellanox NIC to DPDK driver because the driver mlx4_core should be used for Mellanox NICs. Currently I am using 2MB hugepages (number: 1024).
> 
> I am very grateful if you can solve my problem. Thank you!
> 
> Best,
> Junzhi

  reply	other threads:[~2018-11-01 16:45 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-10-26 20:27 [dpdk-users] Cannot run DPDK applications using Mellanox NIC - "PMD: net_mlx4: 0x55b8d8015b00: cannot attach flow rules (code 12, "Cannot allocate memory"), flow error type 2, cause 0x7f9ee4028200, message: flow rule rejected by device" Junzhi Gong
2018-11-01 16:45 ` Junzhi Gong [this message]
2018-11-01 16:52   ` [dpdk-users] Cannot run DPDK applications using Mellanox NIC Olga Shern
2018-11-02  0:48     ` Junzhi Gong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=EF9100A2-068F-46E8-8DD5-63154C8AE6C1@g.harvard.edu \
    --to=jug257@g.harvard.edu \
    --cc=users@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).