From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) by dpdk.org (Postfix) with ESMTP id 3B5475F0D for ; Thu, 1 Nov 2018 17:45:10 +0100 (CET) Received: by mail-qt1-f181.google.com with SMTP id q41-v6so21781999qtq.10 for ; Thu, 01 Nov 2018 09:45:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=g-harvard-edu.20150623.gappssmtp.com; s=20150623; h=from:mime-version:subject:date:references:to:in-reply-to:message-id; bh=E9Hu1PqNBvbBdPk3nib63uwODJDlczvFrYoXsVzgbWE=; b=XKJ13ck9R0siwd97GvJweYCuq+D66iN0sGVToTjXDMfrnlwRVOFlNuODkbkMXd+G3q fananinNQdmzX7Q9DBsQw4iwJ8q0Fr2BmrFz/rwrdsmkCqJHnoFJg+tl3wKrGKy3AUgF 9FWH3nE+20htlV1fFzrtLx588RqAeyFGggi7ByQNQjQoMyxbwjpa91aGRXifgHcIYdBa ZS94Ym3zoBSf6W7eBwpdFvU8LJhg+nQ2NyV30U1aGUqffUAZJrYu98+D9FEcqJES8thI B4L0YvEDXBKgI41uFxNuXJcLG1b02VXad6JClPZ/OKu9CJJtImmemGkGzZo6ihrpPV+q bJVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:mime-version:subject:date:references:to :in-reply-to:message-id; bh=E9Hu1PqNBvbBdPk3nib63uwODJDlczvFrYoXsVzgbWE=; b=etDywQ1UWW3iQlPUWFB0EqM0uNLdHp4LkI+9G4a8L9rKxDbKaYnhGEkbVCsHfPKKYX uKT/DkGtoMdXbO85JBdLFH5uh/wYj56w9elozEeqgO5iSTfGSOjy867DGp5ujlOsbRHH b6Ep8KmJw8yUnhK17gZ+LO41KFSudLVVeYlplQP+QrSlGGSv0dzx8x+CJ+7kt2s2wEAP nDpB1yqxCwGyrGk1rHuhnO8bdzFh+zHawgMdh86UNGirZra7oAeFASBR2XQLuiVsWfMW DFGQIjzsu3VebjEAtb3GefkKslNQIaE2c6TyrwGGCTLc7dYO12lI2z7l7iZQYtnS+6pJ /U4A== X-Gm-Message-State: AGRZ1gJXsVTYIKqwsCgkPmPXaDEfZUZZyGB9i6XwOYkhbRSYeRMod+ag gM4CjWpetqUBQA1bkuLkRs9DyT1Cy4Q= X-Google-Smtp-Source: AJdET5eDmwfzUzbHKxdEYV/V3GSMC+rfuIgWb4XSOFnSV+9MYnpng5C4n3r5PrPg13Em32HfaDKAUQ== X-Received: by 2002:a0c:9311:: with SMTP id d17mr7588734qvd.54.1541090708974; Thu, 01 Nov 2018 09:45:08 -0700 (PDT) Received: from [10.252.24.115] ([65.112.8.19]) by smtp.gmail.com with ESMTPSA id d6-v6sm7481666qth.18.2018.11.01.09.45.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 01 Nov 2018 09:45:07 -0700 (PDT) From: Junzhi Gong Mime-Version: 1.0 (Mac OS X Mail 12.0 \(3445.100.39\)) Date: Thu, 1 Nov 2018 12:45:06 -0400 References: To: users@dpdk.org In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3445.100.39) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] Cannot run DPDK applications using Mellanox NIC X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Nov 2018 16:45:10 -0000 Hi everyone, I am sorry that I did not give many examples and details last time. The Mellanox NIC is=20 Device #1: ---------- Device Type: ConnectX3Pro Part Number: MCX313A-BCC_Ax Description: ConnectX-3 Pro EN network interface card; 40GigE; = single-port QSFP; PCIe3.0 x8 8GT/s; RoHS R6 PSID: MT_1060111023 PCI Device Name: /dev/mst/mt4103_pci_cr0 Port1 MAC: ec0d9a07dc90 Port2 MAC: ec0d9a07dc91 Versions: Current Available =20 FW 2.42.5000 N/A =20 PXE 3.4.0752 N/A =20 The Mellanox OFED version is 4.4. I set 12 VFs on the host, and I can = look at them like this: $ lspci | grep Mellanox 04:00.0 Ethernet controller: Mellanox Technologies MT27520 Family = [ConnectX-3 Pro] 04:00.1 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.2 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.3 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.4 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.5 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.6 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.7 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:01.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:01.1 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:01.2 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:01.3 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:01.4 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] All port is set to Ethernet Adapter: $ connectx_port_config -s -------------------------------- Port configuration for PCI device: 0000:04:00.0 is: eth -------------------------------- -------------------------------- Port configuration for PCI device: 0000:04:00.1 is: eth -------------------------------- -------------------------------- Port configuration for PCI device: 0000:04:00.2 is: eth -------------------------------- -------------------------------- Port configuration for PCI device: 0000:04:00.3 is: eth -------------------------------- -------------------------------- Port configuration for PCI device: 0000:04:00.4 is: eth -------------------------------- -------------------------------- Port configuration for PCI device: 0000:04:00.5 is: eth -------------------------------- -------------------------------- Port configuration for PCI device: 0000:04:00.6 is: eth -------------------------------- -------------------------------- Port configuration for PCI device: 0000:04:00.7 is: eth -------------------------------- -------------------------------- Port configuration for PCI device: 0000:04:01.0 is: eth -------------------------------- -------------------------------- Port configuration for PCI device: 0000:04:01.1 is: eth -------------------------------- -------------------------------- Port configuration for PCI device: 0000:04:01.2 is: eth -------------------------------- -------------------------------- Port configuration for PCI device: 0000:04:01.3 is: eth -------------------------------- -------------------------------- Port configuration for PCI device: 0000:04:01.4 is: eth -------------------------------- In file /etc/modprobe.d/mlx4_core.conf: options mlx4_core log_num_mgm_entry_size=3D-7 num_vfs=3D12 = port_type_array=3D2 probe_vf=3D12 My DPDK version is 18.08, and in file /config/common_base, I set: CONFIG_RTE_LIBRTE_MLX4_PMD=3Dy After doing modprobe -a ib_uverbs mlx4_en mlx4_core mlx4_ib I run several DPDK applications, and I got the same error message as = follows. Thank you! Junzhi > On Oct 26, 2018, at 4:27 PM, Junzhi Gong wrote: >=20 > Hi everyone, >=20 > Currently I have some trouble running DPDK applications using Mellanox = NIC. All DPDK applications running with the Mellanox NIC (using SR-IOV = VFs) will crash with the following error message: >=20 > PMD: net_mlx4: 0x559fa4a25b00: cannot attach flow rules (code 12, = "Cannot allocate memory"), flow error type 2, cause 0x7f0ca7e47a80, = message: flow rule rejected by device > EAL: Error - exiting with code: 1 > Cause: rte_eth_dev_start:err=3D-12, port=3D0 >=20 > An example running click has the following error messages: >=20 > $ click --dpdk -l 1 -n 4 -w 0000:04:00.1 =E2=80=94- xxx.click >=20 > EAL: Detected 12 lcore(s) > EAL: Detected 1 NUMA nodes > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > EAL: No free hugepages reported in hugepages-1048576kB > EAL: Probing VFIO support... > EAL: PCI device 0000:04:00.1 on NUMA socket 0 > EAL: probe driver: 15b3:1004 net_mlx4 > PMD: net_mlx4: PCI information matches, using device "mlx4_1" (VF: = true) > PMD: net_mlx4: 1 port(s) detected > PMD: net_mlx4: port 1 MAC address is 6a:f5:c3:1e:09:eb > Initializing DPDK > PMD: net_mlx4: 0x559471c5b700: cannot attach flow rules (code 12, = "Cannot allocate memory"), flow error type 2, cause 0x7f620462c940, = message: flow rule rejected by device > mazu-nat.click:68: While initializing =E2=80=98FromDPDKDevice@3 :: = FromDPDKDevice=E2=80=99: > Cannot start DPDK port 0: error -12 >=20 > 0000:04:00.1 is the PCI address of the VF of the Mellanox NIC. I have = tried several different DPDK applications (all applications have been = re-compiled using the new configured DPDK), and I have also tried reboot = the OS, but the problem still exists.=20 >=20 > I am using Mellanox NIC and its SR-IOV functions ('MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function]' in details). The OS = is Ubuntu 18.04. I have installed the Mellanox OFED (version 4.4, only = the newest version is compatible to Ubuntu 18.04) and configure it by = instructions in DPDK doc page = (https://doc.dpdk.org/guides/nics/mlx4.html = ). The DPDK version is = 18.08. I do not bind the Mellanox NIC to DPDK driver because the driver = mlx4_core should be used for Mellanox NICs. Currently I am using 2MB = hugepages (number: 1024). >=20 > I am very grateful if you can solve my problem. Thank you! >=20 > Best, > Junzhi