From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) by dpdk.org (Postfix) with ESMTP id 977241B0F7 for ; Fri, 2 Nov 2018 01:48:38 +0100 (CET) Received: by mail-qk1-f176.google.com with SMTP id n12so517042qkh.11 for ; Thu, 01 Nov 2018 17:48:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=g-harvard-edu.20150623.gappssmtp.com; s=20150623; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=wxFq5WlXqVxcuk3g9axjlKOzQ9uGtVeUWF7fLCzaa/8=; b=Mnxx2hPoC36kYRDIPuvTMzEgkJc2EqxnvyDuKj1WiLuI3oWLiqi6JCjBh2kjNXO4xY 36ibyliIKZviw1osYxwbDoSclas78fQfh7RCwNuvCa/XCbJ3Aa94000HnCo/DvhvK3mb ZTfiVBx/QmCHPL2YcDOZyu8Pm/qZ69TMglfgXzYxayk1ZwRfxQGsxxzQ3OJIODxBE8Fp NlupzyTKY0kZQs+oRbXvySQb+jw1TL+aEaAevGtLaqd0/5Ubdcs12oOZPxjpIzJNOf9H kfMYPPWXdC48gTv8OZ5yCBcogDGj13T4Y3Lo8on9KGvHknPuUW6MGULM0gjozDWr8IPO YTHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=wxFq5WlXqVxcuk3g9axjlKOzQ9uGtVeUWF7fLCzaa/8=; b=UQu+olmKEHhVRG+RKFlXBBMoh7yXadRi5AuuPlhLm4g3orG2LsVmYL6Ev3JFsyJB/n bnUU0Xc+Ai03RSFKbtWYSPd81CtvlRlDKlfe8WSsBi+qxgr1q1YvaUF6lYfKjTh8UqMd dsDilyF9Ay/0/ZrHRf7Fh0hQnB5vrt7jmfXuv64CEWwRcQHo5BkwkPqNTVieG25tP8bA RETB+Z3sb1jzHH2A7P/pYXjqk5a2B9A9AtLLQG0rFUXQoIjz+EiNy4dkQRn2hpZAyTll qPQvJq3dlF7X89rdLTA/Sy+GIngaqW53bHuo3fDROyRsne2YgVR3y3rZsvqwm4p0nchP dTBw== X-Gm-Message-State: AGRZ1gLuftUywcMNLAoHNP4II95LAybjHwrDlqdScSxAeSUlnrGJhMw7 uOP8lddvD/0nq+bP97rCrNlOZw== X-Google-Smtp-Source: AJdET5dNq8YhzNgt10zuTnuuNq9rnJWCktPq1DDrDxIioeCG2n7DAKdcGecNywseZFLqq/hMfPDr0A== X-Received: by 2002:aed:24b7:: with SMTP id t52mr9133101qtc.131.1541119717775; Thu, 01 Nov 2018 17:48:37 -0700 (PDT) Received: from [10.252.147.78] ([65.112.8.200]) by smtp.gmail.com with ESMTPSA id l13sm12950109qkh.45.2018.11.01.17.48.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 01 Nov 2018 17:48:37 -0700 (PDT) From: Junzhi Gong Message-Id: <18DE9990-4BFD-434A-AE0B-970A087894FE@g.harvard.edu> Mime-Version: 1.0 (Mac OS X Mail 12.0 \(3445.100.39\)) Date: Thu, 1 Nov 2018 20:48:35 -0400 In-Reply-To: Cc: "users@dpdk.org" , Erez Ferber , Raslan Darawsheh To: Olga Shern References: X-Mailer: Apple Mail (2.3445.100.39) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] Cannot run DPDK applications using Mellanox NIC X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2018 00:48:39 -0000 Hi Olga, Thank you very much! It works after I changing = log_num_mgm_entry_size=3D-1! Junzhi > On Nov 1, 2018, at 12:52 PM, Olga Shern wrote: >=20 > Hi Junzhi,=20 >=20 > Can you please change log_num_mgm_entry_size=3D-1 > Can you please run testpmd and specify only pf 04:00.0 >=20 > Thanks, > Olga >=20 >=20 > -----Original Message----- > From: users [mailto:users-bounces@dpdk.org = ] On Behalf Of Junzhi Gong > Sent: Thursday, November 1, 2018 6:45 PM > To: users@dpdk.org > Subject: [dpdk-users] Cannot run DPDK applications using Mellanox NIC >=20 > Hi everyone, >=20 > I am sorry that I did not give many examples and details last time. >=20 > The Mellanox NIC is=20 >=20 > Device #1: > ---------- >=20 > Device Type: ConnectX3Pro > Part Number: MCX313A-BCC_Ax > Description: ConnectX-3 Pro EN network interface card; 40GigE; = single-port QSFP; PCIe3.0 x8 8GT/s; RoHS R6 > PSID: MT_1060111023 > PCI Device Name: /dev/mst/mt4103_pci_cr0 > Port1 MAC: ec0d9a07dc90 > Port2 MAC: ec0d9a07dc91 > Versions: Current Available =20 > FW 2.42.5000 N/A =20 > PXE 3.4.0752 N/A =20 >=20 > The Mellanox OFED version is 4.4. I set 12 VFs on the host, and I can = look at them like this: >=20 > $ lspci | grep Mellanox > 04:00.0 Ethernet controller: Mellanox Technologies MT27520 Family = [ConnectX-3 Pro] > 04:00.1 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] > 04:00.2 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] > 04:00.3 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] > 04:00.4 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] > 04:00.5 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] > 04:00.6 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] > 04:00.7 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] > 04:01.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] > 04:01.1 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] > 04:01.2 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] > 04:01.3 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] > 04:01.4 Ethernet controller: Mellanox Technologies MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function] >=20 > All port is set to Ethernet Adapter: >=20 > $ connectx_port_config -s > -------------------------------- > Port configuration for PCI device: 0000:04:00.0 is: > eth > -------------------------------- > -------------------------------- > Port configuration for PCI device: 0000:04:00.1 is: > eth > -------------------------------- > -------------------------------- > Port configuration for PCI device: 0000:04:00.2 is: > eth > -------------------------------- > -------------------------------- > Port configuration for PCI device: 0000:04:00.3 is: > eth > -------------------------------- > -------------------------------- > Port configuration for PCI device: 0000:04:00.4 is: > eth > -------------------------------- > -------------------------------- > Port configuration for PCI device: 0000:04:00.5 is: > eth > -------------------------------- > -------------------------------- > Port configuration for PCI device: 0000:04:00.6 is: > eth > -------------------------------- > -------------------------------- > Port configuration for PCI device: 0000:04:00.7 is: > eth > -------------------------------- > -------------------------------- > Port configuration for PCI device: 0000:04:01.0 is: > eth > -------------------------------- > -------------------------------- > Port configuration for PCI device: 0000:04:01.1 is: > eth > -------------------------------- > -------------------------------- > Port configuration for PCI device: 0000:04:01.2 is: > eth > -------------------------------- > -------------------------------- > Port configuration for PCI device: 0000:04:01.3 is: > eth > -------------------------------- > -------------------------------- > Port configuration for PCI device: 0000:04:01.4 is: > eth > -------------------------------- >=20 > In file /etc/modprobe.d/mlx4_core.conf: >=20 > options mlx4_core log_num_mgm_entry_size=3D-7 num_vfs=3D12 = port_type_array=3D2 probe_vf=3D12 >=20 > My DPDK version is 18.08, and in file /config/common_base, I set: >=20 > CONFIG_RTE_LIBRTE_MLX4_PMD=3Dy >=20 > After doing modprobe -a ib_uverbs mlx4_en mlx4_core mlx4_ib >=20 > I run several DPDK applications, and I got the same error message as = follows. >=20 > Thank you! >=20 > Junzhi >=20 >=20 >> On Oct 26, 2018, at 4:27 PM, Junzhi Gong = wrote: >>=20 >> Hi everyone, >>=20 >> Currently I have some trouble running DPDK applications using = Mellanox NIC. All DPDK applications running with the Mellanox NIC (using = SR-IOV VFs) will crash with the following error message: >>=20 >> PMD: net_mlx4: 0x559fa4a25b00: cannot attach flow rules (code 12,=20 >> "Cannot allocate memory"), flow error type 2, cause 0x7f0ca7e47a80,=20= >> message: flow rule rejected by device >> EAL: Error - exiting with code: 1 >> Cause: rte_eth_dev_start:err=3D-12, port=3D0 >>=20 >> An example running click has the following error messages: >>=20 >> $ click --dpdk -l 1 -n 4 -w 0000:04:00.1 =E2=80=94- xxx.click >>=20 >> EAL: Detected 12 lcore(s) >> EAL: Detected 1 NUMA nodes >> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket >> EAL: No free hugepages reported in hugepages-1048576kB >> EAL: Probing VFIO support... >> EAL: PCI device 0000:04:00.1 on NUMA socket 0 >> EAL: probe driver: 15b3:1004 net_mlx4 >> PMD: net_mlx4: PCI information matches, using device "mlx4_1" (VF:=20 >> true) >> PMD: net_mlx4: 1 port(s) detected >> PMD: net_mlx4: port 1 MAC address is 6a:f5:c3:1e:09:eb Initializing=20= >> DPDK >> PMD: net_mlx4: 0x559471c5b700: cannot attach flow rules (code 12,=20 >> "Cannot allocate memory"), flow error type 2, cause 0x7f620462c940,=20= >> message: flow rule rejected by device >> mazu-nat.click:68: While initializing =E2=80=98FromDPDKDevice@3 :: = FromDPDKDevice=E2=80=99: >> Cannot start DPDK port 0: error -12 >>=20 >> 0000:04:00.1 is the PCI address of the VF of the Mellanox NIC. I have = tried several different DPDK applications (all applications have been = re-compiled using the new configured DPDK), and I have also tried reboot = the OS, but the problem still exists.=20 >>=20 >> I am using Mellanox NIC and its SR-IOV functions ('MT27500/MT27520 = Family [ConnectX-3/ConnectX-3 Pro Virtual Function]' in details). The OS = is Ubuntu 18.04. I have installed the Mellanox OFED (version 4.4, only = the newest version is compatible to Ubuntu 18.04) and configure it by = instructions in DPDK doc page = (https://emea01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fdoc.= dpdk.org%2Fguides%2Fnics%2Fmlx4.html&data=3D02%7C01%7Colgas%40mellanox= .com%7Cda3acc6743514ffaf81f08d640196525%7Ca652971c7d2e4d9ba6a4d149256f461b= %7C0%7C0%7C636766875152604596&sdata=3DHgF%2FynJD%2B5U2ZegDVUc%2BmqCv5P= y5Ak7xdIk2Nhyemg4%3D&reserved=3D0 = = >). The DPDK version is 18.08. I = do not bind the Mellanox NIC to DPDK driver because the driver mlx4_core = should be used for Mellanox NICs. Currently I am using 2MB hugepages = (number: 1024). >>=20 >> I am very grateful if you can solve my problem. Thank you! >>=20 >> Best, >> Junzhi