From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-vk0-f45.google.com (mail-vk0-f45.google.com [209.85.213.45]) by dpdk.org (Postfix) with ESMTP id 7BE998D94 for ; Mon, 25 Jan 2016 18:20:03 +0100 (CET) Received: by mail-vk0-f45.google.com with SMTP id k1so77477341vkb.2 for ; Mon, 25 Jan 2016 09:20:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Qu59c/FFQFkbH/KQV7SzXR7yaaq3yIhne1ofk+tDA3g=; b=I/pAUjBy6lWr2JYhITM89BNnaNuPAKSV+oj8mryowYJT9OyrxZo240jSOO+3zWtrwe 5EPL0GgdQcbdvoIaqTMJ9oGwV6GUo/FjE9gSASwRGuQ03mNc/9rVAkTCAoq76K5BE+ir 8pTsMqBeTsW7wO22nL7614L0t4WaGH1n7P2OS7TVuXaD3gOzSxZHJ7qf9m/jv19ehKJL vXGxXCSM5ORUnoy9lmzvS3rbc/geAtVXXlZhEVJyUCdLmxKCcDsAi7nTWHqNGqEyV0KQ 3tl/d1NkOyCTmN9a7qttuEQgLJAx88ZnMyJ+Rd08Rvy3/duR44MEqTokH/QD+rEhEAd4 HU4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=Qu59c/FFQFkbH/KQV7SzXR7yaaq3yIhne1ofk+tDA3g=; b=lqYHqOXwAJON7Jt28pPwDHoDLA5Jc8DVw7hEZxdsa4ABoKNf+SPrxqvrg1USoC+usj eEXc6IBm7oTr+Oqzus7iXDM1dbrHb8YSPldsXEZP8olmsomlKMSh6hnXiHtIH7sIO9Lv 5XgOYbmIMz8KpSmT5L+eSg5huGVdu7/4hjvKOhhA/ZX1laCLLWPmQgCrpW88KmrEjeIt Apzwqwbgb6PGGdtk7xWRv4jSq/WuFacWYCnDfklP9WZgjcO3KKkEtVyX8T8dTa53968C cAIhc/F6+f7OYU5RXkAXiQYs7cl+XJkRhmJaI1OJCZNLjcYscp0VRRj+TDCtS4lx1gWS snSQ== X-Gm-Message-State: AG10YOSKQK91DOB5C2HblvI7VEcJGRH533DqWGah4T1//rOEnFhTYs2HftmByq8wFY7MQqJVe46IfgvKlQudNg== MIME-Version: 1.0 X-Received: by 10.31.50.213 with SMTP id y204mr10945032vky.109.1453742402885; Mon, 25 Jan 2016 09:20:02 -0800 (PST) Received: by 10.31.152.208 with HTTP; Mon, 25 Jan 2016 09:20:02 -0800 (PST) In-Reply-To: <82F45D86ADE5454A95A89742C8D1410E031BD0EE@shsmsx102.ccr.corp.intel.com> References: <1802842.TfXaaEq7Xm@xps13> <82F45D86ADE5454A95A89742C8D1410E031BD0EE@shsmsx102.ccr.corp.intel.com> Date: Mon, 25 Jan 2016 09:20:02 -0800 Message-ID: From: Saurabh Mishra To: "Xu, Qian Q" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: "Zhang, Helin" , "users@dpdk.org" Subject: Re: [dpdk-users] i40e with DPDK exits abruptly in rte_eal_init() X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2016 17:20:04 -0000 Hi Qian -- >Mishra, could you help answer below questions? *>* What=E2=80=99s your i40e=E2=80=99s vmware driver on host? We are doing PCI-passthrough of i40e. vmnic7 0000:07:00.1 i40e Up 10000Mbps Full 3c:fd:fe:04:11:c2 1500 Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [root@localhost:~] vmkload_mod -s i40e |grep Version Version: Version 1.3.38, Build: 1331820, Interface: 9.2 Built on: Aug 5 2015 [root@localhost:~] vmkload_mod -s i40e vmkload_mod module information input file: /usr/lib/vmware/vmkmod/i40e Version: Version 1.3.38, Build: 1331820, Interface: 9.2 Built on: Aug 5 2015 License: GPL Required name-spaces: com.vmware.driverAPI#9.2.2.0 com.vmware.vmkapi#v2_2_0_0 Parameters: skb_mpool_max: int Maximum attainable private socket buffer memory pool size for the driver. skb_mpool_initial: int Driver's minimum private socket buffer memory pool size. heap_max: int Maximum attainable heap size for the driver. heap_initial: int Initial heap size allocated for the driver. debug: int Debug level (0=3Dnone,...,16=3Dall) RSS: array of int Number of Receive-Side Scaling Descriptor Queues: 0 =3D disable/default= , 1-4 =3D enable (number of cpus) VMDQ: array of int Number of Virtual Machine Device Queues: 0/1 =3D disable, 2-16 enable (default =3DI40E_ESX_DEFAULT_VMDQ) max_vfs: array of int Number of Virtual Functions: 0 =3D disable (default), 1-128 =3D enable = this many VFs [root@localhost:~] >What=E2=80=99s the firmware of FVL? Is your FVL 4X10G or 2x10G? Hmm..how do I check that? It has two ports. >Could you tell us your detailed test step? I only launch one dpdk app, no secondary process, but your case have 2 processes. When pass through the PCI device, I just pass through 2 ports to the VM and start the VM in my case. So I was passing one port to VM. I do launch primary and a secondary process. This is the error we got. It seems we were not able to map to the VA. So I moved the rte_eal_init() and mbuf alloc to a separate process and now it seems to work with one primary and six secondary process. EAL: PCI memory mapped at 0x7ffff7faa000 EAL: Cannot mmap device resource file /sys/bus/pci/devices/0000:1b:00.0/resource3 to address: 0x7ffff7fac000 EAL: Error - exiting with code: 1 Cause: Requested device 0000:1b:00.0 cannot be used Thanks, /Saurabh On Sun, Jan 24, 2016 at 5:17 PM, Xu, Qian Q wrote: > We have tried the I40E driver with FVL4 on ESXi in r2.2 testing and PCI > pass through can work but SRIOV will have issues. I didn=E2=80=99t see Mi= shra=E2=80=99s > error . Now we are stuck at setting up the ESXi environment, and need > > Vmware=E2=80=99s support to set up the env again. > > Mishra, could you help answer below questions? > > 1. What=E2=80=99s your i40e=E2=80=99s vmware driver on host? > > 2. What=E2=80=99s the firmware of FVL? Is your FVL 4X10G or 2x10G? > > 3. Could you tell us your detailed test step? I only launch one > dpdk app, no secondary process, but your case have 2 processes. When pass > through the PCI device, I just pass through 2 ports to the VM and start t= he > VM in my case. > > > > Helin, rombar=3D0 seems for KVM only, on VMWARE, not see the issue. > > > > > > Thanks > > Qian > > > > *From:* Zhang, Helin > *Sent:* Sunday, January 24, 2016 11:17 AM > *To:* Saurabh Mishra; Thomas Monjalon; Xu, Qian Q > *Cc:* users@dpdk.org; Mcnamara, John > *Subject:* RE: [dpdk-users] i40e with DPDK exits abruptly in > rte_eal_init() > > > > Hi Saurabh > > > > Were you talking about assigning NIC PF to the VMWARE guest? > > I remember that there is an issue of assigning PF to KVM guest, then > rombar=3D0 is needed for grub. > > Thank you for the issue report! > > > > Can Qian tell the details of that? > > > > Regards, > > Helin > > > > *From:* Saurabh Mishra [mailto:saurabh.globe@gmail.com > ] > *Sent:* Saturday, January 23, 2016 1:30 AM > *To:* Thomas Monjalon > *Cc:* users@dpdk.org; Zhang, Helin ; Mcnamara, > John > *Subject:* Re: [dpdk-users] i40e with DPDK exits abruptly in > rte_eal_init() > > > > Hi, > > > > Thanks Thomas.. So this is the error we see on i40e PCI-passthrough of > whole NIC: > > > > > > *Secondary Process:* > > > > EAL: Detected lcore 0 as core 0 on socket 0 > > EAL: Detected lcore 1 as core 0 on socket 0 > > EAL: Support maximum 128 logical core(s) by configuration. > > EAL: Detected 2 lcore(s) > > EAL: Setting up physically contiguous memory... > > EAL: Analysing 1024 files > > EAL: Mapped segment 0 of size 0x22800000 > > EAL: Mapped segment 1 of size 0x200000 > > EAL: Mapped segment 2 of size 0x200000 > > EAL: Mapped segment 3 of size 0x57600000 > > EAL: Mapped segment 4 of size 0x400000 > > EAL: Mapped segment 5 of size 0x400000 > > EAL: Mapped segment 6 of size 0x400000 > > EAL: Mapped segment 7 of size 0x200000 > > EAL: Mapped segment 8 of size 0x2200000 > > EAL: Mapped segment 9 of size 0x200000 > > EAL: Mapped segment 10 of size 0x800000 > > EAL: Mapped segment 11 of size 0x600000 > > EAL: Mapped segment 12 of size 0x800000 > > EAL: Mapped segment 13 of size 0xa00000 > > EAL: Mapped segment 14 of size 0x400000 > > EAL: Mapped segment 15 of size 0x200000 > > EAL: Mapped segment 16 of size 0x200000 > > EAL: Mapped segment 17 of size 0x200000 > > EAL: Mapped segment 18 of size 0x200000 > > EAL: memzone_reserve_aligned_thread_unsafe(): memzone > already exists > > RING: Cannot reserve memory > > EAL: TSC frequency is ~1799997 KHz > > EAL: Master lcore 1 is ready (tid=3Df7fe78c0;cpuset=3D[1]) > > EAL: PCI device 0000:03:00.0 on NUMA socket 0 > > EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd > > EAL: Not managed by a supported kernel driver, skipped > > EAL: PCI device 0000:1b:00.0 on NUMA socket 0 > > EAL: probe driver: 8086:1572 rte_i40e_pmd > > EAL: PCI memory mapped at 0x7fff6f1f3000 > > EAL: PCI memory mapped at 0x7ffff7faa000 > > EAL: Cannot mmap device resource file > /sys/bus/pci/devices/0000:1b:00.0/resource3 to address: 0x7ffff7fac000 > > EAL: Error - exiting with code: 1 > > Cause: Requested device 0000:1b:00.0 cannot be used > > > > > > # ./dpdk-2.2.0/tools/dpdk_nic_bind.py --status > > > > Network devices using DPDK-compatible driver > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > 0000:1b:00.0 'Device 1572' drv=3Duio_pci_generic unused=3Di40e > > > > Network devices using kernel driver > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > 0000:03:00.0 'VMXNET3 Ethernet Controller' if=3Deth0 drv=3Dvmxnet3 > unused=3Duio_pci_generic *Active* > > > > Other network devices > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > # grep Huge /proc/meminfo > > AnonHugePages: 118784 kB > > HugePages_Total: 1024 > > HugePages_Free: 0 > > HugePages_Rsvd: 0 > > HugePages_Surp: 0 > > Hugepagesize: 2048 kB > > # > > > > > > *Primary Process:* > > > > EAL: Detected lcore 0 as core 0 on socket 0 > > EAL: Detected lcore 1 as core 0 on socket 0 > > EAL: Support maximum 128 logical core(s) by configuration. > > EAL: Detected 2 lcore(s) > > EAL: Setting up physically contiguous memory... > > EAL: cannot open /proc/self/numa_maps, consider that all memory is in > socket_id 0 > > EAL: Ask a virtual area of 0x22800000 bytes > > EAL: Virtual area found at 0x7fffd0a00000 (size =3D 0x22800000) > > EAL: Ask a virtual area of 0x200000 bytes > > EAL: Virtual area found at 0x7fffd0600000 (size =3D 0x200000) > > EAL: Ask a virtual area of 0x200000 bytes > > EAL: Virtual area found at 0x7fffd0200000 (size =3D 0x200000) > > EAL: Ask a virtual area of 0x57600000 bytes > > EAL: Virtual area found at 0x7fff78a00000 (size =3D 0x57600000) > > EAL: Ask a virtual area of 0x400000 bytes > > EAL: Virtual area found at 0x7fff78400000 (size =3D 0x400000) > > EAL: Ask a virtual area of 0x400000 bytes > > EAL: Virtual area found at 0x7fff77e00000 (size =3D 0x400000) > > EAL: Ask a virtual area of 0x400000 bytes > > EAL: Virtual area found at 0x7fff77800000 (size =3D 0x400000) > > EAL: Ask a virtual area of 0x200000 bytes > > EAL: Virtual area found at 0x7fff77400000 (size =3D 0x200000) > > EAL: Ask a virtual area of 0x2200000 bytes > > EAL: Virtual area found at 0x7fff75000000 (size =3D 0x2200000) > > EAL: Ask a virtual area of 0x200000 bytes > > EAL: Virtual area found at 0x7fff74c00000 (size =3D 0x200000) > > EAL: Ask a virtual area of 0x800000 bytes > > EAL: Virtual area found at 0x7fff74200000 (size =3D 0x800000) > > EAL: Ask a virtual area of 0x600000 bytes > > EAL: Virtual area found at 0x7fff73a00000 (size =3D 0x600000) > > EAL: Ask a virtual area of 0x800000 bytes > > EAL: Virtual area found at 0x7fff73000000 (size =3D 0x800000) > > EAL: Ask a virtual area of 0xa00000 bytes > > EAL: Virtual area found at 0x7fff72400000 (size =3D 0xa00000) > > EAL: Ask a virtual area of 0x400000 bytes > > EAL: Virtual area found at 0x7fff71e00000 (size =3D 0x400000) > > EAL: Ask a virtual area of 0x200000 bytes > > EAL: Virtual area found at 0x7fff71a00000 (size =3D 0x200000) > > EAL: Ask a virtual area of 0x200000 bytes > > EAL: Virtual area found at 0x7fff71600000 (size =3D 0x200000) > > EAL: Ask a virtual area of 0x200000 bytes > > EAL: Virtual area found at 0x7fff71200000 (size =3D 0x200000) > > EAL: Ask a virtual area of 0x200000 bytes > > EAL: Virtual area found at 0x7fff70e00000 (size =3D 0x200000) > > EAL: Requesting 1024 pages of size 2MB from socket 0 > > EAL: TSC frequency is ~1799997 KHz > > EAL: Master lcore 1 is ready (tid=3Df7fe78a0;cpuset=3D[1]) > > EAL: PCI device 0000:03:00.0 on NUMA socket 0 > > EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd > > EAL: Not managed by a supported kernel driver, skipped > > EAL: PCI device 0000:1b:00.0 on NUMA socket 0 > > EAL: probe driver: 8086:1572 rte_i40e_pmd > > EAL: PCI memory mapped at 0x7fff6f1f3000 > > EAL: PCI memory mapped at 0x7ffff7fac000 > > # > > > > On Fri, Jan 22, 2016 at 2:12 AM, Thomas Monjalon < > thomas.monjalon@6wind.com> wrote: > > Hi, > > Thanks for asking. > There are a couple of requests hidden in this message. > The maintainer of i40e is Helin (CC'ed). > > 2016-01-21 14:54, Saurabh Mishra: > > Hi, > > > > We have noticed that i40e if we do PCI-pass through of whole NIC to VM = on > > ESXi 6.0, the DPDK exits abruptly in rte_eal_init()? > > This looks to be a bug report :) > > > We are passing following parameters: > > > > char *eal_argv[] =3D {"fakeelf", > > > > "-c2", > > > > "-n4", > > > > "--proc-type=3Dprimary",}; > > > > > > int ret =3D rte_eal_init(4, eal_argv); > > > > The code works with Intel '82599ES 10-Gigabit SFI/SFP+ ' adapter in > > PCI-passthrough or SR-IOV mode however i40e it does not work. > > You probably have the feeling that i40e does not work as other PMDs, > maybe wondering what are the special extended PCI configs. > > > [root@localhost:~] esxcfg-nics -l > > > > [.] > > > > vmnic6 0000:07:00.0 i40e Up 10000Mbps Full 3c:fd:fe:04:11:c= 0 > > 1500 Intel Corporation Ethernet Controller X710 for 10GbE SFP+ > > > > vmnic7 0000:07:00.1 i40e Up 10000Mbps Full 3c:fd:fe:04:11:c= 2 > > 1500 Intel Corporation Ethernet Controller X710 for 10GbE SFP+ > > > > > > We have turned on following config in DPDK: > > > > CONFIG_RTE_PCI_CONFIG=3Dy > > > > CONFIG_RTE_PCI_EXTENDED_TAG=3D"on" > > CONFIG_RTE_LIBRTE_I40E_16BYTE_RX_DESC=3Dy > > > > > > Is there any special handling in DPDK for i40e adapter in terms of > config? > > So you had no help when reading the code comments neither in the doc. > Indeed the only doc about i40e is the SR-IOV VF page: > http://dpdk.org/doc/guides/nics/intel_vf.html > > Please Helin, check the issue and the lack of documentation. > Thanks > > >