From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f68.google.com (mail-wm0-f68.google.com [74.125.82.68]) by dpdk.org (Postfix) with ESMTP id 6B5CC36E for ; Sun, 12 Mar 2017 04:30:12 +0100 (CET) Received: by mail-wm0-f68.google.com with SMTP id v190so4744488wme.3 for ; Sat, 11 Mar 2017 19:30:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=o2yxlFek+rnCMyvds4+AUTbV2QCDZuwNGXXuDNcUOsA=; b=FL2yEjmZr8HSX8gNvKwOJDmk6Da7R3SHNkslLFWd0GxjPLfeK8QYZ9F3QR7js3tHeR Nft6dHR+V9syfkbtMEitVWOkWzFbpKZMBYvfPScPVviu4qom5tiHezDWTgBiY+sREqFq 1sagKbFwPNZ5g9+G9wzSRgJsCgMY+1huMQNpMCN7bxLNRAVOcctyUOsENeIkjHTc7rP/ rGhKQGLoWHF1Z7BypW6jFIzaXoRX0FZDH2IoVpLrLEg3qhaDyhR5vYkb8ztIFagVpA3O 9gIAVa7rJZ9dOMT7Ije5WnWHbZ7aMVpyVNQdQuVegMSGPw6fKFWwXf+1N4KB4LeT0uCc RARg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=o2yxlFek+rnCMyvds4+AUTbV2QCDZuwNGXXuDNcUOsA=; b=SE3yYaG9ZkqLsaFGg/yKVctaJ2wr/dgplW5LPnP4HZbEg74JIXxqg4W+Ev7pJIWHaU rnCHilBo8N27eN1HzQrnG3bumAqKsyqkjB2xZb1awcbRlPPkeUfi1cQV4qe6jnILZc21 n2o6nVWMJcFjL6aYWBMF0+nEMXMB/7o3Puve2W9Z8ko9nbbgMVI5BTiOjff3R94CUwpz GsQ+TmoxFWsL1ipiq+hknITo7QOFRol6C3xdgLNovgBrTSAyrb6lU1vSdDFp9WUu9P02 PraUS3b6wq6uka8BlTQCcwE7jjxZlJySihQBSmyd/rMc3uvPkpfn0l1gewnsy4fOGkp7 N0Vg== X-Gm-Message-State: AFeK/H2Pe3ejmW8neWP6RCDVbac6PPnOSTeabTA3MmEZnKcQpxDdl9YdWC/4Tfc+VI8Z9q7PsuHEaIenlDIX5g== X-Received: by 10.28.101.68 with SMTP id z65mr4676049wmb.102.1489289409034; Sat, 11 Mar 2017 19:30:09 -0800 (PST) MIME-Version: 1.0 Received: by 10.28.141.80 with HTTP; Sat, 11 Mar 2017 19:29:48 -0800 (PST) In-Reply-To: References: From: Kai Zhang Date: Sun, 12 Mar 2017 11:29:48 +0800 Message-ID: To: "Wiles, Keith" Cc: "users@dpdk.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] Issue with more Cores assigned: Cannot mmap device resource file X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Mar 2017 03:30:12 -0000 Update: After unbinding the ports on machine A, the program passes rte_eal_init() and works correctly with the primary process for any number of cores. >>From the error message when binding the port, I think there are some resource allocation issues with the bound port. But why is it related with the number of cores ... EAL: Cannot mmap device resource file /sys/bus/pci/devices/*0000:02:00.0*/resource0 to address: 0x7fff65bfc000 EAL: Error - exiting with code: 1 Cause: Requested device *0000:02:00.0* cannot be used Regards, Kai On Sun, Mar 12, 2017 at 11:21 AM, Kai Zhang wrote: > > Command line: > primary: sudo ./primary -l 0,1,2,3 -n 4 --proc-type=3Dprimary > secondary: sudo ./secondary -l 4,5,6,7,8 -n 4 --proc-type=3Dsecondary > > The configurations are as follows: > A) 1 x Intel E5-2650 v4, 12 cores [UMA], XL710 40GbE, bind 02:00.0, > 2048 x 4k huge page > 02:00.0 *Ethernet* controller: Intel Corporation *Ethernet* Controller > XL710 for 40GbE QSFP+ (rev 02) * [<<- Only bind this one]* > 02:00.1 *Ethernet* controller: Intel Corporation *Ethernet* Controller > XL710 for 40GbE QSFP+ (rev 02) > 05:00.0 *Ethernet* controller: Intel Corporation I210 Gigabit Network > Connection (rev 03) > 06:00.0 *Ethernet* controller: Intel Corporation I210 Gigabit Network > Connection (rev 03) > Socket 0 > -------- > Core 0 [0, 12] > Core 1 [1, 13] > Core 2 [2, 14] > Core 3 [3, 15] > Core 4 [4, 16] > Core 5 [5, 17] > Core 8 [6, 18] > Core 9 [7, 19] > Core 10 [8, 20] > Core 11 [9, 21] > Core 12 [10, 22] > Core 13 [11, 23] > > B) 2 x Intel E5-2640 v4, 10 cores [NUMA], No Port Bind, 2048 x 4k > huge page > 05:00.0 *Ethernet* controller: Intel Corporation I210 Gigabit Network > Connection (rev 03) > 06:00.0 *Ethernet* controller: Intel Corporation I210 Gigabit Network > Connection (rev 03) > Socket 0 Socket 1 > -------- -------- > Core 0 [0, 20] [10, 30] > Core 1 [1, 21] [11, 31] > Core 2 [2, 22] [12, 32] > Core 3 [3, 23] [13, 33] > Core 4 [4, 24] [14, 34] > Core 8 [5, 25] [15, 35] > Core 9 [6, 26] [16, 36] > Core 10 [7, 27] [17, 37] > Core 11 [8, 28] [18, 38] > Core 12 [9, 29] [19, 39] > > Ah, as machine B does not have a 40GbE, I did not bind any NIC and run my > program with locally generated packets. But I am using other DPDK feature= s, > such as memory sharing and message passing. Maybe that is the reason it > works correctly? I can only access machine B remotely, so I am unable to > install a NIC on it. I have another PC that is used as a client that only > has four cores, which also cannot be used for verification... > > Regards, > Kai > > > On Sun, Mar 12, 2017 at 2:59 AM, Wiles, Keith > wrote: > >> >> > On Mar 11, 2017, at 9:45 AM, Kai Zhang wrote: >> > >> > Hi Keith, >> > >> > Thank you for your reply. >> > >> > I have tested my program on two machines >> > A) 1 x Intel E5-2650 v4, 12 cores [UMA] >> > B) 2 x Intel E5-2640 v4, 10 cores [NUMA] >> > >> > I am very sure that the primary process uses different cores with the >> secondary process. The strange thing is that my program works correctly = on >> machine B. But on machine A, the above issue happens with more than 4 co= res >> assigned to the secondary process. >> > >> > I have tried to assign cores 1-5 to the secondary process and also >> tried other core assignment policies, but the error still happens >> rte_eal_init() with more than 4 cores. >> >> It would be nice to see both command lines. I am not sure I can help mor= e >> all I can do is suggest some ideas to look at. >> >> Does machine B have the same number and type of NICs? Use =E2=80=98lspci= | grep >> Ethernet=E2=80=99 to get a list of all Ethernet devices on both machines= . >> >> What is the number of hugepages you have allocated for both machines. >> >> Also look at the cpu_layout.py script to see why adding the 5th core >> would be different on the two machines and try to make them the same. >> >> > >> > Regards, >> > Kai >> > >> > On Sat, Mar 11, 2017 at 10:52 PM, Wiles, Keith >> wrote: >> > >> > > On Mar 10, 2017, at 9:35 PM, Kai Zhang wrote: >> > > >> > > Hi, there >> > > >> > > I am using DPDK-16.11 on XL710 40GbE NIC. OS: CentOS 7.3.1611 with >> Linux >> > > kernel version 3.8.0-30. >> > > >> > > I have a master process and a secondary process. When I run the >> secondary >> > > process with less than or equal to 4 cores, it works correctly. Such >> as: >> > > sudo ./program -l 4,5,6,7 -n 4 --proc-type=3Dsecondary >> > > sudo ./program -c 0x0f -n 4 --proc-type=3Dsecondary >> > > >> > > However, there will be error in the rte_eal_init if I assign more >> than 4 >> > > cores. >> > > sudo ./program -l 0,1,2,3,4 -n 4 --proc-type=3Dsecondary >> > > sudo ./program -c 0x1f -n 4 --proc-type=3Dsecondary >> > > >> > > EAL: Cannot mmap device resource file >> > > /sys/bus/pci/devices/0000:02:00.0/resource0 to address: >> 0x7fff65bfc000 >> > > EAL: Error - exiting with code: 1 >> > > Cause: Requested device 0000:02:00.0 cannot be used >> > >> > I assume you have at least 8 cores. Have you tried -l 1-5 on the >> secondary process. >> > >> > You did not show the primary process command line, but the if you use >> 1-5 then you can only give primary process -l 6-7 or two cores. It is >> always a reasonable thing is to leave core zero for linux to use. >> > >> > Also it could be you ran out of memory or hugepages you allocated to >> the system. >> > >> > > >> > > Anyone knows why this happens? >> > > >> > > Thanks a lot, >> > > Kai Zhang >> > >> > Regards, >> > Keith >> > >> > >> >> Regards, >> Keith >> >> >