* [dpdk-dev] problem: Cannot allocate memzone for ethernet port data @ 2015-04-22 10:32 ChenXiaodong 2015-04-22 11:01 ` De Lara Guarch, Pablo 0 siblings, 1 reply; 6+ messages in thread From: ChenXiaodong @ 2015-04-22 10:32 UTC (permalink / raw) To: dev Hi, every one Is there any one who can help me ? I'm new to DPDK, here is a problem that I don't know how to deal with. The program is running on an machine with two Intel 82576 NICs. One of the NICs is binded with the uio_pci_generic driver。1024 2M hugepages have been setup. Any suggestion about the cause of the problem or solution to it will be appreciated! Here is the output of the program: [root@test dpdk-2.0.0]# build/app/testpmd -c ff -n 4 --socket-mem=1024 EAL: Cannot read numa node link for lcore 0 - using physical package id instead EAL: Detected lcore 0 as core 0 on socket 1 EAL: Cannot read numa node link for lcore 1 - using physical package id instead EAL: Detected lcore 1 as core 1 on socket 1 EAL: Cannot read numa node link for lcore 2 - using physical package id instead EAL: Detected lcore 2 as core 2 on socket 1 EAL: Cannot read numa node link for lcore 3 - using physical package id instead EAL: Detected lcore 3 as core 3 on socket 1 EAL: Cannot read numa node link for lcore 4 - using physical package id instead EAL: Detected lcore 4 as core 0 on socket 1 EAL: Cannot read numa node link for lcore 5 - using physical package id instead EAL: Detected lcore 5 as core 1 on socket 1 EAL: Cannot read numa node link for lcore 6 - using physical package id instead EAL: Detected lcore 6 as core 2 on socket 1 EAL: Cannot read numa node link for lcore 7 - using physical package id instead EAL: Detected lcore 7 as core 3 on socket 1 EAL: Support maximum 128 logical core(s) by configuration. EAL: Detected 8 lcore(s) EAL: Setting up memory... EAL: Ask a virtual area of 0x8000000 bytes EAL: Virtual area found at 0x7f83fae00000 (size = 0x8000000) EAL: Ask a virtual area of 0x38000000 bytes EAL: Virtual area found at 0x7f83c2c00000 (size = 0x38000000) EAL: Requesting 512 pages of size 2MB from socket 0 EAL: TSC frequency is ~2394002 KHz EAL: WARNING: Master core has no memory on local socket! EAL: Cannot read numa node link for lcore 0 - using physical package id instead EAL: Cannot read numa node link for lcore 0 - using physical package id instead EAL: Master lcore 0 is ready (tid=319e880;cpuset=[0]) PMD: ENICPMD trace: rte_enic_pmd_init EAL: Cannot read numa node link for lcore 4 - using physical package id instead EAL: Cannot read numa node link for lcore 5 - using physical package id instead EAL: Cannot read numa node link for lcore 1 - using physical package id instead EAL: Cannot read numa node link for lcore 1 - using physical package id instead EAL: Cannot read numa node link for lcore 2 - using physical package id instead EAL: Cannot read numa node link for lcore 3 - using physical package id instead EAL: Cannot read numa node link for lcore 5 - using physical package id instead EAL: Cannot read numa node link for lcore 6 - using physical package id instead EAL: Cannot read numa node link for lcore 7 - using physical package id instead EAL: Cannot read numa node link for lcore 2 - using physical package id instead EAL: Cannot read numa node link for lcore 7 - using physical package id instead EAL: lcore 1 is ready (tid=c1ff8700;cpuset=[1]) EAL: lcore 2 is ready (tid=c15f7700;cpuset=[2]) EAL: Cannot read numa node link for lcore 4 - using physical package id instead EAL: Cannot read numa node link for lcore 3 - using physical package id instead EAL: lcore 5 is ready (tid=bf7f4700;cpuset=[5]) EAL: Cannot read numa node link for lcore 6 - using physical package id instead EAL: lcore 7 is ready (tid=be3f2700;cpuset=[7]) EAL: lcore 3 is ready (tid=c0bf6700;cpuset=[3]) EAL: lcore 6 is ready (tid=bedf3700;cpuset=[6]) EAL: lcore 4 is ready (tid=c01f5700;cpuset=[4]) EAL: PCI device 0000:07:00.0 on NUMA socket -1 EAL: probe driver: 8086:10c9 rte_igb_pmd EAL: PCI memory mapped at 0x7f8402e00000 EAL: PCI memory mapped at 0x7f83bd5f2000 EAL: PCI memory mapped at 0x7f84031ba000 PANIC in rte_eth_dev_data_alloc(): Cannot allocate memzone for ethernet port data 8: [build/app/testpmd() [0x4285d9]] 7: [/lib64/libc.so.6(__libc_start_main+0xfd) [0x3d18e1ecdd]] 6: [build/app/testpmd(main+0x18) [0x42dce8]] 5: [build/app/testpmd(rte_eal_init+0xab8) [0x4982c8]] 4: [build/app/testpmd(rte_eal_pci_probe+0xe2) [0x4a27e2]] 3: [build/app/testpmd() [0x491ea7]] 2: [build/app/testpmd(__rte_panic+0xc0) [0x428520]] 1: [build/app/testpmd(rte_dump_stack+0x1e) [0x49e91e]] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] problem: Cannot allocate memzone for ethernet port data 2015-04-22 10:32 [dpdk-dev] problem: Cannot allocate memzone for ethernet port data ChenXiaodong @ 2015-04-22 11:01 ` De Lara Guarch, Pablo 2015-04-22 11:36 ` ChenXiaodong 0 siblings, 1 reply; 6+ messages in thread From: De Lara Guarch, Pablo @ 2015-04-22 11:01 UTC (permalink / raw) To: ChenXiaodong, dev Hi , > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of ChenXiaodong > Sent: Wednesday, April 22, 2015 11:33 AM > To: dev@dpdk.org > Subject: [dpdk-dev] problem: Cannot allocate memzone for ethernet port > data > > Hi, every one > > Is there any one who can help me ? I'm new to DPDK, here is a problem that > I don't know how to deal with. The program is running on an machine with > two Intel 82576 NICs. One of the NICs is binded with the uio_pci_generic > driver。1024 2M hugepages have been setup. > > Any suggestion about the cause of the problem or solution to it will be > appreciated! Could you tell us what these commands return? ./tools/cpu_layout.py cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages cat /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages (assuming you have 2 NUMA nodes) Pablo > > Here is the output of the program: > > [root@test dpdk-2.0.0]# build/app/testpmd -c ff -n 4 --socket-mem=1024 > EAL: Cannot read numa node link for lcore 0 - using physical package id > instead > EAL: Detected lcore 0 as core 0 on socket 1 > EAL: Cannot read numa node link for lcore 1 - using physical package id > instead > EAL: Detected lcore 1 as core 1 on socket 1 > EAL: Cannot read numa node link for lcore 2 - using physical package id > instead > EAL: Detected lcore 2 as core 2 on socket 1 > EAL: Cannot read numa node link for lcore 3 - using physical package id > instead > EAL: Detected lcore 3 as core 3 on socket 1 > EAL: Cannot read numa node link for lcore 4 - using physical package id > instead > EAL: Detected lcore 4 as core 0 on socket 1 > EAL: Cannot read numa node link for lcore 5 - using physical package id > instead > EAL: Detected lcore 5 as core 1 on socket 1 > EAL: Cannot read numa node link for lcore 6 - using physical package id > instead > EAL: Detected lcore 6 as core 2 on socket 1 > EAL: Cannot read numa node link for lcore 7 - using physical package id > instead > EAL: Detected lcore 7 as core 3 on socket 1 > EAL: Support maximum 128 logical core(s) by configuration. > EAL: Detected 8 lcore(s) > EAL: Setting up memory... > EAL: Ask a virtual area of 0x8000000 bytes > EAL: Virtual area found at 0x7f83fae00000 (size = 0x8000000) > EAL: Ask a virtual area of 0x38000000 bytes > EAL: Virtual area found at 0x7f83c2c00000 (size = 0x38000000) > EAL: Requesting 512 pages of size 2MB from socket 0 > EAL: TSC frequency is ~2394002 KHz > EAL: WARNING: Master core has no memory on local socket! > EAL: Cannot read numa node link for lcore 0 - using physical package id > instead > EAL: Cannot read numa node link for lcore 0 - using physical package id > instead > EAL: Master lcore 0 is ready (tid=319e880;cpuset=[0]) > PMD: ENICPMD trace: rte_enic_pmd_init > EAL: Cannot read numa node link for lcore 4 - using physical package id > instead > EAL: Cannot read numa node link for lcore 5 - using physical package id > instead > EAL: Cannot read numa node link for lcore 1 - using physical package id > instead > EAL: Cannot read numa node link for lcore 1 - using physical package id > instead > EAL: Cannot read numa node link for lcore 2 - using physical package id > instead > EAL: Cannot read numa node link for lcore 3 - using physical package id > instead > EAL: Cannot read numa node link for lcore 5 - using physical package id > instead > EAL: Cannot read numa node link for lcore 6 - using physical package id > instead > EAL: Cannot read numa node link for lcore 7 - using physical package id > instead > EAL: Cannot read numa node link for lcore 2 - using physical package id > instead > EAL: Cannot read numa node link for lcore 7 - using physical package id > instead > EAL: lcore 1 is ready (tid=c1ff8700;cpuset=[1]) > EAL: lcore 2 is ready (tid=c15f7700;cpuset=[2]) > EAL: Cannot read numa node link for lcore 4 - using physical package id > instead > EAL: Cannot read numa node link for lcore 3 - using physical package id > instead > EAL: lcore 5 is ready (tid=bf7f4700;cpuset=[5]) > EAL: Cannot read numa node link for lcore 6 - using physical package id > instead > EAL: lcore 7 is ready (tid=be3f2700;cpuset=[7]) > EAL: lcore 3 is ready (tid=c0bf6700;cpuset=[3]) > EAL: lcore 6 is ready (tid=bedf3700;cpuset=[6]) > EAL: lcore 4 is ready (tid=c01f5700;cpuset=[4]) > EAL: PCI device 0000:07:00.0 on NUMA socket -1 > EAL: probe driver: 8086:10c9 rte_igb_pmd > EAL: PCI memory mapped at 0x7f8402e00000 > EAL: PCI memory mapped at 0x7f83bd5f2000 > EAL: PCI memory mapped at 0x7f84031ba000 > PANIC in rte_eth_dev_data_alloc(): > Cannot allocate memzone for ethernet port data > 8: [build/app/testpmd() [0x4285d9]] > 7: [/lib64/libc.so.6(__libc_start_main+0xfd) [0x3d18e1ecdd]] > 6: [build/app/testpmd(main+0x18) [0x42dce8]] > 5: [build/app/testpmd(rte_eal_init+0xab8) [0x4982c8]] > 4: [build/app/testpmd(rte_eal_pci_probe+0xe2) [0x4a27e2]] > 3: [build/app/testpmd() [0x491ea7]] > 2: [build/app/testpmd(__rte_panic+0xc0) [0x428520]] > 1: [build/app/testpmd(rte_dump_stack+0x1e) [0x49e91e]] > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] problem: Cannot allocate memzone for ethernet port data 2015-04-22 11:01 ` De Lara Guarch, Pablo @ 2015-04-22 11:36 ` ChenXiaodong 2015-04-22 13:34 ` Bruce Richardson 0 siblings, 1 reply; 6+ messages in thread From: ChenXiaodong @ 2015-04-22 11:36 UTC (permalink / raw) To: De Lara Guarch, Pablo, dev Thanks for you reply! There is actually only one NUMA node. here is what the commands return: [root@test dpdk-2.0.0]# tools/cpu_layout.py ============================================================ Core and Socket Information (as reported by '/proc/cpuinfo') ============================================================ cores = [0, 1, 2, 3] sockets = [1] Socket 1 -------- Core 0 [0, 4] Core 1 [1, 5] Core 2 [2, 6] Core 3 [3, 7] [root@test dpdk-2.0.0]# cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages 1024 > From: pablo.de.lara.guarch@intel.com > To: ch.xd@live.cn; dev@dpdk.org > Subject: RE: [dpdk-dev] problem: Cannot allocate memzone for ethernet port data > Date: Wed, 22 Apr 2015 11:01:08 +0000 > > Hi , > > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of ChenXiaodong > > Sent: Wednesday, April 22, 2015 11:33 AM > > To: dev@dpdk.org > > Subject: [dpdk-dev] problem: Cannot allocate memzone for ethernet port > > data > > > > Hi, every one > > > > Is there any one who can help me ? I'm new to DPDK, here is a problem that > > I don't know how to deal with. The program is running on an machine with > > two Intel 82576 NICs. One of the NICs is binded with the uio_pci_generic > > driver。1024 2M hugepages have been setup. > > > > Any suggestion about the cause of the problem or solution to it will be > > appreciated! > > Could you tell us what these commands return? > > ./tools/cpu_layout.py > cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages > cat /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages > > (assuming you have 2 NUMA nodes) > > Pablo > > > > Here is the output of the program: > > > > [root@test dpdk-2.0.0]# build/app/testpmd -c ff -n 4 --socket-mem=1024 > > EAL: Cannot read numa node link for lcore 0 - using physical package id > > instead > > EAL: Detected lcore 0 as core 0 on socket 1 > > EAL: Cannot read numa node link for lcore 1 - using physical package id > > instead > > EAL: Detected lcore 1 as core 1 on socket 1 > > EAL: Cannot read numa node link for lcore 2 - using physical package id > > instead > > EAL: Detected lcore 2 as core 2 on socket 1 > > EAL: Cannot read numa node link for lcore 3 - using physical package id > > instead > > EAL: Detected lcore 3 as core 3 on socket 1 > > EAL: Cannot read numa node link for lcore 4 - using physical package id > > instead > > EAL: Detected lcore 4 as core 0 on socket 1 > > EAL: Cannot read numa node link for lcore 5 - using physical package id > > instead > > EAL: Detected lcore 5 as core 1 on socket 1 > > EAL: Cannot read numa node link for lcore 6 - using physical package id > > instead > > EAL: Detected lcore 6 as core 2 on socket 1 > > EAL: Cannot read numa node link for lcore 7 - using physical package id > > instead > > EAL: Detected lcore 7 as core 3 on socket 1 > > EAL: Support maximum 128 logical core(s) by configuration. > > EAL: Detected 8 lcore(s) > > EAL: Setting up memory... > > EAL: Ask a virtual area of 0x8000000 bytes > > EAL: Virtual area found at 0x7f83fae00000 (size = 0x8000000) > > EAL: Ask a virtual area of 0x38000000 bytes > > EAL: Virtual area found at 0x7f83c2c00000 (size = 0x38000000) > > EAL: Requesting 512 pages of size 2MB from socket 0 > > EAL: TSC frequency is ~2394002 KHz > > EAL: WARNING: Master core has no memory on local socket! > > EAL: Cannot read numa node link for lcore 0 - using physical package id > > instead > > EAL: Cannot read numa node link for lcore 0 - using physical package id > > instead > > EAL: Master lcore 0 is ready (tid=319e880;cpuset=[0]) > > PMD: ENICPMD trace: rte_enic_pmd_init > > EAL: Cannot read numa node link for lcore 4 - using physical package id > > instead > > EAL: Cannot read numa node link for lcore 5 - using physical package id > > instead > > EAL: Cannot read numa node link for lcore 1 - using physical package id > > instead > > EAL: Cannot read numa node link for lcore 1 - using physical package id > > instead > > EAL: Cannot read numa node link for lcore 2 - using physical package id > > instead > > EAL: Cannot read numa node link for lcore 3 - using physical package id > > instead > > EAL: Cannot read numa node link for lcore 5 - using physical package id > > instead > > EAL: Cannot read numa node link for lcore 6 - using physical package id > > instead > > EAL: Cannot read numa node link for lcore 7 - using physical package id > > instead > > EAL: Cannot read numa node link for lcore 2 - using physical package id > > instead > > EAL: Cannot read numa node link for lcore 7 - using physical package id > > instead > > EAL: lcore 1 is ready (tid=c1ff8700;cpuset=[1]) > > EAL: lcore 2 is ready (tid=c15f7700;cpuset=[2]) > > EAL: Cannot read numa node link for lcore 4 - using physical package id > > instead > > EAL: Cannot read numa node link for lcore 3 - using physical package id > > instead > > EAL: lcore 5 is ready (tid=bf7f4700;cpuset=[5]) > > EAL: Cannot read numa node link for lcore 6 - using physical package id > > instead > > EAL: lcore 7 is ready (tid=be3f2700;cpuset=[7]) > > EAL: lcore 3 is ready (tid=c0bf6700;cpuset=[3]) > > EAL: lcore 6 is ready (tid=bedf3700;cpuset=[6]) > > EAL: lcore 4 is ready (tid=c01f5700;cpuset=[4]) > > EAL: PCI device 0000:07:00.0 on NUMA socket -1 > > EAL: probe driver: 8086:10c9 rte_igb_pmd > > EAL: PCI memory mapped at 0x7f8402e00000 > > EAL: PCI memory mapped at 0x7f83bd5f2000 > > EAL: PCI memory mapped at 0x7f84031ba000 > > PANIC in rte_eth_dev_data_alloc(): > > Cannot allocate memzone for ethernet port data > > 8: [build/app/testpmd() [0x4285d9]] > > 7: [/lib64/libc.so.6(__libc_start_main+0xfd) [0x3d18e1ecdd]] > > 6: [build/app/testpmd(main+0x18) [0x42dce8]] > > 5: [build/app/testpmd(rte_eal_init+0xab8) [0x4982c8]] > > 4: [build/app/testpmd(rte_eal_pci_probe+0xe2) [0x4a27e2]] > > 3: [build/app/testpmd() [0x491ea7]] > > 2: [build/app/testpmd(__rte_panic+0xc0) [0x428520]] > > 1: [build/app/testpmd(rte_dump_stack+0x1e) [0x49e91e]] > > > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] problem: Cannot allocate memzone for ethernet port data 2015-04-22 11:36 ` ChenXiaodong @ 2015-04-22 13:34 ` Bruce Richardson 2015-04-22 15:20 ` ChenXiaodong 2015-04-22 15:41 ` Stephen Hemminger 0 siblings, 2 replies; 6+ messages in thread From: Bruce Richardson @ 2015-04-22 13:34 UTC (permalink / raw) To: ChenXiaodong; +Cc: dev On Wed, Apr 22, 2015 at 07:36:59PM +0800, ChenXiaodong wrote: > Thanks for you reply! There is actually only one NUMA node. here is what the commands return: > Hi, there seems to be a mismatch in terms of the numa node the memory is reported as being on (node 0) and the numa node that the cores are detected as being on - as reported by the physical package id value. That's why the app is failing as it's looking for the memory on the wrong numa node. Can you try applying the patch I've just posted to change how we detect the numa node of a core, and see if it helps in your case. thanks, /Bruce > [root@test dpdk-2.0.0]# tools/cpu_layout.py > ============================================================ > Core and Socket Information (as reported by '/proc/cpuinfo') > ============================================================ > > cores = [0, 1, 2, 3] > sockets = [1] > > Socket 1 > -------- > Core 0 [0, 4] > > Core 1 [1, 5] > > Core 2 [2, 6] > > Core 3 [3, 7] > > [root@test dpdk-2.0.0]# cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages > 1024 > > > From: pablo.de.lara.guarch@intel.com > > To: ch.xd@live.cn; dev@dpdk.org > > Subject: RE: [dpdk-dev] problem: Cannot allocate memzone for ethernet port data > > Date: Wed, 22 Apr 2015 11:01:08 +0000 > > > > Hi , > > > > > -----Original Message----- > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of ChenXiaodong > > > Sent: Wednesday, April 22, 2015 11:33 AM > > > To: dev@dpdk.org > > > Subject: [dpdk-dev] problem: Cannot allocate memzone for ethernet port > > > data > > > > > > Hi, every one > > > > > > Is there any one who can help me ? I'm new to DPDK, here is a problem that > > > I don't know how to deal with. The program is running on an machine with > > > two Intel 82576 NICs. One of the NICs is binded with the uio_pci_generic > > > driver。1024 2M hugepages have been setup. > > > > > > Any suggestion about the cause of the problem or solution to it will be > > > appreciated! > > > > Could you tell us what these commands return? > > > > ./tools/cpu_layout.py > > cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages > > cat /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages > > > > (assuming you have 2 NUMA nodes) > > > > Pablo > > > > > > Here is the output of the program: > > > > > > [root@test dpdk-2.0.0]# build/app/testpmd -c ff -n 4 --socket-mem=1024 > > > EAL: Cannot read numa node link for lcore 0 - using physical package id > > > instead > > > EAL: Detected lcore 0 as core 0 on socket 1 > > > EAL: Cannot read numa node link for lcore 1 - using physical package id > > > instead > > > EAL: Detected lcore 1 as core 1 on socket 1 > > > EAL: Cannot read numa node link for lcore 2 - using physical package id > > > instead > > > EAL: Detected lcore 2 as core 2 on socket 1 > > > EAL: Cannot read numa node link for lcore 3 - using physical package id > > > instead > > > EAL: Detected lcore 3 as core 3 on socket 1 > > > EAL: Cannot read numa node link for lcore 4 - using physical package id > > > instead > > > EAL: Detected lcore 4 as core 0 on socket 1 > > > EAL: Cannot read numa node link for lcore 5 - using physical package id > > > instead > > > EAL: Detected lcore 5 as core 1 on socket 1 > > > EAL: Cannot read numa node link for lcore 6 - using physical package id > > > instead > > > EAL: Detected lcore 6 as core 2 on socket 1 > > > EAL: Cannot read numa node link for lcore 7 - using physical package id > > > instead > > > EAL: Detected lcore 7 as core 3 on socket 1 > > > EAL: Support maximum 128 logical core(s) by configuration. > > > EAL: Detected 8 lcore(s) > > > EAL: Setting up memory... > > > EAL: Ask a virtual area of 0x8000000 bytes > > > EAL: Virtual area found at 0x7f83fae00000 (size = 0x8000000) > > > EAL: Ask a virtual area of 0x38000000 bytes > > > EAL: Virtual area found at 0x7f83c2c00000 (size = 0x38000000) > > > EAL: Requesting 512 pages of size 2MB from socket 0 > > > EAL: TSC frequency is ~2394002 KHz > > > EAL: WARNING: Master core has no memory on local socket! > > > EAL: Cannot read numa node link for lcore 0 - using physical package id > > > instead > > > EAL: Cannot read numa node link for lcore 0 - using physical package id > > > instead > > > EAL: Master lcore 0 is ready (tid=319e880;cpuset=[0]) > > > PMD: ENICPMD trace: rte_enic_pmd_init > > > EAL: Cannot read numa node link for lcore 4 - using physical package id > > > instead > > > EAL: Cannot read numa node link for lcore 5 - using physical package id > > > instead > > > EAL: Cannot read numa node link for lcore 1 - using physical package id > > > instead > > > EAL: Cannot read numa node link for lcore 1 - using physical package id > > > instead > > > EAL: Cannot read numa node link for lcore 2 - using physical package id > > > instead > > > EAL: Cannot read numa node link for lcore 3 - using physical package id > > > instead > > > EAL: Cannot read numa node link for lcore 5 - using physical package id > > > instead > > > EAL: Cannot read numa node link for lcore 6 - using physical package id > > > instead > > > EAL: Cannot read numa node link for lcore 7 - using physical package id > > > instead > > > EAL: Cannot read numa node link for lcore 2 - using physical package id > > > instead > > > EAL: Cannot read numa node link for lcore 7 - using physical package id > > > instead > > > EAL: lcore 1 is ready (tid=c1ff8700;cpuset=[1]) > > > EAL: lcore 2 is ready (tid=c15f7700;cpuset=[2]) > > > EAL: Cannot read numa node link for lcore 4 - using physical package id > > > instead > > > EAL: Cannot read numa node link for lcore 3 - using physical package id > > > instead > > > EAL: lcore 5 is ready (tid=bf7f4700;cpuset=[5]) > > > EAL: Cannot read numa node link for lcore 6 - using physical package id > > > instead > > > EAL: lcore 7 is ready (tid=be3f2700;cpuset=[7]) > > > EAL: lcore 3 is ready (tid=c0bf6700;cpuset=[3]) > > > EAL: lcore 6 is ready (tid=bedf3700;cpuset=[6]) > > > EAL: lcore 4 is ready (tid=c01f5700;cpuset=[4]) > > > EAL: PCI device 0000:07:00.0 on NUMA socket -1 > > > EAL: probe driver: 8086:10c9 rte_igb_pmd > > > EAL: PCI memory mapped at 0x7f8402e00000 > > > EAL: PCI memory mapped at 0x7f83bd5f2000 > > > EAL: PCI memory mapped at 0x7f84031ba000 > > > PANIC in rte_eth_dev_data_alloc(): > > > Cannot allocate memzone for ethernet port data > > > 8: [build/app/testpmd() [0x4285d9]] > > > 7: [/lib64/libc.so.6(__libc_start_main+0xfd) [0x3d18e1ecdd]] > > > 6: [build/app/testpmd(main+0x18) [0x42dce8]] > > > 5: [build/app/testpmd(rte_eal_init+0xab8) [0x4982c8]] > > > 4: [build/app/testpmd(rte_eal_pci_probe+0xe2) [0x4a27e2]] > > > 3: [build/app/testpmd() [0x491ea7]] > > > 2: [build/app/testpmd(__rte_panic+0xc0) [0x428520]] > > > 1: [build/app/testpmd(rte_dump_stack+0x1e) [0x49e91e]] > > > > > > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] problem: Cannot allocate memzone for ethernet port data 2015-04-22 13:34 ` Bruce Richardson @ 2015-04-22 15:20 ` ChenXiaodong 2015-04-22 15:41 ` Stephen Hemminger 1 sibling, 0 replies; 6+ messages in thread From: ChenXiaodong @ 2015-04-22 15:20 UTC (permalink / raw) To: Bruce Richardson; +Cc: dev Hi, Bruce Your patch works! I applied the patch, and the problem disappeared. Thanks ^_^ /ChenXiaodong > Date: Wed, 22 Apr 2015 14:34:17 +0100 > From: bruce.richardson@intel.com > To: ch.xd@live.cn > CC: pablo.de.lara.guarch@intel.com; dev@dpdk.org > Subject: Re: [dpdk-dev] problem: Cannot allocate memzone for ethernet port data > > On Wed, Apr 22, 2015 at 07:36:59PM +0800, ChenXiaodong wrote: > > Thanks for you reply! There is actually only one NUMA node. here is what the commands return: > > > > Hi, > > there seems to be a mismatch in terms of the numa node the memory is reported > as being on (node 0) and the numa node that the cores are detected as being on > - as reported by the physical package id value. That's why the app is failing > as it's looking for the memory on the wrong numa node. > Can you try applying the patch I've just posted to change how we detect > the numa node of a core, and see if it helps in your case. > > thanks, > /Bruce > > > [root@test dpdk-2.0.0]# tools/cpu_layout.py > > ============================================================ > > Core and Socket Information (as reported by '/proc/cpuinfo') > > ============================================================ > > > > cores = [0, 1, 2, 3] > > sockets = [1] > > > > Socket 1 > > -------- > > Core 0 [0, 4] > > > > Core 1 [1, 5] > > > > Core 2 [2, 6] > > > > Core 3 [3, 7] > > > > [root@test dpdk-2.0.0]# cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages > > 1024 > > > > > From: pablo.de.lara.guarch@intel.com > > > To: ch.xd@live.cn; dev@dpdk.org > > > Subject: RE: [dpdk-dev] problem: Cannot allocate memzone for ethernet port data > > > Date: Wed, 22 Apr 2015 11:01:08 +0000 > > > > > > Hi , > > > > > > > -----Original Message----- > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of ChenXiaodong > > > > Sent: Wednesday, April 22, 2015 11:33 AM > > > > To: dev@dpdk.org > > > > Subject: [dpdk-dev] problem: Cannot allocate memzone for ethernet port > > > > data > > > > > > > > Hi, every one > > > > > > > > Is there any one who can help me ? I'm new to DPDK, here is a problem that > > > > I don't know how to deal with. The program is running on an machine with > > > > two Intel 82576 NICs. One of the NICs is binded with the uio_pci_generic > > > > driver。1024 2M hugepages have been setup. > > > > > > > > Any suggestion about the cause of the problem or solution to it will be > > > > appreciated! > > > > > > Could you tell us what these commands return? > > > > > > ./tools/cpu_layout.py > > > cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages > > > cat /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages > > > > > > (assuming you have 2 NUMA nodes) > > > > > > Pablo > > > > > > > > Here is the output of the program: > > > > > > > > [root@test dpdk-2.0.0]# build/app/testpmd -c ff -n 4 --socket-mem=1024 > > > > EAL: Cannot read numa node link for lcore 0 - using physical package id > > > > instead > > > > EAL: Detected lcore 0 as core 0 on socket 1 > > > > EAL: Cannot read numa node link for lcore 1 - using physical package id > > > > instead > > > > EAL: Detected lcore 1 as core 1 on socket 1 > > > > EAL: Cannot read numa node link for lcore 2 - using physical package id > > > > instead > > > > EAL: Detected lcore 2 as core 2 on socket 1 > > > > EAL: Cannot read numa node link for lcore 3 - using physical package id > > > > instead > > > > EAL: Detected lcore 3 as core 3 on socket 1 > > > > EAL: Cannot read numa node link for lcore 4 - using physical package id > > > > instead > > > > EAL: Detected lcore 4 as core 0 on socket 1 > > > > EAL: Cannot read numa node link for lcore 5 - using physical package id > > > > instead > > > > EAL: Detected lcore 5 as core 1 on socket 1 > > > > EAL: Cannot read numa node link for lcore 6 - using physical package id > > > > instead > > > > EAL: Detected lcore 6 as core 2 on socket 1 > > > > EAL: Cannot read numa node link for lcore 7 - using physical package id > > > > instead > > > > EAL: Detected lcore 7 as core 3 on socket 1 > > > > EAL: Support maximum 128 logical core(s) by configuration. > > > > EAL: Detected 8 lcore(s) > > > > EAL: Setting up memory... > > > > EAL: Ask a virtual area of 0x8000000 bytes > > > > EAL: Virtual area found at 0x7f83fae00000 (size = 0x8000000) > > > > EAL: Ask a virtual area of 0x38000000 bytes > > > > EAL: Virtual area found at 0x7f83c2c00000 (size = 0x38000000) > > > > EAL: Requesting 512 pages of size 2MB from socket 0 > > > > EAL: TSC frequency is ~2394002 KHz > > > > EAL: WARNING: Master core has no memory on local socket! > > > > EAL: Cannot read numa node link for lcore 0 - using physical package id > > > > instead > > > > EAL: Cannot read numa node link for lcore 0 - using physical package id > > > > instead > > > > EAL: Master lcore 0 is ready (tid=319e880;cpuset=[0]) > > > > PMD: ENICPMD trace: rte_enic_pmd_init > > > > EAL: Cannot read numa node link for lcore 4 - using physical package id > > > > instead > > > > EAL: Cannot read numa node link for lcore 5 - using physical package id > > > > instead > > > > EAL: Cannot read numa node link for lcore 1 - using physical package id > > > > instead > > > > EAL: Cannot read numa node link for lcore 1 - using physical package id > > > > instead > > > > EAL: Cannot read numa node link for lcore 2 - using physical package id > > > > instead > > > > EAL: Cannot read numa node link for lcore 3 - using physical package id > > > > instead > > > > EAL: Cannot read numa node link for lcore 5 - using physical package id > > > > instead > > > > EAL: Cannot read numa node link for lcore 6 - using physical package id > > > > instead > > > > EAL: Cannot read numa node link for lcore 7 - using physical package id > > > > instead > > > > EAL: Cannot read numa node link for lcore 2 - using physical package id > > > > instead > > > > EAL: Cannot read numa node link for lcore 7 - using physical package id > > > > instead > > > > EAL: lcore 1 is ready (tid=c1ff8700;cpuset=[1]) > > > > EAL: lcore 2 is ready (tid=c15f7700;cpuset=[2]) > > > > EAL: Cannot read numa node link for lcore 4 - using physical package id > > > > instead > > > > EAL: Cannot read numa node link for lcore 3 - using physical package id > > > > instead > > > > EAL: lcore 5 is ready (tid=bf7f4700;cpuset=[5]) > > > > EAL: Cannot read numa node link for lcore 6 - using physical package id > > > > instead > > > > EAL: lcore 7 is ready (tid=be3f2700;cpuset=[7]) > > > > EAL: lcore 3 is ready (tid=c0bf6700;cpuset=[3]) > > > > EAL: lcore 6 is ready (tid=bedf3700;cpuset=[6]) > > > > EAL: lcore 4 is ready (tid=c01f5700;cpuset=[4]) > > > > EAL: PCI device 0000:07:00.0 on NUMA socket -1 > > > > EAL: probe driver: 8086:10c9 rte_igb_pmd > > > > EAL: PCI memory mapped at 0x7f8402e00000 > > > > EAL: PCI memory mapped at 0x7f83bd5f2000 > > > > EAL: PCI memory mapped at 0x7f84031ba000 > > > > PANIC in rte_eth_dev_data_alloc(): > > > > Cannot allocate memzone for ethernet port data > > > > 8: [build/app/testpmd() [0x4285d9]] > > > > 7: [/lib64/libc.so.6(__libc_start_main+0xfd) [0x3d18e1ecdd]] > > > > 6: [build/app/testpmd(main+0x18) [0x42dce8]] > > > > 5: [build/app/testpmd(rte_eal_init+0xab8) [0x4982c8]] > > > > 4: [build/app/testpmd(rte_eal_pci_probe+0xe2) [0x4a27e2]] > > > > 3: [build/app/testpmd() [0x491ea7]] > > > > 2: [build/app/testpmd(__rte_panic+0xc0) [0x428520]] > > > > 1: [build/app/testpmd(rte_dump_stack+0x1e) [0x49e91e]] > > > > > > > > > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] problem: Cannot allocate memzone for ethernet port data 2015-04-22 13:34 ` Bruce Richardson 2015-04-22 15:20 ` ChenXiaodong @ 2015-04-22 15:41 ` Stephen Hemminger 1 sibling, 0 replies; 6+ messages in thread From: Stephen Hemminger @ 2015-04-22 15:41 UTC (permalink / raw) To: Bruce Richardson; +Cc: dev On Wed, 22 Apr 2015 14:34:17 +0100 Bruce Richardson <bruce.richardson@intel.com> wrote: > On Wed, Apr 22, 2015 at 07:36:59PM +0800, ChenXiaodong wrote: > > Thanks for you reply! There is actually only one NUMA node. here is what the commands return: > > > > Hi, > > there seems to be a mismatch in terms of the numa node the memory is reported > as being on (node 0) and the numa node that the cores are detected as being on > - as reported by the physical package id value. That's why the app is failing > as it's looking for the memory on the wrong numa node. > Can you try applying the patch I've just posted to change how we detect > the numa node of a core, and see if it helps in your case. > > thanks, > /Bruce This issue has been around a long time. Older kernels (pre 3.3) don't have a good solution and are really broken. Some old history: > On Fri, 31 May 2013 11:29:09 +0200 > Damien Millescamps <damien.millescamps at 6wind.com> wrote: > > > On 05/30/2013 07:12 PM, Stephen Hemminger wrote: > > > One fix not included is the NUMA cpu assignment. The original Intel code > > > in 1.2 incorrectly used /proc/cpuinfo to try and assign CPU's to NUMA socket. > > > The problem is that /proc/cpuinfo physical_id corresponds to what the BIOS > > > tells the kernel and is intended for messages only. For example, on our > > > Dell boxes the first CPU and only CPU is reported as physical_id 1! > > > The fix is to use sysfs instead, > > Hi Stephen, > > > > Are you using the > > /sys/devices/system/node/nodeX/cpuX/topology/physical_package_id special > > file ? > > > > If so, then it is only usable starting from Kernel 3.3 according to this > > fix: > > http://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/commit/?id=64be4c1c2428e148de6081af235e2418e6a66dda > > > > The value returned for kernel prior to 3.3 is really not better than the > > one from /proc/cpuinfo, so that's kind of a robbing Peter to pay Paul > > example... > > > > There is obviously a problem with the NUMA node ID detection right now > > since both /proc/cpuinfo and /sys can return incorrect values, however > > the node and cpu numbering is always good in the kernel boot log from > > what I know. So there might be a better way to find the real node ID > > whatever the kernel version used. > > > > Cheers, > > You need to use /sys/devices/system/cpu/cpuN/topology/physical_package_id > as documented in kernel (Documentation/cputopology.txt). > This was confirmed by several kernel developers including Andi Kleen from > Intel. The value in /proc/cpuinfo comes from the APCI tables and is the value > reported by the BIOS. There are machines that report socket 1 and 2. > Haven't played with older kernels, but the topology information in sysfs > has existed since 2.6.16. > > The final Intel solution was to use physical_packate_id and fall back > to /proc/cpuinfo as last resort. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2015-04-22 15:41 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-04-22 10:32 [dpdk-dev] problem: Cannot allocate memzone for ethernet port data ChenXiaodong 2015-04-22 11:01 ` De Lara Guarch, Pablo 2015-04-22 11:36 ` ChenXiaodong 2015-04-22 13:34 ` Bruce Richardson 2015-04-22 15:20 ` ChenXiaodong 2015-04-22 15:41 ` Stephen Hemminger
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).