From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by dpdk.org (Postfix, from userid 33) id 072821B157; Tue, 16 Oct 2018 18:38:40 +0200 (CEST) From: bugzilla@dpdk.org To: dev@dpdk.org Date: Tue, 16 Oct 2018 16:38:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: DPDK X-Bugzilla-Component: core X-Bugzilla-Version: 18.11 X-Bugzilla-Keywords: X-Bugzilla-Severity: normal X-Bugzilla-Who: maxime.leroy@6wind.com X-Bugzilla-Status: CONFIRMED X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: dev@dpdk.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter target_milestone Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://bugs.dpdk.org/ Auto-Submitted: auto-generated X-Auto-Response-Suppress: All MIME-Version: 1.0 Subject: [dpdk-dev] [Bug 95] eal/memory: testpmd failed to start with --socket-mem > 16384 with 2MB huge pages and with memory hotplug X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2018 16:38:40 -0000 https://bugs.dpdk.org/show_bug.cgi?id=3D95 Bug ID: 95 Summary: eal/memory: testpmd failed to start with --socket-mem > 16384 with 2MB huge pages and with memory hotplug Product: DPDK Version: 18.11 Hardware: x86 OS: Linux Status: CONFIRMED Severity: normal Priority: Normal Component: core Assignee: dev@dpdk.org Reporter: maxime.leroy@6wind.com Target Milestone: --- With the lastest version on dpdk.org (i.e. commit id 94d31549c380ee8b9108c4621d6fad8d624a395e) First configure 2Mb hugepages: mkdir -p /mnt/huge mount -t hugetlbfs nodev /mnt/huge echo 12598 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages Start test pmd with 25Gb memory: root@ubuntu1604:~/dpdk# ./build/app/testpmd -c7 --socket-mem 25196 -- -i --nb-cores=3D2 - --total-num-mbufs=3D2048 EAL: Detected 24 lcore(s) EAL: Detected 1 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list EAL: FATAL: Cannot init memory EAL: Cannot init memory PANIC in main(): Cannot init EAL 5: [./build/app/testpmd(_start+0x29) [0x48b509]] 4: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f28f9adb830= ]] 3: [./build/app/testpmd(main+0xb0e) [0x4825ae]] 2: [./build/app/testpmd(__rte_panic+0xc3) [0x475920]] 1: [./build/app/testpmd(rte_dump_stack+0x2b) [0x5ba22b]] Aborted (core dumped) It failed. With --legacy-mem option, it's work fine: ./build/app/testpmd -c7 --socket-mem 25196 --legacy-mem -- -i --nb-cores= =3D2 - --total-num-mbufs=3D2048 EAL: Detected 24 lcore(s) EAL: Detected 1 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: VFIO support initialized .. Interactive-mode selected testpmd: create a new mbuf pool : n=3D2048, size=3D2176, socket=3D0 testpmd: preferred mempool ops selected: ring_mp_mc Done testpmd>=20 testpmd> dump_physmem Segment 0-0: IOVA:0x2400000, len:2097152, virt:0x7f3491a00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:11 Segment 0-1: IOVA:0x2600000, len:2097152, virt:0x7f3491c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12 Segment 0-2: IOVA:0x2800000, len:2097152, virt:0x7f3491e00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:13 Segment 0-3: IOVA:0x2a00000, len:2097152, virt:0x7f3492000000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:14 Segment 0-4: IOVA:0x2c00000, len:2097152, virt:0x7f3492200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:15 Segment 0-5: IOVA:0x2e00000, len:2097152, virt:0x7f3492400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:16 Segment 0-6: IOVA:0x3000000, len:2097152, virt:0x7f3492600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:17 Segment 0-7: IOVA:0x3200000, len:2097152, virt:0x7f3492800000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:18 Segment 0-8: IOVA:0x3400000, len:2097152, virt:0x7f3492a00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:19 Segment 0-9: IOVA:0x3600000, len:2097152, virt:0x7f3492c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:20 Segment 0-10: IOVA:0x3800000, len:2097152, virt:0x7f3492e00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:21 Segment 0-11: IOVA:0x3a00000, len:2097152, virt:0x7f3493000000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:22 Segment 0-12: IOVA:0x3c00000, len:2097152, virt:0x7f3493200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:23 Segment 0-13: IOVA:0x3e00000, len:2097152, virt:0x7f3493400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:24 Segment 0-14: IOVA:0x4000000, len:2097152, virt:0x7f3493600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:25 ... Segment 0-8188: IOVA:0x539200000, len:2097152, virt:0x7f3891200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:7617 Segment 0-8189: IOVA:0x539400000, len:2097152, virt:0x7f3891400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:7618 Segment 0-8190: IOVA:0x539600000, len:2097152, virt:0x7f3891600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:7619 Segment 1-0: IOVA:0x514800000, len:2097152, virt:0x7f3091800000, socket_id:= 0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:7335 Segment 1-1: IOVA:0x514a00000, len:2097152, virt:0x7f3091a00000, socket_id:= 0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:7336 Segment 1-2: IOVA:0x514c00000, len:2097152, virt:0x7f3091c00000, socket_id:= 0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:7337 Segment 1-3: IOVA:0x514e00000, len:2097152, virt:0x7f3091e00000, socket_id:= 0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:7338 Segment 1-4: IOVA:0x515000000, len:2097152, virt:0x7f3092000000, socket_id:= 0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:7339 ... Segment 1-5517: IOVA:0x7f3c00000, len:2097152, virt:0x7f3343200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12602 Segment 1-5519: IOVA:0x7f4c00000, len:2097152, virt:0x7f3343600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12603 Segment 1-5520: IOVA:0x7f4e00000, len:2097152, virt:0x7f3343800000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12604 Segment 1-5522: IOVA:0x7f7000000, len:2097152, virt:0x7f3343c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12605 Segment 1-5524: IOVA:0x7f7a00000, len:2097152, virt:0x7f3344000000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12606 Segment 1-5526: IOVA:0x7fc600000, len:2097152, virt:0x7f3344400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12607 Segment 1-5528: IOVA:0x800c00000, len:2097152, virt:0x7f3344800000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12608 We have 2 segments. The first one has 8192 pages and the second one 5529 segment. If we increaze the MAX_MEMSEG_LIST: --- a/config/common_base +++ b/config/common_base @@ -65,7 +65,7 @@ CONFIG_RTE_MAX_HEAPS=3D32 CONFIG_RTE_MAX_MEMSEG_LISTS=3D64 # each memseg list will be limited to either RTE_MAX_MEMSEG_PER_LIST pages # or RTE_MAX_MEM_MB_PER_LIST megabytes worth of memory, whichever is small= er -CONFIG_RTE_MAX_MEMSEG_PER_LIST=3D8192 +CONFIG_RTE_MAX_MEMSEG_PER_LIST=3D16384 ./build/app/testpmd -c7 --socket-mem 25196 -- -i --nb-cores=3D2 - --total-num-mbufs=3D2048=20 Interactive-mode selected testpmd: create a new mbuf pool : n=3D2048, size=3D2176, socket=3D0 testpmd: preferred mempool ops selected: ring_mp_mc Done testpmd>=20 It's working fine. testpmd> dump_physmem Segment 0-0: IOVA:0x2400000, len:2097152, virt:0x7f3491a00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:11 Segment 0-1: IOVA:0x2600000, len:2097152, virt:0x7f3491c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12 Segment 0-2: IOVA:0x2800000, len:2097152, virt:0x7f3491e00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:13 Segment 0-3: IOVA:0x2a00000, len:2097152, virt:0x7f3492000000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:14 Segment 0-4: IOVA:0x2c00000, len:2097152, virt:0x7f3492200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:15 Segment 0-5: IOVA:0x2e00000, len:2097152, virt:0x7f3492400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:16 Segment 0-6: IOVA:0x3000000, len:2097152, virt:0x7f3492600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:17 Segment 0-7: IOVA:0x3200000, len:2097152, virt:0x7f3492800000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:18 Segment 0-8: IOVA:0x3400000, len:2097152, virt:0x7f3492a00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:19 Segment 0-9: IOVA:0x3600000, len:2097152, virt:0x7f3492c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:20 Segment 0-10: IOVA:0x3800000, len:2097152, virt:0x7f3492e00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:21 Segment 0-11: IOVA:0x3a00000, len:2097152, virt:0x7f3493000000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:22 Segment 0-12: IOVA:0x3c00000, len:2097152, virt:0x7f3493200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:23 Segment 0-13: IOVA:0x3e00000, len:2097152, virt:0x7f3493400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:24 Segment 0-14: IOVA:0x4000000, len:2097152, virt:0x7f3493600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:25 Segment 0-15: IOVA:0x4200000, len:2097152, virt:0x7f3493800000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:26 .. Segment 0-12588: IOVA:0x4cd000000, len:2097152, virt:0x7fb892c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12599 Segment 0-12589: IOVA:0x4d2e00000, len:2097152, virt:0x7fb892e00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12600 Segment 0-12590: IOVA:0x73ba00000, len:2097152, virt:0x7fb893000000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12601 Segment 0-12591: IOVA:0x4d4a00000, len:2097152, virt:0x7fb893200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12602 Segment 0-12592: IOVA:0xc4600000, len:2097152, virt:0x7fb893400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12603 Segment 0-12593: IOVA:0x514400000, len:2097152, virt:0x7fb893600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12604 Segment 0-12594: IOVA:0x7e3000000, len:2097152, virt:0x7fb893800000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12605 Segment 0-12595: IOVA:0x4f9800000, len:2097152, virt:0x7fb893a00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12606 Segment 0-12596: IOVA:0x294400000, len:2097152, virt:0x7fb893c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12607 Segment 0-12597: IOVA:0x4d5000000, len:2097152, virt:0x7fb893e00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:12608 But we have only one segment in this case. Question -------- 1) It's normal that we don't support anymore to allocate more that 16GB with hotplug memory and default config value ? It was working with previous dpdk version. 2) Should we increase the default value of CONFIG_RTE_MAX_MEMSEG_PER_LIST ? Note: This http://patches.dpdk.org/patch/46112/ patch doesn't fix the issue. --=20 You are receiving this mail because: You are the assignee for the bug.=