* Re: [dpdk-users] Using DPDK for contiguous physical memory allocation
@ 2016-01-25 21:02 Alain Gautherot
2016-01-25 22:27 ` Sergio Gonzalez Monroy
2016-01-25 22:46 ` Alain Gautherot
0 siblings, 2 replies; 8+ messages in thread
From: Alain Gautherot @ 2016-01-25 21:02 UTC (permalink / raw)
To: Sergio Gonzalez Monroy, users
[resend with enclosed log instead of attachment]
Hello Sergio,
I'm running the following command
$ ./build/helloworld -c fff -n 1
And get the attached log (hope it goes through). Using "-n 2" (I'm not sure how many channels) gives the same SIGSEGV error.
Here's the configuration:
$ numactl -H
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11
node 0 size: 65431 MB
node 0 free: 62040 MB
node distances:
node 0
0: 10
$ cat /proc/meminfo
MemTotal: 65867360 kB
MemFree: 63529276 kB
Buffers: 93996 kB
Cached: 562160 kB
SwapCached: 0 kB
Active: 314816 kB
Inactive: 483752 kB
Active(anon): 144372 kB
Inactive(anon): 28 kB
Active(file): 170444 kB
Inactive(file): 483724 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 12 kB
Writeback: 0 kB
AnonPages: 144184 kB
Mapped: 49004 kB
Shmem: 280 kB
Slab: 77572 kB
SReclaimable: 31580 kB
SUnreclaim: 45992 kB
KernelStack: 2904 kB
PageTables: 7744 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 32421680 kB
Committed_AS: 383316 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 378992 kB
VmallocChunk: 34359352736 kB
HardwareCorrupted: 0 kB
AnonHugePages: 73728 kB
HugePa ges_Total: 500
HugePages_Free: 9
HugePages_Rsvd: 9
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 4096 kB
DirectMap2M: 2027520 kB
DirectMap1G: 65011712 kB
Log:
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 5 on socket 0
EAL: Detected lcore 6 as core 0 on socket 0
EAL: Detected lcore 7 as core 1 on socket 0
EAL: Detected lcore 8 as core 2 on socket 0
EAL: Detected lcore 9 as core 3 on socket 0
EAL: Detected lcore 10 as core 4 on socket 0
EAL: Detected lcore 11 as core 5 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 12 lcore(s)
EAL: Setting up memory...
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fd26c800000 (size = 0x200000)
EAL: Ask a virtual area of 0x35800000 bytes
EAL: Virtual area found at 0x7fd236e00000 (size = 0x35800000)
EAL: Requesting 429 pages of size 2MB from socket 0
EAL: TSC frequency is ~2400001 KHz
EAL: Master lcore 0 is ready (tid=6cd40880;cpuset=[0])
PMD: ENICPMD trace: rte_enic_pmd_init
EAL: lcore 6 is ready (tid=331f7700;cpuset=[6])
EAL: lcore 5 is ready (tid=33bf8700;cpuset=[5])
EAL: lcore 9 is ready (tid=313f4700;cpuset=[9])
EAL: lcore 11 is ready (tid=2fff2700;cpuset=[11])
EAL: lcore 4 is ready (tid=345f9700;cpuset=[4])
EAL: lcore 8 is ready (tid=31df5700;cpuset=[8])
EAL: lcore 1 is ready (tid=363fc700;cpuset=[1])
EAL: lcore 10 is ready (tid=309f3700;cpuset=[10])
EAL: lcore 3 is ready (tid=34ffa700;cpuset=[3])
EAL: lcore 2 is ready (tid=359fb700;cpuset=[2])
EAL: lcore 7 is ready (tid=327f6700;cpuset=[7])
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL: probe driver: 8086:10fb rte_ixgbe_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:03:00.1 on NUMA socket 0
EAL: probe driver: 8086:10fb rte_ixgbe_pmd
EAL: Not managed by a supported kernel driver, skipped
Allocating 0.1GB: PASS
Allocating 0.2GB: PASS
Allocating 0.3GB: PASS
Allocating 0.4GB: fail
Allocating 0.5GB: fail
Allocating 0.6GB: fail
Allocating 0.7GB: fail
Allocating 0.8GB: fail
Allocating 0.9GB: fail
Allocating 1.0GB: fail
Allocating 1.1GB: fail
Allocating 1.2GB: fail
Allocating 1.3GB: fail
Allocating 1.4GB: fail
Allocating 1.5GB: fail
Allocating 1.6GB: fail
Allocating 1.7GB: fail
Allocating 1.8GB: fail
Allocating 1.9GB: fail
Allocating 2.0GB: fail
Allocating 2.1GB: fail
Allocating 2.2GB:
Thanks,
Alain
-----Original Message-----
From: Sergio Gonzalez Monroy [mailto:sergio.gonzalez.monroy@intel.com]
Sent: Monday, January 25, 2016 5:50 AM
To: Alain Gautherot <alain@edicogenome.com>; users@dpdk.org
Subject: Re: [dpdk-users] Using DPDK for contiguous physical memory allocation
On 23/01/2016 00:20, Alain Gautherot wrote:
> Hello,
>
> I came across DPDK in a thread @ http://stackoverflow.com/questions/4401912/linux-contiguous-physical-memory-from-userspace (bottom reply from mrsmith) and wanted to see if I can use rte_malloc() to allocate large blocks of contiguous physical memory (16GB or even 32GB at some point).
>
> The platform I'm working on has an FPGA that shares host memory with the x86_64 cores via a QPI link.
> The FPGA crunches data directly from host memory and uses physical addresses (mostly a QPI limitation, but it is also dictated by performance considerations and the ability to make the best possible use of multiple memory controllers).
> The data shared is 16GB or up to 32GB and could be provided as multiple descriptors to the FPGA, but that still means that each descriptor is in the order of several GBytes each.
> I understand that allocation may fail, but is ok for now, since I'm still in the proof-of-concept stage, trying to rule things out.
>
> My sample application attempts to allocate memory by chunks of 100MB like so:
>
> int main(int argc, char **argv)
> {
> int ret;
>
> ret = rte_eal_init(argc, argv);
> if (ret < 0) {
> rte_panic("Cannot init EAL\n");
> }
>
> int i;
> for (i = 1; i <= 100; ++i) {
> size_t allocsize = i * 100*1000*1000;
>
> printf(" Allocating %3.1fGB: ", ((float )i)/10.0f);
> fflush(stdout);
> void* ptr = rte_malloc(NULL, allocsize, 0U);
> if (ptr != NULL) {
> printf("PASS\n");
> rte_free(ptr);
> } else {
> printf("fail\n");
> }
> }
>
> printf("Done\n");
> return 0;
> }
>
> I get a consistent crash @ the 2.2GB mark:
> (gdb) r -c f -n 4
> ...
> EAL: PCI device 0000:06:00.1 on NUMA socket 0
> EAL: probe driver: 8086:1521 rte_igb_pmd
> EAL: Not managed by a supported kernel driver, skipped
> Allocating 0.1GB: fail
> Allocating 0.2GB: fail
> ...
> Allocating 2.0GB: fail
> Allocating 2.1GB: fail
> Allocating 2.2GB:
> Program received signal SIGSEGV, Segmentation fault.
> 0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
> 61 elem->heap = heap;
> Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.5.x86_64
> (gdb) bt
> ...
> #0 0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
> #1 0x00000000004c694e in split_elem (elem=0x7ffff3e00000, split_pt=0x800070eaa880) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:121
> #2 0x00000000004c6bda in malloc_elem_alloc (elem=0x7ffff3e00000, size=18446744071614584320, align=64)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:223
> #3 0x00000000004c736e in malloc_heap_alloc (heap=0x7ffff7fe561c, type=0x0, size=18446744071614584320, align=64)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_heap.c:167
> #4 0x00000000004c0aa1 in rte_malloc_socket (type=0x0, size=18446744071614584320, align=0, socket_arg=-1)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:89
> #5 0x00000000004c0b5b in rte_malloc (type=0x0, size=18446744071614584320, align=0) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:115
> #6 0x000000000041ca6e in main (argc=5, argv=0x7fffffffdd48) at /home/alaing/INTEL/dpdk-2.0.0/examples/hugephymem/main.c:66
>
>
> Has anybody seen such an issue?
> Could I be misusing RTE somehow?
>
What options are you running your DPDK app with?
Can you also provide the full initialization log and hugepage info?
Sergio
> Thanks for your time,
> Alain
>
>
> --
> Alain Gautherot
> Edico Genome
>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: dpdk_log.txt
URL: <http://dpdk.org/ml/archives/users/attachments/20160125/3fa0b05c/attachment.txt>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-users] Using DPDK for contiguous physical memory allocation
2016-01-25 21:02 [dpdk-users] Using DPDK for contiguous physical memory allocation Alain Gautherot
@ 2016-01-25 22:27 ` Sergio Gonzalez Monroy
2016-01-25 22:51 ` Alain Gautherot
2016-01-25 22:46 ` Alain Gautherot
1 sibling, 1 reply; 8+ messages in thread
From: Sergio Gonzalez Monroy @ 2016-01-25 22:27 UTC (permalink / raw)
To: Alain Gautherot; +Cc: users
Hi Alain,
On 25/01/2016 21:02, Alain Gautherot wrote:
> [resend with enclosed log instead of attachment]
>
> Hello Sergio,
>
> I'm running the following command
>
> $ ./build/helloworld -c fff -n 1
>
> And get the attached log (hope it goes through). Using "-n 2" (I'm not sure how many channels) gives the same SIGSEGV error.
>
> Here's the configuration:
>
> $ numactl -H
> available: 1 nodes (0)
> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11
> node 0 size: 65431 MB
> node 0 free: 62040 MB
> node distances:
> node 0
> 0: 10
>
> $ cat /proc/meminfo
> MemTotal: 65867360 kB
> MemFree: 63529276 kB
> Buffers: 93996 kB
> Cached: 562160 kB
> SwapCached: 0 kB
> Active: 314816 kB
> Inactive: 483752 kB
> Active(anon): 144372 kB
> Inactive(anon): 28 kB
> Active(file): 170444 kB
> Inactive(file): 483724 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 0 kB
> SwapFree: 0 kB
> Dirty: 12 kB
> Writeback: 0 kB
> AnonPages: 144184 kB
> Mapped: 49004 kB
> Shmem: 280 kB
> Slab: 77572 kB
> SReclaimable: 31580 kB
> SUnreclaim: 45992 kB
> KernelStack: 2904 kB
> PageTables: 7744 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 32421680 kB
> Committed_AS: 383316 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 378992 kB
> VmallocChunk: 34359352736 kB
> HardwareCorrupted: 0 kB
> AnonHugePages: 73728 kB
> HugePa ges_Total: 500
> HugePages_Free: 9
> HugePages_Rsvd: 9
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
> DirectMap4k: 4096 kB
> DirectMap2M: 2027520 kB
> DirectMap1G: 65011712 kB
>
>
> Log:
> EAL: Detected lcore 0 as core 0 on socket 0
> EAL: Detected lcore 1 as core 1 on socket 0
> EAL: Detected lcore 2 as core 2 on socket 0
> EAL: Detected lcore 3 as core 3 on socket 0
> EAL: Detected lcore 4 as core 4 on socket 0
> EAL: Detected lcore 5 as core 5 on socket 0
> EAL: Detected lcore 6 as core 0 on socket 0
> EAL: Detected lcore 7 as core 1 on socket 0
> EAL: Detected lcore 8 as core 2 on socket 0
> EAL: Detected lcore 9 as core 3 on socket 0
> EAL: Detected lcore 10 as core 4 on socket 0
> EAL: Detected lcore 11 as core 5 on socket 0
> EAL: Support maximum 128 logical core(s) by configuration.
> EAL: Detected 12 lcore(s)
> EAL: Setting up memory...
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7fd26c800000 (size = 0x200000)
> EAL: Ask a virtual area of 0x35800000 bytes
> EAL: Virtual area found at 0x7fd236e00000 (size = 0x35800000)
> EAL: Requesting 429 pages of size 2MB from socket 0
> EAL: TSC frequency is ~2400001 KHz
> EAL: Master lcore 0 is ready (tid=6cd40880;cpuset=[0])
> PMD: ENICPMD trace: rte_enic_pmd_init
> EAL: lcore 6 is ready (tid=331f7700;cpuset=[6])
> EAL: lcore 5 is ready (tid=33bf8700;cpuset=[5])
> EAL: lcore 9 is ready (tid=313f4700;cpuset=[9])
> EAL: lcore 11 is ready (tid=2fff2700;cpuset=[11])
> EAL: lcore 4 is ready (tid=345f9700;cpuset=[4])
> EAL: lcore 8 is ready (tid=31df5700;cpuset=[8])
> EAL: lcore 1 is ready (tid=363fc700;cpuset=[1])
> EAL: lcore 10 is ready (tid=309f3700;cpuset=[10])
> EAL: lcore 3 is ready (tid=34ffa700;cpuset=[3])
> EAL: lcore 2 is ready (tid=359fb700;cpuset=[2])
> EAL: lcore 7 is ready (tid=327f6700;cpuset=[7])
> EAL: PCI device 0000:01:00.0 on NUMA socket 0
> EAL: probe driver: 8086:1521 rte_igb_pmd
> EAL: Not managed by a supported kernel driver, skipped
> EAL: PCI device 0000:01:00.1 on NUMA socket 0
> EAL: probe driver: 8086:1521 rte_igb_pmd
> EAL: Not managed by a supported kernel driver, skipped
> EAL: PCI device 0000:03:00.0 on NUMA socket 0
> EAL: probe driver: 8086:10fb rte_ixgbe_pmd
> EAL: Not managed by a supported kernel driver, skipped
> EAL: PCI device 0000:03:00.1 on NUMA socket 0
> EAL: probe driver: 8086:10fb rte_ixgbe_pmd
> EAL: Not managed by a supported kernel driver, skipped
> Allocating 0.1GB: PASS
> Allocating 0.2GB: PASS
> Allocating 0.3GB: PASS
> Allocating 0.4GB: fail
> Allocating 0.5GB: fail
> Allocating 0.6GB: fail
> Allocating 0.7GB: fail
> Allocating 0.8GB: fail
> Allocating 0.9GB: fail
> Allocating 1.0GB: fail
> Allocating 1.1GB: fail
> Allocating 1.2GB: fail
> Allocating 1.3GB: fail
> Allocating 1.4GB: fail
> Allocating 1.5GB: fail
> Allocating 1.6GB: fail
> Allocating 1.7GB: fail
> Allocating 1.8GB: fail
> Allocating 1.9GB: fail
> Allocating 2.0GB: fail
> Allocating 2.1GB: fail
> Allocating 2.2GB:
>
> Thanks,
> Alain
>
> -----Original Message-----
> From: Sergio Gonzalez Monroy [mailto:sergio.gonzalez.monroy@intel.com]
> Sent: Monday, January 25, 2016 5:50 AM
> To: Alain Gautherot <alain@edicogenome.com>; users@dpdk.org
> Subject: Re: [dpdk-users] Using DPDK for contiguous physical memory allocation
>
> On 23/01/2016 00:20, Alain Gautherot wrote:
>> Hello,
>>
>> I came across DPDK in a thread @ http://stackoverflow.com/questions/4401912/linux-contiguous-physical-memory-from-userspace (bottom reply from mrsmith) and wanted to see if I can use rte_malloc() to allocate large blocks of contiguous physical memory (16GB or even 32GB at some point).
>>
>> The platform I'm working on has an FPGA that shares host memory with the x86_64 cores via a QPI link.
>> The FPGA crunches data directly from host memory and uses physical addresses (mostly a QPI limitation, but it is also dictated by performance considerations and the ability to make the best possible use of multiple memory controllers).
>> The data shared is 16GB or up to 32GB and could be provided as multiple descriptors to the FPGA, but that still means that each descriptor is in the order of several GBytes each.
>> I understand that allocation may fail, but is ok for now, since I'm still in the proof-of-concept stage, trying to rule things out.
>>
>> My sample application attempts to allocate memory by chunks of 100MB like so:
>>
>> int main(int argc, char **argv)
>> {
>> int ret;
>>
>> ret = rte_eal_init(argc, argv);
>> if (ret < 0) {
>> rte_panic("Cannot init EAL\n");
>> }
>>
>> int i;
I get warning with this code. It warns of undefined behavior because of
signed integer overflow.
Could you change the above 'int i' to 'size_t i' and run it again?
Sergio
>> for (i = 1; i <= 100; ++i) {
>> size_t allocsize = i * 100*1000*1000;
>>
>> printf(" Allocating %3.1fGB: ", ((float )i)/10.0f);
>> fflush(stdout);
>> void* ptr = rte_malloc(NULL, allocsize, 0U);
>> if (ptr != NULL) {
>> printf("PASS\n");
>> rte_free(ptr);
>> } else {
>> printf("fail\n");
>> }
>> }
>>
>> printf("Done\n");
>> return 0;
>> }
>>
>> I get a consistent crash @ the 2.2GB mark:
>> (gdb) r -c f -n 4
>> ...
>> EAL: PCI device 0000:06:00.1 on NUMA socket 0
>> EAL: probe driver: 8086:1521 rte_igb_pmd
>> EAL: Not managed by a supported kernel driver, skipped
>> Allocating 0.1GB: fail
>> Allocating 0.2GB: fail
>> ...
>> Allocating 2.0GB: fail
>> Allocating 2.1GB: fail
>> Allocating 2.2GB:
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
>> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
>> 61 elem->heap = heap;
>> Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.5.x86_64
>> (gdb) bt
>> ...
>> #0 0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
>> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
>> #1 0x00000000004c694e in split_elem (elem=0x7ffff3e00000, split_pt=0x800070eaa880) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:121
>> #2 0x00000000004c6bda in malloc_elem_alloc (elem=0x7ffff3e00000, size=18446744071614584320, align=64)
>> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:223
>> #3 0x00000000004c736e in malloc_heap_alloc (heap=0x7ffff7fe561c, type=0x0, size=18446744071614584320, align=64)
>> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_heap.c:167
>> #4 0x00000000004c0aa1 in rte_malloc_socket (type=0x0, size=18446744071614584320, align=0, socket_arg=-1)
>> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:89
>> #5 0x00000000004c0b5b in rte_malloc (type=0x0, size=18446744071614584320, align=0) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:115
>> #6 0x000000000041ca6e in main (argc=5, argv=0x7fffffffdd48) at /home/alaing/INTEL/dpdk-2.0.0/examples/hugephymem/main.c:66
>>
>>
>> Has anybody seen such an issue?
>> Could I be misusing RTE somehow?
>>
> What options are you running your DPDK app with?
>
> Can you also provide the full initialization log and hugepage info?
>
> Sergio
>> Thanks for your time,
>> Alain
>>
>>
>> --
>> Alain Gautherot
>> Edico Genome
>>
> -------------- next part --------------
> An embedded and charset-unspecified text was scrubbed...
> Name: dpdk_log.txt
> URL: <http://dpdk.org/ml/archives/users/attachments/20160125/3fa0b05c/attachment.txt>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-users] Using DPDK for contiguous physical memory allocation
2016-01-25 21:02 [dpdk-users] Using DPDK for contiguous physical memory allocation Alain Gautherot
2016-01-25 22:27 ` Sergio Gonzalez Monroy
@ 2016-01-25 22:46 ` Alain Gautherot
1 sibling, 0 replies; 8+ messages in thread
From: Alain Gautherot @ 2016-01-25 22:46 UTC (permalink / raw)
To: Alain Gautherot, Sergio Gonzalez Monroy, users
Hola Sergio,
Actually, that may just have been a bug in DPDK 2.0.0. I realized there is a newer DPDK 2.2.0 so I tried it and SIGSEGV error is gone.
So issue closed!
FYI, I just had to workaround a compilation issue in VIRTIO lib (I'm on my centos6.6/kernel 2.6.32-504 currently):
== Build lib/librte_vhost
CC virtio-net.o
In file included from /home/alaing/INTEL/dpdk-2.2.0/lib/librte_vhost/virtio-net.c:35:
/usr/include/linux/virtio_net.h:171: error: expected specifier-qualifier-list before 'u16'
make[4]: *** [virtio-net.o] Error 1
make[3]: *** [librte_vhost] Error 2
make[2]: *** [lib] Error 2
make[1]: *** [all] Error 2
make: *** [all] Error 2
I bypassed VIRTIO and was able to build.
Using 10k x 2MB huge page tables, it seems I can allocate up to 18GB of these 20GB, which is very hopeful for what I intend to do.
$ ./build/helloworld
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 5 on socket 0
EAL: Detected lcore 6 as core 0 on socket 0
EAL: Detected lcore 7 as core 1 on socket 0
EAL: Detected lcore 8 as core 2 on socket 0
EAL: Detected lcore 9 as core 3 on socket 0
EAL: Detected lcore 10 as core 4 on socket 0
EAL: Detected lcore 11 as core 5 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 12 lcore(s)
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f1b9a600000 (size = 0x200000)
EAL: Ask a virtual area of 0x483400000 bytes
EAL: Virtual area found at 0x7f1717000000 (size = 0x483400000)
EAL: Ask a virtual area of 0x8800000 bytes
EAL: Virtual area found at 0x7f170e600000 (size = 0x8800000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f170e000000 (size = 0x400000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f170da00000 (size = 0x400000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f170d400000 (size = 0x400000)
EAL: Ask a virtual area of 0x3000000 bytes
EAL: Virtual area found at 0x7f170a200000 (size = 0x3000000)
EAL: Ask a virtual area of 0x1000000 bytes
EAL: Virtual area found at 0x7f1709000000 (size = 0x1000000)
EAL: Ask a virtual area of 0xc00000 bytes
EAL: Virtual area found at 0x7f1708200000 (size = 0xc00000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f1707c00000 (size = 0x400000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f1707600000 (size = 0x400000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f1707000000 (size = 0x400000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f1706a00000 (size = 0x400000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f1706400000 (size = 0x400000)
EAL: Ask a virtual area of 0x1400000 bytes
EAL: Virtual area found at 0x7f1704e00000 (size = 0x1400000)
EAL: Ask a virtual area of 0x1000000 bytes
EAL: Virtual area found at 0x7f1703c00000 (size = 0x1000000)
EAL: Ask a virtual area of 0x800000 bytes
EAL: Virtual area found at 0x7f1703200000 (size = 0x800000)
EAL: Ask a virtual area of 0x1c00000 bytes
EAL: Virtual area found at 0x7f1701400000 (size = 0x1c00000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f1700e00000 (size = 0x400000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f1700800000 (size = 0x400000)
EAL: Ask a virtual area of 0xc00000 bytes
EAL: Virtual area found at 0x7f16ffa00000 (size = 0xc00000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f16ff400000 (size = 0x400000)
EAL: Ask a virtual area of 0x4800000 bytes
EAL: Virtual area found at 0x7f16faa00000 (size = 0x4800000)
EAL: Ask a virtual area of 0xc00000 bytes
EAL: Virtual area found at 0x7f16f9c00000 (size = 0xc00000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f16f9600000 (size = 0x400000)
EAL: Ask a virtual area of 0xc00000 bytes
EAL: Virtual area found at 0x7f16f8800000 (size = 0xc00000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f16f8200000 (size = 0x400000)
EAL: Ask a virtual area of 0x1000000 bytes
EAL: Virtual area found at 0x7f16f7000000 (size = 0x1000000)
EAL: Ask a virtual area of 0x1400000 bytes
EAL: Virtual area found at 0x7f16f5a00000 (size = 0x1400000)
EAL: Ask a virtual area of 0x800000 bytes
EAL: Virtual area found at 0x7f16f5000000 (size = 0x800000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f16f4a00000 (size = 0x400000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f16f4400000 (size = 0x400000)
EAL: Ask a virtual area of 0x800000 bytes
EAL: Virtual area found at 0x7f16f3a00000 (size = 0x800000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f16f3400000 (size = 0x400000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f16f3000000 (size = 0x200000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f16f2a00000 (size = 0x400000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f16f2600000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f16f2200000 (size = 0x200000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f16f1c00000 (size = 0x400000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f16f1800000 (size = 0x200000)
EAL: Ask a virtual area of 0x35800000 bytes
EAL: Virtual area found at 0x7f16bbe00000 (size = 0x35800000)
EAL: Requesting 9933 pages of size 2MB from socket 0
EAL: TSC frequency is ~2399945 KHz
EAL: Master lcore 0 is ready (tid=9d146880;cpuset=[0])
EAL: lcore 6 is ready (tid=b82bb700;cpuset=[6])
EAL: lcore 3 is ready (tid=9b30f700;cpuset=[3])
EAL: lcore 8 is ready (tid=b6eb9700;cpuset=[8])
EAL: lcore 7 is ready (tid=b78ba700;cpuset=[7])
EAL: lcore 11 is ready (tid=b50b6700;cpuset=[11])
EAL: lcore 9 is ready (tid=b64b8700;cpuset=[9])
EAL: lcore 4 is ready (tid=b96bd700;cpuset=[4])
EAL: lcore 2 is ready (tid=9bd10700;cpuset=[2])
EAL: lcore 10 is ready (tid=b5ab7700;cpuset=[10])
EAL: lcore 1 is ready (tid=9c711700;cpuset=[1])
EAL: lcore 5 is ready (tid=b8cbc700;cpuset=[5])
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL: probe driver: 8086:10fb rte_ixgbe_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:03:00.1 on NUMA socket 0
EAL: probe driver: 8086:10fb rte_ixgbe_pmd
EAL: Not managed by a supported kernel driver, skipped
Allocating 0.1GB: PASS
Allocating 0.2GB: PASS
Allocating 0.3GB: PASS
Allocating 0.4GB: PASS
Allocating 0.5GB: PASS
Allocating 0.6GB: PASS
Allocating 0.7GB: PASS
Allocating 0.8GB: PASS
Allocating 0.9GB: PASS
Allocating 1.0GB: PASS
Allocating 1.1GB: PASS
Allocating 1.2GB: PASS
Allocating 1.3GB: PASS
Allocating 1.4GB: PASS
Allocating 1.5GB: PASS
Allocating 1.6GB: PASS
Allocating 1.7GB: PASS
Allocating 1.8GB: PASS
Allocating 1.9GB: PASS
Allocating 2.0GB: PASS
Allocating 2.1GB: PASS
Allocating 2.2GB: PASS
Allocating 2.3GB: PASS
Allocating 2.4GB: PASS
Allocating 2.5GB: PASS
Allocating 2.6GB: PASS
Allocating 2.7GB: PASS
Allocating 2.8GB: PASS
Allocating 2.9GB: PASS
Allocating 3.0GB: PASS
Allocating 3.1GB: PASS
Allocating 3.2GB: PASS
Allocating 3.3GB: PASS
Allocating 3.4GB: PASS
Allocating 3.5GB: PASS
Allocating 3.6GB: PASS
Allocating 3.7GB: PASS
Allocating 3.8GB: PASS
Allocating 3.9GB: PASS
Allocating 4.0GB: PASS
Allocating 4.1GB: PASS
Allocating 4.2GB: PASS
Allocating 4.3GB: PASS
Allocating 4.4GB: PASS
Allocating 4.5GB: PASS
Allocating 4.6GB: PASS
Allocating 4.7GB: PASS
Allocating 4.8GB: PASS
Allocating 4.9GB: PASS
Allocating 5.0GB: PASS
Allocating 5.1GB: PASS
Allocating 5.2GB: PASS
Allocating 5.3GB: PASS
Allocating 5.4GB: PASS
Allocating 5.5GB: PASS
Allocating 5.6GB: PASS
Allocating 5.7GB: PASS
Allocating 5.8GB: PASS
Allocating 5.9GB: PASS
Allocating 6.0GB: PASS
Allocating 6.1GB: PASS
Allocating 6.2GB: PASS
Allocating 6.3GB: PASS
Allocating 6.4GB: PASS
Allocating 6.5GB: PASS
Allocating 6.6GB: PASS
Allocating 6.7GB: PASS
Allocating 6.8GB: PASS
Allocating 6.9GB: PASS
Allocating 7.0GB: PASS
Allocating 7.1GB: PASS
Allocating 7.2GB: PASS
Allocating 7.3GB: PASS
Allocating 7.4GB: PASS
Allocating 7.5GB: PASS
Allocating 7.6GB: PASS
Allocating 7.7GB: PASS
Allocating 7.8GB: PASS
Allocating 7.9GB: PASS
Allocating 8.0GB: PASS
Allocating 8.1GB: PASS
Allocating 8.2GB: PASS
Allocating 8.3GB: PASS
Allocating 8.4GB: PASS
Allocating 8.5GB: PASS
Allocating 8.6GB: PASS
Allocating 8.7GB: PASS
Allocating 8.8GB: PASS
Allocating 8.9GB: PASS
Allocating 9.0GB: PASS
Allocating 9.1GB: PASS
Allocating 9.2GB: PASS
Allocating 9.3GB: PASS
Allocating 9.4GB: PASS
Allocating 9.5GB: PASS
Allocating 9.6GB: PASS
Allocating 9.7GB: PASS
Allocating 9.8GB: PASS
Allocating 9.9GB: PASS
Allocating 10.0GB: PASS
Allocating 10.1GB: PASS
Allocating 10.2GB: PASS
Allocating 10.3GB: PASS
Allocating 10.4GB: PASS
Allocating 10.5GB: PASS
Allocating 10.6GB: PASS
Allocating 10.7GB: PASS
Allocating 10.8GB: PASS
Allocating 10.9GB: PASS
Allocating 11.0GB: PASS
Allocating 11.1GB: PASS
Allocating 11.2GB: PASS
Allocating 11.3GB: PASS
Allocating 11.4GB: PASS
Allocating 11.5GB: PASS
Allocating 11.6GB: PASS
Allocating 11.7GB: PASS
Allocating 11.8GB: PASS
Allocating 11.9GB: PASS
Allocating 12.0GB: PASS
Allocating 12.1GB: PASS
Allocating 12.2GB: PASS
Allocating 12.3GB: PASS
Allocating 12.4GB: PASS
Allocating 12.5GB: PASS
Allocating 12.6GB: PASS
Allocating 12.7GB: PASS
Allocating 12.8GB: PASS
Allocating 12.9GB: PASS
Allocating 13.0GB: PASS
Allocating 13.1GB: PASS
Allocating 13.2GB: PASS
Allocating 13.3GB: PASS
Allocating 13.4GB: PASS
Allocating 13.5GB: PASS
Allocating 13.6GB: PASS
Allocating 13.7GB: PASS
Allocating 13.8GB: PASS
Allocating 13.9GB: PASS
Allocating 14.0GB: PASS
Allocating 14.1GB: PASS
Allocating 14.2GB: PASS
Allocating 14.3GB: PASS
Allocating 14.4GB: PASS
Allocating 14.5GB: PASS
Allocating 14.6GB: PASS
Allocating 14.7GB: PASS
Allocating 14.8GB: PASS
Allocating 14.9GB: PASS
Allocating 15.0GB: PASS
Allocating 15.1GB: PASS
Allocating 15.2GB: PASS
Allocating 15.3GB: PASS
Allocating 15.4GB: PASS
Allocating 15.5GB: PASS
Allocating 15.6GB: PASS
Allocating 15.7GB: PASS
Allocating 15.8GB: PASS
Allocating 15.9GB: PASS
Allocating 16.0GB: PASS
Allocating 16.1GB: PASS
Allocating 16.2GB: PASS
Allocating 16.3GB: PASS
Allocating 16.4GB: PASS
Allocating 16.5GB: PASS
Allocating 16.6GB: PASS
Allocating 16.7GB: PASS
Allocating 16.8GB: PASS
Allocating 16.9GB: PASS
Allocating 17.0GB: PASS
Allocating 17.1GB: PASS
Allocating 17.2GB: PASS
Allocating 17.3GB: PASS
Allocating 17.4GB: PASS
Allocating 17.5GB: PASS
Allocating 17.6GB: PASS
Allocating 17.7GB: PASS
Allocating 17.8GB: PASS
Allocating 17.9GB: PASS
Allocating 18.0GB: PASS
Allocating 18.1GB: fail
Allocating 18.2GB: fail
Allocating 18.3GB: fail
Allocating 18.4GB: fail
Allocating 18.5GB: fail
Allocating 18.6GB: fail
Allocating 18.7GB: fail
Allocating 18.8GB: fail
Allocating 18.9GB: fail
Allocating 19.0GB: fail
Allocating 19.1GB: fail
Allocating 19.2GB: fail
Allocating 19.3GB: fail
Allocating 19.4GB: fail
Allocating 19.5GB: fail
Allocating 19.6GB: fail
Allocating 19.7GB: fail
Allocating 19.8GB: fail
Allocating 19.9GB: fail
Allocating 20.0GB: fail
Done
Thanks,
Alain
-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Alain Gautherot
Sent: Monday, January 25, 2016 1:03 PM
To: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>; users@dpdk.org
Subject: Re: [dpdk-users] Using DPDK for contiguous physical memory allocation
[resend with enclosed log instead of attachment]
Hello Sergio,
I'm running the following command
$ ./build/helloworld -c fff -n 1
And get the attached log (hope it goes through). Using "-n 2" (I'm not sure how many channels) gives the same SIGSEGV error.
Here's the configuration:
$ numactl -H
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11
node 0 size: 65431 MB
node 0 free: 62040 MB
node distances:
node 0
0: 10
$ cat /proc/meminfo
MemTotal: 65867360 kB
MemFree: 63529276 kB
Buffers: 93996 kB
Cached: 562160 kB
SwapCached: 0 kB
Active: 314816 kB
Inactive: 483752 kB
Active(anon): 144372 kB
Inactive(anon): 28 kB
Active(file): 170444 kB
Inactive(file): 483724 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 12 kB
Writeback: 0 kB
AnonPages: 144184 kB
Mapped: 49004 kB
Shmem: 280 kB
Slab: 77572 kB
SReclaimable: 31580 kB
SUnreclaim: 45992 kB
KernelStack: 2904 kB
PageTables: 7744 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 32421680 kB
Committed_AS: 383316 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 378992 kB
VmallocChunk: 34359352736 kB
HardwareCorrupted: 0 kB
AnonHugePages: 73728 kB
HugePa ges_Total: 500
HugePages_Free: 9
HugePages_Rsvd: 9
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 4096 kB
DirectMap2M: 2027520 kB
DirectMap1G: 65011712 kB
Log:
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 5 on socket 0
EAL: Detected lcore 6 as core 0 on socket 0
EAL: Detected lcore 7 as core 1 on socket 0
EAL: Detected lcore 8 as core 2 on socket 0
EAL: Detected lcore 9 as core 3 on socket 0
EAL: Detected lcore 10 as core 4 on socket 0
EAL: Detected lcore 11 as core 5 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 12 lcore(s)
EAL: Setting up memory...
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fd26c800000 (size = 0x200000)
EAL: Ask a virtual area of 0x35800000 bytes
EAL: Virtual area found at 0x7fd236e00000 (size = 0x35800000)
EAL: Requesting 429 pages of size 2MB from socket 0
EAL: TSC frequency is ~2400001 KHz
EAL: Master lcore 0 is ready (tid=6cd40880;cpuset=[0])
PMD: ENICPMD trace: rte_enic_pmd_init
EAL: lcore 6 is ready (tid=331f7700;cpuset=[6])
EAL: lcore 5 is ready (tid=33bf8700;cpuset=[5])
EAL: lcore 9 is ready (tid=313f4700;cpuset=[9])
EAL: lcore 11 is ready (tid=2fff2700;cpuset=[11])
EAL: lcore 4 is ready (tid=345f9700;cpuset=[4])
EAL: lcore 8 is ready (tid=31df5700;cpuset=[8])
EAL: lcore 1 is ready (tid=363fc700;cpuset=[1])
EAL: lcore 10 is ready (tid=309f3700;cpuset=[10])
EAL: lcore 3 is ready (tid=34ffa700;cpuset=[3])
EAL: lcore 2 is ready (tid=359fb700;cpuset=[2])
EAL: lcore 7 is ready (tid=327f6700;cpuset=[7])
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL: probe driver: 8086:10fb rte_ixgbe_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:03:00.1 on NUMA socket 0
EAL: probe driver: 8086:10fb rte_ixgbe_pmd
EAL: Not managed by a supported kernel driver, skipped
Allocating 0.1GB: PASS
Allocating 0.2GB: PASS
Allocating 0.3GB: PASS
Allocating 0.4GB: fail
Allocating 0.5GB: fail
Allocating 0.6GB: fail
Allocating 0.7GB: fail
Allocating 0.8GB: fail
Allocating 0.9GB: fail
Allocating 1.0GB: fail
Allocating 1.1GB: fail
Allocating 1.2GB: fail
Allocating 1.3GB: fail
Allocating 1.4GB: fail
Allocating 1.5GB: fail
Allocating 1.6GB: fail
Allocating 1.7GB: fail
Allocating 1.8GB: fail
Allocating 1.9GB: fail
Allocating 2.0GB: fail
Allocating 2.1GB: fail
Allocating 2.2GB:
Thanks,
Alain
-----Original Message-----
From: Sergio Gonzalez Monroy [mailto:sergio.gonzalez.monroy@intel.com]
Sent: Monday, January 25, 2016 5:50 AM
To: Alain Gautherot <alain@edicogenome.com>; users@dpdk.org
Subject: Re: [dpdk-users] Using DPDK for contiguous physical memory allocation
On 23/01/2016 00:20, Alain Gautherot wrote:
> Hello,
>
> I came across DPDK in a thread @ http://stackoverflow.com/questions/4401912/linux-contiguous-physical-memory-from-userspace (bottom reply from mrsmith) and wanted to see if I can use rte_malloc() to allocate large blocks of contiguous physical memory (16GB or even 32GB at some point).
>
> The platform I'm working on has an FPGA that shares host memory with the x86_64 cores via a QPI link.
> The FPGA crunches data directly from host memory and uses physical addresses (mostly a QPI limitation, but it is also dictated by performance considerations and the ability to make the best possible use of multiple memory controllers).
> The data shared is 16GB or up to 32GB and could be provided as multiple descriptors to the FPGA, but that still means that each descriptor is in the order of several GBytes each.
> I understand that allocation may fail, but is ok for now, since I'm still in the proof-of-concept stage, trying to rule things out.
>
> My sample application attempts to allocate memory by chunks of 100MB like so:
>
> int main(int argc, char **argv)
> {
> int ret;
>
> ret = rte_eal_init(argc, argv);
> if (ret < 0) {
> rte_panic("Cannot init EAL\n");
> }
>
> int i;
> for (i = 1; i <= 100; ++i) {
> size_t allocsize = i * 100*1000*1000;
>
> printf(" Allocating %3.1fGB: ", ((float )i)/10.0f);
> fflush(stdout);
> void* ptr = rte_malloc(NULL, allocsize, 0U);
> if (ptr != NULL) {
> printf("PASS\n");
> rte_free(ptr);
> } else {
> printf("fail\n");
> }
> }
>
> printf("Done\n");
> return 0;
> }
>
> I get a consistent crash @ the 2.2GB mark:
> (gdb) r -c f -n 4
> ...
> EAL: PCI device 0000:06:00.1 on NUMA socket 0
> EAL: probe driver: 8086:1521 rte_igb_pmd
> EAL: Not managed by a supported kernel driver, skipped
> Allocating 0.1GB: fail
> Allocating 0.2GB: fail
> ...
> Allocating 2.0GB: fail
> Allocating 2.1GB: fail
> Allocating 2.2GB:
> Program received signal SIGSEGV, Segmentation fault.
> 0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
> 61 elem->heap = heap;
> Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.5.x86_64
> (gdb) bt
> ...
> #0 0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
> #1 0x00000000004c694e in split_elem (elem=0x7ffff3e00000, split_pt=0x800070eaa880) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:121
> #2 0x00000000004c6bda in malloc_elem_alloc (elem=0x7ffff3e00000, size=18446744071614584320, align=64)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:223
> #3 0x00000000004c736e in malloc_heap_alloc (heap=0x7ffff7fe561c, type=0x0, size=18446744071614584320, align=64)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_heap.c:167
> #4 0x00000000004c0aa1 in rte_malloc_socket (type=0x0, size=18446744071614584320, align=0, socket_arg=-1)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:89
> #5 0x00000000004c0b5b in rte_malloc (type=0x0, size=18446744071614584320, align=0) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:115
> #6 0x000000000041ca6e in main (argc=5, argv=0x7fffffffdd48) at /home/alaing/INTEL/dpdk-2.0.0/examples/hugephymem/main.c:66
>
>
> Has anybody seen such an issue?
> Could I be misusing RTE somehow?
>
What options are you running your DPDK app with?
Can you also provide the full initialization log and hugepage info?
Sergio
> Thanks for your time,
> Alain
>
>
> --
> Alain Gautherot
> Edico Genome
>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: dpdk_log.txt
URL: <http://dpdk.org/ml/archives/users/attachments/20160125/3fa0b05c/attachment.txt>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-users] Using DPDK for contiguous physical memory allocation
2016-01-25 22:27 ` Sergio Gonzalez Monroy
@ 2016-01-25 22:51 ` Alain Gautherot
0 siblings, 0 replies; 8+ messages in thread
From: Alain Gautherot @ 2016-01-25 22:51 UTC (permalink / raw)
To: Sergio Gonzalez Monroy; +Cc: users
Sergio,
Yes, I made that change and am now using the following:
size_t i;
for (i = 1; i <= 200; ++i) {
size_t allocsize = (i << 30) / 10U;
printf(" Allocating %3.1fGB: ", ((float )i)/10.0f);
fflush(stdout);
void* ptr = rte_malloc(NULL, allocsize, 0U);
if (ptr != NULL) {
printf("PASS\n");
rte_free(ptr);
} else {
printf("fail\n");
}
}
printf("Done\n");
That seems to be running fine now with DPDK 2.2.0 (was not with 2.0.0).
Thanks,
Alain
-----Original Message-----
From: Sergio Gonzalez Monroy [mailto:sergio.gonzalez.monroy@intel.com]
Sent: Monday, January 25, 2016 2:27 PM
To: Alain Gautherot <alain@edicogenome.com>
Cc: users@dpdk.org
Subject: Re: [dpdk-users] Using DPDK for contiguous physical memory allocation
Hi Alain,
On 25/01/2016 21:02, Alain Gautherot wrote:
> [resend with enclosed log instead of attachment]
>
> Hello Sergio,
>
> I'm running the following command
>
> $ ./build/helloworld -c fff -n 1
>
> And get the attached log (hope it goes through). Using "-n 2" (I'm not sure how many channels) gives the same SIGSEGV error.
>
> Here's the configuration:
>
> $ numactl -H
> available: 1 nodes (0)
> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11
> node 0 size: 65431 MB
> node 0 free: 62040 MB
> node distances:
> node 0
> 0: 10
>
> $ cat /proc/meminfo
> MemTotal: 65867360 kB
> MemFree: 63529276 kB
> Buffers: 93996 kB
> Cached: 562160 kB
> SwapCached: 0 kB
> Active: 314816 kB
> Inactive: 483752 kB
> Active(anon): 144372 kB
> Inactive(anon): 28 kB
> Active(file): 170444 kB
> Inactive(file): 483724 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 0 kB
> SwapFree: 0 kB
> Dirty: 12 kB
> Writeback: 0 kB
> AnonPages: 144184 kB
> Mapped: 49004 kB
> Shmem: 280 kB
> Slab: 77572 kB
> SReclaimable: 31580 kB
> SUnreclaim: 45992 kB
> KernelStack: 2904 kB
> PageTables: 7744 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 32421680 kB
> Committed_AS: 383316 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 378992 kB
> VmallocChunk: 34359352736 kB
> HardwareCorrupted: 0 kB
> AnonHugePages: 73728 kB
> HugePa ges_Total: 500
> HugePages_Free: 9
> HugePages_Rsvd: 9
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
> DirectMap4k: 4096 kB
> DirectMap2M: 2027520 kB
> DirectMap1G: 65011712 kB
>
>
> Log:
> EAL: Detected lcore 0 as core 0 on socket 0
> EAL: Detected lcore 1 as core 1 on socket 0
> EAL: Detected lcore 2 as core 2 on socket 0
> EAL: Detected lcore 3 as core 3 on socket 0
> EAL: Detected lcore 4 as core 4 on socket 0
> EAL: Detected lcore 5 as core 5 on socket 0
> EAL: Detected lcore 6 as core 0 on socket 0
> EAL: Detected lcore 7 as core 1 on socket 0
> EAL: Detected lcore 8 as core 2 on socket 0
> EAL: Detected lcore 9 as core 3 on socket 0
> EAL: Detected lcore 10 as core 4 on socket 0
> EAL: Detected lcore 11 as core 5 on socket 0
> EAL: Support maximum 128 logical core(s) by configuration.
> EAL: Detected 12 lcore(s)
> EAL: Setting up memory...
> EAL: Ask a virtual area of 0x200000 bytes
> EAL: Virtual area found at 0x7fd26c800000 (size = 0x200000)
> EAL: Ask a virtual area of 0x35800000 bytes
> EAL: Virtual area found at 0x7fd236e00000 (size = 0x35800000)
> EAL: Requesting 429 pages of size 2MB from socket 0
> EAL: TSC frequency is ~2400001 KHz
> EAL: Master lcore 0 is ready (tid=6cd40880;cpuset=[0])
> PMD: ENICPMD trace: rte_enic_pmd_init
> EAL: lcore 6 is ready (tid=331f7700;cpuset=[6])
> EAL: lcore 5 is ready (tid=33bf8700;cpuset=[5])
> EAL: lcore 9 is ready (tid=313f4700;cpuset=[9])
> EAL: lcore 11 is ready (tid=2fff2700;cpuset=[11])
> EAL: lcore 4 is ready (tid=345f9700;cpuset=[4])
> EAL: lcore 8 is ready (tid=31df5700;cpuset=[8])
> EAL: lcore 1 is ready (tid=363fc700;cpuset=[1])
> EAL: lcore 10 is ready (tid=309f3700;cpuset=[10])
> EAL: lcore 3 is ready (tid=34ffa700;cpuset=[3])
> EAL: lcore 2 is ready (tid=359fb700;cpuset=[2])
> EAL: lcore 7 is ready (tid=327f6700;cpuset=[7])
> EAL: PCI device 0000:01:00.0 on NUMA socket 0
> EAL: probe driver: 8086:1521 rte_igb_pmd
> EAL: Not managed by a supported kernel driver, skipped
> EAL: PCI device 0000:01:00.1 on NUMA socket 0
> EAL: probe driver: 8086:1521 rte_igb_pmd
> EAL: Not managed by a supported kernel driver, skipped
> EAL: PCI device 0000:03:00.0 on NUMA socket 0
> EAL: probe driver: 8086:10fb rte_ixgbe_pmd
> EAL: Not managed by a supported kernel driver, skipped
> EAL: PCI device 0000:03:00.1 on NUMA socket 0
> EAL: probe driver: 8086:10fb rte_ixgbe_pmd
> EAL: Not managed by a supported kernel driver, skipped
> Allocating 0.1GB: PASS
> Allocating 0.2GB: PASS
> Allocating 0.3GB: PASS
> Allocating 0.4GB: fail
> Allocating 0.5GB: fail
> Allocating 0.6GB: fail
> Allocating 0.7GB: fail
> Allocating 0.8GB: fail
> Allocating 0.9GB: fail
> Allocating 1.0GB: fail
> Allocating 1.1GB: fail
> Allocating 1.2GB: fail
> Allocating 1.3GB: fail
> Allocating 1.4GB: fail
> Allocating 1.5GB: fail
> Allocating 1.6GB: fail
> Allocating 1.7GB: fail
> Allocating 1.8GB: fail
> Allocating 1.9GB: fail
> Allocating 2.0GB: fail
> Allocating 2.1GB: fail
> Allocating 2.2GB:
>
> Thanks,
> Alain
>
> -----Original Message-----
> From: Sergio Gonzalez Monroy [mailto:sergio.gonzalez.monroy@intel.com]
> Sent: Monday, January 25, 2016 5:50 AM
> To: Alain Gautherot <alain@edicogenome.com>; users@dpdk.org
> Subject: Re: [dpdk-users] Using DPDK for contiguous physical memory
> allocation
>
> On 23/01/2016 00:20, Alain Gautherot wrote:
>> Hello,
>>
>> I came across DPDK in a thread @ http://stackoverflow.com/questions/4401912/linux-contiguous-physical-memory-from-userspace (bottom reply from mrsmith) and wanted to see if I can use rte_malloc() to allocate large blocks of contiguous physical memory (16GB or even 32GB at some point).
>>
>> The platform I'm working on has an FPGA that shares host memory with the x86_64 cores via a QPI link.
>> The FPGA crunches data directly from host memory and uses physical addresses (mostly a QPI limitation, but it is also dictated by performance considerations and the ability to make the best possible use of multiple memory controllers).
>> The data shared is 16GB or up to 32GB and could be provided as multiple descriptors to the FPGA, but that still means that each descriptor is in the order of several GBytes each.
>> I understand that allocation may fail, but is ok for now, since I'm still in the proof-of-concept stage, trying to rule things out.
>>
>> My sample application attempts to allocate memory by chunks of 100MB like so:
>>
>> int main(int argc, char **argv)
>> {
>> int ret;
>>
>> ret = rte_eal_init(argc, argv);
>> if (ret < 0) {
>> rte_panic("Cannot init EAL\n");
>> }
>>
>> int i;
I get warning with this code. It warns of undefined behavior because of signed integer overflow.
Could you change the above 'int i' to 'size_t i' and run it again?
Sergio
>> for (i = 1; i <= 100; ++i) {
>> size_t allocsize = i * 100*1000*1000;
>>
>> printf(" Allocating %3.1fGB: ", ((float )i)/10.0f);
>> fflush(stdout);
>> void* ptr = rte_malloc(NULL, allocsize, 0U);
>> if (ptr != NULL) {
>> printf("PASS\n");
>> rte_free(ptr);
>> } else {
>> printf("fail\n");
>> }
>> }
>>
>> printf("Done\n");
>> return 0;
>> }
>>
>> I get a consistent crash @ the 2.2GB mark:
>> (gdb) r -c f -n 4
>> ...
>> EAL: PCI device 0000:06:00.1 on NUMA socket 0
>> EAL: probe driver: 8086:1521 rte_igb_pmd
>> EAL: Not managed by a supported kernel driver, skipped
>> Allocating 0.1GB: fail
>> Allocating 0.2GB: fail
>> ...
>> Allocating 2.0GB: fail
>> Allocating 2.1GB: fail
>> Allocating 2.2GB:
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
>> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
>> 61 elem->heap = heap;
>> Missing separate debuginfos, use: debuginfo-install
>> glibc-2.12-1.149.el6_6.5.x86_64
>> (gdb) bt
>> ...
>> #0 0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
>> at
>> /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
>> #1 0x00000000004c694e in split_elem (elem=0x7ffff3e00000,
>> split_pt=0x800070eaa880) at
>> /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:121
>> #2 0x00000000004c6bda in malloc_elem_alloc (elem=0x7ffff3e00000, size=18446744071614584320, align=64)
>> at
>> /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:223
>> #3 0x00000000004c736e in malloc_heap_alloc (heap=0x7ffff7fe561c, type=0x0, size=18446744071614584320, align=64)
>> at
>> /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_heap.c:167
>> #4 0x00000000004c0aa1 in rte_malloc_socket (type=0x0, size=18446744071614584320, align=0, socket_arg=-1)
>> at
>> /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:89
>> #5 0x00000000004c0b5b in rte_malloc (type=0x0,
>> size=18446744071614584320, align=0) at
>> /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:115
>> #6 0x000000000041ca6e in main (argc=5, argv=0x7fffffffdd48) at
>> /home/alaing/INTEL/dpdk-2.0.0/examples/hugephymem/main.c:66
>>
>>
>> Has anybody seen such an issue?
>> Could I be misusing RTE somehow?
>>
> What options are you running your DPDK app with?
>
> Can you also provide the full initialization log and hugepage info?
>
> Sergio
>> Thanks for your time,
>> Alain
>>
>>
>> --
>> Alain Gautherot
>> Edico Genome
>>
> -------------- next part -------------- An embedded and
> charset-unspecified text was scrubbed...
> Name: dpdk_log.txt
> URL:
> <http://dpdk.org/ml/archives/users/attachments/20160125/3fa0b05c/attac
> hment.txt>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-users] Using DPDK for contiguous physical memory allocation
2016-01-25 13:50 ` Sergio Gonzalez Monroy
2016-01-25 14:51 ` Freynet, Marc (Nokia - FR)
@ 2016-01-25 20:59 ` Alain Gautherot
1 sibling, 0 replies; 8+ messages in thread
From: Alain Gautherot @ 2016-01-25 20:59 UTC (permalink / raw)
To: Sergio Gonzalez Monroy, users
Hello Sergio,
I'm running the following command
$ ./build/helloworld -c fff -n 1
And get the attached log (hope it goes through). Using "-n 2" (I'm not sure how many channels) gives the same SIGSEGV error.
Here's the configuration:
$ numactl -H
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11
node 0 size: 65431 MB
node 0 free: 62040 MB
node distances:
node 0
0: 10
$ cat /proc/meminfo
MemTotal: 65867360 kB
MemFree: 63529276 kB
Buffers: 93996 kB
Cached: 562160 kB
SwapCached: 0 kB
Active: 314816 kB
Inactive: 483752 kB
Active(anon): 144372 kB
Inactive(anon): 28 kB
Active(file): 170444 kB
Inactive(file): 483724 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 12 kB
Writeback: 0 kB
AnonPages: 144184 kB
Mapped: 49004 kB
Shmem: 280 kB
Slab: 77572 kB
SReclaimable: 31580 kB
SUnreclaim: 45992 kB
KernelStack: 2904 kB
PageTables: 7744 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 32421680 kB
Committed_AS: 383316 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 378992 kB
VmallocChunk: 34359352736 kB
HardwareCorrupted: 0 kB
AnonHugePages: 73728 kB
HugePa ges_Total: 500
HugePages_Free: 9
HugePages_Rsvd: 9
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 4096 kB
DirectMap2M: 2027520 kB
DirectMap1G: 65011712 kB
Thanks,
Alain
-----Original Message-----
From: Sergio Gonzalez Monroy [mailto:sergio.gonzalez.monroy@intel.com]
Sent: Monday, January 25, 2016 5:50 AM
To: Alain Gautherot <alain@edicogenome.com>; users@dpdk.org
Subject: Re: [dpdk-users] Using DPDK for contiguous physical memory allocation
On 23/01/2016 00:20, Alain Gautherot wrote:
> Hello,
>
> I came across DPDK in a thread @ http://stackoverflow.com/questions/4401912/linux-contiguous-physical-memory-from-userspace (bottom reply from mrsmith) and wanted to see if I can use rte_malloc() to allocate large blocks of contiguous physical memory (16GB or even 32GB at some point).
>
> The platform I'm working on has an FPGA that shares host memory with the x86_64 cores via a QPI link.
> The FPGA crunches data directly from host memory and uses physical addresses (mostly a QPI limitation, but it is also dictated by performance considerations and the ability to make the best possible use of multiple memory controllers).
> The data shared is 16GB or up to 32GB and could be provided as multiple descriptors to the FPGA, but that still means that each descriptor is in the order of several GBytes each.
> I understand that allocation may fail, but is ok for now, since I'm still in the proof-of-concept stage, trying to rule things out.
>
> My sample application attempts to allocate memory by chunks of 100MB like so:
>
> int main(int argc, char **argv)
> {
> int ret;
>
> ret = rte_eal_init(argc, argv);
> if (ret < 0) {
> rte_panic("Cannot init EAL\n");
> }
>
> int i;
> for (i = 1; i <= 100; ++i) {
> size_t allocsize = i * 100*1000*1000;
>
> printf(" Allocating %3.1fGB: ", ((float )i)/10.0f);
> fflush(stdout);
> void* ptr = rte_malloc(NULL, allocsize, 0U);
> if (ptr != NULL) {
> printf("PASS\n");
> rte_free(ptr);
> } else {
> printf("fail\n");
> }
> }
>
> printf("Done\n");
> return 0;
> }
>
> I get a consistent crash @ the 2.2GB mark:
> (gdb) r -c f -n 4
> ...
> EAL: PCI device 0000:06:00.1 on NUMA socket 0
> EAL: probe driver: 8086:1521 rte_igb_pmd
> EAL: Not managed by a supported kernel driver, skipped
> Allocating 0.1GB: fail
> Allocating 0.2GB: fail
> ...
> Allocating 2.0GB: fail
> Allocating 2.1GB: fail
> Allocating 2.2GB:
> Program received signal SIGSEGV, Segmentation fault.
> 0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
> 61 elem->heap = heap;
> Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.5.x86_64
> (gdb) bt
> ...
> #0 0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
> #1 0x00000000004c694e in split_elem (elem=0x7ffff3e00000, split_pt=0x800070eaa880) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:121
> #2 0x00000000004c6bda in malloc_elem_alloc (elem=0x7ffff3e00000, size=18446744071614584320, align=64)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:223
> #3 0x00000000004c736e in malloc_heap_alloc (heap=0x7ffff7fe561c, type=0x0, size=18446744071614584320, align=64)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_heap.c:167
> #4 0x00000000004c0aa1 in rte_malloc_socket (type=0x0, size=18446744071614584320, align=0, socket_arg=-1)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:89
> #5 0x00000000004c0b5b in rte_malloc (type=0x0, size=18446744071614584320, align=0) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:115
> #6 0x000000000041ca6e in main (argc=5, argv=0x7fffffffdd48) at /home/alaing/INTEL/dpdk-2.0.0/examples/hugephymem/main.c:66
>
>
> Has anybody seen such an issue?
> Could I be misusing RTE somehow?
>
What options are you running your DPDK app with?
Can you also provide the full initialization log and hugepage info?
Sergio
> Thanks for your time,
> Alain
>
>
> --
> Alain Gautherot
> Edico Genome
>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: dpdk_log.txt
URL: <http://dpdk.org/ml/archives/users/attachments/20160125/3fa0b05c/attachment.txt>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-users] Using DPDK for contiguous physical memory allocation
2016-01-25 13:50 ` Sergio Gonzalez Monroy
@ 2016-01-25 14:51 ` Freynet, Marc (Nokia - FR)
2016-01-25 20:59 ` Alain Gautherot
1 sibling, 0 replies; 8+ messages in thread
From: Freynet, Marc (Nokia - FR) @ 2016-01-25 14:51 UTC (permalink / raw)
To: EXT Sergio Gonzalez Monroy, Alain Gautherot, users
I can see that your system has a NUMA architecture.
My experience on DPDK and NUMA is 3 years old and I remembered that there was some issues.
I never manage to allocate huge pages in a NUMA architecture.
I think that something need to be done at kernel boot time to avoid the fragmentation of the physical memory in the virtual address space but I never spend enough time to find out what should be done.
And also, in a NUMA architecture, you can only access the physical memory that is dedicated to the CPU where your DPDK "core" are running.
"Bowl of rice will raise a benefactor, a bucket of rice will raise a enemy.", Chinese proverb.
FREYNET Marc
Alcatel-Lucent France
Centre de Villarceaux
Route de Villejust
91620 NOZAY France
Tel: +33 (0)1 6040 1960
Intranet: 2103 1960
marc.freynet@nokia.com
-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of EXT Sergio Gonzalez Monroy
Sent: lundi 25 janvier 2016 14:50
To: Alain Gautherot; users@dpdk.org
Subject: Re: [dpdk-users] Using DPDK for contiguous physical memory allocation
On 23/01/2016 00:20, Alain Gautherot wrote:
> Hello,
>
> I came across DPDK in a thread @ http://stackoverflow.com/questions/4401912/linux-contiguous-physical-memory-from-userspace (bottom reply from mrsmith) and wanted to see if I can use rte_malloc() to allocate large blocks of contiguous physical memory (16GB or even 32GB at some point).
>
> The platform I'm working on has an FPGA that shares host memory with the x86_64 cores via a QPI link.
> The FPGA crunches data directly from host memory and uses physical addresses (mostly a QPI limitation, but it is also dictated by performance considerations and the ability to make the best possible use of multiple memory controllers).
> The data shared is 16GB or up to 32GB and could be provided as multiple descriptors to the FPGA, but that still means that each descriptor is in the order of several GBytes each.
> I understand that allocation may fail, but is ok for now, since I'm still in the proof-of-concept stage, trying to rule things out.
>
> My sample application attempts to allocate memory by chunks of 100MB like so:
>
> int main(int argc, char **argv)
> {
> int ret;
>
> ret = rte_eal_init(argc, argv);
> if (ret < 0) {
> rte_panic("Cannot init EAL\n");
> }
>
> int i;
> for (i = 1; i <= 100; ++i) {
> size_t allocsize = i * 100*1000*1000;
>
> printf(" Allocating %3.1fGB: ", ((float )i)/10.0f);
> fflush(stdout);
> void* ptr = rte_malloc(NULL, allocsize, 0U);
> if (ptr != NULL) {
> printf("PASS\n");
> rte_free(ptr);
> } else {
> printf("fail\n");
> }
> }
>
> printf("Done\n");
> return 0;
> }
>
> I get a consistent crash @ the 2.2GB mark:
> (gdb) r -c f -n 4
> ...
> EAL: PCI device 0000:06:00.1 on NUMA socket 0
> EAL: probe driver: 8086:1521 rte_igb_pmd
> EAL: Not managed by a supported kernel driver, skipped
> Allocating 0.1GB: fail
> Allocating 0.2GB: fail
> ...
> Allocating 2.0GB: fail
> Allocating 2.1GB: fail
> Allocating 2.2GB:
> Program received signal SIGSEGV, Segmentation fault.
> 0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
> 61 elem->heap = heap;
> Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.5.x86_64
> (gdb) bt
> ...
> #0 0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
> #1 0x00000000004c694e in split_elem (elem=0x7ffff3e00000, split_pt=0x800070eaa880) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:121
> #2 0x00000000004c6bda in malloc_elem_alloc (elem=0x7ffff3e00000, size=18446744071614584320, align=64)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:223
> #3 0x00000000004c736e in malloc_heap_alloc (heap=0x7ffff7fe561c, type=0x0, size=18446744071614584320, align=64)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_heap.c:167
> #4 0x00000000004c0aa1 in rte_malloc_socket (type=0x0, size=18446744071614584320, align=0, socket_arg=-1)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:89
> #5 0x00000000004c0b5b in rte_malloc (type=0x0, size=18446744071614584320, align=0) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:115
> #6 0x000000000041ca6e in main (argc=5, argv=0x7fffffffdd48) at /home/alaing/INTEL/dpdk-2.0.0/examples/hugephymem/main.c:66
>
>
> Has anybody seen such an issue?
> Could I be misusing RTE somehow?
>
What options are you running your DPDK app with?
Can you also provide the full initialization log and hugepage info?
Sergio
> Thanks for your time,
> Alain
>
>
> --
> Alain Gautherot
> Edico Genome
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-users] Using DPDK for contiguous physical memory allocation
2016-01-23 0:20 Alain Gautherot
@ 2016-01-25 13:50 ` Sergio Gonzalez Monroy
2016-01-25 14:51 ` Freynet, Marc (Nokia - FR)
2016-01-25 20:59 ` Alain Gautherot
0 siblings, 2 replies; 8+ messages in thread
From: Sergio Gonzalez Monroy @ 2016-01-25 13:50 UTC (permalink / raw)
To: Alain Gautherot, users
On 23/01/2016 00:20, Alain Gautherot wrote:
> Hello,
>
> I came across DPDK in a thread @ http://stackoverflow.com/questions/4401912/linux-contiguous-physical-memory-from-userspace (bottom reply from mrsmith) and wanted to see if I can use rte_malloc() to allocate large blocks of contiguous physical memory (16GB or even 32GB at some point).
>
> The platform I'm working on has an FPGA that shares host memory with the x86_64 cores via a QPI link.
> The FPGA crunches data directly from host memory and uses physical addresses (mostly a QPI limitation, but it is also dictated by performance considerations and the ability to make the best possible use of multiple memory controllers).
> The data shared is 16GB or up to 32GB and could be provided as multiple descriptors to the FPGA, but that still means that each descriptor is in the order of several GBytes each.
> I understand that allocation may fail, but is ok for now, since I'm still in the proof-of-concept stage, trying to rule things out.
>
> My sample application attempts to allocate memory by chunks of 100MB like so:
>
> int main(int argc, char **argv)
> {
> int ret;
>
> ret = rte_eal_init(argc, argv);
> if (ret < 0) {
> rte_panic("Cannot init EAL\n");
> }
>
> int i;
> for (i = 1; i <= 100; ++i) {
> size_t allocsize = i * 100*1000*1000;
>
> printf(" Allocating %3.1fGB: ", ((float )i)/10.0f);
> fflush(stdout);
> void* ptr = rte_malloc(NULL, allocsize, 0U);
> if (ptr != NULL) {
> printf("PASS\n");
> rte_free(ptr);
> } else {
> printf("fail\n");
> }
> }
>
> printf("Done\n");
> return 0;
> }
>
> I get a consistent crash @ the 2.2GB mark:
> (gdb) r -c f -n 4
> ...
> EAL: PCI device 0000:06:00.1 on NUMA socket 0
> EAL: probe driver: 8086:1521 rte_igb_pmd
> EAL: Not managed by a supported kernel driver, skipped
> Allocating 0.1GB: fail
> Allocating 0.2GB: fail
> ...
> Allocating 2.0GB: fail
> Allocating 2.1GB: fail
> Allocating 2.2GB:
> Program received signal SIGSEGV, Segmentation fault.
> 0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
> 61 elem->heap = heap;
> Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.5.x86_64
> (gdb) bt
> ...
> #0 0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
> #1 0x00000000004c694e in split_elem (elem=0x7ffff3e00000, split_pt=0x800070eaa880) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:121
> #2 0x00000000004c6bda in malloc_elem_alloc (elem=0x7ffff3e00000, size=18446744071614584320, align=64)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:223
> #3 0x00000000004c736e in malloc_heap_alloc (heap=0x7ffff7fe561c, type=0x0, size=18446744071614584320, align=64)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_heap.c:167
> #4 0x00000000004c0aa1 in rte_malloc_socket (type=0x0, size=18446744071614584320, align=0, socket_arg=-1)
> at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:89
> #5 0x00000000004c0b5b in rte_malloc (type=0x0, size=18446744071614584320, align=0) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:115
> #6 0x000000000041ca6e in main (argc=5, argv=0x7fffffffdd48) at /home/alaing/INTEL/dpdk-2.0.0/examples/hugephymem/main.c:66
>
>
> Has anybody seen such an issue?
> Could I be misusing RTE somehow?
>
What options are you running your DPDK app with?
Can you also provide the full initialization log and hugepage info?
Sergio
> Thanks for your time,
> Alain
>
>
> --
> Alain Gautherot
> Edico Genome
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* [dpdk-users] Using DPDK for contiguous physical memory allocation
@ 2016-01-23 0:20 Alain Gautherot
2016-01-25 13:50 ` Sergio Gonzalez Monroy
0 siblings, 1 reply; 8+ messages in thread
From: Alain Gautherot @ 2016-01-23 0:20 UTC (permalink / raw)
To: users
Hello,
I came across DPDK in a thread @ http://stackoverflow.com/questions/4401912/linux-contiguous-physical-memory-from-userspace (bottom reply from mrsmith) and wanted to see if I can use rte_malloc() to allocate large blocks of contiguous physical memory (16GB or even 32GB at some point).
The platform I'm working on has an FPGA that shares host memory with the x86_64 cores via a QPI link.
The FPGA crunches data directly from host memory and uses physical addresses (mostly a QPI limitation, but it is also dictated by performance considerations and the ability to make the best possible use of multiple memory controllers).
The data shared is 16GB or up to 32GB and could be provided as multiple descriptors to the FPGA, but that still means that each descriptor is in the order of several GBytes each.
I understand that allocation may fail, but is ok for now, since I'm still in the proof-of-concept stage, trying to rule things out.
My sample application attempts to allocate memory by chunks of 100MB like so:
int main(int argc, char **argv)
{
int ret;
ret = rte_eal_init(argc, argv);
if (ret < 0) {
rte_panic("Cannot init EAL\n");
}
int i;
for (i = 1; i <= 100; ++i) {
size_t allocsize = i * 100*1000*1000;
printf(" Allocating %3.1fGB: ", ((float )i)/10.0f);
fflush(stdout);
void* ptr = rte_malloc(NULL, allocsize, 0U);
if (ptr != NULL) {
printf("PASS\n");
rte_free(ptr);
} else {
printf("fail\n");
}
}
printf("Done\n");
return 0;
}
I get a consistent crash @ the 2.2GB mark:
(gdb) r -c f -n 4
...
EAL: PCI device 0000:06:00.1 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
Allocating 0.1GB: fail
Allocating 0.2GB: fail
...
Allocating 2.0GB: fail
Allocating 2.1GB: fail
Allocating 2.2GB:
Program received signal SIGSEGV, Segmentation fault.
0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
61 elem->heap = heap;
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.5.x86_64
(gdb) bt
...
#0 0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
#1 0x00000000004c694e in split_elem (elem=0x7ffff3e00000, split_pt=0x800070eaa880) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:121
#2 0x00000000004c6bda in malloc_elem_alloc (elem=0x7ffff3e00000, size=18446744071614584320, align=64)
at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:223
#3 0x00000000004c736e in malloc_heap_alloc (heap=0x7ffff7fe561c, type=0x0, size=18446744071614584320, align=64)
at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_heap.c:167
#4 0x00000000004c0aa1 in rte_malloc_socket (type=0x0, size=18446744071614584320, align=0, socket_arg=-1)
at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:89
#5 0x00000000004c0b5b in rte_malloc (type=0x0, size=18446744071614584320, align=0) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:115
#6 0x000000000041ca6e in main (argc=5, argv=0x7fffffffdd48) at /home/alaing/INTEL/dpdk-2.0.0/examples/hugephymem/main.c:66
Has anybody seen such an issue?
Could I be misusing RTE somehow?
Thanks for your time,
Alain
--
Alain Gautherot
Edico Genome
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2016-01-25 22:51 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-01-25 21:02 [dpdk-users] Using DPDK for contiguous physical memory allocation Alain Gautherot
2016-01-25 22:27 ` Sergio Gonzalez Monroy
2016-01-25 22:51 ` Alain Gautherot
2016-01-25 22:46 ` Alain Gautherot
-- strict thread matches above, loose matches on Subject: below --
2016-01-23 0:20 Alain Gautherot
2016-01-25 13:50 ` Sergio Gonzalez Monroy
2016-01-25 14:51 ` Freynet, Marc (Nokia - FR)
2016-01-25 20:59 ` Alain Gautherot
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).