Re: [dpdk-users] Using DPDK for contiguous physical memory allocation

[Alain Gautherot <alain@edicogenome.com>]

Sergio,

Yes, I made that change and am now using the following:
    size_t   i;
    for (i = 1; i <= 200; ++i) {
      size_t  allocsize = (i << 30) / 10U;

      printf("  Allocating %3.1fGB: ", ((float )i)/10.0f);
      fflush(stdout);
      void*  ptr = rte_malloc(NULL, allocsize, 0U);
      if (ptr != NULL) {
        printf("PASS\n");
        rte_free(ptr);
      } else {
        printf("fail\n");
      }
    }
    printf("Done\n");


That seems to be running fine now with DPDK 2.2.0 (was not with 2.0.0).

Thanks,
Alain

-----Original Message-----
From: Sergio Gonzalez Monroy [mailto:sergio.gonzalez.monroy@intel.com] 
Sent: Monday, January 25, 2016 2:27 PM
To: Alain Gautherot <alain@edicogenome.com>
Cc: users@dpdk.org
Subject: Re: [dpdk-users] Using DPDK for contiguous physical memory allocation

Hi Alain,

On 25/01/2016 21:02, Alain Gautherot wrote:
> [resend with enclosed log instead of attachment]
>
> Hello Sergio,
>
> I'm running the following command
>
>   	$ ./build/helloworld -c fff -n 1
>
> And get the attached log (hope it goes through). Using "-n 2" (I'm not sure how many channels) gives the same SIGSEGV error.
>
> Here's the configuration:
>
> $ numactl -H
> 	available: 1 nodes (0)
> 	node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11
> 	node 0 size: 65431 MB
> 	node 0 free: 62040 MB
> 	node distances:
> 	node   0
> 		0:  10
>
> $ cat /proc/meminfo
> 	MemTotal:       65867360 kB
> 	MemFree:        63529276 kB
> 	Buffers:           93996 kB
> 	Cached:           562160 kB
> 	SwapCached:            0 kB
> 	Active:           314816 kB
> 	Inactive:         483752 kB
> 	Active(anon):     144372 kB
> 	Inactive(anon):       28 kB
> 	Active(file):     170444 kB
> 	Inactive(file):   483724 kB
> 	Unevictable:           0 kB
> 	Mlocked:               0 kB
> 	SwapTotal:             0 kB
> 	SwapFree:              0 kB
> 	Dirty:                12 kB
> 	Writeback:             0 kB
> 	AnonPages:        144184 kB
> 	Mapped:            49004 kB
> 	Shmem:               280 kB
> 	Slab:              77572 kB
> 	SReclaimable:      31580 kB
> 	SUnreclaim:        45992 kB
> 	KernelStack:        2904 kB
> 	PageTables:         7744 kB
> 	NFS_Unstable:          0 kB
> 	Bounce:                0 kB
> 	WritebackTmp:          0 kB
> 	CommitLimit:    32421680 kB
> 	Committed_AS:     383316 kB
> 	VmallocTotal:   34359738367 kB
> 	VmallocUsed:      378992 kB
> 	VmallocChunk:   34359352736 kB
> 	HardwareCorrupted:     0 kB
> 	AnonHugePages:     73728 kB
> 	HugePa	ges_Total:     500
> 	HugePages_Free:        9
> 	HugePages_Rsvd:        9
> 	HugePages_Surp:        0
> 	Hugepagesize:       2048 kB
> 	DirectMap4k:        4096 kB
> 	DirectMap2M:     2027520 kB
> 	DirectMap1G:    65011712 kB
>
>
> Log:
> 	EAL: Detected lcore 0 as core 0 on socket 0
> 	EAL: Detected lcore 1 as core 1 on socket 0
> 	EAL: Detected lcore 2 as core 2 on socket 0
> 	EAL: Detected lcore 3 as core 3 on socket 0
> 	EAL: Detected lcore 4 as core 4 on socket 0
> 	EAL: Detected lcore 5 as core 5 on socket 0
> 	EAL: Detected lcore 6 as core 0 on socket 0
> 	EAL: Detected lcore 7 as core 1 on socket 0
> 	EAL: Detected lcore 8 as core 2 on socket 0
> 	EAL: Detected lcore 9 as core 3 on socket 0
> 	EAL: Detected lcore 10 as core 4 on socket 0
> 	EAL: Detected lcore 11 as core 5 on socket 0
> 	EAL: Support maximum 128 logical core(s) by configuration.
> 	EAL: Detected 12 lcore(s)
> 	EAL: Setting up memory...
> 	EAL: Ask a virtual area of 0x200000 bytes
> 	EAL: Virtual area found at 0x7fd26c800000 (size = 0x200000)
> 	EAL: Ask a virtual area of 0x35800000 bytes
> 	EAL: Virtual area found at 0x7fd236e00000 (size = 0x35800000)
> 	EAL: Requesting 429 pages of size 2MB from socket 0
> 	EAL: TSC frequency is ~2400001 KHz
> 	EAL: Master lcore 0 is ready (tid=6cd40880;cpuset=[0])
> 	PMD: ENICPMD trace: rte_enic_pmd_init
> 	EAL: lcore 6 is ready (tid=331f7700;cpuset=[6])
> 	EAL: lcore 5 is ready (tid=33bf8700;cpuset=[5])
> 	EAL: lcore 9 is ready (tid=313f4700;cpuset=[9])
> 	EAL: lcore 11 is ready (tid=2fff2700;cpuset=[11])
> 	EAL: lcore 4 is ready (tid=345f9700;cpuset=[4])
> 	EAL: lcore 8 is ready (tid=31df5700;cpuset=[8])
> 	EAL: lcore 1 is ready (tid=363fc700;cpuset=[1])
> 	EAL: lcore 10 is ready (tid=309f3700;cpuset=[10])
> 	EAL: lcore 3 is ready (tid=34ffa700;cpuset=[3])
> 	EAL: lcore 2 is ready (tid=359fb700;cpuset=[2])
> 	EAL: lcore 7 is ready (tid=327f6700;cpuset=[7])
> 	EAL: PCI device 0000:01:00.0 on NUMA socket 0
> 	EAL:   probe driver: 8086:1521 rte_igb_pmd
> 	EAL:   Not managed by a supported kernel driver, skipped
> 	EAL: PCI device 0000:01:00.1 on NUMA socket 0
> 	EAL:   probe driver: 8086:1521 rte_igb_pmd
> 	EAL:   Not managed by a supported kernel driver, skipped
> 	EAL: PCI device 0000:03:00.0 on NUMA socket 0
> 	EAL:   probe driver: 8086:10fb rte_ixgbe_pmd
> 	EAL:   Not managed by a supported kernel driver, skipped
> 	EAL: PCI device 0000:03:00.1 on NUMA socket 0
> 	EAL:   probe driver: 8086:10fb rte_ixgbe_pmd
> 	EAL:   Not managed by a supported kernel driver, skipped
> 	  Allocating 0.1GB: PASS
> 	  Allocating 0.2GB: PASS
> 	  Allocating 0.3GB: PASS
> 	  Allocating 0.4GB: fail
> 	  Allocating 0.5GB: fail
> 	  Allocating 0.6GB: fail
> 	  Allocating 0.7GB: fail
> 	  Allocating 0.8GB: fail
> 	  Allocating 0.9GB: fail
> 	  Allocating 1.0GB: fail
> 	  Allocating 1.1GB: fail
> 	  Allocating 1.2GB: fail
> 	  Allocating 1.3GB: fail
> 	  Allocating 1.4GB: fail
> 	  Allocating 1.5GB: fail
> 	  Allocating 1.6GB: fail
> 	  Allocating 1.7GB: fail
> 	  Allocating 1.8GB: fail
> 	  Allocating 1.9GB: fail
> 	  Allocating 2.0GB: fail
> 	  Allocating 2.1GB: fail
> 	  Allocating 2.2GB:
>
> Thanks,
> Alain
>
> -----Original Message-----
> From: Sergio Gonzalez Monroy [mailto:sergio.gonzalez.monroy@intel.com]
> Sent: Monday, January 25, 2016 5:50 AM
> To: Alain Gautherot <alain@edicogenome.com>; users@dpdk.org
> Subject: Re: [dpdk-users] Using DPDK for contiguous physical memory 
> allocation
>
> On 23/01/2016 00:20, Alain Gautherot wrote:
>> Hello,
>>
>> I came across DPDK in a thread @ http://stackoverflow.com/questions/4401912/linux-contiguous-physical-memory-from-userspace (bottom reply from mrsmith) and wanted to see if I can use rte_malloc() to allocate large blocks of contiguous physical memory (16GB or even 32GB at some point).
>>
>> The platform I'm working on has an FPGA that shares host memory with the x86_64 cores via a QPI link.
>> The FPGA crunches data directly from host memory and uses physical addresses (mostly a QPI limitation, but it is also dictated by performance considerations and the ability to make the best possible use of multiple memory controllers).
>> The data shared is 16GB or up to 32GB and could be provided as multiple descriptors to the FPGA, but that still means that each descriptor is in the order of several GBytes each.
>> I understand that allocation may fail, but is ok for now, since I'm still in the proof-of-concept stage, trying to rule things out.
>>
>> My sample application attempts to allocate memory by chunks of 100MB like so:
>>
>> int main(int argc, char **argv)
>> {
>>     int ret;
>>
>>     ret = rte_eal_init(argc, argv);
>>     if (ret < 0) {
>>       rte_panic("Cannot init EAL\n");
>>     }
>>
>>     int  i;

I get warning with this code. It warns of undefined behavior because of signed integer overflow.
Could you change the above 'int i' to 'size_t i' and run it again?

Sergio
>>     for (i = 1; i <= 100; ++i) {
>>       size_t  allocsize = i * 100*1000*1000;
>>
>>       printf("  Allocating %3.1fGB: ", ((float )i)/10.0f);
>>       fflush(stdout);
>>       void*  ptr = rte_malloc(NULL, allocsize, 0U);
>>       if (ptr != NULL) {
>>         printf("PASS\n");
>>         rte_free(ptr);
>>       } else {
>>         printf("fail\n");
>>       }
>>     }
>>
>>     printf("Done\n");
>>     return 0;
>> }
>>
>> I get a consistent crash @ the 2.2GB mark:
>> (gdb) r -c f -n 4
>> ...
>> EAL: PCI device 0000:06:00.1 on NUMA socket 0
>> EAL:   probe driver: 8086:1521 rte_igb_pmd
>> EAL:   Not managed by a supported kernel driver, skipped
>>     Allocating 0.1GB: fail
>>     Allocating 0.2GB: fail
>>     ...
>>     Allocating 2.0GB: fail
>>     Allocating 2.1GB: fail
>>     Allocating 2.2GB:
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
>>       at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
>> 61              elem->heap = heap;
>> Missing separate debuginfos, use: debuginfo-install 
>> glibc-2.12-1.149.el6_6.5.x86_64
>> (gdb) bt
>> ...
>> #0  0x00000000004c6770 in malloc_elem_init (elem=0x800070eaa880, heap=0x7ffff7fe561c, mz=0x7ffff7fb2c1c, size=2200000064)
>>       at 
>> /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61
>> #1  0x00000000004c694e in split_elem (elem=0x7ffff3e00000, 
>> split_pt=0x800070eaa880) at 
>> /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:121
>> #2  0x00000000004c6bda in malloc_elem_alloc (elem=0x7ffff3e00000, size=18446744071614584320, align=64)
>>       at 
>> /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:223
>> #3  0x00000000004c736e in malloc_heap_alloc (heap=0x7ffff7fe561c, type=0x0, size=18446744071614584320, align=64)
>>       at 
>> /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_heap.c:167
>> #4  0x00000000004c0aa1 in rte_malloc_socket (type=0x0, size=18446744071614584320, align=0, socket_arg=-1)
>>       at 
>> /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:89
>> #5  0x00000000004c0b5b in rte_malloc (type=0x0, 
>> size=18446744071614584320, align=0) at 
>> /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:115
>> #6  0x000000000041ca6e in main (argc=5, argv=0x7fffffffdd48) at 
>> /home/alaing/INTEL/dpdk-2.0.0/examples/hugephymem/main.c:66
>>
>>
>> Has anybody seen such an issue?
>> Could I be misusing RTE somehow?
>>
> What options are you running your DPDK app with?
>
> Can you also provide the full initialization log and hugepage info?
>
> Sergio
>> Thanks for your time,
>> Alain
>>
>>
>> --
>> Alain Gautherot
>> Edico Genome
>>
> -------------- next part -------------- An embedded and 
> charset-unspecified text was scrubbed...
> Name: dpdk_log.txt
> URL: 
> <http://dpdk.org/ml/archives/users/attachments/20160125/3fa0b05c/attac
> hment.txt>