From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0119.outbound.protection.outlook.com [157.56.110.119]) by dpdk.org (Postfix) with ESMTP id 25B12567A for ; Mon, 25 Jan 2016 22:02:59 +0100 (CET) Received: from BY2PR07MB057.namprd07.prod.outlook.com (10.255.240.149) by BY2PR07MB060.namprd07.prod.outlook.com (10.255.240.152) with Microsoft SMTP Server (TLS) id 15.1.365.19; Mon, 25 Jan 2016 21:02:57 +0000 Received: from BY2PR07MB057.namprd07.prod.outlook.com ([169.254.11.152]) by BY2PR07MB057.namprd07.prod.outlook.com ([169.254.11.152]) with mapi id 15.01.0365.024; Mon, 25 Jan 2016 21:02:57 +0000 From: Alain Gautherot To: Sergio Gonzalez Monroy , "users@dpdk.org" Thread-Topic: [dpdk-users] Using DPDK for contiguous physical memory allocation Thread-Index: AdFXs8IboG1gyaOuSxWKRzvG0vqHRA== Date: Mon, 25 Jan 2016 21:02:56 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-Auto-Response-Suppress: All X-MS-TNEF-Correlator: received-spf: None (protection.outlook.com: edicogenome.com does not designate permitted sender hosts) authentication-results: spf=none (sender IP is ) smtp.mailfrom=alain@edicogenome.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [70.167.8.242] x-microsoft-exchange-diagnostics: 1; BY2PR07MB060; 5:sr4m5YVAhp/U6MGx2vtZrk46nkxrwV/Q9dczBoFUmk7CuIfolanw5typ0Y2VDRaON5/SGV3Q879jYV5yfoRdbU8slhuU4FImvbobRAmJq1DtQeYnJpcigRbxNqetInFbw5k9rOUtKe0GtzRcT1mwnQ==; 24:o4u3eLZhl4DRNx3zWY5f7xq/fgzr8hlgB0ofs7P2heCQc3BQOmUdbbyexP/3jTRwUNMXLgB5hyTAZbUzczo7Ml+C1v4wHFL+2LDLlqtS8vs= x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(421252001); SRVR:BY2PR07MB060; x-ms-office365-filtering-correlation-id: 93b5195b-ae2a-4ad6-d74a-08d325ca79cc x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(2401047)(5005006)(520078)(8121501046)(3002001)(10201501046); SRVR:BY2PR07MB060; BCL:0; PCL:0; RULEID:; SRVR:BY2PR07MB060; x-forefront-prvs: 083289FD26 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(479174004)(156002)(24454002)(164054003)(377454003)(189002)(199003)(24434003)(13464003)(54356999)(5001960100002)(74316001)(15975445007)(3846002)(76806002)(1720100001)(1096002)(1220700001)(102836003)(76576001)(2900100001)(11100500001)(107886002)(47976999)(105586002)(106356001)(50986999)(3280700002)(76786999)(5002640100001)(5003600100002)(586003)(5008740100001)(97736004)(19580395003)(33656002)(5001770100001)(6116002)(81156007)(99286002)(66066001)(189998001)(101416001)(122556002)(19580405001)(2906002)(87936001)(146001)(10400500002)(5004730100002)(40100003)(86362001)(575784001)(5890100001)(92566002)(15395725005)(2501003); DIR:OUT; SFP:1102; SCL:1; SRVR:BY2PR07MB060; H:BY2PR07MB057.namprd07.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: edicogenome.com X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Jan 2016 21:02:56.8978 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 715cf729-d8ae-46d7-a6f6-5d36941a1b6c X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR07MB060 Subject: Re: [dpdk-users] Using DPDK for contiguous physical memory allocation X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2016 21:02:59 -0000 [resend with enclosed log instead of attachment] Hello Sergio, I'm running the following command $ ./build/helloworld -c fff -n 1 And get the attached log (hope it goes through). Using "-n 2" (I'm not sure= how many channels) gives the same SIGSEGV error. Here's the configuration: $ numactl -H =20 available: 1 nodes (0) =20 node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 =20 node 0 size: 65431 MB =20 node 0 free: 62040 MB =20 node distances: =20 node 0 =20 0: 10 =20 $ cat /proc/meminfo MemTotal: 65867360 kB MemFree: 63529276 kB Buffers: 93996 kB Cached: 562160 kB SwapCached: 0 kB Active: 314816 kB Inactive: 483752 kB Active(anon): 144372 kB Inactive(anon): 28 kB Active(file): 170444 kB Inactive(file): 483724 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 12 kB Writeback: 0 kB AnonPages: 144184 kB Mapped: 49004 kB Shmem: 280 kB Slab: 77572 kB SReclaimable: 31580 kB SUnreclaim: 45992 kB KernelStack: 2904 kB PageTables: 7744 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 32421680 kB Committed_AS: 383316 kB VmallocTotal: 34359738367 kB VmallocUsed: 378992 kB VmallocChunk: 34359352736 kB HardwareCorrupted: 0 kB AnonHugePages: 73728 kB HugePa ges_Total: 500 HugePages_Free: 9 HugePages_Rsvd: 9 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 4096 kB DirectMap2M: 2027520 kB DirectMap1G: 65011712 kB Log: EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 1 on socket 0 EAL: Detected lcore 2 as core 2 on socket 0 EAL: Detected lcore 3 as core 3 on socket 0 EAL: Detected lcore 4 as core 4 on socket 0 EAL: Detected lcore 5 as core 5 on socket 0 EAL: Detected lcore 6 as core 0 on socket 0 EAL: Detected lcore 7 as core 1 on socket 0 EAL: Detected lcore 8 as core 2 on socket 0 EAL: Detected lcore 9 as core 3 on socket 0 EAL: Detected lcore 10 as core 4 on socket 0 EAL: Detected lcore 11 as core 5 on socket 0 EAL: Support maximum 128 logical core(s) by configuration. EAL: Detected 12 lcore(s) EAL: Setting up memory... EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fd26c800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x35800000 bytes EAL: Virtual area found at 0x7fd236e00000 (size =3D 0x35800000) EAL: Requesting 429 pages of size 2MB from socket 0 EAL: TSC frequency is ~2400001 KHz EAL: Master lcore 0 is ready (tid=3D6cd40880;cpuset=3D[0]) PMD: ENICPMD trace: rte_enic_pmd_init EAL: lcore 6 is ready (tid=3D331f7700;cpuset=3D[6]) EAL: lcore 5 is ready (tid=3D33bf8700;cpuset=3D[5]) EAL: lcore 9 is ready (tid=3D313f4700;cpuset=3D[9]) EAL: lcore 11 is ready (tid=3D2fff2700;cpuset=3D[11]) EAL: lcore 4 is ready (tid=3D345f9700;cpuset=3D[4]) EAL: lcore 8 is ready (tid=3D31df5700;cpuset=3D[8]) EAL: lcore 1 is ready (tid=3D363fc700;cpuset=3D[1]) EAL: lcore 10 is ready (tid=3D309f3700;cpuset=3D[10]) EAL: lcore 3 is ready (tid=3D34ffa700;cpuset=3D[3]) EAL: lcore 2 is ready (tid=3D359fb700;cpuset=3D[2]) EAL: lcore 7 is ready (tid=3D327f6700;cpuset=3D[7]) EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:01:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:03:00.0 on NUMA socket 0 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:03:00.1 on NUMA socket 0 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: Not managed by a supported kernel driver, skipped Allocating 0.1GB: PASS Allocating 0.2GB: PASS Allocating 0.3GB: PASS Allocating 0.4GB: fail Allocating 0.5GB: fail Allocating 0.6GB: fail Allocating 0.7GB: fail Allocating 0.8GB: fail Allocating 0.9GB: fail Allocating 1.0GB: fail Allocating 1.1GB: fail Allocating 1.2GB: fail Allocating 1.3GB: fail Allocating 1.4GB: fail Allocating 1.5GB: fail Allocating 1.6GB: fail Allocating 1.7GB: fail Allocating 1.8GB: fail Allocating 1.9GB: fail Allocating 2.0GB: fail Allocating 2.1GB: fail Allocating 2.2GB: Thanks, Alain -----Original Message----- From: Sergio Gonzalez Monroy [mailto:sergio.gonzalez.monroy@intel.com]=20 Sent: Monday, January 25, 2016 5:50 AM To: Alain Gautherot ; users@dpdk.org Subject: Re: [dpdk-users] Using DPDK for contiguous physical memory allocat= ion On 23/01/2016 00:20, Alain Gautherot wrote: > Hello, > > I came across DPDK in a thread @ http://stackoverflow.com/questions/44019= 12/linux-contiguous-physical-memory-from-userspace (bottom reply from mrsmi= th) and wanted to see if I can use rte_malloc() to allocate large blocks of= contiguous physical memory (16GB or even 32GB at some point). > > The platform I'm working on has an FPGA that shares host memory with the = x86_64 cores via a QPI link. > The FPGA crunches data directly from host memory and uses physical addres= ses (mostly a QPI limitation, but it is also dictated by performance consid= erations and the ability to make the best possible use of multiple memory c= ontrollers). > The data shared is 16GB or up to 32GB and could be provided as multiple d= escriptors to the FPGA, but that still means that each descriptor is in the= order of several GBytes each. > I understand that allocation may fail, but is ok for now, since I'm still= in the proof-of-concept stage, trying to rule things out. > > My sample application attempts to allocate memory by chunks of 100MB like= so: > > int main(int argc, char **argv) > { > int ret; > > ret =3D rte_eal_init(argc, argv); > if (ret < 0) { > rte_panic("Cannot init EAL\n"); > } > > int i; > for (i =3D 1; i <=3D 100; ++i) { > size_t allocsize =3D i * 100*1000*1000; > > printf(" Allocating %3.1fGB: ", ((float )i)/10.0f); > fflush(stdout); > void* ptr =3D rte_malloc(NULL, allocsize, 0U); > if (ptr !=3D NULL) { > printf("PASS\n"); > rte_free(ptr); > } else { > printf("fail\n"); > } > } > > printf("Done\n"); > return 0; > } > > I get a consistent crash @ the 2.2GB mark: > (gdb) r -c f -n 4 > ... > EAL: PCI device 0000:06:00.1 on NUMA socket 0 > EAL: probe driver: 8086:1521 rte_igb_pmd > EAL: Not managed by a supported kernel driver, skipped > Allocating 0.1GB: fail > Allocating 0.2GB: fail > ... > Allocating 2.0GB: fail > Allocating 2.1GB: fail > Allocating 2.2GB: > Program received signal SIGSEGV, Segmentation fault. > 0x00000000004c6770 in malloc_elem_init (elem=3D0x800070eaa880, heap=3D0x7= ffff7fe561c, mz=3D0x7ffff7fb2c1c, size=3D2200000064) > at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61 > 61 elem->heap =3D heap; > Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_= 6.5.x86_64 > (gdb) bt > ... > #0 0x00000000004c6770 in malloc_elem_init (elem=3D0x800070eaa880, heap= =3D0x7ffff7fe561c, mz=3D0x7ffff7fb2c1c, size=3D2200000064) > at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61 > #1 0x00000000004c694e in split_elem (elem=3D0x7ffff3e00000, split_pt=3D0= x800070eaa880) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_el= em.c:121 > #2 0x00000000004c6bda in malloc_elem_alloc (elem=3D0x7ffff3e00000, size= =3D18446744071614584320, align=3D64) > at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:223 > #3 0x00000000004c736e in malloc_heap_alloc (heap=3D0x7ffff7fe561c, type= =3D0x0, size=3D18446744071614584320, align=3D64) > at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_heap.c:167 > #4 0x00000000004c0aa1 in rte_malloc_socket (type=3D0x0, size=3D184467440= 71614584320, align=3D0, socket_arg=3D-1) > at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:89 > #5 0x00000000004c0b5b in rte_malloc (type=3D0x0, size=3D1844674407161458= 4320, align=3D0) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_mal= loc.c:115 > #6 0x000000000041ca6e in main (argc=3D5, argv=3D0x7fffffffdd48) at /home= /alaing/INTEL/dpdk-2.0.0/examples/hugephymem/main.c:66 > > > Has anybody seen such an issue? > Could I be misusing RTE somehow? > What options are you running your DPDK app with? Can you also provide the full initialization log and hugepage info? Sergio > Thanks for your time, > Alain > > > -- > Alain Gautherot > Edico Genome > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: dpdk_log.txt URL: