From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0115.outbound.protection.outlook.com [157.56.110.115]) by dpdk.org (Postfix) with ESMTP id 5170B56A9 for ; Sat, 23 Jan 2016 01:20:52 +0100 (CET) Received: from BY2PR07MB057.namprd07.prod.outlook.com (10.255.240.149) by BY2PR07MB059.namprd07.prod.outlook.com (10.255.240.151) with Microsoft SMTP Server (TLS) id 15.1.365.19; Sat, 23 Jan 2016 00:20:50 +0000 Received: from BY2PR07MB057.namprd07.prod.outlook.com ([169.254.11.152]) by BY2PR07MB057.namprd07.prod.outlook.com ([169.254.11.152]) with mapi id 15.01.0365.024; Sat, 23 Jan 2016 00:20:50 +0000 From: Alain Gautherot To: "users@dpdk.org" Thread-Topic: Using DPDK for contiguous physical memory allocation Thread-Index: AdFVcytLoG1gyaOuSxWKRzvG0vqHRA== Date: Sat, 23 Jan 2016 00:20:49 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=alain@edicogenome.com; x-originating-ip: [70.167.8.242] x-microsoft-exchange-diagnostics: 1; BY2PR07MB059; 5:XLs30lXMrhXq4EX+g46rKhYklbasByn6JtlrTYyo0c5nSdesAvMu0fEIfJQ2pRSa2elHFaiLdqGV1jXNOxFvsBXmAbCFbmpKWf4qSRU+tnUeQTfXjVRg2oIBHS3kuaz8DoCHKUNqoZtHD5/NY8YFLQ==; 24:IlPyvfBoWpM0K2Ik7whdlrs1Q+Sj7mRlVEieKjXAzIql9BKEWYBPknqiBrzBAaDjuHfA8kixdlkfN4Ji9Lf/wGhFMel/eqptqm5FepwH4Y0= x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BY2PR07MB059; x-ms-office365-filtering-correlation-id: c7c8d61d-f47c-4927-0923-08d3238b0c20 x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(2401047)(8121501046)(5005006)(520078)(10201501046)(3002001); SRVR:BY2PR07MB059; BCL:0; PCL:0; RULEID:; SRVR:BY2PR07MB059; x-forefront-prvs: 0830866D19 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(189002)(199003)(5002640100001)(11100500001)(87936001)(1730700002)(1096002)(586003)(5008740100001)(101416001)(81156007)(6116002)(3846002)(1220700001)(102836003)(19580395003)(40100003)(5004730100002)(66066001)(10400500002)(2501003)(92566002)(99286002)(97736004)(86362001)(122556002)(76576001)(1720100001)(450100001)(229853001)(5001960100002)(2351001)(33656002)(107886002)(105586002)(74316001)(54356999)(2906002)(15395725005)(2900100001)(50986999)(189998001)(575784001)(15975445007)(5003600100002)(106356001)(110136002)(568214009); DIR:OUT; SFP:1102; SCL:1; SRVR:BY2PR07MB059; H:BY2PR07MB057.namprd07.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: edicogenome.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: edicogenome.com X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Jan 2016 00:20:49.7728 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 715cf729-d8ae-46d7-a6f6-5d36941a1b6c X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR07MB059 Subject: [dpdk-users] Using DPDK for contiguous physical memory allocation X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 00:20:52 -0000 Hello, I came across DPDK in a thread @ http://stackoverflow.com/questions/4401912= /linux-contiguous-physical-memory-from-userspace (bottom reply from mrsmith= ) and wanted to see if I can use rte_malloc() to allocate large blocks of c= ontiguous physical memory (16GB or even 32GB at some point). The platform I'm working on has an FPGA that shares host memory with the x8= 6_64 cores via a QPI link. The FPGA crunches data directly from host memory and uses physical addresse= s (mostly a QPI limitation, but it is also dictated by performance consider= ations and the ability to make the best possible use of multiple memory con= trollers). The data shared is 16GB or up to 32GB and could be provided as multiple des= criptors to the FPGA, but that still means that each descriptor is in the o= rder of several GBytes each. I understand that allocation may fail, but is ok for now, since I'm still i= n the proof-of-concept stage, trying to rule things out. My sample application attempts to allocate memory by chunks of 100MB like s= o: int main(int argc, char **argv) { int ret; ret =3D rte_eal_init(argc, argv); if (ret < 0) { rte_panic("Cannot init EAL\n"); } int i; for (i =3D 1; i <=3D 100; ++i) { size_t allocsize =3D i * 100*1000*1000; printf(" Allocating %3.1fGB: ", ((float )i)/10.0f); fflush(stdout); void* ptr =3D rte_malloc(NULL, allocsize, 0U); if (ptr !=3D NULL) { printf("PASS\n"); rte_free(ptr); } else { printf("fail\n"); } } printf("Done\n"); return 0; } I get a consistent crash @ the 2.2GB mark: (gdb) r -c f -n 4 ... EAL: PCI device 0000:06:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 rte_igb_pmd EAL: Not managed by a supported kernel driver, skipped Allocating 0.1GB: fail Allocating 0.2GB: fail ... Allocating 2.0GB: fail Allocating 2.1GB: fail Allocating 2.2GB: Program received signal SIGSEGV, Segmentation fault. 0x00000000004c6770 in malloc_elem_init (elem=3D0x800070eaa880, heap=3D0x7ff= ff7fe561c, mz=3D0x7ffff7fb2c1c, size=3D2200000064) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61 61 elem->heap =3D heap; Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.= 5.x86_64 (gdb) bt ... #0 0x00000000004c6770 in malloc_elem_init (elem=3D0x800070eaa880, heap=3D0= x7ffff7fe561c, mz=3D0x7ffff7fb2c1c, size=3D2200000064) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:61 #1 0x00000000004c694e in split_elem (elem=3D0x7ffff3e00000, split_pt=3D0x8= 00070eaa880) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem= .c:121 #2 0x00000000004c6bda in malloc_elem_alloc (elem=3D0x7ffff3e00000, size=3D= 18446744071614584320, align=3D64) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_elem.c:223 #3 0x00000000004c736e in malloc_heap_alloc (heap=3D0x7ffff7fe561c, type=3D= 0x0, size=3D18446744071614584320, align=3D64) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/malloc_heap.c:167 #4 0x00000000004c0aa1 in rte_malloc_socket (type=3D0x0, size=3D18446744071= 614584320, align=3D0, socket_arg=3D-1) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_malloc.c:89 #5 0x00000000004c0b5b in rte_malloc (type=3D0x0, size=3D184467440716145843= 20, align=3D0) at /home/alaing/INTEL/dpdk-2.0.0/lib/librte_malloc/rte_mallo= c.c:115 #6 0x000000000041ca6e in main (argc=3D5, argv=3D0x7fffffffdd48) at /home/a= laing/INTEL/dpdk-2.0.0/examples/hugephymem/main.c:66 Has anybody seen such an issue? Could I be misusing RTE somehow? Thanks for your time, Alain -- Alain Gautherot Edico Genome