From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2on0080.outbound.protection.outlook.com [207.46.100.80]) by dpdk.org (Postfix) with ESMTP id 4AEE36CA3 for ; Mon, 29 Feb 2016 13:06:35 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=SonusNetworks.onmicrosoft.com; s=selector1-sonusnet-com; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=mNjSPXTe1d1Kcdv0bH+TFVzpdLHcSIEjB22NXSYpBRk=; b=tQaMwukzxG9BP+gEdMV5UqvF8mr+HBtVUTXQKdn1mOecGpucjS9l1RuTbKccKzdMA3dEX8w5rxWPE4jKRsXRkg/d3FoizHtgKQ5Gi8pjhNd/ZV8cy742K957M9twiuBBtG2UOw52oB0Bv/l/ZNArsSQfk1XNfMi2KMiN3AYV+WU= Received: from BY2PR0301MB1542.namprd03.prod.outlook.com (10.163.27.156) by BY2PR0301MB1543.namprd03.prod.outlook.com (10.163.27.157) with Microsoft SMTP Server (TLS) id 15.1.415.20; Mon, 29 Feb 2016 12:06:33 +0000 Received: from BY2PR0301MB1542.namprd03.prod.outlook.com ([10.163.27.156]) by BY2PR0301MB1542.namprd03.prod.outlook.com ([10.163.27.156]) with mapi id 15.01.0415.022; Mon, 29 Feb 2016 12:06:33 +0000 From: "Rapelly, Varun" To: "dev@dpdk.org" Thread-Topic: ACL memory allocation failures Thread-Index: AdFwf4kdB1whTVNNQG6hirscCjj/gAAANPTAAADEVXAAAf5KEAABgkhQAIaC0iAAD0BR4A== Date: Mon, 29 Feb 2016 12:06:32 +0000 Message-ID: References: <2601191342CEEE43887BDE71AB97725836B0BA25@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB97725836B0BADD@irsmsx105.ger.corp.intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=none action=none header.from=sonusnet.com; x-originating-ip: [121.242.142.135] x-ms-office365-filtering-correlation-id: ebfdbbf4-cca9-4be9-1158-08d34100c413 x-microsoft-exchange-diagnostics: 1; BY2PR0301MB1543; 5:M/dRbp/t2wwJRZLXNORg/DYn1bbdEYvdcFS/+58oUMebrCgtMB+kYK7bWbPfNiW0RHGymy/2b8IoBgUPrXxhvsyNO3ilACyCQuRqdTr9LauAY1Xc3kmM2oOatFn5z02XLQAVtHumXahYDVtEcCaOLg==; 24:+TU/+k0r/E0thLUyuc33B8Ak4svsm4O7CaNc7QVR7P02SQDwFZ4CJgAIw3bIrYvQz5g+YB4eNZiLhkH7rxBuAkDVwm0I0ozZ/JWiYVMrgZk= x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BY2PR0301MB1543; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001); SRVR:BY2PR0301MB1543; BCL:0; PCL:0; RULEID:; SRVR:BY2PR0301MB1543; x-forefront-prvs: 0867F4F1AA x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(13464003)(24434003)(377454003)(86362001)(110136002)(5001960100002)(19580405001)(19580395003)(93886004)(76576001)(5002640100001)(1096002)(1220700001)(102836003)(586003)(99286002)(2501003)(1730700002)(3846002)(6116002)(74316001)(2351001)(92566002)(5004730100002)(122556002)(76176999)(11100500001)(50986999)(54356999)(3280700002)(77096005)(2950100001)(15975445007)(3480700003)(2900100001)(3660700001)(15395725005)(10400500002)(40100003)(5003600100002)(87936001)(5008740100001)(2906002)(4326007)(189998001)(33656002); DIR:OUT; SFP:1101; SCL:1; SRVR:BY2PR0301MB1543; H:BY2PR0301MB1542.namprd03.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: sonusnet.com X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Feb 2016 12:06:32.4760 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 29a671dc-ed7e-4a54-b1e5-8da1eb495dc3 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR0301MB1543 Subject: Re: [dpdk-dev] ACL memory allocation failures X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Feb 2016 12:06:35 -0000 Thanks Konstantin. Few more questions in line: >=20 > Previous allocation error was coming with 1024 huge pages of 2 MB size. >=20 > After increasing the huge pages to 2048, I was able to add another > ~140 rules [IPv4 rule data--> with src, dst IP address & port, next heade= r ] more, ie., 950 rules were added. That's strange according to your log, all you need is ~13MB of hugepage mem= ory: ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 Wonder wh= at consumed rest of 4GB? >> We are creating mem pools (for DPDK compatible 3 ports) for packet proce= ssing. >>> And there are no free huge pages available after our DPDK app initializ= ation. Again do you re-build your table after every rule you add? If so, then it seems a bit strange approach (and definitely not the fastest= one). >>Yes, we are rebuilding the rules every time and is due to 2 reasons:=20 >>1. Our application, gives full list of rules every time you add new rule.= =20 >>2. There is no way to delete a specific rule in the trie. Is there any wa= y to delete a specific ACL rule? What you can do instead: create context; add all your rules into it; build;= =20 >>> By following the same approach (what I explained above, rebuilding the = ACL trie everytime), can we fix this memory allocation issue? >>>If yes, please provide me some pointers to modify the code. >=20 > Logically it did not increase number of rules [expected 2*817, but only 9= 50 were added]. Is it really using huge pages memory only? >=20 > From the code it looks like heap memory. [ ret =3D=20 > malloc_heap_alloc(&mcfg->malloc_heaps[i], type, size, 0, align =3D=3D 0 ? > 1 : align, 0) ] As I can see from the log it fails at GEN phase, when trying to allocate hu= gepages for RT table. At lib/librte_acl/acl_gen.c:509 rte_acl_gen(struct rte_acl_ctx *ctx, struct rte_acl_trie *trie, struct rte_acl_bld_trie *node_bld_trie, uint32_t num_tries, uint32_t num_categories, uint32_t data_index_sz, size_t max_size) {= ... mem =3D rte_zmalloc_socket(ctx->name, total_size, RTE_CACHE_LINE_SIZE, ctx->socket_id); if (mem =3D=3D NULL) { RTE_LOG(ERR, ACL, "allocation of %zu bytes on socket %d for %s failed= \n", total_size, ctx->socket_id, ctx->name); return -ENOMEM; } >>> Is there any way to reserve some particular amount of huge page memory = for ACL trie (in eal_init())? Konstantin >=20 > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun > > Sent: Friday, February 26, 2016 10:28 AM > > To: dev@dpdk.org > > Subject: Re: [dpdk-dev] ACL memory allocation failures > > > > Hi All, > > > > When I'm trying to configure some 5000+ ACL rules with different=20 > > source IP addresses, getting ACL memory allocation failure. I'm using D= PDK 2.1. > > > > [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log match=20 > > nodes/bytes > > used: 816/104448 > > total: 12940832 bytes > > ACL: Build phase for ACL "ipv4_acl_table2": > > memory consumed: 947913495 > > ACL: trie 0: number of rules: 816 > > ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1=20 > > failed > > ACL: Build phase for ACL "ipv4_acl_table1": > > memory consumed: 947913495 > > ACL: trie 0: number of rules: 817 > > EAL: Error - exiting with code: 1 > > Cause: Failed to build ACL trie > > > > Again sourced the ACL config file. After adding around 77 again the sam= e error came. > > > > total: 14912784 bytes > > ACL: Build phase for ACL "ipv4_acl_table1": > > memory consumed: 1040188260 > > ACL: trie 0: number of rules: 893 > > ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2=20 > > failed >=20 > You are running out of hugepages memory. >=20 > > ACL: Build phase for ACL "ipv4_acl_table2": > > memory consumed: 1040188260 > > ACL: trie 0: number of rules: 894 > > EAL: Error - exiting with code: 1 > > Cause: Failed to build ACL trie > > > > Where to increase the memory to avoid this issue? >=20 > Refer to: > http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applic > ations > Section 2.3.2 >=20 > Konstantin