From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2on0096.outbound.protection.outlook.com [65.55.169.96]) by dpdk.org (Postfix) with ESMTP id 969D15598 for ; Mon, 29 Feb 2016 06:00:03 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=SonusNetworks.onmicrosoft.com; s=selector1-sonusnet-com; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=4QTSyhJXYvx5ddAmz78F0QFZ3LkzumuPR4bZHQ6Onw8=; b=WleccKsOvHZnr76fDV3Eps5jo79Pk/UvCZUQYZZik0f5d1IyAi93JAm/PnnPtEfGvXPpAS1peiZQyvIsqaV1KSjO2rMZxe59l6Yzr+n8dEEgcvzI1GleJIEIej8vHkWNcpHizqGq6lFSBToqY5FKxPXXfrIPgY+O25XS0soaUL4= Received: from BY2PR0301MB1542.namprd03.prod.outlook.com (10.163.27.156) by BY2PR0301MB1541.namprd03.prod.outlook.com (10.163.27.155) with Microsoft SMTP Server (TLS) id 15.1.415.20; Mon, 29 Feb 2016 05:00:00 +0000 Received: from BY2PR0301MB1542.namprd03.prod.outlook.com ([10.163.27.156]) by BY2PR0301MB1542.namprd03.prod.outlook.com ([10.163.27.156]) with mapi id 15.01.0415.022; Mon, 29 Feb 2016 05:00:00 +0000 From: "Rapelly, Varun" To: "Ananyev, Konstantin" , "dev@dpdk.org" Thread-Topic: ACL memory allocation failures Thread-Index: AdFwf4kdB1whTVNNQG6hirscCjj/gAAANPTAAADEVXAAAf5KEAABgkhQAIaC0iA= Date: Mon, 29 Feb 2016 05:00:00 +0000 Message-ID: References: <2601191342CEEE43887BDE71AB97725836B0BA25@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB97725836B0BADD@irsmsx105.ger.corp.intel.com> In-Reply-To: <2601191342CEEE43887BDE71AB97725836B0BADD@irsmsx105.ger.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=sonusnet.com; x-originating-ip: [121.242.142.135] x-ms-office365-filtering-correlation-id: 151ca811-1f10-403d-027c-08d340c52d8a x-microsoft-exchange-diagnostics: 1; BY2PR0301MB1541; 5:Bbidaexa7Jet76gZGVUK3XbR7lFWUdFGD72hN28WmyOb1GOUWK7u5+qoThSC4LCTNyZtvWGPDWzKKM3JkfyQI/EWnAGEKkxVt5mixyPM5UiwuZe3pGweuytfjQCSNMdZCPFwBrG2GECJfzEk1LZqQQ==; 24:8dZuW8qb/H1OJfLwrQOZI+P6fp5lH52kckfMawfJoScvakZCEQSKzzkyDSNLl9sK9DcgrnKWfWWzpk8U9jYeDpXvdPrgWeG7o5b9M5n09xA= x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BY2PR0301MB1541; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001); SRVR:BY2PR0301MB1541; BCL:0; PCL:0; RULEID:; SRVR:BY2PR0301MB1541; x-forefront-prvs: 0867F4F1AA x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(24434003)(377454003)(13464003)(53754006)(5003600100002)(2906002)(2501003)(5001960100002)(107886002)(74316001)(93886004)(5002640100001)(5008740100001)(19580405001)(92566002)(3280700002)(15395725005)(19580395003)(3480700003)(66066001)(3660700001)(99286002)(86362001)(50986999)(1096002)(11100500001)(2900100001)(1220700001)(5001770100001)(54356999)(15975445007)(77096005)(33656002)(189998001)(586003)(2950100001)(6116002)(102836003)(3846002)(5004730100002)(87936001)(122556002)(40100003)(76576001); DIR:OUT; SFP:1101; SCL:1; SRVR:BY2PR0301MB1541; H:BY2PR0301MB1542.namprd03.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: sonusnet.com X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Feb 2016 05:00:00.4407 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 29a671dc-ed7e-4a54-b1e5-8da1eb495dc3 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR0301MB1541 Subject: Re: [dpdk-dev] ACL memory allocation failures X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Feb 2016 05:00:04 -0000 >=20 > Thanks Konstantin. >=20 > Previous allocation error was coming with 1024 huge pages of 2 MB size. >=20 > After increasing the huge pages to 2048, I was able to add another=20 > ~140 rules [IPv4 rule data--> with src, dst IP address & port, next heade= r ] more, ie., 950 rules were added. That's strange according to your log, all you need is ~13MB of hugepage mem= ory: ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 Wonder wh= at consumed rest of 4GB? >> We are creating mem pools (for DPDK compatible 3 ports) for packet proce= ssing. Again do you re-build your table after every rule you add? If so, then it seems a bit strange approach (and definitely not the fastest= one). >>Yes, we are rebuilding the rules every time and is due to 2 reasons:=20 >>1. Our application, gives full list of rules every time you add new rule.= =20 >>2. There is no way to delete a specific rule in the trie. Is there any wa= y to delete a specific ACL rule? What you can do instead: create context; add all your rules into it; build;= =20 >=20 > Logically it did not increase number of rules [expected 2*817, but only 9= 50 were added]. Is it really using huge pages memory only? >=20 > From the code it looks like heap memory. [ ret =3D=20 > malloc_heap_alloc(&mcfg->malloc_heaps[i], type, size, 0, align =3D=3D 0 ?= =20 > 1 : align, 0) ] As I can see from the log it fails at GEN phase, when trying to allocate hu= gepages for RT table. At lib/librte_acl/acl_gen.c:509 rte_acl_gen(struct rte_acl_ctx *ctx, struct rte_acl_trie *trie, struct rte_acl_bld_trie *node_bld_trie, uint32_t num_tries, uint32_t num_categories, uint32_t data_index_sz, size_t max_size) {= ... mem =3D rte_zmalloc_socket(ctx->name, total_size, RTE_CACHE_LINE_SIZE, ctx->socket_id); if (mem =3D=3D NULL) { RTE_LOG(ERR, ACL, "allocation of %zu bytes on socket %d for %s failed= \n", total_size, ctx->socket_id, ctx->name); return -ENOMEM; } Konstantin >=20 > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun > > Sent: Friday, February 26, 2016 10:28 AM > > To: dev@dpdk.org > > Subject: Re: [dpdk-dev] ACL memory allocation failures > > > > Hi All, > > > > When I'm trying to configure some 5000+ ACL rules with different=20 > > source IP addresses, getting ACL memory allocation failure. I'm using D= PDK 2.1. > > > > [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log match=20 > > nodes/bytes > > used: 816/104448 > > total: 12940832 bytes > > ACL: Build phase for ACL "ipv4_acl_table2": > > memory consumed: 947913495 > > ACL: trie 0: number of rules: 816 > > ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1=20 > > failed > > ACL: Build phase for ACL "ipv4_acl_table1": > > memory consumed: 947913495 > > ACL: trie 0: number of rules: 817 > > EAL: Error - exiting with code: 1 > > Cause: Failed to build ACL trie > > > > Again sourced the ACL config file. After adding around 77 again the sam= e error came. > > > > total: 14912784 bytes > > ACL: Build phase for ACL "ipv4_acl_table1": > > memory consumed: 1040188260 > > ACL: trie 0: number of rules: 893 > > ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2=20 > > failed >=20 > You are running out of hugepages memory. >=20 > > ACL: Build phase for ACL "ipv4_acl_table2": > > memory consumed: 1040188260 > > ACL: trie 0: number of rules: 894 > > EAL: Error - exiting with code: 1 > > Cause: Failed to build ACL trie > > > > Where to increase the memory to avoid this issue? >=20 > Refer to: > http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applic > ations > Section 2.3.2 >=20 > Konstantin