* [dpdk-dev] ACL memory allocation failures @ 2016-02-26 10:27 Rapelly, Varun 0 siblings, 0 replies; 7+ messages in thread From: Rapelly, Varun @ 2016-02-26 10:27 UTC (permalink / raw) To: dev Hi All, When I'm trying to configure some 5000+ ACL rules with different source IP addresses, getting ACL memory allocation failure.\ [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log match nodes/bytes used: 816/104448 total: 12940832 bytes ACL: Build phase for ACL "ipv4_acl_table2": memory consumed: 947913495 ACL: trie 0: number of rules: 816 ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 failed ACL: Build phase for ACL "ipv4_acl_table1": memory consumed: 947913495 ACL: trie 0: number of rules: 817 EAL: Error - exiting with code: 1 Cause: Failed to build ACL trie Again sourced the ACL config file. After adding around 77 again the same error came. total: 14912784 bytes ACL: Build phase for ACL "ipv4_acl_table1": memory consumed: 1040188260 ACL: trie 0: number of rules: 893 ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2 failed ACL: Build phase for ACL "ipv4_acl_table2": memory consumed: 1040188260 ACL: trie 0: number of rules: 894 EAL: Error - exiting with code: 1 Cause: Failed to build ACL trie Where to increase the memory to avoid this issue? Regards, Varun ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-dev] ACL memory allocation failures @ 2016-02-26 10:28 Rapelly, Varun 2016-02-26 10:53 ` Ananyev, Konstantin 0 siblings, 1 reply; 7+ messages in thread From: Rapelly, Varun @ 2016-02-26 10:28 UTC (permalink / raw) To: dev Hi All, When I'm trying to configure some 5000+ ACL rules with different source IP addresses, getting ACL memory allocation failure. I'm using DPDK 2.1. [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log match nodes/bytes used: 816/104448 total: 12940832 bytes ACL: Build phase for ACL "ipv4_acl_table2": memory consumed: 947913495 ACL: trie 0: number of rules: 816 ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 failed ACL: Build phase for ACL "ipv4_acl_table1": memory consumed: 947913495 ACL: trie 0: number of rules: 817 EAL: Error - exiting with code: 1 Cause: Failed to build ACL trie Again sourced the ACL config file. After adding around 77 again the same error came. total: 14912784 bytes ACL: Build phase for ACL "ipv4_acl_table1": memory consumed: 1040188260 ACL: trie 0: number of rules: 893 ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2 failed ACL: Build phase for ACL "ipv4_acl_table2": memory consumed: 1040188260 ACL: trie 0: number of rules: 894 EAL: Error - exiting with code: 1 Cause: Failed to build ACL trie Where to increase the memory to avoid this issue? Regards, Varun ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-dev] ACL memory allocation failures 2016-02-26 10:28 Rapelly, Varun @ 2016-02-26 10:53 ` Ananyev, Konstantin 2016-02-26 11:55 ` Rapelly, Varun 0 siblings, 1 reply; 7+ messages in thread From: Ananyev, Konstantin @ 2016-02-26 10:53 UTC (permalink / raw) To: Rapelly, Varun, dev > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun > Sent: Friday, February 26, 2016 10:28 AM > To: dev@dpdk.org > Subject: Re: [dpdk-dev] ACL memory allocation failures > > Hi All, > > When I'm trying to configure some 5000+ ACL rules with different source IP addresses, getting ACL memory allocation failure. I'm using > DPDK 2.1. > > [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log > match nodes/bytes used: 816/104448 > total: 12940832 bytes > ACL: Build phase for ACL "ipv4_acl_table2": > memory consumed: 947913495 > ACL: trie 0: number of rules: 816 > ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 failed > ACL: Build phase for ACL "ipv4_acl_table1": > memory consumed: 947913495 > ACL: trie 0: number of rules: 817 > EAL: Error - exiting with code: 1 > Cause: Failed to build ACL trie > > Again sourced the ACL config file. After adding around 77 again the same error came. > > total: 14912784 bytes > ACL: Build phase for ACL "ipv4_acl_table1": > memory consumed: 1040188260 > ACL: trie 0: number of rules: 893 > ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2 failed You are running out of hugepages memory. > ACL: Build phase for ACL "ipv4_acl_table2": > memory consumed: 1040188260 > ACL: trie 0: number of rules: 894 > EAL: Error - exiting with code: 1 > Cause: Failed to build ACL trie > > Where to increase the memory to avoid this issue? Refer to: http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applications Section 2.3.2 Konstantin ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-dev] ACL memory allocation failures 2016-02-26 10:53 ` Ananyev, Konstantin @ 2016-02-26 11:55 ` Rapelly, Varun 2016-02-26 12:39 ` Ananyev, Konstantin 0 siblings, 1 reply; 7+ messages in thread From: Rapelly, Varun @ 2016-02-26 11:55 UTC (permalink / raw) To: Ananyev, Konstantin, dev Thanks Konstantin. Previous allocation error was coming with 1024 huge pages of 2 MB size. After increasing the huge pages to 2048, I was able to add another ~140 rules [IPv4 rule data--> with src, dst IP address & port, next header ] more, ie., 950 rules were added. Logically it did not increase number of rules [expected 2*817, but only 950 were added]. Is it really using huge pages memory only? >From the code it looks like heap memory. [ ret = malloc_heap_alloc(&mcfg->malloc_heaps[i], type, size, 0, align == 0 ? 1 : align, 0) ] > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun > Sent: Friday, February 26, 2016 10:28 AM > To: dev@dpdk.org > Subject: Re: [dpdk-dev] ACL memory allocation failures > > Hi All, > > When I'm trying to configure some 5000+ ACL rules with different > source IP addresses, getting ACL memory allocation failure. I'm using DPDK 2.1. > > [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log match nodes/bytes > used: 816/104448 > total: 12940832 bytes > ACL: Build phase for ACL "ipv4_acl_table2": > memory consumed: 947913495 > ACL: trie 0: number of rules: 816 > ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 > failed > ACL: Build phase for ACL "ipv4_acl_table1": > memory consumed: 947913495 > ACL: trie 0: number of rules: 817 > EAL: Error - exiting with code: 1 > Cause: Failed to build ACL trie > > Again sourced the ACL config file. After adding around 77 again the same error came. > > total: 14912784 bytes > ACL: Build phase for ACL "ipv4_acl_table1": > memory consumed: 1040188260 > ACL: trie 0: number of rules: 893 > ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2 > failed You are running out of hugepages memory. > ACL: Build phase for ACL "ipv4_acl_table2": > memory consumed: 1040188260 > ACL: trie 0: number of rules: 894 > EAL: Error - exiting with code: 1 > Cause: Failed to build ACL trie > > Where to increase the memory to avoid this issue? Refer to: http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applications Section 2.3.2 Konstantin ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-dev] ACL memory allocation failures 2016-02-26 11:55 ` Rapelly, Varun @ 2016-02-26 12:39 ` Ananyev, Konstantin 2016-02-29 5:00 ` Rapelly, Varun 0 siblings, 1 reply; 7+ messages in thread From: Ananyev, Konstantin @ 2016-02-26 12:39 UTC (permalink / raw) To: Rapelly, Varun, dev > > Thanks Konstantin. > > Previous allocation error was coming with 1024 huge pages of 2 MB size. > > After increasing the huge pages to 2048, I was able to add another ~140 rules [IPv4 rule data--> with src, dst IP address & port, next > header ] more, ie., 950 rules were added. That's strange according to your log, all you need is ~13MB of hugepage memory: ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 Wonder what consumed rest of 4GB? Again do you re-build your table after every rule you add? If so, then it seems a bit strange approach (and definitely not the fastest one). What you can do instead: create context; add all your rules into it; build; > > Logically it did not increase number of rules [expected 2*817, but only 950 were added]. Is it really using huge pages memory only? > > From the code it looks like heap memory. [ ret = malloc_heap_alloc(&mcfg->malloc_heaps[i], type, size, 0, align == 0 ? 1 : align, 0) ] As I can see from the log it fails at GEN phase, when trying to allocate hugepages for RT table. At lib/librte_acl/acl_gen.c:509 rte_acl_gen(struct rte_acl_ctx *ctx, struct rte_acl_trie *trie, struct rte_acl_bld_trie *node_bld_trie, uint32_t num_tries, uint32_t num_categories, uint32_t data_index_sz, size_t max_size) { ... mem = rte_zmalloc_socket(ctx->name, total_size, RTE_CACHE_LINE_SIZE, ctx->socket_id); if (mem == NULL) { RTE_LOG(ERR, ACL, "allocation of %zu bytes on socket %d for %s failed\n", total_size, ctx->socket_id, ctx->name); return -ENOMEM; } Konstantin > > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun > > Sent: Friday, February 26, 2016 10:28 AM > > To: dev@dpdk.org > > Subject: Re: [dpdk-dev] ACL memory allocation failures > > > > Hi All, > > > > When I'm trying to configure some 5000+ ACL rules with different > > source IP addresses, getting ACL memory allocation failure. I'm using DPDK 2.1. > > > > [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log match nodes/bytes > > used: 816/104448 > > total: 12940832 bytes > > ACL: Build phase for ACL "ipv4_acl_table2": > > memory consumed: 947913495 > > ACL: trie 0: number of rules: 816 > > ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 > > failed > > ACL: Build phase for ACL "ipv4_acl_table1": > > memory consumed: 947913495 > > ACL: trie 0: number of rules: 817 > > EAL: Error - exiting with code: 1 > > Cause: Failed to build ACL trie > > > > Again sourced the ACL config file. After adding around 77 again the same error came. > > > > total: 14912784 bytes > > ACL: Build phase for ACL "ipv4_acl_table1": > > memory consumed: 1040188260 > > ACL: trie 0: number of rules: 893 > > ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2 > > failed > > You are running out of hugepages memory. > > > ACL: Build phase for ACL "ipv4_acl_table2": > > memory consumed: 1040188260 > > ACL: trie 0: number of rules: 894 > > EAL: Error - exiting with code: 1 > > Cause: Failed to build ACL trie > > > > Where to increase the memory to avoid this issue? > > Refer to: > http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applications > Section 2.3.2 > > Konstantin ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-dev] ACL memory allocation failures 2016-02-26 12:39 ` Ananyev, Konstantin @ 2016-02-29 5:00 ` Rapelly, Varun 2016-02-29 12:06 ` Rapelly, Varun 0 siblings, 1 reply; 7+ messages in thread From: Rapelly, Varun @ 2016-02-29 5:00 UTC (permalink / raw) To: Ananyev, Konstantin, dev > > Thanks Konstantin. > > Previous allocation error was coming with 1024 huge pages of 2 MB size. > > After increasing the huge pages to 2048, I was able to add another > ~140 rules [IPv4 rule data--> with src, dst IP address & port, next header ] more, ie., 950 rules were added. That's strange according to your log, all you need is ~13MB of hugepage memory: ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 Wonder what consumed rest of 4GB? >> We are creating mem pools (for DPDK compatible 3 ports) for packet processing. Again do you re-build your table after every rule you add? If so, then it seems a bit strange approach (and definitely not the fastest one). >>Yes, we are rebuilding the rules every time and is due to 2 reasons: >>1. Our application, gives full list of rules every time you add new rule. >>2. There is no way to delete a specific rule in the trie. Is there any way to delete a specific ACL rule? What you can do instead: create context; add all your rules into it; build; > > Logically it did not increase number of rules [expected 2*817, but only 950 were added]. Is it really using huge pages memory only? > > From the code it looks like heap memory. [ ret = > malloc_heap_alloc(&mcfg->malloc_heaps[i], type, size, 0, align == 0 ? > 1 : align, 0) ] As I can see from the log it fails at GEN phase, when trying to allocate hugepages for RT table. At lib/librte_acl/acl_gen.c:509 rte_acl_gen(struct rte_acl_ctx *ctx, struct rte_acl_trie *trie, struct rte_acl_bld_trie *node_bld_trie, uint32_t num_tries, uint32_t num_categories, uint32_t data_index_sz, size_t max_size) { ... mem = rte_zmalloc_socket(ctx->name, total_size, RTE_CACHE_LINE_SIZE, ctx->socket_id); if (mem == NULL) { RTE_LOG(ERR, ACL, "allocation of %zu bytes on socket %d for %s failed\n", total_size, ctx->socket_id, ctx->name); return -ENOMEM; } Konstantin > > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun > > Sent: Friday, February 26, 2016 10:28 AM > > To: dev@dpdk.org > > Subject: Re: [dpdk-dev] ACL memory allocation failures > > > > Hi All, > > > > When I'm trying to configure some 5000+ ACL rules with different > > source IP addresses, getting ACL memory allocation failure. I'm using DPDK 2.1. > > > > [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log match > > nodes/bytes > > used: 816/104448 > > total: 12940832 bytes > > ACL: Build phase for ACL "ipv4_acl_table2": > > memory consumed: 947913495 > > ACL: trie 0: number of rules: 816 > > ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 > > failed > > ACL: Build phase for ACL "ipv4_acl_table1": > > memory consumed: 947913495 > > ACL: trie 0: number of rules: 817 > > EAL: Error - exiting with code: 1 > > Cause: Failed to build ACL trie > > > > Again sourced the ACL config file. After adding around 77 again the same error came. > > > > total: 14912784 bytes > > ACL: Build phase for ACL "ipv4_acl_table1": > > memory consumed: 1040188260 > > ACL: trie 0: number of rules: 893 > > ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2 > > failed > > You are running out of hugepages memory. > > > ACL: Build phase for ACL "ipv4_acl_table2": > > memory consumed: 1040188260 > > ACL: trie 0: number of rules: 894 > > EAL: Error - exiting with code: 1 > > Cause: Failed to build ACL trie > > > > Where to increase the memory to avoid this issue? > > Refer to: > http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applic > ations > Section 2.3.2 > > Konstantin ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-dev] ACL memory allocation failures 2016-02-29 5:00 ` Rapelly, Varun @ 2016-02-29 12:06 ` Rapelly, Varun 0 siblings, 0 replies; 7+ messages in thread From: Rapelly, Varun @ 2016-02-29 12:06 UTC (permalink / raw) To: dev Thanks Konstantin. Few more questions in line: > > Previous allocation error was coming with 1024 huge pages of 2 MB size. > > After increasing the huge pages to 2048, I was able to add another > ~140 rules [IPv4 rule data--> with src, dst IP address & port, next header ] more, ie., 950 rules were added. That's strange according to your log, all you need is ~13MB of hugepage memory: ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 Wonder what consumed rest of 4GB? >> We are creating mem pools (for DPDK compatible 3 ports) for packet processing. >>> And there are no free huge pages available after our DPDK app initialization. Again do you re-build your table after every rule you add? If so, then it seems a bit strange approach (and definitely not the fastest one). >>Yes, we are rebuilding the rules every time and is due to 2 reasons: >>1. Our application, gives full list of rules every time you add new rule. >>2. There is no way to delete a specific rule in the trie. Is there any way to delete a specific ACL rule? What you can do instead: create context; add all your rules into it; build; >>> By following the same approach (what I explained above, rebuilding the ACL trie everytime), can we fix this memory allocation issue? >>>If yes, please provide me some pointers to modify the code. > > Logically it did not increase number of rules [expected 2*817, but only 950 were added]. Is it really using huge pages memory only? > > From the code it looks like heap memory. [ ret = > malloc_heap_alloc(&mcfg->malloc_heaps[i], type, size, 0, align == 0 ? > 1 : align, 0) ] As I can see from the log it fails at GEN phase, when trying to allocate hugepages for RT table. At lib/librte_acl/acl_gen.c:509 rte_acl_gen(struct rte_acl_ctx *ctx, struct rte_acl_trie *trie, struct rte_acl_bld_trie *node_bld_trie, uint32_t num_tries, uint32_t num_categories, uint32_t data_index_sz, size_t max_size) { ... mem = rte_zmalloc_socket(ctx->name, total_size, RTE_CACHE_LINE_SIZE, ctx->socket_id); if (mem == NULL) { RTE_LOG(ERR, ACL, "allocation of %zu bytes on socket %d for %s failed\n", total_size, ctx->socket_id, ctx->name); return -ENOMEM; } >>> Is there any way to reserve some particular amount of huge page memory for ACL trie (in eal_init())? Konstantin > > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun > > Sent: Friday, February 26, 2016 10:28 AM > > To: dev@dpdk.org > > Subject: Re: [dpdk-dev] ACL memory allocation failures > > > > Hi All, > > > > When I'm trying to configure some 5000+ ACL rules with different > > source IP addresses, getting ACL memory allocation failure. I'm using DPDK 2.1. > > > > [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log match > > nodes/bytes > > used: 816/104448 > > total: 12940832 bytes > > ACL: Build phase for ACL "ipv4_acl_table2": > > memory consumed: 947913495 > > ACL: trie 0: number of rules: 816 > > ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 > > failed > > ACL: Build phase for ACL "ipv4_acl_table1": > > memory consumed: 947913495 > > ACL: trie 0: number of rules: 817 > > EAL: Error - exiting with code: 1 > > Cause: Failed to build ACL trie > > > > Again sourced the ACL config file. After adding around 77 again the same error came. > > > > total: 14912784 bytes > > ACL: Build phase for ACL "ipv4_acl_table1": > > memory consumed: 1040188260 > > ACL: trie 0: number of rules: 893 > > ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2 > > failed > > You are running out of hugepages memory. > > > ACL: Build phase for ACL "ipv4_acl_table2": > > memory consumed: 1040188260 > > ACL: trie 0: number of rules: 894 > > EAL: Error - exiting with code: 1 > > Cause: Failed to build ACL trie > > > > Where to increase the memory to avoid this issue? > > Refer to: > http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applic > ations > Section 2.3.2 > > Konstantin ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2016-02-29 12:06 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-02-26 10:27 [dpdk-dev] ACL memory allocation failures Rapelly, Varun 2016-02-26 10:28 Rapelly, Varun 2016-02-26 10:53 ` Ananyev, Konstantin 2016-02-26 11:55 ` Rapelly, Varun 2016-02-26 12:39 ` Ananyev, Konstantin 2016-02-29 5:00 ` Rapelly, Varun 2016-02-29 12:06 ` Rapelly, Varun
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).