* [dpdk-dev] ACL memory allocation failures
@ 2016-02-26 10:27 Rapelly, Varun
0 siblings, 0 replies; 7+ messages in thread
From: Rapelly, Varun @ 2016-02-26 10:27 UTC (permalink / raw)
To: dev
Hi All,
When I'm trying to configure some 5000+ ACL rules with different source IP addresses, getting ACL memory allocation failure.\
[root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log
match nodes/bytes used: 816/104448
total: 12940832 bytes
ACL: Build phase for ACL "ipv4_acl_table2":
memory consumed: 947913495
ACL: trie 0: number of rules: 816
ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 failed
ACL: Build phase for ACL "ipv4_acl_table1":
memory consumed: 947913495
ACL: trie 0: number of rules: 817
EAL: Error - exiting with code: 1
Cause: Failed to build ACL trie
Again sourced the ACL config file. After adding around 77 again the same error came.
total: 14912784 bytes
ACL: Build phase for ACL "ipv4_acl_table1":
memory consumed: 1040188260
ACL: trie 0: number of rules: 893
ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2 failed
ACL: Build phase for ACL "ipv4_acl_table2":
memory consumed: 1040188260
ACL: trie 0: number of rules: 894
EAL: Error - exiting with code: 1
Cause: Failed to build ACL trie
Where to increase the memory to avoid this issue?
Regards,
Varun
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-dev] ACL memory allocation failures
2016-02-29 5:00 ` Rapelly, Varun
@ 2016-02-29 12:06 ` Rapelly, Varun
0 siblings, 0 replies; 7+ messages in thread
From: Rapelly, Varun @ 2016-02-29 12:06 UTC (permalink / raw)
To: dev
Thanks Konstantin. Few more questions in line:
>
> Previous allocation error was coming with 1024 huge pages of 2 MB size.
>
> After increasing the huge pages to 2048, I was able to add another
> ~140 rules [IPv4 rule data--> with src, dst IP address & port, next header ] more, ie., 950 rules were added.
That's strange according to your log, all you need is ~13MB of hugepage memory:
ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 Wonder what consumed rest of 4GB?
>> We are creating mem pools (for DPDK compatible 3 ports) for packet processing.
>>> And there are no free huge pages available after our DPDK app initialization.
Again do you re-build your table after every rule you add?
If so, then it seems a bit strange approach (and definitely not the fastest one).
>>Yes, we are rebuilding the rules every time and is due to 2 reasons:
>>1. Our application, gives full list of rules every time you add new rule.
>>2. There is no way to delete a specific rule in the trie. Is there any way to delete a specific ACL rule?
What you can do instead: create context; add all your rules into it; build;
>>> By following the same approach (what I explained above, rebuilding the ACL trie everytime), can we fix this memory allocation issue?
>>>If yes, please provide me some pointers to modify the code.
>
> Logically it did not increase number of rules [expected 2*817, but only 950 were added]. Is it really using huge pages memory only?
>
> From the code it looks like heap memory. [ ret =
> malloc_heap_alloc(&mcfg->malloc_heaps[i], type, size, 0, align == 0 ?
> 1 : align, 0) ]
As I can see from the log it fails at GEN phase, when trying to allocate hugepages for RT table.
At lib/librte_acl/acl_gen.c:509
rte_acl_gen(struct rte_acl_ctx *ctx, struct rte_acl_trie *trie,
struct rte_acl_bld_trie *node_bld_trie, uint32_t num_tries,
uint32_t num_categories, uint32_t data_index_sz, size_t max_size) { ...
mem = rte_zmalloc_socket(ctx->name, total_size, RTE_CACHE_LINE_SIZE,
ctx->socket_id); if (mem == NULL) {
RTE_LOG(ERR, ACL,
"allocation of %zu bytes on socket %d for %s failed\n",
total_size, ctx->socket_id, ctx->name);
return -ENOMEM;
}
>>> Is there any way to reserve some particular amount of huge page memory for ACL trie (in eal_init())?
Konstantin
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun
> > Sent: Friday, February 26, 2016 10:28 AM
> > To: dev@dpdk.org
> > Subject: Re: [dpdk-dev] ACL memory allocation failures
> >
> > Hi All,
> >
> > When I'm trying to configure some 5000+ ACL rules with different
> > source IP addresses, getting ACL memory allocation failure. I'm using DPDK 2.1.
> >
> > [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log match
> > nodes/bytes
> > used: 816/104448
> > total: 12940832 bytes
> > ACL: Build phase for ACL "ipv4_acl_table2":
> > memory consumed: 947913495
> > ACL: trie 0: number of rules: 816
> > ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1
> > failed
> > ACL: Build phase for ACL "ipv4_acl_table1":
> > memory consumed: 947913495
> > ACL: trie 0: number of rules: 817
> > EAL: Error - exiting with code: 1
> > Cause: Failed to build ACL trie
> >
> > Again sourced the ACL config file. After adding around 77 again the same error came.
> >
> > total: 14912784 bytes
> > ACL: Build phase for ACL "ipv4_acl_table1":
> > memory consumed: 1040188260
> > ACL: trie 0: number of rules: 893
> > ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2
> > failed
>
> You are running out of hugepages memory.
>
> > ACL: Build phase for ACL "ipv4_acl_table2":
> > memory consumed: 1040188260
> > ACL: trie 0: number of rules: 894
> > EAL: Error - exiting with code: 1
> > Cause: Failed to build ACL trie
> >
> > Where to increase the memory to avoid this issue?
>
> Refer to:
> http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applic
> ations
> Section 2.3.2
>
> Konstantin
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-dev] ACL memory allocation failures
2016-02-26 12:39 ` Ananyev, Konstantin
@ 2016-02-29 5:00 ` Rapelly, Varun
2016-02-29 12:06 ` Rapelly, Varun
0 siblings, 1 reply; 7+ messages in thread
From: Rapelly, Varun @ 2016-02-29 5:00 UTC (permalink / raw)
To: Ananyev, Konstantin, dev
>
> Thanks Konstantin.
>
> Previous allocation error was coming with 1024 huge pages of 2 MB size.
>
> After increasing the huge pages to 2048, I was able to add another
> ~140 rules [IPv4 rule data--> with src, dst IP address & port, next header ] more, ie., 950 rules were added.
That's strange according to your log, all you need is ~13MB of hugepage memory:
ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 Wonder what consumed rest of 4GB?
>> We are creating mem pools (for DPDK compatible 3 ports) for packet processing.
Again do you re-build your table after every rule you add?
If so, then it seems a bit strange approach (and definitely not the fastest one).
>>Yes, we are rebuilding the rules every time and is due to 2 reasons:
>>1. Our application, gives full list of rules every time you add new rule.
>>2. There is no way to delete a specific rule in the trie. Is there any way to delete a specific ACL rule?
What you can do instead: create context; add all your rules into it; build;
>
> Logically it did not increase number of rules [expected 2*817, but only 950 were added]. Is it really using huge pages memory only?
>
> From the code it looks like heap memory. [ ret =
> malloc_heap_alloc(&mcfg->malloc_heaps[i], type, size, 0, align == 0 ?
> 1 : align, 0) ]
As I can see from the log it fails at GEN phase, when trying to allocate hugepages for RT table.
At lib/librte_acl/acl_gen.c:509
rte_acl_gen(struct rte_acl_ctx *ctx, struct rte_acl_trie *trie,
struct rte_acl_bld_trie *node_bld_trie, uint32_t num_tries,
uint32_t num_categories, uint32_t data_index_sz, size_t max_size) { ...
mem = rte_zmalloc_socket(ctx->name, total_size, RTE_CACHE_LINE_SIZE,
ctx->socket_id); if (mem == NULL) {
RTE_LOG(ERR, ACL,
"allocation of %zu bytes on socket %d for %s failed\n",
total_size, ctx->socket_id, ctx->name);
return -ENOMEM;
}
Konstantin
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun
> > Sent: Friday, February 26, 2016 10:28 AM
> > To: dev@dpdk.org
> > Subject: Re: [dpdk-dev] ACL memory allocation failures
> >
> > Hi All,
> >
> > When I'm trying to configure some 5000+ ACL rules with different
> > source IP addresses, getting ACL memory allocation failure. I'm using DPDK 2.1.
> >
> > [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log match
> > nodes/bytes
> > used: 816/104448
> > total: 12940832 bytes
> > ACL: Build phase for ACL "ipv4_acl_table2":
> > memory consumed: 947913495
> > ACL: trie 0: number of rules: 816
> > ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1
> > failed
> > ACL: Build phase for ACL "ipv4_acl_table1":
> > memory consumed: 947913495
> > ACL: trie 0: number of rules: 817
> > EAL: Error - exiting with code: 1
> > Cause: Failed to build ACL trie
> >
> > Again sourced the ACL config file. After adding around 77 again the same error came.
> >
> > total: 14912784 bytes
> > ACL: Build phase for ACL "ipv4_acl_table1":
> > memory consumed: 1040188260
> > ACL: trie 0: number of rules: 893
> > ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2
> > failed
>
> You are running out of hugepages memory.
>
> > ACL: Build phase for ACL "ipv4_acl_table2":
> > memory consumed: 1040188260
> > ACL: trie 0: number of rules: 894
> > EAL: Error - exiting with code: 1
> > Cause: Failed to build ACL trie
> >
> > Where to increase the memory to avoid this issue?
>
> Refer to:
> http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applic
> ations
> Section 2.3.2
>
> Konstantin
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-dev] ACL memory allocation failures
2016-02-26 11:55 ` Rapelly, Varun
@ 2016-02-26 12:39 ` Ananyev, Konstantin
2016-02-29 5:00 ` Rapelly, Varun
0 siblings, 1 reply; 7+ messages in thread
From: Ananyev, Konstantin @ 2016-02-26 12:39 UTC (permalink / raw)
To: Rapelly, Varun, dev
>
> Thanks Konstantin.
>
> Previous allocation error was coming with 1024 huge pages of 2 MB size.
>
> After increasing the huge pages to 2048, I was able to add another ~140 rules [IPv4 rule data--> with src, dst IP address & port, next
> header ] more, ie., 950 rules were added.
That's strange according to your log, all you need is ~13MB of hugepage memory:
ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1
Wonder what consumed rest of 4GB?
Again do you re-build your table after every rule you add?
If so, then it seems a bit strange approach (and definitely not the fastest one).
What you can do instead: create context; add all your rules into it; build;
>
> Logically it did not increase number of rules [expected 2*817, but only 950 were added]. Is it really using huge pages memory only?
>
> From the code it looks like heap memory. [ ret = malloc_heap_alloc(&mcfg->malloc_heaps[i], type, size, 0, align == 0 ? 1 : align, 0) ]
As I can see from the log it fails at GEN phase, when trying to allocate hugepages for RT table.
At lib/librte_acl/acl_gen.c:509
rte_acl_gen(struct rte_acl_ctx *ctx, struct rte_acl_trie *trie,
struct rte_acl_bld_trie *node_bld_trie, uint32_t num_tries,
uint32_t num_categories, uint32_t data_index_sz, size_t max_size)
{
...
mem = rte_zmalloc_socket(ctx->name, total_size, RTE_CACHE_LINE_SIZE,
ctx->socket_id);
if (mem == NULL) {
RTE_LOG(ERR, ACL,
"allocation of %zu bytes on socket %d for %s failed\n",
total_size, ctx->socket_id, ctx->name);
return -ENOMEM;
}
Konstantin
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun
> > Sent: Friday, February 26, 2016 10:28 AM
> > To: dev@dpdk.org
> > Subject: Re: [dpdk-dev] ACL memory allocation failures
> >
> > Hi All,
> >
> > When I'm trying to configure some 5000+ ACL rules with different
> > source IP addresses, getting ACL memory allocation failure. I'm using DPDK 2.1.
> >
> > [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log match nodes/bytes
> > used: 816/104448
> > total: 12940832 bytes
> > ACL: Build phase for ACL "ipv4_acl_table2":
> > memory consumed: 947913495
> > ACL: trie 0: number of rules: 816
> > ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1
> > failed
> > ACL: Build phase for ACL "ipv4_acl_table1":
> > memory consumed: 947913495
> > ACL: trie 0: number of rules: 817
> > EAL: Error - exiting with code: 1
> > Cause: Failed to build ACL trie
> >
> > Again sourced the ACL config file. After adding around 77 again the same error came.
> >
> > total: 14912784 bytes
> > ACL: Build phase for ACL "ipv4_acl_table1":
> > memory consumed: 1040188260
> > ACL: trie 0: number of rules: 893
> > ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2
> > failed
>
> You are running out of hugepages memory.
>
> > ACL: Build phase for ACL "ipv4_acl_table2":
> > memory consumed: 1040188260
> > ACL: trie 0: number of rules: 894
> > EAL: Error - exiting with code: 1
> > Cause: Failed to build ACL trie
> >
> > Where to increase the memory to avoid this issue?
>
> Refer to:
> http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applications
> Section 2.3.2
>
> Konstantin
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-dev] ACL memory allocation failures
2016-02-26 10:53 ` Ananyev, Konstantin
@ 2016-02-26 11:55 ` Rapelly, Varun
2016-02-26 12:39 ` Ananyev, Konstantin
0 siblings, 1 reply; 7+ messages in thread
From: Rapelly, Varun @ 2016-02-26 11:55 UTC (permalink / raw)
To: Ananyev, Konstantin, dev
Thanks Konstantin.
Previous allocation error was coming with 1024 huge pages of 2 MB size.
After increasing the huge pages to 2048, I was able to add another ~140 rules [IPv4 rule data--> with src, dst IP address & port, next header ] more, ie., 950 rules were added.
Logically it did not increase number of rules [expected 2*817, but only 950 were added]. Is it really using huge pages memory only?
>From the code it looks like heap memory. [ ret = malloc_heap_alloc(&mcfg->malloc_heaps[i], type, size, 0, align == 0 ? 1 : align, 0) ]
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun
> Sent: Friday, February 26, 2016 10:28 AM
> To: dev@dpdk.org
> Subject: Re: [dpdk-dev] ACL memory allocation failures
>
> Hi All,
>
> When I'm trying to configure some 5000+ ACL rules with different
> source IP addresses, getting ACL memory allocation failure. I'm using DPDK 2.1.
>
> [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log match nodes/bytes
> used: 816/104448
> total: 12940832 bytes
> ACL: Build phase for ACL "ipv4_acl_table2":
> memory consumed: 947913495
> ACL: trie 0: number of rules: 816
> ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1
> failed
> ACL: Build phase for ACL "ipv4_acl_table1":
> memory consumed: 947913495
> ACL: trie 0: number of rules: 817
> EAL: Error - exiting with code: 1
> Cause: Failed to build ACL trie
>
> Again sourced the ACL config file. After adding around 77 again the same error came.
>
> total: 14912784 bytes
> ACL: Build phase for ACL "ipv4_acl_table1":
> memory consumed: 1040188260
> ACL: trie 0: number of rules: 893
> ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2
> failed
You are running out of hugepages memory.
> ACL: Build phase for ACL "ipv4_acl_table2":
> memory consumed: 1040188260
> ACL: trie 0: number of rules: 894
> EAL: Error - exiting with code: 1
> Cause: Failed to build ACL trie
>
> Where to increase the memory to avoid this issue?
Refer to:
http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applications
Section 2.3.2
Konstantin
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-dev] ACL memory allocation failures
2016-02-26 10:28 Rapelly, Varun
@ 2016-02-26 10:53 ` Ananyev, Konstantin
2016-02-26 11:55 ` Rapelly, Varun
0 siblings, 1 reply; 7+ messages in thread
From: Ananyev, Konstantin @ 2016-02-26 10:53 UTC (permalink / raw)
To: Rapelly, Varun, dev
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun
> Sent: Friday, February 26, 2016 10:28 AM
> To: dev@dpdk.org
> Subject: Re: [dpdk-dev] ACL memory allocation failures
>
> Hi All,
>
> When I'm trying to configure some 5000+ ACL rules with different source IP addresses, getting ACL memory allocation failure. I'm using
> DPDK 2.1.
>
> [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log
> match nodes/bytes used: 816/104448
> total: 12940832 bytes
> ACL: Build phase for ACL "ipv4_acl_table2":
> memory consumed: 947913495
> ACL: trie 0: number of rules: 816
> ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 failed
> ACL: Build phase for ACL "ipv4_acl_table1":
> memory consumed: 947913495
> ACL: trie 0: number of rules: 817
> EAL: Error - exiting with code: 1
> Cause: Failed to build ACL trie
>
> Again sourced the ACL config file. After adding around 77 again the same error came.
>
> total: 14912784 bytes
> ACL: Build phase for ACL "ipv4_acl_table1":
> memory consumed: 1040188260
> ACL: trie 0: number of rules: 893
> ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2 failed
You are running out of hugepages memory.
> ACL: Build phase for ACL "ipv4_acl_table2":
> memory consumed: 1040188260
> ACL: trie 0: number of rules: 894
> EAL: Error - exiting with code: 1
> Cause: Failed to build ACL trie
>
> Where to increase the memory to avoid this issue?
Refer to:
http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applications
Section 2.3.2
Konstantin
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-dev] ACL memory allocation failures
@ 2016-02-26 10:28 Rapelly, Varun
2016-02-26 10:53 ` Ananyev, Konstantin
0 siblings, 1 reply; 7+ messages in thread
From: Rapelly, Varun @ 2016-02-26 10:28 UTC (permalink / raw)
To: dev
Hi All,
When I'm trying to configure some 5000+ ACL rules with different source IP addresses, getting ACL memory allocation failure. I'm using DPDK 2.1.
[root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log
match nodes/bytes used: 816/104448
total: 12940832 bytes
ACL: Build phase for ACL "ipv4_acl_table2":
memory consumed: 947913495
ACL: trie 0: number of rules: 816
ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 failed
ACL: Build phase for ACL "ipv4_acl_table1":
memory consumed: 947913495
ACL: trie 0: number of rules: 817
EAL: Error - exiting with code: 1
Cause: Failed to build ACL trie
Again sourced the ACL config file. After adding around 77 again the same error came.
total: 14912784 bytes
ACL: Build phase for ACL "ipv4_acl_table1":
memory consumed: 1040188260
ACL: trie 0: number of rules: 893
ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2 failed
ACL: Build phase for ACL "ipv4_acl_table2":
memory consumed: 1040188260
ACL: trie 0: number of rules: 894
EAL: Error - exiting with code: 1
Cause: Failed to build ACL trie
Where to increase the memory to avoid this issue?
Regards,
Varun
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2016-02-29 12:06 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-02-26 10:27 [dpdk-dev] ACL memory allocation failures Rapelly, Varun
2016-02-26 10:28 Rapelly, Varun
2016-02-26 10:53 ` Ananyev, Konstantin
2016-02-26 11:55 ` Rapelly, Varun
2016-02-26 12:39 ` Ananyev, Konstantin
2016-02-29 5:00 ` Rapelly, Varun
2016-02-29 12:06 ` Rapelly, Varun
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).