Re: [dpdk-dev] ACL memory allocation failures

Re: [dpdk-dev] ACL memory allocation failures

[Rapelly, Varun]

Hi All,

When I'm trying to configure some 5000+ ACL rules with different source IP addresses, getting ACL memory allocation failure. I'm using DPDK 2.1.

[root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log
match nodes/bytes used: 816/104448
total: 12940832 bytes
ACL: Build phase for ACL "ipv4_acl_table2":
memory consumed: 947913495
ACL: trie 0: number of rules: 816
ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 failed
ACL: Build phase for ACL "ipv4_acl_table1":
memory consumed: 947913495
ACL: trie 0: number of rules: 817
EAL: Error - exiting with code: 1
  Cause: Failed to build ACL trie

Again sourced the ACL config file. After adding around 77 again the same error came.

total: 14912784 bytes
ACL: Build phase for ACL "ipv4_acl_table1":
memory consumed: 1040188260
ACL: trie 0: number of rules: 893
ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2 failed
ACL: Build phase for ACL "ipv4_acl_table2":
memory consumed: 1040188260
ACL: trie 0: number of rules: 894
EAL: Error - exiting with code: 1
  Cause: Failed to build ACL trie

Where to increase the memory to avoid this issue?

Regards,
Varun

Re: [dpdk-dev] ACL memory allocation failures

[Ananyev, Konstantin]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun
> Sent: Friday, February 26, 2016 10:28 AM
> To: dev@dpdk.org
> Subject: Re: [dpdk-dev] ACL memory allocation failures
> 
> Hi All,
> 
> When I'm trying to configure some 5000+ ACL rules with different source IP addresses, getting ACL memory allocation failure. I'm using
> DPDK 2.1.
> 
> [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log
> match nodes/bytes used: 816/104448
> total: 12940832 bytes
> ACL: Build phase for ACL "ipv4_acl_table2":
> memory consumed: 947913495
> ACL: trie 0: number of rules: 816
> ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 failed
> ACL: Build phase for ACL "ipv4_acl_table1":
> memory consumed: 947913495
> ACL: trie 0: number of rules: 817
> EAL: Error - exiting with code: 1
>   Cause: Failed to build ACL trie
> 
> Again sourced the ACL config file. After adding around 77 again the same error came.
> 
> total: 14912784 bytes
> ACL: Build phase for ACL "ipv4_acl_table1":
> memory consumed: 1040188260
> ACL: trie 0: number of rules: 893
> ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2 failed

You are running out of hugepages memory.

> ACL: Build phase for ACL "ipv4_acl_table2":
> memory consumed: 1040188260
> ACL: trie 0: number of rules: 894
> EAL: Error - exiting with code: 1
>   Cause: Failed to build ACL trie
> 
> Where to increase the memory to avoid this issue?

 Refer to:
http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applications
Section 2.3.2 

Konstantin

Re: [dpdk-dev] ACL memory allocation failures

[Rapelly, Varun]

Thanks Konstantin.

Previous allocation error was coming with 1024 huge pages of 2 MB size. 

After increasing the huge pages to 2048, I was able to add another ~140 rules [IPv4 rule data--> with src, dst IP address & port, next header ] more, ie., 950 rules were added.

Logically it did not increase number of rules [expected 2*817, but only 950 were added]. Is it really using huge pages memory only? 

>From the code it looks like heap memory. [ ret = malloc_heap_alloc(&mcfg->malloc_heaps[i], type, size, 0, align == 0 ? 1 : align, 0) ]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun
> Sent: Friday, February 26, 2016 10:28 AM
> To: dev@dpdk.org
> Subject: Re: [dpdk-dev] ACL memory allocation failures
> 
> Hi All,
> 
> When I'm trying to configure some 5000+ ACL rules with different 
> source IP addresses, getting ACL memory allocation failure. I'm using DPDK 2.1.
> 
> [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log match nodes/bytes 
> used: 816/104448
> total: 12940832 bytes
> ACL: Build phase for ACL "ipv4_acl_table2":
> memory consumed: 947913495
> ACL: trie 0: number of rules: 816
> ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 
> failed
> ACL: Build phase for ACL "ipv4_acl_table1":
> memory consumed: 947913495
> ACL: trie 0: number of rules: 817
> EAL: Error - exiting with code: 1
>   Cause: Failed to build ACL trie
> 
> Again sourced the ACL config file. After adding around 77 again the same error came.
> 
> total: 14912784 bytes
> ACL: Build phase for ACL "ipv4_acl_table1":
> memory consumed: 1040188260
> ACL: trie 0: number of rules: 893
> ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2 
> failed

You are running out of hugepages memory.

> ACL: Build phase for ACL "ipv4_acl_table2":
> memory consumed: 1040188260
> ACL: trie 0: number of rules: 894
> EAL: Error - exiting with code: 1
>   Cause: Failed to build ACL trie
> 
> Where to increase the memory to avoid this issue?

 Refer to:
http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applications
Section 2.3.2 

Konstantin

Re: [dpdk-dev] ACL memory allocation failures

[Ananyev, Konstantin]

> 
> Thanks Konstantin.
> 
> Previous allocation error was coming with 1024 huge pages of 2 MB size.
> 
> After increasing the huge pages to 2048, I was able to add another ~140 rules [IPv4 rule data--> with src, dst IP address & port, next
> header ] more, ie., 950 rules were added.

That's strange according to your log, all you need is ~13MB of hugepage memory:
ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1
Wonder what consumed rest of 4GB?
Again do you re-build your table after every rule you add?
If so, then it seems a bit strange approach (and definitely not the fastest one).
What you can do instead: create context; add all your rules into it; build; 

> 
> Logically it did not increase number of rules [expected 2*817, but only 950 were added]. Is it really using huge pages memory only?
> 
> From the code it looks like heap memory. [ ret = malloc_heap_alloc(&mcfg->malloc_heaps[i], type, size, 0, align == 0 ? 1 : align, 0) ]

As I can see from the log it fails at GEN phase, when trying to allocate hugepages for RT table.
At lib/librte_acl/acl_gen.c:509

rte_acl_gen(struct rte_acl_ctx *ctx, struct rte_acl_trie *trie,
        struct rte_acl_bld_trie *node_bld_trie, uint32_t num_tries,
        uint32_t num_categories, uint32_t data_index_sz, size_t max_size)
{
...
mem = rte_zmalloc_socket(ctx->name, total_size, RTE_CACHE_LINE_SIZE,
                        ctx->socket_id);
if (mem == NULL) {
                RTE_LOG(ERR, ACL,
                        "allocation of %zu bytes on socket %d for %s failed\n",
                        total_size, ctx->socket_id, ctx->name);
                return -ENOMEM;
}

Konstantin

> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun
> > Sent: Friday, February 26, 2016 10:28 AM
> > To: dev@dpdk.org
> > Subject: Re: [dpdk-dev] ACL memory allocation failures
> >
> > Hi All,
> >
> > When I'm trying to configure some 5000+ ACL rules with different
> > source IP addresses, getting ACL memory allocation failure. I'm using DPDK 2.1.
> >
> > [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log match nodes/bytes
> > used: 816/104448
> > total: 12940832 bytes
> > ACL: Build phase for ACL "ipv4_acl_table2":
> > memory consumed: 947913495
> > ACL: trie 0: number of rules: 816
> > ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1
> > failed
> > ACL: Build phase for ACL "ipv4_acl_table1":
> > memory consumed: 947913495
> > ACL: trie 0: number of rules: 817
> > EAL: Error - exiting with code: 1
> >   Cause: Failed to build ACL trie
> >
> > Again sourced the ACL config file. After adding around 77 again the same error came.
> >
> > total: 14912784 bytes
> > ACL: Build phase for ACL "ipv4_acl_table1":
> > memory consumed: 1040188260
> > ACL: trie 0: number of rules: 893
> > ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2
> > failed
> 
> You are running out of hugepages memory.
> 
> > ACL: Build phase for ACL "ipv4_acl_table2":
> > memory consumed: 1040188260
> > ACL: trie 0: number of rules: 894
> > EAL: Error - exiting with code: 1
> >   Cause: Failed to build ACL trie
> >
> > Where to increase the memory to avoid this issue?
> 
>  Refer to:
> http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applications
> Section 2.3.2
> 
> Konstantin

Re: [dpdk-dev] ACL memory allocation failures

[Rapelly, Varun]

> 
> Thanks Konstantin.
> 
> Previous allocation error was coming with 1024 huge pages of 2 MB size.
> 
> After increasing the huge pages to 2048, I was able to add another 
> ~140 rules [IPv4 rule data--> with src, dst IP address & port, next header ] more, ie., 950 rules were added.

That's strange according to your log, all you need is ~13MB of hugepage memory:
ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 Wonder what consumed rest of 4GB?


>> We are creating mem pools (for DPDK compatible 3 ports) for packet processing.

Again do you re-build your table after every rule you add?
If so, then it seems a bit strange approach (and definitely not the fastest one).
>>Yes, we are rebuilding the rules every time and is due to 2 reasons: 
>>1. Our application, gives full list of rules every time you add new rule. 
>>2. There is no way to delete a specific rule in the trie. Is there any way to delete a specific ACL rule?
What you can do instead: create context; add all your rules into it; build; 

> 
> Logically it did not increase number of rules [expected 2*817, but only 950 were added]. Is it really using huge pages memory only?
> 
> From the code it looks like heap memory. [ ret = 
> malloc_heap_alloc(&mcfg->malloc_heaps[i], type, size, 0, align == 0 ? 
> 1 : align, 0) ]

As I can see from the log it fails at GEN phase, when trying to allocate hugepages for RT table.
At lib/librte_acl/acl_gen.c:509

rte_acl_gen(struct rte_acl_ctx *ctx, struct rte_acl_trie *trie,
        struct rte_acl_bld_trie *node_bld_trie, uint32_t num_tries,
        uint32_t num_categories, uint32_t data_index_sz, size_t max_size) { ...
mem = rte_zmalloc_socket(ctx->name, total_size, RTE_CACHE_LINE_SIZE,
                        ctx->socket_id); if (mem == NULL) {
                RTE_LOG(ERR, ACL,
                        "allocation of %zu bytes on socket %d for %s failed\n",
                        total_size, ctx->socket_id, ctx->name);
                return -ENOMEM;
}

Konstantin

> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun
> > Sent: Friday, February 26, 2016 10:28 AM
> > To: dev@dpdk.org
> > Subject: Re: [dpdk-dev] ACL memory allocation failures
> >
> > Hi All,
> >
> > When I'm trying to configure some 5000+ ACL rules with different 
> > source IP addresses, getting ACL memory allocation failure. I'm using DPDK 2.1.
> >
> > [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log match 
> > nodes/bytes
> > used: 816/104448
> > total: 12940832 bytes
> > ACL: Build phase for ACL "ipv4_acl_table2":
> > memory consumed: 947913495
> > ACL: trie 0: number of rules: 816
> > ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 
> > failed
> > ACL: Build phase for ACL "ipv4_acl_table1":
> > memory consumed: 947913495
> > ACL: trie 0: number of rules: 817
> > EAL: Error - exiting with code: 1
> >   Cause: Failed to build ACL trie
> >
> > Again sourced the ACL config file. After adding around 77 again the same error came.
> >
> > total: 14912784 bytes
> > ACL: Build phase for ACL "ipv4_acl_table1":
> > memory consumed: 1040188260
> > ACL: trie 0: number of rules: 893
> > ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2 
> > failed
> 
> You are running out of hugepages memory.
> 
> > ACL: Build phase for ACL "ipv4_acl_table2":
> > memory consumed: 1040188260
> > ACL: trie 0: number of rules: 894
> > EAL: Error - exiting with code: 1
> >   Cause: Failed to build ACL trie
> >
> > Where to increase the memory to avoid this issue?
> 
>  Refer to:
> http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applic
> ations
> Section 2.3.2
> 
> Konstantin

Re: [dpdk-dev] ACL memory allocation failures

[Rapelly, Varun]

Thanks Konstantin. Few more questions in line:

> 
> Previous allocation error was coming with 1024 huge pages of 2 MB size.
> 
> After increasing the huge pages to 2048, I was able to add another
> ~140 rules [IPv4 rule data--> with src, dst IP address & port, next header ] more, ie., 950 rules were added.

That's strange according to your log, all you need is ~13MB of hugepage memory:
ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 Wonder what consumed rest of 4GB?
>> We are creating mem pools (for DPDK compatible 3 ports) for packet processing.
>>> And there are no free huge pages available after our DPDK app initialization.

Again do you re-build your table after every rule you add?
If so, then it seems a bit strange approach (and definitely not the fastest one).
>>Yes, we are rebuilding the rules every time and is due to 2 reasons: 
>>1. Our application, gives full list of rules every time you add new rule. 
>>2. There is no way to delete a specific rule in the trie. Is there any way to delete a specific ACL rule?


What you can do instead: create context; add all your rules into it; build; 
>>> By following the same approach (what I explained above, rebuilding the ACL trie everytime), can we fix this memory allocation issue?
>>>If yes, please provide me some pointers to modify the code.

> 
> Logically it did not increase number of rules [expected 2*817, but only 950 were added]. Is it really using huge pages memory only?
> 
> From the code it looks like heap memory. [ ret = 
> malloc_heap_alloc(&mcfg->malloc_heaps[i], type, size, 0, align == 0 ?
> 1 : align, 0) ]

As I can see from the log it fails at GEN phase, when trying to allocate hugepages for RT table.
At lib/librte_acl/acl_gen.c:509

rte_acl_gen(struct rte_acl_ctx *ctx, struct rte_acl_trie *trie,
        struct rte_acl_bld_trie *node_bld_trie, uint32_t num_tries,
        uint32_t num_categories, uint32_t data_index_sz, size_t max_size) { ...
mem = rte_zmalloc_socket(ctx->name, total_size, RTE_CACHE_LINE_SIZE,
                        ctx->socket_id); if (mem == NULL) {
                RTE_LOG(ERR, ACL,
                        "allocation of %zu bytes on socket %d for %s failed\n",
                        total_size, ctx->socket_id, ctx->name);
                return -ENOMEM;
}
>>> Is there any way to reserve some particular amount of huge page memory for ACL trie (in eal_init())?

Konstantin

> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rapelly, Varun
> > Sent: Friday, February 26, 2016 10:28 AM
> > To: dev@dpdk.org
> > Subject: Re: [dpdk-dev] ACL memory allocation failures
> >
> > Hi All,
> >
> > When I'm trying to configure some 5000+ ACL rules with different 
> > source IP addresses, getting ACL memory allocation failure. I'm using DPDK 2.1.
> >
> > [root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log match 
> > nodes/bytes
> > used: 816/104448
> > total: 12940832 bytes
> > ACL: Build phase for ACL "ipv4_acl_table2":
> > memory consumed: 947913495
> > ACL: trie 0: number of rules: 816
> > ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 
> > failed
> > ACL: Build phase for ACL "ipv4_acl_table1":
> > memory consumed: 947913495
> > ACL: trie 0: number of rules: 817
> > EAL: Error - exiting with code: 1
> >   Cause: Failed to build ACL trie
> >
> > Again sourced the ACL config file. After adding around 77 again the same error came.
> >
> > total: 14912784 bytes
> > ACL: Build phase for ACL "ipv4_acl_table1":
> > memory consumed: 1040188260
> > ACL: trie 0: number of rules: 893
> > ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2 
> > failed
> 
> You are running out of hugepages memory.
> 
> > ACL: Build phase for ACL "ipv4_acl_table2":
> > memory consumed: 1040188260
> > ACL: trie 0: number of rules: 894
> > EAL: Error - exiting with code: 1
> >   Cause: Failed to build ACL trie
> >
> > Where to increase the memory to avoid this issue?
> 
>  Refer to:
> http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html#running-dpdk-applic
> ations
> Section 2.3.2
> 
> Konstantin

[dpdk-dev] ACL memory allocation failures

[Rapelly, Varun]

Hi All,

When I'm trying to configure some 5000+ ACL rules with different source IP addresses, getting ACL memory allocation failure.\

[root@ACLISSUE log_2015_10_26_08_19_42]# vim np.log
match nodes/bytes used: 816/104448
total: 12940832 bytes
ACL: Build phase for ACL "ipv4_acl_table2":
memory consumed: 947913495
ACL: trie 0: number of rules: 816
ACL: allocation of 12966528 bytes on socket 0 for ipv4_acl_table1 failed
ACL: Build phase for ACL "ipv4_acl_table1":
memory consumed: 947913495
ACL: trie 0: number of rules: 817
EAL: Error - exiting with code: 1
  Cause: Failed to build ACL trie

Again sourced the ACL config file. After adding around 77 again the same error came.

total: 14912784 bytes
ACL: Build phase for ACL "ipv4_acl_table1":
memory consumed: 1040188260
ACL: trie 0: number of rules: 893
ACL: allocation of 14938480 bytes on socket 0 for ipv4_acl_table2 failed
ACL: Build phase for ACL "ipv4_acl_table2":
memory consumed: 1040188260
ACL: trie 0: number of rules: 894
EAL: Error - exiting with code: 1
  Cause: Failed to build ACL trie

Where to increase the memory to avoid this issue?

Regards,
Varun