[dpdk-users] mbuf free cnt not decreasing

[dpdk-users] mbuf free cnt not decreasing

[Javier Coleto Fernández]

Hello,

I've developed an application using DPDK to make forwarding and
encapsulation/decapsulation between two end nodes.
This application uses two eth interfaces and creates two KNI interfaces to
forward the packets:
- the encapsulated ones through the eth interfaces
- the not encapsulated (standard packets) through the KNI interfaces
The problem I'm facing is my mbuf free count (as given by
rte_mempool_free_count()) keeps increasing and tops at ~8130. Being 8192
the size of the mempool, that stops the packet processing and my
application stops forwarding.
Any clues about what I could do to liberate the free mbufs back to the
mempool?

Regards,
Javier

Re: [dpdk-users] mbuf free cnt not decreasing

[Andriy Berestovskyy]

Hi Javier,
Please see below.

On Fri, Apr 15, 2016 at 10:34 AM, Javier Coleto Fernández
<javicoleto44@gmail.com> wrote:
> The problem I'm facing is my mbuf free count (as given by
> rte_mempool_free_count()) keeps increasing and tops at ~8130. Being 8192
> the size of the mempool, that stops the packet processing and my
> application stops forwarding.

Try rte_ring_count(mempool->ring) instead of rte_mempool_free_count().
The latter adds cached mbufs, hence might be the difference 8192 vs
8130.

> Any clues about what I could do to liberate the free mbufs back to the
> mempool?

Try to increase the number of mbufs in the mempool. There might be two cases:
1. For whatever mempool size you always lack mbufs -> it is an mbuf
leak, so you have to find and fix it.

2. Number of free mbufs stabilizes at some point -> you could either
increase the mempool size to accommodate that usage or reduce the
caches/descriptors/thresholds.

Andriy

Re: [dpdk-users] mbuf free cnt not decreasing

[Javier Coleto Fernández]

Hi Andriy,

Thank you for your answer.
I've tried what you said and determined it's an mbuf leak, because the
number of free mbufs doesn't stabilize, even when I increase the number of
mbufs in the mempool.
Is there any way to easily debug this mbuf leak? I didn't make the full
code, so I don't really know where to start looking for leaks.

Regards,
Javier

Re: [dpdk-users] mbuf free cnt not decreasing

[Andriy Berestovskyy]

Hi Javier,
Please make sure you check the return value of rte_kni_tx_burst() and
rte_eth_tx_burst() and free the unsent mbufs. Might be something
else...

Overall the technique should be the same as with memory leaks: debug
logs and patience ;)

Regards,
Andriy

On Mon, Apr 18, 2016 at 10:41 AM, Javier Coleto Fernández
<javicoleto44@gmail.com> wrote:
> Hi Andriy,
>
> Thank you for your answer.
> I've tried what you said and determined it's an mbuf leak, because the
> number of free mbufs doesn't stabilize, even when I increase the number of
> mbufs in the mempool.
> Is there any way to easily debug this mbuf leak? I didn't make the full
> code, so I don't really know where to start looking for leaks.
>
> Regards,
> Javier



-- 
Andriy Berestovskyy

Re: [dpdk-users] mbuf free cnt not decreasing

[Wiles, Keith]

>Hi Javier,
>Please make sure you check the return value of rte_kni_tx_burst() and
>rte_eth_tx_burst() and free the unsent mbufs. Might be something
>else...
>
>Overall the technique should be the same as with memory leaks: debug
>logs and patience ;)

The rte_eth_tx_burst() and other APIs will not handle the case where the ring below fills up. This means if you send 32 packets it may only send part of the packets in the and if you do not check the return value you could be losing the packets here. You need to handle the case were the number of TX sent are not send because of rings filling up or some other reason.
>
>Regards,
>Andriy
>
>On Mon, Apr 18, 2016 at 10:41 AM, Javier Coleto Fernández
><javicoleto44@gmail.com> wrote:
>> Hi Andriy,
>>
>> Thank you for your answer.
>> I've tried what you said and determined it's an mbuf leak, because the
>> number of free mbufs doesn't stabilize, even when I increase the number of
>> mbufs in the mempool.
>> Is there any way to easily debug this mbuf leak? I didn't make the full
>> code, so I don't really know where to start looking for leaks.
>>
>> Regards,
>> Javier
>
>
>
>-- 
>Andriy Berestovskyy
>


Regards,
Keith

Re: [dpdk-users] mbuf free cnt not decreasing

[Javier Coleto Fernández]

I've checked every return value for the rte_eth_tx_burst() and it's always
the same as the 'n' I provided, so I assume it's sending all the packets i
put in the ring.
Basing on what you say, is that return value supposed to be less than 'n'
in case the ring is filled up or do I have to check the ring size before
calling rte_eth_tx_burst()?

Regards,
Javier

Re: [dpdk-users] mbuf free cnt not decreasing

[Andriy Berestovskyy]

On Mon, Apr 18, 2016 at 5:34 PM, Javier Coleto Fernández
<javicoleto44@gmail.com> wrote:
> Basing on what you say, is that return value supposed to be less than 'n' in
> case the ring is filled up or do I have to check the ring size before
> calling rte_eth_tx_burst()?

You just have to check the return value and free the unsent mbufs.
Here is an example:

ret = rte_eth_tx_burst(port, queueid, m_table, n);
if (unlikely(ret < n)) {
    do {
        rte_pktmbuf_free(m_table[ret]);
    } while (++ret < n);
}


Regards,
Andiry

Re: [dpdk-users] mbuf free cnt not decreasing

[Javier Coleto Fernández]

This is exactly what I'm doing at the moment, but the free count (both the
rte_mempool_free_count() and rte_ring_free_count()) keeps increasing
nonetheless.

Regards,
Javier

2016-04-18 17:57 GMT+02:00 Andriy Berestovskyy <aber@semihalf.com>:

> On Mon, Apr 18, 2016 at 5:34 PM, Javier Coleto Fernández
> <javicoleto44@gmail.com> wrote:
> > Basing on what you say, is that return value supposed to be less than
> 'n' in
> > case the ring is filled up or do I have to check the ring size before
> > calling rte_eth_tx_burst()?
>
> You just have to check the return value and free the unsent mbufs.
> Here is an example:
>
> ret = rte_eth_tx_burst(port, queueid, m_table, n);
> if (unlikely(ret < n)) {
>     do {
>         rte_pktmbuf_free(m_table[ret]);
>     } while (++ret < n);
> }
>
>
> Regards,
> Andiry
>

Re: [dpdk-users] mbuf free cnt not decreasing

[Andriy Berestovskyy]

Javier,
That was just our guess, you might have the leak somewhere else...

Andriy

On Mon, Apr 18, 2016 at 6:01 PM, Javier Coleto Fernández
<javicoleto44@gmail.com> wrote:
> This is exactly what I'm doing at the moment, but the free count (both the
> rte_mempool_free_count() and rte_ring_free_count()) keeps increasing
> nonetheless.
>
> Regards,
> Javier
>
> 2016-04-18 17:57 GMT+02:00 Andriy Berestovskyy <aber@semihalf.com>:
>>
>> On Mon, Apr 18, 2016 at 5:34 PM, Javier Coleto Fernández
>> <javicoleto44@gmail.com> wrote:
>> > Basing on what you say, is that return value supposed to be less than
>> > 'n' in
>> > case the ring is filled up or do I have to check the ring size before
>> > calling rte_eth_tx_burst()?
>>
>> You just have to check the return value and free the unsent mbufs.
>> Here is an example:
>>
>> ret = rte_eth_tx_burst(port, queueid, m_table, n);
>> if (unlikely(ret < n)) {
>>     do {
>>         rte_pktmbuf_free(m_table[ret]);
>>     } while (++ret < n);
>> }

Re: [dpdk-users] mbuf free cnt not decreasing

[Wiles, Keith]

>Javier,
>That was just our guess, you might have the leak somewhere else...
>
>Andriy
>
>On Mon, Apr 18, 2016 at 6:01 PM, Javier Coleto Fernández
><javicoleto44@gmail.com> wrote:
>> This is exactly what I'm doing at the moment, but the free count (both the
>> rte_mempool_free_count() and rte_ring_free_count()) keeps increasing
>> nonetheless.

Check to make sure you are not increasing the reference count in the mbuf, these two are the most common places to lose packets. Also check your loops to make sure you are not breaking out of the packet processing loops too soon and skipping some packets. In one case I have seen someone converted a while loop into a for loop and forgot to remove the i++ some place in the body of the code. Try adding counter on your code to see if you are missing processing or freeing the packets.

I would not expect the DPDK code to be the problem, so focus on your code is all I can tell you.

>>
>> Regards,
>> Javier
>>
>> 2016-04-18 17:57 GMT+02:00 Andriy Berestovskyy <aber@semihalf.com>:
>>>
>>> On Mon, Apr 18, 2016 at 5:34 PM, Javier Coleto Fernández
>>> <javicoleto44@gmail.com> wrote:
>>> > Basing on what you say, is that return value supposed to be less than
>>> > 'n' in
>>> > case the ring is filled up or do I have to check the ring size before
>>> > calling rte_eth_tx_burst()?
>>>
>>> You just have to check the return value and free the unsent mbufs.
>>> Here is an example:
>>>
>>> ret = rte_eth_tx_burst(port, queueid, m_table, n);
>>> if (unlikely(ret < n)) {
>>>     do {
>>>         rte_pktmbuf_free(m_table[ret]);
>>>     } while (++ret < n);
>>> }
>


Regards,
Keith

Re: [dpdk-users] mbuf free cnt not decreasing

[Javier Coleto Fernández]

2016-04-18 18:29 GMT+02:00 Wiles, Keith <keith.wiles@intel.com>:

> >Javier,
> >That was just our guess, you might have the leak somewhere else...
> >
> >Andriy
> >
> >On Mon, Apr 18, 2016 at 6:01 PM, Javier Coleto Fernández
> ><javicoleto44@gmail.com> wrote:
> >> This is exactly what I'm doing at the moment, but the free count (both
> the
> >> rte_mempool_free_count() and rte_ring_free_count()) keeps increasing
> >> nonetheless.
>
> Check to make sure you are not increasing the reference count in the mbuf,
> these two are the most common places to lose packets.


I don't make any call to reference count changing functions (like
rte_mbuf_refcnt_update/set) but, is there any other way this reference
count could end up having a "bad" value?

Also check your loops to make sure you are not breaking out of the packet
> processing loops too soon and skipping some packets. In one case I have
> seen someone converted a while loop into a for loop and forgot to remove
> the i++ some place in the body of the code. Try adding counter on your code
> to see if you are missing processing or freeing the packets.


> I would not expect the DPDK code to be the problem, so focus on your code
> is all I can tell you.
>
> >>
> >> Regards,
> >> Javier
> >>
> >> 2016-04-18 17:57 GMT+02:00 Andriy Berestovskyy <aber@semihalf.com>:
> >>>
> >>> On Mon, Apr 18, 2016 at 5:34 PM, Javier Coleto Fernández
> >>> <javicoleto44@gmail.com> wrote:
> >>> > Basing on what you say, is that return value supposed to be less than
> >>> > 'n' in
> >>> > case the ring is filled up or do I have to check the ring size before
> >>> > calling rte_eth_tx_burst()?
> >>>
> >>> You just have to check the return value and free the unsent mbufs.
> >>> Here is an example:
> >>>
> >>> ret = rte_eth_tx_burst(port, queueid, m_table, n);
> >>> if (unlikely(ret < n)) {
> >>>     do {
> >>>         rte_pktmbuf_free(m_table[ret]);
> >>>     } while (++ret < n);
> >>> }
> >
>
>
> Regards,
> Keith
>

Regards,
Javier

Re: [dpdk-users] mbuf free cnt not decreasing

[Wiles, Keith]

From: Javier Coleto Fernández <javicoleto44@gmail.com<mailto:javicoleto44@gmail.com>>
Date: Monday, April 18, 2016 at 11:41 AM
To: Keith Wiles <keith.wiles@intel.com<mailto:keith.wiles@intel.com>>
Cc: Andriy Berestovskyy <aber@semihalf.com<mailto:aber@semihalf.com>>, users <users@dpdk.org<mailto:users@dpdk.org>>
Subject: Re: [dpdk-users] mbuf free cnt not decreasing


2016-04-18 18:29 GMT+02:00 Wiles, Keith <keith.wiles@intel.com<mailto:keith.wiles@intel.com>>:
>Javier,
>That was just our guess, you might have the leak somewhere else...
>
>Andriy
>
>On Mon, Apr 18, 2016 at 6:01 PM, Javier Coleto Fernández
><javicoleto44@gmail.com<mailto:javicoleto44@gmail.com>> wrote:
>> This is exactly what I'm doing at the moment, but the free count (both the
>> rte_mempool_free_count() and rte_ring_free_count()) keeps increasing
>> nonetheless.

Check to make sure you are not increasing the reference count in the mbuf, these two are the most common places to lose packets.

I don't make any call to reference count changing functions (like rte_mbuf_refcnt_update/set) but, is there any other way this reference count could end up having a "bad" value?

The one way is overwriting the value in the mbuf that I know of.


Also check your loops to make sure you are not breaking out of the packet processing loops too soon and skipping some packets. In one case I have seen someone converted a while loop into a for loop and forgot to remove the i++ some place in the body of the code. Try adding counter on your code to see if you are missing processing or freeing the packets.

I would not expect the DPDK code to be the problem, so focus on your code is all I can tell you.

>>
>> Regards,
>> Javier
>>
>> 2016-04-18 17:57 GMT+02:00 Andriy Berestovskyy <aber@semihalf.com<mailto:aber@semihalf.com>>:
>>>
>>> On Mon, Apr 18, 2016 at 5:34 PM, Javier Coleto Fernández
>>> <javicoleto44@gmail.com<mailto:javicoleto44@gmail.com>> wrote:
>>> > Basing on what you say, is that return value supposed to be less than
>>> > 'n' in
>>> > case the ring is filled up or do I have to check the ring size before
>>> > calling rte_eth_tx_burst()?
>>>
>>> You just have to check the return value and free the unsent mbufs.
>>> Here is an example:
>>>
>>> ret = rte_eth_tx_burst(port, queueid, m_table, n);
>>> if (unlikely(ret < n)) {
>>>     do {
>>>         rte_pktmbuf_free(m_table[ret]);
>>>     } while (++ret < n);
>>> }
>


Regards,
Keith

Regards,
Javier


Regards,
Keith

Re: [dpdk-users] mbuf free cnt not decreasing

[Javier Coleto Fernández]

I don't use any update/set call to modify the reference counter, but I have
a theory about it.
Is it possible the mbufs are being accessed too fast after being set and
just before the reference counter has been updated internally?
I say so because I got a PANIC error after enabling MEMPOOL debugging. It
said 'bad ref cnt' and the dump stack pointed to rte_kni_tx_burst(), so I
thought this could be an explanation for that error.
Any ideas?

Regards,
Javier

Re: [dpdk-users] mbuf free cnt not decreasing

[Ferruh Yigit]

On 4/18/2016 9:41 AM, Javier Coleto Fernández wrote:
> Hi Andriy,
> 
> Thank you for your answer.
> I've tried what you said and determined it's an mbuf leak, because the
> number of free mbufs doesn't stabilize, even when I increase the number of
> mbufs in the mempool.
> Is there any way to easily debug this mbuf leak? I didn't make the full
> code, so I don't really know where to start looking for leaks.
> 

Can you please check free_q?
kni_fifo_free_count(kni->free_q)

If you call rte_kni_tx_burst() for more than 32 packets at a time, mbufs
may stuck in free_q.

Regards,
ferruh