rte_pktmbuf_free_bulk vs rte_pktmbuf_free

rte_pktmbuf_free_bulk vs rte_pktmbuf_free

[Filip Janiszewski]

Hi,

Is there any specific reason why using rte_pktmbuf_free_bulk seems to be
much slower than rte_pktmbuf_free in a loop? (DPDK 21.11)

I ran a bunch of tests on a 50GbE link where I'm getting packet drops
(running with too few RX cores on purpose, to make some performance
verification) and when the time comes to release the packets, i did a
quick change like this:

.
            //rte_pktmbuf_free_bulk( data, pkt_cnt );
            for( idx = 0 ; idx < pkt_cnt ; ++idx ) {
                rte_pktmbuf_free( data[ idx ] );
            }
.

And suddenly I'm dropping around 10% less packets (The traffic rate is
around ~95Mpps). In case that's relevant, RX from the nic is done on a
separate core than where the pkts are released (processed and released)

I did also the following experiment: Found the MPPs speed value where i
get around 2-5% drops using rte_pktmbuf_free_bulk, executed a bunch of
readings where I consistently get drops.. Then switched to the loop with
rte_pktmbuf_free and executed the same tests again, of a sudden I can't
drop anymore.

Isn't this strange? I was sure rte_pktmbuf_free_bulk would be kind of
optimized for bulk releases so people don't have to loop themselves.

Thanks

-- 
BR, Filip
+48 666 369 823

Re: rte_pktmbuf_free_bulk vs rte_pktmbuf_free

[Stephen Hemminger]

On Tue, 11 Jan 2022 13:12:24 +0100
Filip Janiszewski <contact@filipjaniszewski.com> wrote:

> Hi,
> 
> Is there any specific reason why using rte_pktmbuf_free_bulk seems to be
> much slower than rte_pktmbuf_free in a loop? (DPDK 21.11)
> 
> I ran a bunch of tests on a 50GbE link where I'm getting packet drops
> (running with too few RX cores on purpose, to make some performance
> verification) and when the time comes to release the packets, i did a
> quick change like this:
> 
> .
>             //rte_pktmbuf_free_bulk( data, pkt_cnt );
>             for( idx = 0 ; idx < pkt_cnt ; ++idx ) {
>                 rte_pktmbuf_free( data[ idx ] );
>             }
> .
> 
> And suddenly I'm dropping around 10% less packets (The traffic rate is
> around ~95Mpps). In case that's relevant, RX from the nic is done on a
> separate core than where the pkts are released (processed and released)
> 
> I did also the following experiment: Found the MPPs speed value where i
> get around 2-5% drops using rte_pktmbuf_free_bulk, executed a bunch of
> readings where I consistently get drops.. Then switched to the loop with
> rte_pktmbuf_free and executed the same tests again, of a sudden I can't
> drop anymore.
> 
> Isn't this strange? I was sure rte_pktmbuf_free_bulk would be kind of
> optimized for bulk releases so people don't have to loop themselves.
> 
> Thanks
> 

Is your mbuf pool close to exhausted? How big is your bulk size?
It might be with that with larger bulk sizes, the loop is giving packets
back that instantly get consumed by incoming packets. So either pool is almost
empty or the non-bulk is keeping packets in cache more.

Re: rte_pktmbuf_free_bulk vs rte_pktmbuf_free

[Filip Janiszewski]

Il 1/11/22 7:02 PM, Stephen Hemminger ha scritto:
> On Tue, 11 Jan 2022 13:12:24 +0100
> Filip Janiszewski <contact@filipjaniszewski.com> wrote:
> 
>> Hi,
>>
>> Is there any specific reason why using rte_pktmbuf_free_bulk seems to be
>> much slower than rte_pktmbuf_free in a loop? (DPDK 21.11)
>>
>> I ran a bunch of tests on a 50GbE link where I'm getting packet drops
>> (running with too few RX cores on purpose, to make some performance
>> verification) and when the time comes to release the packets, i did a
>> quick change like this:
>>
>> .
>>             //rte_pktmbuf_free_bulk( data, pkt_cnt );
>>             for( idx = 0 ; idx < pkt_cnt ; ++idx ) {
>>                 rte_pktmbuf_free( data[ idx ] );
>>             }
>> .
>>
>> And suddenly I'm dropping around 10% less packets (The traffic rate is
>> around ~95Mpps). In case that's relevant, RX from the nic is done on a
>> separate core than where the pkts are released (processed and released)
>>
>> I did also the following experiment: Found the MPPs speed value where i
>> get around 2-5% drops using rte_pktmbuf_free_bulk, executed a bunch of
>> readings where I consistently get drops.. Then switched to the loop with
>> rte_pktmbuf_free and executed the same tests again, of a sudden I can't
>> drop anymore.
>>
>> Isn't this strange? I was sure rte_pktmbuf_free_bulk would be kind of
>> optimized for bulk releases so people don't have to loop themselves.
>>
>> Thanks
>>
> 
> Is your mbuf pool close to exhausted? How big is your bulk size?
> It might be with that with larger bulk sizes, the loop is giving packets
> back that instantly get consumed by incoming packets. So either pool is almost
> empty or the non-bulk is keeping packets in cache more.
> 

Well, yes, once it starts dropping the buffer is full, but for quite a
while before that event the mem usage is pretty low.

In fact, I've added few diagnostics and here they are for the
rte_pktmbuf_free_bulk test:

.
Mem usage: 0.244141%, captured 0 pkts
Mem usage: 0.244141%, captured 0 pkts
Mem usage: 0.241852%, captured 11,681,034 pkts
Mem usage: 0.243807%, captured 44,327,015 pkts
Mem usage: 0.243855%, captured 78,834,947 pkts
Mem usage: 0.243235%, captured 113,343,787 pkts
Mem usage: 0.246191%, captured 147,867,507 pkts
Mem usage: 0.264502%, captured 182,367,926 pkts
Mem usage: 0.244856%, captured 216,917,982 pkts
Mem usage: 0.248837%, captured 251,445,720 pkts
Mem usage: 0.257087%, captured 285,985,575 pkts
Mem usage: 0.338078%, captured 320,509,279 pkts
Mem usage: 0.362778%, captured 355,016,693 pkts
Mem usage: 0.692415%, captured 389,521,441 pkts
Mem usage: 52.050495%, captured 424,066,179 pkts
Mem usage: 99.960041%, captured 456,936,573 pkts // DROPPING FROM HERE
Mem usage: 99.962330%, captured 485,568,660 pkts
Mem usage: 0.241208%, captured 491,178,294 pkts
Mem usage: 0.241208%, captured 491,178,294 pkts
.

The % value is for the pool usage, it's a 8M items pool. As you can see
all of a sudden it sharply gets exhausted and it never recover (the test
stops at 500M packets.). Please note the prints have a 500ms interval.

Attempting the same test with 1 billion packets lead to a similar
result, the pool is exhausted after a while and there are plenty of drops:

.
Mem usage: 0.244141%, captured 0 pkts
Mem usage: 0.244141%, captured 0 pkts
Mem usage: 0.242686%, captured 1,994,944 pkts
Mem usage: 0.243521%, captured 23,094,546 pkts
Mem usage: 0.350094%, captured 57,594,139 pkts
Mem usage: 0.245333%, captured 92,103,632 pkts
Mem usage: 0.243330%, captured 126,616,534 pkts
Mem usage: 0.244308%, captured 161,136,760 pkts
Mem usage: 0.244093%, captured 195,633,863 pkts
Mem usage: 0.245523%, captured 230,149,916 pkts
Mem usage: 0.249910%, captured 264,648,839 pkts
Mem usage: 0.258422%, captured 299,165,901 pkts
Mem usage: 0.301266%, captured 333,678,228 pkts
Mem usage: 0.425720%, captured 368,197,372 pkts
Mem usage: 0.542426%, captured 402,699,822 pkts
Mem usage: 21.447337%, captured 437,244,879 pkts
Mem usage: 86.296201%, captured 471,804,014 pkts
Mem usage: 99.958158%, captured 501,730,958 pkts // DROPPING FROM HERE
Mem usage: 99.954629%, captured 529,462,253 pkts
Mem usage: 99.958587%, captured 556,391,644 pkts
Mem usage: 99.932027%, captured 582,999,427 pkts
Mem usage: 99.959493%, captured 609,456,194 pkts
Mem usage: 99.959779%, captured 635,641,696 pkts
Mem usage: 99.958920%, captured 661,792,475 pkts
Mem usage: 99.954844%, captured 687,919,194 pkts
Mem usage: 99.957728%, captured 713,992,293 pkts
Mem usage: 99.960685%, captured 740,042,732 pkts
Mem usage: 99.956965%, captured 766,240,304 pkts
Mem usage: 99.960780%, captured 792,423,477 pkts
Mem usage: 99.960351%, captured 818,629,881 pkts
Mem usage: 99.959016%, captured 844,904,955 pkts
Mem usage: 99.960637%, captured 871,162,327 pkts
Mem usage: 0.241995%, captured 878,826,100 pkts
Mem usage: 0.241995%, captured 878,826,100 pkts
.

I can fix the issue switching from rte_pktmbuf_free_bulk to
rte_pktmbuf_free in a loop (Not dropping at all, no matter how many
packets I capture)..

I would like to understand this issue better, also I'm a little confused
on why after a while the performance degrade, you're suggesting there's
some packet cache going on, can you elaborate on that?

Thanks

-- 
BR, Filip
+48 666 369 823