[dpdk-dev] How to fight forwarding performance regression on large mempool sizes.

[dpdk-dev] How to fight forwarding performance regression on large mempool sizes.

[Dmitry Vyal]

Good day everyone,

While working on IP packet defragmenting I had to enlarge mempool size. 
I did this to provide large enough time window for assembling a fragment 
sequence. Unfortunately, I got a performance regression: if I enlarge 
mempool size from 2**12 to 2**20 MBufs, packet performance for not 
fragmented packets drops from ~8.5mpps to ~5.5mpps for single core. I 
made only a single measure, so the data are noisy, but the trend is evident:
SIZE 4096 - 8.47mpps
SIZE 8192 - 8.26mpps
SIZE 16384 - 8.29mpps
SIZE 32768 - 8.31mpps
SIZE 65536 - 8.12mpps
SIZE 131072 - 7.93mpps
SIZE 262144 - 6.22mpps
SIZE 524288 - 5.72mpps
SIZE 1048576 - 5.63mpps

And I need even larger sizes.

I want to ask for an advice, how best to tackle this? One way I'm 
thinking about is to make two mempools, one large for fragments (we may 
accumulate a big number of them) and one small for full packets, we just 
forward them burst by burst. Is it possible to configure RSS to 
distribute packets between queues according to this scheme? Perhaps, 
there are better ways?

Thanks,
Dmitry

Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes.

[Robert Sanford]

Hi Dmitry,

The biggest drop-off seems to be from size 128K to 256K. Are you using 1GB
huge pages already (rather than 2MB)?

I would think that it would not use over 1GB until you ask for 512K mbufs
or more.

--
Regards,
Robert



On Thu, Sep 19, 2013 at 3:50 AM, Dmitry Vyal <dmitryvyal@gmail.com> wrote:

> Good day everyone,
>
> While working on IP packet defragmenting I had to enlarge mempool size. I
> did this to provide large enough time window for assembling a fragment
> sequence. Unfortunately, I got a performance regression: if I enlarge
> mempool size from 2**12 to 2**20 MBufs, packet performance for not
> fragmented packets drops from ~8.5mpps to ~5.5mpps for single core. I made
> only a single measure, so the data are noisy, but the trend is evident:
> SIZE 4096 - 8.47mpps
> SIZE 8192 - 8.26mpps
> SIZE 16384 - 8.29mpps
> SIZE 32768 - 8.31mpps
> SIZE 65536 - 8.12mpps
> SIZE 131072 - 7.93mpps
> SIZE 262144 - 6.22mpps
> SIZE 524288 - 5.72mpps
> SIZE 1048576 - 5.63mpps
>
> And I need even larger sizes.
>
> I want to ask for an advice, how best to tackle this? One way I'm thinking
> about is to make two mempools, one large for fragments (we may accumulate a
> big number of them) and one small for full packets, we just forward them
> burst by burst. Is it possible to configure RSS to distribute packets
> between queues according to this scheme? Perhaps, there are better ways?
>
> Thanks,
> Dmitry
>

Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes.

[Venkatesan, Venky]

Dmitry, 
One other question - what version of DPDK are you doing on?  
-Venky

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Robert Sanford
Sent: Thursday, September 19, 2013 12:40 PM
To: Dmitry Vyal
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes.

Hi Dmitry,

The biggest drop-off seems to be from size 128K to 256K. Are you using 1GB huge pages already (rather than 2MB)?

I would think that it would not use over 1GB until you ask for 512K mbufs or more.

--
Regards,
Robert



On Thu, Sep 19, 2013 at 3:50 AM, Dmitry Vyal <dmitryvyal@gmail.com> wrote:

> Good day everyone,
>
> While working on IP packet defragmenting I had to enlarge mempool 
> size. I did this to provide large enough time window for assembling a 
> fragment sequence. Unfortunately, I got a performance regression: if I 
> enlarge mempool size from 2**12 to 2**20 MBufs, packet performance for 
> not fragmented packets drops from ~8.5mpps to ~5.5mpps for single 
> core. I made only a single measure, so the data are noisy, but the trend is evident:
> SIZE 4096 - 8.47mpps
> SIZE 8192 - 8.26mpps
> SIZE 16384 - 8.29mpps
> SIZE 32768 - 8.31mpps
> SIZE 65536 - 8.12mpps
> SIZE 131072 - 7.93mpps
> SIZE 262144 - 6.22mpps
> SIZE 524288 - 5.72mpps
> SIZE 1048576 - 5.63mpps
>
> And I need even larger sizes.
>
> I want to ask for an advice, how best to tackle this? One way I'm 
> thinking about is to make two mempools, one large for fragments (we 
> may accumulate a big number of them) and one small for full packets, 
> we just forward them burst by burst. Is it possible to configure RSS 
> to distribute packets between queues according to this scheme? Perhaps, there are better ways?
>
> Thanks,
> Dmitry
>

Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes.

[Dmitry Vyal]

On 09/19/2013 11:39 PM, Robert Sanford wrote:
> Hi Dmitry,
>
> The biggest drop-off seems to be from size 128K to 256K. Are you using 
> 1GB huge pages already (rather than 2MB)?
>
> I would think that it would not use over 1GB until you ask for 512K 
> mbufs or more.
>

Hi Robert,

Yes, I've been using 1GB pages for a while. My L3 cache is 20MB and 
mbufs are 2240 bytes of size. So something strange indeed happens then 
we move from ~200MB to ~400MB. Any ideas?

Regards,
Dmitry

Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes.

[Dmitry Vyal]

On 09/19/2013 11:43 PM, Venkatesan, Venky wrote:
> Dmitry,
> One other question - what version of DPDK are you doing on?
> -Venky
>
It's DPDK-1.3.1-7 downloaded from intel.com. Should I try upgrading?

Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes.

[Robert Sanford]

It might be interesting to see that start/end address of the 256K-item (256
* 1024 * 2240 = 560 MB) mbuf memory pool. Maybe it's the first size that
straddles two 1GB pages.

Perhaps you should try a tool that reports cache misses, TLB misses, and
related statistics. I don't know much about this area, yet, but this looks
like a good starting place:

http://software.intel.com/en-us/articles/intel-performance-counter-monitor-a-better-way-to-measure-cpu-utilization

--
Regards,
Robert



On Fri, Sep 20, 2013 at 2:48 AM, Dmitry Vyal <dmitryvyal@gmail.com> wrote:

> On 09/19/2013 11:39 PM, Robert Sanford wrote:
>
>> Hi Dmitry,
>>
>> The biggest drop-off seems to be from size 128K to 256K. Are you using
>> 1GB huge pages already (rather than 2MB)?
>>
>> I would think that it would not use over 1GB until you ask for 512K mbufs
>> or more.
>>
>>
> Hi Robert,
>
> Yes, I've been using 1GB pages for a while. My L3 cache is 20MB and mbufs
> are 2240 bytes of size. So something strange indeed happens then we move
> from ~200MB to ~400MB. Any ideas?
>
> Regards,
> Dmitry
>
>

Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes.

[Robert Sanford]

One more point, if you're not doing this already: Allocate 2^N-1 mbufs, not
2^N. According to the code and comments: "The optimum size (in terms of
memory usage) for a mempool is when n is a power of two minus one: n = (2^q
- 1)."

The reason: rte_mempool_create(... n ...) invokes rte_ring_create(...
rte_align32power(n+1) ...), i.e., it wants to allocate n+1 slots in the
ring, but it has to round that up to the next power of 2. So, when you
allocate 2^N bufs, its ring comes with twice as many slots (2^(N+1)).

--
Robert



On Fri, Sep 20, 2013 at 2:48 AM, Dmitry Vyal <dmitryvyal@gmail.com> wrote:

> On 09/19/2013 11:39 PM, Robert Sanford wrote:
>
>> Hi Dmitry,
>>
>> The biggest drop-off seems to be from size 128K to 256K. Are you using
>> 1GB huge pages already (rather than 2MB)?
>>
>> I would think that it would not use over 1GB until you ask for 512K mbufs
>> or more.
>>
>>
> Hi Robert,
>
> Yes, I've been using 1GB pages for a while. My L3 cache is 20MB and mbufs
> are 2240 bytes of size. So something strange indeed happens then we move
> from ~200MB to ~400MB. Any ideas?
>
> Regards,
> Dmitry
>
>

Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes.

[Dmitry Vyal]

On 09/20/2013 07:34 PM, Robert Sanford wrote:
> One more point, if you're not doing this already: Allocate 2^N-1 
> mbufs, not 2^N. According to the code and comments: "The optimum size 
> (in terms of memory usage) for a mempool is when n is a power of two 
> minus one: n = (2^q - 1)."
>
Many thanks! Didn't know about it.