* [dpdk-dev] How to fight forwarding performance regression on large mempool sizes.
@ 2013-09-19 7:50 Dmitry Vyal
2013-09-19 19:39 ` Robert Sanford
0 siblings, 1 reply; 8+ messages in thread
From: Dmitry Vyal @ 2013-09-19 7:50 UTC (permalink / raw)
To: dev
Good day everyone,
While working on IP packet defragmenting I had to enlarge mempool size.
I did this to provide large enough time window for assembling a fragment
sequence. Unfortunately, I got a performance regression: if I enlarge
mempool size from 2**12 to 2**20 MBufs, packet performance for not
fragmented packets drops from ~8.5mpps to ~5.5mpps for single core. I
made only a single measure, so the data are noisy, but the trend is evident:
SIZE 4096 - 8.47mpps
SIZE 8192 - 8.26mpps
SIZE 16384 - 8.29mpps
SIZE 32768 - 8.31mpps
SIZE 65536 - 8.12mpps
SIZE 131072 - 7.93mpps
SIZE 262144 - 6.22mpps
SIZE 524288 - 5.72mpps
SIZE 1048576 - 5.63mpps
And I need even larger sizes.
I want to ask for an advice, how best to tackle this? One way I'm
thinking about is to make two mempools, one large for fragments (we may
accumulate a big number of them) and one small for full packets, we just
forward them burst by burst. Is it possible to configure RSS to
distribute packets between queues according to this scheme? Perhaps,
there are better ways?
Thanks,
Dmitry
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes.
2013-09-19 7:50 [dpdk-dev] How to fight forwarding performance regression on large mempool sizes Dmitry Vyal
@ 2013-09-19 19:39 ` Robert Sanford
2013-09-19 19:43 ` Venkatesan, Venky
2013-09-20 6:48 ` Dmitry Vyal
0 siblings, 2 replies; 8+ messages in thread
From: Robert Sanford @ 2013-09-19 19:39 UTC (permalink / raw)
To: Dmitry Vyal; +Cc: dev
Hi Dmitry,
The biggest drop-off seems to be from size 128K to 256K. Are you using 1GB
huge pages already (rather than 2MB)?
I would think that it would not use over 1GB until you ask for 512K mbufs
or more.
--
Regards,
Robert
On Thu, Sep 19, 2013 at 3:50 AM, Dmitry Vyal <dmitryvyal@gmail.com> wrote:
> Good day everyone,
>
> While working on IP packet defragmenting I had to enlarge mempool size. I
> did this to provide large enough time window for assembling a fragment
> sequence. Unfortunately, I got a performance regression: if I enlarge
> mempool size from 2**12 to 2**20 MBufs, packet performance for not
> fragmented packets drops from ~8.5mpps to ~5.5mpps for single core. I made
> only a single measure, so the data are noisy, but the trend is evident:
> SIZE 4096 - 8.47mpps
> SIZE 8192 - 8.26mpps
> SIZE 16384 - 8.29mpps
> SIZE 32768 - 8.31mpps
> SIZE 65536 - 8.12mpps
> SIZE 131072 - 7.93mpps
> SIZE 262144 - 6.22mpps
> SIZE 524288 - 5.72mpps
> SIZE 1048576 - 5.63mpps
>
> And I need even larger sizes.
>
> I want to ask for an advice, how best to tackle this? One way I'm thinking
> about is to make two mempools, one large for fragments (we may accumulate a
> big number of them) and one small for full packets, we just forward them
> burst by burst. Is it possible to configure RSS to distribute packets
> between queues according to this scheme? Perhaps, there are better ways?
>
> Thanks,
> Dmitry
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes.
2013-09-19 19:39 ` Robert Sanford
@ 2013-09-19 19:43 ` Venkatesan, Venky
2013-09-20 6:50 ` Dmitry Vyal
2013-09-20 6:48 ` Dmitry Vyal
1 sibling, 1 reply; 8+ messages in thread
From: Venkatesan, Venky @ 2013-09-19 19:43 UTC (permalink / raw)
To: Robert Sanford, Dmitry Vyal; +Cc: dev
Dmitry,
One other question - what version of DPDK are you doing on?
-Venky
-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Robert Sanford
Sent: Thursday, September 19, 2013 12:40 PM
To: Dmitry Vyal
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes.
Hi Dmitry,
The biggest drop-off seems to be from size 128K to 256K. Are you using 1GB huge pages already (rather than 2MB)?
I would think that it would not use over 1GB until you ask for 512K mbufs or more.
--
Regards,
Robert
On Thu, Sep 19, 2013 at 3:50 AM, Dmitry Vyal <dmitryvyal@gmail.com> wrote:
> Good day everyone,
>
> While working on IP packet defragmenting I had to enlarge mempool
> size. I did this to provide large enough time window for assembling a
> fragment sequence. Unfortunately, I got a performance regression: if I
> enlarge mempool size from 2**12 to 2**20 MBufs, packet performance for
> not fragmented packets drops from ~8.5mpps to ~5.5mpps for single
> core. I made only a single measure, so the data are noisy, but the trend is evident:
> SIZE 4096 - 8.47mpps
> SIZE 8192 - 8.26mpps
> SIZE 16384 - 8.29mpps
> SIZE 32768 - 8.31mpps
> SIZE 65536 - 8.12mpps
> SIZE 131072 - 7.93mpps
> SIZE 262144 - 6.22mpps
> SIZE 524288 - 5.72mpps
> SIZE 1048576 - 5.63mpps
>
> And I need even larger sizes.
>
> I want to ask for an advice, how best to tackle this? One way I'm
> thinking about is to make two mempools, one large for fragments (we
> may accumulate a big number of them) and one small for full packets,
> we just forward them burst by burst. Is it possible to configure RSS
> to distribute packets between queues according to this scheme? Perhaps, there are better ways?
>
> Thanks,
> Dmitry
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes.
2013-09-19 19:39 ` Robert Sanford
2013-09-19 19:43 ` Venkatesan, Venky
@ 2013-09-20 6:48 ` Dmitry Vyal
2013-09-20 14:41 ` Robert Sanford
2013-09-20 15:34 ` Robert Sanford
1 sibling, 2 replies; 8+ messages in thread
From: Dmitry Vyal @ 2013-09-20 6:48 UTC (permalink / raw)
To: Robert Sanford; +Cc: dev
On 09/19/2013 11:39 PM, Robert Sanford wrote:
> Hi Dmitry,
>
> The biggest drop-off seems to be from size 128K to 256K. Are you using
> 1GB huge pages already (rather than 2MB)?
>
> I would think that it would not use over 1GB until you ask for 512K
> mbufs or more.
>
Hi Robert,
Yes, I've been using 1GB pages for a while. My L3 cache is 20MB and
mbufs are 2240 bytes of size. So something strange indeed happens then
we move from ~200MB to ~400MB. Any ideas?
Regards,
Dmitry
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes.
2013-09-20 6:48 ` Dmitry Vyal
@ 2013-09-20 14:41 ` Robert Sanford
2013-09-20 15:34 ` Robert Sanford
1 sibling, 0 replies; 8+ messages in thread
From: Robert Sanford @ 2013-09-20 14:41 UTC (permalink / raw)
To: Dmitry Vyal; +Cc: dev
It might be interesting to see that start/end address of the 256K-item (256
* 1024 * 2240 = 560 MB) mbuf memory pool. Maybe it's the first size that
straddles two 1GB pages.
Perhaps you should try a tool that reports cache misses, TLB misses, and
related statistics. I don't know much about this area, yet, but this looks
like a good starting place:
http://software.intel.com/en-us/articles/intel-performance-counter-monitor-a-better-way-to-measure-cpu-utilization
--
Regards,
Robert
On Fri, Sep 20, 2013 at 2:48 AM, Dmitry Vyal <dmitryvyal@gmail.com> wrote:
> On 09/19/2013 11:39 PM, Robert Sanford wrote:
>
>> Hi Dmitry,
>>
>> The biggest drop-off seems to be from size 128K to 256K. Are you using
>> 1GB huge pages already (rather than 2MB)?
>>
>> I would think that it would not use over 1GB until you ask for 512K mbufs
>> or more.
>>
>>
> Hi Robert,
>
> Yes, I've been using 1GB pages for a while. My L3 cache is 20MB and mbufs
> are 2240 bytes of size. So something strange indeed happens then we move
> from ~200MB to ~400MB. Any ideas?
>
> Regards,
> Dmitry
>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes.
2013-09-20 6:48 ` Dmitry Vyal
2013-09-20 14:41 ` Robert Sanford
@ 2013-09-20 15:34 ` Robert Sanford
2013-09-22 19:03 ` Dmitry Vyal
1 sibling, 1 reply; 8+ messages in thread
From: Robert Sanford @ 2013-09-20 15:34 UTC (permalink / raw)
To: Dmitry Vyal; +Cc: dev
One more point, if you're not doing this already: Allocate 2^N-1 mbufs, not
2^N. According to the code and comments: "The optimum size (in terms of
memory usage) for a mempool is when n is a power of two minus one: n = (2^q
- 1)."
The reason: rte_mempool_create(... n ...) invokes rte_ring_create(...
rte_align32power(n+1) ...), i.e., it wants to allocate n+1 slots in the
ring, but it has to round that up to the next power of 2. So, when you
allocate 2^N bufs, its ring comes with twice as many slots (2^(N+1)).
--
Robert
On Fri, Sep 20, 2013 at 2:48 AM, Dmitry Vyal <dmitryvyal@gmail.com> wrote:
> On 09/19/2013 11:39 PM, Robert Sanford wrote:
>
>> Hi Dmitry,
>>
>> The biggest drop-off seems to be from size 128K to 256K. Are you using
>> 1GB huge pages already (rather than 2MB)?
>>
>> I would think that it would not use over 1GB until you ask for 512K mbufs
>> or more.
>>
>>
> Hi Robert,
>
> Yes, I've been using 1GB pages for a while. My L3 cache is 20MB and mbufs
> are 2240 bytes of size. So something strange indeed happens then we move
> from ~200MB to ~400MB. Any ideas?
>
> Regards,
> Dmitry
>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes.
2013-09-20 15:34 ` Robert Sanford
@ 2013-09-22 19:03 ` Dmitry Vyal
0 siblings, 0 replies; 8+ messages in thread
From: Dmitry Vyal @ 2013-09-22 19:03 UTC (permalink / raw)
To: Robert Sanford; +Cc: dev
On 09/20/2013 07:34 PM, Robert Sanford wrote:
> One more point, if you're not doing this already: Allocate 2^N-1
> mbufs, not 2^N. According to the code and comments: "The optimum size
> (in terms of memory usage) for a mempool is when n is a power of two
> minus one: n = (2^q - 1)."
>
Many thanks! Didn't know about it.
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2013-09-22 19:03 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-09-19 7:50 [dpdk-dev] How to fight forwarding performance regression on large mempool sizes Dmitry Vyal
2013-09-19 19:39 ` Robert Sanford
2013-09-19 19:43 ` Venkatesan, Venky
2013-09-20 6:50 ` Dmitry Vyal
2013-09-20 6:48 ` Dmitry Vyal
2013-09-20 14:41 ` Robert Sanford
2013-09-20 15:34 ` Robert Sanford
2013-09-22 19:03 ` Dmitry Vyal
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).