[dpdk-users] Optimal number of elements in mempool n = (2^q - 1) vs examples, what is the right thing to do?

[dpdk-users] Optimal number of elements in mempool n = (2^q - 1) vs examples, what is the right thing to do?

[Lazarenko, Vlad (WorldQuant)]

Hello,

I'm new to DPDK and have noticed that documentation for rte_mempool_create states that the optimal size for a number of elements in the pool is n = (2^q-1). But in many examples it is simply set to 2^q (multi_process/simple_mp/main.c uses 2014, for example). This is a bit confusing. Is 2^q - 1 really the optimal number but examples don't use it, or maybe the documentation for the mempool is wrong, or...? If anyone could shed some light on this that'd be helpful.

Thanks,
Vlad



###################################################################################

The information contained in this communication is confidential, may be

subject to legal privilege, and is intended only for the individual named.

If you are not the named addressee, please notify the sender immediately and

delete this email from your system.  The views expressed in this email are

the views of the sender only.  Outgoing and incoming electronic communications

to this address are electronically archived and subject to review and/or disclosure

to someone other than the recipient.

###################################################################################

Re: [dpdk-users] Optimal number of elements in mempool n = (2^q - 1) vs examples, what is the right thing to do?

[Olivier Matz]

Hi Vlad,

On Wed, 1 Feb 2017 16:54:57 +0000, Vlad.Lazarenko at worldquant.com
(Lazarenko, Vlad (WorldQuant)) wrote:
> Hello,
> 
> I'm new to DPDK and have noticed that documentation for
> rte_mempool_create states that the optimal size for a number of
> elements in the pool is n = (2^q-1). But in many examples it is
> simply set to 2^q (multi_process/simple_mp/main.c uses 2014, for
> example). This is a bit confusing. Is 2^q - 1 really the optimal
> number but examples don't use it, or maybe the documentation for the
> mempool is wrong, or...? If anyone could shed some light on this
> that'd be helpful.

That's true for rte_mempool based on a rte_ring (this is the default,
but since recently, it's possible to use another handler).

The size of a rte_ring is (2^n - 1), because one element in the ring is
reserved to distinguish between a full an an empty ring. So, when a
mempool uses a ring, if we ask for 2^n elements, a ring of size
(2^(n+1) - 1) is created, which can consume additional memory.

On the other hand, the mempool object size is often much larger than
a ring entry (usually 8 bytes, the size of a pointer), especially
knowing that by default, the objects are cache aligned (usually 64
bytes).

So we may remove this note in the future since it's not very relevant.

Regards,
Olivier

> 
> ###################################################################################
> 
> The information contained in this communication is confidential, may
> be
> 
> subject to legal privilege, and is intended only for the individual
> named.
> 
> If you are not the named addressee, please notify the sender
> immediately and
> 
> delete this email from your system.  The views expressed in this
> email are
> 
> the views of the sender only.  Outgoing and incoming electronic
> communications
> 
> to this address are electronically archived and subject to review
> and/or disclosure
> 
> to someone other than the recipient.
> 
> ###################################################################################


You can remove this confidential notice for public mailing list :)

Re: [dpdk-users] Optimal number of elements in mempool n = (2^q - 1) vs examples, what is the right thing to do?

[Pavel Shirshov]

Hi Olivier,

It's a good explanation of rte_mempool internals. I think it would be
good to have your comment in the rte_mempool documentation. Could we
add it there?

Thanks

On Mon, Feb 6, 2017 at 5:51 AM, Olivier Matz <olivier.matz@6wind.com> wrote:
> Hi Vlad,
>
> On Wed, 1 Feb 2017 16:54:57 +0000, Vlad.Lazarenko at worldquant.com
> (Lazarenko, Vlad (WorldQuant)) wrote:
>> Hello,
>>
>> I'm new to DPDK and have noticed that documentation for
>> rte_mempool_create states that the optimal size for a number of
>> elements in the pool is n = (2^q-1). But in many examples it is
>> simply set to 2^q (multi_process/simple_mp/main.c uses 2014, for
>> example). This is a bit confusing. Is 2^q - 1 really the optimal
>> number but examples don't use it, or maybe the documentation for the
>> mempool is wrong, or...? If anyone could shed some light on this
>> that'd be helpful.
>
> That's true for rte_mempool based on a rte_ring (this is the default,
> but since recently, it's possible to use another handler).
>
> The size of a rte_ring is (2^n - 1), because one element in the ring is
> reserved to distinguish between a full an an empty ring. So, when a
> mempool uses a ring, if we ask for 2^n elements, a ring of size
> (2^(n+1) - 1) is created, which can consume additional memory.
>
> On the other hand, the mempool object size is often much larger than
> a ring entry (usually 8 bytes, the size of a pointer), especially
> knowing that by default, the objects are cache aligned (usually 64
> bytes).
>
> So we may remove this note in the future since it's not very relevant.
>
> Regards,
> Olivier
>
>>
>> ###################################################################################
>>
>> The information contained in this communication is confidential, may
>> be
>>
>> subject to legal privilege, and is intended only for the individual
>> named.
>>
>> If you are not the named addressee, please notify the sender
>> immediately and
>>
>> delete this email from your system.  The views expressed in this
>> email are
>>
>> the views of the sender only.  Outgoing and incoming electronic
>> communications
>>
>> to this address are electronically archived and subject to review
>> and/or disclosure
>>
>> to someone other than the recipient.
>>
>> ###################################################################################
>
>
> You can remove this confidential notice for public mailing list :)
>

Re: [dpdk-users] Optimal number of elements in mempool n = (2^q - 1) vs examples, what is the right thing to do?

[Olivier MATZ]

Hi Pavel,

On Mon, 6 Feb 2017 13:07:35 -0800, Pavel Shirshov
<pavel.shirshov@gmail.com> wrote:
> On Mon, Feb 6, 2017 at 5:51 AM, Olivier Matz <olivier.matz@6wind.com>
> wrote:
> > Hi Vlad,
> >
> > On Wed, 1 Feb 2017 16:54:57 +0000, Vlad.Lazarenko at worldquant.com
> > (Lazarenko, Vlad (WorldQuant)) wrote:  
> >> Hello,
> >>
> >> I'm new to DPDK and have noticed that documentation for
> >> rte_mempool_create states that the optimal size for a number of
> >> elements in the pool is n = (2^q-1). But in many examples it is
> >> simply set to 2^q (multi_process/simple_mp/main.c uses 2014, for
> >> example). This is a bit confusing. Is 2^q - 1 really the optimal
> >> number but examples don't use it, or maybe the documentation for
> >> the mempool is wrong, or...? If anyone could shed some light on
> >> this that'd be helpful.  
> >
> > That's true for rte_mempool based on a rte_ring (this is the
> > default, but since recently, it's possible to use another handler).
> >
> > The size of a rte_ring is (2^n - 1), because one element in the
> > ring is reserved to distinguish between a full an an empty ring.
> > So, when a mempool uses a ring, if we ask for 2^n elements, a ring
> > of size (2^(n+1) - 1) is created, which can consume additional
> > memory.
> >
> > On the other hand, the mempool object size is often much larger than
> > a ring entry (usually 8 bytes, the size of a pointer), especially
> > knowing that by default, the objects are cache aligned (usually 64
> > bytes).
> >
> > So we may remove this note in the future since it's not very
> > relevant.
>
> Hi Olivier,
> 
> It's a good explanation of rte_mempool internals. I think it would be
> good to have your comment in the rte_mempool documentation. Could we
> add it there?

My comment applies on mempool based on rings (default), but it becomes
wrong when using another handler (ex: stack, or upcoming hw handlers).
Moreover, the size of the pointer array in ring is often negligible in
comparison with the size of the objects array in mempool, making this
comment no so useful.

So I'll probably remove this part from API guide to the programmer's
guide.

Regards,
Olivier

Re: [dpdk-users] Optimal number of elements in mempool n = (2^q - 1) vs examples, what is the right thing to do?

[Pavel Shirshov]

Hi Olivier,

I got your point. Thank you for explaining.
My point is about following. DPDK is not so traditional library. This
library is used for high performance applications, when every detail
is important.
Your initial answer with details how mempool is implemented showed
some DPDK internal details. Which was very valuable for me.
I'm reading dpdk-users to get this kind of internal information about
DPDK, because DPDK has a good documentation, but not comprehensive
one. dpdk-users gives much more valuable information, than DPDK
documentation itself.

Thanks

On Tue, Feb 7, 2017 at 12:53 AM, Olivier MATZ <olivier.matz@6wind.com> wrote:
> Hi Pavel,
>
> On Mon, 6 Feb 2017 13:07:35 -0800, Pavel Shirshov
> <pavel.shirshov@gmail.com> wrote:
>> On Mon, Feb 6, 2017 at 5:51 AM, Olivier Matz <olivier.matz@6wind.com>
>> wrote:
>> > Hi Vlad,
>> >
>> > On Wed, 1 Feb 2017 16:54:57 +0000, Vlad.Lazarenko at worldquant.com
>> > (Lazarenko, Vlad (WorldQuant)) wrote:
>> >> Hello,
>> >>
>> >> I'm new to DPDK and have noticed that documentation for
>> >> rte_mempool_create states that the optimal size for a number of
>> >> elements in the pool is n = (2^q-1). But in many examples it is
>> >> simply set to 2^q (multi_process/simple_mp/main.c uses 2014, for
>> >> example). This is a bit confusing. Is 2^q - 1 really the optimal
>> >> number but examples don't use it, or maybe the documentation for
>> >> the mempool is wrong, or...? If anyone could shed some light on
>> >> this that'd be helpful.
>> >
>> > That's true for rte_mempool based on a rte_ring (this is the
>> > default, but since recently, it's possible to use another handler).
>> >
>> > The size of a rte_ring is (2^n - 1), because one element in the
>> > ring is reserved to distinguish between a full an an empty ring.
>> > So, when a mempool uses a ring, if we ask for 2^n elements, a ring
>> > of size (2^(n+1) - 1) is created, which can consume additional
>> > memory.
>> >
>> > On the other hand, the mempool object size is often much larger than
>> > a ring entry (usually 8 bytes, the size of a pointer), especially
>> > knowing that by default, the objects are cache aligned (usually 64
>> > bytes).
>> >
>> > So we may remove this note in the future since it's not very
>> > relevant.
>>
>> Hi Olivier,
>>
>> It's a good explanation of rte_mempool internals. I think it would be
>> good to have your comment in the rte_mempool documentation. Could we
>> add it there?
>
> My comment applies on mempool based on rings (default), but it becomes
> wrong when using another handler (ex: stack, or upcoming hw handlers).
> Moreover, the size of the pointer array in ring is often negligible in
> comparison with the size of the objects array in mempool, making this
> comment no so useful.
>
> So I'll probably remove this part from API guide to the programmer's
> guide.
>
> Regards,
> Olivier