Re: [dpdk-dev] Ability to/impact of running with smaller page sizes

[Jeff Shaw <jeffrey.b.shaw@intel.com>]

Hi Matt,

On Mon, Jun 30, 2014 at 05:43:39PM -0500, Matt Laswell wrote:
> Hey Folks,
> 
> In my application, I'm seeing some design considerations in a project I'm
> working on that push me towards the use of smaller memory page sizes.  I'm
> curious - is it possible in practical terms to run DPDK without hugepages?

Yes, but I do not believe an implementation exists.

>  If so, does anybody have any practical experience (or a
> back-of-the-envelop estimate) of how badly such a configuration would hurt
> performance?  For sake of argument, assume that virtually all of the memory
> being used is in pre-allocated mempools (e.g lots of rte_mempool_create(),
> very little rte_malloc().
> 

It is possible, though not recommended if you want "good performance", to use
smaller memory page sizes.  Poor performance results from penalties incurred
due to DTLB misses.  Please consider the following example.
 
An application pre-allocates several thousand buffers to use for packet
reception and transmission using 4KB pages.  Each buffer contains 2KB worth
of data space, or enough to store the typical maximum Ethernet frame size. 
Since the page size is only 4KB, each DTLB entry can cache a maximum of two
packet buffer address translations.  If the first level DTLB has, for
instance, 64 x 4KB entries, you would only be able to cache about 128 address
translations at any given time (+1,024 if you include the second level DTLB). 
With 32 x 2MB entries, each DTLB entry can cache address translations for 32K
packet buffers at any given time.
 
If you believe that your application performance will be negatively impacted
by latencies incurred due to DTLB misses, it is recommended to take steps
which would maximize the DTLB hit rate.
 
Of course, you will not know how this impacts performance for you application
unless it is tried under realistic conditions.  If you end up doing so, could
you please update the list?


Thanks,
Jeff