Re: [dpdk-dev] [PATCH] parray: introduce internal API for dynamic arrays

[Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>]

2021-06-16 18:29 (UTC+0530), Jerin Jacob:
> On Wed, Jun 16, 2021 at 5:52 PM Burakov, Anatoly
> <anatoly.burakov@intel.com> wrote:
> >
> > On 16-Jun-21 10:42 AM, Jerin Jacob wrote:  
> > > On Tue, Jun 15, 2021 at 12:18 PM Thomas Monjalon <thomas@monjalon.net> wrote:  
> > >>
> > >> 14/06/2021 17:48, Morten Brørup:  
> > >>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas Monjalon  
> > >>> It would be much simpler to just increase RTE_MAX_ETHPORTS to something big enough to hold a sufficiently large array. And possibly add an rte_max_ethports variable to indicate the number of populated entries in the array, for use when iterating over the array.
> > >>>
> > >>> Can we come up with another example than RTE_MAX_ETHPORTS where this library provides a better benefit?  
> > >>
> > >> What is big enough?
> > >> Is 640KB enough for RAM? ;)  
> > >
> > > If I understand it correctly, Linux process allocates 640KB due to
> > > that fact currently
> > > struct rte_eth_dev rte_eth_devices[RTE_MAX_ETHPORTS] is global and it
> > > is from BSS.
> > >
> > > If we make this from heap i.e use malloc() to allocate this memory
> > > then in my understanding Linux
> > > really won't allocate the real page for backend memory until unless,
> > > someone write/read to this memory.
> > >
> > > i.e it will be free virtual memory using Linux memory management help.
> > > If so, we can keep large values for RTE_MAX_ETHPORTS
> > > without wasting any "real" memory even though the system has a few ports.
> > >
> > > Thoughts?
> > >  
> >
> > mmap works this way with anonymous memory, i'm not so sure about
> > malloc()'ed memory.  
> 
> Looking at online documentation scatters over the internet, sbrk(), is
> based on demand paging.
> So I am not sure as well. I am also not sure how we can write some
> test case to verify it.
> Allocating a huge memory through malloc() not failing, not sure it is
> due to demand pagging
> or Linux over commit feature or combination of both,
> 
> if mmap works in this way, we could have EAL abstraction for such
> memory alloc like
> eal_malloc_demand_page() or so and if Windows also supports it.
> 
> 
> 
> > Plus, we can't base these decisions on what Linux
> > does because we support other OS's. Do they do this as well?  
> 
> + Windows OS maintainers

Yes, Windows uses demand paging.

Is it true that BSS is eagerly allocated (i. e. RAM consumed)? If not, and it
shouldn't be, malloc() isn't needed unless hugepages are required.