[RFC] config: customize max memzones configuration

[RFC] config: customize max memzones configuration

[Ophir Munk]

In current DPDK the RTE_MAX_MEMZONE definition is unconditionally hard
coded as 2560.  For applications requiring different values of this
parameter – it is more convenient to set its value as part of the meson
command line or to set the max value via an rte API - rather than
changing the dpdk source code per application.

An example would be of an application that uses the DPDK mempool library
which is based on DPDK memzone library.  The application may need to
create a number of steering tables, each of which will require its own
mempool allocation.  This RFC is not about how to optimize the
application usage of mempool nor about how to improve the mempool
implementation based on memzone.  It is about how to make the max
memzone definition - build-time or run-time customized.

I would like to suggest three options.

Option 1
========
Add a Meson option in meson options.txt and remove the
RTE_MAX_MEMZONE definition from config/rte_config.h

For example,

 config/meson.build

 # set other values pulled from the build options
 dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
 +dpdk_conf.set('RTE_MAX_MEMZONE', get_option('max_memzones'))
 dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))

 meson_options.txt

 option('max_lcores', type: 'integer', value: 128,
        description: 'maximum number of cores/threads supported by EAL')
 +option('max_memzones', type: 'integer', value: 2560,
 +       description: 'maximum number of memory zones supported by EAL')
 option('max_numa_nodes', type: 'integer', value: 32,
        description: 'maximum number of NUMA nodes supported by EAL')

 config/rte_config.h

 #define RTE_MAX_MEM_MB_PER_TYPE 65536
 -#define RTE_MAX_MEMZONE 2560
 #define RTE_MAX_TAILQ 32

Option 2
========
Use Meson setup -Dc_args="-DRTE_MAX_MEMZONE=XXX" and
make RTE_MAX_MEMZONE conditional in config/rte_config.h

For example, see the code of this commit.

Option 3
========
Add a function which must be called before rte_eal_init():
void rte_memzone_set_max(int max) {memzone_max = max;}
If not called, the default memzone (RTE_MAX_MEMZONE) is used.

With this option there is no need to recompile DPDK and it allows
using an in-box packaged DPDK.

Signed-off-by: Ophir Munk <ophirmu@nvidia.com>
---
 config/rte_config.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/config/rte_config.h b/config/rte_config.h
index fed174a..ca653cc 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -35,7 +35,9 @@
 #define RTE_MAX_MEM_MB_PER_LIST 32768
 #define RTE_MAX_MEMSEG_PER_TYPE 32768
 #define RTE_MAX_MEM_MB_PER_TYPE 65536
+#ifndef RTE_MAX_MEMZONE
 #define RTE_MAX_MEMZONE 2560
+#endif
 #define RTE_MAX_TAILQ 32
 #define RTE_LOG_DP_LEVEL RTE_LOG_INFO
 #define RTE_BACKTRACE 1
-- 
2.8.4

Re: [RFC] config: customize max memzones configuration

[Bruce Richardson]

On Mon, Jan 30, 2023 at 11:23:02AM +0200, Ophir Munk wrote:
> In current DPDK the RTE_MAX_MEMZONE definition is unconditionally hard
> coded as 2560.  For applications requiring different values of this
> parameter – it is more convenient to set its value as part of the meson
> command line or to set the max value via an rte API - rather than
> changing the dpdk source code per application.
> 
> An example would be of an application that uses the DPDK mempool library
> which is based on DPDK memzone library.  The application may need to
> create a number of steering tables, each of which will require its own
> mempool allocation.  This RFC is not about how to optimize the
> application usage of mempool nor about how to improve the mempool
> implementation based on memzone.  It is about how to make the max
> memzone definition - build-time or run-time customized.
> 
> I would like to suggest three options.
> 
> Option 1
> ========
> Add a Meson option in meson options.txt and remove the
> RTE_MAX_MEMZONE definition from config/rte_config.h
> 
> For example,
> 
>  config/meson.build
> 
>  # set other values pulled from the build options
>  dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
>  +dpdk_conf.set('RTE_MAX_MEMZONE', get_option('max_memzones'))
>  dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
> 
>  meson_options.txt
> 
>  option('max_lcores', type: 'integer', value: 128,
>         description: 'maximum number of cores/threads supported by EAL')
>  +option('max_memzones', type: 'integer', value: 2560,
>  +       description: 'maximum number of memory zones supported by EAL')
>  option('max_numa_nodes', type: 'integer', value: 32,
>         description: 'maximum number of NUMA nodes supported by EAL')
> 
>  config/rte_config.h
> 
>  #define RTE_MAX_MEM_MB_PER_TYPE 65536
>  -#define RTE_MAX_MEMZONE 2560
>  #define RTE_MAX_TAILQ 32
> 
Of the 3 options, I think this first option would probably be my preferred one.

/Bruce

Re: [RFC] config: customize max memzones configuration

[Dmitry Kozlyuk]

2023-01-30 11:23 (UTC+0200), Ophir Munk:
> In current DPDK the RTE_MAX_MEMZONE definition is unconditionally hard
> coded as 2560.  For applications requiring different values of this
> parameter – it is more convenient to set its value as part of the meson
> command line or to set the max value via an rte API - rather than
> changing the dpdk source code per application.
> 
> An example would be of an application that uses the DPDK mempool library
> which is based on DPDK memzone library.  The application may need to
> create a number of steering tables, each of which will require its own
> mempool allocation.  This RFC is not about how to optimize the
> application usage of mempool nor about how to improve the mempool
> implementation based on memzone.  It is about how to make the max
> memzone definition - build-time or run-time customized.
> 
> I would like to suggest three options.
> 
> Option 1
> ========
> Add a Meson option in meson options.txt and remove the
> RTE_MAX_MEMZONE definition from config/rte_config.h
> 
[...]
> 
> Option 2
> ========
> Use Meson setup -Dc_args="-DRTE_MAX_MEMZONE=XXX" and
> make RTE_MAX_MEMZONE conditional in config/rte_config.h
> 
> For example, see the code of this commit.
> 
> Option 3
> ========
> Add a function which must be called before rte_eal_init():
> void rte_memzone_set_max(int max) {memzone_max = max;}
> If not called, the default memzone (RTE_MAX_MEMZONE) is used.
> 
> With this option there is no need to recompile DPDK and it allows
> using an in-box packaged DPDK.
[...]

Ideally, there should be no limitation at all.
I vote for some compile-time solution for now
with a plan to remove the restriction inside EAL later.
Option 2 does not expose a user-facing option that would be obsoleted anyway,
so it seems preferable to me, but Option 1 is also OK and consistent.

`RTE_MAX_MEMZONE` is needed currently, because `struct rte_memzone` are stored
in `rte_fbarray` (mapped file-backed array) with a fixed capacity.
Unlike e.g. `rte_eth_devices`, this is not used for efficient access by index
and is not even exposed to PMDs, so the storage can be changed painlessly.
In DPDK, only net/qede uses this constant and also for slow-path bookkeeping.

RE: [RFC] config: customize max memzones configuration

[Ophir Munk]

Thanks Dmitry and Bruce for your inputs. Option 1 is my favorite.
We all agree on Option 1. I will send a patch based on it.

Thank you.

> -----Original Message-----
> From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> Sent: Monday, 30 January 2023 12:01
> To: Ophir Munk <ophirmu@nvidia.com>
> Cc: dev@dpdk.org; Ophir Munk <ophirmu@mellanox.com>; Matan Azrad
> <matan@nvidia.com>; Thomas Monjalon <homas@monjalon.net>; Bruce
> Richardson <bruce.richardson@intel.com>; Lior Margalit
> <lmargalit@nvidia.com>
> Subject: Re: [RFC] config: customize max memzones configuration
> 
> 2023-01-30 11:23 (UTC+0200), Ophir Munk:
> > In current DPDK the RTE_MAX_MEMZONE definition is unconditionally hard
> > coded as 2560.  For applications requiring different values of this
> > parameter – it is more convenient to set its value as part of the
> > meson command line or to set the max value via an rte API - rather
> > than changing the dpdk source code per application.
> >
> > An example would be of an application that uses the DPDK mempool
> > library which is based on DPDK memzone library.  The application may
> > need to create a number of steering tables, each of which will require
> > its own mempool allocation.  This RFC is not about how to optimize the
> > application usage of mempool nor about how to improve the mempool
> > implementation based on memzone.  It is about how to make the max
> > memzone definition - build-time or run-time customized.
> >
> > I would like to suggest three options.
> >
> > Option 1
> > ========
> > Add a Meson option in meson options.txt and remove the
> RTE_MAX_MEMZONE
> > definition from config/rte_config.h
> >
> [...]
> >
> > Option 2
> > ========
> > Use Meson setup -Dc_args="-DRTE_MAX_MEMZONE=XXX" and make
> > RTE_MAX_MEMZONE conditional in config/rte_config.h
> >
> > For example, see the code of this commit.
> >
> > Option 3
> > ========
> > Add a function which must be called before rte_eal_init():
> > void rte_memzone_set_max(int max) {memzone_max = max;} If not called,
> > the default memzone (RTE_MAX_MEMZONE) is used.
> >
> > With this option there is no need to recompile DPDK and it allows
> > using an in-box packaged DPDK.
> [...]
> 
> Ideally, there should be no limitation at all.
> I vote for some compile-time solution for now with a plan to remove the
> restriction inside EAL later.
> Option 2 does not expose a user-facing option that would be obsoleted
> anyway, so it seems preferable to me, but Option 1 is also OK and consistent.
> 
> `RTE_MAX_MEMZONE` is needed currently, because `struct rte_memzone`
> are stored in `rte_fbarray` (mapped file-backed array) with a fixed capacity.
> Unlike e.g. `rte_eth_devices`, this is not used for efficient access by index and
> is not even exposed to PMDs, so the storage can be changed painlessly.
> In DPDK, only net/qede uses this constant and also for slow-path
> bookkeeping.