[dpdk-dev] [RFC] net/mlx5: add sys_mem_en devarg

[Suanming Mou <suanmingm@mellanox.com>]

Currently, for MLX5 PMD, once millions of flows created, the memory 
consumption of the flows are also very huge.
For the system with limited memory, it means the system need to reserve 
most of the memory as huge page memory to serve the flows in advance.
And other normal applications will have no chance to use this reserved 
memory any more.
While most of the time, the system will not have lots of flows, the 
reserved huge page memory becomes a bit waste of memory at most of the time.

One solution maybe not to use the huge page memory for the DPDK 
application, but in this case it hurts the datapath performance.
Since the flows memory is just for the control path, allocate the memory 
from system will only affects the flow insertion rate, so the other 
solution becomes provide a new option which allows flows allocated 
memory from the system memory.
In this case, for the system with limited memory with this option, it 
gets a balance with the huge page memory consumption and datapath 
performance not be hurted.

This feature will add a new devarg for MLX5 PMD which calls 
"sys_mem_en". With this devarg set to be true, the system memory 
allocate functions will be registered for flow objects memory allocate.
The "sys_mem_en" devarg can be set to 0 or 1.
0 means the flow objects will allocate from rte memory.
1 means the flow objects will allocate from system memory.
The sample configuration as below:
./testpmd -n 4  -w 0000:d8:00.0,sys_mem_en=1 -- --burst=64 --mbcache=512 
-i  --nb-cores=7

For DPDK application without this devarg set, it is same with 
"sys_mem_en=0", the default memory allocator for flow objects will still 
be the default DPDK rte memory allocator.


Thanks,

SuanmingMou