Re: Optimizing memory access with DPDK allocated memory

[Antonio Di Bacco <a.dibacco.ks@gmail.com>]

I read a couple of articles
(https://www.thomas-krenn.com/en/wiki/Optimize_memory_performance_of_Intel_Xeon_Scalable_systems?xtxsearchselecthit=1
and this https://www.exxactcorp.com/blog/HPC/balance-memory-guidelines-for-intel-xeon-scalable-family-processors)
and I understood a little bit more.

If the XEON memory controller is able to spread contiguous memory
accesses onto different channels in hardware (as Stepphen correctly
stated), then, how DPDK with option -n can benefit an application?
I also coded a test application to write a 1GB hugepage and calculate
time needed but, equipping an additional two DIMM on two unused
channels of my available six channels motherboard (X11DPi-NT) , I
didn't observe any improvement. This is strange because adding two
channels to the 4 already equipped should make a noticeable
difference.

For reference this is the small program for allocating and writing memory.
https://github.com/adibacco/simple_mp_mem_2
and the results with 4 memory channels:
https://docs.google.com/spreadsheets/d/1mDoKYLMhMMKDaOS3RuGEnpPgRNKuZOy4lMIhG-1N7B8/edit?usp=sharing


On Fri, May 20, 2022 at 5:48 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Fri, 20 May 2022 10:34:46 +0200
> Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
>
> > Let us say I have two memory channels each one with its own 16GB memory
> > module, I suppose the first memory channel will be used when addressing
> > physical memory in the range 0 to 0x4 0000 0000 and the second when
> > addressing physical memory in the range 0x4 0000 0000 to  0x7 ffff ffff.
> > Correct?
> > Now, I need to have a 2GB buffer with one "writer" and one "reader", the
> > writer writes on half of the buffer (call it A) and, in the meantime, the
> > reader reads on the other half (B). When the writer finishes writing its
> > half buffer (A), signal it to the reader and they swap,  the reader starts
> > to read from A and writer starts to write to B.
> > If I allocate the whole buffer (on two 1GB hugepages) across the two memory
> > channels, one half of the buffer is allocated on the end of first channel
> > while the other half is allocated on the start of the second memory
> > channel, would this increase performances compared to the whole buffer
> > allocated within the same memory channel?
>
> Most systems just interleave memory chips based on number of filled slots.
> This is handled by BIOS before kernel even starts.
> The DPDK has a number of memory channels parameter and what it does
> is try and optimize memory allocation by spreading.
>
> Looks like you are inventing your own limited version of what memif does.