[dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems

[dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems

[Kumar, Ravi1]

Hi,

As the memory subsystem in DPDK 18.05 is reworked, it has introduced a problem for AMD EPYC 2P platforms.
The issue is that DPDK 18.05 only works with up to 4 NUMAs. For AMD EPYC 2P platforms, DPDK now only works with P0 (NUMA 0-3) and does not work with P1 (NUMA 4-7).

The problem can be fixed by reducing some of the default settings of the memory subsystem.

To solve this issue:
-              We can create our own config file for our integrated 10G NIC, that is for amd_xgbe PMD. This will make amd_xgbe immune to this problem.
-              However, when any other NIC (Intel, Mellanox, Cavium or Broadcom etc.) is plugged into NUMA 4-7, the problem will still be exposed.
-              If we only fix it in "config/common_base", it will cover all cases.

Our current workaround is:
Edit config file "./config/common_base" and change the following line
                CONFIG_RTE_MAX_MEM_MB_PER_TYPE=131072
TO
                CONFIG_RTE_MAX_MEM_MB_PER_TYPE=65536

Any better solution for this issue is welcome.

We would appreciate if this issue can be fixed in the next release (18.08) so the STOCK version of DPDK works on AMD EPYC 2P platforms.

Regards,
Ravi

Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems

[Burakov, Anatoly]

On 22-Jun-18 5:37 PM, Kumar, Ravi1 wrote:
> Hi,
> 
> As the memory subsystem in DPDK 18.05 is reworked, it has introduced a problem for AMD EPYC 2P platforms.
> The issue is that DPDK 18.05 only works with up to 4 NUMAs. For AMD EPYC 2P platforms, DPDK now only works with P0 (NUMA 0-3) and does not work with P1 (NUMA 4-7).
> 
> The problem can be fixed by reducing some of the default settings of the memory subsystem.
> 
> To solve this issue:
> -              We can create our own config file for our integrated 10G NIC, that is for amd_xgbe PMD. This will make amd_xgbe immune to this problem.
> -              However, when any other NIC (Intel, Mellanox, Cavium or Broadcom etc.) is plugged into NUMA 4-7, the problem will still be exposed.
> -              If we only fix it in "config/common_base", it will cover all cases.
> 
> Our current workaround is:
> Edit config file "./config/common_base" and change the following line
>                  CONFIG_RTE_MAX_MEM_MB_PER_TYPE=131072
> TO
>                  CONFIG_RTE_MAX_MEM_MB_PER_TYPE=65536
> 
> Any better solution for this issue is welcome.
> 
> We would appreciate if this issue can be fixed in the next release (18.08) so the STOCK version of DPDK works on AMD EPYC 2P platforms.
> 
> Regards,
> Ravi
> 

Hi Ravi,

What is the reason behind this limitation? Is it too much virtual memory 
being preallocated?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems

[Kumar, Ravi1]

>On 22-Jun-18 5:37 PM, Kumar, Ravi1 wrote:
>> Hi,
>> 
>> As the memory subsystem in DPDK 18.05 is reworked, it has introduced a problem for AMD EPYC 2P platforms.
>> The issue is that DPDK 18.05 only works with up to 4 NUMAs. For AMD EPYC 2P platforms, DPDK now only works with P0 (NUMA 0-3) and does not work with P1 (NUMA 4-7).
>> 
>> The problem can be fixed by reducing some of the default settings of the memory subsystem.
>> 
>> To solve this issue:
>> -              We can create our own config file for our integrated 10G NIC, that is for amd_xgbe PMD. This will make amd_xgbe immune to this problem.
>> -              However, when any other NIC (Intel, Mellanox, Cavium or Broadcom etc.) is plugged into NUMA 4-7, the problem will still be exposed.
>> -              If we only fix it in "config/common_base", it will cover all cases.
>> 
>> Our current workaround is:
>> Edit config file "./config/common_base" and change the following line
>>                  CONFIG_RTE_MAX_MEM_MB_PER_TYPE=131072
>> TO
>>                  CONFIG_RTE_MAX_MEM_MB_PER_TYPE=65536
>> 
>> Any better solution for this issue is welcome.
>> 
>> We would appreciate if this issue can be fixed in the next release (18.08) so the STOCK version of DPDK works on AMD EPYC 2P platforms.
>> 
>> Regards,
>> Ravi
>> 
>
>Hi Ravi,
>
>What is the reason behind this limitation? Is it too much virtual memory being preallocated?
>
>--
>Thanks,
>Anatoly
>
Hi Anatoly,

We believe this is true.  By default, too much virtual memory is being preallocated. The result is it can only support up to 4 NUMAs.

Our workaround is to reduce the amount of preallocated virtual memory by half, so to support up to 8 NUMAs.

Regards,
Ravi

Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems

[Burakov, Anatoly]

On 28-Jun-18 8:03 AM, Kumar, Ravi1 wrote:
>> On 22-Jun-18 5:37 PM, Kumar, Ravi1 wrote:
>>> Hi,
>>>
>>> As the memory subsystem in DPDK 18.05 is reworked, it has introduced a problem for AMD EPYC 2P platforms.
>>> The issue is that DPDK 18.05 only works with up to 4 NUMAs. For AMD EPYC 2P platforms, DPDK now only works with P0 (NUMA 0-3) and does not work with P1 (NUMA 4-7).
>>>
>>> The problem can be fixed by reducing some of the default settings of the memory subsystem.
>>>
>>> To solve this issue:
>>> -              We can create our own config file for our integrated 10G NIC, that is for amd_xgbe PMD. This will make amd_xgbe immune to this problem.
>>> -              However, when any other NIC (Intel, Mellanox, Cavium or Broadcom etc.) is plugged into NUMA 4-7, the problem will still be exposed.
>>> -              If we only fix it in "config/common_base", it will cover all cases.
>>>
>>> Our current workaround is:
>>> Edit config file "./config/common_base" and change the following line
>>>                   CONFIG_RTE_MAX_MEM_MB_PER_TYPE=131072
>>> TO
>>>                   CONFIG_RTE_MAX_MEM_MB_PER_TYPE=65536
>>>
>>> Any better solution for this issue is welcome.
>>>
>>> We would appreciate if this issue can be fixed in the next release (18.08) so the STOCK version of DPDK works on AMD EPYC 2P platforms.
>>>
>>> Regards,
>>> Ravi
>>>
>>
>> Hi Ravi,
>>
>> What is the reason behind this limitation? Is it too much virtual memory being preallocated?
>>
>> --
>> Thanks,
>> Anatoly
>>
> Hi Anatoly,
> 
> We believe this is true.  By default, too much virtual memory is being preallocated. The result is it can only support up to 4 NUMAs.
> 
> Our workaround is to reduce the amount of preallocated virtual memory by half, so to support up to 8 NUMAs.
> 
> Regards,
> Ravi
> 

I assume you see a bunch of failed mmap() calls with ENOMEM?

In general, changing base config that way is an OK change, and it won't 
even be an ABI break since this memory is allocated at runtime. I just 
want to make sure that we fix the underlying problem, rather than the 
symptom.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems

[Kumar, Ravi1]

>On 28-Jun-18 8:03 AM, Kumar, Ravi1 wrote:

>>> On 22-Jun-18 5:37 PM, Kumar, Ravi1 wrote:

>>>> Hi,

>>>>

>>>> As the memory subsystem in DPDK 18.05 is reworked, it has introduced a problem for AMD EPYC 2P platforms.

>>>> The issue is that DPDK 18.05 only works with up to 4 NUMAs. For AMD EPYC 2P platforms, DPDK now only works with P0 (NUMA 0-3) and does not work with P1 (NUMA 4-7).

>>>>

>>>> The problem can be fixed by reducing some of the default settings of the memory subsystem.

>>>>

>>>> To solve this issue:

>>>> -              We can create our own config file for our integrated 10G NIC, that is for amd_xgbe PMD. This will make amd_xgbe immune to this problem.

>>>> -              However, when any other NIC (Intel, Mellanox, Cavium or Broadcom etc.) is plugged into NUMA 4-7, the problem will still be exposed.

>>>> -              If we only fix it in "config/common_base", it will cover all cases.

>>>>

>>>> Our current workaround is:

>>>> Edit config file "./config/common_base" and change the following line

>>>>                   CONFIG_RTE_MAX_MEM_MB_PER_TYPE=131072

>>>> TO

>>>>                   CONFIG_RTE_MAX_MEM_MB_PER_TYPE=65536

>>>>

>>>> Any better solution for this issue is welcome.

>>>>

>>>> We would appreciate if this issue can be fixed in the next release (18.08) so the STOCK version of DPDK works on AMD EPYC 2P platforms.

>>>>

>>>> Regards,

>>>> Ravi

>>>>

>>>

>>> Hi Ravi,

>>>

>>> What is the reason behind this limitation? Is it too much virtual memory being preallocated?

>>>

>>> --

>>> Thanks,

>>> Anatoly

>>>

>> Hi Anatoly,

>>

>> We believe this is true.  By default, too much virtual memory is being preallocated. The result is it can only support up to 4 NUMAs.

>>

>> Our workaround is to reduce the amount of preallocated virtual memory by half, so to support up to 8 NUMAs.

>>

>> Regards,

>> Ravi

>>

>

>I assume you see a bunch of failed mmap() calls with ENOMEM?

>

>In general, changing base config that way is an OK change, and it won't even be an ABI break since this memory is allocated at runtime. I just want to make sure that we fix the underlying problem, rather than the symptom.

>

>--

>Thanks,

>Anatoly

Hi Anatoly,



Sorry for the late reply. I have been away and took me some time to get the logs.



Here are some more details.



Dpdk-18.05/config/common_base contains the constants used to configure the memory subsystem.



CONFIG_RTE_MAX_NUMA_NODES=8

CONFIG_RTE_MAX_MEMSEG_LISTS=64

# each memseg list will be limited to either RTE_MAX_MEMSEG_PER_LIST pages

# or RTE_MAX_MEM_MB_PER_LIST megabytes worth of memory, whichever is smaller

CONFIG_RTE_MAX_MEMSEG_PER_LIST=8192

CONFIG_RTE_MAX_MEM_MB_PER_LIST=32768

# a "type" is a combination of page size and NUMA node. total number of memseg

# lists per type will be limited to either RTE_MAX_MEMSEG_PER_TYPE pages (split

# over multiple lists of RTE_MAX_MEMSEG_PER_LIST pages), or

# RTE_MAX_MEM_MB_PER_TYPE megabytes of memory (split over multiple lists of

# RTE_MAX_MEM_MB_PER_LIST), whichever is smaller

CONFIG_RTE_MAX_MEMSEG_PER_TYPE=32768

CONFIG_RTE_MAX_MEM_MB_PER_TYPE=131072

# global maximum usable amount of VA, in megabytes

CONFIG_RTE_MAX_MEM_MB=524288



From the documentation.

Dpdk-18.05/doc/guides/prog_guide/env_abstraction_layer.rst



All possible virtual memory space that can ever be used for hugepage mapping in a DPDK process is preallocated at startup, thereby placing an upper limit on how much memory a DPDK application can have. DPDK memory is stored in segment lists, each segment is strictly one physical page. It is possible to change the amount of virtual memory being preallocated at startup by editing the following config variables:



* ``CONFIG_RTE_MAX_MEMSEG_LISTS`` controls how many segment lists can DPDK have

* ``CONFIG_RTE_MAX_MEM_MB_PER_LIST`` controls how much megabytes of memory each segment list can address

* ``CONFIG_RTE_MAX_MEMSEG_PER_LIST`` controls how many segments each segment can have

* ``CONFIG_RTE_MAX_MEMSEG_PER_TYPE`` controls how many segments each memory typ can have (where "type" is defined as "page size + NUMA node" combination)

* ``CONFIG_RTE_MAX_MEM_MB_PER_TYPE`` controls how much megabytes of memory each memory type can address

* ``CONFIG_RTE_MAX_MEM_MB`` places a global maximum on the amount of memory DPDK can reserve



Normally, these options do not need to be changed.



.. note::



Preallocated virtual memory is not to be confused with preallocated hugepage memory! All DPDK processes preallocate virtual memory at startup. Hugepages  can later be mapped into that preallocated VA space (if dynamic memory mode is enabled), and can optionally be mapped into it at startup.



Memory setup with 2M pages works with the default configuration.  With the default configuration and 2M hugepages



1.            Total amount of memory for each NUMA zone does not exceed 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE).

2.            Total number of segment lists per NUMA is limited to 32768 (CONFIG_RTE_MAX_MEMSEG_PER_TYPE).   This constraint is met for each numa zone.  This is the limiting factor for memory per numa with 2M hugepages and the default configuration.

3.            The data structures are capable of supporting 64G of memory for each numa zone (32768 segments * 2M hugepagesize).

4.            8 NUMA zones * 64G = 512G.   Therefore the total for all numa zones does not exceed 512G (CONFIG_RTE_MAX_MEM_MB).

5.            Resources are capable of allocating up to 64G per NUMA zone.  Things will work as long as there are enough 2M hugepages  to cover the memory  needs of the DPDK applications AND no memory zone needs more than 64G.



With the default configuration and 1G hugepages



1.            Total amount of memory for each NUMA zone is limited to 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE).  This constraint is hit for each numa zone.  This is the limiting factor for memory per numa.

2.            Total number of segment lists (128) does not exceed 32768 (CONFIG_RTE_MAX_MEMSEG_PER_TYPE).    There are 128 segments per NUMA.

3.            The data structures are capable of supporting 128G of memory for each numa zone (128 segments * 1G hugepagesize).     However, only the first four NUMA zones get initialized before we hit CONFIG_RTE_MAX_MEM_MB (512G).

4.            The total for all numa zones is limited to 512G (CONFIG_RTE_MAX_MEM_MB).  This  limit is  hit after configuring the first four NUMA zones (4 x 128G = 512G).   The rest of the NUMA zones cannot allocate memory.



Apparently, it is intended to support max 8 NUMAs by default (CONFIG_RTE_MAX_NUMA_NODES=8), but when 1G hugepages are use, it can only support up to 4 NUMAs.



Possible workarounds when using 1G hugepages:

1.            Decrease CONFIG_RTE_MAX_MEM_MB_PER_TYPE to 65536 (limit of 64G per NUMA zone).  This is probably the best option unless you need a lot of memory in any given NUMA.

2.            Or, increase CONFIG_RTE_MAX_MEM_MB to 1048576.



With default settings, I got the following errors.



amd@amd:~/dpdk/app/test-pmd$ sudo -E ./testpmd -c 0x7 -n 4 -- -i  --portmask=0x3 --nb-cores=2

EAL: Detected 64 lcore(s)

EAL: Detected 8 NUMA nodes

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Probing VFIO support...

EAL: PCI device 0000:01:00.0 on NUMA socket 0

EAL:   probe driver: 8086:1563 net_ixgbe

EAL: PCI device 0000:01:00.1 on NUMA socket 0

EAL:   probe driver: 8086:1563 net_ixgbe

EAL: PCI device 0000:13:00.0 on NUMA socket 1

EAL:   probe driver: 8086:10d3 net_e1000_em

EAL: PCI device 0000:31:00.0 on NUMA socket 3

EAL:   probe driver: 8086:1563 net_ixgbe

EAL: PCI device 0000:31:00.1 on NUMA socket 3

EAL:   probe driver: 8086:1563 net_ixgbe

Interactive-mode selected

testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0

testpmd: preferred mempool ops selected: ring_mp_mc

testpmd: create a new mbuf pool <mbuf_pool_socket_1>: n=163456, size=2176, socket=1

testpmd: preferred mempool ops selected: ring_mp_mc

testpmd: create a new mbuf pool <mbuf_pool_socket_2>: n=163456, size=2176, socket=2

testpmd: preferred mempool ops selected: ring_mp_mc

testpmd: create a new mbuf pool <mbuf_pool_socket_3>: n=163456, size=2176, socket=3

testpmd: preferred mempool ops selected: ring_mp_mc

testpmd: create a new mbuf pool <mbuf_pool_socket_4>: n=163456, size=2176, socket=4

testpmd: preferred mempool ops selected: ring_mp_mc

EAL: Error - exiting with code: 1

  Cause: Creation of mbuf pool for socket 4 failed: Cannot allocate memory

amd@amd:~/dpdk/app/test-pmd$



After changing  CONFIG_RTE_MAX_MEM_MB_PER_TYPE=131072 to CONFIG_RTE_MAX_MEM_MB_PER_TYPE=65536, the error is gone.



amd@amd:~/dpdk/app/test-pmd$ sudo -E ./testpmd -c 0x7 -n 4 -- -i  --portmask=0x3 --nb-cores=2

EAL: Detected 64 lcore(s)

EAL: Detected 8 NUMA nodes

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Probing VFIO support...

EAL: PCI device 0000:01:00.0 on NUMA socket 0

EAL:   probe driver: 8086:1563 net_ixgbe

EAL: PCI device 0000:01:00.1 on NUMA socket 0

EAL:   probe driver: 8086:1563 net_ixgbe

EAL: PCI device 0000:13:00.0 on NUMA socket 1

EAL:   probe driver: 8086:10d3 net_e1000_em

EAL: PCI device 0000:31:00.0 on NUMA socket 3

EAL:   probe driver: 8086:1563 net_ixgbe

EAL: PCI device 0000:31:00.1 on NUMA socket 3

EAL:   probe driver: 8086:1563 net_ixgbe

Interactive-mode selected

testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0

testpmd: preferred mempool ops selected: ring_mp_mc

testpmd: create a new mbuf pool <mbuf_pool_socket_1>: n=163456, size=2176, socket=1

testpmd: preferred mempool ops selected: ring_mp_mc

testpmd: create a new mbuf pool <mbuf_pool_socket_2>: n=163456, size=2176, socket=2

testpmd: preferred mempool ops selected: ring_mp_mc

testpmd: create a new mbuf pool <mbuf_pool_socket_3>: n=163456, size=2176, socket=3

testpmd: preferred mempool ops selected: ring_mp_mc

testpmd: create a new mbuf pool <mbuf_pool_socket_4>: n=163456, size=2176, socket=4

testpmd: preferred mempool ops selected: ring_mp_mc

testpmd: create a new mbuf pool <mbuf_pool_socket_5>: n=163456, size=2176, socket=5

testpmd: preferred mempool ops selected: ring_mp_mc

testpmd: create a new mbuf pool <mbuf_pool_socket_6>: n=163456, size=2176, socket=6

testpmd: preferred mempool ops selected: ring_mp_mc

testpmd: create a new mbuf pool <mbuf_pool_socket_7>: n=163456, size=2176, socket=7

testpmd: preferred mempool ops selected: ring_mp_mc

Configuring Port 0 (socket 0)

Port 0: A0:36:9F:F7:94:68

Configuring Port 1 (socket 0)

Port 1: A0:36:9F:F7:94:69

Checking link statuses...

Done

testpmd>



Regards,

Ravi

Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems

[Burakov, Anatoly]

On 14-Jul-18 10:44 AM, Kumar, Ravi1 wrote:
> 
> Memory setup with 2M pages works with the default configuration.  With 
> the default configuration and 2M hugepages
> 
> 1.            Total amount of memory for each NUMA zone does not exceed 
> 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE).
> 
> 2.            Total number of segment lists per NUMA is limited to 32768 
> (CONFIG_RTE_MAX_MEMSEG_PER_TYPE).   This constraint is met for each numa 
> zone.  This is the limiting factor for memory per numa with 2M hugepages 
> and the default configuration.
> 
> 3.            The data structures are capable of supporting 64G of 
> memory for each numa zone (32768 segments * 2M hugepagesize).
> 
> 4.            8 NUMA zones * 64G = 512G.   Therefore the total for all 
> numa zones does not exceed 512G (CONFIG_RTE_MAX_MEM_MB).
> 
> 5.            Resources are capable of allocating up to 64G per NUMA 
> zone.  Things will work as long as there are enough 2M hugepages  to 
> cover the memory  needs of the DPDK applications AND no memory zone 
> needs more than 64G.
> 
> With the default configuration and 1G hugepages
> 
> 1.            Total amount of memory for each NUMA zone is limited to 
> 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE).  This constraint is hit for each 
> numa zone.  This is the limiting factor for memory per numa.
> 
> 2.            Total number of segment lists (128) does not exceed 32768 
> (CONFIG_RTE_MAX_MEMSEG_PER_TYPE).    There are 128 segments per NUMA.
> 
> 3.            The data structures are capable of supporting 128G of 
> memory for each numa zone (128 segments * 1G hugepagesize).     However, 
> only the first four NUMA zones get initialized before we hit 
> CONFIG_RTE_MAX_MEM_MB (512G).
> 
> 4.            The total for all numa zones is limited to 512G 
> (CONFIG_RTE_MAX_MEM_MB).  This  limit is  hit after configuring the 
> first four NUMA zones (4 x 128G = 512G).   The rest of the NUMA zones 
> cannot allocate memory.
> 
> Apparently, it is intended to support max 8 NUMAs by default 
> (CONFIG_RTE_MAX_NUMA_NODES=8), but when 1G hugepages are use, it can 
> only support up to 4 NUMAs.
> 
> Possible workarounds when using 1G hugepages:
> 
> 1.            Decrease CONFIG_RTE_MAX_MEM_MB_PER_TYPE to 65536 (limit of 
> 64G per NUMA zone).  This is probably the best option unless you need a 
> lot of memory in any given NUMA.
> 
> 2.            Or, increase CONFIG_RTE_MAX_MEM_MB to 1048576.

Hi Ravi,

OK this makes it much clearer, thanks!

I think the first one should be done. I think 64G per NUMA node is still 
a reasonable amount of memory and it makes the default work (i think we 
can go as far as reducing this limit to 32G per type!), and whoever has 
issues with it can change CONFIG_RTE_MAX_MEM_MB_PER_TYPE or 
CONFIG_RTE_MAX_MEM_MB for their use case. That's what these options are 
there for :)

-- 
Thanks,
Anatoly

Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems

[Kumar, Ravi1]

>
>
>-----Original Message-----
>From: Burakov, Anatoly <anatoly.burakov@intel.com> 
>Sent: Monday, July 16, 2018 4:05 PM
>To: Kumar, Ravi1 <Ravi1.Kumar@amd.com>; dev@dpdk.org
>Subject: Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems
>
>On 14-Jul-18 10:44 AM, Kumar, Ravi1 wrote:
>> 
>> Memory setup with 2M pages works with the default configuration.  With 
>> the default configuration and 2M hugepages
>> 
>> 1.            Total amount of memory for each NUMA zone does not 
>> exceed 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE).
>> 
>> 2.            Total number of segment lists per NUMA is limited to 
>> 32768 (CONFIG_RTE_MAX_MEMSEG_PER_TYPE).   This constraint is met for 
>> each numa zone.  This is the limiting factor for memory per numa with 
>> 2M hugepages and the default configuration.
>> 
>> 3.            The data structures are capable of supporting 64G of 
>> memory for each numa zone (32768 segments * 2M hugepagesize).
>> 
>> 4.            8 NUMA zones * 64G = 512G.   Therefore the total for all 
>> numa zones does not exceed 512G (CONFIG_RTE_MAX_MEM_MB).
>> 
>> 5.            Resources are capable of allocating up to 64G per NUMA 
>> zone.  Things will work as long as there are enough 2M hugepages  to 
>> cover the memory  needs of the DPDK applications AND no memory zone 
>> needs more than 64G.
>> 
>> With the default configuration and 1G hugepages
>> 
>> 1.            Total amount of memory for each NUMA zone is limited to 
>> 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE).  This constraint is hit for 
>> each numa zone.  This is the limiting factor for memory per numa.
>> 
>> 2.            Total number of segment lists (128) does not exceed 
>> 32768 (CONFIG_RTE_MAX_MEMSEG_PER_TYPE).    There are 128 segments per NUMA.
>> 
>> 3.            The data structures are capable of supporting 128G of 
>> memory for each numa zone (128 segments * 1G hugepagesize).     
>> However, only the first four NUMA zones get initialized before we hit 
>> CONFIG_RTE_MAX_MEM_MB (512G).
>> 
>> 4.            The total for all numa zones is limited to 512G 
>> (CONFIG_RTE_MAX_MEM_MB).  This  limit is  hit after configuring the 
>> first four NUMA zones (4 x 128G = 512G).   The rest of the NUMA zones 
>> cannot allocate memory.
>> 
>> Apparently, it is intended to support max 8 NUMAs by default 
>> (CONFIG_RTE_MAX_NUMA_NODES=8), but when 1G hugepages are use, it can 
>> only support up to 4 NUMAs.
>> 
>> Possible workarounds when using 1G hugepages:
>> 
>> 1.            Decrease CONFIG_RTE_MAX_MEM_MB_PER_TYPE to 65536 (limit 
>> of 64G per NUMA zone).  This is probably the best option unless you 
>> need a lot of memory in any given NUMA.
>> 
>> 2.            Or, increase CONFIG_RTE_MAX_MEM_MB to 1048576.
>
>Hi Ravi,
>
>OK this makes it much clearer, thanks!
>
>I think the first one should be done. I think 64G per NUMA node is still a reasonable amount of memory and it makes the default work (i think we can go as far as reducing this limit to 32G per type!), and whoever has issues with it can change CONFIG_RTE_MAX_MEM_MB_PER_TYPE or CONFIG_RTE_MAX_MEM_MB for their use case. That's what these options are there for :)
>
>--
>Thanks,
>Anatoly
>

Hi Anatoly,

Thanks a lot. Will the next release include this change? 

Regards,
Ravi

Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems

[Burakov, Anatoly]

On 24-Jul-18 9:09 AM, Kumar, Ravi1 wrote:
>>
>>
>> -----Original Message-----
>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>> Sent: Monday, July 16, 2018 4:05 PM
>> To: Kumar, Ravi1 <Ravi1.Kumar@amd.com>; dev@dpdk.org
>> Subject: Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems
>>
>> On 14-Jul-18 10:44 AM, Kumar, Ravi1 wrote:
>>>
>>> Memory setup with 2M pages works with the default configuration.  With
>>> the default configuration and 2M hugepages
>>>
>>> 1.            Total amount of memory for each NUMA zone does not
>>> exceed 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE).
>>>
>>> 2.            Total number of segment lists per NUMA is limited to
>>> 32768 (CONFIG_RTE_MAX_MEMSEG_PER_TYPE).   This constraint is met for
>>> each numa zone.  This is the limiting factor for memory per numa with
>>> 2M hugepages and the default configuration.
>>>
>>> 3.            The data structures are capable of supporting 64G of
>>> memory for each numa zone (32768 segments * 2M hugepagesize).
>>>
>>> 4.            8 NUMA zones * 64G = 512G.   Therefore the total for all
>>> numa zones does not exceed 512G (CONFIG_RTE_MAX_MEM_MB).
>>>
>>> 5.            Resources are capable of allocating up to 64G per NUMA
>>> zone.  Things will work as long as there are enough 2M hugepages  to
>>> cover the memory  needs of the DPDK applications AND no memory zone
>>> needs more than 64G.
>>>
>>> With the default configuration and 1G hugepages
>>>
>>> 1.            Total amount of memory for each NUMA zone is limited to
>>> 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE).  This constraint is hit for
>>> each numa zone.  This is the limiting factor for memory per numa.
>>>
>>> 2.            Total number of segment lists (128) does not exceed
>>> 32768 (CONFIG_RTE_MAX_MEMSEG_PER_TYPE).    There are 128 segments per NUMA.
>>>
>>> 3.            The data structures are capable of supporting 128G of
>>> memory for each numa zone (128 segments * 1G hugepagesize).
>>> However, only the first four NUMA zones get initialized before we hit
>>> CONFIG_RTE_MAX_MEM_MB (512G).
>>>
>>> 4.            The total for all numa zones is limited to 512G
>>> (CONFIG_RTE_MAX_MEM_MB).  This  limit is  hit after configuring the
>>> first four NUMA zones (4 x 128G = 512G).   The rest of the NUMA zones
>>> cannot allocate memory.
>>>
>>> Apparently, it is intended to support max 8 NUMAs by default
>>> (CONFIG_RTE_MAX_NUMA_NODES=8), but when 1G hugepages are use, it can
>>> only support up to 4 NUMAs.
>>>
>>> Possible workarounds when using 1G hugepages:
>>>
>>> 1.            Decrease CONFIG_RTE_MAX_MEM_MB_PER_TYPE to 65536 (limit
>>> of 64G per NUMA zone).  This is probably the best option unless you
>>> need a lot of memory in any given NUMA.
>>>
>>> 2.            Or, increase CONFIG_RTE_MAX_MEM_MB to 1048576.
>>
>> Hi Ravi,
>>
>> OK this makes it much clearer, thanks!
>>
>> I think the first one should be done. I think 64G per NUMA node is still a reasonable amount of memory and it makes the default work (i think we can go as far as reducing this limit to 32G per type!), and whoever has issues with it can change CONFIG_RTE_MAX_MEM_MB_PER_TYPE or CONFIG_RTE_MAX_MEM_MB for their use case. That's what these options are there for :)
>>
>> --
>> Thanks,
>> Anatoly
>>
> 
> Hi Anatoly,
> 
> Thanks a lot. Will the next release include this change?
> 
> Regards,
> Ravi
> 

No one has submitted a patch for this, so not at this moment. I will do 
so now, but i cannot guarantee it getting merged in 18.08 since it's 
almost RC2 time, and introducing such a change may be too big a risk.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems

[Kumar, Ravi1]

>
>
>-----Original Message-----
>From: Burakov, Anatoly <anatoly.burakov@intel.com> 
>Sent: Tuesday, July 24, 2018 2:33 PM
>To: Kumar, Ravi1 <Ravi1.Kumar@amd.com>; dev@dpdk.org
>Subject: Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems
>
>On 24-Jul-18 9:09 AM, Kumar, Ravi1 wrote:
>>>
>>>
>>> -----Original Message-----
>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>>> Sent: Monday, July 16, 2018 4:05 PM
>>> To: Kumar, Ravi1 <Ravi1.Kumar@amd.com>; dev@dpdk.org
>>> Subject: Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs 
>>> systems
>>>
>>> On 14-Jul-18 10:44 AM, Kumar, Ravi1 wrote:
>>>>
>>>> Memory setup with 2M pages works with the default configuration.  
>>>> With the default configuration and 2M hugepages
>>>>
>>>> 1.            Total amount of memory for each NUMA zone does not
>>>> exceed 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE).
>>>>
>>>> 2.            Total number of segment lists per NUMA is limited to
>>>> 32768 (CONFIG_RTE_MAX_MEMSEG_PER_TYPE).   This constraint is met for
>>>> each numa zone.  This is the limiting factor for memory per numa 
>>>> with 2M hugepages and the default configuration.
>>>>
>>>> 3.            The data structures are capable of supporting 64G of
>>>> memory for each numa zone (32768 segments * 2M hugepagesize).
>>>>
>>>> 4.            8 NUMA zones * 64G = 512G.   Therefore the total for all
>>>> numa zones does not exceed 512G (CONFIG_RTE_MAX_MEM_MB).
>>>>
>>>> 5.            Resources are capable of allocating up to 64G per NUMA
>>>> zone.  Things will work as long as there are enough 2M hugepages  to 
>>>> cover the memory  needs of the DPDK applications AND no memory zone 
>>>> needs more than 64G.
>>>>
>>>> With the default configuration and 1G hugepages
>>>>
>>>> 1.            Total amount of memory for each NUMA zone is limited to
>>>> 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE).  This constraint is hit for 
>>>> each numa zone.  This is the limiting factor for memory per numa.
>>>>
>>>> 2.            Total number of segment lists (128) does not exceed
>>>> 32768 (CONFIG_RTE_MAX_MEMSEG_PER_TYPE).    There are 128 segments per NUMA.
>>>>
>>>> 3.            The data structures are capable of supporting 128G of
>>>> memory for each numa zone (128 segments * 1G hugepagesize).
>>>> However, only the first four NUMA zones get initialized before we 
>>>> hit CONFIG_RTE_MAX_MEM_MB (512G).
>>>>
>>>> 4.            The total for all numa zones is limited to 512G
>>>> (CONFIG_RTE_MAX_MEM_MB).  This  limit is  hit after configuring the
>>>> first four NUMA zones (4 x 128G = 512G).   The rest of the NUMA zones
>>>> cannot allocate memory.
>>>>
>>>> Apparently, it is intended to support max 8 NUMAs by default 
>>>> (CONFIG_RTE_MAX_NUMA_NODES=8), but when 1G hugepages are use, it can 
>>>> only support up to 4 NUMAs.
>>>>
>>>> Possible workarounds when using 1G hugepages:
>>>>
>>>> 1.            Decrease CONFIG_RTE_MAX_MEM_MB_PER_TYPE to 65536 (limit
>>>> of 64G per NUMA zone).  This is probably the best option unless you 
>>>> need a lot of memory in any given NUMA.
>>>>
>>>> 2.            Or, increase CONFIG_RTE_MAX_MEM_MB to 1048576.
>>>
>>> Hi Ravi,
>>>
>>> OK this makes it much clearer, thanks!
>>>
>>> I think the first one should be done. I think 64G per NUMA node is 
>>> still a reasonable amount of memory and it makes the default work (i 
>>> think we can go as far as reducing this limit to 32G per type!), and 
>>> whoever has issues with it can change CONFIG_RTE_MAX_MEM_MB_PER_TYPE 
>>> or CONFIG_RTE_MAX_MEM_MB for their use case. That's what these 
>>> options are there for :)
>>>
>>> --
>>> Thanks,
>>> Anatoly
>>>
>> 
>> Hi Anatoly,
>> 
>> Thanks a lot. Will the next release include this change?
>> 
>> Regards,
>> Ravi
>> 
>
>No one has submitted a patch for this, so not at this moment. I will do so now, but i cannot guarantee it getting merged in 18.08 since it's almost RC2 time, and introducing such a change may be too big a risk.
>
>--
>Thanks,
>Anatoly
Thanks Anatoly. I understand.

Regards,
Ravi

Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems

[Burakov, Anatoly]

On 24-Jul-18 10:39 AM, Kumar, Ravi1 wrote:
>>
>>
>> -----Original Message-----
>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>> Sent: Tuesday, July 24, 2018 2:33 PM
>> To: Kumar, Ravi1 <Ravi1.Kumar@amd.com>; dev@dpdk.org
>> Subject: Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems
>>
>> On 24-Jul-18 9:09 AM, Kumar, Ravi1 wrote:
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>>>> Sent: Monday, July 16, 2018 4:05 PM
>>>> To: Kumar, Ravi1 <Ravi1.Kumar@amd.com>; dev@dpdk.org
>>>> Subject: Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs
>>>> systems
>>>>
>>>> On 14-Jul-18 10:44 AM, Kumar, Ravi1 wrote:
>>>>>
>>>>> Memory setup with 2M pages works with the default configuration.
>>>>> With the default configuration and 2M hugepages
>>>>>
>>>>> 1.            Total amount of memory for each NUMA zone does not
>>>>> exceed 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE).
>>>>>
>>>>> 2.            Total number of segment lists per NUMA is limited to
>>>>> 32768 (CONFIG_RTE_MAX_MEMSEG_PER_TYPE).   This constraint is met for
>>>>> each numa zone.  This is the limiting factor for memory per numa
>>>>> with 2M hugepages and the default configuration.
>>>>>
>>>>> 3.            The data structures are capable of supporting 64G of
>>>>> memory for each numa zone (32768 segments * 2M hugepagesize).
>>>>>
>>>>> 4.            8 NUMA zones * 64G = 512G.   Therefore the total for all
>>>>> numa zones does not exceed 512G (CONFIG_RTE_MAX_MEM_MB).
>>>>>
>>>>> 5.            Resources are capable of allocating up to 64G per NUMA
>>>>> zone.  Things will work as long as there are enough 2M hugepages  to
>>>>> cover the memory  needs of the DPDK applications AND no memory zone
>>>>> needs more than 64G.
>>>>>
>>>>> With the default configuration and 1G hugepages
>>>>>
>>>>> 1.            Total amount of memory for each NUMA zone is limited to
>>>>> 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE).  This constraint is hit for
>>>>> each numa zone.  This is the limiting factor for memory per numa.
>>>>>
>>>>> 2.            Total number of segment lists (128) does not exceed
>>>>> 32768 (CONFIG_RTE_MAX_MEMSEG_PER_TYPE).    There are 128 segments per NUMA.
>>>>>
>>>>> 3.            The data structures are capable of supporting 128G of
>>>>> memory for each numa zone (128 segments * 1G hugepagesize).
>>>>> However, only the first four NUMA zones get initialized before we
>>>>> hit CONFIG_RTE_MAX_MEM_MB (512G).
>>>>>
>>>>> 4.            The total for all numa zones is limited to 512G
>>>>> (CONFIG_RTE_MAX_MEM_MB).  This  limit is  hit after configuring the
>>>>> first four NUMA zones (4 x 128G = 512G).   The rest of the NUMA zones
>>>>> cannot allocate memory.
>>>>>
>>>>> Apparently, it is intended to support max 8 NUMAs by default
>>>>> (CONFIG_RTE_MAX_NUMA_NODES=8), but when 1G hugepages are use, it can
>>>>> only support up to 4 NUMAs.
>>>>>
>>>>> Possible workarounds when using 1G hugepages:
>>>>>
>>>>> 1.            Decrease CONFIG_RTE_MAX_MEM_MB_PER_TYPE to 65536 (limit
>>>>> of 64G per NUMA zone).  This is probably the best option unless you
>>>>> need a lot of memory in any given NUMA.
>>>>>
>>>>> 2.            Or, increase CONFIG_RTE_MAX_MEM_MB to 1048576.
>>>>
>>>> Hi Ravi,
>>>>
>>>> OK this makes it much clearer, thanks!
>>>>
>>>> I think the first one should be done. I think 64G per NUMA node is
>>>> still a reasonable amount of memory and it makes the default work (i
>>>> think we can go as far as reducing this limit to 32G per type!), and
>>>> whoever has issues with it can change CONFIG_RTE_MAX_MEM_MB_PER_TYPE
>>>> or CONFIG_RTE_MAX_MEM_MB for their use case. That's what these
>>>> options are there for :)
>>>>
>>>> --
>>>> Thanks,
>>>> Anatoly
>>>>
>>>
>>> Hi Anatoly,
>>>
>>> Thanks a lot. Will the next release include this change?
>>>
>>> Regards,
>>> Ravi
>>>
>>
>> No one has submitted a patch for this, so not at this moment. I will do so now, but i cannot guarantee it getting merged in 18.08 since it's almost RC2 time, and introducing such a change may be too big a risk.
>>
>> --
>> Thanks,
>> Anatoly
> Thanks Anatoly. I understand.
> 
> Regards,
> Ravi
> 

Hi Ravi,

In addition to predefined limits that aren't suitable for platforms with 
high numbers of sockets, the calculation method itself has had numerous 
bugs that prevented it from working if limits were changed. I've come up 
with a patch that should fix the issue without the need to change the 
config:

http://patches.dpdk.org/patch/46112/

It would be great if you could test it!

-- 
Thanks,
Anatoly

Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems

[Kumar, Ravi1]

>On 24-Jul-18 10:39 AM, Kumar, Ravi1 wrote:
>>>
>>>
>>> -----Original Message-----
>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>>> Sent: Tuesday, July 24, 2018 2:33 PM
>>> To: Kumar, Ravi1 <Ravi1.Kumar@amd.com>; dev@dpdk.org
>>> Subject: Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs 
>>> systems
>>>
>>> On 24-Jul-18 9:09 AM, Kumar, Ravi1 wrote:
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>>>>> Sent: Monday, July 16, 2018 4:05 PM
>>>>> To: Kumar, Ravi1 <Ravi1.Kumar@amd.com>; dev@dpdk.org
>>>>> Subject: Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs 
>>>>> systems
>>>>>
>>>>> On 14-Jul-18 10:44 AM, Kumar, Ravi1 wrote:
>>>>>>
>>>>>> Memory setup with 2M pages works with the default configuration.
>>>>>> With the default configuration and 2M hugepages
>>>>>>
>>>>>> 1.            Total amount of memory for each NUMA zone does not
>>>>>> exceed 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE).
>>>>>>
>>>>>> 2.            Total number of segment lists per NUMA is limited to
>>>>>> 32768 (CONFIG_RTE_MAX_MEMSEG_PER_TYPE).   This constraint is met for
>>>>>> each numa zone.  This is the limiting factor for memory per numa 
>>>>>> with 2M hugepages and the default configuration.
>>>>>>
>>>>>> 3.            The data structures are capable of supporting 64G of
>>>>>> memory for each numa zone (32768 segments * 2M hugepagesize).
>>>>>>
>>>>>> 4.            8 NUMA zones * 64G = 512G.   Therefore the total for all
>>>>>> numa zones does not exceed 512G (CONFIG_RTE_MAX_MEM_MB).
>>>>>>
>>>>>> 5.            Resources are capable of allocating up to 64G per NUMA
>>>>>> zone.  Things will work as long as there are enough 2M hugepages  
>>>>>> to cover the memory  needs of the DPDK applications AND no memory 
>>>>>> zone needs more than 64G.
>>>>>>
>>>>>> With the default configuration and 1G hugepages
>>>>>>
>>>>>> 1.            Total amount of memory for each NUMA zone is limited to
>>>>>> 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE).  This constraint is hit for 
>>>>>> each numa zone.  This is the limiting factor for memory per numa.
>>>>>>
>>>>>> 2.            Total number of segment lists (128) does not exceed
>>>>>> 32768 (CONFIG_RTE_MAX_MEMSEG_PER_TYPE).    There are 128 segments per NUMA.
>>>>>>
>>>>>> 3.            The data structures are capable of supporting 128G of
>>>>>> memory for each numa zone (128 segments * 1G hugepagesize).
>>>>>> However, only the first four NUMA zones get initialized before we 
>>>>>> hit CONFIG_RTE_MAX_MEM_MB (512G).
>>>>>>
>>>>>> 4.            The total for all numa zones is limited to 512G
>>>>>> (CONFIG_RTE_MAX_MEM_MB).  This  limit is  hit after configuring the
>>>>>> first four NUMA zones (4 x 128G = 512G).   The rest of the NUMA zones
>>>>>> cannot allocate memory.
>>>>>>
>>>>>> Apparently, it is intended to support max 8 NUMAs by default 
>>>>>> (CONFIG_RTE_MAX_NUMA_NODES=8), but when 1G hugepages are use, it 
>>>>>> can only support up to 4 NUMAs.
>>>>>>
>>>>>> Possible workarounds when using 1G hugepages:
>>>>>>
>>>>>> 1.            Decrease CONFIG_RTE_MAX_MEM_MB_PER_TYPE to 65536 (limit
>>>>>> of 64G per NUMA zone).  This is probably the best option unless 
>>>>>> you need a lot of memory in any given NUMA.
>>>>>>
>>>>>> 2.            Or, increase CONFIG_RTE_MAX_MEM_MB to 1048576.
>>>>>
>>>>> Hi Ravi,
>>>>>
>>>>> OK this makes it much clearer, thanks!
>>>>>
>>>>> I think the first one should be done. I think 64G per NUMA node is 
>>>>> still a reasonable amount of memory and it makes the default work 
>>>>> (i think we can go as far as reducing this limit to 32G per type!), 
>>>>> and whoever has issues with it can change 
>>>>> CONFIG_RTE_MAX_MEM_MB_PER_TYPE or CONFIG_RTE_MAX_MEM_MB for their 
>>>>> use case. That's what these options are there for :)
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Anatoly
>>>>>
>>>>
>>>> Hi Anatoly,
>>>>
>>>> Thanks a lot. Will the next release include this change?
>>>>
>>>> Regards,
>>>> Ravi
>>>>
>>>
>>> No one has submitted a patch for this, so not at this moment. I will do so now, but i cannot guarantee it getting merged in 18.08 since it's almost RC2 time, and introducing such a change may be too big a risk.
>>>
>>> --
>>> Thanks,
>>> Anatoly
>> Thanks Anatoly. I understand.
>> 
>> Regards,
>> Ravi
>> 
>
>Hi Ravi,
>
>In addition to predefined limits that aren't suitable for platforms with high numbers of sockets, the calculation method itself has had numerous bugs that prevented it from working if limits were changed. I've come up with a patch that should fix the issue without the need to change the
>config:
>
>http://patches.dpdk.org/patch/46112/
>
>It would be great if you could test it!
>
>--
>Thanks,
>Anatoly

Hi Anatoly,

Thank you very much. This looks really good. I will test this and get back to you. 

Regards,
Ravi

Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems

[Sandeep Raman]

Hi  Anatoly,

I made the changes per the patch http://patches.dpdk.org/patch/43298/,
however testpmd failed.
Alongwith the patch, CONFIG_RTE_MAX_MEM_MB was increased to 1048576.

Logs at https://pastebin.com/XPCDRD2h.

Cheers,

Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems

[Burakov, Anatoly]

On 04-Oct-18 6:07 PM, Sandeep Raman wrote:
> Hi  Anatoly,
> 
> I made the changes per the patch http://patches.dpdk.org/patch/43298/,
> however testpmd failed.
> Alongwith the patch, CONFIG_RTE_MAX_MEM_MB was increased to 1048576.
> 
> Logs at https://pastebin.com/XPCDRD2h.
> 
> Cheers,
> 

Hi,

Please try this patch instead:

http://patches.dpdk.org/patch/46112/

No need to change config.

-- 
Thanks,
Anatoly