[dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Asaf Sinai]

Hi,

We have 2 NUMAs in our system, and we try to allocate a single DPDK memory pool on each NUMA.
However, we see no difference when enabling/disabling "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES" configuration.
We expected that disabling it will allocate pools only on one NUMA (probably NUMA0), but it actually allocates pools on both NUMAs, according to "socket_id" parameter passed to "rte_mempool_create" API.
We have 192GB memory, so NUMA1 memory starts from address: 0x1800000000.
As you can see below, "undDpdkPoolNameSocket_1" was indeed allocated on NUMA1, as we wanted, although "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES" is disabled:

CONFIG_RTE_LIBRTE_VHOST_NUMA=n
CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n

created poolName=undDpdkPoolNameSocket_0, nbufs=887808, bufferSize=2432, total=2059MB
(memZone: name=MP_undDpdkPoolNameSocket_0, socket_id=0, vaddr=0x1f2c0427d00-0x1f2c05abe00, paddr=0x178e627d00-0x178e7abe00, len=1589504, hugepage_sz=2MB)
created poolName=undDpdkPoolNameSocket_1, nbufs=887808, bufferSize=2432, total=2059MB
(memZone: name=MP_undDpdkPoolNameSocket_1, socket_id=1, vaddr=0x1f57fa7be40-0x1f57fbfff40, paddr=0x2f8247be40-0x2f825fff40, len=1589504, hugepage_sz=2MB)

Does anyone know what is "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES" configuration used for?

Thanks,
Asaf

[Radware]

Asaf Sinai
ND SW Engineer
Email: asafsi@radware.com<mailto:asafsi@radware.com>

T:+972-72-3917050
M:+972-50-6518541
F:+972-3-6488662


[Check out the latest and greatest from Radware]<https://www.radware.com/Resources/CampaignRedirector.html>


www.radware.com<https://www.radware.com>



[Blog]<https://blog.radware.com/>  [https://www.radware.com/images/signature/linkedin.jpg] <https://www.linkedin.com/companies/165642> [https://www.radware.com/images/signature/twitter.jpg] <file://twitter.com/radware>   [youtube] <https://www.youtube.com/user/radwareinc>


Confidentiality note: This message, and any attachments to it, contains privileged/confidential information of RADWARE Ltd./RADWARE Inc. and may not be disclosed, used, copied, or transmitted in any form or by any means without prior written permission from RADWARE. If you are not the intended recipient, delete the message and any attachments from your system without reading or copying it, and kindly notify the sender by e-mail. Thank you.

P Please consider your environmental responsibility before printing this e-mail

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Burakov, Anatoly]

On 26-Nov-18 9:15 AM, Asaf Sinai wrote:
> Hi,
> 
> We have 2 NUMAs in our system, and we try to allocate a single DPDK memory pool on each NUMA.
> However, we see no difference when enabling/disabling "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES" configuration.
> We expected that disabling it will allocate pools only on one NUMA (probably NUMA0), but it actually allocates pools on both NUMAs, according to "socket_id" parameter passed to "rte_mempool_create" API.
> We have 192GB memory, so NUMA1 memory starts from address: 0x1800000000.
> As you can see below, "undDpdkPoolNameSocket_1" was indeed allocated on NUMA1, as we wanted, although "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES" is disabled:
> 
> CONFIG_RTE_LIBRTE_VHOST_NUMA=n
> CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n
> 
> created poolName=undDpdkPoolNameSocket_0, nbufs=887808, bufferSize=2432, total=2059MB
> (memZone: name=MP_undDpdkPoolNameSocket_0, socket_id=0, vaddr=0x1f2c0427d00-0x1f2c05abe00, paddr=0x178e627d00-0x178e7abe00, len=1589504, hugepage_sz=2MB)
> created poolName=undDpdkPoolNameSocket_1, nbufs=887808, bufferSize=2432, total=2059MB
> (memZone: name=MP_undDpdkPoolNameSocket_1, socket_id=1, vaddr=0x1f57fa7be40-0x1f57fbfff40, paddr=0x2f8247be40-0x2f825fff40, len=1589504, hugepage_sz=2MB)
> 
> Does anyone know what is "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES" configuration used for?
> 
> Thanks,
> Asaf
> 

Hi Asaf,

I cannot reproduce this behavior. Just tried running testpmd with DPDK 
18.08 as well as latest master [1], and DPDK could not successfully 
allocate a mempool on socket 1.

Did you reconfigure and recompile DPDK after this config change?

[1] Latest master will crash on init in this configuration, fix: 
http://patches.dpdk.org/patch/48338/

-- 
Thanks,
Anatoly

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Asaf Sinai]

[-- Attachment #1: Type: text/plain, Size: 2727 bytes --]

Hi Anatoly,

We did not check it with "testpmd", only with our application.
From the beginning, we did not enable this configuration (look at attached files), and everything works fine.
Of course we rebuild DPDK, when we change configuration.
Please note that we use DPDK 17.11.3, maybe this is why it works fine?

Thanks,
Asaf

-----Original Message-----
From: Burakov, Anatoly <anatoly.burakov@intel.com> 
Sent: Monday, November 26, 2018 01:10 PM
To: Asaf Sinai <AsafSi@Radware.com>; dev@dpdk.org
Subject: Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

On 26-Nov-18 9:15 AM, Asaf Sinai wrote:
> Hi,
> 
> We have 2 NUMAs in our system, and we try to allocate a single DPDK memory pool on each NUMA.
> However, we see no difference when enabling/disabling "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES" configuration.
> We expected that disabling it will allocate pools only on one NUMA (probably NUMA0), but it actually allocates pools on both NUMAs, according to "socket_id" parameter passed to "rte_mempool_create" API.
> We have 192GB memory, so NUMA1 memory starts from address: 0x1800000000.
> As you can see below, "undDpdkPoolNameSocket_1" was indeed allocated on NUMA1, as we wanted, although "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES" is disabled:
> 
> CONFIG_RTE_LIBRTE_VHOST_NUMA=n
> CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n
> 
> created poolName=undDpdkPoolNameSocket_0, nbufs=887808, bufferSize=2432, total=2059MB
> (memZone: name=MP_undDpdkPoolNameSocket_0, socket_id=0, vaddr=0x1f2c0427d00-0x1f2c05abe00, paddr=0x178e627d00-0x178e7abe00, len=1589504, hugepage_sz=2MB)
> created poolName=undDpdkPoolNameSocket_1, nbufs=887808, bufferSize=2432, total=2059MB
> (memZone: name=MP_undDpdkPoolNameSocket_1, socket_id=1, vaddr=0x1f57fa7be40-0x1f57fbfff40, paddr=0x2f8247be40-0x2f825fff40, len=1589504, hugepage_sz=2MB)
> 
> Does anyone know what is "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES" configuration used for?
> 
> Thanks,
> Asaf
> 

Hi Asaf,

I cannot reproduce this behavior. Just tried running testpmd with DPDK 
18.08 as well as latest master [1], and DPDK could not successfully 
allocate a mempool on socket 1.

Did you reconfigure and recompile DPDK after this config change?

[1] Latest master will crash on init in this configuration, fix: 
https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpatches.dpdk.org%2Fpatch%2F48338%2F&amp;data=02%7C01%7CAsafSi%40radware.com%7C8abb9fa1f2534a424b8e08d6538fb6ef%7C6ae4e000b5d04f48a766402d46119b76%7C0%7C0%7C636788274062104056&amp;sdata=LvREwJCBJ25pQ2va8r6US%2F%2B4fPcUQCjPl6cfuc%2B0gGA%3D&amp;reserved=0

-- 
Thanks,
Anatoly

[-- Attachment #2: rte_config.h --]
[-- Type: text/plain, Size: 15600 bytes --]

#ifndef __RTE_CONFIG_H
#define __RTE_CONFIG_H
#undef RTE_EXEC_ENV
#define RTE_EXEC_ENV "linuxapp"
#undef RTE_ARCH
#define RTE_ARCH "x86_64"
#undef RTE_MACHINE
#define RTE_MACHINE "default"
#undef RTE_TOOLCHAIN
#define RTE_TOOLCHAIN "gcc"
#undef RTE_FORCE_INTRINSICS
#undef RTE_ARCH_STRICT_ALIGN
#undef RTE_BUILD_SHARED_LIB
#undef RTE_NEXT_ABI
#define RTE_NEXT_ABI 1
#undef RTE_MAJOR_ABI
#define RTE_MAJOR_ABI 
#undef RTE_CACHE_LINE_SIZE
#define RTE_CACHE_LINE_SIZE 64
#undef RTE_LIBRTE_EAL
#define RTE_LIBRTE_EAL 1
#undef RTE_MAX_LCORE
#define RTE_MAX_LCORE 128
#undef RTE_MAX_NUMA_NODES
#define RTE_MAX_NUMA_NODES 8
#undef RTE_MAX_MEMSEG
#define RTE_MAX_MEMSEG 256
#undef RTE_MAX_MEMZONE
#define RTE_MAX_MEMZONE 8192
#undef RTE_MAX_TAILQ
#define RTE_MAX_TAILQ 32
#undef RTE_ENABLE_ASSERT
#undef RTE_LOG_LEVEL
#define RTE_LOG_LEVEL RTE_LOG_DEBUG
#undef RTE_LOG_DP_LEVEL
#define RTE_LOG_DP_LEVEL RTE_LOG_INFO
#undef RTE_LOG_HISTORY
#define RTE_LOG_HISTORY 256
#undef RTE_BACKTRACE
#define RTE_BACKTRACE 1
#undef RTE_LIBEAL_USE_HPET
#undef RTE_EAL_ALLOW_INV_SOCKET_ID
#undef RTE_EAL_ALWAYS_PANIC_ON_ERROR
#undef RTE_EAL_IGB_UIO
#define RTE_EAL_IGB_UIO 1
#undef RTE_EAL_VFIO
#define RTE_EAL_VFIO 1
#undef RTE_MALLOC_DEBUG
#define RTE_MALLOC_DEBUG 1
#undef RTE_EAL_NUMA_AWARE_HUGEPAGES
#undef RTE_ENABLE_AVX
#define RTE_ENABLE_AVX 1
#undef RTE_ENABLE_AVX512
#undef RTE_EAL_PMD_PATH
#define RTE_EAL_PMD_PATH ""
#undef RTE_LIBRTE_EAL_VMWARE_TSC_MAP_SUPPORT
#define RTE_LIBRTE_EAL_VMWARE_TSC_MAP_SUPPORT 1
#undef RTE_LIBRTE_PCI
#define RTE_LIBRTE_PCI 1
#undef RTE_LIBRTE_KVARGS
#define RTE_LIBRTE_KVARGS 1
#undef RTE_LIBRTE_ETHER
#define RTE_LIBRTE_ETHER 1
#undef RTE_LIBRTE_ETHDEV_DEBUG
#define RTE_LIBRTE_ETHDEV_DEBUG 1
#undef RTE_MAX_ETHPORTS
#define RTE_MAX_ETHPORTS 32
#undef RTE_MAX_QUEUES_PER_PORT
#define RTE_MAX_QUEUES_PER_PORT 1024
#undef RTE_LIBRTE_IEEE1588
#undef RTE_ETHDEV_QUEUE_STAT_CNTRS
#define RTE_ETHDEV_QUEUE_STAT_CNTRS 16
#undef RTE_ETHDEV_RXTX_CALLBACKS
#define RTE_ETHDEV_RXTX_CALLBACKS 1
#undef RTE_ETHDEV_PROFILE_ITT_WASTED_RX_ITERATIONS
#undef RTE_ETHDEV_TX_PREPARE_NOOP
#undef RTE_LIBRTE_PCI_BUS
#define RTE_LIBRTE_PCI_BUS 1
#undef RTE_LIBRTE_VDEV_BUS
#define RTE_LIBRTE_VDEV_BUS 1
#undef RTE_LIBRTE_ENA_PMD
#define RTE_LIBRTE_ENA_PMD 1
#undef RTE_LIBRTE_ENA_DEBUG_RX
#undef RTE_LIBRTE_ENA_DEBUG_TX
#undef RTE_LIBRTE_ENA_DEBUG_TX_FREE
#undef RTE_LIBRTE_ENA_DEBUG_DRIVER
#undef RTE_LIBRTE_ENA_COM_DEBUG
#undef RTE_LIBRTE_EM_PMD
#define RTE_LIBRTE_EM_PMD 1
#undef RTE_LIBRTE_IGB_PMD
#define RTE_LIBRTE_IGB_PMD 1
#undef RTE_LIBRTE_E1000_DEBUG_INIT
#define RTE_LIBRTE_E1000_DEBUG_INIT 1
#undef RTE_LIBRTE_E1000_DEBUG_RX
#define RTE_LIBRTE_E1000_DEBUG_RX 1
#undef RTE_LIBRTE_E1000_DEBUG_TX
#define RTE_LIBRTE_E1000_DEBUG_TX 1
#undef RTE_LIBRTE_E1000_DEBUG_TX_FREE
#define RTE_LIBRTE_E1000_DEBUG_TX_FREE 1
#undef RTE_LIBRTE_E1000_DEBUG_DRIVER
#define RTE_LIBRTE_E1000_DEBUG_DRIVER 1
#undef RTE_LIBRTE_E1000_PF_DISABLE_STRIP_CRC
#undef RTE_LIBRTE_IXGBE_PMD
#define RTE_LIBRTE_IXGBE_PMD 1
#undef RTE_LIBRTE_IXGBE_DEBUG_INIT
#define RTE_LIBRTE_IXGBE_DEBUG_INIT 1
#undef RTE_LIBRTE_IXGBE_DEBUG_RX
#define RTE_LIBRTE_IXGBE_DEBUG_RX 1
#undef RTE_LIBRTE_IXGBE_DEBUG_TX
#define RTE_LIBRTE_IXGBE_DEBUG_TX 1
#undef RTE_LIBRTE_IXGBE_DEBUG_TX_FREE
#define RTE_LIBRTE_IXGBE_DEBUG_TX_FREE 1
#undef RTE_LIBRTE_IXGBE_DEBUG_DRIVER
#define RTE_LIBRTE_IXGBE_DEBUG_DRIVER 1
#undef RTE_LIBRTE_IXGBE_PF_DISABLE_STRIP_CRC
#undef RTE_IXGBE_INC_VECTOR
#define RTE_IXGBE_INC_VECTOR 1
#undef RTE_LIBRTE_IXGBE_BYPASS
#undef RTE_LIBRTE_I40E_PMD
#define RTE_LIBRTE_I40E_PMD 1
#undef RTE_LIBRTE_I40E_DEBUG_RX
#define RTE_LIBRTE_I40E_DEBUG_RX 1
#undef RTE_LIBRTE_I40E_DEBUG_TX
#define RTE_LIBRTE_I40E_DEBUG_TX 1
#undef RTE_LIBRTE_I40E_DEBUG_TX_FREE
#define RTE_LIBRTE_I40E_DEBUG_TX_FREE 1
#undef RTE_LIBRTE_I40E_RX_ALLOW_BULK_ALLOC
#define RTE_LIBRTE_I40E_RX_ALLOW_BULK_ALLOC 1
#undef RTE_LIBRTE_I40E_INC_VECTOR
#define RTE_LIBRTE_I40E_INC_VECTOR 1
#undef RTE_LIBRTE_I40E_16BYTE_RX_DESC
#undef RTE_LIBRTE_I40E_QUEUE_NUM_PER_PF
#define RTE_LIBRTE_I40E_QUEUE_NUM_PER_PF 64
#undef RTE_LIBRTE_I40E_QUEUE_NUM_PER_VF
#define RTE_LIBRTE_I40E_QUEUE_NUM_PER_VF 4
#undef RTE_LIBRTE_I40E_QUEUE_NUM_PER_VM
#define RTE_LIBRTE_I40E_QUEUE_NUM_PER_VM 4
#undef RTE_LIBRTE_I40E_ITR_INTERVAL
#define RTE_LIBRTE_I40E_ITR_INTERVAL -1
#undef RTE_LIBRTE_FM10K_PMD
#define RTE_LIBRTE_FM10K_PMD 1
#undef RTE_LIBRTE_FM10K_DEBUG_INIT
#define RTE_LIBRTE_FM10K_DEBUG_INIT 1
#undef RTE_LIBRTE_FM10K_DEBUG_RX
#define RTE_LIBRTE_FM10K_DEBUG_RX 1
#undef RTE_LIBRTE_FM10K_DEBUG_TX
#define RTE_LIBRTE_FM10K_DEBUG_TX 1
#undef RTE_LIBRTE_FM10K_DEBUG_TX_FREE
#define RTE_LIBRTE_FM10K_DEBUG_TX_FREE 1
#undef RTE_LIBRTE_FM10K_DEBUG_DRIVER
#define RTE_LIBRTE_FM10K_DEBUG_DRIVER 1
#undef RTE_LIBRTE_FM10K_RX_OLFLAGS_ENABLE
#define RTE_LIBRTE_FM10K_RX_OLFLAGS_ENABLE 1
#undef RTE_LIBRTE_FM10K_INC_VECTOR
#undef RTE_LIBRTE_MLX4_PMD
#undef RTE_LIBRTE_MLX4_DEBUG
#undef RTE_LIBRTE_MLX4_TX_MP_CACHE
#define RTE_LIBRTE_MLX4_TX_MP_CACHE 8
#undef RTE_LIBRTE_MLX5_PMD
#define RTE_LIBRTE_MLX5_PMD 1
#undef RTE_LIBRTE_MLX5_DEBUG
#define RTE_LIBRTE_MLX5_DEBUG 1
#undef RTE_LIBRTE_MLX5_TX_MP_CACHE
#define RTE_LIBRTE_MLX5_TX_MP_CACHE 8
#undef RTE_LIBRTE_BNX2X_PMD
#undef RTE_LIBRTE_BNX2X_DEBUG
#undef RTE_LIBRTE_BNX2X_DEBUG_INIT
#undef RTE_LIBRTE_BNX2X_DEBUG_RX
#undef RTE_LIBRTE_BNX2X_DEBUG_TX
#undef RTE_LIBRTE_BNX2X_MF_SUPPORT
#undef RTE_LIBRTE_BNX2X_DEBUG_PERIODIC
#undef RTE_LIBRTE_CXGBE_PMD
#define RTE_LIBRTE_CXGBE_PMD 1
#undef RTE_LIBRTE_CXGBE_DEBUG
#undef RTE_LIBRTE_CXGBE_DEBUG_REG
#undef RTE_LIBRTE_CXGBE_DEBUG_MBOX
#undef RTE_LIBRTE_CXGBE_DEBUG_TX
#undef RTE_LIBRTE_CXGBE_DEBUG_RX
#undef RTE_LIBRTE_CXGBE_TPUT
#define RTE_LIBRTE_CXGBE_TPUT 1
#undef RTE_LIBRTE_ENIC_PMD
#define RTE_LIBRTE_ENIC_PMD 1
#undef RTE_LIBRTE_ENIC_DEBUG
#undef RTE_LIBRTE_ENIC_DEBUG_FLOW
#undef RTE_LIBRTE_NFP_PMD
#define RTE_LIBRTE_NFP_PMD 1
#undef RTE_LIBRTE_NFP_DEBUG
#undef RTE_LIBRTE_MRVL_PMD
#undef RTE_LIBRTE_BNXT_PMD
#define RTE_LIBRTE_BNXT_PMD 1
#undef RTE_LIBRTE_SFC_EFX_PMD
#define RTE_LIBRTE_SFC_EFX_PMD 1
#undef RTE_LIBRTE_SFC_EFX_DEBUG
#undef RTE_LIBRTE_PMD_SOFTNIC
#define RTE_LIBRTE_PMD_SOFTNIC 1
#undef RTE_LIBRTE_PMD_SZEDATA2
#undef RTE_LIBRTE_PMD_SZEDATA2_AS
#define RTE_LIBRTE_PMD_SZEDATA2_AS 0
#undef RTE_LIBRTE_THUNDERX_NICVF_PMD
#define RTE_LIBRTE_THUNDERX_NICVF_PMD 1
#undef RTE_LIBRTE_THUNDERX_NICVF_DEBUG_INIT
#undef RTE_LIBRTE_THUNDERX_NICVF_DEBUG_RX
#undef RTE_LIBRTE_THUNDERX_NICVF_DEBUG_TX
#undef RTE_LIBRTE_THUNDERX_NICVF_DEBUG_DRIVER
#undef RTE_LIBRTE_THUNDERX_NICVF_DEBUG_MBOX
#undef RTE_LIBRTE_LIO_PMD
#define RTE_LIBRTE_LIO_PMD 1
#undef RTE_LIBRTE_LIO_DEBUG_DRIVER
#undef RTE_LIBRTE_LIO_DEBUG_INIT
#undef RTE_LIBRTE_LIO_DEBUG_RX
#undef RTE_LIBRTE_LIO_DEBUG_TX
#undef RTE_LIBRTE_LIO_DEBUG_MBOX
#undef RTE_LIBRTE_LIO_DEBUG_REGS
#undef RTE_LIBRTE_DPAA_BUS
#undef RTE_LIBRTE_DPAA_MEMPOOL
#undef RTE_LIBRTE_DPAA_PMD
#undef RTE_LIBRTE_OCTEONTX_PMD
#define RTE_LIBRTE_OCTEONTX_PMD 1
#undef RTE_LIBRTE_OCTEONTX_DEBUG_INIT
#undef RTE_LIBRTE_OCTEONTX_DEBUG_RX
#undef RTE_LIBRTE_OCTEONTX_DEBUG_TX
#undef RTE_LIBRTE_OCTEONTX_DEBUG_DRIVER
#undef RTE_LIBRTE_OCTEONTX_DEBUG_MBOX
#undef RTE_LIBRTE_FSLMC_BUS
#undef RTE_LIBRTE_DPAA2_MEMPOOL
#undef RTE_LIBRTE_DPAA2_USE_PHYS_IOVA
#define RTE_LIBRTE_DPAA2_USE_PHYS_IOVA 1
#undef RTE_LIBRTE_DPAA2_PMD
#undef RTE_LIBRTE_DPAA2_DEBUG_INIT
#undef RTE_LIBRTE_DPAA2_DEBUG_DRIVER
#undef RTE_LIBRTE_DPAA2_DEBUG_RX
#undef RTE_LIBRTE_DPAA2_DEBUG_TX
#undef RTE_LIBRTE_DPAA2_DEBUG_TX_FREE
#undef RTE_LIBRTE_VIRTIO_PMD
#define RTE_LIBRTE_VIRTIO_PMD 1
#undef RTE_LIBRTE_VIRTIO_DEBUG_INIT
#define RTE_LIBRTE_VIRTIO_DEBUG_INIT 1
#undef RTE_LIBRTE_VIRTIO_DEBUG_RX
#define RTE_LIBRTE_VIRTIO_DEBUG_RX 1
#undef RTE_LIBRTE_VIRTIO_DEBUG_TX
#define RTE_LIBRTE_VIRTIO_DEBUG_TX 1
#undef RTE_LIBRTE_VIRTIO_DEBUG_DRIVER
#define RTE_LIBRTE_VIRTIO_DEBUG_DRIVER 1
#undef RTE_LIBRTE_VIRTIO_DEBUG_DUMP
#define RTE_LIBRTE_VIRTIO_DEBUG_DUMP 1
#undef RTE_VIRTIO_USER
#define RTE_VIRTIO_USER 1
#undef RTE_LIBRTE_VMXNET3_PMD
#define RTE_LIBRTE_VMXNET3_PMD 1
#undef RTE_LIBRTE_VMXNET3_DEBUG_INIT
#undef RTE_LIBRTE_VMXNET3_DEBUG_RX
#undef RTE_LIBRTE_VMXNET3_DEBUG_TX
#undef RTE_LIBRTE_VMXNET3_DEBUG_TX_FREE
#undef RTE_LIBRTE_VMXNET3_DEBUG_DRIVER
#undef RTE_LIBRTE_PMD_RING
#define RTE_LIBRTE_PMD_RING 1
#undef RTE_PMD_RING_MAX_RX_RINGS
#define RTE_PMD_RING_MAX_RX_RINGS 16
#undef RTE_PMD_RING_MAX_TX_RINGS
#define RTE_PMD_RING_MAX_TX_RINGS 16
#undef RTE_LIBRTE_PMD_PCAP
#undef RTE_LIBRTE_PMD_BOND
#define RTE_LIBRTE_PMD_BOND 1
#undef RTE_LIBRTE_BOND_DEBUG_ALB
#undef RTE_LIBRTE_BOND_DEBUG_ALB_L1
#undef RTE_LIBRTE_QEDE_PMD
#define RTE_LIBRTE_QEDE_PMD 1
#undef RTE_LIBRTE_QEDE_DEBUG_INIT
#undef RTE_LIBRTE_QEDE_DEBUG_INFO
#undef RTE_LIBRTE_QEDE_DEBUG_DRIVER
#undef RTE_LIBRTE_QEDE_DEBUG_TX
#undef RTE_LIBRTE_QEDE_DEBUG_RX
#undef RTE_LIBRTE_QEDE_FW
#define RTE_LIBRTE_QEDE_FW ""
#undef RTE_LIBRTE_PMD_AF_PACKET
#define RTE_LIBRTE_PMD_AF_PACKET 1
#undef RTE_LIBRTE_ARK_PMD
#undef RTE_LIBRTE_ARK_PAD_TX
#undef RTE_LIBRTE_ARK_DEBUG_RX
#undef RTE_LIBRTE_ARK_DEBUG_TX
#undef RTE_LIBRTE_ARK_DEBUG_STATS
#undef RTE_LIBRTE_ARK_DEBUG_TRACE
#undef RTE_LIBRTE_AVP_PMD
#define RTE_LIBRTE_AVP_PMD 1
#undef RTE_LIBRTE_AVP_DEBUG_RX
#undef RTE_LIBRTE_AVP_DEBUG_TX
#undef RTE_LIBRTE_AVP_DEBUG_DRIVER
#define RTE_LIBRTE_AVP_DEBUG_DRIVER 1
#undef RTE_LIBRTE_AVP_DEBUG_BUFFERS
#undef RTE_LIBRTE_PMD_TAP
#define RTE_LIBRTE_PMD_TAP 1
#undef RTE_LIBRTE_PMD_NULL
#define RTE_LIBRTE_PMD_NULL 1
#undef RTE_LIBRTE_PMD_FAILSAFE
#define RTE_LIBRTE_PMD_FAILSAFE 1
#undef RTE_PMD_PACKET_PREFETCH
#define RTE_PMD_PACKET_PREFETCH 1
#undef RTE_LIBRTE_CRYPTODEV
#define RTE_LIBRTE_CRYPTODEV 1
#undef RTE_LIBRTE_CRYPTODEV_DEBUG
#undef RTE_CRYPTO_MAX_DEVS
#define RTE_CRYPTO_MAX_DEVS 64
#undef RTE_CRYPTODEV_NAME_LEN
#define RTE_CRYPTODEV_NAME_LEN 64
#undef RTE_LIBRTE_PMD_ARMV8_CRYPTO
#undef RTE_LIBRTE_PMD_ARMV8_CRYPTO_DEBUG
#undef RTE_LIBRTE_PMD_DPAA2_SEC
#undef RTE_LIBRTE_DPAA2_SEC_DEBUG_INIT
#undef RTE_LIBRTE_DPAA2_SEC_DEBUG_DRIVER
#undef RTE_LIBRTE_DPAA2_SEC_DEBUG_RX
#undef RTE_LIBRTE_PMD_DPAA_SEC
#undef RTE_LIBRTE_DPAA_SEC_DEBUG_INIT
#undef RTE_LIBRTE_DPAA_SEC_DEBUG_DRIVER
#undef RTE_LIBRTE_DPAA_SEC_DEBUG_RX
#undef RTE_LIBRTE_PMD_QAT
#undef RTE_LIBRTE_PMD_QAT_DEBUG_INIT
#undef RTE_LIBRTE_PMD_QAT_DEBUG_TX
#undef RTE_LIBRTE_PMD_QAT_DEBUG_RX
#undef RTE_LIBRTE_PMD_QAT_DEBUG_DRIVER
#undef RTE_QAT_PMD_MAX_NB_SESSIONS
#define RTE_QAT_PMD_MAX_NB_SESSIONS 2048
#undef RTE_LIBRTE_PMD_AESNI_MB
#undef RTE_LIBRTE_PMD_AESNI_MB_DEBUG
#undef RTE_LIBRTE_PMD_OPENSSL
#undef RTE_LIBRTE_PMD_OPENSSL_DEBUG
#undef RTE_LIBRTE_PMD_AESNI_GCM
#undef RTE_LIBRTE_PMD_AESNI_GCM_DEBUG
#undef RTE_LIBRTE_PMD_SNOW3G
#undef RTE_LIBRTE_PMD_SNOW3G_DEBUG
#undef RTE_LIBRTE_PMD_KASUMI
#undef RTE_LIBRTE_PMD_KASUMI_DEBUG
#undef RTE_LIBRTE_PMD_ZUC
#undef RTE_LIBRTE_PMD_ZUC_DEBUG
#undef RTE_LIBRTE_PMD_CRYPTO_SCHEDULER
#define RTE_LIBRTE_PMD_CRYPTO_SCHEDULER 1
#undef RTE_LIBRTE_PMD_CRYPTO_SCHEDULER_DEBUG
#undef RTE_LIBRTE_PMD_NULL_CRYPTO
#define RTE_LIBRTE_PMD_NULL_CRYPTO 1
#undef RTE_LIBRTE_PMD_MRVL_CRYPTO
#undef RTE_LIBRTE_PMD_MRVL_CRYPTO_DEBUG
#undef RTE_LIBRTE_SECURITY
#define RTE_LIBRTE_SECURITY 1
#undef RTE_LIBRTE_EVENTDEV
#define RTE_LIBRTE_EVENTDEV 1
#undef RTE_LIBRTE_EVENTDEV_DEBUG
#define RTE_LIBRTE_EVENTDEV_DEBUG 1
#undef RTE_EVENT_MAX_DEVS
#define RTE_EVENT_MAX_DEVS 16
#undef RTE_EVENT_MAX_QUEUES_PER_DEV
#define RTE_EVENT_MAX_QUEUES_PER_DEV 64
#undef RTE_LIBRTE_PMD_SKELETON_EVENTDEV
#define RTE_LIBRTE_PMD_SKELETON_EVENTDEV 1
#undef RTE_LIBRTE_PMD_SKELETON_EVENTDEV_DEBUG
#undef RTE_LIBRTE_PMD_SW_EVENTDEV
#define RTE_LIBRTE_PMD_SW_EVENTDEV 1
#undef RTE_LIBRTE_PMD_SW_EVENTDEV_DEBUG
#undef RTE_LIBRTE_PMD_OCTEONTX_SSOVF
#define RTE_LIBRTE_PMD_OCTEONTX_SSOVF 1
#undef RTE_LIBRTE_PMD_OCTEONTX_SSOVF_DEBUG
#undef RTE_LIBRTE_RING
#define RTE_LIBRTE_RING 1
#undef RTE_LIBRTE_MEMPOOL
#define RTE_LIBRTE_MEMPOOL 1
#undef RTE_MEMPOOL_CACHE_MAX_SIZE
#define RTE_MEMPOOL_CACHE_MAX_SIZE 512
#undef RTE_LIBRTE_MEMPOOL_DEBUG
#define RTE_LIBRTE_MEMPOOL_DEBUG 1
#undef RTE_DRIVER_MEMPOOL_RING
#define RTE_DRIVER_MEMPOOL_RING 1
#undef RTE_DRIVER_MEMPOOL_STACK
#define RTE_DRIVER_MEMPOOL_STACK 1
#undef RTE_LIBRTE_OCTEONTX_MEMPOOL
#define RTE_LIBRTE_OCTEONTX_MEMPOOL 1
#undef RTE_LIBRTE_OCTEONTX_MEMPOOL_DEBUG
#undef RTE_LIBRTE_MBUF
#define RTE_LIBRTE_MBUF 1
#undef RTE_LIBRTE_MBUF_DEBUG
#define RTE_LIBRTE_MBUF_DEBUG 1
#undef RTE_MBUF_DEFAULT_MEMPOOL_OPS
#define RTE_MBUF_DEFAULT_MEMPOOL_OPS "ring_mp_mc"
#undef RTE_MBUF_REFCNT_ATOMIC
#define RTE_MBUF_REFCNT_ATOMIC 1
#undef RTE_PKTMBUF_HEADROOM
#define RTE_PKTMBUF_HEADROOM 256
#undef RTE_LIBRTE_TIMER
#define RTE_LIBRTE_TIMER 1
#undef RTE_LIBRTE_TIMER_DEBUG
#undef RTE_LIBRTE_CFGFILE
#define RTE_LIBRTE_CFGFILE 1
#undef RTE_LIBRTE_CMDLINE
#define RTE_LIBRTE_CMDLINE 1
#undef RTE_LIBRTE_CMDLINE_DEBUG
#undef RTE_LIBRTE_HASH
#define RTE_LIBRTE_HASH 1
#undef RTE_LIBRTE_HASH_DEBUG
#undef RTE_LIBRTE_EFD
#define RTE_LIBRTE_EFD 1
#undef RTE_LIBRTE_MEMBER
#define RTE_LIBRTE_MEMBER 1
#undef RTE_LIBRTE_JOBSTATS
#define RTE_LIBRTE_JOBSTATS 1
#undef RTE_LIBRTE_METRICS
#define RTE_LIBRTE_METRICS 1
#undef RTE_LIBRTE_BITRATE
#define RTE_LIBRTE_BITRATE 1
#undef RTE_LIBRTE_LATENCY_STATS
#define RTE_LIBRTE_LATENCY_STATS 1
#undef RTE_LIBRTE_LPM
#define RTE_LIBRTE_LPM 1
#undef RTE_LIBRTE_LPM_DEBUG
#undef RTE_LIBRTE_ACL
#define RTE_LIBRTE_ACL 1
#undef RTE_LIBRTE_ACL_DEBUG
#define RTE_LIBRTE_ACL_DEBUG 1
#undef RTE_LIBRTE_POWER
#define RTE_LIBRTE_POWER 1
#undef RTE_LIBRTE_POWER_DEBUG
#undef RTE_MAX_LCORE_FREQS
#define RTE_MAX_LCORE_FREQS 64
#undef RTE_LIBRTE_NET
#define RTE_LIBRTE_NET 1
#undef RTE_LIBRTE_IP_FRAG
#define RTE_LIBRTE_IP_FRAG 1
#undef RTE_LIBRTE_IP_FRAG_DEBUG
#undef RTE_LIBRTE_IP_FRAG_MAX_FRAG
#define RTE_LIBRTE_IP_FRAG_MAX_FRAG 4
#undef RTE_LIBRTE_IP_FRAG_TBL_STAT
#undef RTE_LIBRTE_GRO
#define RTE_LIBRTE_GRO 1
#undef RTE_LIBRTE_GSO
#define RTE_LIBRTE_GSO 1
#undef RTE_LIBRTE_METER
#define RTE_LIBRTE_METER 1
#undef RTE_LIBRTE_FLOW_CLASSIFY
#define RTE_LIBRTE_FLOW_CLASSIFY 1
#undef RTE_LIBRTE_SCHED
#define RTE_LIBRTE_SCHED 1
#undef RTE_SCHED_DEBUG
#undef RTE_SCHED_RED
#undef RTE_SCHED_COLLECT_STATS
#undef RTE_SCHED_SUBPORT_TC_OV
#undef RTE_SCHED_PORT_N_GRINDERS
#define RTE_SCHED_PORT_N_GRINDERS 8
#undef RTE_SCHED_VECTOR
#undef RTE_LIBRTE_DISTRIBUTOR
#define RTE_LIBRTE_DISTRIBUTOR 1
#undef RTE_LIBRTE_REORDER
#define RTE_LIBRTE_REORDER 1
#undef RTE_LIBRTE_PORT
#define RTE_LIBRTE_PORT 1
#undef RTE_PORT_STATS_COLLECT
#undef RTE_PORT_PCAP
#undef RTE_LIBRTE_TABLE
#define RTE_LIBRTE_TABLE 1
#undef RTE_TABLE_STATS_COLLECT
#undef RTE_LIBRTE_PIPELINE
#define RTE_LIBRTE_PIPELINE 1
#undef RTE_PIPELINE_STATS_COLLECT
#undef RTE_LIBRTE_KNI
#undef RTE_LIBRTE_PMD_KNI
#define RTE_LIBRTE_PMD_KNI 1
#undef RTE_KNI_KMOD
#undef RTE_KNI_KMOD_ETHTOOL
#undef RTE_KNI_PREEMPT_DEFAULT
#define RTE_KNI_PREEMPT_DEFAULT 1
#undef RTE_LIBRTE_PDUMP
#define RTE_LIBRTE_PDUMP 1
#undef RTE_LIBRTE_VHOST
#define RTE_LIBRTE_VHOST 1
#undef RTE_LIBRTE_VHOST_NUMA
#undef RTE_LIBRTE_VHOST_DEBUG
#undef RTE_LIBRTE_PMD_VHOST
#define RTE_LIBRTE_PMD_VHOST 1
#undef RTE_APP_TEST
#define RTE_APP_TEST 1
#undef RTE_APP_TEST_RESOURCE_TAR
#undef RTE_PROC_INFO
#define RTE_PROC_INFO 1
#undef RTE_TEST_PMD
#define RTE_TEST_PMD 1
#undef RTE_TEST_PMD_RECORD_CORE_CYCLES
#undef RTE_TEST_PMD_RECORD_BURST_STATS
#undef RTE_APP_CRYPTO_PERF
#define RTE_APP_CRYPTO_PERF 1
#undef RTE_APP_EVENTDEV
#define RTE_APP_EVENTDEV 1
#undef RTE_EXEC_ENV_LINUXAPP
#define RTE_EXEC_ENV_LINUXAPP 1
#undef RTE_ARCH_X86_64
#define RTE_ARCH_X86_64 1
#undef RTE_ARCH_X86
#define RTE_ARCH_X86 1
#undef RTE_ARCH_64
#define RTE_ARCH_64 1
#undef RTE_TOOLCHAIN_GCC
#define RTE_TOOLCHAIN_GCC 1
#endif /* __RTE_CONFIG_H */

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Burakov, Anatoly]

On 26-Nov-18 11:33 AM, Asaf Sinai wrote:
> Hi Anatoly,
> 
> We did not check it with "testpmd", only with our application.
>  From the beginning, we did not enable this configuration (look at attached files), and everything works fine.
> Of course we rebuild DPDK, when we change configuration.
> Please note that we use DPDK 17.11.3, maybe this is why it works fine?

Just tested with DPDK 17.11, and yes, it does work the way you are 
describing. This is not intended behavior. I will look into it.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations

[Ilya Maximets]

> Hi,
> 
> We have 2 NUMAs in our system, and we try to allocate a single DPDK memory pool on each NUMA.
> However, we see no difference when enabling/disabling "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES" configuration.
> We expected that disabling it will allocate pools only on one NUMA (probably NUMA0), but it actually allocates pools on both NUMAs, according to "socket_id" parameter passed to "rte_mempool_create" API.
> We have 192GB memory, so NUMA1 memory starts from address: 0x1800000000.
> As you can see below, "undDpdkPoolNameSocket_1" was indeed allocated on NUMA1, as we wanted, although "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES" is disabled:
> 
> CONFIG_RTE_LIBRTE_VHOST_NUMA=n
> CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n
> 
> created poolName=undDpdkPoolNameSocket_0, nbufs=887808, bufferSize=2432, total=2059MB
> (memZone: name=MP_undDpdkPoolNameSocket_0, socket_id=0, vaddr=0x1f2c0427d00-0x1f2c05abe00, paddr=0x178e627d00-0x178e7abe00, len=1589504, hugepage_sz=2MB)
> created poolName=undDpdkPoolNameSocket_1, nbufs=887808, bufferSize=2432, total=2059MB
> (memZone: name=MP_undDpdkPoolNameSocket_1, socket_id=1, vaddr=0x1f57fa7be40-0x1f57fbfff40, paddr=0x2f8247be40-0x2f825fff40, len=1589504, hugepage_sz=2MB)
> 
> Does anyone know what is "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES" configuration used for?
> 

This config option was introduced to force DPDK to allocate memory
from NUMA nodes that was requested by 'socket_id'. That is exactly
what you're observing.

Look at the commit 1b72605d2416 ("mem: balanced allocation of hugepages")
for the original issue fixed by this option.

----

> Hi Asaf,
> 
> I cannot reproduce this behavior. Just tried running testpmd with DPDK 
> 18.08 as well as latest master [1], and DPDK could not successfully 
> allocate a mempool on socket 1.

I think that this is a bug. Because, if option enabled, you should successfully
allocate the memory from the requested NUMA if it's available.

If option disabled, we just requesting pages from the kernel and it could
return them from any NUMA node. With option enabled, we're trying to
force kernel to allocate from the nodes we need.

Best regards, Ilya Maximets.

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations

[Ilya Maximets]

On 26.11.2018 15:23, Ilya Maximets wrote:
>> Hi,
>>
>> We have 2 NUMAs in our system, and we try to allocate a single DPDK memory pool on each NUMA.
>> However, we see no difference when enabling/disabling "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES" configuration.
>> We expected that disabling it will allocate pools only on one NUMA (probably NUMA0), but it actually allocates pools on both NUMAs, according to "socket_id" parameter passed to "rte_mempool_create" API.
>> We have 192GB memory, so NUMA1 memory starts from address: 0x1800000000.
>> As you can see below, "undDpdkPoolNameSocket_1" was indeed allocated on NUMA1, as we wanted, although "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES" is disabled:
>>
>> CONFIG_RTE_LIBRTE_VHOST_NUMA=n
>> CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n
>>
>> created poolName=undDpdkPoolNameSocket_0, nbufs=887808, bufferSize=2432, total=2059MB
>> (memZone: name=MP_undDpdkPoolNameSocket_0, socket_id=0, vaddr=0x1f2c0427d00-0x1f2c05abe00, paddr=0x178e627d00-0x178e7abe00, len=1589504, hugepage_sz=2MB)
>> created poolName=undDpdkPoolNameSocket_1, nbufs=887808, bufferSize=2432, total=2059MB
>> (memZone: name=MP_undDpdkPoolNameSocket_1, socket_id=1, vaddr=0x1f57fa7be40-0x1f57fbfff40, paddr=0x2f8247be40-0x2f825fff40, len=1589504, hugepage_sz=2MB)
>>
>> Does anyone know what is "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES" configuration used for?
>>
> 
> This config option was introduced to force DPDK to allocate memory
> from NUMA nodes that was requested by 'socket_id'. That is exactly
> what you're observing.

I meant '--socket-mem' (not the 'socket_id'), of course.

> 
> Look at the commit 1b72605d2416 ("mem: balanced allocation of hugepages")
> for the original issue fixed by this option.
> 
> ----
> 
>> Hi Asaf,
>>
>> I cannot reproduce this behavior. Just tried running testpmd with DPDK 
>> 18.08 as well as latest master [1], and DPDK could not successfully 
>> allocate a mempool on socket 1.
> 
> I think that this is a bug. Because, if option enabled, you should successfully
> allocate the memory from the requested NUMA if it's available.

Anyway, we can't guarantee the failure here because if option disabled
the kernel will decide from which NUMA to allocate memory. And it
could eventually allocate it from the requested one.

And I guess, the meaning of this option was a bit changed with a
new memory model.

> 
> If option disabled, we just requesting pages from the kernel and it could
> return them from any NUMA node. With option enabled, we're trying to
> force kernel to allocate from the nodes we need.
> 
> Best regards, Ilya Maximets.

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Burakov, Anatoly]

On 26-Nov-18 11:43 AM, Burakov, Anatoly wrote:
> On 26-Nov-18 11:33 AM, Asaf Sinai wrote:
>> Hi Anatoly,
>>
>> We did not check it with "testpmd", only with our application.
>>  From the beginning, we did not enable this configuration (look at 
>> attached files), and everything works fine.
>> Of course we rebuild DPDK, when we change configuration.
>> Please note that we use DPDK 17.11.3, maybe this is why it works fine?
> 
> Just tested with DPDK 17.11, and yes, it does work the way you are 
> describing. This is not intended behavior. I will look into it.
> 

+CC author of commit introducing CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES.

Looking at the code, i think this config option needs to be reworked and 
we should clarify what we mean by this option. It appears that i've 
misunderstood what this option actually intended to do, and i also think 
it's naming could be improved because it's confusing and misleading.

In 17.11, this option does *not* prevent EAL from using NUMA - it merely 
disables using libnuma to perform memory allocation. This looks like 
intended (if counter-intuitive) behavior - disabling this option will 
simply revert DPDK to working as it did before this option was 
introduced (i.e. best-effort allocation). This is why your code still 
works - because EAL still does allocate memory on socket 1, and *knows* 
that it's socket 1 memory. It still supports NUMA.

The commit message for these changes states that the actual purpose of 
this option is to enable "balanced" hugepage allocation. In case of 
cgroups limitations, previously, DPDK would've exhausted all hugepages 
on master core's socket before attempting to allocate from other 
sockets, but by the time we've reached cgroups limits on numbers of 
hugepages, we might not have reached socket 1 and thus missed out on the 
pages we could've allocated, but didn't. Using libnuma solves this 
issue, because now we can allocate pages on sockets we want, instead of 
hoping we won't run out of hugepages before we get the memory we need.

In 18.05 onwards, this option works differently (and arguably wrong). 
More specifically, it disallows allocations on sockets other than 0, and 
it also makes it so that EAL does not check which socket the memory 
*actually* came from. So, not only allocating memory from socket 1 is 
disabled, but allocating from socket 0 may even get you memory from 
socket 1!

+CC Thomas

The CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES option is a misnomer, because it 
makes it seem like this option disables NUMA support, which is not the case.

I would also argue that it is not relevant to 18.05+ memory subsystem, 
and should only work in legacy mode, because it is *impossible* to make 
it work right in the new memory subsystem, and here's why:

Without libnuma, we have no way of "asking" the kernel to allocate a 
hugepage on a specific socket - instead, any allocation will most likely 
happen on socket from which the allocation came from. For example, if 
user program's lcore is on socket 1, allocation on socket 0 will 
actually allocate a page on socket 1.

If we don't check for page's NUMA node affinity (which is what currently 
happens) - we get performance degradation because we may unintentionally 
allocate memory on wrong NUMA node. If we do check for this - then 
allocation of memory on socket 1 from lcore on socket 0 will almost 
never succeed, because kernel will always give us pages on socket 0.

Put it simply, there is no sane way to make this option work for the new 
memory subsystem - IMO it should be dropped, and libnuma should be made 
a hard dependency on Linux.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Ilya Maximets]

On 26.11.2018 15:50, Burakov, Anatoly wrote:
> On 26-Nov-18 11:43 AM, Burakov, Anatoly wrote:
>> On 26-Nov-18 11:33 AM, Asaf Sinai wrote:
>>> Hi Anatoly,
>>>
>>> We did not check it with "testpmd", only with our application.
>>>  From the beginning, we did not enable this configuration (look at attached files), and everything works fine.
>>> Of course we rebuild DPDK, when we change configuration.
>>> Please note that we use DPDK 17.11.3, maybe this is why it works fine?
>>
>> Just tested with DPDK 17.11, and yes, it does work the way you are describing. This is not intended behavior. I will look into it.
>>
> 
> +CC author of commit introducing CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES.
> 
> Looking at the code, i think this config option needs to be reworked and we should clarify what we mean by this option. It appears that i've misunderstood what this option actually intended to do, and i also think it's naming could be improved because it's confusing and misleading.
> 
> In 17.11, this option does *not* prevent EAL from using NUMA - it merely disables using libnuma to perform memory allocation. This looks like intended (if counter-intuitive) behavior - disabling this option will simply revert DPDK to working as it did before this option was introduced (i.e. best-effort allocation). This is why your code still works - because EAL still does allocate memory on socket 1, and *knows* that it's socket 1 memory. It still supports NUMA.
> 
> The commit message for these changes states that the actual purpose of this option is to enable "balanced" hugepage allocation. In case of cgroups limitations, previously, DPDK would've exhausted all hugepages on master core's socket before attempting to allocate from other sockets, but by the time we've reached cgroups limits on numbers of hugepages, we might not have reached socket 1 and thus missed out on the pages we could've allocated, but didn't. Using libnuma solves this issue, because now we can allocate pages on sockets we want, instead of hoping we won't run out of hugepages before we get the memory we need.
> 
> In 18.05 onwards, this option works differently (and arguably wrong). More specifically, it disallows allocations on sockets other than 0, and it also makes it so that EAL does not check which socket the memory *actually* came from. So, not only allocating memory from socket 1 is disabled, but allocating from socket 0 may even get you memory from socket 1!

I'd consider this as a bug.

> 
> +CC Thomas
> 
> The CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES option is a misnomer, because it makes it seem like this option disables NUMA support, which is not the case.
> 
> I would also argue that it is not relevant to 18.05+ memory subsystem, and should only work in legacy mode, because it is *impossible* to make it work right in the new memory subsystem, and here's why:
> 
> Without libnuma, we have no way of "asking" the kernel to allocate a hugepage on a specific socket - instead, any allocation will most likely happen on socket from which the allocation came from. For example, if user program's lcore is on socket 1, allocation on socket 0 will actually allocate a page on socket 1.
> 
> If we don't check for page's NUMA node affinity (which is what currently happens) - we get performance degradation because we may unintentionally allocate memory on wrong NUMA node. If we do check for this - then allocation of memory on socket 1 from lcore on socket 0 will almost never succeed, because kernel will always give us pages on socket 0.
> 
> Put it simply, there is no sane way to make this option work for the new memory subsystem - IMO it should be dropped, and libnuma should be made a hard dependency on Linux.

I agree that new memory model could not work without libnuma, i.e. will
lead to unpredictable memory allocations with no any respect to requested
socket_id's. I also agree that CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES is only
sane for a legacy memory model.
It looks like we have no other choice than just drop the option and make
the code unconditional, i.e. have hard dependency on libnuma.

Best regards, Ilya Maximets.

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Ilya Maximets]

On 26.11.2018 16:16, Ilya Maximets wrote:
> On 26.11.2018 15:50, Burakov, Anatoly wrote:
>> On 26-Nov-18 11:43 AM, Burakov, Anatoly wrote:
>>> On 26-Nov-18 11:33 AM, Asaf Sinai wrote:
>>>> Hi Anatoly,
>>>>
>>>> We did not check it with "testpmd", only with our application.
>>>>  From the beginning, we did not enable this configuration (look at attached files), and everything works fine.
>>>> Of course we rebuild DPDK, when we change configuration.
>>>> Please note that we use DPDK 17.11.3, maybe this is why it works fine?
>>>
>>> Just tested with DPDK 17.11, and yes, it does work the way you are describing. This is not intended behavior. I will look into it.
>>>
>>
>> +CC author of commit introducing CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES.
>>
>> Looking at the code, i think this config option needs to be reworked and we should clarify what we mean by this option. It appears that i've misunderstood what this option actually intended to do, and i also think it's naming could be improved because it's confusing and misleading.
>>
>> In 17.11, this option does *not* prevent EAL from using NUMA - it merely disables using libnuma to perform memory allocation. This looks like intended (if counter-intuitive) behavior - disabling this option will simply revert DPDK to working as it did before this option was introduced (i.e. best-effort allocation). This is why your code still works - because EAL still does allocate memory on socket 1, and *knows* that it's socket 1 memory. It still supports NUMA.
>>
>> The commit message for these changes states that the actual purpose of this option is to enable "balanced" hugepage allocation. In case of cgroups limitations, previously, DPDK would've exhausted all hugepages on master core's socket before attempting to allocate from other sockets, but by the time we've reached cgroups limits on numbers of hugepages, we might not have reached socket 1 and thus missed out on the pages we could've allocated, but didn't. Using libnuma solves this issue, because now we can allocate pages on sockets we want, instead of hoping we won't run out of hugepages before we get the memory we need.
>>
>> In 18.05 onwards, this option works differently (and arguably wrong). More specifically, it disallows allocations on sockets other than 0, and it also makes it so that EAL does not check which socket the memory *actually* came from. So, not only allocating memory from socket 1 is disabled, but allocating from socket 0 may even get you memory from socket 1!
> 
> I'd consider this as a bug.
> 
>>
>> +CC Thomas
>>
>> The CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES option is a misnomer, because it makes it seem like this option disables NUMA support, which is not the case.
>>
>> I would also argue that it is not relevant to 18.05+ memory subsystem, and should only work in legacy mode, because it is *impossible* to make it work right in the new memory subsystem, and here's why:
>>
>> Without libnuma, we have no way of "asking" the kernel to allocate a hugepage on a specific socket - instead, any allocation will most likely happen on socket from which the allocation came from. For example, if user program's lcore is on socket 1, allocation on socket 0 will actually allocate a page on socket 1.
>>
>> If we don't check for page's NUMA node affinity (which is what currently happens) - we get performance degradation because we may unintentionally allocate memory on wrong NUMA node. If we do check for this - then allocation of memory on socket 1 from lcore on socket 0 will almost never succeed, because kernel will always give us pages on socket 0.
>>
>> Put it simply, there is no sane way to make this option work for the new memory subsystem - IMO it should be dropped, and libnuma should be made a hard dependency on Linux.
> 
> I agree that new memory model could not work without libnuma, i.e. will
> lead to unpredictable memory allocations with no any respect to requested
> socket_id's. I also agree that CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES is only
> sane for a legacy memory model.
> It looks like we have no other choice than just drop the option and make
> the code unconditional, i.e. have hard dependency on libnuma.
> 

We, probably, could compile this code and have hard dependency only for
platforms with 'RTE_MAX_NUMA_NODES > 1'.

> Best regards, Ilya Maximets.

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Burakov, Anatoly]

On 26-Nov-18 1:20 PM, Ilya Maximets wrote:
> On 26.11.2018 16:16, Ilya Maximets wrote:
>> On 26.11.2018 15:50, Burakov, Anatoly wrote:
>>> On 26-Nov-18 11:43 AM, Burakov, Anatoly wrote:
>>>> On 26-Nov-18 11:33 AM, Asaf Sinai wrote:
>>>>> Hi Anatoly,
>>>>>
>>>>> We did not check it with "testpmd", only with our application.
>>>>>   From the beginning, we did not enable this configuration (look at attached files), and everything works fine.
>>>>> Of course we rebuild DPDK, when we change configuration.
>>>>> Please note that we use DPDK 17.11.3, maybe this is why it works fine?
>>>>
>>>> Just tested with DPDK 17.11, and yes, it does work the way you are describing. This is not intended behavior. I will look into it.
>>>>
>>>
>>> +CC author of commit introducing CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES.
>>>
>>> Looking at the code, i think this config option needs to be reworked and we should clarify what we mean by this option. It appears that i've misunderstood what this option actually intended to do, and i also think it's naming could be improved because it's confusing and misleading.
>>>
>>> In 17.11, this option does *not* prevent EAL from using NUMA - it merely disables using libnuma to perform memory allocation. This looks like intended (if counter-intuitive) behavior - disabling this option will simply revert DPDK to working as it did before this option was introduced (i.e. best-effort allocation). This is why your code still works - because EAL still does allocate memory on socket 1, and *knows* that it's socket 1 memory. It still supports NUMA.
>>>
>>> The commit message for these changes states that the actual purpose of this option is to enable "balanced" hugepage allocation. In case of cgroups limitations, previously, DPDK would've exhausted all hugepages on master core's socket before attempting to allocate from other sockets, but by the time we've reached cgroups limits on numbers of hugepages, we might not have reached socket 1 and thus missed out on the pages we could've allocated, but didn't. Using libnuma solves this issue, because now we can allocate pages on sockets we want, instead of hoping we won't run out of hugepages before we get the memory we need.
>>>
>>> In 18.05 onwards, this option works differently (and arguably wrong). More specifically, it disallows allocations on sockets other than 0, and it also makes it so that EAL does not check which socket the memory *actually* came from. So, not only allocating memory from socket 1 is disabled, but allocating from socket 0 may even get you memory from socket 1!
>>
>> I'd consider this as a bug.
>>
>>>
>>> +CC Thomas
>>>
>>> The CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES option is a misnomer, because it makes it seem like this option disables NUMA support, which is not the case.
>>>
>>> I would also argue that it is not relevant to 18.05+ memory subsystem, and should only work in legacy mode, because it is *impossible* to make it work right in the new memory subsystem, and here's why:
>>>
>>> Without libnuma, we have no way of "asking" the kernel to allocate a hugepage on a specific socket - instead, any allocation will most likely happen on socket from which the allocation came from. For example, if user program's lcore is on socket 1, allocation on socket 0 will actually allocate a page on socket 1.
>>>
>>> If we don't check for page's NUMA node affinity (which is what currently happens) - we get performance degradation because we may unintentionally allocate memory on wrong NUMA node. If we do check for this - then allocation of memory on socket 1 from lcore on socket 0 will almost never succeed, because kernel will always give us pages on socket 0.
>>>
>>> Put it simply, there is no sane way to make this option work for the new memory subsystem - IMO it should be dropped, and libnuma should be made a hard dependency on Linux.
>>
>> I agree that new memory model could not work without libnuma, i.e. will
>> lead to unpredictable memory allocations with no any respect to requested
>> socket_id's. I also agree that CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES is only
>> sane for a legacy memory model.
>> It looks like we have no other choice than just drop the option and make
>> the code unconditional, i.e. have hard dependency on libnuma.
>>
> 
> We, probably, could compile this code and have hard dependency only for
> platforms with 'RTE_MAX_NUMA_NODES > 1'.

Well, as long as legacy mode stays supported, we have to keep the 
option. The "drop" part was referring to supporting it under the new 
memory system, not a literal drop from config files.

As for using RTE_MAX_NUMA_NODES, i don't think it's merited. 
Distributions cannot deliver different DPDK versions based on the number 
of sockets on a particular machine - so it would have to be a hard 
dependency for distributions anyway (does any distribution ship DPDK 
without libnuma?).

For those compiling from source - are there any supported distributions 
which don't package libnuma? I don't see much sense in keeping libnuma 
optional, IMO. This is of course up to the tech board to decide, but IMO 
the "without libnuma it's basically broken" argument is very strong in 
my opinion :)

-- 
Thanks,
Anatoly

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Ilya Maximets]

On 26.11.2018 16:42, Burakov, Anatoly wrote:
> On 26-Nov-18 1:20 PM, Ilya Maximets wrote:
>> On 26.11.2018 16:16, Ilya Maximets wrote:
>>> On 26.11.2018 15:50, Burakov, Anatoly wrote:
>>>> On 26-Nov-18 11:43 AM, Burakov, Anatoly wrote:
>>>>> On 26-Nov-18 11:33 AM, Asaf Sinai wrote:
>>>>>> Hi Anatoly,
>>>>>>
>>>>>> We did not check it with "testpmd", only with our application.
>>>>>>   From the beginning, we did not enable this configuration (look at attached files), and everything works fine.
>>>>>> Of course we rebuild DPDK, when we change configuration.
>>>>>> Please note that we use DPDK 17.11.3, maybe this is why it works fine?
>>>>>
>>>>> Just tested with DPDK 17.11, and yes, it does work the way you are describing. This is not intended behavior. I will look into it.
>>>>>
>>>>
>>>> +CC author of commit introducing CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES.
>>>>
>>>> Looking at the code, i think this config option needs to be reworked and we should clarify what we mean by this option. It appears that i've misunderstood what this option actually intended to do, and i also think it's naming could be improved because it's confusing and misleading.
>>>>
>>>> In 17.11, this option does *not* prevent EAL from using NUMA - it merely disables using libnuma to perform memory allocation. This looks like intended (if counter-intuitive) behavior - disabling this option will simply revert DPDK to working as it did before this option was introduced (i.e. best-effort allocation). This is why your code still works - because EAL still does allocate memory on socket 1, and *knows* that it's socket 1 memory. It still supports NUMA.
>>>>
>>>> The commit message for these changes states that the actual purpose of this option is to enable "balanced" hugepage allocation. In case of cgroups limitations, previously, DPDK would've exhausted all hugepages on master core's socket before attempting to allocate from other sockets, but by the time we've reached cgroups limits on numbers of hugepages, we might not have reached socket 1 and thus missed out on the pages we could've allocated, but didn't. Using libnuma solves this issue, because now we can allocate pages on sockets we want, instead of hoping we won't run out of hugepages before we get the memory we need.
>>>>
>>>> In 18.05 onwards, this option works differently (and arguably wrong). More specifically, it disallows allocations on sockets other than 0, and it also makes it so that EAL does not check which socket the memory *actually* came from. So, not only allocating memory from socket 1 is disabled, but allocating from socket 0 may even get you memory from socket 1!
>>>
>>> I'd consider this as a bug.
>>>
>>>>
>>>> +CC Thomas
>>>>
>>>> The CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES option is a misnomer, because it makes it seem like this option disables NUMA support, which is not the case.
>>>>
>>>> I would also argue that it is not relevant to 18.05+ memory subsystem, and should only work in legacy mode, because it is *impossible* to make it work right in the new memory subsystem, and here's why:
>>>>
>>>> Without libnuma, we have no way of "asking" the kernel to allocate a hugepage on a specific socket - instead, any allocation will most likely happen on socket from which the allocation came from. For example, if user program's lcore is on socket 1, allocation on socket 0 will actually allocate a page on socket 1.
>>>>
>>>> If we don't check for page's NUMA node affinity (which is what currently happens) - we get performance degradation because we may unintentionally allocate memory on wrong NUMA node. If we do check for this - then allocation of memory on socket 1 from lcore on socket 0 will almost never succeed, because kernel will always give us pages on socket 0.
>>>>
>>>> Put it simply, there is no sane way to make this option work for the new memory subsystem - IMO it should be dropped, and libnuma should be made a hard dependency on Linux.
>>>
>>> I agree that new memory model could not work without libnuma, i.e. will
>>> lead to unpredictable memory allocations with no any respect to requested
>>> socket_id's. I also agree that CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES is only
>>> sane for a legacy memory model.
>>> It looks like we have no other choice than just drop the option and make
>>> the code unconditional, i.e. have hard dependency on libnuma.
>>>
>>
>> We, probably, could compile this code and have hard dependency only for
>> platforms with 'RTE_MAX_NUMA_NODES > 1'.
> 
> Well, as long as legacy mode stays supported, we have to keep the option. The "drop" part was referring to supporting it under the new memory system, not a literal drop from config files.

The option was introduced because we didn't want to introduce the
new hard dependency. Since we'll have it anyway, I'm not sure if
keeping the option for legacy mode makes any sense.

> 
> As for using RTE_MAX_NUMA_NODES, i don't think it's merited. Distributions cannot deliver different DPDK versions based on the number of sockets on a particular machine - so it would have to be a hard dependency for distributions anyway (does any distribution ship DPDK without libnuma?).

At least ARMv7 builds commonly does not ship libnuma package.

> 
> For those compiling from source - are there any supported distributions which don't package libnuma? I don't see much sense in keeping libnuma optional, IMO. This is of course up to the tech board to decide, but IMO the "without libnuma it's basically broken" argument is very strong in my opinion :)
> 

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Burakov, Anatoly]

On 26-Nov-18 2:10 PM, Ilya Maximets wrote:
> On 26.11.2018 16:42, Burakov, Anatoly wrote:
>> On 26-Nov-18 1:20 PM, Ilya Maximets wrote:
>>> On 26.11.2018 16:16, Ilya Maximets wrote:
>>>> On 26.11.2018 15:50, Burakov, Anatoly wrote:
>>>>> On 26-Nov-18 11:43 AM, Burakov, Anatoly wrote:
>>>>>> On 26-Nov-18 11:33 AM, Asaf Sinai wrote:
>>>>>>> Hi Anatoly,
>>>>>>>
>>>>>>> We did not check it with "testpmd", only with our application.
>>>>>>>    From the beginning, we did not enable this configuration (look at attached files), and everything works fine.
>>>>>>> Of course we rebuild DPDK, when we change configuration.
>>>>>>> Please note that we use DPDK 17.11.3, maybe this is why it works fine?
>>>>>>
>>>>>> Just tested with DPDK 17.11, and yes, it does work the way you are describing. This is not intended behavior. I will look into it.
>>>>>>
>>>>>
>>>>> +CC author of commit introducing CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES.
>>>>>
>>>>> Looking at the code, i think this config option needs to be reworked and we should clarify what we mean by this option. It appears that i've misunderstood what this option actually intended to do, and i also think it's naming could be improved because it's confusing and misleading.
>>>>>
>>>>> In 17.11, this option does *not* prevent EAL from using NUMA - it merely disables using libnuma to perform memory allocation. This looks like intended (if counter-intuitive) behavior - disabling this option will simply revert DPDK to working as it did before this option was introduced (i.e. best-effort allocation). This is why your code still works - because EAL still does allocate memory on socket 1, and *knows* that it's socket 1 memory. It still supports NUMA.
>>>>>
>>>>> The commit message for these changes states that the actual purpose of this option is to enable "balanced" hugepage allocation. In case of cgroups limitations, previously, DPDK would've exhausted all hugepages on master core's socket before attempting to allocate from other sockets, but by the time we've reached cgroups limits on numbers of hugepages, we might not have reached socket 1 and thus missed out on the pages we could've allocated, but didn't. Using libnuma solves this issue, because now we can allocate pages on sockets we want, instead of hoping we won't run out of hugepages before we get the memory we need.
>>>>>
>>>>> In 18.05 onwards, this option works differently (and arguably wrong). More specifically, it disallows allocations on sockets other than 0, and it also makes it so that EAL does not check which socket the memory *actually* came from. So, not only allocating memory from socket 1 is disabled, but allocating from socket 0 may even get you memory from socket 1!
>>>>
>>>> I'd consider this as a bug.
>>>>
>>>>>
>>>>> +CC Thomas
>>>>>
>>>>> The CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES option is a misnomer, because it makes it seem like this option disables NUMA support, which is not the case.
>>>>>
>>>>> I would also argue that it is not relevant to 18.05+ memory subsystem, and should only work in legacy mode, because it is *impossible* to make it work right in the new memory subsystem, and here's why:
>>>>>
>>>>> Without libnuma, we have no way of "asking" the kernel to allocate a hugepage on a specific socket - instead, any allocation will most likely happen on socket from which the allocation came from. For example, if user program's lcore is on socket 1, allocation on socket 0 will actually allocate a page on socket 1.
>>>>>
>>>>> If we don't check for page's NUMA node affinity (which is what currently happens) - we get performance degradation because we may unintentionally allocate memory on wrong NUMA node. If we do check for this - then allocation of memory on socket 1 from lcore on socket 0 will almost never succeed, because kernel will always give us pages on socket 0.
>>>>>
>>>>> Put it simply, there is no sane way to make this option work for the new memory subsystem - IMO it should be dropped, and libnuma should be made a hard dependency on Linux.
>>>>
>>>> I agree that new memory model could not work without libnuma, i.e. will
>>>> lead to unpredictable memory allocations with no any respect to requested
>>>> socket_id's. I also agree that CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES is only
>>>> sane for a legacy memory model.
>>>> It looks like we have no other choice than just drop the option and make
>>>> the code unconditional, i.e. have hard dependency on libnuma.
>>>>
>>>
>>> We, probably, could compile this code and have hard dependency only for
>>> platforms with 'RTE_MAX_NUMA_NODES > 1'.
>>
>> Well, as long as legacy mode stays supported, we have to keep the option. The "drop" part was referring to supporting it under the new memory system, not a literal drop from config files.
> 
> The option was introduced because we didn't want to introduce the
> new hard dependency. Since we'll have it anyway, I'm not sure if
> keeping the option for legacy mode makes any sense.

Oh yes, you're right. Drop it is!

> 
>>
>> As for using RTE_MAX_NUMA_NODES, i don't think it's merited. Distributions cannot deliver different DPDK versions based on the number of sockets on a particular machine - so it would have to be a hard dependency for distributions anyway (does any distribution ship DPDK without libnuma?).
> 
> At least ARMv7 builds commonly does not ship libnuma package.

Do you mean libnuma builds for ARMv7 are not available? Or do you mean 
the libnuma package is not installed by default?

If it's the latter, then i believe it's not installed by default 
anywhere, but if using distribution version of DPDK, libnuma will be 
taken care of via package manager. Presumably building from source can 
be taken care of with pkg-config/meson.

Or do you mean ARMv7 does not have libnuma for their arch at all, in any 
distro?

> 
>>
>> For those compiling from source - are there any supported distributions which don't package libnuma? I don't see much sense in keeping libnuma optional, IMO. This is of course up to the tech board to decide, but IMO the "without libnuma it's basically broken" argument is very strong in my opinion :)
>>
> 


-- 
Thanks,
Anatoly

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Ilya Maximets]

On 26.11.2018 17:21, Burakov, Anatoly wrote:
> On 26-Nov-18 2:10 PM, Ilya Maximets wrote:
>> On 26.11.2018 16:42, Burakov, Anatoly wrote:
>>> On 26-Nov-18 1:20 PM, Ilya Maximets wrote:
>>>> On 26.11.2018 16:16, Ilya Maximets wrote:
>>>>> On 26.11.2018 15:50, Burakov, Anatoly wrote:
>>>>>> On 26-Nov-18 11:43 AM, Burakov, Anatoly wrote:
>>>>>>> On 26-Nov-18 11:33 AM, Asaf Sinai wrote:
>>>>>>>> Hi Anatoly,
>>>>>>>>
>>>>>>>> We did not check it with "testpmd", only with our application.
>>>>>>>>    From the beginning, we did not enable this configuration (look at attached files), and everything works fine.
>>>>>>>> Of course we rebuild DPDK, when we change configuration.
>>>>>>>> Please note that we use DPDK 17.11.3, maybe this is why it works fine?
>>>>>>>
>>>>>>> Just tested with DPDK 17.11, and yes, it does work the way you are describing. This is not intended behavior. I will look into it.
>>>>>>>
>>>>>>
>>>>>> +CC author of commit introducing CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES.
>>>>>>
>>>>>> Looking at the code, i think this config option needs to be reworked and we should clarify what we mean by this option. It appears that i've misunderstood what this option actually intended to do, and i also think it's naming could be improved because it's confusing and misleading.
>>>>>>
>>>>>> In 17.11, this option does *not* prevent EAL from using NUMA - it merely disables using libnuma to perform memory allocation. This looks like intended (if counter-intuitive) behavior - disabling this option will simply revert DPDK to working as it did before this option was introduced (i.e. best-effort allocation). This is why your code still works - because EAL still does allocate memory on socket 1, and *knows* that it's socket 1 memory. It still supports NUMA.
>>>>>>
>>>>>> The commit message for these changes states that the actual purpose of this option is to enable "balanced" hugepage allocation. In case of cgroups limitations, previously, DPDK would've exhausted all hugepages on master core's socket before attempting to allocate from other sockets, but by the time we've reached cgroups limits on numbers of hugepages, we might not have reached socket 1 and thus missed out on the pages we could've allocated, but didn't. Using libnuma solves this issue, because now we can allocate pages on sockets we want, instead of hoping we won't run out of hugepages before we get the memory we need.
>>>>>>
>>>>>> In 18.05 onwards, this option works differently (and arguably wrong). More specifically, it disallows allocations on sockets other than 0, and it also makes it so that EAL does not check which socket the memory *actually* came from. So, not only allocating memory from socket 1 is disabled, but allocating from socket 0 may even get you memory from socket 1!
>>>>>
>>>>> I'd consider this as a bug.
>>>>>
>>>>>>
>>>>>> +CC Thomas
>>>>>>
>>>>>> The CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES option is a misnomer, because it makes it seem like this option disables NUMA support, which is not the case.
>>>>>>
>>>>>> I would also argue that it is not relevant to 18.05+ memory subsystem, and should only work in legacy mode, because it is *impossible* to make it work right in the new memory subsystem, and here's why:
>>>>>>
>>>>>> Without libnuma, we have no way of "asking" the kernel to allocate a hugepage on a specific socket - instead, any allocation will most likely happen on socket from which the allocation came from. For example, if user program's lcore is on socket 1, allocation on socket 0 will actually allocate a page on socket 1.
>>>>>>
>>>>>> If we don't check for page's NUMA node affinity (which is what currently happens) - we get performance degradation because we may unintentionally allocate memory on wrong NUMA node. If we do check for this - then allocation of memory on socket 1 from lcore on socket 0 will almost never succeed, because kernel will always give us pages on socket 0.
>>>>>>
>>>>>> Put it simply, there is no sane way to make this option work for the new memory subsystem - IMO it should be dropped, and libnuma should be made a hard dependency on Linux.
>>>>>
>>>>> I agree that new memory model could not work without libnuma, i.e. will
>>>>> lead to unpredictable memory allocations with no any respect to requested
>>>>> socket_id's. I also agree that CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES is only
>>>>> sane for a legacy memory model.
>>>>> It looks like we have no other choice than just drop the option and make
>>>>> the code unconditional, i.e. have hard dependency on libnuma.
>>>>>
>>>>
>>>> We, probably, could compile this code and have hard dependency only for
>>>> platforms with 'RTE_MAX_NUMA_NODES > 1'.
>>>
>>> Well, as long as legacy mode stays supported, we have to keep the option. The "drop" part was referring to supporting it under the new memory system, not a literal drop from config files.
>>
>> The option was introduced because we didn't want to introduce the
>> new hard dependency. Since we'll have it anyway, I'm not sure if
>> keeping the option for legacy mode makes any sense.
> 
> Oh yes, you're right. Drop it is!
> 
>>
>>>
>>> As for using RTE_MAX_NUMA_NODES, i don't think it's merited. Distributions cannot deliver different DPDK versions based on the number of sockets on a particular machine - so it would have to be a hard dependency for distributions anyway (does any distribution ship DPDK without libnuma?).
>>
>> At least ARMv7 builds commonly does not ship libnuma package.
> 
> Do you mean libnuma builds for ARMv7 are not available? Or do you mean the libnuma package is not installed by default?
> 
> If it's the latter, then i believe it's not installed by default anywhere, but if using distribution version of DPDK, libnuma will be taken care of via package manager. Presumably building from source can be taken care of with pkg-config/meson.
> 
> Or do you mean ARMv7 does not have libnuma for their arch at all, in any distro?

libnuma builds for ARMv7 are not available in most of the distros. I didn't check all,
but here is results for Ubuntu:
    https://packages.ubuntu.com/search?suite=bionic&arch=armhf&searchon=names&keywords=libnuma

You may see that Ubuntu 18.04 (bionic) has no libnuma package for 'armhf' and
also 'powerpc' platforms.

> 
>>
>>>
>>> For those compiling from source - are there any supported distributions which don't package libnuma? I don't see much sense in keeping libnuma optional, IMO. This is of course up to the tech board to decide, but IMO the "without libnuma it's basically broken" argument is very strong in my opinion :)
>>>
>>
> 
> 

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Burakov, Anatoly]

On 26-Nov-18 2:32 PM, Ilya Maximets wrote:
> On 26.11.2018 17:21, Burakov, Anatoly wrote:
>> On 26-Nov-18 2:10 PM, Ilya Maximets wrote:
>>> On 26.11.2018 16:42, Burakov, Anatoly wrote:
>>>> On 26-Nov-18 1:20 PM, Ilya Maximets wrote:
>>>>> On 26.11.2018 16:16, Ilya Maximets wrote:
>>>>>> On 26.11.2018 15:50, Burakov, Anatoly wrote:
>>>>>>> On 26-Nov-18 11:43 AM, Burakov, Anatoly wrote:
>>>>>>>> On 26-Nov-18 11:33 AM, Asaf Sinai wrote:
>>>>>>>>> Hi Anatoly,
>>>>>>>>>
>>>>>>>>> We did not check it with "testpmd", only with our application.
>>>>>>>>>     From the beginning, we did not enable this configuration (look at attached files), and everything works fine.
>>>>>>>>> Of course we rebuild DPDK, when we change configuration.
>>>>>>>>> Please note that we use DPDK 17.11.3, maybe this is why it works fine?
>>>>>>>>
>>>>>>>> Just tested with DPDK 17.11, and yes, it does work the way you are describing. This is not intended behavior. I will look into it.
>>>>>>>>
>>>>>>>
>>>>>>> +CC author of commit introducing CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES.
>>>>>>>
>>>>>>> Looking at the code, i think this config option needs to be reworked and we should clarify what we mean by this option. It appears that i've misunderstood what this option actually intended to do, and i also think it's naming could be improved because it's confusing and misleading.
>>>>>>>
>>>>>>> In 17.11, this option does *not* prevent EAL from using NUMA - it merely disables using libnuma to perform memory allocation. This looks like intended (if counter-intuitive) behavior - disabling this option will simply revert DPDK to working as it did before this option was introduced (i.e. best-effort allocation). This is why your code still works - because EAL still does allocate memory on socket 1, and *knows* that it's socket 1 memory. It still supports NUMA.
>>>>>>>
>>>>>>> The commit message for these changes states that the actual purpose of this option is to enable "balanced" hugepage allocation. In case of cgroups limitations, previously, DPDK would've exhausted all hugepages on master core's socket before attempting to allocate from other sockets, but by the time we've reached cgroups limits on numbers of hugepages, we might not have reached socket 1 and thus missed out on the pages we could've allocated, but didn't. Using libnuma solves this issue, because now we can allocate pages on sockets we want, instead of hoping we won't run out of hugepages before we get the memory we need.
>>>>>>>
>>>>>>> In 18.05 onwards, this option works differently (and arguably wrong). More specifically, it disallows allocations on sockets other than 0, and it also makes it so that EAL does not check which socket the memory *actually* came from. So, not only allocating memory from socket 1 is disabled, but allocating from socket 0 may even get you memory from socket 1!
>>>>>>
>>>>>> I'd consider this as a bug.
>>>>>>
>>>>>>>
>>>>>>> +CC Thomas
>>>>>>>
>>>>>>> The CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES option is a misnomer, because it makes it seem like this option disables NUMA support, which is not the case.
>>>>>>>
>>>>>>> I would also argue that it is not relevant to 18.05+ memory subsystem, and should only work in legacy mode, because it is *impossible* to make it work right in the new memory subsystem, and here's why:
>>>>>>>
>>>>>>> Without libnuma, we have no way of "asking" the kernel to allocate a hugepage on a specific socket - instead, any allocation will most likely happen on socket from which the allocation came from. For example, if user program's lcore is on socket 1, allocation on socket 0 will actually allocate a page on socket 1.
>>>>>>>
>>>>>>> If we don't check for page's NUMA node affinity (which is what currently happens) - we get performance degradation because we may unintentionally allocate memory on wrong NUMA node. If we do check for this - then allocation of memory on socket 1 from lcore on socket 0 will almost never succeed, because kernel will always give us pages on socket 0.
>>>>>>>
>>>>>>> Put it simply, there is no sane way to make this option work for the new memory subsystem - IMO it should be dropped, and libnuma should be made a hard dependency on Linux.
>>>>>>
>>>>>> I agree that new memory model could not work without libnuma, i.e. will
>>>>>> lead to unpredictable memory allocations with no any respect to requested
>>>>>> socket_id's. I also agree that CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES is only
>>>>>> sane for a legacy memory model.
>>>>>> It looks like we have no other choice than just drop the option and make
>>>>>> the code unconditional, i.e. have hard dependency on libnuma.
>>>>>>
>>>>>
>>>>> We, probably, could compile this code and have hard dependency only for
>>>>> platforms with 'RTE_MAX_NUMA_NODES > 1'.
>>>>
>>>> Well, as long as legacy mode stays supported, we have to keep the option. The "drop" part was referring to supporting it under the new memory system, not a literal drop from config files.
>>>
>>> The option was introduced because we didn't want to introduce the
>>> new hard dependency. Since we'll have it anyway, I'm not sure if
>>> keeping the option for legacy mode makes any sense.
>>
>> Oh yes, you're right. Drop it is!
>>
>>>
>>>>
>>>> As for using RTE_MAX_NUMA_NODES, i don't think it's merited. Distributions cannot deliver different DPDK versions based on the number of sockets on a particular machine - so it would have to be a hard dependency for distributions anyway (does any distribution ship DPDK without libnuma?).
>>>
>>> At least ARMv7 builds commonly does not ship libnuma package.
>>
>> Do you mean libnuma builds for ARMv7 are not available? Or do you mean the libnuma package is not installed by default?
>>
>> If it's the latter, then i believe it's not installed by default anywhere, but if using distribution version of DPDK, libnuma will be taken care of via package manager. Presumably building from source can be taken care of with pkg-config/meson.
>>
>> Or do you mean ARMv7 does not have libnuma for their arch at all, in any distro?
> 
> libnuma builds for ARMv7 are not available in most of the distros. I didn't check all,
> but here is results for Ubuntu:
>      https://packages.ubuntu.com/search?suite=bionic&arch=armhf&searchon=names&keywords=libnuma
> 
> You may see that Ubuntu 18.04 (bionic) has no libnuma package for 'armhf' and
> also 'powerpc' platforms.
> 

That's a difficulty. Do these platforms support NUMA? In other words, 
could we replace this flag with just outright disabling NUMA support?

>>
>>>
>>>>
>>>> For those compiling from source - are there any supported distributions which don't package libnuma? I don't see much sense in keeping libnuma optional, IMO. This is of course up to the tech board to decide, but IMO the "without libnuma it's basically broken" argument is very strong in my opinion :)
>>>>
>>>
>>
>>
> 


-- 
Thanks,
Anatoly

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Asaf Sinai]

+CC Ilia & Sasha.

-----Original Message-----
From: Burakov, Anatoly <anatoly.burakov@intel.com> 
Sent: Monday, November 26, 2018 04:57 PM
To: Ilya Maximets <i.maximets@samsung.com>; Asaf Sinai <AsafSi@Radware.com>; dev@dpdk.org; Thomas Monjalon <thomas@monjalon.net>
Subject: Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

On 26-Nov-18 2:32 PM, Ilya Maximets wrote:
> On 26.11.2018 17:21, Burakov, Anatoly wrote:
>> On 26-Nov-18 2:10 PM, Ilya Maximets wrote:
>>> On 26.11.2018 16:42, Burakov, Anatoly wrote:
>>>> On 26-Nov-18 1:20 PM, Ilya Maximets wrote:
>>>>> On 26.11.2018 16:16, Ilya Maximets wrote:
>>>>>> On 26.11.2018 15:50, Burakov, Anatoly wrote:
>>>>>>> On 26-Nov-18 11:43 AM, Burakov, Anatoly wrote:
>>>>>>>> On 26-Nov-18 11:33 AM, Asaf Sinai wrote:
>>>>>>>>> Hi Anatoly,
>>>>>>>>>
>>>>>>>>> We did not check it with "testpmd", only with our application.
>>>>>>>>>     From the beginning, we did not enable this configuration (look at attached files), and everything works fine.
>>>>>>>>> Of course we rebuild DPDK, when we change configuration.
>>>>>>>>> Please note that we use DPDK 17.11.3, maybe this is why it works fine?
>>>>>>>>
>>>>>>>> Just tested with DPDK 17.11, and yes, it does work the way you are describing. This is not intended behavior. I will look into it.
>>>>>>>>
>>>>>>>
>>>>>>> +CC author of commit introducing CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES.
>>>>>>>
>>>>>>> Looking at the code, i think this config option needs to be reworked and we should clarify what we mean by this option. It appears that i've misunderstood what this option actually intended to do, and i also think it's naming could be improved because it's confusing and misleading.
>>>>>>>
>>>>>>> In 17.11, this option does *not* prevent EAL from using NUMA - it merely disables using libnuma to perform memory allocation. This looks like intended (if counter-intuitive) behavior - disabling this option will simply revert DPDK to working as it did before this option was introduced (i.e. best-effort allocation). This is why your code still works - because EAL still does allocate memory on socket 1, and *knows* that it's socket 1 memory. It still supports NUMA.
>>>>>>>
>>>>>>> The commit message for these changes states that the actual purpose of this option is to enable "balanced" hugepage allocation. In case of cgroups limitations, previously, DPDK would've exhausted all hugepages on master core's socket before attempting to allocate from other sockets, but by the time we've reached cgroups limits on numbers of hugepages, we might not have reached socket 1 and thus missed out on the pages we could've allocated, but didn't. Using libnuma solves this issue, because now we can allocate pages on sockets we want, instead of hoping we won't run out of hugepages before we get the memory we need.
>>>>>>>
>>>>>>> In 18.05 onwards, this option works differently (and arguably wrong). More specifically, it disallows allocations on sockets other than 0, and it also makes it so that EAL does not check which socket the memory *actually* came from. So, not only allocating memory from socket 1 is disabled, but allocating from socket 0 may even get you memory from socket 1!
>>>>>>
>>>>>> I'd consider this as a bug.
>>>>>>
>>>>>>>
>>>>>>> +CC Thomas
>>>>>>>
>>>>>>> The CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES option is a misnomer, because it makes it seem like this option disables NUMA support, which is not the case.
>>>>>>>
>>>>>>> I would also argue that it is not relevant to 18.05+ memory subsystem, and should only work in legacy mode, because it is *impossible* to make it work right in the new memory subsystem, and here's why:
>>>>>>>
>>>>>>> Without libnuma, we have no way of "asking" the kernel to allocate a hugepage on a specific socket - instead, any allocation will most likely happen on socket from which the allocation came from. For example, if user program's lcore is on socket 1, allocation on socket 0 will actually allocate a page on socket 1.
>>>>>>>
>>>>>>> If we don't check for page's NUMA node affinity (which is what currently happens) - we get performance degradation because we may unintentionally allocate memory on wrong NUMA node. If we do check for this - then allocation of memory on socket 1 from lcore on socket 0 will almost never succeed, because kernel will always give us pages on socket 0.
>>>>>>>
>>>>>>> Put it simply, there is no sane way to make this option work for the new memory subsystem - IMO it should be dropped, and libnuma should be made a hard dependency on Linux.
>>>>>>
>>>>>> I agree that new memory model could not work without libnuma, 
>>>>>> i.e. will lead to unpredictable memory allocations with no any 
>>>>>> respect to requested socket_id's. I also agree that 
>>>>>> CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES is only sane for a legacy memory model.
>>>>>> It looks like we have no other choice than just drop the option 
>>>>>> and make the code unconditional, i.e. have hard dependency on libnuma.
>>>>>>
>>>>>
>>>>> We, probably, could compile this code and have hard dependency 
>>>>> only for platforms with 'RTE_MAX_NUMA_NODES > 1'.
>>>>
>>>> Well, as long as legacy mode stays supported, we have to keep the option. The "drop" part was referring to supporting it under the new memory system, not a literal drop from config files.
>>>
>>> The option was introduced because we didn't want to introduce the 
>>> new hard dependency. Since we'll have it anyway, I'm not sure if 
>>> keeping the option for legacy mode makes any sense.
>>
>> Oh yes, you're right. Drop it is!
>>
>>>
>>>>
>>>> As for using RTE_MAX_NUMA_NODES, i don't think it's merited. Distributions cannot deliver different DPDK versions based on the number of sockets on a particular machine - so it would have to be a hard dependency for distributions anyway (does any distribution ship DPDK without libnuma?).
>>>
>>> At least ARMv7 builds commonly does not ship libnuma package.
>>
>> Do you mean libnuma builds for ARMv7 are not available? Or do you mean the libnuma package is not installed by default?
>>
>> If it's the latter, then i believe it's not installed by default anywhere, but if using distribution version of DPDK, libnuma will be taken care of via package manager. Presumably building from source can be taken care of with pkg-config/meson.
>>
>> Or do you mean ARMv7 does not have libnuma for their arch at all, in any distro?
> 
> libnuma builds for ARMv7 are not available in most of the distros. I 
> didn't check all, but here is results for Ubuntu:
>      
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpac
> kages.ubuntu.com%2Fsearch%3Fsuite%3Dbionic%26arch%3Darmhf%26searchon%3
> Dnames%26keywords%3Dlibnuma&amp;data=02%7C01%7CAsafSi%40radware.com%7C
> a44f84bca42d4a52acac08d653af83b8%7C6ae4e000b5d04f48a766402d46119b76%7C
> 0%7C0%7C636788410626179927&amp;sdata=1pJ0WkAs6Y%2Bv3w%2BhKAELBw%2BjMra
> BnhiqqpsXkRv2ifI%3D&amp;reserved=0
> 
> You may see that Ubuntu 18.04 (bionic) has no libnuma package for 
> 'armhf' and also 'powerpc' platforms.
> 

That's a difficulty. Do these platforms support NUMA? In other words, could we replace this flag with just outright disabling NUMA support?

>>
>>>
>>>>
>>>> For those compiling from source - are there any supported 
>>>> distributions which don't package libnuma? I don't see much sense 
>>>> in keeping libnuma optional, IMO. This is of course up to the tech 
>>>> board to decide, but IMO the "without libnuma it's basically 
>>>> broken" argument is very strong in my opinion :)
>>>>
>>>
>>
>>
> 


--
Thanks,
Anatoly

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Hemant Agrawal]

On 11/26/2018 8:55 PM, Asaf Sinai wrote:
> +CC Ilia & Sasha.
>
> -----Original Message-----
> From: Burakov, Anatoly <anatoly.burakov@intel.com>
> Sent: Monday, November 26, 2018 04:57 PM
> To: Ilya Maximets <i.maximets@samsung.com>; Asaf Sinai <AsafSi@Radware.com>; dev@dpdk.org; Thomas Monjalon <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration
>
> On 26-Nov-18 2:32 PM, Ilya Maximets wrote:
>> On 26.11.2018 17:21, Burakov, Anatoly wrote:
>>> On 26-Nov-18 2:10 PM, Ilya Maximets wrote:
>>>> On 26.11.2018 16:42, Burakov, Anatoly wrote:
>>>>> On 26-Nov-18 1:20 PM, Ilya Maximets wrote:
>>>>>> On 26.11.2018 16:16, Ilya Maximets wrote:
>>>>>>> On 26.11.2018 15:50, Burakov, Anatoly wrote:
>>>>>>>> On 26-Nov-18 11:43 AM, Burakov, Anatoly wrote:
>>>>>>>>> On 26-Nov-18 11:33 AM, Asaf Sinai wrote:
>>>>>>>>>> Hi Anatoly,
>>>>>>>>>>
>>>>>>>>>> We did not check it with "testpmd", only with our application.
>>>>>>>>>>      From the beginning, we did not enable this configuration (look at attached files), and everything works fine.
>>>>>>>>>> Of course we rebuild DPDK, when we change configuration.
>>>>>>>>>> Please note that we use DPDK 17.11.3, maybe this is why it works fine?
>>>>>>>>> Just tested with DPDK 17.11, and yes, it does work the way you are describing. This is not intended behavior. I will look into it.
>>>>>>>>>
>>>>>>>> +CC author of commit introducing CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES.
>>>>>>>>
>>>>>>>> Looking at the code, i think this config option needs to be reworked and we should clarify what we mean by this option. It appears that i've misunderstood what this option actually intended to do, and i also think it's naming could be improved because it's confusing and misleading.
>>>>>>>>
>>>>>>>> In 17.11, this option does *not* prevent EAL from using NUMA - it merely disables using libnuma to perform memory allocation. This looks like intended (if counter-intuitive) behavior - disabling this option will simply revert DPDK to working as it did before this option was introduced (i.e. best-effort allocation). This is why your code still works - because EAL still does allocate memory on socket 1, and *knows* that it's socket 1 memory. It still supports NUMA.
>>>>>>>>
>>>>>>>> The commit message for these changes states that the actual purpose of this option is to enable "balanced" hugepage allocation. In case of cgroups limitations, previously, DPDK would've exhausted all hugepages on master core's socket before attempting to allocate from other sockets, but by the time we've reached cgroups limits on numbers of hugepages, we might not have reached socket 1 and thus missed out on the pages we could've allocated, but didn't. Using libnuma solves this issue, because now we can allocate pages on sockets we want, instead of hoping we won't run out of hugepages before we get the memory we need.
>>>>>>>>
>>>>>>>> In 18.05 onwards, this option works differently (and arguably wrong). More specifically, it disallows allocations on sockets other than 0, and it also makes it so that EAL does not check which socket the memory *actually* came from. So, not only allocating memory from socket 1 is disabled, but allocating from socket 0 may even get you memory from socket 1!
>>>>>>> I'd consider this as a bug.
>>>>>>>
>>>>>>>> +CC Thomas
>>>>>>>>
>>>>>>>> The CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES option is a misnomer, because it makes it seem like this option disables NUMA support, which is not the case.
>>>>>>>>
>>>>>>>> I would also argue that it is not relevant to 18.05+ memory subsystem, and should only work in legacy mode, because it is *impossible* to make it work right in the new memory subsystem, and here's why:
>>>>>>>>
>>>>>>>> Without libnuma, we have no way of "asking" the kernel to allocate a hugepage on a specific socket - instead, any allocation will most likely happen on socket from which the allocation came from. For example, if user program's lcore is on socket 1, allocation on socket 0 will actually allocate a page on socket 1.
>>>>>>>>
>>>>>>>> If we don't check for page's NUMA node affinity (which is what currently happens) - we get performance degradation because we may unintentionally allocate memory on wrong NUMA node. If we do check for this - then allocation of memory on socket 1 from lcore on socket 0 will almost never succeed, because kernel will always give us pages on socket 0.
>>>>>>>>
>>>>>>>> Put it simply, there is no sane way to make this option work for the new memory subsystem - IMO it should be dropped, and libnuma should be made a hard dependency on Linux.
>>>>>>> I agree that new memory model could not work without libnuma,
>>>>>>> i.e. will lead to unpredictable memory allocations with no any
>>>>>>> respect to requested socket_id's. I also agree that
>>>>>>> CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES is only sane for a legacy memory model.
>>>>>>> It looks like we have no other choice than just drop the option
>>>>>>> and make the code unconditional, i.e. have hard dependency on libnuma.
>>>>>>>
>>>>>> We, probably, could compile this code and have hard dependency
>>>>>> only for platforms with 'RTE_MAX_NUMA_NODES > 1'.
>>>>> Well, as long as legacy mode stays supported, we have to keep the option. The "drop" part was referring to supporting it under the new memory system, not a literal drop from config files.
>>>> The option was introduced because we didn't want to introduce the
>>>> new hard dependency. Since we'll have it anyway, I'm not sure if
>>>> keeping the option for legacy mode makes any sense.
>>> Oh yes, you're right. Drop it is!
>>>
>>>>> As for using RTE_MAX_NUMA_NODES, i don't think it's merited. Distributions cannot deliver different DPDK versions based on the number of sockets on a particular machine - so it would have to be a hard dependency for distributions anyway (does any distribution ship DPDK without libnuma?).
>>>> At least ARMv7 builds commonly does not ship libnuma package.
>>> Do you mean libnuma builds for ARMv7 are not available? Or do you mean the libnuma package is not installed by default?
>>>
>>> If it's the latter, then i believe it's not installed by default anywhere, but if using distribution version of DPDK, libnuma will be taken care of via package manager. Presumably building from source can be taken care of with pkg-config/meson.
>>>
>>> Or do you mean ARMv7 does not have libnuma for their arch at all, in any distro?
>> libnuma builds for ARMv7 are not available in most of the distros. I
>> didn't check all, but here is results for Ubuntu:
>>       
>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpac
>> kages.ubuntu.com%2Fsearch%3Fsuite%3Dbionic%26arch%3Darmhf%26searchon%3
>> Dnames%26keywords%3Dlibnuma&amp;data=02%7C01%7CAsafSi%40radware.com%7C
>> a44f84bca42d4a52acac08d653af83b8%7C6ae4e000b5d04f48a766402d46119b76%7C
>> 0%7C0%7C636788410626179927&amp;sdata=1pJ0WkAs6Y%2Bv3w%2BhKAELBw%2BjMra
>> BnhiqqpsXkRv2ifI%3D&amp;reserved=0
>>
>> You may see that Ubuntu 18.04 (bionic) has no libnuma package for
>> 'armhf' and also 'powerpc' platforms.
>>
> That's a difficulty. Do these platforms support NUMA? In other words, could we replace this flag with just outright disabling NUMA support?

Many platforms don't support NUMA, so they dont' really need libnuma.

Mandating libnuma will also break several things:

   - cross build for ARM on x86 - which is among the preferred method 
for build by many in ARM community.

  - many of the embedded SoCs are without NUMA support, they use smaller 
rootf (e.g. Yocto).  It will be a burden to add libnuma there.


>
>>>>> For those compiling from source - are there any supported
>>>>> distributions which don't package libnuma? I don't see much sense
>>>>> in keeping libnuma optional, IMO. This is of course up to the tech
>>>>> board to decide, but IMO the "without libnuma it's basically
>>>>> broken" argument is very strong in my opinion :)
>>>>>
>>>
>
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Burakov, Anatoly]

On 27-Nov-18 10:26 AM, Hemant Agrawal wrote:
> 
> On 11/26/2018 8:55 PM, Asaf Sinai wrote:
>> +CC Ilia & Sasha.
>>
>> -----Original Message-----
>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>> Sent: Monday, November 26, 2018 04:57 PM
>> To: Ilya Maximets <i.maximets@samsung.com>; Asaf Sinai <AsafSi@Radware.com>; dev@dpdk.org; Thomas Monjalon <thomas@monjalon.net>
>> Subject: Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration
>>
>> On 26-Nov-18 2:32 PM, Ilya Maximets wrote:
>>> On 26.11.2018 17:21, Burakov, Anatoly wrote:
>>>> On 26-Nov-18 2:10 PM, Ilya Maximets wrote:
>>>>> On 26.11.2018 16:42, Burakov, Anatoly wrote:
>>>>>> On 26-Nov-18 1:20 PM, Ilya Maximets wrote:
>>>>>>> On 26.11.2018 16:16, Ilya Maximets wrote:
>>>>>>>> On 26.11.2018 15:50, Burakov, Anatoly wrote:
>>>>>>>>> On 26-Nov-18 11:43 AM, Burakov, Anatoly wrote:
>>>>>>>>>> On 26-Nov-18 11:33 AM, Asaf Sinai wrote:
>>>>>>>>>>> Hi Anatoly,
>>>>>>>>>>>
>>>>>>>>>>> We did not check it with "testpmd", only with our application.
>>>>>>>>>>>       From the beginning, we did not enable this configuration (look at attached files), and everything works fine.
>>>>>>>>>>> Of course we rebuild DPDK, when we change configuration.
>>>>>>>>>>> Please note that we use DPDK 17.11.3, maybe this is why it works fine?
>>>>>>>>>> Just tested with DPDK 17.11, and yes, it does work the way you are describing. This is not intended behavior. I will look into it.
>>>>>>>>>>
>>>>>>>>> +CC author of commit introducing CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES.
>>>>>>>>>
>>>>>>>>> Looking at the code, i think this config option needs to be reworked and we should clarify what we mean by this option. It appears that i've misunderstood what this option actually intended to do, and i also think it's naming could be improved because it's confusing and misleading.
>>>>>>>>>
>>>>>>>>> In 17.11, this option does *not* prevent EAL from using NUMA - it merely disables using libnuma to perform memory allocation. This looks like intended (if counter-intuitive) behavior - disabling this option will simply revert DPDK to working as it did before this option was introduced (i.e. best-effort allocation). This is why your code still works - because EAL still does allocate memory on socket 1, and *knows* that it's socket 1 memory. It still supports NUMA.
>>>>>>>>>
>>>>>>>>> The commit message for these changes states that the actual purpose of this option is to enable "balanced" hugepage allocation. In case of cgroups limitations, previously, DPDK would've exhausted all hugepages on master core's socket before attempting to allocate from other sockets, but by the time we've reached cgroups limits on numbers of hugepages, we might not have reached socket 1 and thus missed out on the pages we could've allocated, but didn't. Using libnuma solves this issue, because now we can allocate pages on sockets we want, instead of hoping we won't run out of hugepages before we get the memory we need.
>>>>>>>>>
>>>>>>>>> In 18.05 onwards, this option works differently (and arguably wrong). More specifically, it disallows allocations on sockets other than 0, and it also makes it so that EAL does not check which socket the memory *actually* came from. So, not only allocating memory from socket 1 is disabled, but allocating from socket 0 may even get you memory from socket 1!
>>>>>>>> I'd consider this as a bug.
>>>>>>>>
>>>>>>>>> +CC Thomas
>>>>>>>>>
>>>>>>>>> The CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES option is a misnomer, because it makes it seem like this option disables NUMA support, which is not the case.
>>>>>>>>>
>>>>>>>>> I would also argue that it is not relevant to 18.05+ memory subsystem, and should only work in legacy mode, because it is *impossible* to make it work right in the new memory subsystem, and here's why:
>>>>>>>>>
>>>>>>>>> Without libnuma, we have no way of "asking" the kernel to allocate a hugepage on a specific socket - instead, any allocation will most likely happen on socket from which the allocation came from. For example, if user program's lcore is on socket 1, allocation on socket 0 will actually allocate a page on socket 1.
>>>>>>>>>
>>>>>>>>> If we don't check for page's NUMA node affinity (which is what currently happens) - we get performance degradation because we may unintentionally allocate memory on wrong NUMA node. If we do check for this - then allocation of memory on socket 1 from lcore on socket 0 will almost never succeed, because kernel will always give us pages on socket 0.
>>>>>>>>>
>>>>>>>>> Put it simply, there is no sane way to make this option work for the new memory subsystem - IMO it should be dropped, and libnuma should be made a hard dependency on Linux.
>>>>>>>> I agree that new memory model could not work without libnuma,
>>>>>>>> i.e. will lead to unpredictable memory allocations with no any
>>>>>>>> respect to requested socket_id's. I also agree that
>>>>>>>> CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES is only sane for a legacy memory model.
>>>>>>>> It looks like we have no other choice than just drop the option
>>>>>>>> and make the code unconditional, i.e. have hard dependency on libnuma.
>>>>>>>>
>>>>>>> We, probably, could compile this code and have hard dependency
>>>>>>> only for platforms with 'RTE_MAX_NUMA_NODES > 1'.
>>>>>> Well, as long as legacy mode stays supported, we have to keep the option. The "drop" part was referring to supporting it under the new memory system, not a literal drop from config files.
>>>>> The option was introduced because we didn't want to introduce the
>>>>> new hard dependency. Since we'll have it anyway, I'm not sure if
>>>>> keeping the option for legacy mode makes any sense.
>>>> Oh yes, you're right. Drop it is!
>>>>
>>>>>> As for using RTE_MAX_NUMA_NODES, i don't think it's merited. Distributions cannot deliver different DPDK versions based on the number of sockets on a particular machine - so it would have to be a hard dependency for distributions anyway (does any distribution ship DPDK without libnuma?).
>>>>> At least ARMv7 builds commonly does not ship libnuma package.
>>>> Do you mean libnuma builds for ARMv7 are not available? Or do you mean the libnuma package is not installed by default?
>>>>
>>>> If it's the latter, then i believe it's not installed by default anywhere, but if using distribution version of DPDK, libnuma will be taken care of via package manager. Presumably building from source can be taken care of with pkg-config/meson.
>>>>
>>>> Or do you mean ARMv7 does not have libnuma for their arch at all, in any distro?
>>> libnuma builds for ARMv7 are not available in most of the distros. I
>>> didn't check all, but here is results for Ubuntu:
>>>        
>>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpac
>>> kages.ubuntu.com%2Fsearch%3Fsuite%3Dbionic%26arch%3Darmhf%26searchon%3
>>> Dnames%26keywords%3Dlibnuma&amp;data=02%7C01%7CAsafSi%40radware.com%7C
>>> a44f84bca42d4a52acac08d653af83b8%7C6ae4e000b5d04f48a766402d46119b76%7C
>>> 0%7C0%7C636788410626179927&amp;sdata=1pJ0WkAs6Y%2Bv3w%2BhKAELBw%2BjMra
>>> BnhiqqpsXkRv2ifI%3D&amp;reserved=0
>>>
>>> You may see that Ubuntu 18.04 (bionic) has no libnuma package for
>>> 'armhf' and also 'powerpc' platforms.
>>>
>> That's a difficulty. Do these platforms support NUMA? In other words, could we replace this flag with just outright disabling NUMA support?
> 
> Many platforms don't support NUMA, so they dont' really need libnuma.
> 
> Mandating libnuma will also break several things:
> 
>     - cross build for ARM on x86 - which is among the preferred method
> for build by many in ARM community.
> 
>    - many of the embedded SoCs are without NUMA support, they use smaller
> rootf (e.g. Yocto).  It will be a burden to add libnuma there.
> 

OK, point taken.

So, the alternative would be to have the ability to outright disable 
NUMA support (either with a new option, or reworking this one - i would 
prefer a new one, since this one is confusingly named). Meaning, report 
all cores as socket 0, report all hardware as socket 0, report all 
memory as socket 0 and never care about NUMA nodes anywhere.

Would that work? E.g. by default, make libnuma a hard dependency on x86 
Linux (but allow to disable it), but disable it everywhere else?

> 
>>
>>>>>> For those compiling from source - are there any supported
>>>>>> distributions which don't package libnuma? I don't see much sense
>>>>>> in keeping libnuma optional, IMO. This is of course up to the tech
>>>>>> board to decide, but IMO the "without libnuma it's basically
>>>>>> broken" argument is very strong in my opinion :)
>>>>>>
>>>>
>>
>> --
>> Thanks,
>> Anatoly


-- 
Thanks,
Anatoly

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Ilya Maximets]

On 27.11.2018 13:33, Burakov, Anatoly wrote:
> On 27-Nov-18 10:26 AM, Hemant Agrawal wrote:
>>
>> On 11/26/2018 8:55 PM, Asaf Sinai wrote:
>>> +CC Ilia & Sasha.
>>>
>>> -----Original Message-----
>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>>> Sent: Monday, November 26, 2018 04:57 PM
>>> To: Ilya Maximets <i.maximets@samsung.com>; Asaf Sinai <AsafSi@Radware.com>; dev@dpdk.org; Thomas Monjalon <thomas@monjalon.net>
>>> Subject: Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration
>>>
>>> On 26-Nov-18 2:32 PM, Ilya Maximets wrote:
>>>> On 26.11.2018 17:21, Burakov, Anatoly wrote:
>>>>> On 26-Nov-18 2:10 PM, Ilya Maximets wrote:
>>>>>> On 26.11.2018 16:42, Burakov, Anatoly wrote:
>>>>>>> On 26-Nov-18 1:20 PM, Ilya Maximets wrote:
>>>>>>>> On 26.11.2018 16:16, Ilya Maximets wrote:
>>>>>>>>> On 26.11.2018 15:50, Burakov, Anatoly wrote:
>>>>>>>>>> On 26-Nov-18 11:43 AM, Burakov, Anatoly wrote:
>>>>>>>>>>> On 26-Nov-18 11:33 AM, Asaf Sinai wrote:
>>>>>>>>>>>> Hi Anatoly,
>>>>>>>>>>>>
>>>>>>>>>>>> We did not check it with "testpmd", only with our application.
>>>>>>>>>>>>       From the beginning, we did not enable this configuration (look at attached files), and everything works fine.
>>>>>>>>>>>> Of course we rebuild DPDK, when we change configuration.
>>>>>>>>>>>> Please note that we use DPDK 17.11.3, maybe this is why it works fine?
>>>>>>>>>>> Just tested with DPDK 17.11, and yes, it does work the way you are describing. This is not intended behavior. I will look into it.
>>>>>>>>>>>
>>>>>>>>>> +CC author of commit introducing CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES.
>>>>>>>>>>
>>>>>>>>>> Looking at the code, i think this config option needs to be reworked and we should clarify what we mean by this option. It appears that i've misunderstood what this option actually intended to do, and i also think it's naming could be improved because it's confusing and misleading.
>>>>>>>>>>
>>>>>>>>>> In 17.11, this option does *not* prevent EAL from using NUMA - it merely disables using libnuma to perform memory allocation. This looks like intended (if counter-intuitive) behavior - disabling this option will simply revert DPDK to working as it did before this option was introduced (i.e. best-effort allocation). This is why your code still works - because EAL still does allocate memory on socket 1, and *knows* that it's socket 1 memory. It still supports NUMA.
>>>>>>>>>>
>>>>>>>>>> The commit message for these changes states that the actual purpose of this option is to enable "balanced" hugepage allocation. In case of cgroups limitations, previously, DPDK would've exhausted all hugepages on master core's socket before attempting to allocate from other sockets, but by the time we've reached cgroups limits on numbers of hugepages, we might not have reached socket 1 and thus missed out on the pages we could've allocated, but didn't. Using libnuma solves this issue, because now we can allocate pages on sockets we want, instead of hoping we won't run out of hugepages before we get the memory we need.
>>>>>>>>>>
>>>>>>>>>> In 18.05 onwards, this option works differently (and arguably wrong). More specifically, it disallows allocations on sockets other than 0, and it also makes it so that EAL does not check which socket the memory *actually* came from. So, not only allocating memory from socket 1 is disabled, but allocating from socket 0 may even get you memory from socket 1!
>>>>>>>>> I'd consider this as a bug.
>>>>>>>>>
>>>>>>>>>> +CC Thomas
>>>>>>>>>>
>>>>>>>>>> The CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES option is a misnomer, because it makes it seem like this option disables NUMA support, which is not the case.
>>>>>>>>>>
>>>>>>>>>> I would also argue that it is not relevant to 18.05+ memory subsystem, and should only work in legacy mode, because it is *impossible* to make it work right in the new memory subsystem, and here's why:
>>>>>>>>>>
>>>>>>>>>> Without libnuma, we have no way of "asking" the kernel to allocate a hugepage on a specific socket - instead, any allocation will most likely happen on socket from which the allocation came from. For example, if user program's lcore is on socket 1, allocation on socket 0 will actually allocate a page on socket 1.
>>>>>>>>>>
>>>>>>>>>> If we don't check for page's NUMA node affinity (which is what currently happens) - we get performance degradation because we may unintentionally allocate memory on wrong NUMA node. If we do check for this - then allocation of memory on socket 1 from lcore on socket 0 will almost never succeed, because kernel will always give us pages on socket 0.
>>>>>>>>>>
>>>>>>>>>> Put it simply, there is no sane way to make this option work for the new memory subsystem - IMO it should be dropped, and libnuma should be made a hard dependency on Linux.
>>>>>>>>> I agree that new memory model could not work without libnuma,
>>>>>>>>> i.e. will lead to unpredictable memory allocations with no any
>>>>>>>>> respect to requested socket_id's. I also agree that
>>>>>>>>> CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES is only sane for a legacy memory model.
>>>>>>>>> It looks like we have no other choice than just drop the option
>>>>>>>>> and make the code unconditional, i.e. have hard dependency on libnuma.
>>>>>>>>>
>>>>>>>> We, probably, could compile this code and have hard dependency
>>>>>>>> only for platforms with 'RTE_MAX_NUMA_NODES > 1'.
>>>>>>> Well, as long as legacy mode stays supported, we have to keep the option. The "drop" part was referring to supporting it under the new memory system, not a literal drop from config files.
>>>>>> The option was introduced because we didn't want to introduce the
>>>>>> new hard dependency. Since we'll have it anyway, I'm not sure if
>>>>>> keeping the option for legacy mode makes any sense.
>>>>> Oh yes, you're right. Drop it is!
>>>>>
>>>>>>> As for using RTE_MAX_NUMA_NODES, i don't think it's merited. Distributions cannot deliver different DPDK versions based on the number of sockets on a particular machine - so it would have to be a hard dependency for distributions anyway (does any distribution ship DPDK without libnuma?).
>>>>>> At least ARMv7 builds commonly does not ship libnuma package.
>>>>> Do you mean libnuma builds for ARMv7 are not available? Or do you mean the libnuma package is not installed by default?
>>>>>
>>>>> If it's the latter, then i believe it's not installed by default anywhere, but if using distribution version of DPDK, libnuma will be taken care of via package manager. Presumably building from source can be taken care of with pkg-config/meson.
>>>>>
>>>>> Or do you mean ARMv7 does not have libnuma for their arch at all, in any distro?
>>>> libnuma builds for ARMv7 are not available in most of the distros. I
>>>> didn't check all, but here is results for Ubuntu:
>>>>        https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpac
>>>> kages.ubuntu.com%2Fsearch%3Fsuite%3Dbionic%26arch%3Darmhf%26searchon%3
>>>> Dnames%26keywords%3Dlibnuma&amp;data=02%7C01%7CAsafSi%40radware.com%7C
>>>> a44f84bca42d4a52acac08d653af83b8%7C6ae4e000b5d04f48a766402d46119b76%7C
>>>> 0%7C0%7C636788410626179927&amp;sdata=1pJ0WkAs6Y%2Bv3w%2BhKAELBw%2BjMra
>>>> BnhiqqpsXkRv2ifI%3D&amp;reserved=0
>>>>
>>>> You may see that Ubuntu 18.04 (bionic) has no libnuma package for
>>>> 'armhf' and also 'powerpc' platforms.
>>>>
>>> That's a difficulty. Do these platforms support NUMA? In other words, could we replace this flag with just outright disabling NUMA support?
>>
>> Many platforms don't support NUMA, so they dont' really need libnuma.
>>
>> Mandating libnuma will also break several things:
>>
>>     - cross build for ARM on x86 - which is among the preferred method
>> for build by many in ARM community.
>>
>>    - many of the embedded SoCs are without NUMA support, they use smaller
>> rootf (e.g. Yocto).  It will be a burden to add libnuma there.
>>
> 
> OK, point taken.
> 
> So, the alternative would be to have the ability to outright disable NUMA support (either with a new option, or reworking this one - i would prefer a new one, since this one is confusingly named). Meaning, report all cores as socket 0, report all hardware as socket 0, report all memory as socket 0 and never care about NUMA nodes anywhere.
> 
> Would that work? E.g. by default, make libnuma a hard dependency on x86 Linux (but allow to disable it), but disable it everywhere else?

I think, you may just rename the RTE_EAL_NUMA_AWARE_HUGEPAGES to something
like RTE_EAL_NUMA_SUPPORT and keep all the defaults as is, i.e.
* globally disabled
* enabled for linux
* disabled for armv7a, dpaa, dpaa2 and stingray.
Meson could handle everything dynamically.

>>
>>>
>>>>>>> For those compiling from source - are there any supported
>>>>>>> distributions which don't package libnuma? I don't see much sense
>>>>>>> in keeping libnuma optional, IMO. This is of course up to the tech
>>>>>>> board to decide, but IMO the "without libnuma it's basically
>>>>>>> broken" argument is very strong in my opinion :)
>>>>>>>
>>>>>
>>>
>>> -- 
>>> Thanks,
>>> Anatoly
> 
> 

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Asaf Sinai]

Hi all,

Thanks for the detailed explanations!

So, what we understood from that, is the following (please correct, if it is wrong):
Before 18.05 version:
- Dividing huge pages between NUMAs was based, by default, on Linux good will.
- Enforcing Linux to divide huge pages between NUMAs, required enabling configuration option "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES".
- The enforcement was done via "libnuma" library.

From 18.05 version:
- The mentioned configuration option is ignored, so that by default, all huge pages are allocated on NUMA 0.
- if "libnuma" library exists in system, then huge pages will be divided between NUMAs, without any special configuration.
- The above is relevant to architectures that support NUMA, e.g. X86 (which we use).

Thanks,
Asaf

-----Original Message-----
From: Ilya Maximets <i.maximets@samsung.com> 
Sent: Tuesday, November 27, 2018 06:50 PM
To: Burakov, Anatoly <anatoly.burakov@intel.com>; Hemant Agrawal <hemant.agrawal@nxp.com>; Asaf Sinai <AsafSi@Radware.com>; dev@dpdk.org; Thomas Monjalon <thomas@monjalon.net>
Cc: Ilia Ferdman <IliaF@Radware.com>; Sasha Hodos <SashaH@Radware.com>
Subject: Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

On 27.11.2018 13:33, Burakov, Anatoly wrote:
> On 27-Nov-18 10:26 AM, Hemant Agrawal wrote:
>>
>> On 11/26/2018 8:55 PM, Asaf Sinai wrote:
>>> +CC Ilia & Sasha.
>>>
>>> -----Original Message-----
>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>>> Sent: Monday, November 26, 2018 04:57 PM
>>> To: Ilya Maximets <i.maximets@samsung.com>; Asaf Sinai 
>>> <AsafSi@Radware.com>; dev@dpdk.org; Thomas Monjalon 
>>> <thomas@monjalon.net>
>>> Subject: Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no 
>>> difference in memory pool allocations, when enabling/disabling this 
>>> configuration
>>>
>>> On 26-Nov-18 2:32 PM, Ilya Maximets wrote:
>>>> On 26.11.2018 17:21, Burakov, Anatoly wrote:
>>>>> On 26-Nov-18 2:10 PM, Ilya Maximets wrote:
>>>>>> On 26.11.2018 16:42, Burakov, Anatoly wrote:
>>>>>>> On 26-Nov-18 1:20 PM, Ilya Maximets wrote:
>>>>>>>> On 26.11.2018 16:16, Ilya Maximets wrote:
>>>>>>>>> On 26.11.2018 15:50, Burakov, Anatoly wrote:
>>>>>>>>>> On 26-Nov-18 11:43 AM, Burakov, Anatoly wrote:
>>>>>>>>>>> On 26-Nov-18 11:33 AM, Asaf Sinai wrote:
>>>>>>>>>>>> Hi Anatoly,
>>>>>>>>>>>>
>>>>>>>>>>>> We did not check it with "testpmd", only with our application.
>>>>>>>>>>>>       From the beginning, we did not enable this configuration (look at attached files), and everything works fine.
>>>>>>>>>>>> Of course we rebuild DPDK, when we change configuration.
>>>>>>>>>>>> Please note that we use DPDK 17.11.3, maybe this is why it works fine?
>>>>>>>>>>> Just tested with DPDK 17.11, and yes, it does work the way you are describing. This is not intended behavior. I will look into it.
>>>>>>>>>>>
>>>>>>>>>> +CC author of commit introducing CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES.
>>>>>>>>>>
>>>>>>>>>> Looking at the code, i think this config option needs to be reworked and we should clarify what we mean by this option. It appears that i've misunderstood what this option actually intended to do, and i also think it's naming could be improved because it's confusing and misleading.
>>>>>>>>>>
>>>>>>>>>> In 17.11, this option does *not* prevent EAL from using NUMA - it merely disables using libnuma to perform memory allocation. This looks like intended (if counter-intuitive) behavior - disabling this option will simply revert DPDK to working as it did before this option was introduced (i.e. best-effort allocation). This is why your code still works - because EAL still does allocate memory on socket 1, and *knows* that it's socket 1 memory. It still supports NUMA.
>>>>>>>>>>
>>>>>>>>>> The commit message for these changes states that the actual purpose of this option is to enable "balanced" hugepage allocation. In case of cgroups limitations, previously, DPDK would've exhausted all hugepages on master core's socket before attempting to allocate from other sockets, but by the time we've reached cgroups limits on numbers of hugepages, we might not have reached socket 1 and thus missed out on the pages we could've allocated, but didn't. Using libnuma solves this issue, because now we can allocate pages on sockets we want, instead of hoping we won't run out of hugepages before we get the memory we need.
>>>>>>>>>>
>>>>>>>>>> In 18.05 onwards, this option works differently (and arguably wrong). More specifically, it disallows allocations on sockets other than 0, and it also makes it so that EAL does not check which socket the memory *actually* came from. So, not only allocating memory from socket 1 is disabled, but allocating from socket 0 may even get you memory from socket 1!
>>>>>>>>> I'd consider this as a bug.
>>>>>>>>>
>>>>>>>>>> +CC Thomas
>>>>>>>>>>
>>>>>>>>>> The CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES option is a misnomer, because it makes it seem like this option disables NUMA support, which is not the case.
>>>>>>>>>>
>>>>>>>>>> I would also argue that it is not relevant to 18.05+ memory subsystem, and should only work in legacy mode, because it is *impossible* to make it work right in the new memory subsystem, and here's why:
>>>>>>>>>>
>>>>>>>>>> Without libnuma, we have no way of "asking" the kernel to allocate a hugepage on a specific socket - instead, any allocation will most likely happen on socket from which the allocation came from. For example, if user program's lcore is on socket 1, allocation on socket 0 will actually allocate a page on socket 1.
>>>>>>>>>>
>>>>>>>>>> If we don't check for page's NUMA node affinity (which is what currently happens) - we get performance degradation because we may unintentionally allocate memory on wrong NUMA node. If we do check for this - then allocation of memory on socket 1 from lcore on socket 0 will almost never succeed, because kernel will always give us pages on socket 0.
>>>>>>>>>>
>>>>>>>>>> Put it simply, there is no sane way to make this option work for the new memory subsystem - IMO it should be dropped, and libnuma should be made a hard dependency on Linux.
>>>>>>>>> I agree that new memory model could not work without libnuma, 
>>>>>>>>> i.e. will lead to unpredictable memory allocations with no any 
>>>>>>>>> respect to requested socket_id's. I also agree that 
>>>>>>>>> CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES is only sane for a legacy memory model.
>>>>>>>>> It looks like we have no other choice than just drop the 
>>>>>>>>> option and make the code unconditional, i.e. have hard dependency on libnuma.
>>>>>>>>>
>>>>>>>> We, probably, could compile this code and have hard dependency 
>>>>>>>> only for platforms with 'RTE_MAX_NUMA_NODES > 1'.
>>>>>>> Well, as long as legacy mode stays supported, we have to keep the option. The "drop" part was referring to supporting it under the new memory system, not a literal drop from config files.
>>>>>> The option was introduced because we didn't want to introduce the 
>>>>>> new hard dependency. Since we'll have it anyway, I'm not sure if 
>>>>>> keeping the option for legacy mode makes any sense.
>>>>> Oh yes, you're right. Drop it is!
>>>>>
>>>>>>> As for using RTE_MAX_NUMA_NODES, i don't think it's merited. Distributions cannot deliver different DPDK versions based on the number of sockets on a particular machine - so it would have to be a hard dependency for distributions anyway (does any distribution ship DPDK without libnuma?).
>>>>>> At least ARMv7 builds commonly does not ship libnuma package.
>>>>> Do you mean libnuma builds for ARMv7 are not available? Or do you mean the libnuma package is not installed by default?
>>>>>
>>>>> If it's the latter, then i believe it's not installed by default anywhere, but if using distribution version of DPDK, libnuma will be taken care of via package manager. Presumably building from source can be taken care of with pkg-config/meson.
>>>>>
>>>>> Or do you mean ARMv7 does not have libnuma for their arch at all, in any distro?
>>>> libnuma builds for ARMv7 are not available in most of the distros. 
>>>> I didn't check all, but here is results for Ubuntu:
>>>>        
>>>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>> pac
>>>> kages.ubuntu.com%2Fsearch%3Fsuite%3Dbionic%26arch%3Darmhf%26searcho
>>>> n%3 
>>>> Dnames%26keywords%3Dlibnuma&amp;data=02%7C01%7CAsafSi%40radware.com
>>>> %7C 
>>>> a44f84bca42d4a52acac08d653af83b8%7C6ae4e000b5d04f48a766402d46119b76
>>>> %7C 
>>>> 0%7C0%7C636788410626179927&amp;sdata=1pJ0WkAs6Y%2Bv3w%2BhKAELBw%2Bj
>>>> Mra
>>>> BnhiqqpsXkRv2ifI%3D&amp;reserved=0
>>>>
>>>> You may see that Ubuntu 18.04 (bionic) has no libnuma package for 
>>>> 'armhf' and also 'powerpc' platforms.
>>>>
>>> That's a difficulty. Do these platforms support NUMA? In other words, could we replace this flag with just outright disabling NUMA support?
>>
>> Many platforms don't support NUMA, so they dont' really need libnuma.
>>
>> Mandating libnuma will also break several things:
>>
>>     - cross build for ARM on x86 - which is among the preferred 
>> method for build by many in ARM community.
>>
>>    - many of the embedded SoCs are without NUMA support, they use 
>> smaller rootf (e.g. Yocto).  It will be a burden to add libnuma there.
>>
> 
> OK, point taken.
> 
> So, the alternative would be to have the ability to outright disable NUMA support (either with a new option, or reworking this one - i would prefer a new one, since this one is confusingly named). Meaning, report all cores as socket 0, report all hardware as socket 0, report all memory as socket 0 and never care about NUMA nodes anywhere.
> 
> Would that work? E.g. by default, make libnuma a hard dependency on x86 Linux (but allow to disable it), but disable it everywhere else?

I think, you may just rename the RTE_EAL_NUMA_AWARE_HUGEPAGES to something like RTE_EAL_NUMA_SUPPORT and keep all the defaults as is, i.e.
* globally disabled
* enabled for linux
* disabled for armv7a, dpaa, dpaa2 and stingray.
Meson could handle everything dynamically.

>>
>>>
>>>>>>> For those compiling from source - are there any supported 
>>>>>>> distributions which don't package libnuma? I don't see much 
>>>>>>> sense in keeping libnuma optional, IMO. This is of course up to 
>>>>>>> the tech board to decide, but IMO the "without libnuma it's 
>>>>>>> basically broken" argument is very strong in my opinion :)
>>>>>>>
>>>>>
>>>
>>> --
>>> Thanks,
>>> Anatoly
> 
> 

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Burakov, Anatoly]

On 09-Dec-18 8:14 AM, Asaf Sinai wrote:
> Hi all,
> 
> Thanks for the detailed explanations!
> 
> So, what we understood from that, is the following (please correct, if it is wrong):
> Before 18.05 version:
> - Dividing huge pages between NUMAs was based, by default, on Linux good will.
> - Enforcing Linux to divide huge pages between NUMAs, required enabling configuration option "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES".
> - The enforcement was done via "libnuma" library.
> 
>  From 18.05 version:
> - The mentioned configuration option is ignored, so that by default, all huge pages are allocated on NUMA 0.
> - if "libnuma" library exists in system, then huge pages will be divided between NUMAs, without any special configuration.
> - The above is relevant to architectures that support NUMA, e.g. X86 (which we use).
> 
> Thanks,
> Asaf

Hi Asaf,

Before 18.05, the above description is correct.

Since 18.05, it's not _quite_ like that. There are two memory modes in 
18.05 - default and legacy. Legacy mode pretty much behaves like 
pre-18.05 code.

Default memory mode without the CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES for 
all intents and purposes should be considered unsupported for post-18.05 
code, and libnuma should be considered to be a hard dependency for 
non-legacy, NUMA-aware code. Without this option, EAL will disallow 
allocations on sockets other than 0, but on a NUMA-enabled system, you 
won't necessarily get memory from socket 0 - it will *say* it is on 
socket 0, but it may not *actually* be the case, because without libnuma 
we do not check where it was allocated.

Reasons for the above behavior is simple: legacy mem mode preallocates 
all memory in advance. This gives us an opportunity to figure out page 
socket affinity at initialization, and not worry about it afterwards. 
Non-legacy mode doesn't have the luxury of preallocating all memory in 
advance, instead we allocate memory on the fly - which means that 
whenever an allocation is requested, we need memory not just anywhere 
(like in legacy init case), but located on a specific socket - we cannot 
"sort it out later" like we do with legacy mem. Without libnuma, we 
cannot get this functionality.

> 
> -----Original Message-----
> From: Ilya Maximets <i.maximets@samsung.com>
> Sent: Tuesday, November 27, 2018 06:50 PM
> To: Burakov, Anatoly <anatoly.burakov@intel.com>; Hemant Agrawal <hemant.agrawal@nxp.com>; Asaf Sinai <AsafSi@Radware.com>; dev@dpdk.org; Thomas Monjalon <thomas@monjalon.net>
> Cc: Ilia Ferdman <IliaF@Radware.com>; Sasha Hodos <SashaH@Radware.com>
> Subject: Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration
> 
> On 27.11.2018 13:33, Burakov, Anatoly wrote:
>> On 27-Nov-18 10:26 AM, Hemant Agrawal wrote:
>>>
>>> On 11/26/2018 8:55 PM, Asaf Sinai wrote:
>>>> +CC Ilia & Sasha.
>>>>
>>>> -----Original Message-----
>>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>>>> Sent: Monday, November 26, 2018 04:57 PM
>>>> To: Ilya Maximets <i.maximets@samsung.com>; Asaf Sinai
>>>> <AsafSi@Radware.com>; dev@dpdk.org; Thomas Monjalon
>>>> <thomas@monjalon.net>
>>>> Subject: Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no
>>>> difference in memory pool allocations, when enabling/disabling this
>>>> configuration
>>>>
>>>> On 26-Nov-18 2:32 PM, Ilya Maximets wrote:
>>>>> On 26.11.2018 17:21, Burakov, Anatoly wrote:
>>>>>> On 26-Nov-18 2:10 PM, Ilya Maximets wrote:
>>>>>>> On 26.11.2018 16:42, Burakov, Anatoly wrote:
>>>>>>>> On 26-Nov-18 1:20 PM, Ilya Maximets wrote:
>>>>>>>>> On 26.11.2018 16:16, Ilya Maximets wrote:
>>>>>>>>>> On 26.11.2018 15:50, Burakov, Anatoly wrote:
>>>>>>>>>>> On 26-Nov-18 11:43 AM, Burakov, Anatoly wrote:
>>>>>>>>>>>> On 26-Nov-18 11:33 AM, Asaf Sinai wrote:
>>>>>>>>>>>>> Hi Anatoly,
>>>>>>>>>>>>>
>>>>>>>>>>>>> We did not check it with "testpmd", only with our application.
>>>>>>>>>>>>>        From the beginning, we did not enable this configuration (look at attached files), and everything works fine.
>>>>>>>>>>>>> Of course we rebuild DPDK, when we change configuration.
>>>>>>>>>>>>> Please note that we use DPDK 17.11.3, maybe this is why it works fine?
>>>>>>>>>>>> Just tested with DPDK 17.11, and yes, it does work the way you are describing. This is not intended behavior. I will look into it.
>>>>>>>>>>>>
>>>>>>>>>>> +CC author of commit introducing CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES.
>>>>>>>>>>>
>>>>>>>>>>> Looking at the code, i think this config option needs to be reworked and we should clarify what we mean by this option. It appears that i've misunderstood what this option actually intended to do, and i also think it's naming could be improved because it's confusing and misleading.
>>>>>>>>>>>
>>>>>>>>>>> In 17.11, this option does *not* prevent EAL from using NUMA - it merely disables using libnuma to perform memory allocation. This looks like intended (if counter-intuitive) behavior - disabling this option will simply revert DPDK to working as it did before this option was introduced (i.e. best-effort allocation). This is why your code still works - because EAL still does allocate memory on socket 1, and *knows* that it's socket 1 memory. It still supports NUMA.
>>>>>>>>>>>
>>>>>>>>>>> The commit message for these changes states that the actual purpose of this option is to enable "balanced" hugepage allocation. In case of cgroups limitations, previously, DPDK would've exhausted all hugepages on master core's socket before attempting to allocate from other sockets, but by the time we've reached cgroups limits on numbers of hugepages, we might not have reached socket 1 and thus missed out on the pages we could've allocated, but didn't. Using libnuma solves this issue, because now we can allocate pages on sockets we want, instead of hoping we won't run out of hugepages before we get the memory we need.
>>>>>>>>>>>
>>>>>>>>>>> In 18.05 onwards, this option works differently (and arguably wrong). More specifically, it disallows allocations on sockets other than 0, and it also makes it so that EAL does not check which socket the memory *actually* came from. So, not only allocating memory from socket 1 is disabled, but allocating from socket 0 may even get you memory from socket 1!
>>>>>>>>>> I'd consider this as a bug.
>>>>>>>>>>
>>>>>>>>>>> +CC Thomas
>>>>>>>>>>>
>>>>>>>>>>> The CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES option is a misnomer, because it makes it seem like this option disables NUMA support, which is not the case.
>>>>>>>>>>>
>>>>>>>>>>> I would also argue that it is not relevant to 18.05+ memory subsystem, and should only work in legacy mode, because it is *impossible* to make it work right in the new memory subsystem, and here's why:
>>>>>>>>>>>
>>>>>>>>>>> Without libnuma, we have no way of "asking" the kernel to allocate a hugepage on a specific socket - instead, any allocation will most likely happen on socket from which the allocation came from. For example, if user program's lcore is on socket 1, allocation on socket 0 will actually allocate a page on socket 1.
>>>>>>>>>>>
>>>>>>>>>>> If we don't check for page's NUMA node affinity (which is what currently happens) - we get performance degradation because we may unintentionally allocate memory on wrong NUMA node. If we do check for this - then allocation of memory on socket 1 from lcore on socket 0 will almost never succeed, because kernel will always give us pages on socket 0.
>>>>>>>>>>>
>>>>>>>>>>> Put it simply, there is no sane way to make this option work for the new memory subsystem - IMO it should be dropped, and libnuma should be made a hard dependency on Linux.
>>>>>>>>>> I agree that new memory model could not work without libnuma,
>>>>>>>>>> i.e. will lead to unpredictable memory allocations with no any
>>>>>>>>>> respect to requested socket_id's. I also agree that
>>>>>>>>>> CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES is only sane for a legacy memory model.
>>>>>>>>>> It looks like we have no other choice than just drop the
>>>>>>>>>> option and make the code unconditional, i.e. have hard dependency on libnuma.
>>>>>>>>>>
>>>>>>>>> We, probably, could compile this code and have hard dependency
>>>>>>>>> only for platforms with 'RTE_MAX_NUMA_NODES > 1'.
>>>>>>>> Well, as long as legacy mode stays supported, we have to keep the option. The "drop" part was referring to supporting it under the new memory system, not a literal drop from config files.
>>>>>>> The option was introduced because we didn't want to introduce the
>>>>>>> new hard dependency. Since we'll have it anyway, I'm not sure if
>>>>>>> keeping the option for legacy mode makes any sense.
>>>>>> Oh yes, you're right. Drop it is!
>>>>>>
>>>>>>>> As for using RTE_MAX_NUMA_NODES, i don't think it's merited. Distributions cannot deliver different DPDK versions based on the number of sockets on a particular machine - so it would have to be a hard dependency for distributions anyway (does any distribution ship DPDK without libnuma?).
>>>>>>> At least ARMv7 builds commonly does not ship libnuma package.
>>>>>> Do you mean libnuma builds for ARMv7 are not available? Or do you mean the libnuma package is not installed by default?
>>>>>>
>>>>>> If it's the latter, then i believe it's not installed by default anywhere, but if using distribution version of DPDK, libnuma will be taken care of via package manager. Presumably building from source can be taken care of with pkg-config/meson.
>>>>>>
>>>>>> Or do you mean ARMv7 does not have libnuma for their arch at all, in any distro?
>>>>> libnuma builds for ARMv7 are not available in most of the distros.
>>>>> I didn't check all, but here is results for Ubuntu:
>>>>>         
>>>>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>> pac
>>>>> kages.ubuntu.com%2Fsearch%3Fsuite%3Dbionic%26arch%3Darmhf%26searcho
>>>>> n%3
>>>>> Dnames%26keywords%3Dlibnuma&amp;data=02%7C01%7CAsafSi%40radware.com
>>>>> %7C
>>>>> a44f84bca42d4a52acac08d653af83b8%7C6ae4e000b5d04f48a766402d46119b76
>>>>> %7C
>>>>> 0%7C0%7C636788410626179927&amp;sdata=1pJ0WkAs6Y%2Bv3w%2BhKAELBw%2Bj
>>>>> Mra
>>>>> BnhiqqpsXkRv2ifI%3D&amp;reserved=0
>>>>>
>>>>> You may see that Ubuntu 18.04 (bionic) has no libnuma package for
>>>>> 'armhf' and also 'powerpc' platforms.
>>>>>
>>>> That's a difficulty. Do these platforms support NUMA? In other words, could we replace this flag with just outright disabling NUMA support?
>>>
>>> Many platforms don't support NUMA, so they dont' really need libnuma.
>>>
>>> Mandating libnuma will also break several things:
>>>
>>>      - cross build for ARM on x86 - which is among the preferred
>>> method for build by many in ARM community.
>>>
>>>     - many of the embedded SoCs are without NUMA support, they use
>>> smaller rootf (e.g. Yocto).  It will be a burden to add libnuma there.
>>>
>>
>> OK, point taken.
>>
>> So, the alternative would be to have the ability to outright disable NUMA support (either with a new option, or reworking this one - i would prefer a new one, since this one is confusingly named). Meaning, report all cores as socket 0, report all hardware as socket 0, report all memory as socket 0 and never care about NUMA nodes anywhere.
>>
>> Would that work? E.g. by default, make libnuma a hard dependency on x86 Linux (but allow to disable it), but disable it everywhere else?
> 
> I think, you may just rename the RTE_EAL_NUMA_AWARE_HUGEPAGES to something like RTE_EAL_NUMA_SUPPORT and keep all the defaults as is, i.e.
> * globally disabled
> * enabled for linux
> * disabled for armv7a, dpaa, dpaa2 and stingray.
> Meson could handle everything dynamically.
> 
>>>
>>>>
>>>>>>>> For those compiling from source - are there any supported
>>>>>>>> distributions which don't package libnuma? I don't see much
>>>>>>>> sense in keeping libnuma optional, IMO. This is of course up to
>>>>>>>> the tech board to decide, but IMO the "without libnuma it's
>>>>>>>> basically broken" argument is very strong in my opinion :)
>>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Anatoly
>>
>>


-- 
Thanks,
Anatoly

Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

[Asaf Sinai]

Hi Anatoly,

Thank you very much for the useful explanations!

Thanks,
Asaf

-----Original Message-----
From: Burakov, Anatoly <anatoly.burakov@intel.com> 
Sent: Monday, December 10, 2018 12:10 PM
To: Asaf Sinai <AsafSi@Radware.com>; Ilya Maximets <i.maximets@samsung.com>; Hemant Agrawal <hemant.agrawal@nxp.com>; dev@dpdk.org; Thomas Monjalon <thomas@monjalon.net>
Cc: Ilia Ferdman <IliaF@Radware.com>; Sasha Hodos <SashaH@Radware.com>
Subject: Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no difference in memory pool allocations, when enabling/disabling this configuration

On 09-Dec-18 8:14 AM, Asaf Sinai wrote:
> Hi all,
> 
> Thanks for the detailed explanations!
> 
> So, what we understood from that, is the following (please correct, if it is wrong):
> Before 18.05 version:
> - Dividing huge pages between NUMAs was based, by default, on Linux good will.
> - Enforcing Linux to divide huge pages between NUMAs, required enabling configuration option "CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES".
> - The enforcement was done via "libnuma" library.
> 
>  From 18.05 version:
> - The mentioned configuration option is ignored, so that by default, all huge pages are allocated on NUMA 0.
> - if "libnuma" library exists in system, then huge pages will be divided between NUMAs, without any special configuration.
> - The above is relevant to architectures that support NUMA, e.g. X86 (which we use).
> 
> Thanks,
> Asaf

Hi Asaf,

Before 18.05, the above description is correct.

Since 18.05, it's not _quite_ like that. There are two memory modes in
18.05 - default and legacy. Legacy mode pretty much behaves like
pre-18.05 code.

Default memory mode without the CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES for all intents and purposes should be considered unsupported for post-18.05 code, and libnuma should be considered to be a hard dependency for non-legacy, NUMA-aware code. Without this option, EAL will disallow allocations on sockets other than 0, but on a NUMA-enabled system, you won't necessarily get memory from socket 0 - it will *say* it is on socket 0, but it may not *actually* be the case, because without libnuma we do not check where it was allocated.

Reasons for the above behavior is simple: legacy mem mode preallocates all memory in advance. This gives us an opportunity to figure out page socket affinity at initialization, and not worry about it afterwards. 
Non-legacy mode doesn't have the luxury of preallocating all memory in advance, instead we allocate memory on the fly - which means that whenever an allocation is requested, we need memory not just anywhere (like in legacy init case), but located on a specific socket - we cannot "sort it out later" like we do with legacy mem. Without libnuma, we cannot get this functionality.

> 
> -----Original Message-----
> From: Ilya Maximets <i.maximets@samsung.com>
> Sent: Tuesday, November 27, 2018 06:50 PM
> To: Burakov, Anatoly <anatoly.burakov@intel.com>; Hemant Agrawal 
> <hemant.agrawal@nxp.com>; Asaf Sinai <AsafSi@Radware.com>; 
> dev@dpdk.org; Thomas Monjalon <thomas@monjalon.net>
> Cc: Ilia Ferdman <IliaF@Radware.com>; Sasha Hodos <SashaH@Radware.com>
> Subject: Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no 
> difference in memory pool allocations, when enabling/disabling this 
> configuration
> 
> On 27.11.2018 13:33, Burakov, Anatoly wrote:
>> On 27-Nov-18 10:26 AM, Hemant Agrawal wrote:
>>>
>>> On 11/26/2018 8:55 PM, Asaf Sinai wrote:
>>>> +CC Ilia & Sasha.
>>>>
>>>> -----Original Message-----
>>>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
>>>> Sent: Monday, November 26, 2018 04:57 PM
>>>> To: Ilya Maximets <i.maximets@samsung.com>; Asaf Sinai 
>>>> <AsafSi@Radware.com>; dev@dpdk.org; Thomas Monjalon 
>>>> <thomas@monjalon.net>
>>>> Subject: Re: [dpdk-dev] CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES: no 
>>>> difference in memory pool allocations, when enabling/disabling this 
>>>> configuration
>>>>
>>>> On 26-Nov-18 2:32 PM, Ilya Maximets wrote:
>>>>> On 26.11.2018 17:21, Burakov, Anatoly wrote:
>>>>>> On 26-Nov-18 2:10 PM, Ilya Maximets wrote:
>>>>>>> On 26.11.2018 16:42, Burakov, Anatoly wrote:
>>>>>>>> On 26-Nov-18 1:20 PM, Ilya Maximets wrote:
>>>>>>>>> On 26.11.2018 16:16, Ilya Maximets wrote:
>>>>>>>>>> On 26.11.2018 15:50, Burakov, Anatoly wrote:
>>>>>>>>>>> On 26-Nov-18 11:43 AM, Burakov, Anatoly wrote:
>>>>>>>>>>>> On 26-Nov-18 11:33 AM, Asaf Sinai wrote:
>>>>>>>>>>>>> Hi Anatoly,
>>>>>>>>>>>>>
>>>>>>>>>>>>> We did not check it with "testpmd", only with our application.
>>>>>>>>>>>>>        From the beginning, we did not enable this configuration (look at attached files), and everything works fine.
>>>>>>>>>>>>> Of course we rebuild DPDK, when we change configuration.
>>>>>>>>>>>>> Please note that we use DPDK 17.11.3, maybe this is why it works fine?
>>>>>>>>>>>> Just tested with DPDK 17.11, and yes, it does work the way you are describing. This is not intended behavior. I will look into it.
>>>>>>>>>>>>
>>>>>>>>>>> +CC author of commit introducing CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES.
>>>>>>>>>>>
>>>>>>>>>>> Looking at the code, i think this config option needs to be reworked and we should clarify what we mean by this option. It appears that i've misunderstood what this option actually intended to do, and i also think it's naming could be improved because it's confusing and misleading.
>>>>>>>>>>>
>>>>>>>>>>> In 17.11, this option does *not* prevent EAL from using NUMA - it merely disables using libnuma to perform memory allocation. This looks like intended (if counter-intuitive) behavior - disabling this option will simply revert DPDK to working as it did before this option was introduced (i.e. best-effort allocation). This is why your code still works - because EAL still does allocate memory on socket 1, and *knows* that it's socket 1 memory. It still supports NUMA.
>>>>>>>>>>>
>>>>>>>>>>> The commit message for these changes states that the actual purpose of this option is to enable "balanced" hugepage allocation. In case of cgroups limitations, previously, DPDK would've exhausted all hugepages on master core's socket before attempting to allocate from other sockets, but by the time we've reached cgroups limits on numbers of hugepages, we might not have reached socket 1 and thus missed out on the pages we could've allocated, but didn't. Using libnuma solves this issue, because now we can allocate pages on sockets we want, instead of hoping we won't run out of hugepages before we get the memory we need.
>>>>>>>>>>>
>>>>>>>>>>> In 18.05 onwards, this option works differently (and arguably wrong). More specifically, it disallows allocations on sockets other than 0, and it also makes it so that EAL does not check which socket the memory *actually* came from. So, not only allocating memory from socket 1 is disabled, but allocating from socket 0 may even get you memory from socket 1!
>>>>>>>>>> I'd consider this as a bug.
>>>>>>>>>>
>>>>>>>>>>> +CC Thomas
>>>>>>>>>>>
>>>>>>>>>>> The CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES option is a misnomer, because it makes it seem like this option disables NUMA support, which is not the case.
>>>>>>>>>>>
>>>>>>>>>>> I would also argue that it is not relevant to 18.05+ memory subsystem, and should only work in legacy mode, because it is *impossible* to make it work right in the new memory subsystem, and here's why:
>>>>>>>>>>>
>>>>>>>>>>> Without libnuma, we have no way of "asking" the kernel to allocate a hugepage on a specific socket - instead, any allocation will most likely happen on socket from which the allocation came from. For example, if user program's lcore is on socket 1, allocation on socket 0 will actually allocate a page on socket 1.
>>>>>>>>>>>
>>>>>>>>>>> If we don't check for page's NUMA node affinity (which is what currently happens) - we get performance degradation because we may unintentionally allocate memory on wrong NUMA node. If we do check for this - then allocation of memory on socket 1 from lcore on socket 0 will almost never succeed, because kernel will always give us pages on socket 0.
>>>>>>>>>>>
>>>>>>>>>>> Put it simply, there is no sane way to make this option work for the new memory subsystem - IMO it should be dropped, and libnuma should be made a hard dependency on Linux.
>>>>>>>>>> I agree that new memory model could not work without libnuma, 
>>>>>>>>>> i.e. will lead to unpredictable memory allocations with no 
>>>>>>>>>> any respect to requested socket_id's. I also agree that 
>>>>>>>>>> CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES is only sane for a legacy memory model.
>>>>>>>>>> It looks like we have no other choice than just drop the 
>>>>>>>>>> option and make the code unconditional, i.e. have hard dependency on libnuma.
>>>>>>>>>>
>>>>>>>>> We, probably, could compile this code and have hard dependency 
>>>>>>>>> only for platforms with 'RTE_MAX_NUMA_NODES > 1'.
>>>>>>>> Well, as long as legacy mode stays supported, we have to keep the option. The "drop" part was referring to supporting it under the new memory system, not a literal drop from config files.
>>>>>>> The option was introduced because we didn't want to introduce 
>>>>>>> the new hard dependency. Since we'll have it anyway, I'm not 
>>>>>>> sure if keeping the option for legacy mode makes any sense.
>>>>>> Oh yes, you're right. Drop it is!
>>>>>>
>>>>>>>> As for using RTE_MAX_NUMA_NODES, i don't think it's merited. Distributions cannot deliver different DPDK versions based on the number of sockets on a particular machine - so it would have to be a hard dependency for distributions anyway (does any distribution ship DPDK without libnuma?).
>>>>>>> At least ARMv7 builds commonly does not ship libnuma package.
>>>>>> Do you mean libnuma builds for ARMv7 are not available? Or do you mean the libnuma package is not installed by default?
>>>>>>
>>>>>> If it's the latter, then i believe it's not installed by default anywhere, but if using distribution version of DPDK, libnuma will be taken care of via package manager. Presumably building from source can be taken care of with pkg-config/meson.
>>>>>>
>>>>>> Or do you mean ARMv7 does not have libnuma for their arch at all, in any distro?
>>>>> libnuma builds for ARMv7 are not available in most of the distros.
>>>>> I didn't check all, but here is results for Ubuntu:
>>>>>         
>>>>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>>>> F
>>>>> pac
>>>>> kages.ubuntu.com%2Fsearch%3Fsuite%3Dbionic%26arch%3Darmhf%26search
>>>>> o
>>>>> n%3
>>>>> Dnames%26keywords%3Dlibnuma&amp;data=02%7C01%7CAsafSi%40radware.co
>>>>> m
>>>>> %7C
>>>>> a44f84bca42d4a52acac08d653af83b8%7C6ae4e000b5d04f48a766402d46119b7
>>>>> 6
>>>>> %7C
>>>>> 0%7C0%7C636788410626179927&amp;sdata=1pJ0WkAs6Y%2Bv3w%2BhKAELBw%2B
>>>>> j
>>>>> Mra
>>>>> BnhiqqpsXkRv2ifI%3D&amp;reserved=0
>>>>>
>>>>> You may see that Ubuntu 18.04 (bionic) has no libnuma package for 
>>>>> 'armhf' and also 'powerpc' platforms.
>>>>>
>>>> That's a difficulty. Do these platforms support NUMA? In other words, could we replace this flag with just outright disabling NUMA support?
>>>
>>> Many platforms don't support NUMA, so they dont' really need libnuma.
>>>
>>> Mandating libnuma will also break several things:
>>>
>>>      - cross build for ARM on x86 - which is among the preferred 
>>> method for build by many in ARM community.
>>>
>>>     - many of the embedded SoCs are without NUMA support, they use 
>>> smaller rootf (e.g. Yocto).  It will be a burden to add libnuma there.
>>>
>>
>> OK, point taken.
>>
>> So, the alternative would be to have the ability to outright disable NUMA support (either with a new option, or reworking this one - i would prefer a new one, since this one is confusingly named). Meaning, report all cores as socket 0, report all hardware as socket 0, report all memory as socket 0 and never care about NUMA nodes anywhere.
>>
>> Would that work? E.g. by default, make libnuma a hard dependency on x86 Linux (but allow to disable it), but disable it everywhere else?
> 
> I think, you may just rename the RTE_EAL_NUMA_AWARE_HUGEPAGES to something like RTE_EAL_NUMA_SUPPORT and keep all the defaults as is, i.e.
> * globally disabled
> * enabled for linux
> * disabled for armv7a, dpaa, dpaa2 and stingray.
> Meson could handle everything dynamically.
> 
>>>
>>>>
>>>>>>>> For those compiling from source - are there any supported 
>>>>>>>> distributions which don't package libnuma? I don't see much 
>>>>>>>> sense in keeping libnuma optional, IMO. This is of course up to 
>>>>>>>> the tech board to decide, but IMO the "without libnuma it's 
>>>>>>>> basically broken" argument is very strong in my opinion :)
>>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Anatoly
>>
>>


--
Thanks,
Anatoly