Re: [dpdk-dev] [PATCH v5 0/2] Balanced allocation of hugepages

[Jerin Jacob <jerin.jacob@caviumnetworks.com>]

-----Original Message-----
> Date: Wed, 21 Jun 2017 09:25:25 +0100
> From: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>
> To: Hemant Agrawal <hemant.agrawal@nxp.com>, Jerin Jacob
>  <jerin.jacob@caviumnetworks.com>
> CC: Thomas Monjalon <thomas@monjalon.net>, Ilya Maximets
>  <i.maximets@samsung.com>, dev@dpdk.org, Bruce Richardson
>  <bruce.richardson@intel.com>, David Marchand <david.marchand@6wind.com>,
>  Heetae Ahn <heetae82.ahn@samsung.com>, Yuanhan Liu <yliu@fridaylinux.org>,
>  Jianfeng Tan <jianfeng.tan@intel.com>, Neil Horman
>  <nhorman@tuxdriver.com>, Yulong Pei <yulong.pei@intel.com>
> Subject: Re: [PATCH v5 0/2] Balanced allocation of hugepages
> User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101
>  Thunderbird/45.1.1
> 
> On 21/06/2017 09:14, Hemant Agrawal wrote:
> > On 6/20/2017 9:11 PM, Jerin Jacob wrote:
> > > -----Original Message-----
> > > > Date: Tue, 20 Jun 2017 15:58:50 +0100
> > > > From: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>
> > > > To: Thomas Monjalon <thomas@monjalon.net>, Ilya Maximets
> > > >  <i.maximets@samsung.com>
> > > > CC: dev@dpdk.org, Hemant Agrawal <hemant.agrawal@nxp.com>, Bruce
> > > > Richardson
> > > >  <bruce.richardson@intel.com>, David Marchand
> > > > <david.marchand@6wind.com>,
> > > >  Heetae Ahn <heetae82.ahn@samsung.com>, Yuanhan Liu
> > > > <yliu@fridaylinux.org>,
> > > >  Jianfeng Tan <jianfeng.tan@intel.com>, Neil Horman
> > > >  <nhorman@tuxdriver.com>, Yulong Pei <yulong.pei@intel.com>
> > > > Subject: Re: [dpdk-dev] [PATCH v5 0/2] Balanced allocation of hugepages
> > > > User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101
> > > >  Thunderbird/45.1.1
> > > > 
> > > > On 20/06/2017 15:35, Thomas Monjalon wrote:
> > > > > 20/06/2017 15:58, Ilya Maximets:
> > > > > > On 20.06.2017 16:07, Thomas Monjalon wrote:
> > > > > > > 19/06/2017 13:10, Hemant Agrawal:
> > > > > > > > > > > On Thu, Jun 08, 2017 at 02:21:58PM +0300, Ilya Maximets wrote:
> > > > > > > > > > > > So, there are 2 option:
> > > > > > > > > > > > 
> > > > > > > > > > > >      1. Return back config
> > > > > > > > > > > > option
> > > > > > > > > > > > RTE_LIBRTE_EAL_NUMA_AWARE_HUGEPAGES
> > > > > > > > > > > >         from the first version
> > > > > > > > > > > > of the patch and disable it by
> > > > > > > > > > > > default.
> > > > > > > > > > > > 
> > > > > > > > > > > >      2. Keep patch as it is now
> > > > > > > > > > > > and make everyone install
> > > > > > > > > > > > libnuma
> > > > > > > > > > > >         for successful build.
> > > > > > > > +1 for option 1
> > > > > > > > It will be a issue and undesired dependency for
> > > > > > > > SoCs, not supporting
> > > > > > > > NUMA architecture.
> > > > > > > > 
> > > > > > > > It can be added to the config, who desired to use it by default.
> > > > > > > Yes I agree, it cannot be a dependency for architectures which
> > > > > > > do not support NUMA.
> > > > > > > Please can we rework the patch so that only one node is assumed
> > > > > > > if NUMA is disabled for the architecture?
> > > > 
> > > > Ilya, I missed that libnuma is not supported on ARM.
> > > 
> > > It is supported on arm64 and arm64 has NUMA machines(thunderx,
> > > thunderx2) too.
> > > 
> > > [dpdk.org] $ dpkg-query -L libnuma-dev
> > > /.
> > > /usr
> > > /usr/lib
> > > /usr/lib/aarch64-linux-gnu
> > > /usr/lib/aarch64-linux-gnu/libnuma.a
> > > /usr/share
> > > /usr/share/man
> > > /usr/share/man/man3
> > > /usr/share/man/man3/numa.3.gz
> > > /usr/share/doc
> > > /usr/share/doc/libnuma-dev
> > > /usr/share/doc/libnuma-dev/copyright
> > > /usr/include
> > > /usr/include/numaif.h
> > > /usr/include/numa.h
> > > /usr/include/numacompat1.h
> > > /usr/lib/aarch64-linux-gnu/libnuma.so
> > > 
> > 
> > 1. There are many machines (arm/ppc), which do not support NUMA.
> > 
> > https://wiki.linaro.org/LEG/Engineering/Kernel/NUMA
> > 
> 
> I did find that link too, last modified 4 years ago.
> Despite that, I could not find any ARM references in libnuma sources, but
> Jerin proved that there is support for it.
> 
> http://oss.sgi.com/projects/libnuma/
> https://github.com/numactl/numactl

Those Linaro links are very old. ARM64 NUMA supported has been added in 4.7 kernel.
I guess we are talking about build time time dependency with libnuma here.
Correct? I think, Even with old arm64 kernel(< 4.6), You can build against
libnuma if it is present in rootfs. Just that at runtime, it will return
NUMA support not available. Correct?

How hard is detect the presence of "numaif.h" if existing build system does not
support it? If it trivial, we can enable RTE_LIBRTE_EAL_NUMA_AWARE_HUGEPAGES
if build environment has "numaif.h".

Some example in linux kernel build system:
http://lxr.linux.no/linux+v4.10.1/scripts/gcc-goto.sh

> 
> > 2. I could not locate it by default in Linaro toolchains.
> > 
> > 3.  Since this is not a common across all platform. This option should
> > not be added to the common_base or common configs. It can be added to
> > any architecture configuration, which needs it.
> > 
> 
> So is it thunderx the only arm64 to enable this feature by default?
> I thought the dependency was the libnuma library support itself.
> 
> Thanks,
> Sergio
> 
> > Regards,
> > Hemant
> > 
> > > 
> > > > 
> > > > > > We're still don't have dynamic build time configuration system.
> > > > > > To make get/set_mempolicy work we need to include <numaif.h>
> > > > > > and have libnuma for successful linkage.
> > > > > > This means that the only option to not have libnuma as dependency
> > > > > > is to return back configuration option
> > > > > > RTE_LIBRTE_EAL_NUMA_AWARE_HUGEPAGES
> > > > > > as it was in the first version of the patch.
> > > > > > 
> > > > > > There is, actually, the third option (besides 2 already described):
> > > > > > 
> > > > > >     3. Return back config option RTE_LIBRTE_EAL_NUMA_AWARE_HUGEPAGES
> > > > > >        from the first version of the patch and *enable*
> > > > > > it by default.
> > > > > >        In this case anyone who doesn't want to have
> > > > > > libnuma as dependency
> > > > > >        will be able to disable the config option manually.
> > > > > > 
> > > > > > Thomas, what do you think? Bruce? Sergio?
> > > > > It should be enabled on x86 and ppc, and disabled in other
> > > > > default configurations (ARM for now).
> > > > 
> > > > Agree.
> > > > 
> > > > > > P.S. We're always able to implement syscall wrappers by
> > > > > > hands without any
> > > > > >       external dependencies, but I don't think it's a good decision.
> > > > > I agree to use libnuma instead of re-inventing the wheel.
> > > > > Let's just make it optional at build time and fallback on one node
> > > > > if disabled.
> > > > 
> > > > That is the simple way out.
> > > > 
> > > > Sergio
> > > 
> > 
> > 
>