Re: [dts] [PATCH] framework/dut: setup hugepage for all available NUMA nodes

[Phil Yang <Phil.Yang@arm.com>]

Yes. That's very thoughtful.
Reworked. Please review it.

Thanks,
Phil Yang

> -----Original Message-----
> From: Liu, Yong <yong.liu@intel.com>
> Sent: Tuesday, August 28, 2018 10:58 AM
> To: Phil Yang <Phil.Yang@arm.com>; dts <dts-bounces@dpdk.org>;
> dts@dpdk.org
> Cc: nd <nd@arm.com>
> Subject: RE: [dts] [PATCH] framework/dut: setup hugepage for all available
> NUMA nodes
> 
> You're welcome, Phil.
> One more comment, It's better to make total_numa_nodes and
> numa_service_num variables as integer before comparing them.
> Function "min" may not return the expect result when numa number over ten.
> 
> Thanks,
> Marvin
> 
> > -----Original Message-----
> > From: Phil.Yang@arm.com [mailto:Phil.Yang@arm.com]
> > Sent: Tuesday, August 28, 2018 10:01 AM
> > To: Liu, Yong <yong.liu@intel.com>; dts <dts-bounces@dpdk.org>;
> > dts@dpdk.org
> > Cc: nd <nd@arm.com>
> > Subject: RE: [dts] [PATCH] framework/dut: setup hugepage for all
> > available NUMA nodes
> >
> > Hi Marvin,
> >
> > Thanks for your comment.
> > I reworked this patch, please review it.
> >
> > Thanks,
> > Phil Yang
> >
> > > -----Original Message-----
> > > From: dts <dts-bounces@dpdk.org> On Behalf Of Liu, Yong
> > > Sent: Monday, August 27, 2018 1:28 PM
> > > To: dts <dts-bounces@dpdk.org>; dts@dpdk.org
> > > Cc: nd <nd@arm.com>
> > > Subject: Re: [dts] [PATCH] framework/dut: setup hugepage for all
> > > available NUMA nodes
> > >
> > > Hi Phil,
> > > One comment is inline.
> > >
> > > Thanks,
> > > Marvin
> > >
> > > > -----Original Message-----
> > > > From: dts [mailto:dts-bounces@dpdk.org]
> > > > Sent: Monday, August 27, 2018 9:16 AM
> > > > To: dts@dpdk.org
> > > > Cc: nd@arm.com
> > > > Subject: [dts] [PATCH] framework/dut: setup hugepage for all
> > > > available NUMA nodes
> > > >
> > > > On a NUMA machine, we need to setup hugepage for all NUMA nodes.
> > > >
> > > > The default approach is by setting /sys/kernel/mm/hugepages, but
> > > > it only works on a single-node system.
> > > >
> > > > This fix needs numactl tool support on Linux.
> > > >
> > > > Signed-off-by: Phil Yang <phil.yang@arm.com>
> > > > ---
> > > >  framework/crb.py | 6 ++++--
> > > >  framework/dut.py | 9 ++++++++-
> > > >  2 files changed, 12 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/framework/crb.py b/framework/crb.py index
> > > > 5c555db..97bebfe 100644
> > > > --- a/framework/crb.py
> > > > +++ b/framework/crb.py
> > > > @@ -204,8 +204,10 @@ class Crb(object):
> > > >          if numa == -1:
> > > >              self.send_expect('echo %d >
> > > > /sys/kernel/mm/hugepages/hugepages-
> > > > %skB/nr_hugepages' % (huge_pages, page_size), '# ', 5)
> > > >          else:
> > > > -            # sometimes we set hugepage on kernel cmdline, so we need
> > clear
> > > > default hugepage
> > > > -            self.send_expect('echo 0 >
> > /sys/kernel/mm/hugepages/hugepages-
> > > > %skB/nr_hugepages' % (page_size), '# ', 5)
> > > > +            # sometimes we set hugepage on kernel cmdline, so we
> > > > + clear all
> > > > nodes' default hugepages at the first time.
> > > > +            if numa == 0:
> > > > +                self.send_expect('echo 0 >
> > > > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' %
> > > > (page_size), '# ', 5)
> > > > +
> > > >              # some platform not support numa, example vm dut
> > > >              try:
> > > >                  self.send_expect('echo %d >
> > > > /sys/devices/system/node/node%d/hugepages/hugepages-
> > > %skB/nr_hugepages'
> > > > % (huge_pages, numa, page_size), '# ', 5) diff --git
> > > > a/framework/dut.py b/framework/dut.py index 18f0b39..b358578
> > > > 100644
> > > > --- a/framework/dut.py
> > > > +++ b/framework/dut.py
> > > > @@ -305,6 +305,12 @@ class Dut(Crb):
> > > >              return
> > > >          hugepages_size = self.send_expect("awk '/Hugepagesize/
> > > > {print
> > $2}'
> > > > /proc/meminfo", "# ")
> > > >          total_huge_pages = self.get_total_huge_pages()
> > > > +        total_numa_nodes = self.send_expect("numactl -H | awk
> > > > + '/available/
> > > > {print $2}'", "# ")
> > >
> > > We may not expect that numactl is installed on the DUT. It will be
> > > more
> > portable
> > > by checking sysfs like "ls /sys/devices/system/node | grep node* | wc -l".
> > >
> > > > +        numa_service_num =
> > > > self.get_def_rte_config('CONFIG_RTE_MAX_NUMA_NODES')
> > > > +        if numa_service_num:
> > > > +            numa = min(total_numa_nodes, numa_service_num)
> > > > +        else:
> > > > +            numa = total_numa_nodes
> > > >          force_socket = False
> > > >
> > > >          if int(hugepages_size) < (1024 * 1024):
> > > > @@ -332,7 +338,8 @@ class Dut(Crb):
> > > >                  if force_socket:
> > > >                      self.set_huge_pages(arch_huge_pages, 0)
> > > >                  else:
> > > > -                    self.set_huge_pages(arch_huge_pages)
> > > > +                    for numa_id in range(0, int(numa)):
> > > > +                        self.set_huge_pages(arch_huge_pages,
> > > > + numa_id)
> > > >
> > > >          self.mount_huge_pages()
> > > >          self.hugepage_path = self.strip_hugepage_path()
> > > > --
> > > > 2.7.4