* [dts] [PATCH] framework/dut: setup hugepage for all available NUMA nodes
@ 2018-08-27 1:16 Phil Yang
2018-08-27 5:27 ` Liu, Yong
2018-08-28 1:59 ` [dts] [PATCH v2] framework/dut: setup hugepage for all available numa nodes Phil Yang
0 siblings, 2 replies; 8+ messages in thread
From: Phil Yang @ 2018-08-27 1:16 UTC (permalink / raw)
To: dts; +Cc: nd
On a NUMA machine, we need to setup hugepage for all NUMA nodes.
The default approach is by setting /sys/kernel/mm/hugepages, but it
only works on a single-node system.
This fix needs numactl tool support on Linux.
Signed-off-by: Phil Yang <phil.yang@arm.com>
---
framework/crb.py | 6 ++++--
framework/dut.py | 9 ++++++++-
2 files changed, 12 insertions(+), 3 deletions(-)
diff --git a/framework/crb.py b/framework/crb.py
index 5c555db..97bebfe 100644
--- a/framework/crb.py
+++ b/framework/crb.py
@@ -204,8 +204,10 @@ class Crb(object):
if numa == -1:
self.send_expect('echo %d > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (huge_pages, page_size), '# ', 5)
else:
- # sometimes we set hugepage on kernel cmdline, so we need clear default hugepage
- self.send_expect('echo 0 > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (page_size), '# ', 5)
+ # sometimes we set hugepage on kernel cmdline, so we clear all nodes' default hugepages at the first time.
+ if numa == 0:
+ self.send_expect('echo 0 > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (page_size), '# ', 5)
+
# some platform not support numa, example vm dut
try:
self.send_expect('echo %d > /sys/devices/system/node/node%d/hugepages/hugepages-%skB/nr_hugepages' % (huge_pages, numa, page_size), '# ', 5)
diff --git a/framework/dut.py b/framework/dut.py
index 18f0b39..b358578 100644
--- a/framework/dut.py
+++ b/framework/dut.py
@@ -305,6 +305,12 @@ class Dut(Crb):
return
hugepages_size = self.send_expect("awk '/Hugepagesize/ {print $2}' /proc/meminfo", "# ")
total_huge_pages = self.get_total_huge_pages()
+ total_numa_nodes = self.send_expect("numactl -H | awk '/available/ {print $2}'", "# ")
+ numa_service_num = self.get_def_rte_config('CONFIG_RTE_MAX_NUMA_NODES')
+ if numa_service_num:
+ numa = min(total_numa_nodes, numa_service_num)
+ else:
+ numa = total_numa_nodes
force_socket = False
if int(hugepages_size) < (1024 * 1024):
@@ -332,7 +338,8 @@ class Dut(Crb):
if force_socket:
self.set_huge_pages(arch_huge_pages, 0)
else:
- self.set_huge_pages(arch_huge_pages)
+ for numa_id in range(0, int(numa)):
+ self.set_huge_pages(arch_huge_pages, numa_id)
self.mount_huge_pages()
self.hugepage_path = self.strip_hugepage_path()
--
2.7.4
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dts] [PATCH] framework/dut: setup hugepage for all available NUMA nodes
2018-08-27 1:16 [dts] [PATCH] framework/dut: setup hugepage for all available NUMA nodes Phil Yang
@ 2018-08-27 5:27 ` Liu, Yong
2018-08-28 2:00 ` Phil Yang
2018-08-28 1:59 ` [dts] [PATCH v2] framework/dut: setup hugepage for all available numa nodes Phil Yang
1 sibling, 1 reply; 8+ messages in thread
From: Liu, Yong @ 2018-08-27 5:27 UTC (permalink / raw)
To: dts, dts; +Cc: nd
Hi Phil,
One comment is inline.
Thanks,
Marvin
> -----Original Message-----
> From: dts [mailto:dts-bounces@dpdk.org]
> Sent: Monday, August 27, 2018 9:16 AM
> To: dts@dpdk.org
> Cc: nd@arm.com
> Subject: [dts] [PATCH] framework/dut: setup hugepage for all available NUMA
> nodes
>
> On a NUMA machine, we need to setup hugepage for all NUMA nodes.
>
> The default approach is by setting /sys/kernel/mm/hugepages, but it
> only works on a single-node system.
>
> This fix needs numactl tool support on Linux.
>
> Signed-off-by: Phil Yang <phil.yang@arm.com>
> ---
> framework/crb.py | 6 ++++--
> framework/dut.py | 9 ++++++++-
> 2 files changed, 12 insertions(+), 3 deletions(-)
>
> diff --git a/framework/crb.py b/framework/crb.py
> index 5c555db..97bebfe 100644
> --- a/framework/crb.py
> +++ b/framework/crb.py
> @@ -204,8 +204,10 @@ class Crb(object):
> if numa == -1:
> self.send_expect('echo %d > /sys/kernel/mm/hugepages/hugepages-
> %skB/nr_hugepages' % (huge_pages, page_size), '# ', 5)
> else:
> - # sometimes we set hugepage on kernel cmdline, so we need clear
> default hugepage
> - self.send_expect('echo 0 > /sys/kernel/mm/hugepages/hugepages-
> %skB/nr_hugepages' % (page_size), '# ', 5)
> + # sometimes we set hugepage on kernel cmdline, so we clear all
> nodes' default hugepages at the first time.
> + if numa == 0:
> + self.send_expect('echo 0 >
> /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (page_size), '# ', 5)
> +
> # some platform not support numa, example vm dut
> try:
> self.send_expect('echo %d >
> /sys/devices/system/node/node%d/hugepages/hugepages-%skB/nr_hugepages' %
> (huge_pages, numa, page_size), '# ', 5)
> diff --git a/framework/dut.py b/framework/dut.py
> index 18f0b39..b358578 100644
> --- a/framework/dut.py
> +++ b/framework/dut.py
> @@ -305,6 +305,12 @@ class Dut(Crb):
> return
> hugepages_size = self.send_expect("awk '/Hugepagesize/ {print $2}'
> /proc/meminfo", "# ")
> total_huge_pages = self.get_total_huge_pages()
> + total_numa_nodes = self.send_expect("numactl -H | awk '/available/
> {print $2}'", "# ")
We may not expect that numactl is installed on the DUT. It will be more portable by checking sysfs like "ls /sys/devices/system/node | grep node* | wc -l".
> + numa_service_num =
> self.get_def_rte_config('CONFIG_RTE_MAX_NUMA_NODES')
> + if numa_service_num:
> + numa = min(total_numa_nodes, numa_service_num)
> + else:
> + numa = total_numa_nodes
> force_socket = False
>
> if int(hugepages_size) < (1024 * 1024):
> @@ -332,7 +338,8 @@ class Dut(Crb):
> if force_socket:
> self.set_huge_pages(arch_huge_pages, 0)
> else:
> - self.set_huge_pages(arch_huge_pages)
> + for numa_id in range(0, int(numa)):
> + self.set_huge_pages(arch_huge_pages, numa_id)
>
> self.mount_huge_pages()
> self.hugepage_path = self.strip_hugepage_path()
> --
> 2.7.4
^ permalink raw reply [flat|nested] 8+ messages in thread
* [dts] [PATCH v2] framework/dut: setup hugepage for all available numa nodes
2018-08-27 1:16 [dts] [PATCH] framework/dut: setup hugepage for all available NUMA nodes Phil Yang
2018-08-27 5:27 ` Liu, Yong
@ 2018-08-28 1:59 ` Phil Yang
2018-08-28 3:08 ` [dts] [PATCH v3] " Phil Yang
1 sibling, 1 reply; 8+ messages in thread
From: Phil Yang @ 2018-08-28 1:59 UTC (permalink / raw)
To: dts; +Cc: nd, yong.liu, Phil Yang
From: Phil Yang <phil.yang@arm.com>
On a NUMA machine, we need to setup hugepage for all NUMA nodes.
The default approach is by setting /sys/kernel/mm/hugepages, but it
only works on a single-node system.
Signed-off-by: Phil Yang <phil.yang@arm.com>
---
framework/crb.py | 6 ++++--
framework/dut.py | 9 ++++++++-
2 files changed, 12 insertions(+), 3 deletions(-)
diff --git a/framework/crb.py b/framework/crb.py
index 5c555db..97bebfe 100644
--- a/framework/crb.py
+++ b/framework/crb.py
@@ -204,8 +204,10 @@ class Crb(object):
if numa == -1:
self.send_expect('echo %d > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (huge_pages, page_size), '# ', 5)
else:
- # sometimes we set hugepage on kernel cmdline, so we need clear default hugepage
- self.send_expect('echo 0 > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (page_size), '# ', 5)
+ # sometimes we set hugepage on kernel cmdline, so we clear all nodes' default hugepages at the first time.
+ if numa == 0:
+ self.send_expect('echo 0 > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (page_size), '# ', 5)
+
# some platform not support numa, example vm dut
try:
self.send_expect('echo %d > /sys/devices/system/node/node%d/hugepages/hugepages-%skB/nr_hugepages' % (huge_pages, numa, page_size), '# ', 5)
diff --git a/framework/dut.py b/framework/dut.py
index 18f0b39..3e21509 100644
--- a/framework/dut.py
+++ b/framework/dut.py
@@ -305,6 +305,12 @@ class Dut(Crb):
return
hugepages_size = self.send_expect("awk '/Hugepagesize/ {print $2}' /proc/meminfo", "# ")
total_huge_pages = self.get_total_huge_pages()
+ total_numa_nodes = self.send_expect("ls /sys/devices/system/node | grep node* | wc -l", "# ")
+ numa_service_num = self.get_def_rte_config('CONFIG_RTE_MAX_NUMA_NODES')
+ if numa_service_num:
+ numa = min(total_numa_nodes, numa_service_num)
+ else:
+ numa = total_numa_nodes
force_socket = False
if int(hugepages_size) < (1024 * 1024):
@@ -332,7 +338,8 @@ class Dut(Crb):
if force_socket:
self.set_huge_pages(arch_huge_pages, 0)
else:
- self.set_huge_pages(arch_huge_pages)
+ for numa_id in range(0, int(numa)):
+ self.set_huge_pages(arch_huge_pages, numa_id)
self.mount_huge_pages()
self.hugepage_path = self.strip_hugepage_path()
--
2.7.4
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dts] [PATCH] framework/dut: setup hugepage for all available NUMA nodes
2018-08-27 5:27 ` Liu, Yong
@ 2018-08-28 2:00 ` Phil Yang
2018-08-28 2:57 ` Liu, Yong
0 siblings, 1 reply; 8+ messages in thread
From: Phil Yang @ 2018-08-28 2:00 UTC (permalink / raw)
To: Liu, Yong, dts, dts; +Cc: nd
Hi Marvin,
Thanks for your comment.
I reworked this patch, please review it.
Thanks,
Phil Yang
> -----Original Message-----
> From: dts <dts-bounces@dpdk.org> On Behalf Of Liu, Yong
> Sent: Monday, August 27, 2018 1:28 PM
> To: dts <dts-bounces@dpdk.org>; dts@dpdk.org
> Cc: nd <nd@arm.com>
> Subject: Re: [dts] [PATCH] framework/dut: setup hugepage for all available
> NUMA nodes
>
> Hi Phil,
> One comment is inline.
>
> Thanks,
> Marvin
>
> > -----Original Message-----
> > From: dts [mailto:dts-bounces@dpdk.org]
> > Sent: Monday, August 27, 2018 9:16 AM
> > To: dts@dpdk.org
> > Cc: nd@arm.com
> > Subject: [dts] [PATCH] framework/dut: setup hugepage for all available
> > NUMA nodes
> >
> > On a NUMA machine, we need to setup hugepage for all NUMA nodes.
> >
> > The default approach is by setting /sys/kernel/mm/hugepages, but it
> > only works on a single-node system.
> >
> > This fix needs numactl tool support on Linux.
> >
> > Signed-off-by: Phil Yang <phil.yang@arm.com>
> > ---
> > framework/crb.py | 6 ++++--
> > framework/dut.py | 9 ++++++++-
> > 2 files changed, 12 insertions(+), 3 deletions(-)
> >
> > diff --git a/framework/crb.py b/framework/crb.py index
> > 5c555db..97bebfe 100644
> > --- a/framework/crb.py
> > +++ b/framework/crb.py
> > @@ -204,8 +204,10 @@ class Crb(object):
> > if numa == -1:
> > self.send_expect('echo %d >
> > /sys/kernel/mm/hugepages/hugepages-
> > %skB/nr_hugepages' % (huge_pages, page_size), '# ', 5)
> > else:
> > - # sometimes we set hugepage on kernel cmdline, so we need clear
> > default hugepage
> > - self.send_expect('echo 0 > /sys/kernel/mm/hugepages/hugepages-
> > %skB/nr_hugepages' % (page_size), '# ', 5)
> > + # sometimes we set hugepage on kernel cmdline, so we
> > + clear all
> > nodes' default hugepages at the first time.
> > + if numa == 0:
> > + self.send_expect('echo 0 >
> > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (page_size),
> > '# ', 5)
> > +
> > # some platform not support numa, example vm dut
> > try:
> > self.send_expect('echo %d >
> > /sys/devices/system/node/node%d/hugepages/hugepages-
> %skB/nr_hugepages'
> > % (huge_pages, numa, page_size), '# ', 5) diff --git
> > a/framework/dut.py b/framework/dut.py index 18f0b39..b358578 100644
> > --- a/framework/dut.py
> > +++ b/framework/dut.py
> > @@ -305,6 +305,12 @@ class Dut(Crb):
> > return
> > hugepages_size = self.send_expect("awk '/Hugepagesize/ {print $2}'
> > /proc/meminfo", "# ")
> > total_huge_pages = self.get_total_huge_pages()
> > + total_numa_nodes = self.send_expect("numactl -H | awk
> > + '/available/
> > {print $2}'", "# ")
>
> We may not expect that numactl is installed on the DUT. It will be more portable
> by checking sysfs like "ls /sys/devices/system/node | grep node* | wc -l".
>
> > + numa_service_num =
> > self.get_def_rte_config('CONFIG_RTE_MAX_NUMA_NODES')
> > + if numa_service_num:
> > + numa = min(total_numa_nodes, numa_service_num)
> > + else:
> > + numa = total_numa_nodes
> > force_socket = False
> >
> > if int(hugepages_size) < (1024 * 1024):
> > @@ -332,7 +338,8 @@ class Dut(Crb):
> > if force_socket:
> > self.set_huge_pages(arch_huge_pages, 0)
> > else:
> > - self.set_huge_pages(arch_huge_pages)
> > + for numa_id in range(0, int(numa)):
> > + self.set_huge_pages(arch_huge_pages, numa_id)
> >
> > self.mount_huge_pages()
> > self.hugepage_path = self.strip_hugepage_path()
> > --
> > 2.7.4
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dts] [PATCH] framework/dut: setup hugepage for all available NUMA nodes
2018-08-28 2:00 ` Phil Yang
@ 2018-08-28 2:57 ` Liu, Yong
2018-08-28 3:10 ` Phil Yang
0 siblings, 1 reply; 8+ messages in thread
From: Liu, Yong @ 2018-08-28 2:57 UTC (permalink / raw)
To: Phil.Yang, dts, dts; +Cc: nd
You're welcome, Phil.
One more comment, It's better to make total_numa_nodes and numa_service_num variables as integer before comparing them.
Function "min" may not return the expect result when numa number over ten.
Thanks,
Marvin
> -----Original Message-----
> From: Phil.Yang@arm.com [mailto:Phil.Yang@arm.com]
> Sent: Tuesday, August 28, 2018 10:01 AM
> To: Liu, Yong <yong.liu@intel.com>; dts <dts-bounces@dpdk.org>; dts@dpdk.org
> Cc: nd <nd@arm.com>
> Subject: RE: [dts] [PATCH] framework/dut: setup hugepage for all available
> NUMA nodes
>
> Hi Marvin,
>
> Thanks for your comment.
> I reworked this patch, please review it.
>
> Thanks,
> Phil Yang
>
> > -----Original Message-----
> > From: dts <dts-bounces@dpdk.org> On Behalf Of Liu, Yong
> > Sent: Monday, August 27, 2018 1:28 PM
> > To: dts <dts-bounces@dpdk.org>; dts@dpdk.org
> > Cc: nd <nd@arm.com>
> > Subject: Re: [dts] [PATCH] framework/dut: setup hugepage for all available
> > NUMA nodes
> >
> > Hi Phil,
> > One comment is inline.
> >
> > Thanks,
> > Marvin
> >
> > > -----Original Message-----
> > > From: dts [mailto:dts-bounces@dpdk.org]
> > > Sent: Monday, August 27, 2018 9:16 AM
> > > To: dts@dpdk.org
> > > Cc: nd@arm.com
> > > Subject: [dts] [PATCH] framework/dut: setup hugepage for all available
> > > NUMA nodes
> > >
> > > On a NUMA machine, we need to setup hugepage for all NUMA nodes.
> > >
> > > The default approach is by setting /sys/kernel/mm/hugepages, but it
> > > only works on a single-node system.
> > >
> > > This fix needs numactl tool support on Linux.
> > >
> > > Signed-off-by: Phil Yang <phil.yang@arm.com>
> > > ---
> > > framework/crb.py | 6 ++++--
> > > framework/dut.py | 9 ++++++++-
> > > 2 files changed, 12 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/framework/crb.py b/framework/crb.py index
> > > 5c555db..97bebfe 100644
> > > --- a/framework/crb.py
> > > +++ b/framework/crb.py
> > > @@ -204,8 +204,10 @@ class Crb(object):
> > > if numa == -1:
> > > self.send_expect('echo %d >
> > > /sys/kernel/mm/hugepages/hugepages-
> > > %skB/nr_hugepages' % (huge_pages, page_size), '# ', 5)
> > > else:
> > > - # sometimes we set hugepage on kernel cmdline, so we need
> clear
> > > default hugepage
> > > - self.send_expect('echo 0 >
> /sys/kernel/mm/hugepages/hugepages-
> > > %skB/nr_hugepages' % (page_size), '# ', 5)
> > > + # sometimes we set hugepage on kernel cmdline, so we
> > > + clear all
> > > nodes' default hugepages at the first time.
> > > + if numa == 0:
> > > + self.send_expect('echo 0 >
> > > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (page_size),
> > > '# ', 5)
> > > +
> > > # some platform not support numa, example vm dut
> > > try:
> > > self.send_expect('echo %d >
> > > /sys/devices/system/node/node%d/hugepages/hugepages-
> > %skB/nr_hugepages'
> > > % (huge_pages, numa, page_size), '# ', 5) diff --git
> > > a/framework/dut.py b/framework/dut.py index 18f0b39..b358578 100644
> > > --- a/framework/dut.py
> > > +++ b/framework/dut.py
> > > @@ -305,6 +305,12 @@ class Dut(Crb):
> > > return
> > > hugepages_size = self.send_expect("awk '/Hugepagesize/ {print
> $2}'
> > > /proc/meminfo", "# ")
> > > total_huge_pages = self.get_total_huge_pages()
> > > + total_numa_nodes = self.send_expect("numactl -H | awk
> > > + '/available/
> > > {print $2}'", "# ")
> >
> > We may not expect that numactl is installed on the DUT. It will be more
> portable
> > by checking sysfs like "ls /sys/devices/system/node | grep node* | wc -l".
> >
> > > + numa_service_num =
> > > self.get_def_rte_config('CONFIG_RTE_MAX_NUMA_NODES')
> > > + if numa_service_num:
> > > + numa = min(total_numa_nodes, numa_service_num)
> > > + else:
> > > + numa = total_numa_nodes
> > > force_socket = False
> > >
> > > if int(hugepages_size) < (1024 * 1024):
> > > @@ -332,7 +338,8 @@ class Dut(Crb):
> > > if force_socket:
> > > self.set_huge_pages(arch_huge_pages, 0)
> > > else:
> > > - self.set_huge_pages(arch_huge_pages)
> > > + for numa_id in range(0, int(numa)):
> > > + self.set_huge_pages(arch_huge_pages, numa_id)
> > >
> > > self.mount_huge_pages()
> > > self.hugepage_path = self.strip_hugepage_path()
> > > --
> > > 2.7.4
^ permalink raw reply [flat|nested] 8+ messages in thread
* [dts] [PATCH v3] framework/dut: setup hugepage for all available numa nodes
2018-08-28 1:59 ` [dts] [PATCH v2] framework/dut: setup hugepage for all available numa nodes Phil Yang
@ 2018-08-28 3:08 ` Phil Yang
2018-09-03 8:32 ` Liu, Yong
0 siblings, 1 reply; 8+ messages in thread
From: Phil Yang @ 2018-08-28 3:08 UTC (permalink / raw)
To: dts; +Cc: nd, yong.liu, Phil Yang
From: Phil Yang <phil.yang@arm.com>
On a NUMA machine, we need to setup hugepage for all NUMA nodes.
The default approach is by setting /sys/kernel/mm/hugepages, but it
only works on a single-node system.
Signed-off-by: Phil Yang <phil.yang@arm.com>
---
framework/crb.py | 6 ++++--
framework/dut.py | 9 ++++++++-
2 files changed, 12 insertions(+), 3 deletions(-)
diff --git a/framework/crb.py b/framework/crb.py
index 5c555db..97bebfe 100644
--- a/framework/crb.py
+++ b/framework/crb.py
@@ -204,8 +204,10 @@ class Crb(object):
if numa == -1:
self.send_expect('echo %d > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (huge_pages, page_size), '# ', 5)
else:
- # sometimes we set hugepage on kernel cmdline, so we need clear default hugepage
- self.send_expect('echo 0 > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (page_size), '# ', 5)
+ # sometimes we set hugepage on kernel cmdline, so we clear all nodes' default hugepages at the first time.
+ if numa == 0:
+ self.send_expect('echo 0 > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (page_size), '# ', 5)
+
# some platform not support numa, example vm dut
try:
self.send_expect('echo %d > /sys/devices/system/node/node%d/hugepages/hugepages-%skB/nr_hugepages' % (huge_pages, numa, page_size), '# ', 5)
diff --git a/framework/dut.py b/framework/dut.py
index 18f0b39..a8116ff 100644
--- a/framework/dut.py
+++ b/framework/dut.py
@@ -305,6 +305,12 @@ class Dut(Crb):
return
hugepages_size = self.send_expect("awk '/Hugepagesize/ {print $2}' /proc/meminfo", "# ")
total_huge_pages = self.get_total_huge_pages()
+ total_numa_nodes = self.send_expect("ls /sys/devices/system/node | grep node* | wc -l", "# ")
+ numa_service_num = self.get_def_rte_config('CONFIG_RTE_MAX_NUMA_NODES')
+ if numa_service_num is not None:
+ numa = min(int(total_numa_nodes), int(numa_service_num))
+ else:
+ numa = total_numa_nodes
force_socket = False
if int(hugepages_size) < (1024 * 1024):
@@ -332,7 +338,8 @@ class Dut(Crb):
if force_socket:
self.set_huge_pages(arch_huge_pages, 0)
else:
- self.set_huge_pages(arch_huge_pages)
+ for numa_id in range(0, int(numa)):
+ self.set_huge_pages(arch_huge_pages, numa_id)
self.mount_huge_pages()
self.hugepage_path = self.strip_hugepage_path()
--
2.7.4
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dts] [PATCH] framework/dut: setup hugepage for all available NUMA nodes
2018-08-28 2:57 ` Liu, Yong
@ 2018-08-28 3:10 ` Phil Yang
0 siblings, 0 replies; 8+ messages in thread
From: Phil Yang @ 2018-08-28 3:10 UTC (permalink / raw)
To: Liu, Yong, dts, dts; +Cc: nd
Yes. That's very thoughtful.
Reworked. Please review it.
Thanks,
Phil Yang
> -----Original Message-----
> From: Liu, Yong <yong.liu@intel.com>
> Sent: Tuesday, August 28, 2018 10:58 AM
> To: Phil Yang <Phil.Yang@arm.com>; dts <dts-bounces@dpdk.org>;
> dts@dpdk.org
> Cc: nd <nd@arm.com>
> Subject: RE: [dts] [PATCH] framework/dut: setup hugepage for all available
> NUMA nodes
>
> You're welcome, Phil.
> One more comment, It's better to make total_numa_nodes and
> numa_service_num variables as integer before comparing them.
> Function "min" may not return the expect result when numa number over ten.
>
> Thanks,
> Marvin
>
> > -----Original Message-----
> > From: Phil.Yang@arm.com [mailto:Phil.Yang@arm.com]
> > Sent: Tuesday, August 28, 2018 10:01 AM
> > To: Liu, Yong <yong.liu@intel.com>; dts <dts-bounces@dpdk.org>;
> > dts@dpdk.org
> > Cc: nd <nd@arm.com>
> > Subject: RE: [dts] [PATCH] framework/dut: setup hugepage for all
> > available NUMA nodes
> >
> > Hi Marvin,
> >
> > Thanks for your comment.
> > I reworked this patch, please review it.
> >
> > Thanks,
> > Phil Yang
> >
> > > -----Original Message-----
> > > From: dts <dts-bounces@dpdk.org> On Behalf Of Liu, Yong
> > > Sent: Monday, August 27, 2018 1:28 PM
> > > To: dts <dts-bounces@dpdk.org>; dts@dpdk.org
> > > Cc: nd <nd@arm.com>
> > > Subject: Re: [dts] [PATCH] framework/dut: setup hugepage for all
> > > available NUMA nodes
> > >
> > > Hi Phil,
> > > One comment is inline.
> > >
> > > Thanks,
> > > Marvin
> > >
> > > > -----Original Message-----
> > > > From: dts [mailto:dts-bounces@dpdk.org]
> > > > Sent: Monday, August 27, 2018 9:16 AM
> > > > To: dts@dpdk.org
> > > > Cc: nd@arm.com
> > > > Subject: [dts] [PATCH] framework/dut: setup hugepage for all
> > > > available NUMA nodes
> > > >
> > > > On a NUMA machine, we need to setup hugepage for all NUMA nodes.
> > > >
> > > > The default approach is by setting /sys/kernel/mm/hugepages, but
> > > > it only works on a single-node system.
> > > >
> > > > This fix needs numactl tool support on Linux.
> > > >
> > > > Signed-off-by: Phil Yang <phil.yang@arm.com>
> > > > ---
> > > > framework/crb.py | 6 ++++--
> > > > framework/dut.py | 9 ++++++++-
> > > > 2 files changed, 12 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/framework/crb.py b/framework/crb.py index
> > > > 5c555db..97bebfe 100644
> > > > --- a/framework/crb.py
> > > > +++ b/framework/crb.py
> > > > @@ -204,8 +204,10 @@ class Crb(object):
> > > > if numa == -1:
> > > > self.send_expect('echo %d >
> > > > /sys/kernel/mm/hugepages/hugepages-
> > > > %skB/nr_hugepages' % (huge_pages, page_size), '# ', 5)
> > > > else:
> > > > - # sometimes we set hugepage on kernel cmdline, so we need
> > clear
> > > > default hugepage
> > > > - self.send_expect('echo 0 >
> > /sys/kernel/mm/hugepages/hugepages-
> > > > %skB/nr_hugepages' % (page_size), '# ', 5)
> > > > + # sometimes we set hugepage on kernel cmdline, so we
> > > > + clear all
> > > > nodes' default hugepages at the first time.
> > > > + if numa == 0:
> > > > + self.send_expect('echo 0 >
> > > > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' %
> > > > (page_size), '# ', 5)
> > > > +
> > > > # some platform not support numa, example vm dut
> > > > try:
> > > > self.send_expect('echo %d >
> > > > /sys/devices/system/node/node%d/hugepages/hugepages-
> > > %skB/nr_hugepages'
> > > > % (huge_pages, numa, page_size), '# ', 5) diff --git
> > > > a/framework/dut.py b/framework/dut.py index 18f0b39..b358578
> > > > 100644
> > > > --- a/framework/dut.py
> > > > +++ b/framework/dut.py
> > > > @@ -305,6 +305,12 @@ class Dut(Crb):
> > > > return
> > > > hugepages_size = self.send_expect("awk '/Hugepagesize/
> > > > {print
> > $2}'
> > > > /proc/meminfo", "# ")
> > > > total_huge_pages = self.get_total_huge_pages()
> > > > + total_numa_nodes = self.send_expect("numactl -H | awk
> > > > + '/available/
> > > > {print $2}'", "# ")
> > >
> > > We may not expect that numactl is installed on the DUT. It will be
> > > more
> > portable
> > > by checking sysfs like "ls /sys/devices/system/node | grep node* | wc -l".
> > >
> > > > + numa_service_num =
> > > > self.get_def_rte_config('CONFIG_RTE_MAX_NUMA_NODES')
> > > > + if numa_service_num:
> > > > + numa = min(total_numa_nodes, numa_service_num)
> > > > + else:
> > > > + numa = total_numa_nodes
> > > > force_socket = False
> > > >
> > > > if int(hugepages_size) < (1024 * 1024):
> > > > @@ -332,7 +338,8 @@ class Dut(Crb):
> > > > if force_socket:
> > > > self.set_huge_pages(arch_huge_pages, 0)
> > > > else:
> > > > - self.set_huge_pages(arch_huge_pages)
> > > > + for numa_id in range(0, int(numa)):
> > > > + self.set_huge_pages(arch_huge_pages,
> > > > + numa_id)
> > > >
> > > > self.mount_huge_pages()
> > > > self.hugepage_path = self.strip_hugepage_path()
> > > > --
> > > > 2.7.4
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [dts] [PATCH v3] framework/dut: setup hugepage for all available numa nodes
2018-08-28 3:08 ` [dts] [PATCH v3] " Phil Yang
@ 2018-09-03 8:32 ` Liu, Yong
0 siblings, 0 replies; 8+ messages in thread
From: Liu, Yong @ 2018-09-03 8:32 UTC (permalink / raw)
To: ,; +Cc: nd, Phil Yang
Thanks, Phil. Applied.
On 08/28/2018 11:08 AM, wrote:
> From: Phil Yang<phil.yang@arm.com>
>
> On a NUMA machine, we need to setup hugepage for all NUMA nodes.
>
> The default approach is by setting /sys/kernel/mm/hugepages, but it
> only works on a single-node system.
>
> Signed-off-by: Phil Yang<phil.yang@arm.com>
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-09-03 0:49 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-08-27 1:16 [dts] [PATCH] framework/dut: setup hugepage for all available NUMA nodes Phil Yang
2018-08-27 5:27 ` Liu, Yong
2018-08-28 2:00 ` Phil Yang
2018-08-28 2:57 ` Liu, Yong
2018-08-28 3:10 ` Phil Yang
2018-08-28 1:59 ` [dts] [PATCH v2] framework/dut: setup hugepage for all available numa nodes Phil Yang
2018-08-28 3:08 ` [dts] [PATCH v3] " Phil Yang
2018-09-03 8:32 ` Liu, Yong
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).