test suite reviews and discussions
 help / color / mirror / Atom feed
* [dts] [DTS] [PATCH v2] Framework:fix the first numa node may not be '0'
@ 2021-09-30 17:20 Thinh Tran
  2021-10-09  8:36 ` Tu, Lijuan
  2021-10-16  6:45 ` [dts] [PATCH v3] " Thinh Tran
  0 siblings, 2 replies; 4+ messages in thread
From: Thinh Tran @ 2021-09-30 17:20 UTC (permalink / raw)
  To: dts; +Cc: Thinh Tran

On certain Virtual Machine system  such as Power Virtual Machine, PowerVM,
assigns the CPUs of its virtual machines (or Logical Partitions LPARs) to
NUMA nodes, and so the VM (or LPARs) 's CPUs may be on different ID
other than '0'.
This patch should fix that.

v2: correct spelling and remove trailing whitespace

Signed-off-by: Thinh Tran <thinhtr@linux.vnet.ibm.com>
---
 framework/crb.py | 13 ++++++-------
 framework/dut.py | 32 +++++++++++++++++++-------------
 2 files changed, 25 insertions(+), 20 deletions(-)

diff --git a/framework/crb.py b/framework/crb.py
index 40fe4fd1..9b0742ba 100644
--- a/framework/crb.py
+++ b/framework/crb.py
@@ -204,24 +204,23 @@ class Crb(object):
         else:
             return ''
 
-    def set_huge_pages(self, huge_pages, numa=-1):
+    def set_huge_pages(self, huge_pages, numa=""):
         """
         Set numbers of huge pages
         """
         page_size = self.send_expect("awk '/Hugepagesize/ {print $2}' /proc/meminfo", "# ")
 
-        if numa == -1:
+        if not numa:
             self.send_expect('echo %d > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (huge_pages, page_size), '# ', 5)
         else:
-            # sometimes we set hugepage on kernel cmdline, so we clear all nodes' default hugepages at the first time.
-            if numa == 0:
-                self.send_expect('echo 0 > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (page_size), '# ', 5)
+            # sometimes we set hugepage on kernel cmdline, so we clear it
+            self.send_expect('echo 0 > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (page_size), '# ', 5)
 
             # some platform not support numa, example vm dut
             try:
-                self.send_expect('echo %d > /sys/devices/system/node/node%d/hugepages/hugepages-%skB/nr_hugepages' % (huge_pages, numa, page_size), '# ', 5)
+                self.send_expect('echo %d > /sys/devices/system/node/%s/hugepages/hugepages-%skB/nr_hugepages' % (huge_pages, numa, page_size), '# ', 5)
             except:
-                self.logger.warning("set %d hugepage on socket %d error" % (huge_pages, numa))
+                self.logger.warning("set %d hugepage on %s error" % (huge_pages, numa))
                 self.send_expect('echo %d > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (huge_pages. page_size), '# ', 5)
 
     def set_speedup_options(self, read_cache, skip_setup):
diff --git a/framework/dut.py b/framework/dut.py
index f5481d06..5c9a6b23 100644
--- a/framework/dut.py
+++ b/framework/dut.py
@@ -547,16 +547,15 @@ class Dut(Crb):
             return
         hugepages_size = self.send_expect("awk '/Hugepagesize/ {print $2}' /proc/meminfo", "# ")
         total_huge_pages = self.get_total_huge_pages()
-        total_numa_nodes = self.send_expect("ls /sys/devices/system/node | grep node* | wc -l", "# ")
-        numa_service_num = self.get_def_rte_config('CONFIG_RTE_MAX_NUMA_NODES')
-        try:
-            int(total_numa_nodes)
-        except ValueError:
+        numa_nodes = self.send_expect("ls /sys/devices/system/node | grep node*", "# ")
+        if not numa_nodes:
             total_numa_nodes = -1
-        if numa_service_num is not None:
-            numa = min(int(total_numa_nodes), int(numa_service_num))
         else:
-            numa = total_numa_nodes
+            numa_nodes = numa_nodes.splitlines()
+            total_numa_nodes = len(numa_nodes)
+            self.logger.info(numa_nodes)
+
+
         force_socket = False
 
         if int(hugepages_size) < (1024 * 1024):
@@ -580,14 +579,21 @@ class Dut(Crb):
             if total_huge_pages != arch_huge_pages:
                 # before all hugepage average distribution  by all socket,
                 # but sometimes create mbuf pool on socket 0 failed when setup testpmd,
-                # so set all huge page on socket 0
+                # so set all huge page on first socket
                 if force_socket:
-                    self.set_huge_pages(arch_huge_pages, 0)
+                    self.set_huge_pages(arch_huge_pages, numa_nodes[0])
+                    self.logger.info("force_socket on %s" % numa_nodes[0])
                 else:
-                    for numa_id in range(0, int(numa)):
-                        self.set_huge_pages(arch_huge_pages, numa_id)
-                    if numa == -1:
+                    if total_numa_nodes == -1:
                         self.set_huge_pages(arch_huge_pages)
+                    else:
+                        numa_service_num = self.get_def_rte_config('CONFIG_RTE_MAX_NUMA_NODES')
+                        if numa_service_num is not None:
+                            total_numa_nodes = min(total_numa_nodes, int(numa_service_num))
+
+                        # set huge pages to configured total_numa_nodes
+                        for numa_node in numa_nodes[:total_numa_nodes]:
+                            self.set_huge_pages(arch_huge_pages, numa_node)
 
         self.mount_huge_pages()
         self.hugepage_path = self.strip_hugepage_path()
-- 
2.25.1


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [dts] [DTS] [PATCH v2] Framework:fix the first numa node may not be '0'
  2021-09-30 17:20 [dts] [DTS] [PATCH v2] Framework:fix the first numa node may not be '0' Thinh Tran
@ 2021-10-09  8:36 ` Tu, Lijuan
  2021-10-16  6:45 ` [dts] [PATCH v3] " Thinh Tran
  1 sibling, 0 replies; 4+ messages in thread
From: Tu, Lijuan @ 2021-10-09  8:36 UTC (permalink / raw)
  To: Thinh Tran, dts

> -----Original Message-----
> From: dts <dts-bounces@dpdk.org> On Behalf Of Thinh Tran
> Sent: 2021年10月1日 1:21
> To: dts@dpdk.org
> Cc: Thinh Tran <thinhtr@linux.vnet.ibm.com>
> Subject: [dts] [DTS] [PATCH v2] Framework:fix the first numa node may not be
> '0'
> 
> On certain Virtual Machine system  such as Power Virtual Machine, PowerVM,
> assigns the CPUs of its virtual machines (or Logical Partitions LPARs) to NUMA
> nodes, and so the VM (or LPARs) 's CPUs may be on different ID other than '0'.
> This patch should fix that.
> 
> v2: correct spelling and remove trailing whitespace
> 
> Signed-off-by: Thinh Tran <thinhtr@linux.vnet.ibm.com>

Applied

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [dts] [PATCH v3] Framework:fix the first numa node may not be '0'
  2021-09-30 17:20 [dts] [DTS] [PATCH v2] Framework:fix the first numa node may not be '0' Thinh Tran
  2021-10-09  8:36 ` Tu, Lijuan
@ 2021-10-16  6:45 ` Thinh Tran
  2021-10-25  3:00   ` Tu, Lijuan
  1 sibling, 1 reply; 4+ messages in thread
From: Thinh Tran @ 2021-10-16  6:45 UTC (permalink / raw)
  To: dts; +Cc: weix.ling, yux.jiang, lijuan.tu, Thinh Tran

On certain Virtual Machine system  such as Power Virtual Machine, PowerVM,
assigns the CPUs of its virtual machines (or Logical Partitions LPARs) to
NUMA nodes, and so the VM (or LPARs) 's CPUs may be on different ID
other than '0'.
This patch should fix that.

v2: correct spelling and remove trailing whitespace

v3: 
* adding a flag to check if default hugepages is already cleared

Signed-off-by: Thinh Tran <thinhtr@linux.vnet.ibm.com>
---
 framework/crb.py | 14 ++++++++------
 framework/dut.py | 40 +++++++++++++++++++++++-----------------
 2 files changed, 31 insertions(+), 23 deletions(-)

diff --git a/framework/crb.py b/framework/crb.py
index 40fe4fd1..89255f00 100644
--- a/framework/crb.py
+++ b/framework/crb.py
@@ -61,6 +61,7 @@ class Crb(object):
         self.stage = 'pre-init'
         self.name = name
         self.trex_prefix = None
+        self.default_hugepages_cleared = False
 
         self.logger = getLogger(name)
         self.session = SSHConnection(self.get_ip_address(), name,
@@ -204,24 +205,25 @@ class Crb(object):
         else:
             return ''
 
-    def set_huge_pages(self, huge_pages, numa=-1):
+    def set_huge_pages(self, huge_pages, numa=""):
         """
         Set numbers of huge pages
         """
         page_size = self.send_expect("awk '/Hugepagesize/ {print $2}' /proc/meminfo", "# ")
 
-        if numa == -1:
+        if not numa:
             self.send_expect('echo %d > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (huge_pages, page_size), '# ', 5)
         else:
-            # sometimes we set hugepage on kernel cmdline, so we clear all nodes' default hugepages at the first time.
-            if numa == 0:
+            # sometimes we set hugepage on kernel cmdline, so we clear it
+            if not self.default_hugepages_cleared:
                 self.send_expect('echo 0 > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (page_size), '# ', 5)
+                self.default_hugepages_cleared = True
 
             # some platform not support numa, example vm dut
             try:
-                self.send_expect('echo %d > /sys/devices/system/node/node%d/hugepages/hugepages-%skB/nr_hugepages' % (huge_pages, numa, page_size), '# ', 5)
+                self.send_expect('echo %d > /sys/devices/system/node/%s/hugepages/hugepages-%skB/nr_hugepages' % (huge_pages, numa, page_size), '# ', 5)
             except:
-                self.logger.warning("set %d hugepage on socket %d error" % (huge_pages, numa))
+                self.logger.warning("set %d hugepage on %s error" % (huge_pages, numa))
                 self.send_expect('echo %d > /sys/kernel/mm/hugepages/hugepages-%skB/nr_hugepages' % (huge_pages. page_size), '# ', 5)
 
     def set_speedup_options(self, read_cache, skip_setup):
diff --git a/framework/dut.py b/framework/dut.py
index f5481d06..65e8d597 100644
--- a/framework/dut.py
+++ b/framework/dut.py
@@ -547,16 +547,15 @@ class Dut(Crb):
             return
         hugepages_size = self.send_expect("awk '/Hugepagesize/ {print $2}' /proc/meminfo", "# ")
         total_huge_pages = self.get_total_huge_pages()
-        total_numa_nodes = self.send_expect("ls /sys/devices/system/node | grep node* | wc -l", "# ")
-        numa_service_num = self.get_def_rte_config('CONFIG_RTE_MAX_NUMA_NODES')
-        try:
-            int(total_numa_nodes)
-        except ValueError:
+        numa_nodes = self.send_expect("ls /sys/devices/system/node | grep node*", "# ")
+        if not numa_nodes:
             total_numa_nodes = -1
-        if numa_service_num is not None:
-            numa = min(int(total_numa_nodes), int(numa_service_num))
         else:
-            numa = total_numa_nodes
+            numa_nodes = numa_nodes.splitlines()
+            total_numa_nodes = len(numa_nodes)
+            self.logger.info(numa_nodes)
+
+
         force_socket = False
 
         if int(hugepages_size) < (1024 * 1024):
@@ -578,16 +577,23 @@ class Dut(Crb):
                     arch_huge_pages = hugepages if hugepages > 0 else 2048
 
             if total_huge_pages != arch_huge_pages:
-                # before all hugepage average distribution  by all socket,
-                # but sometimes create mbuf pool on socket 0 failed when setup testpmd,
-                # so set all huge page on socket 0
-                if force_socket:
-                    self.set_huge_pages(arch_huge_pages, 0)
+                if total_numa_nodes == -1 :
+                    self.set_huge_pages(arch_huge_pages)
                 else:
-                    for numa_id in range(0, int(numa)):
-                        self.set_huge_pages(arch_huge_pages, numa_id)
-                    if numa == -1:
-                        self.set_huge_pages(arch_huge_pages)
+                    # before all hugepage average distribution  by all socket,
+                    # but sometimes create mbuf pool on socket 0 failed when 
+                    # setup testpmd, so set all huge page on first socket
+                    if force_socket:
+                        self.set_huge_pages(arch_huge_pages, numa_nodes[0])
+                        self.logger.info("force_socket on %s" % numa_nodes[0])
+                    else:
+                        numa_service_num = self.get_def_rte_config('CONFIG_RTE_MAX_NUMA_NODES')
+                        if numa_service_num is not None:
+                            total_numa_nodes = min(total_numa_nodes, int(numa_service_num))
+
+                        # set huge pages to configured total_numa_nodes
+                        for numa_node in numa_nodes[:total_numa_nodes]:
+                            self.set_huge_pages(arch_huge_pages, numa_node)
 
         self.mount_huge_pages()
         self.hugepage_path = self.strip_hugepage_path()
-- 
2.25.1


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [dts] [PATCH v3] Framework:fix the first numa node may not be '0'
  2021-10-16  6:45 ` [dts] [PATCH v3] " Thinh Tran
@ 2021-10-25  3:00   ` Tu, Lijuan
  0 siblings, 0 replies; 4+ messages in thread
From: Tu, Lijuan @ 2021-10-25  3:00 UTC (permalink / raw)
  To: Thinh Tran, dts; +Cc: Ling, WeiX, Jiang, YuX

> -----Original Message-----
> From: Thinh Tran <thinhtr@linux.vnet.ibm.com>
> Sent: 2021年10月16日 14:46
> To: dts@dpdk.org
> Cc: Ling, WeiX <weix.ling@intel.com>; Jiang, YuX <yux.jiang@intel.com>; Tu,
> Lijuan <lijuan.tu@intel.com>; Thinh Tran <thinhtr@linux.vnet.ibm.com>
> Subject: [dts] [PATCH v3] Framework:fix the first numa node may not be '0'
> 
> On certain Virtual Machine system  such as Power Virtual Machine, PowerVM,
> assigns the CPUs of its virtual machines (or Logical Partitions LPARs) to NUMA
> nodes, and so the VM (or LPARs) 's CPUs may be on different ID other than '0'.
> This patch should fix that.
> 
> v2: correct spelling and remove trailing whitespace
> 
> v3:
> * adding a flag to check if default hugepages is already cleared
> 
> Signed-off-by: Thinh Tran <thinhtr@linux.vnet.ibm.com>

Applied, thanks

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2021-10-25  3:01 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-30 17:20 [dts] [DTS] [PATCH v2] Framework:fix the first numa node may not be '0' Thinh Tran
2021-10-09  8:36 ` Tu, Lijuan
2021-10-16  6:45 ` [dts] [PATCH v3] " Thinh Tran
2021-10-25  3:00   ` Tu, Lijuan

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).