[dts] [PATCH 0/4] Load CRBs information from configuration file

[dts] [PATCH 0/4] Load CRBs information from configuration file

[Yong Liu]

From: Marvin Liu <yong.liu@intel.com>

All configuration for running DTS will be moved to files. This patch set will
move dut/tester related settings from python module to file. If file missing
prerequisite parameter, config module will load default configuraton.

Marvin Liu (4):
  Support load crbs configuration file
  Add sample crbs configuration file
  Move function accepted_nic to dts module
  Load CRBs information from configuration file

 conf/crbs.cfg             | 28 +++++++++++++++++++++++
 framework/config.py       | 58 ++++++++++++++++++++++++++++++++++++++++++++++-
 framework/crbs.py         | 15 ------------
 framework/dts.py          | 25 ++++++++++++++++++--
 framework/dut.py          |  6 ++---
 framework/project_dpdk.py |  4 ++--
 framework/settings.py     | 17 --------------
 7 files changed, 113 insertions(+), 40 deletions(-)
 create mode 100644 conf/crbs.cfg

-- 
1.9.3

[dts] [PATCH 1/4] Support load crbs configuration file

[Yong Liu]

From: Marvin Liu <yong.liu@intel.com>

Move CRBs configruations from internal python file to configuration file.

Signed-off-by: Marvin Liu <yong.liu@intel.com>

diff --git a/framework/config.py b/framework/config.py
index 65cecd4..f099a5d 100644
--- a/framework/config.py
+++ b/framework/config.py
@@ -36,7 +36,7 @@ Generic port and crbs configuration file load function
 import re
 import ConfigParser  # config parse module
 import argparse      # prase arguments module
-
+from settings import IXIA
 from exception import ConfigParseException, VirtConfigParseException
 
 PORTCONF = "conf/ports.cfg"
@@ -185,6 +185,58 @@ class PortConf(UserConf):
             return False
 
 
+class CrbsConf(UserConf):
+    DEF_CRB = {'IP': '', 'name': 'CrownPassCRB1', 'user': '',
+               'pass': '', 'tester IP': '', 'tester pass': '',
+               IXIA: None, 'memory channels': 4,
+               'bypass core0': True}
+
+    def __init__(self, crbs_conf=CRBCONF):
+        self.config_file = crbs_conf
+        self.crbs_cfg = []
+        try:
+            self.crbs_conf = UserConf(self.config_file)
+        except ConfigParseException:
+            self.crbs_conf = None
+            raise ConfigParseException
+
+    def load_crbs_config(self):
+        sections = self.crbs_conf.get_sections()
+        if not sections:
+            return self.crbs_cfg
+
+        for name in sections:
+            crb = self.DEF_CRB.copy()
+            crb_confs = self.crbs_conf.load_section(name)
+            if not crb_confs:
+                continue
+
+            # covert file configuration to dts crbs
+            for conf in crb_confs:
+                key, value = conf
+                if key == 'dut_ip':
+                    crb['IP'] = value
+                elif key == 'dut_user':
+                    crb['user'] = value
+                elif key == 'dut_passwd':
+                    crb['pass'] = value
+                elif key == 'tester_ip':
+                    crb['tester IP'] = value
+                elif key == 'tester_passwd':
+                    crb['tester pass'] = value
+                elif key == 'ixia_group':
+                    crb[IXIA] = value
+                elif key == 'channels':
+                    crb['memory channels'] = int(value)
+                elif key == 'bypass_core0':
+                    if value == 'True':
+                        crb['bypass core0'] = True
+                    else:
+                        crb['bypass core0'] = False
+
+            self.crbs_cfg.append(crb)
+        return self.crbs_cfg
+
 if __name__ == '__main__':
     parser = argparse.ArgumentParser(
         description="Load DTS configuration files")
@@ -218,3 +270,7 @@ if __name__ == '__main__':
     virtconf = VirtConf(VIRTCONF)
     virtconf.load_virt_config('LIBVIRT')
     print virtconf.get_virt_config()
+
+    # example for crbs configuration file
+    crbsconf = CrbsConf(CRBCONF)
+    print crbsconf.load_crbs_config()
-- 
1.9.3

[dts] [PATCH 2/4] Add sample crbs configuration file

[Yong Liu]

From: Marvin Liu <yong.liu@intel.com>

This file is sample file for CRBS configuration. User need to configure every
option in the file.

Signed-off-by: Marvin Liu <yong.liu@intel.com>

diff --git a/conf/crbs.cfg b/conf/crbs.cfg
new file mode 100644
index 0000000..de8e025
--- /dev/null
+++ b/conf/crbs.cfg
@@ -0,0 +1,28 @@
+#DUT crbs Configuration
+#[DUT IP]
+#  dut_ip: DUT ip address
+#  dut_user: Login DUT username
+#  dut_passwd: Login DUT password
+#  tester_ip: Tester ip address
+#  tester_passwd: Tester password
+#  ixia_group: IXIA group nmae
+#  channels: Board channel number
+#  bypass_core0: Whether by pass core0
+[DUT IP1]
+dut_ip=xxx.xxx.xxx.xxx
+dut_user=root
+dut_passwd=
+tester_ip=xxx.xxx.xxx.xxx
+tester_passwd=
+ixia_group=
+channels=4
+bypass_core0=True
+[DUT IP2]
+dut_ip=yyy.yyy.yyy.yyy
+dut_user=root
+dut_passwd=
+tester_ip=yyy.yyy.yyy.yyy
+tester_passwd=
+ixia_group=
+channels=4
+bypass_core0=True
-- 
1.9.3

[dts] [PATCH 3/4] Move function accepted_nic to dts module

[Yong Liu]

From: Marvin Liu <yong.liu@intel.com>

This function will use varaiable in dts. Moved to dts module will make
settings module independent from it.

Signed-off-by: Marvin Liu <yong.liu@intel.com>

diff --git a/framework/dut.py b/framework/dut.py
index a5c9db3..1199fc6 100644
--- a/framework/dut.py
+++ b/framework/dut.py
@@ -322,7 +322,7 @@ class Dut(Crb):
 
         current_nic = 0
         for (pci_bus, pci_id) in self.pci_devices_info:
-            if settings.accepted_nic(pci_id):
+            if dts.accepted_nic(pci_id):
                 if self.is_ssh_session_port(pci_bus):
                     continue
 
@@ -345,7 +345,7 @@ class Dut(Crb):
 
         current_nic = 0
         for (pci_bus, pci_id) in self.pci_devices_info:
-            if settings.accepted_nic(pci_id):
+            if dts.accepted_nic(pci_id):
                 if self.is_ssh_session_port(pci_bus):
                     continue
 
@@ -671,7 +671,7 @@ class Dut(Crb):
 
         for (pci_bus, pci_id) in self.pci_devices_info:
 
-            if not settings.accepted_nic(pci_id):
+            if not dts.accepted_nic(pci_id):
                 self.logger.info("DUT: [%s %s] %s" % (pci_bus, pci_id,
                                                       skipped))
                 continue
diff --git a/framework/project_dpdk.py b/framework/project_dpdk.py
index 7b6e1ae..e0009d3 100644
--- a/framework/project_dpdk.py
+++ b/framework/project_dpdk.py
@@ -39,7 +39,7 @@ from crb import Crb
 from dut import Dut
 from tester import Tester
 from logger import getLogger
-from settings import IXIA, accepted_nic
+from settings import IXIA
 
 
 class DPDKdut(Dut):
@@ -112,7 +112,7 @@ class DPDKdut(Dut):
         binding_list = ''
 
         for (pci_bus, pci_id) in self.pci_devices_info:
-            if accepted_nic(pci_id):
+            if dts.accepted_nic(pci_id):
                 binding_list += '%s,' % (pci_bus)
 
         self.send_expect("kldunload if_ixgbe.ko", "#")
diff --git a/framework/settings.py b/framework/settings.py
index 631bd32..b371fd7 100644
--- a/framework/settings.py
+++ b/framework/settings.py
@@ -179,23 +179,6 @@ def get_nic_driver(pci_id):
     return driver
 
 
-def accepted_nic(pci_id):
-    """
-    Return True if the pci_id is a known NIC card in the settings file and if
-    it is selected in the execution file, otherwise it returns False.
-    """
-    if pci_id not in NICS.values():
-        return False
-
-    if dts.nic is 'any':
-        return True
-
-    else:
-        if pci_id == NICS[dts.nic]:
-            return True
-
-    return False
-
 def get_netdev(crb, pci):
     for port in crb.ports_info:
         if pci == port['pci']:
-- 
1.9.3

[dts] [PATCH 4/4] Load CRBs information from configuration file

[Yong Liu]

From: Marvin Liu <yong.liu@intel.com>

Now crb instance will be loaded from configuration file. Remove useless staffs
from crbs.py.

Signed-off-by: Marvin Liu <yong.liu@intel.com>

diff --git a/framework/crbs.py b/framework/crbs.py
index 6aa5435..c586caa 100644
--- a/framework/crbs.py
+++ b/framework/crbs.py
@@ -1,21 +1,6 @@
 """
 Static configuration data for any CRBs that can be used.
 """
-from settings import IXIA
-
-# Todo: modify this script to a config file, like crbs.cfg
-crbs = [
-    {'IP': '',
-     'name': 'CrownPassCRB1',
-     'user': '',
-     'pass': '',
-     'tester IP': '',
-     'tester pass': '',
-     IXIA: None,
-     'memory channels': 4,
-     'bypass core0': True},
-]
-
 
 crbs_desc = {
     'CrownPassCRB1':
diff --git a/framework/dts.py b/framework/dts.py
index e9513c6..1b62077 100644
--- a/framework/dts.py
+++ b/framework/dts.py
@@ -41,7 +41,6 @@ import signal       # signal module for debug mode
 import time         # time module for unique output folder
 
 import rst          # rst file support
-from crbs import crbs
 from tester import Tester
 from dut import Dut
 from settings import FOLDERS, NICS, DRIVERS
@@ -57,6 +56,7 @@ from logger import getLogger
 import logger
 import debugger
 from virt_scene import VirtScene
+from config import CrbsConf
 
 import sys
 reload(sys)
@@ -453,6 +453,10 @@ def run_all(config_file, pkgName, git, patch, skip_setup,
     excel_report = ExcelReporter(output_dir + '/test_results.xls')
     stats = StatsReporter(output_dir + '/statistics.txt')
 
+    crbInst = None
+    crbs_conf = CrbsConf()
+    crbs = crbs_conf.load_crbs_config()
+
     # for all Exectuion sections
     for section in config.sections():
         dts_parse_param(section)
@@ -463,7 +467,6 @@ def run_all(config_file, pkgName, git, patch, skip_setup,
         log_handler.info("\nDUT " + dutIP)
 
         # look up in crbs - to find the matching IP
-        crbInst = None
         for crb in crbs:
             if crb['IP'] == dutIP:
                 crbInst = crb
@@ -735,3 +738,21 @@ def save_all_results():
     """
     excel_report.save(result)
     stats.save(result)
+
+def accepted_nic(pci_id):
+    """
+    Return True if the pci_id is a known NIC card in the settings file and if
+    it is selected in the execution file, otherwise it returns False.
+    """
+    global nic
+    if pci_id not in NICS.values():
+        return False
+
+    if nic is 'any':
+        return True
+
+    else:
+        if pci_id == NICS[nic]:
+            return True
+
+    return False
-- 
1.9.3

Re: [dts] [PATCH 1/4] Support load crbs configuration file

[Xu, HuilongX]

Hi yong,
In crbs dict could you add key 'os'.
If not os config, maybe  can't support freebsd test.

> -----Original Message-----
> From: dts [mailto:dts-bounces@dpdk.org] On Behalf Of Yong Liu
> Sent: Thursday, August 06, 2015 8:35 PM
> To: dts@dpdk.org
> Subject: [dts] [PATCH 1/4] Support load crbs configuration file
> 
> From: Marvin Liu <yong.liu@intel.com>
> 
> Move CRBs configruations from internal python file to configuration file.
> 
> Signed-off-by: Marvin Liu <yong.liu@intel.com>
> 
> diff --git a/framework/config.py b/framework/config.py
> index 65cecd4..f099a5d 100644
> --- a/framework/config.py
> +++ b/framework/config.py
> @@ -36,7 +36,7 @@ Generic port and crbs configuration file load function
>  import re
>  import ConfigParser  # config parse module
>  import argparse      # prase arguments module
> -
> +from settings import IXIA
>  from exception import ConfigParseException, VirtConfigParseException
> 
>  PORTCONF = "conf/ports.cfg"
> @@ -185,6 +185,58 @@ class PortConf(UserConf):
>              return False
> 
> 
> +class CrbsConf(UserConf):
> +    DEF_CRB = {'IP': '', 'name': 'CrownPassCRB1', 'user': '',
> +               'pass': '', 'tester IP': '', 'tester pass': '',
> +               IXIA: None, 'memory channels': 4,
> +               'bypass core0': True}
> +
> +    def __init__(self, crbs_conf=CRBCONF):
> +        self.config_file = crbs_conf
> +        self.crbs_cfg = []
> +        try:
> +            self.crbs_conf = UserConf(self.config_file)
> +        except ConfigParseException:
> +            self.crbs_conf = None
> +            raise ConfigParseException
> +
> +    def load_crbs_config(self):
> +        sections = self.crbs_conf.get_sections()
> +        if not sections:
> +            return self.crbs_cfg
> +
> +        for name in sections:
> +            crb = self.DEF_CRB.copy()
> +            crb_confs = self.crbs_conf.load_section(name)
> +            if not crb_confs:
> +                continue
> +
> +            # covert file configuration to dts crbs
> +            for conf in crb_confs:
> +                key, value = conf
> +                if key == 'dut_ip':
> +                    crb['IP'] = value
> +                elif key == 'dut_user':
> +                    crb['user'] = value
> +                elif key == 'dut_passwd':
> +                    crb['pass'] = value
> +                elif key == 'tester_ip':
> +                    crb['tester IP'] = value
> +                elif key == 'tester_passwd':
> +                    crb['tester pass'] = value
> +                elif key == 'ixia_group':
> +                    crb[IXIA] = value
> +                elif key == 'channels':
> +                    crb['memory channels'] = int(value)
> +                elif key == 'bypass_core0':
> +                    if value == 'True':
> +                        crb['bypass core0'] = True
> +                    else:
> +                        crb['bypass core0'] = False
> +
> +            self.crbs_cfg.append(crb)
> +        return self.crbs_cfg
> +
>  if __name__ == '__main__':
>      parser = argparse.ArgumentParser(
>          description="Load DTS configuration files")
> @@ -218,3 +270,7 @@ if __name__ == '__main__':
>      virtconf = VirtConf(VIRTCONF)
>      virtconf.load_virt_config('LIBVIRT')
>      print virtconf.get_virt_config()
> +
> +    # example for crbs configuration file
> +    crbsconf = CrbsConf(CRBCONF)
> +    print crbsconf.load_crbs_config()
> --
> 1.9.3

Re: [dts] [PATCH 2/4] Add sample crbs configuration file

[Xu, HuilongX]

Hi yong,
Could you move "ixia group" to config also.
Thanks a lot

> -----Original Message-----
> From: dts [mailto:dts-bounces@dpdk.org] On Behalf Of Yong Liu
> Sent: Thursday, August 06, 2015 8:35 PM
> To: dts@dpdk.org
> Subject: [dts] [PATCH 2/4] Add sample crbs configuration file
> 
> From: Marvin Liu <yong.liu@intel.com>
> 
> This file is sample file for CRBS configuration. User need to configure
> every
> option in the file.
> 
> Signed-off-by: Marvin Liu <yong.liu@intel.com>
> 
> diff --git a/conf/crbs.cfg b/conf/crbs.cfg
> new file mode 100644
> index 0000000..de8e025
> --- /dev/null
> +++ b/conf/crbs.cfg
> @@ -0,0 +1,28 @@
> +#DUT crbs Configuration
> +#[DUT IP]
> +#  dut_ip: DUT ip address
> +#  dut_user: Login DUT username
> +#  dut_passwd: Login DUT password
> +#  tester_ip: Tester ip address
> +#  tester_passwd: Tester password
> +#  ixia_group: IXIA group nmae
> +#  channels: Board channel number
> +#  bypass_core0: Whether by pass core0
> +[DUT IP1]
> +dut_ip=xxx.xxx.xxx.xxx
> +dut_user=root
> +dut_passwd=
> +tester_ip=xxx.xxx.xxx.xxx
> +tester_passwd=
> +ixia_group=
> +channels=4
> +bypass_core0=True
> +[DUT IP2]
> +dut_ip=yyy.yyy.yyy.yyy
> +dut_user=root
> +dut_passwd=
> +tester_ip=yyy.yyy.yyy.yyy
> +tester_passwd=
> +ixia_group=
> +channels=4
> +bypass_core0=True
> --
> 1.9.3

Re: [dts] [PATCH 1/4] Support load crbs configuration file

[Liu, Yong]

Ok, I'll send out v2 patch.

> -----Original Message-----
> From: Xu, HuilongX
> Sent: Friday, August 07, 2015 9:58 AM
> To: Liu, Yong; dts@dpdk.org
> Subject: RE: [dts] [PATCH 1/4] Support load crbs configuration file
> 
> Hi yong,
> In crbs dict could you add key 'os'.
> If not os config, maybe  can't support freebsd test.
> 
> > -----Original Message-----
> > From: dts [mailto:dts-bounces@dpdk.org] On Behalf Of Yong Liu
> > Sent: Thursday, August 06, 2015 8:35 PM
> > To: dts@dpdk.org
> > Subject: [dts] [PATCH 1/4] Support load crbs configuration file
> >
> > From: Marvin Liu <yong.liu@intel.com>
> >
> > Move CRBs configruations from internal python file to configuration file.
> >
> > Signed-off-by: Marvin Liu <yong.liu@intel.com>
> >
> > diff --git a/framework/config.py b/framework/config.py
> > index 65cecd4..f099a5d 100644
> > --- a/framework/config.py
> > +++ b/framework/config.py
> > @@ -36,7 +36,7 @@ Generic port and crbs configuration file load function
> >  import re
> >  import ConfigParser  # config parse module
> >  import argparse      # prase arguments module
> > -
> > +from settings import IXIA
> >  from exception import ConfigParseException, VirtConfigParseException
> >
> >  PORTCONF = "conf/ports.cfg"
> > @@ -185,6 +185,58 @@ class PortConf(UserConf):
> >              return False
> >
> >
> > +class CrbsConf(UserConf):
> > +    DEF_CRB = {'IP': '', 'name': 'CrownPassCRB1', 'user': '',
> > +               'pass': '', 'tester IP': '', 'tester pass': '',
> > +               IXIA: None, 'memory channels': 4,
> > +               'bypass core0': True}
> > +
> > +    def __init__(self, crbs_conf=CRBCONF):
> > +        self.config_file = crbs_conf
> > +        self.crbs_cfg = []
> > +        try:
> > +            self.crbs_conf = UserConf(self.config_file)
> > +        except ConfigParseException:
> > +            self.crbs_conf = None
> > +            raise ConfigParseException
> > +
> > +    def load_crbs_config(self):
> > +        sections = self.crbs_conf.get_sections()
> > +        if not sections:
> > +            return self.crbs_cfg
> > +
> > +        for name in sections:
> > +            crb = self.DEF_CRB.copy()
> > +            crb_confs = self.crbs_conf.load_section(name)
> > +            if not crb_confs:
> > +                continue
> > +
> > +            # covert file configuration to dts crbs
> > +            for conf in crb_confs:
> > +                key, value = conf
> > +                if key == 'dut_ip':
> > +                    crb['IP'] = value
> > +                elif key == 'dut_user':
> > +                    crb['user'] = value
> > +                elif key == 'dut_passwd':
> > +                    crb['pass'] = value
> > +                elif key == 'tester_ip':
> > +                    crb['tester IP'] = value
> > +                elif key == 'tester_passwd':
> > +                    crb['tester pass'] = value
> > +                elif key == 'ixia_group':
> > +                    crb[IXIA] = value
> > +                elif key == 'channels':
> > +                    crb['memory channels'] = int(value)
> > +                elif key == 'bypass_core0':
> > +                    if value == 'True':
> > +                        crb['bypass core0'] = True
> > +                    else:
> > +                        crb['bypass core0'] = False
> > +
> > +            self.crbs_cfg.append(crb)
> > +        return self.crbs_cfg
> > +
> >  if __name__ == '__main__':
> >      parser = argparse.ArgumentParser(
> >          description="Load DTS configuration files")
> > @@ -218,3 +270,7 @@ if __name__ == '__main__':
> >      virtconf = VirtConf(VIRTCONF)
> >      virtconf.load_virt_config('LIBVIRT')
> >      print virtconf.get_virt_config()
> > +
> > +    # example for crbs configuration file
> > +    crbsconf = CrbsConf(CRBCONF)
> > +    print crbsconf.load_crbs_config()
> > --
> > 1.9.3

Re: [dts] [PATCH 2/4] Add sample crbs configuration file

[Liu, Yong]

Ok, we will implement it later. This patch set will still focus on crbs configuration.

> -----Original Message-----
> From: Xu, HuilongX
> Sent: Friday, August 07, 2015 10:03 AM
> To: Liu, Yong; dts@dpdk.org
> Subject: RE: [dts] [PATCH 2/4] Add sample crbs configuration file
> 
> Hi yong,
> Could you move "ixia group" to config also.
> Thanks a lot
> 
> > -----Original Message-----
> > From: dts [mailto:dts-bounces@dpdk.org] On Behalf Of Yong Liu
> > Sent: Thursday, August 06, 2015 8:35 PM
> > To: dts@dpdk.org
> > Subject: [dts] [PATCH 2/4] Add sample crbs configuration file
> >
> > From: Marvin Liu <yong.liu@intel.com>
> >
> > This file is sample file for CRBS configuration. User need to configure
> > every
> > option in the file.
> >
> > Signed-off-by: Marvin Liu <yong.liu@intel.com>
> >
> > diff --git a/conf/crbs.cfg b/conf/crbs.cfg
> > new file mode 100644
> > index 0000000..de8e025
> > --- /dev/null
> > +++ b/conf/crbs.cfg
> > @@ -0,0 +1,28 @@
> > +#DUT crbs Configuration
> > +#[DUT IP]
> > +#  dut_ip: DUT ip address
> > +#  dut_user: Login DUT username
> > +#  dut_passwd: Login DUT password
> > +#  tester_ip: Tester ip address
> > +#  tester_passwd: Tester password
> > +#  ixia_group: IXIA group nmae
> > +#  channels: Board channel number
> > +#  bypass_core0: Whether by pass core0
> > +[DUT IP1]
> > +dut_ip=xxx.xxx.xxx.xxx
> > +dut_user=root
> > +dut_passwd=
> > +tester_ip=xxx.xxx.xxx.xxx
> > +tester_passwd=
> > +ixia_group=
> > +channels=4
> > +bypass_core0=True
> > +[DUT IP2]
> > +dut_ip=yyy.yyy.yyy.yyy
> > +dut_user=root
> > +dut_passwd=
> > +tester_ip=yyy.yyy.yyy.yyy
> > +tester_passwd=
> > +ixia_group=
> > +channels=4
> > +bypass_core0=True
> > --
> > 1.9.3