[dts] DTS Enhancement Plan

[dts] DTS Enhancement Plan

[Chen, Zhaoyan]

[-- Attachment #1: Type: text/plain, Size: 3301 bytes --]

Hi, DTS developers/users,

With DPDK development rapidly, more and more test suites were developed in DTS. (150+ test suites and 3k+ test cases in DTS). Meanwhile, paralleling execution tests for NICs/crypto/other device on same DUT become important for reducing execution time for covering all test cases. So we plan to enhance the DTS, for

*        Improve test suite quality

*        Unified test capability/function into framework

*        Paralleling execution

So we draft a DTS development plan, and share with you.

Looking forward to hear your thoughts, ideas and suggestions.

/Zhaoyan

DTS framework enhancement
=========================

Unified packet sending for functionality test
              * Implement all packet sending features in Packet Class
              * Remove others sending functions in DTS framework
              * Sending packet from Tester only

Generic EAL options generator for DPDK based application
              * Generate EAL options for DPDK app, and consider the assigned hardware resource for test, instead of whole DUT
              * Test suite can overwrite the options, in restricted range
              * Define hardware resource in crb.cfg, port.cfg and pktgen.cfg

Global resource management
              * Global driver load or unload, vfio-pci, igb_uio, uio_pci_generic
              * Define a parameter to decside if insmod/rmmod, if the module updated. otherwise, keep this module on host.
              * External_memory setup
              * Define DUT's cores mask in crb configuration
              * Define DUT's ports mask
              * Define physical ports in crb config

Clean up unused APIs, unused imported libraries, and unused APIs
              * plotting.py, plotgraph.py, numpy, matplotlib
              * e.g. self.tester.run_rfc2544, has been replaced by pktgen class
              * Replace private texttable.py with public version

All-in-one configuration file
              * Support configure all information in one configuration file for easy using
              * yaml file is better

Prepare to move to Python 3, make code be compatible with py2 and py3

Generate auto-script for all DTS dependencies
              * Update requirements.txt
              * Generate script for binary dependencies for compilation python packages

Release DTS container version


Test Suites cleanness
=================

Move to new packet sending API
              * Update all tests, which generate packets and send packets

Move to generic EAL generator API
              * Update all tests which use DPDK based application with new EAL string

Clean up "sleep"
              * Replace "sleep" with a stable checking mechanism

Replace "pktgen app" with "testpmd app" for testing
              * pktgen app needs be built for special DPDK version and too heavy for a simple function test, replace it in test case with testpmd app
Clean up absolute path
              * Replace "/tmp" with "/tmp/<dts_pid_random>/", or relative path "tmp".
              * Replace specific OS path in test case

Clean up test case dependencies, (files, libraries or tools)
              * Check all test suites, and add dependencies into auto-script or requirements.txt


[-- Attachment #2: Type: text/html, Size: 14655 bytes --]

Re: [dts] DTS Enhancement Plan

[Liu, Yong]

[-- Attachment #1: Type: text/plain, Size: 4132 bytes --]

Hi Zhaoyan,
This is great overall idea. It will be nice to see the roadmap of DTS enhancement work, since all works can't be done in one day:)

One comment for "All-in-one configuration file". It is nice at first glance, but configuration file may will be too huge for reading and modification after add things continuously.
And json format has been supported in DTS, is there any special reason for replacing it?

In my mind "Global resource management" will manage all the resources for test cases. We also need to add test binary into consideration, as it is also shared between suites.

Regards,
Marvin

From: dts [mailto:dts-bounces@dpdk.org] On Behalf Of Chen, Zhaoyan
Sent: Wednesday, July 24, 2019 10:52 AM
To: dts@dpdk.org
Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Subject: [dts] DTS Enhancement Plan

Hi, DTS developers/users,

With DPDK development rapidly, more and more test suites were developed in DTS. (150+ test suites and 3k+ test cases in DTS). Meanwhile, paralleling execution tests for NICs/crypto/other device on same DUT become important for reducing execution time for covering all test cases. So we plan to enhance the DTS, for

*        Improve test suite quality

*        Unified test capability/function into framework

*        Paralleling execution

So we draft a DTS development plan, and share with you.

Looking forward to hear your thoughts, ideas and suggestions.

/Zhaoyan

DTS framework enhancement
=========================

Unified packet sending for functionality test
              * Implement all packet sending features in Packet Class
              * Remove others sending functions in DTS framework
              * Sending packet from Tester only

Generic EAL options generator for DPDK based application
              * Generate EAL options for DPDK app, and consider the assigned hardware resource for test, instead of whole DUT
              * Test suite can overwrite the options, in restricted range
              * Define hardware resource in crb.cfg, port.cfg and pktgen.cfg

Global resource management
              * Global driver load or unload, vfio-pci, igb_uio, uio_pci_generic
              * Define a parameter to decside if insmod/rmmod, if the module updated. otherwise, keep this module on host.
              * External_memory setup
              * Define DUT's cores mask in crb configuration
              * Define DUT's ports mask
              * Define physical ports in crb config

Clean up unused APIs, unused imported libraries, and unused APIs
              * plotting.py, plotgraph.py, numpy, matplotlib
              * e.g. self.tester.run_rfc2544, has been replaced by pktgen class
              * Replace private texttable.py with public version

All-in-one configuration file
              * Support configure all information in one configuration file for easy using
              * yaml file is better

Prepare to move to Python 3, make code be compatible with py2 and py3

Generate auto-script for all DTS dependencies
              * Update requirements.txt
              * Generate script for binary dependencies for compilation python packages

Release DTS container version


Test Suites cleanness
=================

Move to new packet sending API
              * Update all tests, which generate packets and send packets

Move to generic EAL generator API
              * Update all tests which use DPDK based application with new EAL string

Clean up "sleep"
              * Replace "sleep" with a stable checking mechanism

Replace "pktgen app" with "testpmd app" for testing
              * pktgen app needs be built for special DPDK version and too heavy for a simple function test, replace it in test case with testpmd app

Clean up absolute path
              * Replace "/tmp" with "/tmp/<dts_pid_random>/", or relative path "tmp".
              * Replace specific OS path in test case

Clean up test case dependencies, (files, libraries or tools)
              * Check all test suites, and add dependencies into auto-script or requirements.txt


[-- Attachment #2: Type: text/html, Size: 14672 bytes --]

Re: [dts] DTS Enhancement Plan

[Chen, Zhaoyan]

[-- Attachment #1: Type: text/plain, Size: 4914 bytes --]

Yong, Thanks for your feedback.

All-in-one configuration file just a thought that we could combine crb.cfg/port.cfg/pktgen.cfg/ixia.cfg configuration files. Generally, users have to change these configurations for a specific test bed. For making easy use, let users are able to change those information at one place. YAML is similar to current cfg format. Basically, it will not replace current configurations, just add an option for users.

"Global resource management", yes we will consider all dependencies.


/Zhaoyan

From: Liu, Yong
Sent: Wednesday, July 24, 2019 11:17 AM
To: Chen, Zhaoyan <zhaoyan.chen@intel.com>; dts@dpdk.org
Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Subject: RE: DTS Enhancement Plan

Hi Zhaoyan,
This is great overall idea. It will be nice to see the roadmap of DTS enhancement work, since all works can't be done in one day:)

One comment for "All-in-one configuration file". It is nice at first glance, but configuration file may will be too huge for reading and modification after add things continuously.
And json format has been supported in DTS, is there any special reason for replacing it?

In my mind "Global resource management" will manage all the resources for test cases. We also need to add test binary into consideration, as it is also shared between suites.

Regards,
Marvin

From: dts [mailto:dts-bounces@dpdk.org] On Behalf Of Chen, Zhaoyan
Sent: Wednesday, July 24, 2019 10:52 AM
To: dts@dpdk.org<mailto:dts@dpdk.org>
Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>
Subject: [dts] DTS Enhancement Plan

Hi, DTS developers/users,

With DPDK development rapidly, more and more test suites were developed in DTS. (150+ test suites and 3k+ test cases in DTS). Meanwhile, paralleling execution tests for NICs/crypto/other device on same DUT become important for reducing execution time for covering all test cases. So we plan to enhance the DTS, for

*        Improve test suite quality

*        Unified test capability/function into framework

*        Paralleling execution

So we draft a DTS development plan, and share with you.

Looking forward to hear your thoughts, ideas and suggestions.

/Zhaoyan

DTS framework enhancement
=========================

Unified packet sending for functionality test
              * Implement all packet sending features in Packet Class
              * Remove others sending functions in DTS framework
              * Sending packet from Tester only

Generic EAL options generator for DPDK based application
              * Generate EAL options for DPDK app, and consider the assigned hardware resource for test, instead of whole DUT
              * Test suite can overwrite the options, in restricted range
              * Define hardware resource in crb.cfg, port.cfg and pktgen.cfg

Global resource management
              * Global driver load or unload, vfio-pci, igb_uio, uio_pci_generic
              * Define a parameter to decside if insmod/rmmod, if the module updated. otherwise, keep this module on host.
              * External_memory setup
              * Define DUT's cores mask in crb configuration
              * Define DUT's ports mask
              * Define physical ports in crb config

Clean up unused APIs, unused imported libraries, and unused APIs
              * plotting.py, plotgraph.py, numpy, matplotlib
              * e.g. self.tester.run_rfc2544, has been replaced by pktgen class
              * Replace private texttable.py with public version

All-in-one configuration file
              * Support configure all information in one configuration file for easy using
              * yaml file is better

Prepare to move to Python 3, make code be compatible with py2 and py3

Generate auto-script for all DTS dependencies
              * Update requirements.txt
              * Generate script for binary dependencies for compilation python packages

Release DTS container version


Test Suites cleanness
=================

Move to new packet sending API
              * Update all tests, which generate packets and send packets

Move to generic EAL generator API
              * Update all tests which use DPDK based application with new EAL string

Clean up "sleep"
              * Replace "sleep" with a stable checking mechanism

Replace "pktgen app" with "testpmd app" for testing
              * pktgen app needs be built for special DPDK version and too heavy for a simple function test, replace it in test case with testpmd app

Clean up absolute path
              * Replace "/tmp" with "/tmp/<dts_pid_random>/", or relative path "tmp".
              * Replace specific OS path in test case

Clean up test case dependencies, (files, libraries or tools)
              * Check all test suites, and add dependencies into auto-script or requirements.txt


[-- Attachment #2: Type: text/html, Size: 16381 bytes --]