DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Juraj Linkeš" <juraj.linkes@pantheon.tech>
To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com,
	lijuan.tu@intel.com, bruce.richardson@intel.com,
	probb@iol.unh.edu
Cc: dev@dpdk.org, "Juraj Linkeš" <juraj.linkes@pantheon.tech>
Subject: [PATCH v5 10/10] doc: update DTS setup and test suite cookbook
Date: Thu, 23 Feb 2023 16:28:40 +0100	[thread overview]
Message-ID: <20230223152840.634183-11-juraj.linkes@pantheon.tech> (raw)
In-Reply-To: <20230223152840.634183-1-juraj.linkes@pantheon.tech>

Document how to configure and run DTS.
Also add documentation related to new features: SUT setup and a brief
test suite implementation cookbook.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 doc/guides/tools/dts.rst | 165 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 163 insertions(+), 2 deletions(-)

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index daf54359ed..ebd6dceb6a 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -1,5 +1,5 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
-    Copyright(c) 2022 PANTHEON.tech s.r.o.
+    Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 
 DPDK Test Suite
 ===============
@@ -56,7 +56,7 @@ DTS runtime environment or just plain DTS environment are used interchangeably.
 
 
 Setting up DTS environment
---------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 #. **Python Version**
 
@@ -93,6 +93,167 @@ Setting up DTS environment
       poetry install
       poetry shell
 
+#. **SSH Connection**
+
+   DTS uses Python pexpect for SSH connections between DTS environment and the other hosts.
+   The pexpect implementation is a wrapper around the ssh command in the DTS environment.
+   This means it'll use the SSH agent providing the ssh command and its keys.
+
+
+Setting up System Under Test
+----------------------------
+
+There are two areas that need to be set up on a System Under Test:
+
+#. **DPDK dependencies**
+
+   DPDK will be built and run on the SUT.
+   Consult the Getting Started guides for the list of dependencies for each distribution.
+
+#. **Hardware dependencies**
+
+   Any hardware DPDK uses needs a proper driver
+   and most OS distributions provide those, but the version may not be satisfactory.
+   It's up to each user to install the driver they're interested in testing.
+   The hardware also may also need firmware upgrades, which is also left at user discretion.
+
+#. **Hugepages**
+
+   There are two ways to configure hugepages:
+
+   * DTS configuration
+
+     You may specify the optional hugepage configuration in the DTS config file.
+     If you do, DTS will take care of configuring hugepages,
+     overwriting your current SUT hugepage configuration.
+
+   * System under test configuration
+
+     It's possible to use the hugepage configuration already present on the SUT.
+     If you wish to do so, don't specify the hugepage configuration in the DTS config file.
+
+
+Running DTS
+-----------
+
+DTS needs to know which nodes to connect to and what hardware to use on those nodes.
+Once that's configured, DTS needs a DPDK tarball and it's ready to run.
+
+Configuring DTS
+~~~~~~~~~~~~~~~
+
+DTS configuration is split into nodes and executions and build targets within executions.
+By default, DTS will try to use the ``dts/conf.yaml`` config file,
+which is a template that illustrates what can be configured in DTS:
+
+  .. literalinclude:: ../../../dts/conf.yaml
+     :language: yaml
+     :start-at: executions:
+
+
+The user must be root or any other user with prompt starting with ``#``.
+The other fields are mostly self-explanatory
+and documented in more detail in ``dts/framework/config/conf_yaml_schema.json``.
+
+DTS Execution
+~~~~~~~~~~~~~
+
+DTS is run with ``main.py`` located in the ``dts`` directory after entering Poetry shell::
+
+   usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT]
+                  [-v VERBOSE] [-s SKIP_SETUP] [--tarball TARBALL]
+                  [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES]
+                  [--re-run RE_RUN]
+
+   Run DPDK test suites. All options may be specified with the environment variables provided in
+   brackets. Command line arguments have higher priority.
+
+   options:
+     -h, --help            show this help message and exit
+     --config-file CONFIG_FILE
+                           [DTS_CFG_FILE] configuration file that describes the test cases, SUTs
+                           and targets. (default: conf.yaml)
+     --output-dir OUTPUT_DIR, --output OUTPUT_DIR
+                           [DTS_OUTPUT_DIR] Output directory where dts logs and results are
+                           saved. (default: output)
+     -t TIMEOUT, --timeout TIMEOUT
+                           [DTS_TIMEOUT] The default timeout for all DTS operations except for
+                           compiling DPDK. (default: 15)
+     -v VERBOSE, --verbose VERBOSE
+                           [DTS_VERBOSE] Set to 'Y' to enable verbose output, logging all
+                           messages to the console. (default: N)
+     -s SKIP_SETUP, --skip-setup SKIP_SETUP
+                           [DTS_SKIP_SETUP] Set to 'Y' to skip all setup steps on SUT and TG
+                           nodes. (default: N)
+     --tarball TARBALL, --snapshot TARBALL
+                           [DTS_DPDK_TARBALL] Path to DPDK source code tarball which will be
+                           used in testing. (default: dpdk.tar.xz)
+     --compile-timeout COMPILE_TIMEOUT
+                           [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
+     --test-cases TEST_CASES
+                           [DTS_TESTCASES] Comma-separated list of test cases to execute.
+                           Unknown test cases will be silently ignored. (default: )
+     --re-run RE_RUN, --re_run RE_RUN
+                           [DTS_RERUN] Re-run each test case the specified amount of times if a
+                           test failure occurs (default: 0)
+
+
+The brackets contain the names of environment variables that set the same thing.
+The minimum DTS needs is a config file and a DPDK tarball.
+You may pass those to DTS using the command line arguments or use the default paths.
+
+
+DTS Results
+~~~~~~~~~~~
+
+Results are stored in the output dir by default
+which be changed with the ``--output-dir`` command line argument.
+The results contain basic statistics of passed/failed test cases and DPDK version.
+
+
+How To Write a Test Suite
+-------------------------
+
+All test suites inherit from ``TestSuite`` defined in ``dts/framework/test_suite.py``.
+There are four types of methods that comprise a test suite:
+
+#. **Test cases**
+
+   | Test cases are methods that start with a particular prefix.
+   | Functional test cases start with ``test_``, e.g. ``test_hello_world_single_core``.
+   | Performance test cases start with ``test_perf_``, e.g. ``test_perf_nic_single_core``.
+   | A test suite may have any number of functional and/or performance test cases.
+     However, these test cases must test the same feature,
+     following the rule of one feature = one test suite.
+     Test cases for one feature don't need to be grouped in just one test suite, though.
+     If the feature requires many testing scenarios to cover,
+     the test cases would be better off spread over multiple test suites
+     so that each test suite doesn't take too long to execute.
+
+#. **Setup and Teardown methods**
+
+   | There are setup and teardown methods for the whole test suite and each individual test case.
+   | Methods ``set_up_suite`` and ``tear_down_suite`` will be executed
+     before any and after all test cases have been executed, respectively.
+   | Methods ``set_up_test_case`` and ``tear_down_test_case`` will be executed
+     before and after each test case, respectively.
+   | These methods don't need to be implemented if there's no need for them in a test suite.
+     In that case, nothing will happen when they're is executed.
+
+#. **Test case verification**
+
+   Test case verification should be done with the ``verify`` method, which records the result.
+   The method should be called at the end of each test case.
+
+#. **Other methods**
+
+   Of course, all test suite code should adhere to coding standards.
+   Only the above methods will be treated specially and any other methods may be defined
+   (which should be mostly private methods needed by each particular test suite).
+   Any specific features (such as NIC configuration) required by a test suite
+   should be implemented in the ``SutNode`` class (and the underlying classes that ``SutNode`` uses)
+   and used by the test suite via the ``sut_node`` field.
+
 
 DTS Developer Tools
 -------------------
-- 
2.30.2


  parent reply	other threads:[~2023-02-23 15:30 UTC|newest]

Thread overview: 97+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-08-24 16:24 [RFC PATCH v1 00/10] dts: add hello world testcase Juraj Linkeš
2022-08-24 16:24 ` [RFC PATCH v1 01/10] dts: hello world config options Juraj Linkeš
2022-08-24 16:24 ` [RFC PATCH v1 02/10] dts: hello world cli parameters and env vars Juraj Linkeš
2022-08-24 16:24 ` [RFC PATCH v1 03/10] dts: ssh connection additions for hello world Juraj Linkeš
2022-08-24 16:24 ` [RFC PATCH v1 04/10] dts: add basic node management methods Juraj Linkeš
2022-08-24 16:24 ` [RFC PATCH v1 05/10] dts: add system under test node Juraj Linkeš
2022-08-24 16:24 ` [RFC PATCH v1 06/10] dts: add traffic generator node Juraj Linkeš
2022-08-24 16:24 ` [RFC PATCH v1 07/10] dts: add testcase and basic test results Juraj Linkeš
2022-08-24 16:24 ` [RFC PATCH v1 08/10] dts: add test runner and statistics collector Juraj Linkeš
2022-08-24 16:24 ` [RFC PATCH v1 09/10] dts: add hello world testplan Juraj Linkeš
2022-08-24 16:24 ` [RFC PATCH v1 10/10] dts: add hello world testsuite Juraj Linkeš
2022-11-14 16:54 ` [RFC PATCH v2 00/10] dts: add hello world testcase Juraj Linkeš
2022-11-14 16:54   ` [RFC PATCH v2 01/10] dts: add node and os abstractions Juraj Linkeš
2022-11-14 16:54   ` [RFC PATCH v2 02/10] dts: add ssh command verification Juraj Linkeš
2022-11-14 16:54   ` [RFC PATCH v2 03/10] dts: add dpdk build on sut Juraj Linkeš
2022-11-16 13:15     ` Owen Hilyard
     [not found]       ` <30ad4f7d087d4932845b6ca13934b1d2@pantheon.tech>
     [not found]         ` <CAHx6DYDOFMuEm4xc65OTrtUmGBtk8Z6UtSgS2grnR_RBY5HcjQ@mail.gmail.com>
2022-11-23 12:37           ` Juraj Linkeš
2022-11-14 16:54   ` [RFC PATCH v2 04/10] dts: add dpdk execution handling Juraj Linkeš
2022-11-16 13:28     ` Owen Hilyard
     [not found]       ` <df13ee41efb64e7bb37791f21ae5bac1@pantheon.tech>
     [not found]         ` <CAHx6DYCEYxZ0Osm6fKhp3Jx8n7s=r7qVh8R41c6nCan8Or-dpA@mail.gmail.com>
2022-11-23 13:03           ` Juraj Linkeš
2022-11-28 13:05             ` Owen Hilyard
2022-11-14 16:54   ` [RFC PATCH v2 05/10] dts: add node memory setup Juraj Linkeš
2022-11-16 13:47     ` Owen Hilyard
2022-11-23 13:58       ` Juraj Linkeš
2022-11-14 16:54   ` [RFC PATCH v2 06/10] dts: add test results module Juraj Linkeš
2022-11-14 16:54   ` [RFC PATCH v2 07/10] dts: add simple stats report Juraj Linkeš
2022-11-16 13:57     ` Owen Hilyard
2022-11-14 16:54   ` [RFC PATCH v2 08/10] dts: add testsuite class Juraj Linkeš
2022-11-16 15:15     ` Owen Hilyard
2022-11-14 16:54   ` [RFC PATCH v2 09/10] dts: add hello world testplan Juraj Linkeš
2022-11-14 16:54   ` [RFC PATCH v2 10/10] dts: add hello world testsuite Juraj Linkeš
2023-01-17 15:48   ` [PATCH v3 00/10] dts: add hello world testcase Juraj Linkeš
2023-01-17 15:48     ` [PATCH v3 01/10] dts: add node and os abstractions Juraj Linkeš
2023-01-17 15:48     ` [PATCH v3 02/10] dts: add ssh command verification Juraj Linkeš
2023-01-17 15:48     ` [PATCH v3 03/10] dts: add dpdk build on sut Juraj Linkeš
2023-01-17 15:49     ` [PATCH v3 04/10] dts: add dpdk execution handling Juraj Linkeš
2023-01-17 15:49     ` [PATCH v3 05/10] dts: add node memory setup Juraj Linkeš
2023-01-17 15:49     ` [PATCH v3 06/10] dts: add test suite module Juraj Linkeš
2023-01-17 15:49     ` [PATCH v3 07/10] dts: add hello world testplan Juraj Linkeš
2023-01-17 15:49     ` [PATCH v3 08/10] dts: add hello world testsuite Juraj Linkeš
2023-01-17 15:49     ` [PATCH v3 09/10] dts: add test suite config and runner Juraj Linkeš
2023-01-17 15:49     ` [PATCH v3 10/10] dts: add test results module Juraj Linkeš
2023-01-19 16:16     ` [PATCH v3 00/10] dts: add hello world testcase Owen Hilyard
2023-02-09 16:47       ` Patrick Robb
2023-02-13 15:28     ` [PATCH v4 " Juraj Linkeš
2023-02-13 15:28       ` [PATCH v4 01/10] dts: add node and os abstractions Juraj Linkeš
2023-02-17 17:44         ` Bruce Richardson
2023-02-20 13:24           ` Juraj Linkeš
2023-02-13 15:28       ` [PATCH v4 02/10] dts: add ssh command verification Juraj Linkeš
2023-02-13 15:28       ` [PATCH v4 03/10] dts: add dpdk build on sut Juraj Linkeš
2023-02-22 16:44         ` Bruce Richardson
2023-02-13 15:28       ` [PATCH v4 04/10] dts: add dpdk execution handling Juraj Linkeš
2023-02-13 15:28       ` [PATCH v4 05/10] dts: add node memory setup Juraj Linkeš
2023-02-13 15:28       ` [PATCH v4 06/10] dts: add test suite module Juraj Linkeš
2023-02-13 15:28       ` [PATCH v4 07/10] dts: add hello world testsuite Juraj Linkeš
2023-02-13 15:28       ` [PATCH v4 08/10] dts: add test suite config and runner Juraj Linkeš
2023-02-13 15:28       ` [PATCH v4 09/10] dts: add test results module Juraj Linkeš
2023-02-13 15:28       ` [PATCH v4 10/10] doc: update DTS setup and test suite cookbook Juraj Linkeš
2023-02-17 17:26       ` [PATCH v4 00/10] dts: add hello world testcase Bruce Richardson
2023-02-20 10:13         ` Juraj Linkeš
2023-02-20 11:56           ` Bruce Richardson
2023-02-22 16:39           ` Bruce Richardson
2023-02-23  8:27             ` Juraj Linkeš
2023-02-23  9:17               ` Bruce Richardson
2023-02-23 15:28       ` [PATCH v5 " Juraj Linkeš
2023-02-23 15:28         ` [PATCH v5 01/10] dts: add node and os abstractions Juraj Linkeš
2023-02-23 15:28         ` [PATCH v5 02/10] dts: add ssh command verification Juraj Linkeš
2023-02-23 15:28         ` [PATCH v5 03/10] dts: add dpdk build on sut Juraj Linkeš
2023-02-23 15:28         ` [PATCH v5 04/10] dts: add dpdk execution handling Juraj Linkeš
2023-02-23 15:28         ` [PATCH v5 05/10] dts: add node memory setup Juraj Linkeš
2023-02-23 15:28         ` [PATCH v5 06/10] dts: add test suite module Juraj Linkeš
2023-02-23 15:28         ` [PATCH v5 07/10] dts: add hello world testsuite Juraj Linkeš
2023-02-23 15:28         ` [PATCH v5 08/10] dts: add test suite config and runner Juraj Linkeš
2023-02-23 15:28         ` [PATCH v5 09/10] dts: add test results module Juraj Linkeš
2023-02-23 15:28         ` Juraj Linkeš [this message]
2023-03-03  8:31           ` [PATCH v5 10/10] doc: update DTS setup and test suite cookbook Huang, ChenyuX
2023-02-23 16:13         ` [PATCH v5 00/10] dts: add hello world testcase Bruce Richardson
2023-02-26 19:11         ` Wathsala Wathawana Vithanage
2023-02-27  8:28           ` Juraj Linkeš
2023-02-28 15:27             ` Wathsala Wathawana Vithanage
2023-03-01  8:35               ` Juraj Linkeš
2023-03-03 10:24         ` [PATCH v6 00/10] dts: add hello world test case Juraj Linkeš
2023-03-03 10:24           ` [PATCH v6 01/10] dts: add node and os abstractions Juraj Linkeš
2023-03-03 10:24           ` [PATCH v6 02/10] dts: add ssh command verification Juraj Linkeš
2023-03-03 10:25           ` [PATCH v6 03/10] dts: add dpdk build on sut Juraj Linkeš
2023-03-20  8:30             ` David Marchand
2023-03-20 13:12               ` Juraj Linkeš
2023-03-20 13:22                 ` David Marchand
2023-03-03 10:25           ` [PATCH v6 04/10] dts: add dpdk execution handling Juraj Linkeš
2023-03-03 10:25           ` [PATCH v6 05/10] dts: add node memory setup Juraj Linkeš
2023-03-03 10:25           ` [PATCH v6 06/10] dts: add test suite module Juraj Linkeš
2023-03-03 10:25           ` [PATCH v6 07/10] dts: add hello world testsuite Juraj Linkeš
2023-03-03 10:25           ` [PATCH v6 08/10] dts: add test suite config and runner Juraj Linkeš
2023-03-03 10:25           ` [PATCH v6 09/10] dts: add test results module Juraj Linkeš
2023-03-03 10:25           ` [PATCH v6 10/10] doc: update dts setup and test suite cookbook Juraj Linkeš
2023-03-09 21:47             ` Patrick Robb
2023-03-19 15:26           ` [PATCH v6 00/10] dts: add hello world test case Thomas Monjalon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20230223152840.634183-11-juraj.linkes@pantheon.tech \
    --to=juraj.linkes@pantheon.tech \
    --cc=Honnappa.Nagarahalli@arm.com \
    --cc=bruce.richardson@intel.com \
    --cc=dev@dpdk.org \
    --cc=lijuan.tu@intel.com \
    --cc=probb@iol.unh.edu \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).