From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 64F4041E23; Thu, 9 Mar 2023 22:47:25 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 436EC40ED7; Thu, 9 Mar 2023 22:47:25 +0100 (CET) Received: from mail-oa1-f45.google.com (mail-oa1-f45.google.com [209.85.160.45]) by mails.dpdk.org (Postfix) with ESMTP id 3F2AE40695 for ; Thu, 9 Mar 2023 22:47:24 +0100 (CET) Received: by mail-oa1-f45.google.com with SMTP id 586e51a60fabf-1763e201bb4so3857873fac.1 for ; Thu, 09 Mar 2023 13:47:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1678398443; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=CTgqoXe84ApNJ0gGizVGrOE3wcIImBi1gWQz11IC3yM=; b=WBmmQvlnh6QNpUlxZ5vsm2RWhYWhK8zX1p78g6dCDi7vYUeEx7K1XlfV4Muk9biYC9 Hw+T8MXWdFVksKNbf6aniLP4dLtMLU7RFTNDDaUFNzdoiLvQiO0FJdbQ4/YKLi9vcFi4 Byyy4vLGZGyVHbX39kz/dWeiXAw+G1hmL/R6A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678398443; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=CTgqoXe84ApNJ0gGizVGrOE3wcIImBi1gWQz11IC3yM=; b=yL6fEwmmilA7UAOlZbnu5Nx0k5NqhxH6HZ6PkWvvWh85yuQSKykxf01mVZyEOaDlI1 gdkKpE8L2nztyzbDncW0qVnE+flt7L2baqzSZw7iMcUoGsxVj+ufaCLfpR1TWcL5GGLD cJ7CTpOn27cUsoIzy91P5SWO7hL9TAyQCsMamQtx+o2KJV5DJ6YrNqqVnrdSiSC0uusU qmThjEdcmyNMhwgKl7gLPmgYsY22Cp8zwMCl75Mhu0dkiJ+hwpubhbiDb4oWCNSj4bXz ApNzjU/tEnElFMB3VjgAAJeWfSz7L3obF1pqc7mMy5GZ/3AMO/v6I3dmY0iwoUA0up+F 1izg== X-Gm-Message-State: AO0yUKVZUZJ+ib5ntSgjzqh6Ahs4FO1r8/FST8cYqx2yA5luF3KxXBW9 1aXQJylHu3cETL6n+qNRr9Mv0Mxz2JCSaR7jvMi77g== X-Google-Smtp-Source: AK7set9lq/s15Euuv9fN+tKsIylHs+GPQj9oG9SfEvRoqLzohrBOtyaQODhNcEiTdmO4BUfFeuCY4W21hnJqnkbL8fE= X-Received: by 2002:a05:6870:d192:b0:16d:f67f:8258 with SMTP id a18-20020a056870d19200b0016df67f8258mr123701oac.4.1678398443510; Thu, 09 Mar 2023 13:47:23 -0800 (PST) MIME-Version: 1.0 References: <20230223152840.634183-1-juraj.linkes@pantheon.tech> <20230303102507.527790-1-juraj.linkes@pantheon.tech> <20230303102507.527790-11-juraj.linkes@pantheon.tech> In-Reply-To: <20230303102507.527790-11-juraj.linkes@pantheon.tech> From: Patrick Robb Date: Thu, 9 Mar 2023 16:47:12 -0500 Message-ID: Subject: Re: [PATCH v6 10/10] doc: update dts setup and test suite cookbook To: =?UTF-8?Q?Juraj_Linke=C5=A1?= Cc: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, lijuan.tu@intel.com, bruce.richardson@intel.com, dev@dpdk.org Content-Type: multipart/alternative; boundary="00000000000004752705f67e9714" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --00000000000004752705f67e9714 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Tested-by: Patrick Robb On Fri, Mar 3, 2023 at 5:25=E2=80=AFAM Juraj Linke=C5=A1 wrote: > Document how to configure and run DTS. > Also add documentation related to new features: SUT setup and a brief > test suite implementation cookbook. > > Signed-off-by: Juraj Linke=C5=A1 > --- > doc/guides/tools/dts.rst | 165 ++++++++++++++++++++++++++++++++++++++- > 1 file changed, 163 insertions(+), 2 deletions(-) > > diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst > index daf54359ed..ebd6dceb6a 100644 > --- a/doc/guides/tools/dts.rst > +++ b/doc/guides/tools/dts.rst > @@ -1,5 +1,5 @@ > .. SPDX-License-Identifier: BSD-3-Clause > - Copyright(c) 2022 PANTHEON.tech s.r.o. > + Copyright(c) 2022-2023 PANTHEON.tech s.r.o. > > DPDK Test Suite > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > @@ -56,7 +56,7 @@ DTS runtime environment or just plain DTS environment > are used interchangeably. > > > Setting up DTS environment > --------------------------- > +~~~~~~~~~~~~~~~~~~~~~~~~~~ > > #. **Python Version** > > @@ -93,6 +93,167 @@ Setting up DTS environment > poetry install > poetry shell > > +#. **SSH Connection** > + > + DTS uses Python pexpect for SSH connections between DTS environment > and the other hosts. > + The pexpect implementation is a wrapper around the ssh command in the > DTS environment. > + This means it'll use the SSH agent providing the ssh command and its > keys. > + > + > +Setting up System Under Test > +---------------------------- > + > +There are two areas that need to be set up on a System Under Test: > + > +#. **DPDK dependencies** > + > + DPDK will be built and run on the SUT. > + Consult the Getting Started guides for the list of dependencies for > each distribution. > + > +#. **Hardware dependencies** > + > + Any hardware DPDK uses needs a proper driver > + and most OS distributions provide those, but the version may not be > satisfactory. > + It's up to each user to install the driver they're interested in > testing. > + The hardware also may also need firmware upgrades, which is also left > at user discretion. > + > +#. **Hugepages** > + > + There are two ways to configure hugepages: > + > + * DTS configuration > + > + You may specify the optional hugepage configuration in the DTS > config file. > + If you do, DTS will take care of configuring hugepages, > + overwriting your current SUT hugepage configuration. > + > + * System under test configuration > + > + It's possible to use the hugepage configuration already present on > the SUT. > + If you wish to do so, don't specify the hugepage configuration in > the DTS config file. > + > + > +Running DTS > +----------- > + > +DTS needs to know which nodes to connect to and what hardware to use on > those nodes. > +Once that's configured, DTS needs a DPDK tarball and it's ready to run. > + > +Configuring DTS > +~~~~~~~~~~~~~~~ > + > +DTS configuration is split into nodes and executions and build targets > within executions. > +By default, DTS will try to use the ``dts/conf.yaml`` config file, > +which is a template that illustrates what can be configured in DTS: > + > + .. literalinclude:: ../../../dts/conf.yaml > + :language: yaml > + :start-at: executions: > + > + > +The user must be root or any other user with prompt starting with ``#``. > +The other fields are mostly self-explanatory > +and documented in more detail in > ``dts/framework/config/conf_yaml_schema.json``. > + > +DTS Execution > +~~~~~~~~~~~~~ > + > +DTS is run with ``main.py`` located in the ``dts`` directory after > entering Poetry shell:: > + > + usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir > OUTPUT_DIR] [-t TIMEOUT] > + [-v VERBOSE] [-s SKIP_SETUP] [--tarball TARBALL] > + [--compile-timeout COMPILE_TIMEOUT] [--test-cases > TEST_CASES] > + [--re-run RE_RUN] > + > + Run DPDK test suites. All options may be specified with the > environment variables provided in > + brackets. Command line arguments have higher priority. > + > + options: > + -h, --help show this help message and exit > + --config-file CONFIG_FILE > + [DTS_CFG_FILE] configuration file that > describes the test cases, SUTs > + and targets. (default: conf.yaml) > + --output-dir OUTPUT_DIR, --output OUTPUT_DIR > + [DTS_OUTPUT_DIR] Output directory where dts > logs and results are > + saved. (default: output) > + -t TIMEOUT, --timeout TIMEOUT > + [DTS_TIMEOUT] The default timeout for all DTS > operations except for > + compiling DPDK. (default: 15) > + -v VERBOSE, --verbose VERBOSE > + [DTS_VERBOSE] Set to 'Y' to enable verbose > output, logging all > + messages to the console. (default: N) > + -s SKIP_SETUP, --skip-setup SKIP_SETUP > + [DTS_SKIP_SETUP] Set to 'Y' to skip all setup > steps on SUT and TG > + nodes. (default: N) > + --tarball TARBALL, --snapshot TARBALL > + [DTS_DPDK_TARBALL] Path to DPDK source code > tarball which will be > + used in testing. (default: dpdk.tar.xz) > + --compile-timeout COMPILE_TIMEOUT > + [DTS_COMPILE_TIMEOUT] The timeout for > compiling DPDK. (default: 1200) > + --test-cases TEST_CASES > + [DTS_TESTCASES] Comma-separated list of test > cases to execute. > + Unknown test cases will be silently ignored. > (default: ) > + --re-run RE_RUN, --re_run RE_RUN > + [DTS_RERUN] Re-run each test case the > specified amount of times if a > + test failure occurs (default: 0) > + > + > +The brackets contain the names of environment variables that set the sam= e > thing. > +The minimum DTS needs is a config file and a DPDK tarball. > +You may pass those to DTS using the command line arguments or use the > default paths. > + > + > +DTS Results > +~~~~~~~~~~~ > + > +Results are stored in the output dir by default > +which be changed with the ``--output-dir`` command line argument. > +The results contain basic statistics of passed/failed test cases and DPD= K > version. > + > + > +How To Write a Test Suite > +------------------------- > + > +All test suites inherit from ``TestSuite`` defined in > ``dts/framework/test_suite.py``. > +There are four types of methods that comprise a test suite: > + > +#. **Test cases** > + > + | Test cases are methods that start with a particular prefix. > + | Functional test cases start with ``test_``, e.g. > ``test_hello_world_single_core``. > + | Performance test cases start with ``test_perf_``, e.g. > ``test_perf_nic_single_core``. > + | A test suite may have any number of functional and/or performance > test cases. > + However, these test cases must test the same feature, > + following the rule of one feature =3D one test suite. > + Test cases for one feature don't need to be grouped in just one tes= t > suite, though. > + If the feature requires many testing scenarios to cover, > + the test cases would be better off spread over multiple test suites > + so that each test suite doesn't take too long to execute. > + > +#. **Setup and Teardown methods** > + > + | There are setup and teardown methods for the whole test suite and > each individual test case. > + | Methods ``set_up_suite`` and ``tear_down_suite`` will be executed > + before any and after all test cases have been executed, respectivel= y. > + | Methods ``set_up_test_case`` and ``tear_down_test_case`` will be > executed > + before and after each test case, respectively. > + | These methods don't need to be implemented if there's no need for > them in a test suite. > + In that case, nothing will happen when they're is executed. > + > +#. **Test case verification** > + > + Test case verification should be done with the ``verify`` method, > which records the result. > + The method should be called at the end of each test case. > + > +#. **Other methods** > + > + Of course, all test suite code should adhere to coding standards. > + Only the above methods will be treated specially and any other method= s > may be defined > + (which should be mostly private methods needed by each particular tes= t > suite). > + Any specific features (such as NIC configuration) required by a test > suite > + should be implemented in the ``SutNode`` class (and the underlying > classes that ``SutNode`` uses) > + and used by the test suite via the ``sut_node`` field. > + > > DTS Developer Tools > ------------------- > -- > 2.30.2 > > --=20 Patrick Robb Technical Service Manager UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 www.iol.unh.edu --00000000000004752705f67e9714 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Tested-by: Patrick Robb <probb@iol.unh.edu>

On Fri, Mar 3, 2023 at 5:25=E2=80=AFAM Jura= j Linke=C5=A1 <juraj.linkes@pantheon.tech> wrote:
Document how to configure and run D= TS.
Also add documentation related to new features: SUT setup and a brief
test suite implementation cookbook.

Signed-off-by: Juraj Linke=C5=A1 <juraj.linkes@pantheon.tech>
---
=C2=A0doc/guides/tools/dts.rst | 165 ++++++++++++++++++++++++++++++++++++++= -
=C2=A01 file changed, 163 insertions(+), 2 deletions(-)

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index daf54359ed..ebd6dceb6a 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -1,5 +1,5 @@
=C2=A0..=C2=A0 SPDX-License-Identifier: BSD-3-Clause
-=C2=A0 =C2=A0 Copyright(c) 2022 PANTHEON.tech s.r.o.
+=C2=A0 =C2=A0 Copyright(c) 2022-2023 PANTHEON.tech s.r.o.

=C2=A0DPDK Test Suite
=C2=A0=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
@@ -56,7 +56,7 @@ DTS runtime environment or just plain DTS environment are= used interchangeably.


=C2=A0Setting up DTS environment
---------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~

=C2=A0#. **Python Version**

@@ -93,6 +93,167 @@ Setting up DTS environment
=C2=A0 =C2=A0 =C2=A0 =C2=A0poetry install
=C2=A0 =C2=A0 =C2=A0 =C2=A0poetry shell

+#. **SSH Connection**
+
+=C2=A0 =C2=A0DTS uses Python pexpect for SSH connections between DTS envir= onment and the other hosts.
+=C2=A0 =C2=A0The pexpect implementation is a wrapper around the ssh comman= d in the DTS environment.
+=C2=A0 =C2=A0This means it'll use the SSH agent providing the ssh comm= and and its keys.
+
+
+Setting up System Under Test
+----------------------------
+
+There are two areas that need to be set up on a System Under Test:
+
+#. **DPDK dependencies**
+
+=C2=A0 =C2=A0DPDK will be built and run on the SUT.
+=C2=A0 =C2=A0Consult the Getting Started guides for the list of dependenci= es for each distribution.
+
+#. **Hardware dependencies**
+
+=C2=A0 =C2=A0Any hardware DPDK uses needs a proper driver
+=C2=A0 =C2=A0and most OS distributions provide those, but the version may = not be satisfactory.
+=C2=A0 =C2=A0It's up to each user to install the driver they're in= terested in testing.
+=C2=A0 =C2=A0The hardware also may also need firmware upgrades, which is a= lso left at user discretion.
+
+#. **Hugepages**
+
+=C2=A0 =C2=A0There are two ways to configure hugepages:
+
+=C2=A0 =C2=A0* DTS configuration
+
+=C2=A0 =C2=A0 =C2=A0You may specify the optional hugepage configuration in= the DTS config file.
+=C2=A0 =C2=A0 =C2=A0If you do, DTS will take care of configuring hugepages= ,
+=C2=A0 =C2=A0 =C2=A0overwriting your current SUT hugepage configuration. +
+=C2=A0 =C2=A0* System under test configuration
+
+=C2=A0 =C2=A0 =C2=A0It's possible to use the hugepage configuration al= ready present on the SUT.
+=C2=A0 =C2=A0 =C2=A0If you wish to do so, don't specify the hugepage c= onfiguration in the DTS config file.
+
+
+Running DTS
+-----------
+
+DTS needs to know which nodes to connect to and what hardware to use on th= ose nodes.
+Once that's configured, DTS needs a DPDK tarball and it's ready to= run.
+
+Configuring DTS
+~~~~~~~~~~~~~~~
+
+DTS configuration is split into nodes and executions and build targets wit= hin executions.
+By default, DTS will try to use the ``dts/conf.yaml`` config file,
+which is a template that illustrates what can be configured in DTS:
+
+=C2=A0 .. literalinclude:: ../../../dts/conf.yaml
+=C2=A0 =C2=A0 =C2=A0:language: yaml
+=C2=A0 =C2=A0 =C2=A0:start-at: executions:
+
+
+The user must be root or any other user with prompt starting with ``#``. +The other fields are mostly self-explanatory
+and documented in more detail in ``dts/framework/config/conf_yaml_schema.j= son``.
+
+DTS Execution
+~~~~~~~~~~~~~
+
+DTS is run with ``main.py`` located in the ``dts`` directory after enterin= g Poetry shell::
+
+=C2=A0 =C2=A0usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir= OUTPUT_DIR] [-t TIMEOUT]
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 [-v VERBOSE= ] [-s SKIP_SETUP] [--tarball TARBALL]
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 [--compile-= timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES]
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 [--re-run R= E_RUN]
+
+=C2=A0 =C2=A0Run DPDK test suites. All options may be specified with the e= nvironment variables provided in
+=C2=A0 =C2=A0brackets. Command line arguments have higher priority.
+
+=C2=A0 =C2=A0options:
+=C2=A0 =C2=A0 =C2=A0-h, --help=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sh= ow this help message and exit
+=C2=A0 =C2=A0 =C2=A0--config-file CONFIG_FILE
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0[DTS_CFG_FILE] configuration file that describes th= e test cases, SUTs
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0and targets. (default: conf.yaml)
+=C2=A0 =C2=A0 =C2=A0--output-dir OUTPUT_DIR, --output OUTPUT_DIR
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0[DTS_OUTPUT_DIR] Output directory where dts logs an= d results are
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0saved. (default: output)
+=C2=A0 =C2=A0 =C2=A0-t TIMEOUT, --timeout TIMEOUT
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0[DTS_TIMEOUT] The default timeout for all DTS opera= tions except for
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0compiling DPDK. (default: 15)
+=C2=A0 =C2=A0 =C2=A0-v VERBOSE, --verbose VERBOSE
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0[DTS_VERBOSE] Set to 'Y' to enable verbose = output, logging all
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0messages to the console. (default: N)
+=C2=A0 =C2=A0 =C2=A0-s SKIP_SETUP, --skip-setup SKIP_SETUP
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0[DTS_SKIP_SETUP] Set to 'Y' to skip all set= up steps on SUT and TG
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0nodes. (default: N)
+=C2=A0 =C2=A0 =C2=A0--tarball TARBALL, --snapshot TARBALL
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0[DTS_DPDK_TARBALL] Path to DPDK source code tarball= which will be
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0used in testing. (default: dpdk.tar.xz)
+=C2=A0 =C2=A0 =C2=A0--compile-timeout COMPILE_TIMEOUT
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0[DTS_COMPILE_TIMEOUT] The timeout for compiling DPD= K. (default: 1200)
+=C2=A0 =C2=A0 =C2=A0--test-cases TEST_CASES
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0[DTS_TESTCASES] Comma-separated list of test cases = to execute.
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0Unknown test cases will be silently ignored. (defau= lt: )
+=C2=A0 =C2=A0 =C2=A0--re-run RE_RUN, --re_run RE_RUN
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0[DTS_RERUN] Re-run each test case the specified amo= unt of times if a
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0test failure occurs (default: 0)
+
+
+The brackets contain the names of environment variables that set the same = thing.
+The minimum DTS needs is a config file and a DPDK tarball.
+You may pass those to DTS using the command line arguments or use the defa= ult paths.
+
+
+DTS Results
+~~~~~~~~~~~
+
+Results are stored in the output dir by default
+which be changed with the ``--output-dir`` command line argument.
+The results contain basic statistics of passed/failed test cases and DPDK = version.
+
+
+How To Write a Test Suite
+-------------------------
+
+All test suites inherit from ``TestSuite`` defined in ``dts/framework/test= _suite.py``.
+There are four types of methods that comprise a test suite:
+
+#. **Test cases**
+
+=C2=A0 =C2=A0| Test cases are methods that start with a particular prefix.=
+=C2=A0 =C2=A0| Functional test cases start with ``test_``, e.g. ``test_hel= lo_world_single_core``.
+=C2=A0 =C2=A0| Performance test cases start with ``test_perf_``, e.g. ``te= st_perf_nic_single_core``.
+=C2=A0 =C2=A0| A test suite may have any number of functional and/or perfo= rmance test cases.
+=C2=A0 =C2=A0 =C2=A0However, these test cases must test the same feature,<= br> +=C2=A0 =C2=A0 =C2=A0following the rule of one feature =3D one test suite.<= br> +=C2=A0 =C2=A0 =C2=A0Test cases for one feature don't need to be groupe= d in just one test suite, though.
+=C2=A0 =C2=A0 =C2=A0If the feature requires many testing scenarios to cove= r,
+=C2=A0 =C2=A0 =C2=A0the test cases would be better off spread over multipl= e test suites
+=C2=A0 =C2=A0 =C2=A0so that each test suite doesn't take too long to e= xecute.
+
+#. **Setup and Teardown methods**
+
+=C2=A0 =C2=A0| There are setup and teardown methods for the whole test sui= te and each individual test case.
+=C2=A0 =C2=A0| Methods ``set_up_suite`` and ``tear_down_suite`` will be ex= ecuted
+=C2=A0 =C2=A0 =C2=A0before any and after all test cases have been executed= , respectively.
+=C2=A0 =C2=A0| Methods ``set_up_test_case`` and ``tear_down_test_case`` wi= ll be executed
+=C2=A0 =C2=A0 =C2=A0before and after each test case, respectively.
+=C2=A0 =C2=A0| These methods don't need to be implemented if there'= ;s no need for them in a test suite.
+=C2=A0 =C2=A0 =C2=A0In that case, nothing will happen when they're is = executed.
+
+#. **Test case verification**
+
+=C2=A0 =C2=A0Test case verification should be done with the ``verify`` met= hod, which records the result.
+=C2=A0 =C2=A0The method should be called at the end of each test case.
+
+#. **Other methods**
+
+=C2=A0 =C2=A0Of course, all test suite code should adhere to coding standa= rds.
+=C2=A0 =C2=A0Only the above methods will be treated specially and any othe= r methods may be defined
+=C2=A0 =C2=A0(which should be mostly private methods needed by each partic= ular test suite).
+=C2=A0 =C2=A0Any specific features (such as NIC configuration) required by= a test suite
+=C2=A0 =C2=A0should be implemented in the ``SutNode`` class (and the under= lying classes that ``SutNode`` uses)
+=C2=A0 =C2=A0and used by the test suite via the ``sut_node`` field.
+

=C2=A0DTS Developer Tools
=C2=A0-------------------
--
2.30.2



--

Patrick Robb

<= span style=3D"font-size:10pt;font-family:Arial;color:rgb(0,0,0);background-= color:transparent;vertical-align:baseline;white-space:pre-wrap">Technical S= ervice Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu


--00000000000004752705f67e9714--