From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6C6EC41DC6; Fri, 3 Mar 2023 11:26:27 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 155F042D50; Fri, 3 Mar 2023 11:25:26 +0100 (CET) Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com [209.85.208.44]) by mails.dpdk.org (Postfix) with ESMTP id D3B0642D36 for ; Fri, 3 Mar 2023 11:25:20 +0100 (CET) Received: by mail-ed1-f44.google.com with SMTP id o12so8230125edb.9 for ; Fri, 03 Mar 2023 02:25:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon-tech.20210112.gappssmtp.com; s=20210112; t=1677839120; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TtKGqXREbdV24BtUnuvxKsFeNtDkF+SAMUN6axw569Q=; b=aqSjiCq5fsikpmEIfpAcKrOGCg7NnN8QzBCC4Xp0E3O9b5ezWgDX3kRuMqOTf9tMPC Te1jC0vpp9ritvApoKJKwctoEDCM00wsmzpZed4RbejG0HERQ5s7wo6AsgcXZcBhCclH MJ1wlXGGzNYPAhK80M8+hdOK9N6DMqyXiQPGnonmpwMFdklV4lHDKUAQ3YJc0UCUBrtD jx4RAoBunJl7F8rqGAVA+eCq4J+couoztvFU3vGWNZp/w3W0fhvChosnTsyLG1BQ5+D8 f5wK2VINAk+JAH70O2umWrtMSZqF1OO2iZp4c8HlTwJ9oZEfcUtzFZCYhZM20CdC1Lew AY9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677839120; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TtKGqXREbdV24BtUnuvxKsFeNtDkF+SAMUN6axw569Q=; b=Ow6JQO/GEetNvAvSqyPvIwWdiGZqBsVOFHxPNN3S8b8OH+1HftILVJgJpJe/+BoKz7 61M3yBXUp6ER23RAgWyKTkZx5hc56xgiri9bA/0uuk9kpCo8xzyPyiMo69lEuGJVukN0 7M6D4w4Td8ZE1QklT+zVgE5fZ5AdFN1uYMCocYVNXTyeXZwC0fUOBxD2R7s9gs0bkwtG qnUNON+I8MMPXq4TqQ1/P7pW9l5PCVVKNshxIxV0QlWuWUYMOieogG25uoVBh24xtchR N4Ur5aDW0B7Yn0A4rKdOlZ8Bz3NIk8aLndAltz06Vo8QVxwvyBKwlOk3jAHkQ1MoZGCY jHXQ== X-Gm-Message-State: AO0yUKXszdK3YIZqtJScS8IWCuPztnjk51q3P4OrYcXuXELpMcfpHQZe rMecUbnaFtyVIqj/O+RFtQxEgQ== X-Google-Smtp-Source: AK7set/uny2SdpEdqO8Mylsv1lNiPW9lDf6Ff+RfHsjaNrJzV72jqlPuKV45Bc4YO2BBuky5rAR4Bg== X-Received: by 2002:aa7:c40b:0:b0:4c1:f57c:4ee7 with SMTP id j11-20020aa7c40b000000b004c1f57c4ee7mr1455864edq.21.1677839120606; Fri, 03 Mar 2023 02:25:20 -0800 (PST) Received: from localhost.localdomain (ip-46.34.234.35.o2inet.sk. [46.34.234.35]) by smtp.gmail.com with ESMTPSA id j19-20020a508a93000000b004c3e3a6136dsm984028edj.21.2023.03.03.02.25.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Mar 2023 02:25:20 -0800 (PST) From: =?UTF-8?q?Juraj=20Linke=C5=A1?= To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, lijuan.tu@intel.com, bruce.richardson@intel.com, probb@iol.unh.edu Cc: dev@dpdk.org, =?UTF-8?q?Juraj=20Linke=C5=A1?= Subject: [PATCH v6 10/10] doc: update dts setup and test suite cookbook Date: Fri, 3 Mar 2023 11:25:07 +0100 Message-Id: <20230303102507.527790-11-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230303102507.527790-1-juraj.linkes@pantheon.tech> References: <20230223152840.634183-1-juraj.linkes@pantheon.tech> <20230303102507.527790-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Document how to configure and run DTS. Also add documentation related to new features: SUT setup and a brief test suite implementation cookbook. Signed-off-by: Juraj Linkeš --- doc/guides/tools/dts.rst | 165 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 163 insertions(+), 2 deletions(-) diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst index daf54359ed..ebd6dceb6a 100644 --- a/doc/guides/tools/dts.rst +++ b/doc/guides/tools/dts.rst @@ -1,5 +1,5 @@ .. SPDX-License-Identifier: BSD-3-Clause - Copyright(c) 2022 PANTHEON.tech s.r.o. + Copyright(c) 2022-2023 PANTHEON.tech s.r.o. DPDK Test Suite =============== @@ -56,7 +56,7 @@ DTS runtime environment or just plain DTS environment are used interchangeably. Setting up DTS environment --------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~ #. **Python Version** @@ -93,6 +93,167 @@ Setting up DTS environment poetry install poetry shell +#. **SSH Connection** + + DTS uses Python pexpect for SSH connections between DTS environment and the other hosts. + The pexpect implementation is a wrapper around the ssh command in the DTS environment. + This means it'll use the SSH agent providing the ssh command and its keys. + + +Setting up System Under Test +---------------------------- + +There are two areas that need to be set up on a System Under Test: + +#. **DPDK dependencies** + + DPDK will be built and run on the SUT. + Consult the Getting Started guides for the list of dependencies for each distribution. + +#. **Hardware dependencies** + + Any hardware DPDK uses needs a proper driver + and most OS distributions provide those, but the version may not be satisfactory. + It's up to each user to install the driver they're interested in testing. + The hardware also may also need firmware upgrades, which is also left at user discretion. + +#. **Hugepages** + + There are two ways to configure hugepages: + + * DTS configuration + + You may specify the optional hugepage configuration in the DTS config file. + If you do, DTS will take care of configuring hugepages, + overwriting your current SUT hugepage configuration. + + * System under test configuration + + It's possible to use the hugepage configuration already present on the SUT. + If you wish to do so, don't specify the hugepage configuration in the DTS config file. + + +Running DTS +----------- + +DTS needs to know which nodes to connect to and what hardware to use on those nodes. +Once that's configured, DTS needs a DPDK tarball and it's ready to run. + +Configuring DTS +~~~~~~~~~~~~~~~ + +DTS configuration is split into nodes and executions and build targets within executions. +By default, DTS will try to use the ``dts/conf.yaml`` config file, +which is a template that illustrates what can be configured in DTS: + + .. literalinclude:: ../../../dts/conf.yaml + :language: yaml + :start-at: executions: + + +The user must be root or any other user with prompt starting with ``#``. +The other fields are mostly self-explanatory +and documented in more detail in ``dts/framework/config/conf_yaml_schema.json``. + +DTS Execution +~~~~~~~~~~~~~ + +DTS is run with ``main.py`` located in the ``dts`` directory after entering Poetry shell:: + + usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] + [-v VERBOSE] [-s SKIP_SETUP] [--tarball TARBALL] + [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES] + [--re-run RE_RUN] + + Run DPDK test suites. All options may be specified with the environment variables provided in + brackets. Command line arguments have higher priority. + + options: + -h, --help show this help message and exit + --config-file CONFIG_FILE + [DTS_CFG_FILE] configuration file that describes the test cases, SUTs + and targets. (default: conf.yaml) + --output-dir OUTPUT_DIR, --output OUTPUT_DIR + [DTS_OUTPUT_DIR] Output directory where dts logs and results are + saved. (default: output) + -t TIMEOUT, --timeout TIMEOUT + [DTS_TIMEOUT] The default timeout for all DTS operations except for + compiling DPDK. (default: 15) + -v VERBOSE, --verbose VERBOSE + [DTS_VERBOSE] Set to 'Y' to enable verbose output, logging all + messages to the console. (default: N) + -s SKIP_SETUP, --skip-setup SKIP_SETUP + [DTS_SKIP_SETUP] Set to 'Y' to skip all setup steps on SUT and TG + nodes. (default: N) + --tarball TARBALL, --snapshot TARBALL + [DTS_DPDK_TARBALL] Path to DPDK source code tarball which will be + used in testing. (default: dpdk.tar.xz) + --compile-timeout COMPILE_TIMEOUT + [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200) + --test-cases TEST_CASES + [DTS_TESTCASES] Comma-separated list of test cases to execute. + Unknown test cases will be silently ignored. (default: ) + --re-run RE_RUN, --re_run RE_RUN + [DTS_RERUN] Re-run each test case the specified amount of times if a + test failure occurs (default: 0) + + +The brackets contain the names of environment variables that set the same thing. +The minimum DTS needs is a config file and a DPDK tarball. +You may pass those to DTS using the command line arguments or use the default paths. + + +DTS Results +~~~~~~~~~~~ + +Results are stored in the output dir by default +which be changed with the ``--output-dir`` command line argument. +The results contain basic statistics of passed/failed test cases and DPDK version. + + +How To Write a Test Suite +------------------------- + +All test suites inherit from ``TestSuite`` defined in ``dts/framework/test_suite.py``. +There are four types of methods that comprise a test suite: + +#. **Test cases** + + | Test cases are methods that start with a particular prefix. + | Functional test cases start with ``test_``, e.g. ``test_hello_world_single_core``. + | Performance test cases start with ``test_perf_``, e.g. ``test_perf_nic_single_core``. + | A test suite may have any number of functional and/or performance test cases. + However, these test cases must test the same feature, + following the rule of one feature = one test suite. + Test cases for one feature don't need to be grouped in just one test suite, though. + If the feature requires many testing scenarios to cover, + the test cases would be better off spread over multiple test suites + so that each test suite doesn't take too long to execute. + +#. **Setup and Teardown methods** + + | There are setup and teardown methods for the whole test suite and each individual test case. + | Methods ``set_up_suite`` and ``tear_down_suite`` will be executed + before any and after all test cases have been executed, respectively. + | Methods ``set_up_test_case`` and ``tear_down_test_case`` will be executed + before and after each test case, respectively. + | These methods don't need to be implemented if there's no need for them in a test suite. + In that case, nothing will happen when they're is executed. + +#. **Test case verification** + + Test case verification should be done with the ``verify`` method, which records the result. + The method should be called at the end of each test case. + +#. **Other methods** + + Of course, all test suite code should adhere to coding standards. + Only the above methods will be treated specially and any other methods may be defined + (which should be mostly private methods needed by each particular test suite). + Any specific features (such as NIC configuration) required by a test suite + should be implemented in the ``SutNode`` class (and the underlying classes that ``SutNode`` uses) + and used by the test suite via the ``sut_node`` field. + DTS Developer Tools ------------------- -- 2.30.2