From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8684A41D52; Thu, 23 Feb 2023 16:30:11 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0B560430A7; Thu, 23 Feb 2023 16:29:05 +0100 (CET) Received: from mail-ed1-f53.google.com (mail-ed1-f53.google.com [209.85.208.53]) by mails.dpdk.org (Postfix) with ESMTP id 8BF4540ED5 for ; Thu, 23 Feb 2023 16:28:56 +0100 (CET) Received: by mail-ed1-f53.google.com with SMTP id d30so2236499eda.4 for ; Thu, 23 Feb 2023 07:28:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon-tech.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TtKGqXREbdV24BtUnuvxKsFeNtDkF+SAMUN6axw569Q=; b=jm4ldl9g+mmLJ15m7AkT9cPfkIKaniXQXVS2dPrvYqHJe/Zi+V2vxh/Mi+iGtFko6e B1sTFWNytfp+tC1FHzXQI5UyA6HitAj8biw5KJs6brcWYmvkxIXy3EBE0gVsXNKf2VLJ lwSIaSqwGiMyNRo5s4nBjX398tSPZBbWYTmslOYvqk7maRga22J2q/u1jmJRYZri+icL OVKka2EdgCHE4GmJl1PRxTf0c+gnHiq+xWD2nVXQJEHCymPS0TUSNWLoU73fZxT+8NAV 9yux1bY9LiTLAfI/uxsIOvEwDfxV4bf7DSpcGtw0jleSVBq0ieAzRbacUToDHSaFy5P5 hHpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TtKGqXREbdV24BtUnuvxKsFeNtDkF+SAMUN6axw569Q=; b=TTYG9CIoDlN6a3HxIEW42zHMmJ8gEnII499kviFq/tcAxkHtkkGDslkHYiicpPy8Xw Qh0kQ1Uk7vdOvbJRqd0Q77vtUa5jgFnf4ZP9BmTW1RN19VR4LWk/m5UaRbo7MiqSCtAF ylz1KbPj3DVa4ZHgx42czFu8Mn12gGM/NQ6k59KzZ+8fT0BN1pc0Th+YVnNTmQn/ceWJ YXG2ETQ+NGSK3bLGM001YkUm21fcrDAIHt1/SpjxDPgWLTr+BZ0YuP+N0orp76SPmpEN BDhHzJp5HBnIqPijgv4R8S8MUSISzjxOLr9xvN0JDB32f4Vr9re0WVAnw8XNwMNC6qvH CIgg== X-Gm-Message-State: AO0yUKXoyTrde89ssFXcieqQoBhy0hwywDAvd68l67DcUcASEHR0C86T 8/0ruRtSXgOYtaasVk/+0rM+qKkGdN+/+DC6 X-Google-Smtp-Source: AK7set/rz9AxD+NfYTQcQmm8UifGw11Cg5N8OtMJEHGQOeYOdUuOc55UnKqllDfupj+4FTz7hI8NhA== X-Received: by 2002:a17:906:d979:b0:8af:2107:6ce5 with SMTP id rp25-20020a170906d97900b008af21076ce5mr17851785ejb.35.1677166136201; Thu, 23 Feb 2023 07:28:56 -0800 (PST) Received: from localhost.localdomain ([84.245.121.112]) by smtp.gmail.com with ESMTPSA id r6-20020a50c006000000b004af6a8617ffsm1158892edb.46.2023.02.23.07.28.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 07:28:55 -0800 (PST) From: =?UTF-8?q?Juraj=20Linke=C5=A1?= To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, lijuan.tu@intel.com, bruce.richardson@intel.com, probb@iol.unh.edu Cc: dev@dpdk.org, =?UTF-8?q?Juraj=20Linke=C5=A1?= Subject: [PATCH v5 10/10] doc: update DTS setup and test suite cookbook Date: Thu, 23 Feb 2023 16:28:40 +0100 Message-Id: <20230223152840.634183-11-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230223152840.634183-1-juraj.linkes@pantheon.tech> References: <20230213152846.284191-1-juraj.linkes@pantheon.tech> <20230223152840.634183-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Document how to configure and run DTS. Also add documentation related to new features: SUT setup and a brief test suite implementation cookbook. Signed-off-by: Juraj Linkeš --- doc/guides/tools/dts.rst | 165 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 163 insertions(+), 2 deletions(-) diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst index daf54359ed..ebd6dceb6a 100644 --- a/doc/guides/tools/dts.rst +++ b/doc/guides/tools/dts.rst @@ -1,5 +1,5 @@ .. SPDX-License-Identifier: BSD-3-Clause - Copyright(c) 2022 PANTHEON.tech s.r.o. + Copyright(c) 2022-2023 PANTHEON.tech s.r.o. DPDK Test Suite =============== @@ -56,7 +56,7 @@ DTS runtime environment or just plain DTS environment are used interchangeably. Setting up DTS environment --------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~ #. **Python Version** @@ -93,6 +93,167 @@ Setting up DTS environment poetry install poetry shell +#. **SSH Connection** + + DTS uses Python pexpect for SSH connections between DTS environment and the other hosts. + The pexpect implementation is a wrapper around the ssh command in the DTS environment. + This means it'll use the SSH agent providing the ssh command and its keys. + + +Setting up System Under Test +---------------------------- + +There are two areas that need to be set up on a System Under Test: + +#. **DPDK dependencies** + + DPDK will be built and run on the SUT. + Consult the Getting Started guides for the list of dependencies for each distribution. + +#. **Hardware dependencies** + + Any hardware DPDK uses needs a proper driver + and most OS distributions provide those, but the version may not be satisfactory. + It's up to each user to install the driver they're interested in testing. + The hardware also may also need firmware upgrades, which is also left at user discretion. + +#. **Hugepages** + + There are two ways to configure hugepages: + + * DTS configuration + + You may specify the optional hugepage configuration in the DTS config file. + If you do, DTS will take care of configuring hugepages, + overwriting your current SUT hugepage configuration. + + * System under test configuration + + It's possible to use the hugepage configuration already present on the SUT. + If you wish to do so, don't specify the hugepage configuration in the DTS config file. + + +Running DTS +----------- + +DTS needs to know which nodes to connect to and what hardware to use on those nodes. +Once that's configured, DTS needs a DPDK tarball and it's ready to run. + +Configuring DTS +~~~~~~~~~~~~~~~ + +DTS configuration is split into nodes and executions and build targets within executions. +By default, DTS will try to use the ``dts/conf.yaml`` config file, +which is a template that illustrates what can be configured in DTS: + + .. literalinclude:: ../../../dts/conf.yaml + :language: yaml + :start-at: executions: + + +The user must be root or any other user with prompt starting with ``#``. +The other fields are mostly self-explanatory +and documented in more detail in ``dts/framework/config/conf_yaml_schema.json``. + +DTS Execution +~~~~~~~~~~~~~ + +DTS is run with ``main.py`` located in the ``dts`` directory after entering Poetry shell:: + + usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] + [-v VERBOSE] [-s SKIP_SETUP] [--tarball TARBALL] + [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES] + [--re-run RE_RUN] + + Run DPDK test suites. All options may be specified with the environment variables provided in + brackets. Command line arguments have higher priority. + + options: + -h, --help show this help message and exit + --config-file CONFIG_FILE + [DTS_CFG_FILE] configuration file that describes the test cases, SUTs + and targets. (default: conf.yaml) + --output-dir OUTPUT_DIR, --output OUTPUT_DIR + [DTS_OUTPUT_DIR] Output directory where dts logs and results are + saved. (default: output) + -t TIMEOUT, --timeout TIMEOUT + [DTS_TIMEOUT] The default timeout for all DTS operations except for + compiling DPDK. (default: 15) + -v VERBOSE, --verbose VERBOSE + [DTS_VERBOSE] Set to 'Y' to enable verbose output, logging all + messages to the console. (default: N) + -s SKIP_SETUP, --skip-setup SKIP_SETUP + [DTS_SKIP_SETUP] Set to 'Y' to skip all setup steps on SUT and TG + nodes. (default: N) + --tarball TARBALL, --snapshot TARBALL + [DTS_DPDK_TARBALL] Path to DPDK source code tarball which will be + used in testing. (default: dpdk.tar.xz) + --compile-timeout COMPILE_TIMEOUT + [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200) + --test-cases TEST_CASES + [DTS_TESTCASES] Comma-separated list of test cases to execute. + Unknown test cases will be silently ignored. (default: ) + --re-run RE_RUN, --re_run RE_RUN + [DTS_RERUN] Re-run each test case the specified amount of times if a + test failure occurs (default: 0) + + +The brackets contain the names of environment variables that set the same thing. +The minimum DTS needs is a config file and a DPDK tarball. +You may pass those to DTS using the command line arguments or use the default paths. + + +DTS Results +~~~~~~~~~~~ + +Results are stored in the output dir by default +which be changed with the ``--output-dir`` command line argument. +The results contain basic statistics of passed/failed test cases and DPDK version. + + +How To Write a Test Suite +------------------------- + +All test suites inherit from ``TestSuite`` defined in ``dts/framework/test_suite.py``. +There are four types of methods that comprise a test suite: + +#. **Test cases** + + | Test cases are methods that start with a particular prefix. + | Functional test cases start with ``test_``, e.g. ``test_hello_world_single_core``. + | Performance test cases start with ``test_perf_``, e.g. ``test_perf_nic_single_core``. + | A test suite may have any number of functional and/or performance test cases. + However, these test cases must test the same feature, + following the rule of one feature = one test suite. + Test cases for one feature don't need to be grouped in just one test suite, though. + If the feature requires many testing scenarios to cover, + the test cases would be better off spread over multiple test suites + so that each test suite doesn't take too long to execute. + +#. **Setup and Teardown methods** + + | There are setup and teardown methods for the whole test suite and each individual test case. + | Methods ``set_up_suite`` and ``tear_down_suite`` will be executed + before any and after all test cases have been executed, respectively. + | Methods ``set_up_test_case`` and ``tear_down_test_case`` will be executed + before and after each test case, respectively. + | These methods don't need to be implemented if there's no need for them in a test suite. + In that case, nothing will happen when they're is executed. + +#. **Test case verification** + + Test case verification should be done with the ``verify`` method, which records the result. + The method should be called at the end of each test case. + +#. **Other methods** + + Of course, all test suite code should adhere to coding standards. + Only the above methods will be treated specially and any other methods may be defined + (which should be mostly private methods needed by each particular test suite). + Any specific features (such as NIC configuration) required by a test suite + should be implemented in the ``SutNode`` class (and the underlying classes that ``SutNode`` uses) + and used by the test suite via the ``sut_node`` field. + DTS Developer Tools ------------------- -- 2.30.2