From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 00D8FA0547; Mon, 8 Feb 2021 19:03:28 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 768E0160727; Mon, 8 Feb 2021 19:03:28 +0100 (CET) Received: from dal2relay17.mxroute.com (dal2relay17.mxroute.com [64.40.26.17]) by mails.dpdk.org (Postfix) with ESMTP id 41D30160722 for ; Mon, 8 Feb 2021 19:03:25 +0100 (CET) Received: from filter004.mxroute.com ([149.28.56.236] 149.28.56.236.vultr.com) (Authenticated sender: mN4UYu2MZsgR) by dal2relay17.mxroute.com (ZoneMTA) with ESMTPSA id 17782cff78c0008aa5.001 for (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256); Mon, 08 Feb 2021 18:03:23 +0000 X-Zone-Loop: 8f31cd75636b2cb51b118a283bbae642834859be16c2 X-Originating-IP: [149.28.56.236] DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ashroe.eu; s=x; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=6U6WV7NGKpfe89T7eRlFDS8J1h1mxAWnTW2xn/5VvqI=; b=Xul4XOfo1fUF3IMGRkudbPaQ8G bN4dTnF0TjbeCjMBADgq1R9P1uoPfSNHGYbKq5Ht8ycAz6gW5SlyYwp1fNumMaK+s5eGOFwrNeXil aqIpYplDJ3ED05Sk6Rx3TdQ5BJDm5jBSzncQZ9dzPqNH0If1lep+mqLCxys1baJz5qYDE5kf6jwE9 iN1Xh2xBBwQ7OaPFnVmr2ZgTFxxbiuZxmDKXdWob7Np3zbYEJjj//CdzXYgz0/USbbxDF50LwnkZD ZwBn2jiW+Z7k2bsaa4t3zYXvT7MO2DBhckX5mZj+bZfc7OcwyaDOKYADcoxDy2CxqFmUd+8GaLwK7 fIXBkpiQ==; To: Aaron Conole , dev@dpdk.org Cc: Bruce Richardson References: <20210208165234.319578-1-aconole@redhat.com> From: "Kinsella, Ray" Message-ID: Date: Mon, 8 Feb 2021 18:03:19 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <20210208165234.319578-1-aconole@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-AuthUser: mdr@ashroe.eu Subject: Re: [dpdk-dev] [PATCH] guides: add a testing guide for developing tests X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 08/02/2021 16:52, Aaron Conole wrote: > The DPDK testing infrastructure includes a comprehensive set of > libraries, utilities, and CI integrations for developers to test > their code changes. This isn't well documented, however. > > Document the basics for adding a test suite to the infrastructure > and enabling that test suite for continuous integration platforms > so that newer developers can understand how to develop test suites > and test cases. > > Signed-off-by: Aaron Conole > --- > v0: Added information for TEST_SKIPPED and details about generating > code coverage to help with ideas for writing unit test cases. > > doc/guides/contributing/index.rst | 1 + > doc/guides/contributing/testing.rst | 231 ++++++++++++++++++++++++++++ > 2 files changed, 232 insertions(+) > create mode 100644 doc/guides/contributing/testing.rst > > diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst > index 2fefd91931..41909d949b 100644 > --- a/doc/guides/contributing/index.rst > +++ b/doc/guides/contributing/index.rst > @@ -14,6 +14,7 @@ Contributor's Guidelines > abi_versioning > documentation > patches > + testing > vulnerability > stable > cheatsheet > diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst > new file mode 100644 > index 0000000000..3ade654772 > --- /dev/null > +++ b/doc/guides/contributing/testing.rst > @@ -0,0 +1,231 @@ > +.. SPDX-License-Identifier: BSD-3-Clause > + Copyright 2018 The DPDK contributors > + > +.. _testing_guidelines: > + > +DPDK Testing Guidelines > +======================= > + > +This document outlines the guidelines for running and adding new > +tests to the in-tree DPDK test suites. > + > +The DPDK test suite model is loosely based on the xunit model, where > +tests are grouped into test suites, and suites are run by runners. > +For a basic overview, see the basic Wikipedia article on xunit: > +`xUnit - Wikipedia `_. > + > + > +Running a test > +-------------- > + > +DPDK tests are run via the main test runniner, the `dpdk-test` app. runner > +The `dpdk-test` app is a command-line interface that facilitates > +running various tests or test suites. > + > +There are two modes of operation. The first mode is as an interactive > +command shell that allows launching specific test suites. This is > +the default operating mode of `dpdk-test` and can be done by:: is the double :: intentional? > + > + $ ./build/app/test/dpdk-test --dpdk-options-here > + EAL: Detected 4 lcore(s) > + EAL: Detected 1 NUMA nodes > + EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem > + EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket > + EAL: Selected IOVA mode 'VA' > + EAL: Probing VFIO support... > + EAL: PCI device 0000:00:1f.6 on NUMA socket -1 > + EAL: Invalid NUMA socket, default to 0 > + EAL: probe driver: 8086:15d7 net_e1000_em > + APP: HPET is not enabled, using TSC as default timer > + RTE>> > + > +At the prompt, simply type the name of the test suite you wish to run > +and it will execute. > + > +The second form is useful for a scripting environment, and is used by > +the DPDK meson build system. This mode is invoked by assigning a > +specific test suite name to the environment variable `DPDK_TEST` > +before invoking the `dpdk-test` command, such as:: > + > + $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --no-huge better to say --dpdk-options-here instead of --no-huge, as above. > + EAL: Detected 4 lcore(s) > + EAL: Detected 1 NUMA nodes > + EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem > + EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket > + EAL: Selected IOVA mode 'VA' > + EAL: Probing VFIO support... > + EAL: PCI device 0000:00:1f.6 on NUMA socket -1 > + EAL: Invalid NUMA socket, default to 0 > + EAL: probe driver: 8086:15d7 net_e1000_em > + APP: HPET is not enabled, using TSC as default timer > + RTE>>version_autotest > + Version string: 'DPDK 20.02.0-rc0' > + Test OK > + RTE>>$ > + > +The above shows running a specific test case. On success, the return > +code will be '0', otherwise it will be set to some error value (such > +as '255'). > + > + > +Running all tests > +----------------- > + > +In order to allow developers to quickly execute all the standard > +internal tests without needing to remember or look up each test suite > +name, the build system includes a standard way of executing the > +default test suites. After building via `ninja`, the ``meson test`` > +command will execute the standard tests and report errors. > + > +There are four groups of default test suites. The first group is > +the **fast** test suite, which is the largest group of test cases. > +These are the bulk of the unit tests to validate functional blocks. > +The second group is the **perf** tests. These test suites can take > +longer to run and do performance evaluations. The third group is > +the **driver** test suite, which is mostly for special hardware > +related testing (such as `cryptodev`). The last group are the > +**debug** tests. These mostly are used to dump system information. > + > +The suites can be selected by adding the ``--suite`` option to the > +``meson test`` command. Ex: ``meson test --suite fast-tests`` show some console output in this section? > + > + > +Adding test suites > +------------------ > + > +To add a testsuite to the DPDK test application, create a new test > +file for that suite (ex: see *app/test/test_version.c* for the > +``version_autotest`` test suite). There are two useful things: There are two useful things? > + > + 1. REGISTER_TEST_COMMAND(command_name, function_to_execute) > + Registers a test command with the name `command_name` and which > + runs the function `function_to_execute` when `command_name` is > + invoked. > + > + 2. unit_test_suite_runner(struct unit_test_suite \*) > + Returns a runner for a full test suite object, which contains > + a test suite name, setup, teardown, and vector of unit test > + cases. > + > +Each test suite has a setup and teardown function that runs at the > +beginning and end of the test suite execution. Each unit test has > +a similar function for test case setup and teardown. > + > +Adding test cases is controlled via the `.unit_test_cases` element Test cases are added to the .unit_test_cases element of the of the unit test suite structure. > +of the unit test suite. Ex: > + > +.. code-block:: c > + :linenos: > + > + #include > + > + #include > + #include > + #include > + #include > + > + #include "test.h" > + > + static int testsuite_setup(void) { return TEST_SUCCESS; } > + static void testsuite_teardown(void) { } > + > + static int ut_setup(void) { return TEST_SUCCESS; } > + static void ut_teardown(void) { } > + > + static int test_case_first(void) { return TEST_SUCCESS; } > + > + static struct unit_test_suite example_testsuite = { > + .suite_name = "EXAMPLE TEST SUITE", > + .setup = testsuite_setup, > + .teardown = testsuite_teardown, > + .unit_test_cases = { > + TEST_CASE_ST(ut_setup, ut_teardown, test_case_first), > + > + TEST_CASES_END(), /**< NULL terminate unit test array */ > + }, > + }; > + > + static int example_tests() > + { > + return unit_test_suite_runner(&example_testsuite); > + } > + > + REGISTER_TEST_COMMAND(example_autotest, example_tests); > + > +The above code block is a small example that can be used to create a > +complete test suite with test case. > + > + > +Designing a test > +---------------- > + > +Test cases have multiple ways of indicating an error has occurred, > +in order to reflect failure state back to the runner. Using the > +various methods of indicating errors can assist in not only validating > +the requisite functionality is working, but also to help debug when > +a change in environment or code has caused things to go wrong. > + > +The first way to indicate a generic error is by returning a test > +result failure, using the *TEST_FAILED* error code. This is the most > +basic way of indicating that an error has occurred in a test routine. > +It isn't very informative to the user, so it should really be used in > +cases where the test has catastrophically failed. > + > +The preferred method of indicating an error is via the > +`RTE_TEST_ASSERT` family of macros, which will immediately return > +*TEST_FAILED* error condition, but will also log details about the > +failure. The basic form is: > + > +.. code-block:: c > + > + RTE_TEST_ASSERT(cond, msg, ...) > + > +In the above macro, *cond* is the condition to evaluate to **true**. > +Any generic condition can go here. The *msg* parameter will be a > +message to display if *cond* evaluates to **false**. Some specialized > +macros already exist. See `lib/librte_eal/include/rte_test.h` for > +a list of pre-build test assertions. > + > +Sometimes it is important to indicate that a test needs to be > +skipped, either because the environment isn't able to support running > +the test, or because some requisite functionality isn't availble. The available > +test suite supports returning a result of `TEST_SKIPPED` during test > +case setup, or during test case execution to indicate that the > +preconditions of the test aren't available. A skipped test isn't the > +same as a failed test. Failed tests indicate that all required > +functionality was enabled, and the environment was correct but the > +test failed due to some error in logic. A skipped test indicates > +that the test setup wasn't available to run. Simpler to provide an example of a skipped test, than a long explanation? > + > + > +Checking code coverage > +---------------------- > +One of the best ways to help design a test case is to add a new test > +which exercises functionality that doesn't currently have a test case. > +The best way to find these is to run the test suite with some > +additional code coverage reporting turned on. I would drop the previous sentences. They would just confuse a reader looking to generate code coverage. > The meson build system > +supports generating a code coverage report via the `-Db_coverage=true` > +option, in conjunction with a package like **lcov**, to generate an > +HTML code coverage report. Example:: > + > + $ meson covered -Db_coverage=true > + $ meson test -C covered --suite fast-tests > + $ ninja coverage-html -C covered > + > +The above will generate an html report in the > +`covered/meson-logs/coveragereport/` directory that can be explored > +for detailed code covered information. This can be used to assist > +in test development. > + > + > +Adding a suite to the default > +----------------------------- > + > +Adding to one of the default tests involves editing the appropriate > +meson build file `app/test/meson.build` and adding the command to > +the correct test suite class. Once added, the new test suite will > +be run as part of the appropriate class (fast, perf, driver, etc.). > + > +Some of these default test suites are run during continuous integration > +tests, making regression checking automatic for new patches submitted > +to the project. >