From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 597D3A054A; Tue, 9 Feb 2021 21:03:07 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3BFC9160748; Tue, 9 Feb 2021 21:03:07 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id 60AAA160747 for ; Tue, 9 Feb 2021 21:03:05 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1612900984; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=OENi+Y3G6/JRoNYu8jU+7fTGURpCbhOHK9MAR5A98Ww=; b=gfWDB40FIINY+X3yvNKnQVApJyjXVMXOUAgmaJfvFxPOo5VetwNfheOh6RYVELvlPwHQjd YB+VQO/Le7NzouxXASKqyQA+AK59U8nUxzqF3RDSiXWLe4a4let2B2ti7TGvl5JUTJxnGY IAUb3EHNBHc+1PGwlbU63jOKDmvvPSw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-285-wuPoeQ5gOZ-KVhnDJYfm-w-1; Tue, 09 Feb 2021 15:02:57 -0500 X-MC-Unique: wuPoeQ5gOZ-KVhnDJYfm-w-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8C034BBEE2; Tue, 9 Feb 2021 20:02:56 +0000 (UTC) Received: from dhcp-25.97.bos.redhat.com (ovpn-114-243.rdu2.redhat.com [10.10.114.243]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D9D4960C04; Tue, 9 Feb 2021 20:02:55 +0000 (UTC) From: Aaron Conole To: "Kinsella\, Ray" Cc: dev@dpdk.org, Bruce Richardson References: <20210208165234.319578-1-aconole@redhat.com> Date: Tue, 09 Feb 2021 15:02:55 -0500 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=aconole@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain Subject: Re: [dpdk-dev] [PATCH] guides: add a testing guide for developing tests X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" "Kinsella, Ray" writes: > On 08/02/2021 16:52, Aaron Conole wrote: >> The DPDK testing infrastructure includes a comprehensive set of >> libraries, utilities, and CI integrations for developers to test >> their code changes. This isn't well documented, however. >> >> Document the basics for adding a test suite to the infrastructure >> and enabling that test suite for continuous integration platforms >> so that newer developers can understand how to develop test suites >> and test cases. >> >> Signed-off-by: Aaron Conole >> --- >> v0: Added information for TEST_SKIPPED and details about generating >> code coverage to help with ideas for writing unit test cases. >> >> doc/guides/contributing/index.rst | 1 + >> doc/guides/contributing/testing.rst | 231 ++++++++++++++++++++++++++++ >> 2 files changed, 232 insertions(+) >> create mode 100644 doc/guides/contributing/testing.rst >> >> diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst >> index 2fefd91931..41909d949b 100644 >> --- a/doc/guides/contributing/index.rst >> +++ b/doc/guides/contributing/index.rst >> @@ -14,6 +14,7 @@ Contributor's Guidelines >> abi_versioning >> documentation >> patches >> + testing >> vulnerability >> stable >> cheatsheet >> diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst >> new file mode 100644 >> index 0000000000..3ade654772 >> --- /dev/null >> +++ b/doc/guides/contributing/testing.rst >> @@ -0,0 +1,231 @@ >> +.. SPDX-License-Identifier: BSD-3-Clause >> + Copyright 2018 The DPDK contributors >> + >> +.. _testing_guidelines: >> + >> +DPDK Testing Guidelines >> +======================= >> + >> +This document outlines the guidelines for running and adding new >> +tests to the in-tree DPDK test suites. >> + >> +The DPDK test suite model is loosely based on the xunit model, where >> +tests are grouped into test suites, and suites are run by runners. >> +For a basic overview, see the basic Wikipedia article on xunit: >> +`xUnit - Wikipedia `_. >> + >> + >> +Running a test >> +-------------- >> + >> +DPDK tests are run via the main test runniner, the `dpdk-test` app. > > runner D'oh! >> +The `dpdk-test` app is a command-line interface that facilitates >> +running various tests or test suites. >> + >> +There are two modes of operation. The first mode is as an interactive >> +command shell that allows launching specific test suites. This is >> +the default operating mode of `dpdk-test` and can be done by:: > > is the double :: intentional? Yes - it designates the next block as a code block. >> + >> + $ ./build/app/test/dpdk-test --dpdk-options-here >> + EAL: Detected 4 lcore(s) >> + EAL: Detected 1 NUMA nodes >> + EAL: Static memory layout is selected, amount of reserved memory >> can be adjusted with -m or --socket-mem >> + EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket >> + EAL: Selected IOVA mode 'VA' >> + EAL: Probing VFIO support... >> + EAL: PCI device 0000:00:1f.6 on NUMA socket -1 >> + EAL: Invalid NUMA socket, default to 0 >> + EAL: probe driver: 8086:15d7 net_e1000_em >> + APP: HPET is not enabled, using TSC as default timer >> + RTE>> >> + >> +At the prompt, simply type the name of the test suite you wish to run >> +and it will execute. >> + >> +The second form is useful for a scripting environment, and is used by >> +the DPDK meson build system. This mode is invoked by assigning a >> +specific test suite name to the environment variable `DPDK_TEST` >> +before invoking the `dpdk-test` command, such as:: >> + >> + $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --no-huge > > better to say --dpdk-options-here instead of --no-huge, as above. Okay, I can do that. >> + EAL: Detected 4 lcore(s) >> + EAL: Detected 1 NUMA nodes >> + EAL: Static memory layout is selected, amount of reserved memory >> can be adjusted with -m or --socket-mem >> + EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket >> + EAL: Selected IOVA mode 'VA' >> + EAL: Probing VFIO support... >> + EAL: PCI device 0000:00:1f.6 on NUMA socket -1 >> + EAL: Invalid NUMA socket, default to 0 >> + EAL: probe driver: 8086:15d7 net_e1000_em >> + APP: HPET is not enabled, using TSC as default timer >> + RTE>>version_autotest >> + Version string: 'DPDK 20.02.0-rc0' >> + Test OK >> + RTE>>$ >> + >> +The above shows running a specific test case. On success, the return >> +code will be '0', otherwise it will be set to some error value (such >> +as '255'). >> + >> + >> +Running all tests >> +----------------- >> + >> +In order to allow developers to quickly execute all the standard >> +internal tests without needing to remember or look up each test suite >> +name, the build system includes a standard way of executing the >> +default test suites. After building via `ninja`, the ``meson test`` >> +command will execute the standard tests and report errors. >> + >> +There are four groups of default test suites. The first group is >> +the **fast** test suite, which is the largest group of test cases. >> +These are the bulk of the unit tests to validate functional blocks. >> +The second group is the **perf** tests. These test suites can take >> +longer to run and do performance evaluations. The third group is >> +the **driver** test suite, which is mostly for special hardware >> +related testing (such as `cryptodev`). The last group are the >> +**debug** tests. These mostly are used to dump system information. >> + >> +The suites can be selected by adding the ``--suite`` option to the >> +``meson test`` command. Ex: ``meson test --suite fast-tests`` > > show some console output in this section? Okay. >> + >> + >> +Adding test suites >> +------------------ >> + >> +To add a testsuite to the DPDK test application, create a new test >> +file for that suite (ex: see *app/test/test_version.c* for the >> +``version_autotest`` test suite). There are two useful things: > > There are two useful things? Okay, I will rephrase because it sounds awkward. But I don't think it's wrong ;) >> + >> + 1. REGISTER_TEST_COMMAND(command_name, function_to_execute) >> + Registers a test command with the name `command_name` and which >> + runs the function `function_to_execute` when `command_name` is >> + invoked. >> + >> + 2. unit_test_suite_runner(struct unit_test_suite \*) >> + Returns a runner for a full test suite object, which contains >> + a test suite name, setup, teardown, and vector of unit test >> + cases. >> + >> +Each test suite has a setup and teardown function that runs at the >> +beginning and end of the test suite execution. Each unit test has >> +a similar function for test case setup and teardown. >> + >> +Adding test cases is controlled via the `.unit_test_cases` element > > Test cases are added to the .unit_test_cases element of the > of the unit test suite structure. Will rephrase. >> +of the unit test suite. Ex: >> + >> +.. code-block:: c >> + :linenos: >> + >> + #include >> + >> + #include >> + #include >> + #include >> + #include >> + >> + #include "test.h" >> + >> + static int testsuite_setup(void) { return TEST_SUCCESS; } >> + static void testsuite_teardown(void) { } >> + >> + static int ut_setup(void) { return TEST_SUCCESS; } >> + static void ut_teardown(void) { } >> + >> + static int test_case_first(void) { return TEST_SUCCESS; } >> + >> + static struct unit_test_suite example_testsuite = { >> + .suite_name = "EXAMPLE TEST SUITE", >> + .setup = testsuite_setup, >> + .teardown = testsuite_teardown, >> + .unit_test_cases = { >> + TEST_CASE_ST(ut_setup, ut_teardown, test_case_first), >> + >> + TEST_CASES_END(), /**< NULL terminate unit test array */ >> + }, >> + }; >> + >> + static int example_tests() >> + { >> + return unit_test_suite_runner(&example_testsuite); >> + } >> + >> + REGISTER_TEST_COMMAND(example_autotest, example_tests); >> + >> +The above code block is a small example that can be used to create a >> +complete test suite with test case. >> + >> + >> +Designing a test >> +---------------- >> + >> +Test cases have multiple ways of indicating an error has occurred, >> +in order to reflect failure state back to the runner. Using the >> +various methods of indicating errors can assist in not only validating >> +the requisite functionality is working, but also to help debug when >> +a change in environment or code has caused things to go wrong. >> + >> +The first way to indicate a generic error is by returning a test >> +result failure, using the *TEST_FAILED* error code. This is the most >> +basic way of indicating that an error has occurred in a test routine. >> +It isn't very informative to the user, so it should really be used in >> +cases where the test has catastrophically failed. >> + >> +The preferred method of indicating an error is via the >> +`RTE_TEST_ASSERT` family of macros, which will immediately return >> +*TEST_FAILED* error condition, but will also log details about the >> +failure. The basic form is: >> + >> +.. code-block:: c >> + >> + RTE_TEST_ASSERT(cond, msg, ...) >> + >> +In the above macro, *cond* is the condition to evaluate to **true**. >> +Any generic condition can go here. The *msg* parameter will be a >> +message to display if *cond* evaluates to **false**. Some specialized >> +macros already exist. See `lib/librte_eal/include/rte_test.h` for >> +a list of pre-build test assertions. >> + >> +Sometimes it is important to indicate that a test needs to be >> +skipped, either because the environment isn't able to support running >> +the test, or because some requisite functionality isn't availble. The > > available D'oh! >> +test suite supports returning a result of `TEST_SKIPPED` during test >> +case setup, or during test case execution to indicate that the >> +preconditions of the test aren't available. A skipped test isn't the >> +same as a failed test. Failed tests indicate that all required >> +functionality was enabled, and the environment was correct but the >> +test failed due to some error in logic. A skipped test indicates >> +that the test setup wasn't available to run. > > Simpler to provide an example of a skipped test, than a long explanation? I can do both. >> + >> + >> +Checking code coverage >> +---------------------- >> +One of the best ways to help design a test case is to add a new test >> +which exercises functionality that doesn't currently have a test case. >> +The best way to find these is to run the test suite with some >> +additional code coverage reporting turned on. > > I would drop the previous sentences. > They would just confuse a reader looking to generate code coverage. Okay. I felt it was a bit strange to introduce code coverage here, and I thought it was a needed bridge. I can drop it. >> The meson build system >> +supports generating a code coverage report via the `-Db_coverage=true` >> +option, in conjunction with a package like **lcov**, to generate an >> +HTML code coverage report. Example:: >> + >> + $ meson covered -Db_coverage=true >> + $ meson test -C covered --suite fast-tests >> + $ ninja coverage-html -C covered >> + >> +The above will generate an html report in the >> +`covered/meson-logs/coveragereport/` directory that can be explored >> +for detailed code covered information. This can be used to assist >> +in test development. >> + >> + >> +Adding a suite to the default >> +----------------------------- >> + >> +Adding to one of the default tests involves editing the appropriate >> +meson build file `app/test/meson.build` and adding the command to >> +the correct test suite class. Once added, the new test suite will >> +be run as part of the appropriate class (fast, perf, driver, etc.). >> + >> +Some of these default test suites are run during continuous integration >> +tests, making regression checking automatic for new patches submitted >> +to the project. >>