From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id 87F1DA0547;
Mon, 8 Feb 2021 17:52:47 +0100 (CET)
Received: from [217.70.189.124] (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id EFA0916071A;
Mon, 8 Feb 2021 17:52:46 +0100 (CET)
Received: from us-smtp-delivery-124.mimecast.com
(us-smtp-delivery-124.mimecast.com [63.128.21.124])
by mails.dpdk.org (Postfix) with ESMTP id 835AF160714
for <dev@dpdk.org>; Mon, 8 Feb 2021 17:52:45 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
s=mimecast20190719; t=1612803164;
h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
to:to:cc:cc:mime-version:mime-version:content-type:content-type:
content-transfer-encoding:content-transfer-encoding;
bh=06tslKWGxCgMpa070ClBoqvw1jN9/SeVx7kee5gP2XU=;
b=RAoaITx9s5M02hupw8RQhFyC0RZWSgroIGrQiSV6NYpa4wOlpJHf7Bl75m60I5olsbTXdf
/YXBMariJXyGIGvM7MhW//lIEBApptFLkLqmCu7kJ8GQ6Avur3mTeJogigMm5u1NH8CP4Y
cU60ECyFUWfNx/tbbO+n0mqj9yw9Gvo=
Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com
[209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id
us-mta-504-3_NW2UPxOGWcID8ZXT-tyQ-1; Mon, 08 Feb 2021 11:52:39 -0500
X-MC-Unique: 3_NW2UPxOGWcID8ZXT-tyQ-1
Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com
[10.5.11.14])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6993E100CCC5;
Mon, 8 Feb 2021 16:52:37 +0000 (UTC)
Received: from dhcp-25.97.bos.redhat.com (ovpn-114-243.rdu2.redhat.com
[10.10.114.243])
by smtp.corp.redhat.com (Postfix) with ESMTP id 2B54E5D9DC;
Mon, 8 Feb 2021 16:52:35 +0000 (UTC)
From: Aaron Conole <aconole@redhat.com>
To: dev@dpdk.org
Cc: Bruce Richardson <bruce.richardson@intel.com>, Ray Kinsella <mdr@ashroe.eu>
Date: Mon, 8 Feb 2021 11:52:34 -0500
Message-Id: <20210208165234.319578-1-aconole@redhat.com>
MIME-Version: 1.0
X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14
Authentication-Results: relay.mimecast.com;
auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=aconole@redhat.com
X-Mimecast-Spam-Score: 0
X-Mimecast-Originator: redhat.com
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset="US-ASCII"
Subject: [dpdk-dev] [PATCH] guides: add a testing guide for developing tests
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>
The DPDK testing infrastructure includes a comprehensive set of
libraries, utilities, and CI integrations for developers to test
their code changes. This isn't well documented, however.
Document the basics for adding a test suite to the infrastructure
and enabling that test suite for continuous integration platforms
so that newer developers can understand how to develop test suites
and test cases.
Signed-off-by: Aaron Conole <aconole@redhat.com>
---
v0: Added information for TEST_SKIPPED and details about generating
code coverage to help with ideas for writing unit test cases.
doc/guides/contributing/index.rst | 1 +
doc/guides/contributing/testing.rst | 231 ++++++++++++++++++++++++++++
2 files changed, 232 insertions(+)
create mode 100644 doc/guides/contributing/testing.rst
diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst
index 2fefd91931..41909d949b 100644
--- a/doc/guides/contributing/index.rst
+++ b/doc/guides/contributing/index.rst
@@ -14,6 +14,7 @@ Contributor's Guidelines
abi_versioning
documentation
patches
+ testing
vulnerability
stable
cheatsheet
diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
new file mode 100644
index 0000000000..3ade654772
--- /dev/null
+++ b/doc/guides/contributing/testing.rst
@@ -0,0 +1,231 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright 2018 The DPDK contributors
+
+.. _testing_guidelines:
+
+DPDK Testing Guidelines
+=======================
+
+This document outlines the guidelines for running and adding new
+tests to the in-tree DPDK test suites.
+
+The DPDK test suite model is loosely based on the xunit model, where
+tests are grouped into test suites, and suites are run by runners.
+For a basic overview, see the basic Wikipedia article on xunit:
+`xUnit - Wikipedia <https://en.wikipedia.org/wiki/XUnit>`_.
+
+
+Running a test
+--------------
+
+DPDK tests are run via the main test runniner, the `dpdk-test` app.
+The `dpdk-test` app is a command-line interface that facilitates
+running various tests or test suites.
+
+There are two modes of operation. The first mode is as an interactive
+command shell that allows launching specific test suites. This is
+the default operating mode of `dpdk-test` and can be done by::
+
+ $ ./build/app/test/dpdk-test --dpdk-options-here
+ EAL: Detected 4 lcore(s)
+ EAL: Detected 1 NUMA nodes
+ EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+ EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+ EAL: Selected IOVA mode 'VA'
+ EAL: Probing VFIO support...
+ EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+ EAL: Invalid NUMA socket, default to 0
+ EAL: probe driver: 8086:15d7 net_e1000_em
+ APP: HPET is not enabled, using TSC as default timer
+ RTE>>
+
+At the prompt, simply type the name of the test suite you wish to run
+and it will execute.
+
+The second form is useful for a scripting environment, and is used by
+the DPDK meson build system. This mode is invoked by assigning a
+specific test suite name to the environment variable `DPDK_TEST`
+before invoking the `dpdk-test` command, such as::
+
+ $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --no-huge
+ EAL: Detected 4 lcore(s)
+ EAL: Detected 1 NUMA nodes
+ EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+ EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+ EAL: Selected IOVA mode 'VA'
+ EAL: Probing VFIO support...
+ EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+ EAL: Invalid NUMA socket, default to 0
+ EAL: probe driver: 8086:15d7 net_e1000_em
+ APP: HPET is not enabled, using TSC as default timer
+ RTE>>version_autotest
+ Version string: 'DPDK 20.02.0-rc0'
+ Test OK
+ RTE>>$
+
+The above shows running a specific test case. On success, the return
+code will be '0', otherwise it will be set to some error value (such
+as '255').
+
+
+Running all tests
+-----------------
+
+In order to allow developers to quickly execute all the standard
+internal tests without needing to remember or look up each test suite
+name, the build system includes a standard way of executing the
+default test suites. After building via `ninja`, the ``meson test``
+command will execute the standard tests and report errors.
+
+There are four groups of default test suites. The first group is
+the **fast** test suite, which is the largest group of test cases.
+These are the bulk of the unit tests to validate functional blocks.
+The second group is the **perf** tests. These test suites can take
+longer to run and do performance evaluations. The third group is
+the **driver** test suite, which is mostly for special hardware
+related testing (such as `cryptodev`). The last group are the
+**debug** tests. These mostly are used to dump system information.
+
+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command. Ex: ``meson test --suite fast-tests``
+
+
+Adding test suites
+------------------
+
+To add a testsuite to the DPDK test application, create a new test
+file for that suite (ex: see *app/test/test_version.c* for the
+``version_autotest`` test suite). There are two useful things:
+
+ 1. REGISTER_TEST_COMMAND(command_name, function_to_execute)
+ Registers a test command with the name `command_name` and which
+ runs the function `function_to_execute` when `command_name` is
+ invoked.
+
+ 2. unit_test_suite_runner(struct unit_test_suite \*)
+ Returns a runner for a full test suite object, which contains
+ a test suite name, setup, teardown, and vector of unit test
+ cases.
+
+Each test suite has a setup and teardown function that runs at the
+beginning and end of the test suite execution. Each unit test has
+a similar function for test case setup and teardown.
+
+Adding test cases is controlled via the `.unit_test_cases` element
+of the unit test suite. Ex:
+
+.. code-block:: c
+ :linenos:
+
+ #include <time.h>
+
+ #include <rte_common.h>
+ #include <rte_cycles.h>
+ #include <rte_hexdump.h>
+ #include <rte_random.h>
+
+ #include "test.h"
+
+ static int testsuite_setup(void) { return TEST_SUCCESS; }
+ static void testsuite_teardown(void) { }
+
+ static int ut_setup(void) { return TEST_SUCCESS; }
+ static void ut_teardown(void) { }
+
+ static int test_case_first(void) { return TEST_SUCCESS; }
+
+ static struct unit_test_suite example_testsuite = {
+ .suite_name = "EXAMPLE TEST SUITE",
+ .setup = testsuite_setup,
+ .teardown = testsuite_teardown,
+ .unit_test_cases = {
+ TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
+
+ TEST_CASES_END(), /**< NULL terminate unit test array */
+ },
+ };
+
+ static int example_tests()
+ {
+ return unit_test_suite_runner(&example_testsuite);
+ }
+
+ REGISTER_TEST_COMMAND(example_autotest, example_tests);
+
+The above code block is a small example that can be used to create a
+complete test suite with test case.
+
+
+Designing a test
+----------------
+
+Test cases have multiple ways of indicating an error has occurred,
+in order to reflect failure state back to the runner. Using the
+various methods of indicating errors can assist in not only validating
+the requisite functionality is working, but also to help debug when
+a change in environment or code has caused things to go wrong.
+
+The first way to indicate a generic error is by returning a test
+result failure, using the *TEST_FAILED* error code. This is the most
+basic way of indicating that an error has occurred in a test routine.
+It isn't very informative to the user, so it should really be used in
+cases where the test has catastrophically failed.
+
+The preferred method of indicating an error is via the
+`RTE_TEST_ASSERT` family of macros, which will immediately return
+*TEST_FAILED* error condition, but will also log details about the
+failure. The basic form is:
+
+.. code-block:: c
+
+ RTE_TEST_ASSERT(cond, msg, ...)
+
+In the above macro, *cond* is the condition to evaluate to **true**.
+Any generic condition can go here. The *msg* parameter will be a
+message to display if *cond* evaluates to **false**. Some specialized
+macros already exist. See `lib/librte_eal/include/rte_test.h` for
+a list of pre-build test assertions.
+
+Sometimes it is important to indicate that a test needs to be
+skipped, either because the environment isn't able to support running
+the test, or because some requisite functionality isn't availble. The
+test suite supports returning a result of `TEST_SKIPPED` during test
+case setup, or during test case execution to indicate that the
+preconditions of the test aren't available. A skipped test isn't the
+same as a failed test. Failed tests indicate that all required
+functionality was enabled, and the environment was correct but the
+test failed due to some error in logic. A skipped test indicates
+that the test setup wasn't available to run.
+
+
+Checking code coverage
+----------------------
+One of the best ways to help design a test case is to add a new test
+which exercises functionality that doesn't currently have a test case.
+The best way to find these is to run the test suite with some
+additional code coverage reporting turned on. The meson build system
+supports generating a code coverage report via the `-Db_coverage=true`
+option, in conjunction with a package like **lcov**, to generate an
+HTML code coverage report. Example::
+
+ $ meson covered -Db_coverage=true
+ $ meson test -C covered --suite fast-tests
+ $ ninja coverage-html -C covered
+
+The above will generate an html report in the
+`covered/meson-logs/coveragereport/` directory that can be explored
+for detailed code covered information. This can be used to assist
+in test development.
+
+
+Adding a suite to the default
+-----------------------------
+
+Adding to one of the default tests involves editing the appropriate
+meson build file `app/test/meson.build` and adding the command to
+the correct test suite class. Once added, the new test suite will
+be run as part of the appropriate class (fast, perf, driver, etc.).
+
+Some of these default test suites are run during continuous integration
+tests, making regression checking automatic for new patches submitted
+to the project.
--
2.25.4