[dpdk-dev] [PATCH] guides: add a testing guide for developing tests

[dpdk-dev] [PATCH] guides: add a testing guide for developing tests

[Aaron Conole]

The DPDK testing infrastructure includes a comprehensive set of
libraries, utilities, and CI integrations for developers to test
their code changes.  This isn't well documented, however.

Document the basics for adding a test suite to the infrastructure
and enabling that test suite for continuous integration platforms
so that newer developers can understand how to develop test suites
and test cases.

Signed-off-by: Aaron Conole <aconole@redhat.com>
---
v0: Added information for TEST_SKIPPED and details about generating
    code coverage to help with ideas for writing unit test cases.

 doc/guides/contributing/index.rst   |   1 +
 doc/guides/contributing/testing.rst | 231 ++++++++++++++++++++++++++++
 2 files changed, 232 insertions(+)
 create mode 100644 doc/guides/contributing/testing.rst

diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst
index 2fefd91931..41909d949b 100644
--- a/doc/guides/contributing/index.rst
+++ b/doc/guides/contributing/index.rst
@@ -14,6 +14,7 @@ Contributor's Guidelines
     abi_versioning
     documentation
     patches
+    testing
     vulnerability
     stable
     cheatsheet
diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
new file mode 100644
index 0000000000..3ade654772
--- /dev/null
+++ b/doc/guides/contributing/testing.rst
@@ -0,0 +1,231 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright 2018 The DPDK contributors
+
+.. _testing_guidelines:
+
+DPDK Testing Guidelines
+=======================
+
+This document outlines the guidelines for running and adding new
+tests to the in-tree DPDK test suites.
+
+The DPDK test suite model is loosely based on the xunit model, where
+tests are grouped into test suites, and suites are run by runners.
+For a basic overview, see the basic Wikipedia article on xunit:
+`xUnit - Wikipedia <https://en.wikipedia.org/wiki/XUnit>`_.
+
+
+Running a test
+--------------
+
+DPDK tests are run via the main test runniner, the `dpdk-test` app.
+The `dpdk-test` app is a command-line interface that facilitates
+running various tests or test suites.
+
+There are two modes of operation.  The first mode is as an interactive
+command shell that allows launching specific test suites.  This is
+the default operating mode of `dpdk-test` and can be done by::
+
+  $ ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>
+
+At the prompt, simply type the name of the test suite you wish to run
+and it will execute.
+
+The second form is useful for a scripting environment, and is used by
+the DPDK meson build system.  This mode is invoked by assigning a
+specific test suite name to the environment variable `DPDK_TEST`
+before invoking the `dpdk-test` command, such as::
+
+  $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --no-huge
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>version_autotest
+  Version string: 'DPDK 20.02.0-rc0'
+  Test OK
+  RTE>>$
+
+The above shows running a specific test case.  On success, the return
+code will be '0', otherwise it will be set to some error value (such
+as '255').
+
+
+Running all tests
+-----------------
+
+In order to allow developers to quickly execute all the standard
+internal tests without needing to remember or look up each test suite
+name, the build system includes a standard way of executing the
+default test suites.  After building via `ninja`, the ``meson test``
+command will execute the standard tests and report errors.
+
+There are four groups of default test suites.  The first group is
+the **fast** test suite, which is the largest group of test cases.
+These are the bulk of the unit tests to validate functional blocks.
+The second group is the **perf** tests.  These test suites can take
+longer to run and do performance evaluations.  The third group is
+the **driver** test suite, which is mostly for special hardware
+related testing (such as `cryptodev`).  The last group are the
+**debug** tests.  These mostly are used to dump system information.
+
+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command.  Ex: ``meson test --suite fast-tests``
+
+
+Adding test suites
+------------------
+
+To add a testsuite to the DPDK test application, create a new test
+file for that suite (ex: see *app/test/test_version.c* for the
+``version_autotest`` test suite).  There are two useful things:
+
+  1. REGISTER_TEST_COMMAND(command_name, function_to_execute)
+     Registers a test command with the name `command_name` and which
+     runs the function `function_to_execute` when `command_name` is
+     invoked.
+
+  2. unit_test_suite_runner(struct unit_test_suite \*)
+     Returns a runner for a full test suite object, which contains
+     a test suite name, setup, teardown, and vector of unit test
+     cases.
+
+Each test suite has a setup and teardown function that runs at the
+beginning and end of the test suite execution.  Each unit test has
+a similar function for test case setup and teardown.
+
+Adding test cases is controlled via the `.unit_test_cases` element
+of the unit test suite.  Ex:
+
+.. code-block:: c
+   :linenos:
+
+   #include <time.h>
+
+   #include <rte_common.h>
+   #include <rte_cycles.h>
+   #include <rte_hexdump.h>
+   #include <rte_random.h>
+
+   #include "test.h"
+
+   static int testsuite_setup(void) { return TEST_SUCCESS; }
+   static void testsuite_teardown(void) { }
+
+   static int ut_setup(void) { return TEST_SUCCESS; }
+   static void ut_teardown(void) { }
+
+   static int test_case_first(void) { return TEST_SUCCESS; }
+
+   static struct unit_test_suite example_testsuite = {
+          .suite_name = "EXAMPLE TEST SUITE",
+          .setup = testsuite_setup,
+          .teardown = testsuite_teardown,
+          .unit_test_cases = {
+               TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
+
+               TEST_CASES_END(), /**< NULL terminate unit test array */
+          },
+   };
+
+   static int example_tests()
+   {
+       return unit_test_suite_runner(&example_testsuite);
+   }
+
+   REGISTER_TEST_COMMAND(example_autotest, example_tests);
+
+The above code block is a small example that can be used to create a
+complete test suite with test case.
+
+
+Designing a test
+----------------
+
+Test cases have multiple ways of indicating an error has occurred,
+in order to reflect failure state back to the runner.  Using the
+various methods of indicating errors can assist in not only validating
+the requisite functionality is working, but also to help debug when
+a change in environment or code has caused things to go wrong.
+
+The first way to indicate a generic error is by returning a test
+result failure, using the *TEST_FAILED* error code.  This is the most
+basic way of indicating that an error has occurred in a test routine.
+It isn't very informative to the user, so it should really be used in
+cases where the test has catastrophically failed.
+
+The preferred method of indicating an error is via the
+`RTE_TEST_ASSERT` family of macros, which will immediately return
+*TEST_FAILED* error condition, but will also log details about the
+failure.  The basic form is:
+
+.. code-block:: c
+
+   RTE_TEST_ASSERT(cond, msg, ...)
+
+In the above macro, *cond* is the condition to evaluate to **true**.
+Any generic condition can go here.  The *msg* parameter will be a
+message to display if *cond* evaluates to **false**.  Some specialized
+macros already exist.  See `lib/librte_eal/include/rte_test.h` for
+a list of pre-build test assertions.
+
+Sometimes it is important to indicate that a test needs to be
+skipped, either because the environment isn't able to support running
+the test, or because some requisite functionality isn't availble.  The
+test suite supports returning a result of `TEST_SKIPPED` during test
+case setup, or during test case execution to indicate that the
+preconditions of the test aren't available.  A skipped test isn't the
+same as a failed test.  Failed tests indicate that all required
+functionality was enabled, and the environment was correct but the
+test failed due to some error in logic.  A skipped test indicates
+that the test setup wasn't available to run.
+
+
+Checking code coverage
+----------------------
+One of the best ways to help design a test case is to add a new test
+which exercises functionality that doesn't currently have a test case.
+The best way to find these is to run the test suite with some
+additional code coverage reporting turned on.  The meson build system
+supports generating a code coverage report via the `-Db_coverage=true`
+option, in conjunction with a package like **lcov**, to generate an
+HTML code coverage report.  Example::
+
+  $ meson covered -Db_coverage=true
+  $ meson test -C covered --suite fast-tests
+  $ ninja coverage-html -C covered
+
+The above will generate an html report in the
+`covered/meson-logs/coveragereport/` directory that can be explored
+for detailed code covered information.  This can be used to assist
+in test development.
+
+
+Adding a suite to the default
+-----------------------------
+
+Adding to one of the default tests involves editing the appropriate
+meson build file `app/test/meson.build` and adding the command to
+the correct test suite class.  Once added, the new test suite will
+be run as part of the appropriate class (fast, perf, driver, etc.).
+
+Some of these default test suites are run during continuous integration
+tests, making regression checking automatic for new patches submitted
+to the project.
-- 
2.25.4

Re: [dpdk-dev] [PATCH] guides: add a testing guide for developing tests

[Kinsella, Ray]

On 08/02/2021 16:52, Aaron Conole wrote:
> The DPDK testing infrastructure includes a comprehensive set of
> libraries, utilities, and CI integrations for developers to test
> their code changes.  This isn't well documented, however.
> 
> Document the basics for adding a test suite to the infrastructure
> and enabling that test suite for continuous integration platforms
> so that newer developers can understand how to develop test suites
> and test cases.
> 
> Signed-off-by: Aaron Conole <aconole@redhat.com>
> ---
> v0: Added information for TEST_SKIPPED and details about generating
>     code coverage to help with ideas for writing unit test cases.
> 
>  doc/guides/contributing/index.rst   |   1 +
>  doc/guides/contributing/testing.rst | 231 ++++++++++++++++++++++++++++
>  2 files changed, 232 insertions(+)
>  create mode 100644 doc/guides/contributing/testing.rst
> 
> diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst
> index 2fefd91931..41909d949b 100644
> --- a/doc/guides/contributing/index.rst
> +++ b/doc/guides/contributing/index.rst
> @@ -14,6 +14,7 @@ Contributor's Guidelines
>      abi_versioning
>      documentation
>      patches
> +    testing
>      vulnerability
>      stable
>      cheatsheet
> diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
> new file mode 100644
> index 0000000000..3ade654772
> --- /dev/null
> +++ b/doc/guides/contributing/testing.rst
> @@ -0,0 +1,231 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright 2018 The DPDK contributors
> +
> +.. _testing_guidelines:
> +
> +DPDK Testing Guidelines
> +=======================
> +
> +This document outlines the guidelines for running and adding new
> +tests to the in-tree DPDK test suites.
> +
> +The DPDK test suite model is loosely based on the xunit model, where
> +tests are grouped into test suites, and suites are run by runners.
> +For a basic overview, see the basic Wikipedia article on xunit:
> +`xUnit - Wikipedia <https://en.wikipedia.org/wiki/XUnit>`_.
> +
> +
> +Running a test
> +--------------
> +
> +DPDK tests are run via the main test runniner, the `dpdk-test` app.

runner

> +The `dpdk-test` app is a command-line interface that facilitates
> +running various tests or test suites.
> +
> +There are two modes of operation.  The first mode is as an interactive
> +command shell that allows launching specific test suites.  This is
> +the default operating mode of `dpdk-test` and can be done by::

is the double :: intentional?

> +
> +  $ ./build/app/test/dpdk-test --dpdk-options-here
> +  EAL: Detected 4 lcore(s)
> +  EAL: Detected 1 NUMA nodes
> +  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
> +  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
> +  EAL: Selected IOVA mode 'VA'
> +  EAL: Probing VFIO support...
> +  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
> +  EAL:   Invalid NUMA socket, default to 0
> +  EAL:   probe driver: 8086:15d7 net_e1000_em
> +  APP: HPET is not enabled, using TSC as default timer
> +  RTE>>
> +
> +At the prompt, simply type the name of the test suite you wish to run
> +and it will execute.
> +
> +The second form is useful for a scripting environment, and is used by
> +the DPDK meson build system.  This mode is invoked by assigning a
> +specific test suite name to the environment variable `DPDK_TEST`
> +before invoking the `dpdk-test` command, such as::
> +
> +  $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --no-huge

better to say --dpdk-options-here instead of --no-huge, as above. 

> +  EAL: Detected 4 lcore(s)
> +  EAL: Detected 1 NUMA nodes
> +  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
> +  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
> +  EAL: Selected IOVA mode 'VA'
> +  EAL: Probing VFIO support...
> +  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
> +  EAL:   Invalid NUMA socket, default to 0
> +  EAL:   probe driver: 8086:15d7 net_e1000_em
> +  APP: HPET is not enabled, using TSC as default timer
> +  RTE>>version_autotest
> +  Version string: 'DPDK 20.02.0-rc0'
> +  Test OK
> +  RTE>>$
> +
> +The above shows running a specific test case.  On success, the return
> +code will be '0', otherwise it will be set to some error value (such
> +as '255').
> +
> +
> +Running all tests
> +-----------------
> +
> +In order to allow developers to quickly execute all the standard
> +internal tests without needing to remember or look up each test suite
> +name, the build system includes a standard way of executing the
> +default test suites.  After building via `ninja`, the ``meson test``
> +command will execute the standard tests and report errors.
> +
> +There are four groups of default test suites.  The first group is
> +the **fast** test suite, which is the largest group of test cases.
> +These are the bulk of the unit tests to validate functional blocks.
> +The second group is the **perf** tests.  These test suites can take
> +longer to run and do performance evaluations.  The third group is
> +the **driver** test suite, which is mostly for special hardware
> +related testing (such as `cryptodev`).  The last group are the
> +**debug** tests.  These mostly are used to dump system information.
> +
> +The suites can be selected by adding the ``--suite`` option to the
> +``meson test`` command.  Ex: ``meson test --suite fast-tests``

show some console output in this section?

> +
> +
> +Adding test suites
> +------------------
> +
> +To add a testsuite to the DPDK test application, create a new test
> +file for that suite (ex: see *app/test/test_version.c* for the
> +``version_autotest`` test suite).  There are two useful things:

There are two useful things?

> +
> +  1. REGISTER_TEST_COMMAND(command_name, function_to_execute)
> +     Registers a test command with the name `command_name` and which
> +     runs the function `function_to_execute` when `command_name` is
> +     invoked.
> +
> +  2. unit_test_suite_runner(struct unit_test_suite \*)
> +     Returns a runner for a full test suite object, which contains
> +     a test suite name, setup, teardown, and vector of unit test
> +     cases.
> +
> +Each test suite has a setup and teardown function that runs at the
> +beginning and end of the test suite execution.  Each unit test has
> +a similar function for test case setup and teardown.
> +
> +Adding test cases is controlled via the `.unit_test_cases` element

Test cases are added to the .unit_test_cases element of the 
of the unit test suite structure. 

> +of the unit test suite.  Ex:
> +
> +.. code-block:: c
> +   :linenos:
> +
> +   #include <time.h>
> +
> +   #include <rte_common.h>
> +   #include <rte_cycles.h>
> +   #include <rte_hexdump.h>
> +   #include <rte_random.h>
> +
> +   #include "test.h"
> +
> +   static int testsuite_setup(void) { return TEST_SUCCESS; }
> +   static void testsuite_teardown(void) { }
> +
> +   static int ut_setup(void) { return TEST_SUCCESS; }
> +   static void ut_teardown(void) { }
> +
> +   static int test_case_first(void) { return TEST_SUCCESS; }
> +
> +   static struct unit_test_suite example_testsuite = {
> +          .suite_name = "EXAMPLE TEST SUITE",
> +          .setup = testsuite_setup,
> +          .teardown = testsuite_teardown,
> +          .unit_test_cases = {
> +               TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
> +
> +               TEST_CASES_END(), /**< NULL terminate unit test array */
> +          },
> +   };
> +
> +   static int example_tests()
> +   {
> +       return unit_test_suite_runner(&example_testsuite);
> +   }
> +
> +   REGISTER_TEST_COMMAND(example_autotest, example_tests);
> +
> +The above code block is a small example that can be used to create a
> +complete test suite with test case.
> +
> +
> +Designing a test
> +----------------
> +
> +Test cases have multiple ways of indicating an error has occurred,
> +in order to reflect failure state back to the runner.  Using the
> +various methods of indicating errors can assist in not only validating
> +the requisite functionality is working, but also to help debug when
> +a change in environment or code has caused things to go wrong.
> +
> +The first way to indicate a generic error is by returning a test
> +result failure, using the *TEST_FAILED* error code.  This is the most
> +basic way of indicating that an error has occurred in a test routine.
> +It isn't very informative to the user, so it should really be used in
> +cases where the test has catastrophically failed.
> +
> +The preferred method of indicating an error is via the
> +`RTE_TEST_ASSERT` family of macros, which will immediately return
> +*TEST_FAILED* error condition, but will also log details about the
> +failure.  The basic form is:
> +
> +.. code-block:: c
> +
> +   RTE_TEST_ASSERT(cond, msg, ...)
> +
> +In the above macro, *cond* is the condition to evaluate to **true**.
> +Any generic condition can go here.  The *msg* parameter will be a
> +message to display if *cond* evaluates to **false**.  Some specialized
> +macros already exist.  See `lib/librte_eal/include/rte_test.h` for
> +a list of pre-build test assertions.
> +
> +Sometimes it is important to indicate that a test needs to be
> +skipped, either because the environment isn't able to support running
> +the test, or because some requisite functionality isn't availble.  The

available

> +test suite supports returning a result of `TEST_SKIPPED` during test
> +case setup, or during test case execution to indicate that the
> +preconditions of the test aren't available.  A skipped test isn't the
> +same as a failed test.  Failed tests indicate that all required
> +functionality was enabled, and the environment was correct but the
> +test failed due to some error in logic.  A skipped test indicates
> +that the test setup wasn't available to run.

Simpler to provide an example of a skipped test, than a long explanation?

> +
> +
> +Checking code coverage
> +----------------------
> +One of the best ways to help design a test case is to add a new test
> +which exercises functionality that doesn't currently have a test case.
> +The best way to find these is to run the test suite with some
> +additional code coverage reporting turned on.  

I would drop the previous sentences.
They would just confuse a reader looking to generate code coverage.

> The meson build system
> +supports generating a code coverage report via the `-Db_coverage=true`
> +option, in conjunction with a package like **lcov**, to generate an
> +HTML code coverage report.  Example::
> +
> +  $ meson covered -Db_coverage=true
> +  $ meson test -C covered --suite fast-tests
> +  $ ninja coverage-html -C covered
> +
> +The above will generate an html report in the
> +`covered/meson-logs/coveragereport/` directory that can be explored
> +for detailed code covered information.  This can be used to assist
> +in test development.
> +
> +
> +Adding a suite to the default
> +-----------------------------
> +
> +Adding to one of the default tests involves editing the appropriate
> +meson build file `app/test/meson.build` and adding the command to
> +the correct test suite class.  Once added, the new test suite will
> +be run as part of the appropriate class (fast, perf, driver, etc.).
> +
> +Some of these default test suites are run during continuous integration
> +tests, making regression checking automatic for new patches submitted
> +to the project.
> 

Re: [dpdk-dev] [PATCH] guides: add a testing guide for developing tests

[Aaron Conole]

"Kinsella, Ray" <mdr@ashroe.eu> writes:

> On 08/02/2021 16:52, Aaron Conole wrote:
>> The DPDK testing infrastructure includes a comprehensive set of
>> libraries, utilities, and CI integrations for developers to test
>> their code changes.  This isn't well documented, however.
>> 
>> Document the basics for adding a test suite to the infrastructure
>> and enabling that test suite for continuous integration platforms
>> so that newer developers can understand how to develop test suites
>> and test cases.
>> 
>> Signed-off-by: Aaron Conole <aconole@redhat.com>
>> ---
>> v0: Added information for TEST_SKIPPED and details about generating
>>     code coverage to help with ideas for writing unit test cases.
>> 
>>  doc/guides/contributing/index.rst   |   1 +
>>  doc/guides/contributing/testing.rst | 231 ++++++++++++++++++++++++++++
>>  2 files changed, 232 insertions(+)
>>  create mode 100644 doc/guides/contributing/testing.rst
>> 
>> diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst
>> index 2fefd91931..41909d949b 100644
>> --- a/doc/guides/contributing/index.rst
>> +++ b/doc/guides/contributing/index.rst
>> @@ -14,6 +14,7 @@ Contributor's Guidelines
>>      abi_versioning
>>      documentation
>>      patches
>> +    testing
>>      vulnerability
>>      stable
>>      cheatsheet
>> diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
>> new file mode 100644
>> index 0000000000..3ade654772
>> --- /dev/null
>> +++ b/doc/guides/contributing/testing.rst
>> @@ -0,0 +1,231 @@
>> +..  SPDX-License-Identifier: BSD-3-Clause
>> +    Copyright 2018 The DPDK contributors
>> +
>> +.. _testing_guidelines:
>> +
>> +DPDK Testing Guidelines
>> +=======================
>> +
>> +This document outlines the guidelines for running and adding new
>> +tests to the in-tree DPDK test suites.
>> +
>> +The DPDK test suite model is loosely based on the xunit model, where
>> +tests are grouped into test suites, and suites are run by runners.
>> +For a basic overview, see the basic Wikipedia article on xunit:
>> +`xUnit - Wikipedia <https://en.wikipedia.org/wiki/XUnit>`_.
>> +
>> +
>> +Running a test
>> +--------------
>> +
>> +DPDK tests are run via the main test runniner, the `dpdk-test` app.
>
> runner

D'oh!

>> +The `dpdk-test` app is a command-line interface that facilitates
>> +running various tests or test suites.
>> +
>> +There are two modes of operation.  The first mode is as an interactive
>> +command shell that allows launching specific test suites.  This is
>> +the default operating mode of `dpdk-test` and can be done by::
>
> is the double :: intentional?

Yes - it designates the next block as a code block.

>> +
>> +  $ ./build/app/test/dpdk-test --dpdk-options-here
>> +  EAL: Detected 4 lcore(s)
>> +  EAL: Detected 1 NUMA nodes
>> + EAL: Static memory layout is selected, amount of reserved memory
>> can be adjusted with -m or --socket-mem
>> +  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
>> +  EAL: Selected IOVA mode 'VA'
>> +  EAL: Probing VFIO support...
>> +  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
>> +  EAL:   Invalid NUMA socket, default to 0
>> +  EAL:   probe driver: 8086:15d7 net_e1000_em
>> +  APP: HPET is not enabled, using TSC as default timer
>> +  RTE>>
>> +
>> +At the prompt, simply type the name of the test suite you wish to run
>> +and it will execute.
>> +
>> +The second form is useful for a scripting environment, and is used by
>> +the DPDK meson build system.  This mode is invoked by assigning a
>> +specific test suite name to the environment variable `DPDK_TEST`
>> +before invoking the `dpdk-test` command, such as::
>> +
>> +  $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --no-huge
>
> better to say --dpdk-options-here instead of --no-huge, as above. 

Okay, I can do that.

>> +  EAL: Detected 4 lcore(s)
>> +  EAL: Detected 1 NUMA nodes
>> + EAL: Static memory layout is selected, amount of reserved memory
>> can be adjusted with -m or --socket-mem
>> +  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
>> +  EAL: Selected IOVA mode 'VA'
>> +  EAL: Probing VFIO support...
>> +  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
>> +  EAL:   Invalid NUMA socket, default to 0
>> +  EAL:   probe driver: 8086:15d7 net_e1000_em
>> +  APP: HPET is not enabled, using TSC as default timer
>> +  RTE>>version_autotest
>> +  Version string: 'DPDK 20.02.0-rc0'
>> +  Test OK
>> +  RTE>>$
>> +
>> +The above shows running a specific test case.  On success, the return
>> +code will be '0', otherwise it will be set to some error value (such
>> +as '255').
>> +
>> +
>> +Running all tests
>> +-----------------
>> +
>> +In order to allow developers to quickly execute all the standard
>> +internal tests without needing to remember or look up each test suite
>> +name, the build system includes a standard way of executing the
>> +default test suites.  After building via `ninja`, the ``meson test``
>> +command will execute the standard tests and report errors.
>> +
>> +There are four groups of default test suites.  The first group is
>> +the **fast** test suite, which is the largest group of test cases.
>> +These are the bulk of the unit tests to validate functional blocks.
>> +The second group is the **perf** tests.  These test suites can take
>> +longer to run and do performance evaluations.  The third group is
>> +the **driver** test suite, which is mostly for special hardware
>> +related testing (such as `cryptodev`).  The last group are the
>> +**debug** tests.  These mostly are used to dump system information.
>> +
>> +The suites can be selected by adding the ``--suite`` option to the
>> +``meson test`` command.  Ex: ``meson test --suite fast-tests``
>
> show some console output in this section?

Okay.

>> +
>> +
>> +Adding test suites
>> +------------------
>> +
>> +To add a testsuite to the DPDK test application, create a new test
>> +file for that suite (ex: see *app/test/test_version.c* for the
>> +``version_autotest`` test suite).  There are two useful things:
>
> There are two useful things?

Okay, I will rephrase because it sounds awkward.  But I don't think it's
wrong ;)

>> +
>> +  1. REGISTER_TEST_COMMAND(command_name, function_to_execute)
>> +     Registers a test command with the name `command_name` and which
>> +     runs the function `function_to_execute` when `command_name` is
>> +     invoked.
>> +
>> +  2. unit_test_suite_runner(struct unit_test_suite \*)
>> +     Returns a runner for a full test suite object, which contains
>> +     a test suite name, setup, teardown, and vector of unit test
>> +     cases.
>> +
>> +Each test suite has a setup and teardown function that runs at the
>> +beginning and end of the test suite execution.  Each unit test has
>> +a similar function for test case setup and teardown.
>> +
>> +Adding test cases is controlled via the `.unit_test_cases` element
>
> Test cases are added to the .unit_test_cases element of the 
> of the unit test suite structure. 

Will rephrase.

>> +of the unit test suite.  Ex:
>> +
>> +.. code-block:: c
>> +   :linenos:
>> +
>> +   #include <time.h>
>> +
>> +   #include <rte_common.h>
>> +   #include <rte_cycles.h>
>> +   #include <rte_hexdump.h>
>> +   #include <rte_random.h>
>> +
>> +   #include "test.h"
>> +
>> +   static int testsuite_setup(void) { return TEST_SUCCESS; }
>> +   static void testsuite_teardown(void) { }
>> +
>> +   static int ut_setup(void) { return TEST_SUCCESS; }
>> +   static void ut_teardown(void) { }
>> +
>> +   static int test_case_first(void) { return TEST_SUCCESS; }
>> +
>> +   static struct unit_test_suite example_testsuite = {
>> +          .suite_name = "EXAMPLE TEST SUITE",
>> +          .setup = testsuite_setup,
>> +          .teardown = testsuite_teardown,
>> +          .unit_test_cases = {
>> +               TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
>> +
>> +               TEST_CASES_END(), /**< NULL terminate unit test array */
>> +          },
>> +   };
>> +
>> +   static int example_tests()
>> +   {
>> +       return unit_test_suite_runner(&example_testsuite);
>> +   }
>> +
>> +   REGISTER_TEST_COMMAND(example_autotest, example_tests);
>> +
>> +The above code block is a small example that can be used to create a
>> +complete test suite with test case.
>> +
>> +
>> +Designing a test
>> +----------------
>> +
>> +Test cases have multiple ways of indicating an error has occurred,
>> +in order to reflect failure state back to the runner.  Using the
>> +various methods of indicating errors can assist in not only validating
>> +the requisite functionality is working, but also to help debug when
>> +a change in environment or code has caused things to go wrong.
>> +
>> +The first way to indicate a generic error is by returning a test
>> +result failure, using the *TEST_FAILED* error code.  This is the most
>> +basic way of indicating that an error has occurred in a test routine.
>> +It isn't very informative to the user, so it should really be used in
>> +cases where the test has catastrophically failed.
>> +
>> +The preferred method of indicating an error is via the
>> +`RTE_TEST_ASSERT` family of macros, which will immediately return
>> +*TEST_FAILED* error condition, but will also log details about the
>> +failure.  The basic form is:
>> +
>> +.. code-block:: c
>> +
>> +   RTE_TEST_ASSERT(cond, msg, ...)
>> +
>> +In the above macro, *cond* is the condition to evaluate to **true**.
>> +Any generic condition can go here.  The *msg* parameter will be a
>> +message to display if *cond* evaluates to **false**.  Some specialized
>> +macros already exist.  See `lib/librte_eal/include/rte_test.h` for
>> +a list of pre-build test assertions.
>> +
>> +Sometimes it is important to indicate that a test needs to be
>> +skipped, either because the environment isn't able to support running
>> +the test, or because some requisite functionality isn't availble.  The
>
> available

D'oh!

>> +test suite supports returning a result of `TEST_SKIPPED` during test
>> +case setup, or during test case execution to indicate that the
>> +preconditions of the test aren't available.  A skipped test isn't the
>> +same as a failed test.  Failed tests indicate that all required
>> +functionality was enabled, and the environment was correct but the
>> +test failed due to some error in logic.  A skipped test indicates
>> +that the test setup wasn't available to run.
>
> Simpler to provide an example of a skipped test, than a long explanation?

I can do both.

>> +
>> +
>> +Checking code coverage
>> +----------------------
>> +One of the best ways to help design a test case is to add a new test
>> +which exercises functionality that doesn't currently have a test case.
>> +The best way to find these is to run the test suite with some
>> +additional code coverage reporting turned on.  
>
> I would drop the previous sentences.
> They would just confuse a reader looking to generate code coverage.

Okay.  I felt it was a bit strange to introduce code coverage here, and
I thought it was a needed bridge.  I can drop it.

>> The meson build system
>> +supports generating a code coverage report via the `-Db_coverage=true`
>> +option, in conjunction with a package like **lcov**, to generate an
>> +HTML code coverage report.  Example::
>> +
>> +  $ meson covered -Db_coverage=true
>> +  $ meson test -C covered --suite fast-tests
>> +  $ ninja coverage-html -C covered
>> +
>> +The above will generate an html report in the
>> +`covered/meson-logs/coveragereport/` directory that can be explored
>> +for detailed code covered information.  This can be used to assist
>> +in test development.
>> +
>> +
>> +Adding a suite to the default
>> +-----------------------------
>> +
>> +Adding to one of the default tests involves editing the appropriate
>> +meson build file `app/test/meson.build` and adding the command to
>> +the correct test suite class.  Once added, the new test suite will
>> +be run as part of the appropriate class (fast, perf, driver, etc.).
>> +
>> +Some of these default test suites are run during continuous integration
>> +tests, making regression checking automatic for new patches submitted
>> +to the project.
>> 

[dpdk-dev] [PATCH v2] guides: add a testing guide for developing tests

[Aaron Conole]

The DPDK testing infrastructure includes a comprehensive set of
libraries, utilities, and CI integrations for developers to test
their code changes.  This isn't well documented, however.

Document the basics for adding a test suite to the infrastructure
and enabling that test suite for continuous integration platforms
so that newer developers can understand how to develop test suites
and test cases.

Signed-off-by: Aaron Conole <aconole@redhat.com>
---
v0->v1: Added information for TEST_SKIPPED and details about generating
        code coverage to help with ideas for writing unit test cases.
v1->v2: Corrected some spelling, rephrased a bit after suggestions by
        Ray.

 doc/guides/contributing/index.rst   |   1 +
 doc/guides/contributing/testing.rst | 245 ++++++++++++++++++++++++++++
 2 files changed, 246 insertions(+)
 create mode 100644 doc/guides/contributing/testing.rst

diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst
index 2fefd91931..41909d949b 100644
--- a/doc/guides/contributing/index.rst
+++ b/doc/guides/contributing/index.rst
@@ -14,6 +14,7 @@ Contributor's Guidelines
     abi_versioning
     documentation
     patches
+    testing
     vulnerability
     stable
     cheatsheet
diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
new file mode 100644
index 0000000000..86ca24ce43
--- /dev/null
+++ b/doc/guides/contributing/testing.rst
@@ -0,0 +1,245 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright 2018 The DPDK contributors
+
+.. _testing_guidelines:
+
+DPDK Testing Guidelines
+=======================
+
+This document outlines the guidelines for running and adding new
+tests to the in-tree DPDK test suites.
+
+The DPDK test suite model is loosely based on the xunit model, where
+tests are grouped into test suites, and suites are run by runners.
+For a basic overview, see the basic Wikipedia article on xunit:
+`xUnit - Wikipedia <https://en.wikipedia.org/wiki/XUnit>`_.
+
+
+Running a test
+--------------
+
+DPDK tests are run via the main test runner, the `dpdk-test` app.
+The `dpdk-test` app is a command-line interface that facilitates
+running various tests or test suites.
+
+There are two modes of operation.  The first mode is as an interactive
+command shell that allows launching specific test suites.  This is
+the default operating mode of `dpdk-test` and can be done by::
+
+  $ ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>
+
+At the prompt, simply type the name of the test suite you wish to run
+and it will execute.
+
+The second form is useful for a scripting environment, and is used by
+the DPDK meson build system.  This mode is invoked by assigning a
+specific test suite name to the environment variable `DPDK_TEST`
+before invoking the `dpdk-test` command, such as::
+
+  $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>version_autotest
+  Version string: 'DPDK 20.02.0-rc0'
+  Test OK
+  RTE>>$
+
+The above shows running a specific test case.  On success, the return
+code will be '0', otherwise it will be set to some error value (such
+as '255').
+
+
+Running all tests
+-----------------
+
+In order to allow developers to quickly execute all the standard
+internal tests without needing to remember or look up each test suite
+name, the build system includes a standard way of executing the
+default test suites.  After building via `ninja`, the ``meson test``
+command will execute the standard tests and report errors.
+
+There are four groups of default test suites.  The first group is
+the **fast** test suite, which is the largest group of test cases.
+These are the bulk of the unit tests to validate functional blocks.
+The second group is the **perf** tests.  These test suites can take
+longer to run and do performance evaluations.  The third group is
+the **driver** test suite, which is mostly for special hardware
+related testing (such as `cryptodev`).  The last group are the
+**debug** tests.  These mostly are used to dump system information.
+
+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command.  Ex: ``meson test --suite fast-tests``::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
+  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
+  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
+  ...
+
+
+Adding test suites
+------------------
+
+To add a test suite to the DPDK test application, create a new test
+file for that suite (ex: see *app/test/test_version.c* for the
+``version_autotest`` test suite).  There are two important functions
+for interacting with the test harness:
+
+  1. REGISTER_TEST_COMMAND(command_name, function_to_execute)
+     Registers a test command with the name `command_name` and which
+     runs the function `function_to_execute` when `command_name` is
+     invoked.
+
+  2. unit_test_suite_runner(struct unit_test_suite \*)
+     Returns a runner for a full test suite object, which contains
+     a test suite name, setup, tear down, and vector of unit test
+     cases.
+
+Each test suite has a setup and tear down function that runs at the
+beginning and end of the test suite execution.  Each unit test has
+a similar function for test case setup and tear down.
+
+Test cases are added to the `.unit_test_cases` element of the unit
+test suite structure.  Ex:
+
+.. code-block:: c
+   :linenos:
+
+   #include <time.h>
+
+   #include <rte_common.h>
+   #include <rte_cycles.h>
+   #include <rte_hexdump.h>
+   #include <rte_random.h>
+
+   #include "test.h"
+
+   static int testsuite_setup(void) { return TEST_SUCCESS; }
+   static void testsuite_teardown(void) { }
+
+   static int ut_setup(void) { return TEST_SUCCESS; }
+   static void ut_teardown(void) { }
+
+   static int test_case_first(void) { return TEST_SUCCESS; }
+
+   static struct unit_test_suite example_testsuite = {
+          .suite_name = "EXAMPLE TEST SUITE",
+          .setup = testsuite_setup,
+          .teardown = testsuite_teardown,
+          .unit_test_cases = {
+               TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
+
+               TEST_CASES_END(), /**< NULL terminate unit test array */
+          },
+   };
+
+   static int example_tests()
+   {
+       return unit_test_suite_runner(&example_testsuite);
+   }
+
+   REGISTER_TEST_COMMAND(example_autotest, example_tests);
+
+The above code block is a small example that can be used to create a
+complete test suite with test case.
+
+
+Designing a test
+----------------
+
+Test cases have multiple ways of indicating an error has occurred,
+in order to reflect failure state back to the runner.  Using the
+various methods of indicating errors can assist in not only validating
+the requisite functionality is working, but also to help debug when
+a change in environment or code has caused things to go wrong.
+
+The first way to indicate a generic error is by returning a test
+result failure, using the *TEST_FAILED* error code.  This is the most
+basic way of indicating that an error has occurred in a test routine.
+It isn't very informative to the user, so it should really be used in
+cases where the test has catastrophically failed.
+
+The preferred method of indicating an error is via the
+`RTE_TEST_ASSERT` family of macros, which will immediately return
+*TEST_FAILED* error condition, but will also log details about the
+failure.  The basic form is:
+
+.. code-block:: c
+
+   RTE_TEST_ASSERT(cond, msg, ...)
+
+In the above macro, *cond* is the condition to evaluate to **true**.
+Any generic condition can go here.  The *msg* parameter will be a
+message to display if *cond* evaluates to **false**.  Some specialized
+macros already exist.  See `lib/librte_eal/include/rte_test.h` for
+a list of defined test assertions.
+
+Sometimes it is important to indicate that a test needs to be
+skipped, either because the environment isn't able to support running
+the test, or because some requisite functionality isn't available.  The
+test suite supports returning a result of `TEST_SKIPPED` during test
+case setup, or during test case execution to indicate that the
+preconditions of the test aren't available.  Ex::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  ...
+  46/60 DPDK:fast-tests / ipsec_autotest       SKIP     0.22 s
+  ...
+
+
+Checking code coverage
+----------------------
+The meson build system supports generating a code coverage report
+via the `-Db_coverage=true` option, in conjunction with a package
+like **lcov**, to generate an HTML code coverage report.  Example::
+
+  $ meson covered -Db_coverage=true
+  $ meson test -C covered --suite fast-tests
+  $ ninja coverage-html -C covered
+
+The above will generate an html report in the
+`covered/meson-logs/coveragereport/` directory that can be explored
+for detailed code covered information.  This can be used to assist
+in test development.
+
+
+Adding a suite to the default
+-----------------------------
+
+Adding to one of the default tests involves editing the appropriate
+meson build file `app/test/meson.build` and adding the command to
+the correct test suite class.  Once added, the new test suite will
+be run as part of the appropriate class (fast, perf, driver, etc.).
+
+Some of these default test suites are run during continuous integration
+tests, making regression checking automatic for new patches submitted
+to the project.
-- 
2.25.4

Re: [dpdk-dev] [PATCH v2] guides: add a testing guide for developing tests

[David Marchand]

On Wed, Feb 10, 2021 at 3:56 PM Aaron Conole <aconole@redhat.com> wrote:
> diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
> new file mode 100644
> index 0000000000..86ca24ce43
> --- /dev/null
> +++ b/doc/guides/contributing/testing.rst
> @@ -0,0 +1,245 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright 2018 The DPDK contributors

2021?

> +
> +.. _testing_guidelines:

I can't find a call to the testing_guidelines reference, so this can be removed.


[snip]

> +The suites can be selected by adding the ``--suite`` option to the
> +``meson test`` command.  Ex: ``meson test --suite fast-tests``::
> +
> +  $ meson test -C build --suite fast-tests
> +  ninja: Entering directory `/home/aconole/git/dpdk/build'
> +  [2543/2543] Linking target app/test/dpdk-test.
> +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
> +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
> +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
> +  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
> +  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
> +  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
> +  ...

Trying this in my build env, I get all tests failing.
This is because I run this as a normal user, but the system has
hugepages configured.
I figured this out quickly since I know the test framework (simply
added a echo 0; exit at the top of has-hugepages.sh).
But I am not sure a reader of this doc would be able to troubleshoot this.

Not sure if this is worth explaining here, or if we can enhance the
hugepage check (permissions maybe?).


[snip]

> +Checking code coverage
> +----------------------
> +The meson build system supports generating a code coverage report
> +via the `-Db_coverage=true` option, in conjunction with a package
> +like **lcov**, to generate an HTML code coverage report.  Example::
> +
> +  $ meson covered -Db_coverage=true

At first, I read "covered" as a meson command :-).
I prefer an explicit "meson setup covered -Db_coverage=true", but well
this is more a matter of taste.


> +  $ meson test -C covered --suite fast-tests
> +  $ ninja coverage-html -C covered
> +
> +The above will generate an html report in the
> +`covered/meson-logs/coveragereport/` directory that can be explored
> +for detailed code covered information.  This can be used to assist
> +in test development.
> +


The rest of the doc lgtm, and this is a good addition, feel free to
add my review tag.
Thanks Aaron.

-- 
David Marchand

Re: [dpdk-dev] [PATCH v2] guides: add a testing guide for developing tests

[Bruce Richardson]

On Tue, Mar 02, 2021 at 10:07:26AM +0100, David Marchand wrote:
> On Wed, Feb 10, 2021 at 3:56 PM Aaron Conole <aconole@redhat.com> wrote:
> > diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
> > new file mode 100644
> > index 0000000000..86ca24ce43
> > --- /dev/null
> > +++ b/doc/guides/contributing/testing.rst
> > @@ -0,0 +1,245 @@
> > +..  SPDX-License-Identifier: BSD-3-Clause
> > +    Copyright 2018 The DPDK contributors
> 
> 2021?
> 
> > +
> > +.. _testing_guidelines:
> 
> I can't find a call to the testing_guidelines reference, so this can be removed.
> 
> 
> [snip]
> 
> > +The suites can be selected by adding the ``--suite`` option to the
> > +``meson test`` command.  Ex: ``meson test --suite fast-tests``::
> > +
> > +  $ meson test -C build --suite fast-tests
> > +  ninja: Entering directory `/home/aconole/git/dpdk/build'
> > +  [2543/2543] Linking target app/test/dpdk-test.
> > +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
> > +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
> > +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
> > +  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
> > +  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
> > +  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
> > +  ...
> 
> Trying this in my build env, I get all tests failing.
> This is because I run this as a normal user, but the system has
> hugepages configured.
> I figured this out quickly since I know the test framework (simply
> added a echo 0; exit at the top of has-hugepages.sh).
> But I am not sure a reader of this doc would be able to troubleshoot this.
> 
> Not sure if this is worth explaining here, or if we can enhance the
> hugepage check (permissions maybe?).
> 
> 
> [snip]
> 
> > +Checking code coverage
> > +----------------------
> > +The meson build system supports generating a code coverage report
> > +via the `-Db_coverage=true` option, in conjunction with a package
> > +like **lcov**, to generate an HTML code coverage report.  Example::
> > +
> > +  $ meson covered -Db_coverage=true
> 
> At first, I read "covered" as a meson command :-).
> I prefer an explicit "meson setup covered -Db_coverage=true", but well
> this is more a matter of taste.
> 
>

I also tend to prefer the build directory name at the end of the command,
so I'd suggest: "meson setup -Db_coverage=true covered".  Furthermore,
while I can understand the use of "covered" as a build directory name, I
think for consistency across all docs, we should just use "build" here as
the directory name, which again will reduce confusion.  "meson setup
-Db_coverage build"

/Bruce 

Re: [dpdk-dev] [PATCH v2] guides: add a testing guide for developing tests

[Aaron Conole]

Bruce Richardson <bruce.richardson@intel.com> writes:

> On Tue, Mar 02, 2021 at 10:07:26AM +0100, David Marchand wrote:
>> On Wed, Feb 10, 2021 at 3:56 PM Aaron Conole <aconole@redhat.com> wrote:
>> > diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
>> > new file mode 100644
>> > index 0000000000..86ca24ce43
>> > --- /dev/null
>> > +++ b/doc/guides/contributing/testing.rst
>> > @@ -0,0 +1,245 @@
>> > +..  SPDX-License-Identifier: BSD-3-Clause
>> > +    Copyright 2018 The DPDK contributors
>> 
>> 2021?

Whoops, I forgot to update my time machine.

>> > +
>> > +.. _testing_guidelines:
>> 
>> I can't find a call to the testing_guidelines reference, so this can be removed.

done.

>> 
>> [snip]
>> 
>> > +The suites can be selected by adding the ``--suite`` option to the
>> > +``meson test`` command.  Ex: ``meson test --suite fast-tests``::
>> > +
>> > +  $ meson test -C build --suite fast-tests
>> > +  ninja: Entering directory `/home/aconole/git/dpdk/build'
>> > +  [2543/2543] Linking target app/test/dpdk-test.
>> > +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
>> > +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
>> > +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
>> > +  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
>> > +  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
>> > +  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
>> > +  ...
>> 
>> Trying this in my build env, I get all tests failing.
>> This is because I run this as a normal user, but the system has
>> hugepages configured.
>> I figured this out quickly since I know the test framework (simply
>> added a echo 0; exit at the top of has-hugepages.sh).
>> But I am not sure a reader of this doc would be able to troubleshoot this.
>> 
>> Not sure if this is worth explaining here, or if we can enhance the
>> hugepage check (permissions maybe?).

I prefer to fix the hugepage check to make the tests SKIP when we don't
have hugepages accessible (so we need some kind of permission check in
there).  I will submit it separately.

>> 
>> [snip]
>> 
>> > +Checking code coverage
>> > +----------------------
>> > +The meson build system supports generating a code coverage report
>> > +via the `-Db_coverage=true` option, in conjunction with a package
>> > +like **lcov**, to generate an HTML code coverage report.  Example::
>> > +
>> > +  $ meson covered -Db_coverage=true
>> 
>> At first, I read "covered" as a meson command :-).
>> I prefer an explicit "meson setup covered -Db_coverage=true", but well
>> this is more a matter of taste.
>> 
>>
>
> I also tend to prefer the build directory name at the end of the command,
> so I'd suggest: "meson setup -Db_coverage=true covered".  Furthermore,
> while I can understand the use of "covered" as a build directory name, I
> think for consistency across all docs, we should just use "build" here as
> the directory name, which again will reduce confusion.  "meson setup
> -Db_coverage build"

Okay - I will paint this bikeshed like:

   meson setup build -Db_coverage=true

It's a little inconsistent everywhere - so I guess a good janitor
project would be to clean up all the places we have meson commands.

Otherwise, what I've found is that the options generally come after the
build directory / command is specified (ex: see octeontx, the arm64
cross build docs, etc.) so I'll keep that for consistency there.
Hopefully we will consistently become more consistent :)

> /Bruce 

Re: [dpdk-dev] [PATCH v2] guides: add a testing guide for developing tests

[Bruce Richardson]

On Tue, Mar 02, 2021 at 10:26:59AM -0500, Aaron Conole wrote:
> Bruce Richardson <bruce.richardson@intel.com> writes:
> 
> > On Tue, Mar 02, 2021 at 10:07:26AM +0100, David Marchand wrote:
> >> On Wed, Feb 10, 2021 at 3:56 PM Aaron Conole <aconole@redhat.com> wrote:
> >> > diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
> >> > new file mode 100644
> >> > index 0000000000..86ca24ce43
> >> > --- /dev/null
> >> > +++ b/doc/guides/contributing/testing.rst
> >> > @@ -0,0 +1,245 @@
> >> > +..  SPDX-License-Identifier: BSD-3-Clause
> >> > +    Copyright 2018 The DPDK contributors
> >> 
> >> 2021?
> 
> Whoops, I forgot to update my time machine.
> 
> >> > +
> >> > +.. _testing_guidelines:
> >> 
> >> I can't find a call to the testing_guidelines reference, so this can be removed.
> 
> done.
> 
> >> 
> >> [snip]
> >> 
> >> > +The suites can be selected by adding the ``--suite`` option to the
> >> > +``meson test`` command.  Ex: ``meson test --suite fast-tests``::
> >> > +
> >> > +  $ meson test -C build --suite fast-tests
> >> > +  ninja: Entering directory `/home/aconole/git/dpdk/build'
> >> > +  [2543/2543] Linking target app/test/dpdk-test.
> >> > +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
> >> > +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
> >> > +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
> >> > +  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
> >> > +  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
> >> > +  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
> >> > +  ...
> >> 
> >> Trying this in my build env, I get all tests failing.
> >> This is because I run this as a normal user, but the system has
> >> hugepages configured.
> >> I figured this out quickly since I know the test framework (simply
> >> added a echo 0; exit at the top of has-hugepages.sh).
> >> But I am not sure a reader of this doc would be able to troubleshoot this.
> >> 
> >> Not sure if this is worth explaining here, or if we can enhance the
> >> hugepage check (permissions maybe?).
> 
> I prefer to fix the hugepage check to make the tests SKIP when we don't
> have hugepages accessible (so we need some kind of permission check in
> there).  I will submit it separately.
> 
> >> 
> >> [snip]
> >> 
> >> > +Checking code coverage
> >> > +----------------------
> >> > +The meson build system supports generating a code coverage report
> >> > +via the `-Db_coverage=true` option, in conjunction with a package
> >> > +like **lcov**, to generate an HTML code coverage report.  Example::
> >> > +
> >> > +  $ meson covered -Db_coverage=true
> >> 
> >> At first, I read "covered" as a meson command :-).
> >> I prefer an explicit "meson setup covered -Db_coverage=true", but well
> >> this is more a matter of taste.
> >> 
> >>
> >
> > I also tend to prefer the build directory name at the end of the command,
> > so I'd suggest: "meson setup -Db_coverage=true covered".  Furthermore,
> > while I can understand the use of "covered" as a build directory name, I
> > think for consistency across all docs, we should just use "build" here as
> > the directory name, which again will reduce confusion.  "meson setup
> > -Db_coverage build"
> 
> Okay - I will paint this bikeshed like:
> 
>    meson setup build -Db_coverage=true
> 
> It's a little inconsistent everywhere - so I guess a good janitor
> project would be to clean up all the places we have meson commands.
> 
> Otherwise, what I've found is that the options generally come after the
> build directory / command is specified (ex: see octeontx, the arm64
> cross build docs, etc.) so I'll keep that for consistency there.
> Hopefully we will consistently become more consistent :)
> 

That's some fine bikeshed painting! And I'm fine with all your suggestion.

/Bruce

[dpdk-dev] [PATCH v3] guides: add a guide for developing unit tests

[Aaron Conole]

The DPDK testing infrastructure includes a comprehensive set of
libraries, utilities, and CI integrations for developers to test
their code changes.  This isn't well documented, however.

Document the basics for adding a test suite to the infrastructure
and enabling that test suite for continuous integration platforms
so that newer developers can understand how to develop test suites
and test cases.

Reviewed-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Aaron Conole <aconole@redhat.com>
---
v0->v1: Added information for TEST_SKIPPED and details about generating
        code coverage to help with ideas for writing unit test cases.
v1->v2: Corrected some spelling, rephrased a bit after suggestions by
        Ray.
v2->v3: Rewrite the meson build block, updated the copyright section,
        and change the title to be a bit nicer

 doc/guides/contributing/index.rst   |   1 +
 doc/guides/contributing/testing.rst | 243 ++++++++++++++++++++++++++++
 2 files changed, 244 insertions(+)
 create mode 100644 doc/guides/contributing/testing.rst

diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst
index 2fefd91931..41909d949b 100644
--- a/doc/guides/contributing/index.rst
+++ b/doc/guides/contributing/index.rst
@@ -14,6 +14,7 @@ Contributor's Guidelines
     abi_versioning
     documentation
     patches
+    testing
     vulnerability
     stable
     cheatsheet
diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
new file mode 100644
index 0000000000..0757d71ad0
--- /dev/null
+++ b/doc/guides/contributing/testing.rst
@@ -0,0 +1,243 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright 2021 The DPDK contributors
+
+DPDK Testing Guidelines
+=======================
+
+This document outlines the guidelines for running and adding new
+tests to the in-tree DPDK test suites.
+
+The DPDK test suite model is loosely based on the xunit model, where
+tests are grouped into test suites, and suites are run by runners.
+For a basic overview, see the basic Wikipedia article on xunit:
+`xUnit - Wikipedia <https://en.wikipedia.org/wiki/XUnit>`_.
+
+
+Running a test
+--------------
+
+DPDK tests are run via the main test runner, the `dpdk-test` app.
+The `dpdk-test` app is a command-line interface that facilitates
+running various tests or test suites.
+
+There are two modes of operation.  The first mode is as an interactive
+command shell that allows launching specific test suites.  This is
+the default operating mode of `dpdk-test` and can be done by::
+
+  $ ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>
+
+At the prompt, simply type the name of the test suite you wish to run
+and it will execute.
+
+The second form is useful for a scripting environment, and is used by
+the DPDK meson build system.  This mode is invoked by assigning a
+specific test suite name to the environment variable `DPDK_TEST`
+before invoking the `dpdk-test` command, such as::
+
+  $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>version_autotest
+  Version string: 'DPDK 20.02.0-rc0'
+  Test OK
+  RTE>>$
+
+The above shows running a specific test case.  On success, the return
+code will be '0', otherwise it will be set to some error value (such
+as '255').
+
+
+Running all tests
+-----------------
+
+In order to allow developers to quickly execute all the standard
+internal tests without needing to remember or look up each test suite
+name, the build system includes a standard way of executing the
+default test suites.  After building via `ninja`, the ``meson test``
+command will execute the standard tests and report errors.
+
+There are four groups of default test suites.  The first group is
+the **fast** test suite, which is the largest group of test cases.
+These are the bulk of the unit tests to validate functional blocks.
+The second group is the **perf** tests.  These test suites can take
+longer to run and do performance evaluations.  The third group is
+the **driver** test suite, which is mostly for special hardware
+related testing (such as `cryptodev`).  The last group are the
+**debug** tests.  These mostly are used to dump system information.
+
+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command.  Ex: ``meson test --suite fast-tests``::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
+  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
+  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
+  ...
+
+
+Adding test suites
+------------------
+
+To add a test suite to the DPDK test application, create a new test
+file for that suite (ex: see *app/test/test_version.c* for the
+``version_autotest`` test suite).  There are two important functions
+for interacting with the test harness:
+
+  1. REGISTER_TEST_COMMAND(command_name, function_to_execute)
+     Registers a test command with the name `command_name` and which
+     runs the function `function_to_execute` when `command_name` is
+     invoked.
+
+  2. unit_test_suite_runner(struct unit_test_suite \*)
+     Returns a runner for a full test suite object, which contains
+     a test suite name, setup, tear down, and vector of unit test
+     cases.
+
+Each test suite has a setup and tear down function that runs at the
+beginning and end of the test suite execution.  Each unit test has
+a similar function for test case setup and tear down.
+
+Test cases are added to the `.unit_test_cases` element of the unit
+test suite structure.  Ex:
+
+.. code-block:: c
+   :linenos:
+
+   #include <time.h>
+
+   #include <rte_common.h>
+   #include <rte_cycles.h>
+   #include <rte_hexdump.h>
+   #include <rte_random.h>
+
+   #include "test.h"
+
+   static int testsuite_setup(void) { return TEST_SUCCESS; }
+   static void testsuite_teardown(void) { }
+
+   static int ut_setup(void) { return TEST_SUCCESS; }
+   static void ut_teardown(void) { }
+
+   static int test_case_first(void) { return TEST_SUCCESS; }
+
+   static struct unit_test_suite example_testsuite = {
+          .suite_name = "EXAMPLE TEST SUITE",
+          .setup = testsuite_setup,
+          .teardown = testsuite_teardown,
+          .unit_test_cases = {
+               TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
+
+               TEST_CASES_END(), /**< NULL terminate unit test array */
+          },
+   };
+
+   static int example_tests()
+   {
+       return unit_test_suite_runner(&example_testsuite);
+   }
+
+   REGISTER_TEST_COMMAND(example_autotest, example_tests);
+
+The above code block is a small example that can be used to create a
+complete test suite with test case.
+
+
+Designing a test
+----------------
+
+Test cases have multiple ways of indicating an error has occurred,
+in order to reflect failure state back to the runner.  Using the
+various methods of indicating errors can assist in not only validating
+the requisite functionality is working, but also to help debug when
+a change in environment or code has caused things to go wrong.
+
+The first way to indicate a generic error is by returning a test
+result failure, using the *TEST_FAILED* error code.  This is the most
+basic way of indicating that an error has occurred in a test routine.
+It isn't very informative to the user, so it should really be used in
+cases where the test has catastrophically failed.
+
+The preferred method of indicating an error is via the
+`RTE_TEST_ASSERT` family of macros, which will immediately return
+*TEST_FAILED* error condition, but will also log details about the
+failure.  The basic form is:
+
+.. code-block:: c
+
+   RTE_TEST_ASSERT(cond, msg, ...)
+
+In the above macro, *cond* is the condition to evaluate to **true**.
+Any generic condition can go here.  The *msg* parameter will be a
+message to display if *cond* evaluates to **false**.  Some specialized
+macros already exist.  See `lib/librte_eal/include/rte_test.h` for
+a list of defined test assertions.
+
+Sometimes it is important to indicate that a test needs to be
+skipped, either because the environment isn't able to support running
+the test, or because some requisite functionality isn't available.  The
+test suite supports returning a result of `TEST_SKIPPED` during test
+case setup, or during test case execution to indicate that the
+preconditions of the test aren't available.  Ex::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  ...
+  46/60 DPDK:fast-tests / ipsec_autotest       SKIP     0.22 s
+  ...
+
+
+Checking code coverage
+----------------------
+The meson build system supports generating a code coverage report
+via the `-Db_coverage=true` option, in conjunction with a package
+like **lcov**, to generate an HTML code coverage report.  Example::
+
+  $ meson setup build -Db_coverage=true
+  $ meson test -C build --suite fast-tests
+  $ ninja coverage-html -C build
+
+The above will generate an html report in the
+`build/meson-logs/coveragereport/` directory that can be explored
+for detailed code covered information.  This can be used to assist
+in test development.
+
+
+Adding a suite to the default
+-----------------------------
+
+Adding to one of the default tests involves editing the appropriate
+meson build file `app/test/meson.build` and adding the command to
+the correct test suite class.  Once added, the new test suite will
+be run as part of the appropriate class (fast, perf, driver, etc.).
+
+Some of these default test suites are run during continuous integration
+tests, making regression checking automatic for new patches submitted
+to the project.
-- 
2.25.4

Re: [dpdk-dev] [PATCH v2] guides: add a testing guide for developing tests

[Aaron Conole]

Aaron Conole <aconole@redhat.com> writes:

> Bruce Richardson <bruce.richardson@intel.com> writes:
>
>> On Tue, Mar 02, 2021 at 10:07:26AM +0100, David Marchand wrote:
>>> On Wed, Feb 10, 2021 at 3:56 PM Aaron Conole <aconole@redhat.com> wrote:
>>> > diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
>>> > new file mode 100644
>>> > index 0000000000..86ca24ce43
>>> > --- /dev/null
>>> > +++ b/doc/guides/contributing/testing.rst
>>> > @@ -0,0 +1,245 @@
>>> > +..  SPDX-License-Identifier: BSD-3-Clause
>>> > +    Copyright 2018 The DPDK contributors
>>> 
>>> 2021?
>
> Whoops, I forgot to update my time machine.
>
>>> > +
>>> > +.. _testing_guidelines:
>>> 
>>> I can't find a call to the testing_guidelines reference, so this can be removed.
>
> done.
>
>>> 
>>> [snip]
>>> 
>>> > +The suites can be selected by adding the ``--suite`` option to the
>>> > +``meson test`` command.  Ex: ``meson test --suite fast-tests``::
>>> > +
>>> > +  $ meson test -C build --suite fast-tests
>>> > +  ninja: Entering directory `/home/aconole/git/dpdk/build'
>>> > +  [2543/2543] Linking target app/test/dpdk-test.
>>> > +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
>>> > +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
>>> > +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
>>> > +  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
>>> > +  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
>>> > +  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
>>> > +  ...
>>> 
>>> Trying this in my build env, I get all tests failing.
>>> This is because I run this as a normal user, but the system has
>>> hugepages configured.
>>> I figured this out quickly since I know the test framework (simply
>>> added a echo 0; exit at the top of has-hugepages.sh).
>>> But I am not sure a reader of this doc would be able to troubleshoot this.
>>> 
>>> Not sure if this is worth explaining here, or if we can enhance the
>>> hugepage check (permissions maybe?).
>
> I prefer to fix the hugepage check to make the tests SKIP when we don't
> have hugepages accessible (so we need some kind of permission check in
> there).  I will submit it separately.
>

Here is my PoC for this - if you think it's good enough, I'll submit as
formal PATCH.

---
index d600fad319..1c3cfb665a 100755
--- a/app/test/has-hugepage.sh
+++ b/app/test/has-hugepage.sh
@@ -3,7 +3,17 @@
 # Copyright 2020 Mellanox Technologies, Ltd
 
 if [ "$(uname)" = "Linux" ] ; then
-       cat /proc/sys/vm/nr_hugepages || echo 0
+       nr_hugepages=$(cat /proc/sys/vm/nr_hugepages)
+       # Need to check if we have permissions to access hugepages
+       perm=""
+       for mount in `mount | grep hugetlbfs | awk '{ print $3; }'`; do
+               test ! -w $mount/. || perm="$mount"
+       done
+       if [ "$perm" = "" -o "$nr_hugepages" = "0" ]; then
+               echo 0
+       else
+               echo $nr_hugepages
+       fi
 elif [ "$(uname)" = "FreeBSD" ] ; then
        echo 1 # assume FreeBSD always has hugepages
 else
---

Re: [dpdk-dev] [PATCH v2] guides: add a testing guide for developing tests

[David Marchand]

On Tue, Mar 9, 2021 at 5:14 PM Aaron Conole <aconole@redhat.com> wrote:
> >>> > +The suites can be selected by adding the ``--suite`` option to the
> >>> > +``meson test`` command.  Ex: ``meson test --suite fast-tests``::
> >>> > +
> >>> > +  $ meson test -C build --suite fast-tests
> >>> > +  ninja: Entering directory `/home/aconole/git/dpdk/build'
> >>> > +  [2543/2543] Linking target app/test/dpdk-test.
> >>> > +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
> >>> > +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
> >>> > +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
> >>> > +  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
> >>> > +  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
> >>> > +  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
> >>> > +  ...
> >>>
> >>> Trying this in my build env, I get all tests failing.
> >>> This is because I run this as a normal user, but the system has
> >>> hugepages configured.
> >>> I figured this out quickly since I know the test framework (simply
> >>> added a echo 0; exit at the top of has-hugepages.sh).
> >>> But I am not sure a reader of this doc would be able to troubleshoot this.
> >>>
> >>> Not sure if this is worth explaining here, or if we can enhance the
> >>> hugepage check (permissions maybe?).
> >
> > I prefer to fix the hugepage check to make the tests SKIP when we don't
> > have hugepages accessible (so we need some kind of permission check in
> > there).  I will submit it separately.
> >
>
> Here is my PoC for this - if you think it's good enough, I'll submit as
> formal PATCH.
>
> ---
> index d600fad319..1c3cfb665a 100755
> --- a/app/test/has-hugepage.sh
> +++ b/app/test/has-hugepage.sh
> @@ -3,7 +3,17 @@
>  # Copyright 2020 Mellanox Technologies, Ltd
>
>  if [ "$(uname)" = "Linux" ] ; then
> -       cat /proc/sys/vm/nr_hugepages || echo 0
> +       nr_hugepages=$(cat /proc/sys/vm/nr_hugepages)
> +       # Need to check if we have permissions to access hugepages
> +       perm=""
> +       for mount in `mount | grep hugetlbfs | awk '{ print $3; }'`; do
> +               test ! -w $mount/. || perm="$mount"
> +       done
> +       if [ "$perm" = "" -o "$nr_hugepages" = "0" ]; then
> +               echo 0
> +       else
> +               echo $nr_hugepages
> +       fi
>  elif [ "$(uname)" = "FreeBSD" ] ; then
>         echo 1 # assume FreeBSD always has hugepages
>  else
> ---
>

I need to think more about the multiple mountpoints case (but I may be
imagining too much twisted setups..).

At least, this works in my env.
We need tests in travis/GHA, and sending a non-RFC patch is the best
way to know :-)

So +1 for a patch.
Thanks Aaron!


-- 
David Marchand

Re: [dpdk-dev] [PATCH v2] guides: add a testing guide for developing tests

[Aaron Conole]

David Marchand <david.marchand@redhat.com> writes:

> On Tue, Mar 9, 2021 at 5:14 PM Aaron Conole <aconole@redhat.com> wrote:
>> >>> > +The suites can be selected by adding the ``--suite`` option to the
>> >>> > +``meson test`` command.  Ex: ``meson test --suite fast-tests``::
>> >>> > +
>> >>> > +  $ meson test -C build --suite fast-tests
>> >>> > +  ninja: Entering directory `/home/aconole/git/dpdk/build'
>> >>> > +  [2543/2543] Linking target app/test/dpdk-test.
>> >>> > +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
>> >>> > +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
>> >>> > +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
>> >>> > +  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
>> >>> > +  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
>> >>> > +  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
>> >>> > +  ...
>> >>>
>> >>> Trying this in my build env, I get all tests failing.
>> >>> This is because I run this as a normal user, but the system has
>> >>> hugepages configured.
>> >>> I figured this out quickly since I know the test framework (simply
>> >>> added a echo 0; exit at the top of has-hugepages.sh).
>> >>> But I am not sure a reader of this doc would be able to troubleshoot this.
>> >>>
>> >>> Not sure if this is worth explaining here, or if we can enhance the
>> >>> hugepage check (permissions maybe?).
>> >
>> > I prefer to fix the hugepage check to make the tests SKIP when we don't
>> > have hugepages accessible (so we need some kind of permission check in
>> > there).  I will submit it separately.
>> >
>>
>> Here is my PoC for this - if you think it's good enough, I'll submit as
>> formal PATCH.
>>
>> ---
>> index d600fad319..1c3cfb665a 100755
>> --- a/app/test/has-hugepage.sh
>> +++ b/app/test/has-hugepage.sh
>> @@ -3,7 +3,17 @@
>>  # Copyright 2020 Mellanox Technologies, Ltd
>>
>>  if [ "$(uname)" = "Linux" ] ; then
>> -       cat /proc/sys/vm/nr_hugepages || echo 0
>> +       nr_hugepages=$(cat /proc/sys/vm/nr_hugepages)
>> +       # Need to check if we have permissions to access hugepages
>> +       perm=""
>> +       for mount in `mount | grep hugetlbfs | awk '{ print $3; }'`; do
>> +               test ! -w $mount/. || perm="$mount"
>> +       done
>> +       if [ "$perm" = "" -o "$nr_hugepages" = "0" ]; then
>> +               echo 0
>> +       else
>> +               echo $nr_hugepages
>> +       fi
>>  elif [ "$(uname)" = "FreeBSD" ] ; then
>>         echo 1 # assume FreeBSD always has hugepages
>>  else
>> ---
>>
>
> I need to think more about the multiple mountpoints case (but I may be
> imagining too much twisted setups..).
>
> At least, this works in my env.
> We need tests in travis/GHA, and sending a non-RFC patch is the best
> way to know :-)

Sent :)

> So +1 for a patch.
> Thanks Aaron!

Re: [dpdk-dev] [PATCH v3] guides: add a guide for developing unit tests

[Ferruh Yigit]

On 3/9/2021 3:57 PM, Aaron Conole wrote:
> The DPDK testing infrastructure includes a comprehensive set of
> libraries, utilities, and CI integrations for developers to test
> their code changes.  This isn't well documented, however.
> 
> Document the basics for adding a test suite to the infrastructure
> and enabling that test suite for continuous integration platforms
> so that newer developers can understand how to develop test suites
> and test cases.

+1 to adding this long missing documentation, thanks.

> 
> Reviewed-by: David Marchand <david.marchand@redhat.com>
> Signed-off-by: Aaron Conole <aconole@redhat.com>
> ---
> v0->v1: Added information for TEST_SKIPPED and details about generating
>         code coverage to help with ideas for writing unit test cases.
> v1->v2: Corrected some spelling, rephrased a bit after suggestions by
>         Ray.
> v2->v3: Rewrite the meson build block, updated the copyright section,
>         and change the title to be a bit nicer
> 
>  doc/guides/contributing/index.rst   |   1 +
>  doc/guides/contributing/testing.rst | 243 ++++++++++++++++++++++++++++
>  2 files changed, 244 insertions(+)
>  create mode 100644 doc/guides/contributing/testing.rst
> 
> diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst
> index 2fefd91931..41909d949b 100644
> --- a/doc/guides/contributing/index.rst
> +++ b/doc/guides/contributing/index.rst
> @@ -14,6 +14,7 @@ Contributor's Guidelines
>      abi_versioning
>      documentation
>      patches
> +    testing
>      vulnerability
>      stable
>      cheatsheet
> diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
> new file mode 100644
> index 0000000000..0757d71ad0
> --- /dev/null
> +++ b/doc/guides/contributing/testing.rst
> @@ -0,0 +1,243 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright 2021 The DPDK contributors
> +
> +DPDK Testing Guidelines
> +=======================
> +
> +This document outlines the guidelines for running and adding new
> +tests to the in-tree DPDK test suites.
> +

I think both the section name (DPDK Testing Guidelines) and the file name
(testing.rst) is too broad comparing to what is described here.

What about using "DPDK Unit Test Guidelines", and 'unit_test.rst'?

> +The DPDK test suite model is loosely based on the xunit model, where
> +tests are grouped into test suites, and suites are run by runners.
> +For a basic overview, see the basic Wikipedia article on xunit:
> +`xUnit - Wikipedia <https://en.wikipedia.org/wiki/XUnit>`_.
> +
> +
> +Running a test
> +--------------
> +
> +DPDK tests are run via the main test runner, the `dpdk-test` app.
> +The `dpdk-test` app is a command-line interface that facilitates
> +running various tests or test suites.
> +
> +There are two modes of operation.  The first mode is as an interactive
> +command shell that allows launching specific test suites.  This is
> +the default operating mode of `dpdk-test` and can be done by::
> +
> +  $ ./build/app/test/dpdk-test --dpdk-options-here
> +  EAL: Detected 4 lcore(s)
> +  EAL: Detected 1 NUMA nodes
> +  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
> +  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
> +  EAL: Selected IOVA mode 'VA'
> +  EAL: Probing VFIO support...
> +  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
> +  EAL:   Invalid NUMA socket, default to 0
> +  EAL:   probe driver: 8086:15d7 net_e1000_em
> +  APP: HPET is not enabled, using TSC as default timer
> +  RTE>>
> +
> +At the prompt, simply type the name of the test suite you wish to run
> +and it will execute.
> +
> +The second form is useful for a scripting environment, and is used by
> +the DPDK meson build system.  This mode is invoked by assigning a
> +specific test suite name to the environment variable `DPDK_TEST`
> +before invoking the `dpdk-test` command, such as::
> +
> +  $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --dpdk-options-here
> +  EAL: Detected 4 lcore(s)
> +  EAL: Detected 1 NUMA nodes
> +  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
> +  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
> +  EAL: Selected IOVA mode 'VA'
> +  EAL: Probing VFIO support...
> +  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
> +  EAL:   Invalid NUMA socket, default to 0
> +  EAL:   probe driver: 8086:15d7 net_e1000_em
> +  APP: HPET is not enabled, using TSC as default timer
> +  RTE>>version_autotest
> +  Version string: 'DPDK 20.02.0-rc0'
> +  Test OK
> +  RTE>>$
> +

According code, it is also possible to run the unit test by providing its name
as argument to 'dpdk-test', like bellow. And benefit of this is, it is possible
to provide multiple tests at once and run them all, which can't be done via
'DPDK_TEST" env var. Should we document this too?

./build/app/test/dpdk-test version_autotest
EAL: Detected 96 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No free 2048 kB hugepages reported on node 0
EAL: No free 2048 kB hugepages reported on node 1
EAL: No available 2048 kB hugepages reported
APP: HPET is not enabled, using TSC as default timer
RTE>>version_autotest
Version string: 'DPDK 21.05.0'
Test OK


Another env variable it "DPDK_TEST_PARAMS" to provide more parameters, since
meson has args support for test exec, we are not using it in meson, but maybe
good to document it for the individual exec usage.


Also there is a 'RTE_TEST_RECURSIVE' env var used, I am not clear why it is for,
anyone knows the history on it?


> +The above shows running a specific test case.  On success, the return
> +code will be '0', otherwise it will be set to some error value (such
> +as '255').
> +

Also negative value

> +
> +Running all tests
> +-----------------
> +

What about "Running multiple test cases", instead of "all tests"?

Btw, it is possible to run single unit test with meson, perhaps this can be
documented in previous section:
"meson test -C build version_autotest"

Also another useful command is to list all test cases, may worth documenting
that too:
"meson test -C build --list"


We have a legacy runner as well, 'autotest.py', most probably it is outdated
now, if we don't document it we may consider removing it completely
(./app/test/autotest*.py):
"python ./app/test/autotest.py ./build/app/test/dpdk-test build version"

> +In order to allow developers to quickly execute all the standard
> +internal tests without needing to remember or look up each test suite
> +name, the build system includes a standard way of executing the
> +default test suites.  After building via `ninja`, the ``meson test``
> +command will execute the standard tests and report errors.
> +
> +There are four groups of default test suites.  The first group is
> +the **fast** test suite, which is the largest group of test cases.
> +These are the bulk of the unit tests to validate functional blocks.
> +The second group is the **perf** tests.  These test suites can take
> +longer to run and do performance evaluations.  The third group is
> +the **driver** test suite, which is mostly for special hardware
> +related testing (such as `cryptodev`).  The last group are the
> +**debug** tests.  These mostly are used to dump system information.
> +
> +The suites can be selected by adding the ``--suite`` option to the
> +``meson test`` command.  Ex: ``meson test --suite fast-tests``::
> +
> +  $ meson test -C build --suite fast-tests
> +  ninja: Entering directory `/home/aconole/git/dpdk/build'
> +  [2543/2543] Linking target app/test/dpdk-test.
> +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
> +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
> +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
> +  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
> +  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
> +  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
> +  ...
> +
> +
> +Adding test suites
> +------------------
> +

What about renaming 'test suites' to 'test cases', since 'test suites' is just a
group of test case and we can add single test case too.

> +To add a test suite to the DPDK test application, create a new test
> +file for that suite (ex: see *app/test/test_version.c* for the
> +``version_autotest`` test suite).  There are two important functions
> +for interacting with the test harness:
> +
> +  1. REGISTER_TEST_COMMAND(command_name, function_to_execute)
> +     Registers a test command with the name `command_name` and which
> +     runs the function `function_to_execute` when `command_name` is
> +     invoked.
> +
> +  2. unit_test_suite_runner(struct unit_test_suite \*)
> +     Returns a runner for a full test suite object, which contains
> +     a test suite name, setup, tear down, and vector of unit test
> +     cases.
> +

There are two different usage as "test suite" which is confusing I think,

1) First usage within dpdk-test app

a) A test command can be running single test_case, like
'version_autotest' -> 'test_version'

b) Or it can be a group of test cases, like:
'link_bonding_autotest' -> 'test_create_bonded_device',
			'test_create_bonded_device_with_invalid_params',
			'test_add_slave_to_bonded_device',
			'...'

The second one is called as unit_test_suite within the scope of the 'dpdk-test'
application, but at the end of the day it is a single command for test app.

First one has simple execution, but second one has setup/teardown as you already
described below.


2) Second usage with meson,

There is a next level of grouping in meson, as described in previous paragraph,
like "fast-test suite", "perf test suite" etc.., this is grouping of various
test commands.


I wonder if this two different "test suite" can be clarified more. Can we say
"meson test suite" to the meson one?


> +Each test suite has a setup and tear down function that runs at the
> +beginning and end of the test suite execution.  Each unit test has
> +a similar function for test case setup and tear down.
> +
> +Test cases are added to the `.unit_test_cases` element of the unit
> +test suite structure.  Ex:
> +
> +.. code-block:: c
> +   :linenos:
> +
> +   #include <time.h>
> +
> +   #include <rte_common.h>
> +   #include <rte_cycles.h>
> +   #include <rte_hexdump.h>
> +   #include <rte_random.h>
> +
> +   #include "test.h"
> +
> +   static int testsuite_setup(void) { return TEST_SUCCESS; }
> +   static void testsuite_teardown(void) { }
> +
> +   static int ut_setup(void) { return TEST_SUCCESS; }
> +   static void ut_teardown(void) { }
> +
> +   static int test_case_first(void) { return TEST_SUCCESS; }
> +
> +   static struct unit_test_suite example_testsuite = {
> +          .suite_name = "EXAMPLE TEST SUITE",
> +          .setup = testsuite_setup,
> +          .teardown = testsuite_teardown,
> +          .unit_test_cases = {
> +               TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
> +
> +               TEST_CASES_END(), /**< NULL terminate unit test array */
> +          },
> +   };
> +
> +   static int example_tests()
> +   {
> +       return unit_test_suite_runner(&example_testsuite);
> +   }
> +
> +   REGISTER_TEST_COMMAND(example_autotest, example_tests);
> +
> +The above code block is a small example that can be used to create a
> +complete test suite with test case.
> +
> +
> +Designing a test
> +----------------
> +
> +Test cases have multiple ways of indicating an error has occurred,
> +in order to reflect failure state back to the runner.  Using the
> +various methods of indicating errors can assist in not only validating
> +the requisite functionality is working, but also to help debug when
> +a change in environment or code has caused things to go wrong.
> +
> +The first way to indicate a generic error is by returning a test
> +result failure, using the *TEST_FAILED* error code.  This is the most
> +basic way of indicating that an error has occurred in a test routine.
> +It isn't very informative to the user, so it should really be used in
> +cases where the test has catastrophically failed.
> +
> +The preferred method of indicating an error is via the
> +`RTE_TEST_ASSERT` family of macros, which will immediately return
> +*TEST_FAILED* error condition, but will also log details about the
> +failure.  The basic form is:
> +
> +.. code-block:: c
> +
> +   RTE_TEST_ASSERT(cond, msg, ...)
> +
> +In the above macro, *cond* is the condition to evaluate to **true**.
> +Any generic condition can go here.  The *msg* parameter will be a
> +message to display if *cond* evaluates to **false**.  Some specialized
> +macros already exist.  See `lib/librte_eal/include/rte_test.h` for
> +a list of defined test assertions.
> +
> +Sometimes it is important to indicate that a test needs to be
> +skipped, either because the environment isn't able to support running
> +the test, or because some requisite functionality isn't available.  The
> +test suite supports returning a result of `TEST_SKIPPED` during test
> +case setup, or during test case execution to indicate that the
> +preconditions of the test aren't available.  Ex::
> +
> +  $ meson test -C build --suite fast-tests
> +  ninja: Entering directory `/home/aconole/git/dpdk/build
> +  [2543/2543] Linking target app/test/dpdk-test.
> +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
> +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
> +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
> +  ...
> +  46/60 DPDK:fast-tests / ipsec_autotest       SKIP     0.22 s
> +  ...
> +
> +
> +Checking code coverage
> +----------------------
> +The meson build system supports generating a code coverage report
> +via the `-Db_coverage=true` option, in conjunction with a package
> +like **lcov**, to generate an HTML code coverage report.  Example::
> +
> +  $ meson setup build -Db_coverage=true
> +  $ meson test -C build --suite fast-tests
> +  $ ninja coverage-html -C build
> +
> +The above will generate an html report in the
> +`build/meson-logs/coveragereport/` directory that can be explored
> +for detailed code covered information.  This can be used to assist
> +in test development.
> +
> +
> +Adding a suite to the default
> +-----------------------------
> +

What is "the default"?

> +Adding to one of the default tests involves editing the appropriate
> +meson build file `app/test/meson.build` and adding the command to
> +the correct test suite class.  Once added, the new test suite will
> +be run as part of the appropriate class (fast, perf, driver, etc.).
> +

Here "meson test suite" referenced as 'class', lets figure out the naming for it
and use it consistently.

> +Some of these default test suites are run during continuous integration

Same comment for "default test suites" usage.

> +tests, making regression checking automatic for new patches submitted
> +to the project.
> 

Is there a rule to document how to decide to add (or not) a testcase to a meson
test suit?

Re: [dpdk-dev] [PATCH v3] guides: add a guide for developing unit tests

[Aaron Conole]

Ferruh Yigit <ferruh.yigit@intel.com> writes:

> On 3/9/2021 3:57 PM, Aaron Conole wrote:
>> The DPDK testing infrastructure includes a comprehensive set of
>> libraries, utilities, and CI integrations for developers to test
>> their code changes.  This isn't well documented, however.
>> 
>> Document the basics for adding a test suite to the infrastructure
>> and enabling that test suite for continuous integration platforms
>> so that newer developers can understand how to develop test suites
>> and test cases.
>
> +1 to adding this long missing documentation, thanks.
>
>> 
>> Reviewed-by: David Marchand <david.marchand@redhat.com>
>> Signed-off-by: Aaron Conole <aconole@redhat.com>
>> ---
>> v0->v1: Added information for TEST_SKIPPED and details about generating
>>         code coverage to help with ideas for writing unit test cases.
>> v1->v2: Corrected some spelling, rephrased a bit after suggestions by
>>         Ray.
>> v2->v3: Rewrite the meson build block, updated the copyright section,
>>         and change the title to be a bit nicer
>> 
>>  doc/guides/contributing/index.rst   |   1 +
>>  doc/guides/contributing/testing.rst | 243 ++++++++++++++++++++++++++++
>>  2 files changed, 244 insertions(+)
>>  create mode 100644 doc/guides/contributing/testing.rst
>> 
>> diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst
>> index 2fefd91931..41909d949b 100644
>> --- a/doc/guides/contributing/index.rst
>> +++ b/doc/guides/contributing/index.rst
>> @@ -14,6 +14,7 @@ Contributor's Guidelines
>>      abi_versioning
>>      documentation
>>      patches
>> +    testing
>>      vulnerability
>>      stable
>>      cheatsheet
>> diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
>> new file mode 100644
>> index 0000000000..0757d71ad0
>> --- /dev/null
>> +++ b/doc/guides/contributing/testing.rst
>> @@ -0,0 +1,243 @@
>> +..  SPDX-License-Identifier: BSD-3-Clause
>> +    Copyright 2021 The DPDK contributors
>> +
>> +DPDK Testing Guidelines
>> +=======================
>> +
>> +This document outlines the guidelines for running and adding new
>> +tests to the in-tree DPDK test suites.
>> +
>
> I think both the section name (DPDK Testing Guidelines) and the file name
> (testing.rst) is too broad comparing to what is described here.

Makes sense to me.

> What about using "DPDK Unit Test Guidelines", and 'unit_test.rst'?

Sure - I will make this change.

>> +The DPDK test suite model is loosely based on the xunit model, where
>> +tests are grouped into test suites, and suites are run by runners.
>> +For a basic overview, see the basic Wikipedia article on xunit:
>> +`xUnit - Wikipedia <https://en.wikipedia.org/wiki/XUnit>`_.
>> +
>> +
>> +Running a test
>> +--------------
>> +
>> +DPDK tests are run via the main test runner, the `dpdk-test` app.
>> +The `dpdk-test` app is a command-line interface that facilitates
>> +running various tests or test suites.
>> +
>> +There are two modes of operation.  The first mode is as an interactive
>> +command shell that allows launching specific test suites.  This is
>> +the default operating mode of `dpdk-test` and can be done by::
>> +
>> +  $ ./build/app/test/dpdk-test --dpdk-options-here
>> +  EAL: Detected 4 lcore(s)
>> +  EAL: Detected 1 NUMA nodes
>> + EAL: Static memory layout is selected, amount of reserved memory
>> can be adjusted with -m or --socket-mem
>> +  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
>> +  EAL: Selected IOVA mode 'VA'
>> +  EAL: Probing VFIO support...
>> +  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
>> +  EAL:   Invalid NUMA socket, default to 0
>> +  EAL:   probe driver: 8086:15d7 net_e1000_em
>> +  APP: HPET is not enabled, using TSC as default timer
>> +  RTE>>
>> +
>> +At the prompt, simply type the name of the test suite you wish to run
>> +and it will execute.
>> +
>> +The second form is useful for a scripting environment, and is used by
>> +the DPDK meson build system.  This mode is invoked by assigning a
>> +specific test suite name to the environment variable `DPDK_TEST`
>> +before invoking the `dpdk-test` command, such as::
>> +
>> +  $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --dpdk-options-here
>> +  EAL: Detected 4 lcore(s)
>> +  EAL: Detected 1 NUMA nodes
>> + EAL: Static memory layout is selected, amount of reserved memory
>> can be adjusted with -m or --socket-mem
>> +  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
>> +  EAL: Selected IOVA mode 'VA'
>> +  EAL: Probing VFIO support...
>> +  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
>> +  EAL:   Invalid NUMA socket, default to 0
>> +  EAL:   probe driver: 8086:15d7 net_e1000_em
>> +  APP: HPET is not enabled, using TSC as default timer
>> +  RTE>>version_autotest
>> +  Version string: 'DPDK 20.02.0-rc0'
>> +  Test OK
>> +  RTE>>$
>> +
>
> According code, it is also possible to run the unit test by providing its name
> as argument to 'dpdk-test', like bellow. And benefit of this is, it is possible
> to provide multiple tests at once and run them all, which can't be done via
> 'DPDK_TEST" env var. Should we document this too?

Such functionality was added after I posted this series.  I think it's a
worthwhile addition.

> ./build/app/test/dpdk-test version_autotest
> EAL: Detected 96 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Detected static linkage of DPDK
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> EAL: No free 2048 kB hugepages reported on node 0
> EAL: No free 2048 kB hugepages reported on node 1
> EAL: No available 2048 kB hugepages reported
> APP: HPET is not enabled, using TSC as default timer
> RTE>>version_autotest
> Version string: 'DPDK 21.05.0'
> Test OK
>
>
> Another env variable it "DPDK_TEST_PARAMS" to provide more parameters, since
> meson has args support for test exec, we are not using it in meson, but maybe
> good to document it for the individual exec usage.

Makes sense.

>
> Also there is a 'RTE_TEST_RECURSIVE' env var used, I am not clear why it is for,
> anyone knows the history on it?
>
>
>> +The above shows running a specific test case.  On success, the return
>> +code will be '0', otherwise it will be set to some error value (such
>> +as '255').
>> +
>
> Also negative value
>
>> +
>> +Running all tests
>> +-----------------
>> +
>
> What about "Running multiple test cases", instead of "all tests"?
>
> Btw, it is possible to run single unit test with meson, perhaps this can be
> documented in previous section:
> "meson test -C build version_autotest"
>
> Also another useful command is to list all test cases, may worth documenting
> that too:
> "meson test -C build --list"

Agreed - I will add this.

>
> We have a legacy runner as well, 'autotest.py', most probably it is outdated
> now, if we don't document it we may consider removing it completely
> (./app/test/autotest*.py):
> "python ./app/test/autotest.py ./build/app/test/dpdk-test build version"

I guess it might be best to remove it.

>> +In order to allow developers to quickly execute all the standard
>> +internal tests without needing to remember or look up each test suite
>> +name, the build system includes a standard way of executing the
>> +default test suites.  After building via `ninja`, the ``meson test``
>> +command will execute the standard tests and report errors.
>> +
>> +There are four groups of default test suites.  The first group is
>> +the **fast** test suite, which is the largest group of test cases.
>> +These are the bulk of the unit tests to validate functional blocks.
>> +The second group is the **perf** tests.  These test suites can take
>> +longer to run and do performance evaluations.  The third group is
>> +the **driver** test suite, which is mostly for special hardware
>> +related testing (such as `cryptodev`).  The last group are the
>> +**debug** tests.  These mostly are used to dump system information.
>> +
>> +The suites can be selected by adding the ``--suite`` option to the
>> +``meson test`` command.  Ex: ``meson test --suite fast-tests``::
>> +
>> +  $ meson test -C build --suite fast-tests
>> +  ninja: Entering directory `/home/aconole/git/dpdk/build'
>> +  [2543/2543] Linking target app/test/dpdk-test.
>> +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
>> +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
>> +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
>> +  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
>> +  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
>> +  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
>> +  ...
>> +
>> +
>> +Adding test suites
>> +------------------
>> +
>
> What about renaming 'test suites' to 'test cases', since 'test suites' is just a
> group of test case and we can add single test case too.

Okay.

>> +To add a test suite to the DPDK test application, create a new test
>> +file for that suite (ex: see *app/test/test_version.c* for the
>> +``version_autotest`` test suite).  There are two important functions
>> +for interacting with the test harness:
>> +
>> +  1. REGISTER_TEST_COMMAND(command_name, function_to_execute)
>> +     Registers a test command with the name `command_name` and which
>> +     runs the function `function_to_execute` when `command_name` is
>> +     invoked.
>> +
>> +  2. unit_test_suite_runner(struct unit_test_suite \*)
>> +     Returns a runner for a full test suite object, which contains
>> +     a test suite name, setup, tear down, and vector of unit test
>> +     cases.
>> +
>
> There are two different usage as "test suite" which is confusing I think,
>
> 1) First usage within dpdk-test app
>
> a) A test command can be running single test_case, like
> 'version_autotest' -> 'test_version'
>
> b) Or it can be a group of test cases, like:

Well 'link_bonding_autotest' is a "test-suite"

> 'link_bonding_autotest' -> 'test_create_bonded_device',
> 			'test_create_bonded_device_with_invalid_params',
> 			'test_add_slave_to_bonded_device',
> 			'...'
>
> The second one is called as unit_test_suite within the scope of the 'dpdk-test'
> application, but at the end of the day it is a single command for test app.

Sure, the distinction is more important for development rather than
executing cases, so maybe I can make it more clear here.

> First one has simple execution, but second one has setup/teardown as you already
> described below.
>
>
> 2) Second usage with meson,
>
> There is a next level of grouping in meson, as described in previous paragraph,
> like "fast-test suite", "perf test suite" etc.., this is grouping of various
> test commands.
>
>
> I wonder if this two different "test suite" can be clarified more. Can we say
> "meson test suite" to the meson one?

Okay, that makes sense to me.

>> +Each test suite has a setup and tear down function that runs at the
>> +beginning and end of the test suite execution.  Each unit test has
>> +a similar function for test case setup and tear down.
>> +
>> +Test cases are added to the `.unit_test_cases` element of the unit
>> +test suite structure.  Ex:
>> +
>> +.. code-block:: c
>> +   :linenos:
>> +
>> +   #include <time.h>
>> +
>> +   #include <rte_common.h>
>> +   #include <rte_cycles.h>
>> +   #include <rte_hexdump.h>
>> +   #include <rte_random.h>
>> +
>> +   #include "test.h"
>> +
>> +   static int testsuite_setup(void) { return TEST_SUCCESS; }
>> +   static void testsuite_teardown(void) { }
>> +
>> +   static int ut_setup(void) { return TEST_SUCCESS; }
>> +   static void ut_teardown(void) { }
>> +
>> +   static int test_case_first(void) { return TEST_SUCCESS; }
>> +
>> +   static struct unit_test_suite example_testsuite = {
>> +          .suite_name = "EXAMPLE TEST SUITE",
>> +          .setup = testsuite_setup,
>> +          .teardown = testsuite_teardown,
>> +          .unit_test_cases = {
>> +               TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
>> +
>> +               TEST_CASES_END(), /**< NULL terminate unit test array */
>> +          },
>> +   };
>> +
>> +   static int example_tests()
>> +   {
>> +       return unit_test_suite_runner(&example_testsuite);
>> +   }
>> +
>> +   REGISTER_TEST_COMMAND(example_autotest, example_tests);
>> +
>> +The above code block is a small example that can be used to create a
>> +complete test suite with test case.
>> +
>> +
>> +Designing a test
>> +----------------
>> +
>> +Test cases have multiple ways of indicating an error has occurred,
>> +in order to reflect failure state back to the runner.  Using the
>> +various methods of indicating errors can assist in not only validating
>> +the requisite functionality is working, but also to help debug when
>> +a change in environment or code has caused things to go wrong.
>> +
>> +The first way to indicate a generic error is by returning a test
>> +result failure, using the *TEST_FAILED* error code.  This is the most
>> +basic way of indicating that an error has occurred in a test routine.
>> +It isn't very informative to the user, so it should really be used in
>> +cases where the test has catastrophically failed.
>> +
>> +The preferred method of indicating an error is via the
>> +`RTE_TEST_ASSERT` family of macros, which will immediately return
>> +*TEST_FAILED* error condition, but will also log details about the
>> +failure.  The basic form is:
>> +
>> +.. code-block:: c
>> +
>> +   RTE_TEST_ASSERT(cond, msg, ...)
>> +
>> +In the above macro, *cond* is the condition to evaluate to **true**.
>> +Any generic condition can go here.  The *msg* parameter will be a
>> +message to display if *cond* evaluates to **false**.  Some specialized
>> +macros already exist.  See `lib/librte_eal/include/rte_test.h` for
>> +a list of defined test assertions.
>> +
>> +Sometimes it is important to indicate that a test needs to be
>> +skipped, either because the environment isn't able to support running
>> +the test, or because some requisite functionality isn't available.  The
>> +test suite supports returning a result of `TEST_SKIPPED` during test
>> +case setup, or during test case execution to indicate that the
>> +preconditions of the test aren't available.  Ex::
>> +
>> +  $ meson test -C build --suite fast-tests
>> +  ninja: Entering directory `/home/aconole/git/dpdk/build
>> +  [2543/2543] Linking target app/test/dpdk-test.
>> +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
>> +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
>> +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
>> +  ...
>> +  46/60 DPDK:fast-tests / ipsec_autotest       SKIP     0.22 s
>> +  ...
>> +
>> +
>> +Checking code coverage
>> +----------------------
>> +The meson build system supports generating a code coverage report
>> +via the `-Db_coverage=true` option, in conjunction with a package
>> +like **lcov**, to generate an HTML code coverage report.  Example::
>> +
>> +  $ meson setup build -Db_coverage=true
>> +  $ meson test -C build --suite fast-tests
>> +  $ ninja coverage-html -C build
>> +
>> +The above will generate an html report in the
>> +`build/meson-logs/coveragereport/` directory that can be explored
>> +for detailed code covered information.  This can be used to assist
>> +in test development.
>> +
>> +
>> +Adding a suite to the default
>> +-----------------------------
>> +
>
> What is "the default"?

I meant the meson test suite here.

>> +Adding to one of the default tests involves editing the appropriate
>> +meson build file `app/test/meson.build` and adding the command to
>> +the correct test suite class.  Once added, the new test suite will
>> +be run as part of the appropriate class (fast, perf, driver, etc.).
>> +
>
> Here "meson test suite" referenced as 'class', lets figure out the naming for it
> and use it consistently.

Yes, will do.

>> +Some of these default test suites are run during continuous integration
>
> Same comment for "default test suites" usage.

ACK

>> +tests, making regression checking automatic for new patches submitted
>> +to the project.
>> 
>
> Is there a rule to document how to decide to add (or not) a testcase to a meson
> test suit?

No - it's mostly up to developers and maintainers for now, although I
would think the normal thing would be to add the case to the meson test
suite.  I will try to clarify it a bit more.

[dpdk-dev] [PATCH v4] guides: add a guide for developing unit tests

[Aaron Conole]

The DPDK testing infrastructure includes a comprehensive set of
libraries, utilities, and CI integrations for developers to test
their code changes.  This isn't well documented, however.

Document the basics for adding a test suite to the infrastructure
and enabling that test suite for continuous integration platforms
so that newer developers can understand how to develop test suites
and test cases.

Signed-off-by: Aaron Conole <aconole@redhat.com>
---
v0->v1: Added information for TEST_SKIPPED and details about generating
        code coverage to help with ideas for writing unit test cases.
v1->v2: Corrected some spelling, rephrased a bit after suggestions by
        Ray.
v2->v3: Rewrite the meson build block, updated the copyright section,
        and change the title to be a bit nicer.
v3->v4: Dropped 'reviewed-by' tag from David Marchand.
        Incorporated suggestions from Ferruh.
        Clarified 'meson test suite' vs. 'dpdk test suite'.
        Added a bit on when to add tests to meson.
        Renamed to 'unit test' to be more specific.
        Added section about passing test names as command arguments.

 doc/guides/contributing/index.rst     |   1 +
 doc/guides/contributing/unit_test.rst | 327 ++++++++++++++++++++++++++
 2 files changed, 328 insertions(+)
 create mode 100644 doc/guides/contributing/unit_test.rst

diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst
index 2fefd91931..493732a518 100644
--- a/doc/guides/contributing/index.rst
+++ b/doc/guides/contributing/index.rst
@@ -14,6 +14,7 @@ Contributor's Guidelines
     abi_versioning
     documentation
     patches
+    unit_test
     vulnerability
     stable
     cheatsheet
diff --git a/doc/guides/contributing/unit_test.rst b/doc/guides/contributing/unit_test.rst
new file mode 100644
index 0000000000..b274bd5f93
--- /dev/null
+++ b/doc/guides/contributing/unit_test.rst
@@ -0,0 +1,327 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright 2021 The DPDK contributors
+
+DPDK Unit Testing Guidelines
+============================
+
+This document outlines the guidelines for running and adding new
+tests to the in-tree DPDK test suites.
+
+The DPDK test suite model is loosely based on the xunit model, where
+tests are grouped into test suites, and suites are run by runners.
+For a basic overview, see the basic Wikipedia article on xunit:
+`xUnit - Wikipedia <https://en.wikipedia.org/wiki/XUnit>`_.
+
+
+Background
+----------
+
+The in-tree testing infrastructure for DPDK consists of
+multiple applications and support tools.  The primary tools
+are the `dpdk-test` application, and the ``meson test``
+infrastructure.  These two are the primary ways through which
+a user will interact with the DPDK testing infrastructure.
+
+There exists a bit of confusion with the test suite and test case
+separation with respect to `dpdk-test` and ``meson test``.  Both
+have a concept of test suite and test case.  In both, the concept
+is similar.  A test suite is a group of test cases, and a test case
+represents the steps needed to test a particular set of code.  Where
+needed, they will be disambiguated by the word ``meson`` to denote
+a meson test suite / case.
+
+
+Running a test
+--------------
+
+DPDK tests are run via the main test runner, the `dpdk-test` app.
+The `dpdk-test` app is a command-line interface that facilitates
+running various tests or test suites.
+
+There are three modes of operation.  The first mode is as an
+interactive command shell that allows launching specific test
+suites.  This is the default operating mode of `dpdk-test` and
+can be done by::
+
+  $ ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>
+
+At the prompt, simply type the name of the test suite you wish to run
+and it will execute.
+
+The second form is useful for a scripting environment, and is used by
+the DPDK meson build system.  This mode is invoked by assigning a
+specific test suite name to the environment variable `DPDK_TEST`
+before invoking the `dpdk-test` command, such as::
+
+  $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>version_autotest
+  Version string: 'DPDK 20.02.0-rc0'
+  Test OK
+  RTE>>$
+
+The above shows running a specific test case.  On success, the return
+code will be '0', otherwise it will be set to some error value (such
+as '255', or a negative value).
+
+The third form is an alternative to providing the test suite name in
+an environment variable.  The unit test app can accept test suite
+names via command line arguments::
+
+  $ ./build/app/test/dpdk-test --dpdk-options-here version_autotest version_autotest
+  EAL: Detected 8 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+  EAL: Detected static linkage of DPDK
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  TELEMETRY: No legacy callbacks, legacy socket not created
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>version_autotest
+  Version string: 'DPDK 21.08.0-rc0'
+  Test OK
+  RTE>>version_autotest
+  Version string: 'DPDK 21.08.0-rc0'
+  Test OK
+  RTE>>
+
+The primary benefit here is specifying multiple test names, which is
+not possible with the ``DPDK_TEST`` environment variable.
+
+Additionally, it is possible to specify additional test parameters via
+the ``DPDK_TEST_PARAMS`` argument, in case some tests need additional
+configuration.  This isn't currently used in the meson test suites.
+
+
+Running test cases via meson
+----------------------------
+
+In order to allow developers to quickly execute all the standard
+internal tests without needing to remember or look up each test suite
+name, the build system includes a standard way of executing the
+meson test suites.  After building via `ninja`, the ``meson test``
+command with no arguments will execute the meson test suites.
+
+There are four pre-configured meson test suites.  The first is
+the **fast** test suite, which is the largest group of test cases.
+These are the bulk of the unit tests to validate functional blocks.
+The second is the **perf** tests.  These test suites can take
+longer to run and do performance evaluations.  The third is
+the **driver** test suite, which is mostly for special hardware
+related testing (such as `cryptodev`).  The last suite is the
+**debug** suite.  These tests mostly are used to dump system
+information.
+
+The meson test suites can be selected by adding the ``--suite``
+option to the ``meson test`` command.
+Ex: ``meson test --suite fast-tests``::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
+  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
+  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
+  ...
+
+The ``meson test`` command can also execute individual meson test
+cases via the command line by adding the test names as an argument::
+
+  $ meson test -C build version_autotest
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/1 DPDK:fast-tests / version_autotest OK             0.17s
+  ...
+
+Note that these test cases must be known to meson for the ``meson test``
+command to run them.  Simply adding a new test to the `dpdk-test`
+application isn't enough.  See the section titled "Adding a suite or
+test case to meson" for more details.
+
+
+Adding tests to dpdk-test-app
+-----------------------------
+
+Unit tests should be added to the system whenever we introduce new
+functionality to DPDK, as well as whenever a bug is resolved.  This
+helps the DPDK project to catch regressions as they are introduced.
+
+The DPDK test application supports two layers of tests:
+ 1. *test cases* which are individual tests
+ 2. *test suites* which are groups of test cases
+
+To add a new test suite to the DPDK test application, create a new test
+file for that suite (ex: see *app/test/test_version.c* for the
+``version_autotest`` test suite).  There are two important functions
+for interacting with the test harness:
+
+  1. REGISTER_TEST_COMMAND(command_name, function_to_execute)
+     Registers a test command with the name `command_name` and which
+     runs the function `function_to_execute` when `command_name` is
+     invoked.
+
+  2. unit_test_suite_runner(struct unit_test_suite \*)
+     Returns a runner for a full test suite object, which contains
+     a test suite name, setup, tear down, and vector of unit test
+     cases.
+
+Each test suite has a setup and tear down function that runs at the
+beginning and end of the test suite execution.  Each unit test has
+a similar function for test case setup and tear down.
+
+Test cases are added to the `.unit_test_cases` element of the appropriate
+unit test suite structure.  An example of both a test suite and a case:
+
+.. code-block:: c
+   :linenos:
+
+   #include <time.h>
+
+   #include <rte_common.h>
+   #include <rte_cycles.h>
+   #include <rte_hexdump.h>
+   #include <rte_random.h>
+
+   #include "test.h"
+
+   static int testsuite_setup(void) { return TEST_SUCCESS; }
+   static void testsuite_teardown(void) { }
+
+   static int ut_setup(void) { return TEST_SUCCESS; }
+   static void ut_teardown(void) { }
+
+   static int test_case_first(void) { return TEST_SUCCESS; }
+
+   static struct unit_test_suite example_testsuite = {
+          .suite_name = "EXAMPLE TEST SUITE",
+          .setup = testsuite_setup,
+          .teardown = testsuite_teardown,
+          .unit_test_cases = {
+               TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
+
+               TEST_CASES_END(), /**< NULL terminate unit test array */
+          },
+   };
+
+   static int example_tests()
+   {
+       return unit_test_suite_runner(&example_testsuite);
+   }
+
+   REGISTER_TEST_COMMAND(example_autotest, example_tests);
+
+The above code block is a small example that can be used to create a
+complete test suite with test case.
+
+
+Designing a test
+----------------
+
+Test cases have multiple ways of indicating an error has occurred,
+in order to reflect failure state back to the runner.  Using the
+various methods of indicating errors can assist in not only validating
+the requisite functionality is working, but also to help debug when
+a change in environment or code has caused things to go wrong.
+
+The first way to indicate a generic error is by returning a test
+result failure, using the *TEST_FAILED* error code.  This is the most
+basic way of indicating that an error has occurred in a test routine.
+It isn't very informative to the user, so it should really be used in
+cases where the test has catastrophically failed.
+
+The preferred method of indicating an error is via the
+`RTE_TEST_ASSERT` family of macros, which will immediately return
+*TEST_FAILED* error condition, but will also log details about the
+failure.  The basic form is:
+
+.. code-block:: c
+
+   RTE_TEST_ASSERT(cond, msg, ...)
+
+In the above macro, *cond* is the condition to evaluate to **true**.
+Any generic condition can go here.  The *msg* parameter will be a
+message to display if *cond* evaluates to **false**.  Some specialized
+macros already exist.  See `lib/librte_eal/include/rte_test.h` for
+a list of defined test assertions.
+
+Sometimes it is important to indicate that a test needs to be
+skipped, either because the environment isn't able to support running
+the test, or because some requisite functionality isn't available.  The
+test suite supports returning a result of `TEST_SKIPPED` during test
+case setup, or during test case execution to indicate that the
+preconditions of the test aren't available.  Ex::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  ...
+  46/60 DPDK:fast-tests / ipsec_autotest       SKIP     0.22 s
+  ...
+
+
+Checking code coverage
+----------------------
+The meson build system supports generating a code coverage report
+via the `-Db_coverage=true` option, in conjunction with a package
+like **lcov**, to generate an HTML code coverage report.  Example::
+
+  $ meson setup build -Db_coverage=true
+  $ meson test -C build --suite fast-tests
+  $ ninja coverage-html -C build
+
+The above will generate an html report in the
+`build/meson-logs/coveragereport/` directory that can be explored
+for detailed code covered information.  This can be used to assist
+in test development.
+
+
+Adding a suite or test case to meson
+------------------------------------
+
+Adding to one of the meson test suites involves editing the appropriate
+meson build file `app/test/meson.build` and adding the command to
+the correct test suite class.  Once added, the new test will be run as
+part of the appropriate class (fast, perf, driver, etc.).
+
+A user or developer can confirm that a test is known to meson by
+using the ``--list`` option::
+
+  $ meson test -C build --list
+  DPDK:fast-tests / acl_autotest
+  DPDK:fast-tests / bitops_autotest
+  ...
+
+Some of these test suites are run during continuous integration
+tests, making regression checking automatic for new patches submitted
+to the project.
+
+In general, when a test is added to the `dpdk-test` application, it
+probably should be added to a meson test suite, but the choice is
+left to maintainers and individual developers.  Preference is to add
+tests to the meson test suites.
-- 
2.31.1

Re: [dpdk-dev] [PATCH v4] guides: add a guide for developing unit tests

[Power, Ciara]

Hi Aaron,

>From: Aaron Conole <aconole@redhat.com>
>Sent: Wednesday 14 July 2021 17:41
>To: dev@dpdk.org
>Cc: David Marchand <david.marchand@redhat.com>; Richardson, Bruce
><bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>;
>Power, Ciara <ciara.power@intel.com>
>Subject: [PATCH v4] guides: add a guide for developing unit tests
>
>The DPDK testing infrastructure includes a comprehensive set of libraries,
>utilities, and CI integrations for developers to test their code changes.  This
>isn't well documented, however.
>
>Document the basics for adding a test suite to the infrastructure and enabling
>that test suite for continuous integration platforms so that newer developers
>can understand how to develop test suites and test cases.
>
>Signed-off-by: Aaron Conole <aconole@redhat.com>
>---
<snip>

This is a useful doc, looks good.

Acked-by: Ciara Power <ciara.power@intel.com>

Re: [dpdk-dev] [PATCH v4] guides: add a guide for developing unit tests

[Zhang, Roy Fan]

Hi Aaron,

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Aaron Conole
> Sent: Wednesday, July 14, 2021 5:41 PM
> To: dev@dpdk.org
> Cc: David Marchand <david.marchand@redhat.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>;
> Power, Ciara <ciara.power@intel.com>
> Subject: [dpdk-dev] [PATCH v4] guides: add a guide for developing unit tests
> 
> The DPDK testing infrastructure includes a comprehensive set of
> libraries, utilities, and CI integrations for developers to test
> their code changes.  This isn't well documented, however.
> 
> Document the basics for adding a test suite to the infrastructure
> and enabling that test suite for continuous integration platforms
> so that newer developers can understand how to develop test suites
> and test cases.
> 
> Signed-off-by: Aaron Conole <aconole@redhat.com>
> ---

Acked-by: Fan Zhang <roy.fan.zhang@intel.com>

Re: [dpdk-dev] [PATCH v4] guides: add a guide for developing unit tests

[Mcnamara, John]

> +  $ ./build/app/test/dpdk-test --dpdk-options-here
> +  EAL: Detected 4 lcore(s)
> +  EAL: Detected 1 NUMA nodes
> +  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
> +  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
> +  EAL: Selected IOVA mode 'VA'
> +  EAL: Probing VFIO support...
> +  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
> +  EAL:   Invalid NUMA socket, default to 0
> +  EAL:   probe driver: 8086:15d7 net_e1000_em
> +  APP: HPET is not enabled, using TSC as default timer  RTE>>


Minor suggestion. You could truncate the long "Static memory" line in this and other blocks to prevent it invoking the scroll bars. Something, like this:

> +  EAL: Static memory layout is selected ...


Apart from that, great work on trawling through the weeds of the DPDK units test infrastructure and creating this really useful doc.

Thank you.

Acked-by: John McNamara <john.mcnamara@intel.com>

[dpdk-dev] [PATCH v5] guides: add a guide for developing unit tests

[Aaron Conole]

The DPDK testing infrastructure includes a comprehensive set of
libraries, utilities, and CI integrations for developers to test
their code changes.  This isn't well documented, however.

Document the basics for adding a test suite to the infrastructure
and enabling that test suite for continuous integration platforms
so that newer developers can understand how to develop test suites
and test cases.

Signed-off-by: Aaron Conole <aconole@redhat.com>
Acked-by: Ciara Power <ciara.power@intel.com>
Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: John McNamara <john.mcnamara@intel.com>
---
v0->v1: Added information for TEST_SKIPPED and details about generating
        code coverage to help with ideas for writing unit test cases.
v1->v2: Corrected some spelling, rephrased a bit after suggestions by
        Ray.
v2->v3: Rewrite the meson build block, updated the copyright section,
        and change the title to be a bit nicer.
v3->v4: Dropped 'reviewed-by' tag from David Marchand.
        Incorporated suggestions from Ferruh.
        Clarified 'meson test suite' vs. 'dpdk test suite'.
        Added a bit on when to add tests to meson.
        Renamed to 'unit test' to be more specific.
        Added section about passing test names as command arguments.
v4->v5: Update by trimming long line as suggested by John McNamara

 doc/guides/contributing/index.rst     |   1 +
 doc/guides/contributing/unit_test.rst | 327 ++++++++++++++++++++++++++
 2 files changed, 328 insertions(+)
 create mode 100644 doc/guides/contributing/unit_test.rst

diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst
index 2fefd91931..493732a518 100644
--- a/doc/guides/contributing/index.rst
+++ b/doc/guides/contributing/index.rst
@@ -14,6 +14,7 @@ Contributor's Guidelines
     abi_versioning
     documentation
     patches
+    unit_test
     vulnerability
     stable
     cheatsheet
diff --git a/doc/guides/contributing/unit_test.rst b/doc/guides/contributing/unit_test.rst
new file mode 100644
index 0000000000..b207b5aade
--- /dev/null
+++ b/doc/guides/contributing/unit_test.rst
@@ -0,0 +1,327 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright 2021 The DPDK contributors
+
+DPDK Unit Testing Guidelines
+============================
+
+This document outlines the guidelines for running and adding new
+tests to the in-tree DPDK test suites.
+
+The DPDK test suite model is loosely based on the xunit model, where
+tests are grouped into test suites, and suites are run by runners.
+For a basic overview, see the basic Wikipedia article on xunit:
+`xUnit - Wikipedia <https://en.wikipedia.org/wiki/XUnit>`_.
+
+
+Background
+----------
+
+The in-tree testing infrastructure for DPDK consists of
+multiple applications and support tools.  The primary tools
+are the `dpdk-test` application, and the ``meson test``
+infrastructure.  These two are the primary ways through which
+a user will interact with the DPDK testing infrastructure.
+
+There exists a bit of confusion with the test suite and test case
+separation with respect to `dpdk-test` and ``meson test``.  Both
+have a concept of test suite and test case.  In both, the concept
+is similar.  A test suite is a group of test cases, and a test case
+represents the steps needed to test a particular set of code.  Where
+needed, they will be disambiguated by the word ``meson`` to denote
+a meson test suite / case.
+
+
+Running a test
+--------------
+
+DPDK tests are run via the main test runner, the `dpdk-test` app.
+The `dpdk-test` app is a command-line interface that facilitates
+running various tests or test suites.
+
+There are three modes of operation.  The first mode is as an
+interactive command shell that allows launching specific test
+suites.  This is the default operating mode of `dpdk-test` and
+can be done by::
+
+  $ ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory...
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>
+
+At the prompt, simply type the name of the test suite you wish to run
+and it will execute.
+
+The second form is useful for a scripting environment, and is used by
+the DPDK meson build system.  This mode is invoked by assigning a
+specific test suite name to the environment variable `DPDK_TEST`
+before invoking the `dpdk-test` command, such as::
+
+  $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be...
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>version_autotest
+  Version string: 'DPDK 20.02.0-rc0'
+  Test OK
+  RTE>>$
+
+The above shows running a specific test case.  On success, the return
+code will be '0', otherwise it will be set to some error value (such
+as '255', or a negative value).
+
+The third form is an alternative to providing the test suite name in
+an environment variable.  The unit test app can accept test suite
+names via command line arguments::
+
+  $ ./build/app/test/dpdk-test --dpdk-options-here version_autotest version_autotest
+  EAL: Detected 8 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be...
+  EAL: Detected static linkage of DPDK
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  TELEMETRY: No legacy callbacks, legacy socket not created
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>version_autotest
+  Version string: 'DPDK 21.08.0-rc0'
+  Test OK
+  RTE>>version_autotest
+  Version string: 'DPDK 21.08.0-rc0'
+  Test OK
+  RTE>>
+
+The primary benefit here is specifying multiple test names, which is
+not possible with the ``DPDK_TEST`` environment variable.
+
+Additionally, it is possible to specify additional test parameters via
+the ``DPDK_TEST_PARAMS`` argument, in case some tests need additional
+configuration.  This isn't currently used in the meson test suites.
+
+
+Running test cases via meson
+----------------------------
+
+In order to allow developers to quickly execute all the standard
+internal tests without needing to remember or look up each test suite
+name, the build system includes a standard way of executing the
+meson test suites.  After building via `ninja`, the ``meson test``
+command with no arguments will execute the meson test suites.
+
+There are four pre-configured meson test suites.  The first is
+the **fast** test suite, which is the largest group of test cases.
+These are the bulk of the unit tests to validate functional blocks.
+The second is the **perf** tests.  These test suites can take
+longer to run and do performance evaluations.  The third is
+the **driver** test suite, which is mostly for special hardware
+related testing (such as `cryptodev`).  The last suite is the
+**debug** suite.  These tests mostly are used to dump system
+information.
+
+The meson test suites can be selected by adding the ``--suite``
+option to the ``meson test`` command.
+Ex: ``meson test --suite fast-tests``::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
+  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
+  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
+  ...
+
+The ``meson test`` command can also execute individual meson test
+cases via the command line by adding the test names as an argument::
+
+  $ meson test -C build version_autotest
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/1 DPDK:fast-tests / version_autotest OK             0.17s
+  ...
+
+Note that these test cases must be known to meson for the ``meson test``
+command to run them.  Simply adding a new test to the `dpdk-test`
+application isn't enough.  See the section titled "Adding a suite or
+test case to meson" for more details.
+
+
+Adding tests to dpdk-test-app
+-----------------------------
+
+Unit tests should be added to the system whenever we introduce new
+functionality to DPDK, as well as whenever a bug is resolved.  This
+helps the DPDK project to catch regressions as they are introduced.
+
+The DPDK test application supports two layers of tests:
+ 1. *test cases* which are individual tests
+ 2. *test suites* which are groups of test cases
+
+To add a new test suite to the DPDK test application, create a new test
+file for that suite (ex: see *app/test/test_version.c* for the
+``version_autotest`` test suite).  There are two important functions
+for interacting with the test harness:
+
+  1. REGISTER_TEST_COMMAND(command_name, function_to_execute)
+     Registers a test command with the name `command_name` and which
+     runs the function `function_to_execute` when `command_name` is
+     invoked.
+
+  2. unit_test_suite_runner(struct unit_test_suite \*)
+     Returns a runner for a full test suite object, which contains
+     a test suite name, setup, tear down, and vector of unit test
+     cases.
+
+Each test suite has a setup and tear down function that runs at the
+beginning and end of the test suite execution.  Each unit test has
+a similar function for test case setup and tear down.
+
+Test cases are added to the `.unit_test_cases` element of the appropriate
+unit test suite structure.  An example of both a test suite and a case:
+
+.. code-block:: c
+   :linenos:
+
+   #include <time.h>
+
+   #include <rte_common.h>
+   #include <rte_cycles.h>
+   #include <rte_hexdump.h>
+   #include <rte_random.h>
+
+   #include "test.h"
+
+   static int testsuite_setup(void) { return TEST_SUCCESS; }
+   static void testsuite_teardown(void) { }
+
+   static int ut_setup(void) { return TEST_SUCCESS; }
+   static void ut_teardown(void) { }
+
+   static int test_case_first(void) { return TEST_SUCCESS; }
+
+   static struct unit_test_suite example_testsuite = {
+          .suite_name = "EXAMPLE TEST SUITE",
+          .setup = testsuite_setup,
+          .teardown = testsuite_teardown,
+          .unit_test_cases = {
+               TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
+
+               TEST_CASES_END(), /**< NULL terminate unit test array */
+          },
+   };
+
+   static int example_tests()
+   {
+       return unit_test_suite_runner(&example_testsuite);
+   }
+
+   REGISTER_TEST_COMMAND(example_autotest, example_tests);
+
+The above code block is a small example that can be used to create a
+complete test suite with test case.
+
+
+Designing a test
+----------------
+
+Test cases have multiple ways of indicating an error has occurred,
+in order to reflect failure state back to the runner.  Using the
+various methods of indicating errors can assist in not only validating
+the requisite functionality is working, but also to help debug when
+a change in environment or code has caused things to go wrong.
+
+The first way to indicate a generic error is by returning a test
+result failure, using the *TEST_FAILED* error code.  This is the most
+basic way of indicating that an error has occurred in a test routine.
+It isn't very informative to the user, so it should really be used in
+cases where the test has catastrophically failed.
+
+The preferred method of indicating an error is via the
+`RTE_TEST_ASSERT` family of macros, which will immediately return
+*TEST_FAILED* error condition, but will also log details about the
+failure.  The basic form is:
+
+.. code-block:: c
+
+   RTE_TEST_ASSERT(cond, msg, ...)
+
+In the above macro, *cond* is the condition to evaluate to **true**.
+Any generic condition can go here.  The *msg* parameter will be a
+message to display if *cond* evaluates to **false**.  Some specialized
+macros already exist.  See `lib/librte_eal/include/rte_test.h` for
+a list of defined test assertions.
+
+Sometimes it is important to indicate that a test needs to be
+skipped, either because the environment isn't able to support running
+the test, or because some requisite functionality isn't available.  The
+test suite supports returning a result of `TEST_SKIPPED` during test
+case setup, or during test case execution to indicate that the
+preconditions of the test aren't available.  Ex::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  ...
+  46/60 DPDK:fast-tests / ipsec_autotest       SKIP     0.22 s
+  ...
+
+
+Checking code coverage
+----------------------
+The meson build system supports generating a code coverage report
+via the `-Db_coverage=true` option, in conjunction with a package
+like **lcov**, to generate an HTML code coverage report.  Example::
+
+  $ meson setup build -Db_coverage=true
+  $ meson test -C build --suite fast-tests
+  $ ninja coverage-html -C build
+
+The above will generate an html report in the
+`build/meson-logs/coveragereport/` directory that can be explored
+for detailed code covered information.  This can be used to assist
+in test development.
+
+
+Adding a suite or test case to meson
+------------------------------------
+
+Adding to one of the meson test suites involves editing the appropriate
+meson build file `app/test/meson.build` and adding the command to
+the correct test suite class.  Once added, the new test will be run as
+part of the appropriate class (fast, perf, driver, etc.).
+
+A user or developer can confirm that a test is known to meson by
+using the ``--list`` option::
+
+  $ meson test -C build --list
+  DPDK:fast-tests / acl_autotest
+  DPDK:fast-tests / bitops_autotest
+  ...
+
+Some of these test suites are run during continuous integration
+tests, making regression checking automatic for new patches submitted
+to the project.
+
+In general, when a test is added to the `dpdk-test` application, it
+probably should be added to a meson test suite, but the choice is
+left to maintainers and individual developers.  Preference is to add
+tests to the meson test suites.
-- 
2.31.1

Re: [dpdk-dev] [PATCH v5] guides: add a guide for developing unit tests

[Thomas Monjalon]

15/10/2021 19:06, Aaron Conole:
> The DPDK testing infrastructure includes a comprehensive set of
> libraries, utilities, and CI integrations for developers to test
> their code changes.  This isn't well documented, however.
> 
> Document the basics for adding a test suite to the infrastructure
> and enabling that test suite for continuous integration platforms
> so that newer developers can understand how to develop test suites
> and test cases.
> 
> Signed-off-by: Aaron Conole <aconole@redhat.com>
> Acked-by: Ciara Power <ciara.power@intel.com>
> Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
> Acked-by: John McNamara <john.mcnamara@intel.com>

Applied with some formatting changes, and added recent "extra" suite, thanks.