From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id 9EAF9A0A0C;
Mon, 17 May 2021 20:01:52 +0200 (CEST)
Received: from [217.70.189.124] (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id 191954014E;
Mon, 17 May 2021 20:01:52 +0200 (CEST)
Received: from us-smtp-delivery-124.mimecast.com
(us-smtp-delivery-124.mimecast.com [216.205.24.124])
by mails.dpdk.org (Postfix) with ESMTP id A3E7040041
for <dev@dpdk.org>; Mon, 17 May 2021 20:01:50 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
s=mimecast20190719; t=1621274509;
h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
to:to:cc:cc:mime-version:mime-version:content-type:content-type:
in-reply-to:in-reply-to:references:references;
bh=MB4ZpjMXJwbXumGna2UXzj55bxU/G3UXBc5+qHtAHUY=;
b=LT1vzSNIAJI7F7wge0NKSRH4/D7DnfDvWsKaLKsSr9Z/V8P44CDzdLWvo2kjjxH4wJ8qaU
YH81eJ8h2MERkOCeSlvEYQMbS9bAP979t7EDqP1U0CgG5NdXl0imYnN7Qba90u6oPG3zPq
JxhnocMm82f8IMkexsrqHuzJhhGSOF4=
Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com
[209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id
us-mta-155--DL432JeNDKBFPJCgzBw7w-1; Mon, 17 May 2021 14:01:47 -0400
X-MC-Unique: -DL432JeNDKBFPJCgzBw7w-1
Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com
[10.5.11.12])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BFACD801FAF;
Mon, 17 May 2021 18:01:46 +0000 (UTC)
Received: from RHTPC1VM0NT (ovpn-119-199.rdu2.redhat.com [10.10.119.199])
by smtp.corp.redhat.com (Postfix) with ESMTPS id DD1EE60BE5;
Mon, 17 May 2021 18:01:45 +0000 (UTC)
From: Aaron Conole <aconole@redhat.com>
To: Ciara Power <ciara.power@intel.com>
Cc: dev@dpdk.org, roy.fan.zhang@intel.com, declan.doherty@intel.com
References: <20210517155517.806841-1-ciara.power@intel.com>
Date: Mon, 17 May 2021 14:01:39 -0400
In-Reply-To: <20210517155517.806841-1-ciara.power@intel.com> (Ciara Power's
message of "Mon, 17 May 2021 15:55:17 +0000")
Message-ID: <f7to8d9fddo.fsf@redhat.com>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12
Authentication-Results: relay.mimecast.com;
auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=aconole@redhat.com
X-Mimecast-Spam-Score: 0
X-Mimecast-Originator: redhat.com
Content-Type: text/plain
Subject: Re: [dpdk-dev] [PATCH] doc/guides: add details for new test
structure
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>
Ciara Power <ciara.power@intel.com> writes:
> The testing guide is now updated to include details about
> using sub-testsuites. Some example code is given to demonstrate how
> they can be used.
>
> A note is also added to highlight the need for using vdev EAL args when
> running cryptodev tests.
>
> Depends-on: patch-88751 ("guides: add a guide for developing unit tests")
>
> Signed-off-by: Ciara Power <ciara.power@intel.com>
> ---
> doc/guides/contributing/testing.rst | 89 ++++++++++++++++++++++++++++-
> 1 file changed, 87 insertions(+), 2 deletions(-)
>
> diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
> index 0757d71ad0..1188d63a97 100644
> --- a/doc/guides/contributing/testing.rst
> +++ b/doc/guides/contributing/testing.rst
> @@ -114,13 +114,21 @@ for interacting with the test harness:
>
> 2. unit_test_suite_runner(struct unit_test_suite \*)
> Returns a runner for a full test suite object, which contains
> - a test suite name, setup, tear down, and vector of unit test
> - cases.
> + a test suite name, setup, tear down, a pointer to a list of
> + sub-testsuites, and vector of unit test cases.
>
> Each test suite has a setup and tear down function that runs at the
> beginning and end of the test suite execution. Each unit test has
> a similar function for test case setup and tear down.
>
> +Each testsuite may use a nested list of sub-testsuites,
> +which are iterated by the unit_test_suite_runner.
> +This support allows for better granularity when designing test suites.
> +The sub-testsuites list can also be used in parallel with the vector
> +of testcases, in this case the testcases will be run,
> +and then each sub-testsuite is executed. To see an example of a
> +testsuite using sub-testsuites, see *app/test/test_cryptodev.c*.
> +
> Test cases are added to the `.unit_test_cases` element of the unit
> test suite structure. Ex:
>
> @@ -165,6 +173,70 @@ test suite structure. Ex:
> The above code block is a small example that can be used to create a
> complete test suite with test case.
>
> +Sub-testsuites can be added to the `.unit_test_suites` element of
> +the unit test suite structure. Ex:
> +
> +.. code-block:: c
> + :linenos:
> +
> + static int testsuite_setup(void) { return TEST_SUCCESS; }
> + static void testsuite_teardown(void) { }
> +
> + static int ut_setup(void) { return TEST_SUCCESS; }
> + static void ut_teardown(void) { }
> +
> + static int test_case_first(void) { return TEST_SUCCESS; }
> +
> + static struct unit_test_suite example_parent_testsuite = {
> + .suite_name = "EXAMPLE PARENT TEST SUITE",
> + .setup = testsuite_setup,
> + .teardown = testsuite_teardown,
> + .unit_test_cases = {TEST_CASES_END()}
> + };
> +
> + static int sub_testsuite_setup(void) { return TEST_SUCCESS; }
> + static void sub_testsuite_teardown(void) { }
> +
> + static struct unit_test_suite example_sub_testsuite = {
> + .suite_name = "EXAMPLE SUB TEST SUITE",
> + .setup = sub_testsuite_setup,
> + .teardown = sub_testsuite_teardown,
> + .unit_test_cases = {
> + TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
> +
> + TEST_CASES_END(), /**< NULL terminate unit test array */
> + },
> + };
> +
> + static struct unit_test_suite end_testsuite = {
> + .suite_name = NULL,
> + .setup = NULL,
> + .teardown = NULL,
> + .unit_test_suites = NULL
> + };
> +
> + static int example_tests()
> + {
> + uint8_t ret, i = 0;
> + struct unit_test_suite *sub_suites[] = {
> + &example_sub_testsuite,
> + &end_testsuite /**< NULL test suite to indicate end of list */
> + };
> +
> + example_parent_testsuite.unit_test_suites =
> + malloc(sizeof(struct unit_test_suite *) * RTE_DIM(sub_suites));
> +
> + for (i = 0; i < RTE_DIM(sub_suites); i++)
> + example_parent_testsuite.unit_test_suites[i] = sub_suites[i];
> +
> + ret = unit_test_suite_runner(&example_parent_testsuite);
> + free(example_parent_testsuite.unit_test_suites);
> +
> + return ret;
> + }
> +
> + REGISTER_TEST_COMMAND(example_autotest, example_tests);
> +
>
> Designing a test
> ----------------
> @@ -241,3 +313,16 @@ be run as part of the appropriate class (fast, perf, driver, etc.).
> Some of these default test suites are run during continuous integration
> tests, making regression checking automatic for new patches submitted
> to the project.
> +
> +
> +Running Cryptodev Tests
> +-----------------------
> +
> +When running cryptodev tests, the user must create any required virtual
> +device via EAL args, as this is not automatically done by the test.
How does this differ from just running the tests? Will I get 'SKIP' or
'FAIL' if I don't do this? Should the cryptodev tests stay as part of
the driver test suite or be moved out?
> +
> +.. note::
> +
> + The cryptodev_scheduler_autotest is the only exception to this.
> + This vdev will be created automatically by the test app,
> + as it requires a more complex setup than other vdevs.