From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9EAF9A0A0C; Mon, 17 May 2021 20:01:52 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 191954014E; Mon, 17 May 2021 20:01:52 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id A3E7040041 for ; Mon, 17 May 2021 20:01:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1621274509; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=MB4ZpjMXJwbXumGna2UXzj55bxU/G3UXBc5+qHtAHUY=; b=LT1vzSNIAJI7F7wge0NKSRH4/D7DnfDvWsKaLKsSr9Z/V8P44CDzdLWvo2kjjxH4wJ8qaU YH81eJ8h2MERkOCeSlvEYQMbS9bAP979t7EDqP1U0CgG5NdXl0imYnN7Qba90u6oPG3zPq JxhnocMm82f8IMkexsrqHuzJhhGSOF4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-155--DL432JeNDKBFPJCgzBw7w-1; Mon, 17 May 2021 14:01:47 -0400 X-MC-Unique: -DL432JeNDKBFPJCgzBw7w-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BFACD801FAF; Mon, 17 May 2021 18:01:46 +0000 (UTC) Received: from RHTPC1VM0NT (ovpn-119-199.rdu2.redhat.com [10.10.119.199]) by smtp.corp.redhat.com (Postfix) with ESMTPS id DD1EE60BE5; Mon, 17 May 2021 18:01:45 +0000 (UTC) From: Aaron Conole To: Ciara Power Cc: dev@dpdk.org, roy.fan.zhang@intel.com, declan.doherty@intel.com References: <20210517155517.806841-1-ciara.power@intel.com> Date: Mon, 17 May 2021 14:01:39 -0400 In-Reply-To: <20210517155517.806841-1-ciara.power@intel.com> (Ciara Power's message of "Mon, 17 May 2021 15:55:17 +0000") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=aconole@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain Subject: Re: [dpdk-dev] [PATCH] doc/guides: add details for new test structure X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Ciara Power writes: > The testing guide is now updated to include details about > using sub-testsuites. Some example code is given to demonstrate how > they can be used. > > A note is also added to highlight the need for using vdev EAL args when > running cryptodev tests. > > Depends-on: patch-88751 ("guides: add a guide for developing unit tests") > > Signed-off-by: Ciara Power > --- > doc/guides/contributing/testing.rst | 89 ++++++++++++++++++++++++++++- > 1 file changed, 87 insertions(+), 2 deletions(-) > > diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst > index 0757d71ad0..1188d63a97 100644 > --- a/doc/guides/contributing/testing.rst > +++ b/doc/guides/contributing/testing.rst > @@ -114,13 +114,21 @@ for interacting with the test harness: > > 2. unit_test_suite_runner(struct unit_test_suite \*) > Returns a runner for a full test suite object, which contains > - a test suite name, setup, tear down, and vector of unit test > - cases. > + a test suite name, setup, tear down, a pointer to a list of > + sub-testsuites, and vector of unit test cases. > > Each test suite has a setup and tear down function that runs at the > beginning and end of the test suite execution. Each unit test has > a similar function for test case setup and tear down. > > +Each testsuite may use a nested list of sub-testsuites, > +which are iterated by the unit_test_suite_runner. > +This support allows for better granularity when designing test suites. > +The sub-testsuites list can also be used in parallel with the vector > +of testcases, in this case the testcases will be run, > +and then each sub-testsuite is executed. To see an example of a > +testsuite using sub-testsuites, see *app/test/test_cryptodev.c*. > + > Test cases are added to the `.unit_test_cases` element of the unit > test suite structure. Ex: > > @@ -165,6 +173,70 @@ test suite structure. Ex: > The above code block is a small example that can be used to create a > complete test suite with test case. > > +Sub-testsuites can be added to the `.unit_test_suites` element of > +the unit test suite structure. Ex: > + > +.. code-block:: c > + :linenos: > + > + static int testsuite_setup(void) { return TEST_SUCCESS; } > + static void testsuite_teardown(void) { } > + > + static int ut_setup(void) { return TEST_SUCCESS; } > + static void ut_teardown(void) { } > + > + static int test_case_first(void) { return TEST_SUCCESS; } > + > + static struct unit_test_suite example_parent_testsuite = { > + .suite_name = "EXAMPLE PARENT TEST SUITE", > + .setup = testsuite_setup, > + .teardown = testsuite_teardown, > + .unit_test_cases = {TEST_CASES_END()} > + }; > + > + static int sub_testsuite_setup(void) { return TEST_SUCCESS; } > + static void sub_testsuite_teardown(void) { } > + > + static struct unit_test_suite example_sub_testsuite = { > + .suite_name = "EXAMPLE SUB TEST SUITE", > + .setup = sub_testsuite_setup, > + .teardown = sub_testsuite_teardown, > + .unit_test_cases = { > + TEST_CASE_ST(ut_setup, ut_teardown, test_case_first), > + > + TEST_CASES_END(), /**< NULL terminate unit test array */ > + }, > + }; > + > + static struct unit_test_suite end_testsuite = { > + .suite_name = NULL, > + .setup = NULL, > + .teardown = NULL, > + .unit_test_suites = NULL > + }; > + > + static int example_tests() > + { > + uint8_t ret, i = 0; > + struct unit_test_suite *sub_suites[] = { > + &example_sub_testsuite, > + &end_testsuite /**< NULL test suite to indicate end of list */ > + }; > + > + example_parent_testsuite.unit_test_suites = > + malloc(sizeof(struct unit_test_suite *) * RTE_DIM(sub_suites)); > + > + for (i = 0; i < RTE_DIM(sub_suites); i++) > + example_parent_testsuite.unit_test_suites[i] = sub_suites[i]; > + > + ret = unit_test_suite_runner(&example_parent_testsuite); > + free(example_parent_testsuite.unit_test_suites); > + > + return ret; > + } > + > + REGISTER_TEST_COMMAND(example_autotest, example_tests); > + > > Designing a test > ---------------- > @@ -241,3 +313,16 @@ be run as part of the appropriate class (fast, perf, driver, etc.). > Some of these default test suites are run during continuous integration > tests, making regression checking automatic for new patches submitted > to the project. > + > + > +Running Cryptodev Tests > +----------------------- > + > +When running cryptodev tests, the user must create any required virtual > +device via EAL args, as this is not automatically done by the test. How does this differ from just running the tests? Will I get 'SKIP' or 'FAIL' if I don't do this? Should the cryptodev tests stay as part of the driver test suite or be moved out? > + > +.. note:: > + > + The cryptodev_scheduler_autotest is the only exception to this. > + This vdev will be created automatically by the test app, > + as it requires a more complex setup than other vdevs.