From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 788E8A0C42; Tue, 30 Mar 2021 18:15:26 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3C1CB140E8C; Tue, 30 Mar 2021 18:15:26 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 57F7D406B4 for ; Tue, 30 Mar 2021 18:15:24 +0200 (CEST) IronPort-SDR: Z3VM+6Yaln7IBNHuWb4bvU22OfwDogAn2uLMLt6MRCAiyz6vCfulovQev+ShERdwfcY+vbyRcG leSNzmgGh4Dg== X-IronPort-AV: E=McAfee;i="6000,8403,9939"; a="191276188" X-IronPort-AV: E=Sophos;i="5.81,291,1610438400"; d="scan'208";a="191276188" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Mar 2021 09:15:23 -0700 IronPort-SDR: K5NY4QkxCw0TnfcroDSXbaHQx6C2cHj4f4tjcZo4+TrsIj6jtBU6jJTgammC5SeGikTZd6ctAf IqRYLkynFDTA== X-IronPort-AV: E=Sophos;i="5.81,291,1610438400"; d="scan'208";a="418235670" Received: from dwdohert-mobl1.ger.corp.intel.com (HELO [10.213.212.157]) ([10.213.212.157]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Mar 2021 09:15:22 -0700 To: Ciara Power , dev@dpdk.org, Aaron Conole References: <20210316143253.3849182-1-ciara.power@intel.com> From: "Doherty, Declan" Message-ID: Date: Tue, 30 Mar 2021 17:15:17 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.9.0 MIME-Version: 1.0 In-Reply-To: <20210316143253.3849182-1-ciara.power@intel.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH 0/6] test: refactor crypto unit test framework X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hey Aaron, based on the work you've been doing on the unit test documentation we would appreciate if you could take a look over this patchset and get your thoughts. The primary drive here is to make it easier to get a clear picture of what is being executed in the cryptodev testsuite, as at the moment there are test suites masquerading as unit tests and the output doesn't reflect the actual number of unit tests being executed. Thanks Declan On 16/03/2021 2:32 PM, Ciara Power wrote: > The current crypto unit test framework is not granular enough to > accurately track unit test results. This is caused by one testcase > in a suite actually running multiple testcases, but only returning > one result. > > The approach taken in this patchset allows a test suite have either a > list of sub-testsuites, or a list of testcases as previously used. > The unit test suite runner can then recursively iterate and run the > sub-testsuites, until it reaches a suite with testcases, > and it then runs each testcase as it had done previously. > > By allowing this further breakdown into sub-testsuites, > a refactor of the crypto unit tests solves the issue of inaccurate > reporting, as sub-testsuites can be used in place of the testcases > that had multiple testcases hidden on a sub level. > The blockcipher tests previously had these hidden testcases, > but are now sub-testsuites that are dynamically created and added to a > parent test suite, allowing for each testcase status to be reported > directly to the runner. > The cryptodev test suite is broken down into smaller suites that are > used as sub-testsuites, which allows for more flexibility choosing which > sub-testsuites should run for the current device autotest. > The introduction of sub-testsuites also allows for more precise > setup/teardown functions, rather than general ones loaded with > conditions as was seen with the initial setup function used for all > crypto testsuites. > > For example, when running the cryptodev_aesni_mb_autotest, > the AESNI MB parent test suite has its own setup function to initialise > the AESNI MB device. > Various sub-testsuites are added to the parent test suite, such as some > of the static suites that were once in the cryptodev_testsuite, > and blockcipher suites. > The unit test runner can then run the AESNI MB parent test suite, > which in turn will run the sub-testsuites. > > Documentation will be added in a later version of the patchset, > adding to the test document that isn't yet merged. [1] > > --- > [1] https://patchwork.dpdk.org/project/dpdk/patch/20210309155757.615536-1-aconole@redhat.com/ > > Ciara Power (6): > app/test: refactor of unit test suite runner > test: introduce parent testsuite format > test/crypto: refactor to use sub-testsuites > test/crypto: move testsuite params to header file > test/crypto: dynamically build blockcipher suite > doc: add unit test suite change to release notes > > app/test/test.c | 168 +- > app/test/test.h | 22 +- > app/test/test_bitratestats.c | 4 +- > app/test/test_compressdev.c | 4 +- > app/test/test_cryptodev.c | 2020 ++++++++++++++++++------ > app/test/test_cryptodev.h | 20 + > app/test/test_cryptodev_asym.c | 105 +- > app/test/test_cryptodev_blockcipher.c | 121 +- > app/test/test_cryptodev_blockcipher.h | 12 +- > app/test/test_ethdev_link.c | 4 +- > app/test/test_event_crypto_adapter.c | 4 +- > app/test/test_event_eth_rx_adapter.c | 8 +- > app/test/test_event_eth_tx_adapter.c | 4 +- > app/test/test_event_timer_adapter.c | 4 +- > app/test/test_eventdev.c | 4 +- > app/test/test_fbarray.c | 4 +- > app/test/test_fib.c | 8 +- > app/test/test_fib6.c | 8 +- > app/test/test_graph.c | 4 +- > app/test/test_graph_perf.c | 4 +- > app/test/test_ipfrag.c | 4 +- > app/test/test_ipsec.c | 36 +- > app/test/test_ipsec_sad.c | 4 +- > app/test/test_latencystats.c | 4 +- > app/test/test_link_bonding.c | 4 +- > app/test/test_link_bonding_mode4.c | 4 +- > app/test/test_link_bonding_rssconf.c | 4 +- > app/test/test_metrics.c | 4 +- > app/test/test_pmd_ring.c | 4 +- > app/test/test_reorder.c | 4 +- > app/test/test_rib.c | 8 +- > app/test/test_rib6.c | 8 +- > app/test/test_security.c | 4 +- > app/test/test_service_cores.c | 4 +- > app/test/test_trace.c | 4 +- > doc/guides/rel_notes/release_21_05.rst | 5 + > 36 files changed, 1898 insertions(+), 739 deletions(-) >