From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 04F1AA0546; Fri, 30 Apr 2021 23:00:12 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DC83B4014F; Fri, 30 Apr 2021 23:00:11 +0200 (CEST) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id AD0524013F for ; Fri, 30 Apr 2021 23:00:09 +0200 (CEST) IronPort-SDR: nKl11SBy52BU5gc8RlpSnWZvD36yBjB+IV7YTiIPKS15UnWseAeEM5h+JeOc6JV1Fs0yzkd5TJ aBGOAoPNueXw== X-IronPort-AV: E=McAfee;i="6200,9189,9970"; a="258648330" X-IronPort-AV: E=Sophos;i="5.82,263,1613462400"; d="scan'208";a="258648330" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Apr 2021 14:00:08 -0700 IronPort-SDR: uIA7oEUiMHXQ3BMBOPLCsX7pHdHeT0pEbmhlCsaRnkikPDOH4OUZWgeuSRYRzphoiESpjs/kV9 RfIccvsH+Z8g== X-IronPort-AV: E=Sophos;i="5.82,263,1613462400"; d="scan'208";a="431617494" Received: from dwdohert-mobl1.ger.corp.intel.com (HELO [10.213.192.57]) ([10.213.192.57]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Apr 2021 14:00:05 -0700 To: Ciara Power , dev@dpdk.org Cc: thomas@monjalon.net, gakhil@marvell.com, aconole@redhat.com, hemant.agrawal@nxp.com, anoobj@marvell.com, ruifeng.wang@arm.com, asomalap@amd.com, ajit.khaparde@broadcom.com, g.singh@nxp.com References: <20210423161820.2135053-1-ciara.power@intel.com> From: "Doherty, Declan" Message-ID: <365ff71b-92fd-dd9d-9bea-0b579629c810@intel.com> Date: Fri, 30 Apr 2021 22:00:03 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.10.0 MIME-Version: 1.0 In-Reply-To: <20210423161820.2135053-1-ciara.power@intel.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v3 0/7] test: refactor crypto unit test framework X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 23/04/2021 5:18 PM, Ciara Power wrote: > The current crypto unit test framework is not granular enough to > accurately track unit test results. This is caused by one testcase > in a suite actually running multiple testcases, but only returning > one result. > > The approach taken in this patchset allows a test suite have a > list of sub-testsuites, and/or a list of testcases as previously used. > The unit test suite runner can then recursively iterate and run the > sub-testsuites, until it reaches a suite with testcases, > and it then runs each testcase as it had done previously. > In the case of a testsuite with both testcases and sub-testsuites, > the testcases are executed first before iterating through the > sub-testsuites. > > By allowing this further breakdown into sub-testsuites, > a refactor of the crypto unit tests solves the issue of inaccurate > reporting, as sub-testsuites can be used in place of the testcases > that had multiple testcases hidden on a sub level. > The blockcipher tests previously had these hidden testcases, > but are now sub-testsuites that are dynamically created and added to a > parent test suite, allowing for each testcase status to be reported > directly to the runner. > The cryptodev test suite is broken down into smaller suites that are > used as sub-testsuites, which allows for more flexibility choosing which > sub-testsuites should run for the current device. > The introduction of sub-testsuites also allows for more precise > setup/teardown functions, that can check the capabilities required to > run its testcases. > > For example, when running the cryptodev_aesni_mb_autotest, > the parent Cryptodev Test Suite is executed. > Various sub-testsuites are added to the parent test suite, such as > the static suites of testcases that were once in the cryptodev_testsuite, > and blockcipher suites. > The unit test runner can then run the Cryptodev parent test suite, > which in turn will run the sub-testsuites. > > The user is now required to create vdevs via EAL commandline args, > this is no longer done within the test app for crypto autotests. > > Documentation will need to be added at a later stage, > adding to the test document that isn't yet merged. [1] > > [1] https://patchwork.dpdk.org/project/dpdk/patch/20210309155757.615536-1-aconole@redhat.com/ > > > v3: > - Added support for a testsuite having both a list of testcases, > and a list of sub-testsuites. > - Replaced PMD based parent testsuites with a cryptodev testsuite > used by all autotests, with the exception of scheduler autotest. > - Setup functions were added for all sub-testsuites, within which > required capability support is checked. > - The setup functions no longer create vdevs if needed, > this must be done by the user when running the test. > - Patch added to standardise return values for skipped testcases. > v2: > - Added macro in place of testcase/testsuite loops. > - Added more detail in the summary output. > - Moved testcase counts to the testsuite structure. > - Flattened testsuite structure to remove union. > - Added patch for fix of blockcipher test return value. > - Squashed release note into last patch. > > > Ciara Power (7): > app/test: refactor of unit test suite runner > test: introduce parent testsuite format > test/crypto: refactor to use sub-testsuites > test/crypto: replace unsupported with skipped > test/crypto: move testsuite params to header file > test/crypto: fix return value on test skipped > test/crypto: dynamically build blockcipher suite > > app/test/test.c | 176 +- > app/test/test.h | 23 +- > app/test/test_cryptodev.c | 2324 ++++++++++++++---------- > app/test/test_cryptodev.h | 32 + > app/test/test_cryptodev_asym.c | 111 +- > app/test/test_cryptodev_blockcipher.c | 423 ++++- > app/test/test_cryptodev_blockcipher.h | 12 +- > app/test/test_ipsec.c | 32 +- > doc/guides/rel_notes/release_21_05.rst | 5 + > 9 files changed, 2000 insertions(+), 1138 deletions(-) > Series Acked-by: Declan Doherty