From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 31704A0C43; Wed, 12 May 2021 13:37:41 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E3D5041106; Wed, 12 May 2021 13:37:27 +0200 (CEST) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id AB1DC4003E for ; Wed, 12 May 2021 13:37:24 +0200 (CEST) IronPort-SDR: eM8CJ1aUGTh1/XufuZRYndxWeE6Ka8RwZr11+ME51XbFkz0StNTB/8MSU4vkMYL0Zuy4dZaPIM cXKgLWeRynyw== X-IronPort-AV: E=McAfee;i="6200,9189,9981"; a="199733770" X-IronPort-AV: E=Sophos;i="5.82,293,1613462400"; d="scan'208";a="199733770" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 May 2021 04:37:07 -0700 IronPort-SDR: R2+QThTqMvoltFQnMl8mn1Gx1VnGjd9gxZsO5Qk6Q5+CdY00/wzRKz1I4uzX/RcBTRMBiyvM5V 7eKKajiM/yBw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,293,1613462400"; d="scan'208";a="625296843" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.223.148]) by fmsmga005.fm.intel.com with ESMTP; 12 May 2021 04:37:03 -0700 From: Ciara Power To: dev@dpdk.org Cc: declan.doherty@intel.com, gakhil@marvell.com, aconole@redhat.com, hemant.agrawal@nxp.com, anoobj@marvell.com, ruifeng.wang@arm.com, asomalap@amd.com, ajit.khaparde@broadcom.com, g.singh@nxp.com, roy.fan.zhang@intel.com, Ciara Power Date: Wed, 12 May 2021 11:36:48 +0000 Message-Id: <20210512113655.568814-1-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v4 0/7] test: refactor crypto unit test framework X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The current crypto unit test framework is not granular enough to accurately track unit test results. This is caused by one testcase in a suite actually running multiple testcases, but only returning one result. The approach taken in this patchset allows a test suite have a list of sub-testsuites, and/or a list of testcases as previously used. The unit test suite runner can then recursively iterate and run the sub-testsuites, until it reaches a suite with testcases, and it then runs each testcase as it had done previously. In the case of a testsuite with both testcases and sub-testsuites, the testcases are executed first before iterating through the sub-testsuites. By allowing this further breakdown into sub-testsuites, a refactor of the crypto unit tests solves the issue of inaccurate reporting, as sub-testsuites can be used in place of the testcases that had multiple testcases hidden on a sub level. The blockcipher tests previously had these hidden testcases, but are now sub-testsuites that are dynamically created and added to a parent test suite, allowing for each testcase status to be reported directly to the runner. The cryptodev test suite is broken down into smaller suites that are used as sub-testsuites, which allows for more flexibility choosing which sub-testsuites should run for the current device. The introduction of sub-testsuites also allows for more precise setup/teardown functions, that can check the capabilities required to run its testcases. For example, when running the cryptodev_aesni_mb_autotest, the parent Cryptodev Test Suite is executed. Various sub-testsuites are added to the parent test suite, such as the static suites of testcases that were once in the cryptodev_testsuite, and blockcipher suites. The unit test runner can then run the Cryptodev parent test suite, which in turn will run the sub-testsuites. The user is now required to create vdevs via EAL commandline args, this is no longer done within the test app for crypto autotests. Documentation will need to be added at a later stage, adding to the test document that isn't yet merged. [1] [1] https://patchwork.dpdk.org/project/dpdk/patch/20210309155757.615536-1-aconole@redhat.com/ v4: - Fixed some return value bugs related to patch 4. - Rebased onto main. v3: - Added support for a testsuite having both a list of testcases, and a list of sub-testsuites. - Replaced PMD based parent testsuites with a cryptodev testsuite used by all autotests, with the exception of scheduler autotest. - Setup functions were added for all sub-testsuites, within which required capability support is checked. - The setup functions no longer create vdevs if needed, this must be done by the user when running the test. - Patch added to standardise return values for skipped testcases. v2: - Added macro in place of testcase/testsuite loops. - Added more detail in the summary output. - Moved testcase counts to the testsuite structure. - Flattened testsuite structure to remove union. - Added patch for fix of blockcipher test return value. - Squashed release note into last patch. Ciara Power (7): app/test: refactor of unit test suite runner test: introduce parent testsuite format test/crypto: refactor to use sub-testsuites test/crypto: replace unsupported with skipped test/crypto: move testsuite params to header file test/crypto: fix return value on test skipped test/crypto: dynamically build blockcipher suite app/test/test.c | 176 +- app/test/test.h | 23 +- app/test/test_cryptodev.c | 2333 ++++++++++++++---------- app/test/test_cryptodev.h | 32 + app/test/test_cryptodev_asym.c | 111 +- app/test/test_cryptodev_blockcipher.c | 423 ++++- app/test/test_cryptodev_blockcipher.h | 12 +- app/test/test_ipsec.c | 32 +- doc/guides/rel_notes/release_21_05.rst | 5 + 9 files changed, 2004 insertions(+), 1143 deletions(-) -- 2.25.1