DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Doherty, Declan" <declan.doherty@intel.com>
To: Ciara Power <ciara.power@intel.com>, dev@dpdk.org
Cc: thomas@monjalon.net, gakhil@marvell.com, aconole@redhat.com,
	hemant.agrawal@nxp.com, anoobj@marvell.com, ruifeng.wang@arm.com,
	asomalap@amd.com, ajit.khaparde@broadcom.com, g.singh@nxp.com
Subject: Re: [dpdk-dev] [PATCH v3 0/7] test: refactor crypto unit test framework
Date: Fri, 30 Apr 2021 22:00:03 +0100	[thread overview]
Message-ID: <365ff71b-92fd-dd9d-9bea-0b579629c810@intel.com> (raw)
In-Reply-To: <20210423161820.2135053-1-ciara.power@intel.com>



On 23/04/2021 5:18 PM, Ciara Power wrote:
> The current crypto unit test framework is not granular enough to
> accurately track unit test results. This is caused by one testcase
> in a suite actually running multiple testcases, but only returning
> one result.
>   
> The approach taken in this patchset allows a test suite have a
> list of sub-testsuites, and/or a list of testcases as previously used.
> The unit test suite runner can then recursively iterate and run the
> sub-testsuites, until it reaches a suite with testcases,
> and it then runs each testcase as it had done previously.
> In the case of a testsuite with both testcases and sub-testsuites,
> the testcases are executed first before iterating through the
> sub-testsuites.
>   
> By allowing this further breakdown into sub-testsuites,
> a refactor of the crypto unit tests solves the issue of inaccurate
> reporting, as sub-testsuites can be used in place of the testcases
> that had multiple testcases hidden on a sub level.
> The blockcipher tests previously had these hidden testcases,
> but are now sub-testsuites that are dynamically created and added to a
> parent test suite, allowing for each testcase status to be reported
> directly to the runner.
> The cryptodev test suite is broken down into smaller suites that are
> used as sub-testsuites, which allows for more flexibility choosing which
> sub-testsuites should run for the current device.
> The introduction of sub-testsuites also allows for more precise
> setup/teardown functions, that can check the capabilities required to
> run its testcases.
>   
> For example, when running the cryptodev_aesni_mb_autotest,
> the parent Cryptodev Test Suite is executed.
> Various sub-testsuites are added to the parent test suite, such as
> the static suites of testcases that were once in the cryptodev_testsuite,
> and blockcipher suites.
> The unit test runner can then run the Cryptodev parent test suite,
> which in turn will run the sub-testsuites.
> 
> The user is now required to create vdevs via EAL commandline args,
> this is no longer done within the test app for crypto autotests.
> 
> Documentation will need to be added at a later stage,
> adding to the test document that isn't yet merged. [1]
> 
> [1] https://patchwork.dpdk.org/project/dpdk/patch/20210309155757.615536-1-aconole@redhat.com/
> 
> 
> v3:
>    - Added support for a testsuite having both a list of testcases,
>      and a list of sub-testsuites.
>    - Replaced PMD based parent testsuites with a cryptodev testsuite
>      used by all autotests, with the exception of scheduler autotest.
>    - Setup functions were added for all sub-testsuites, within which
>      required capability support is checked.
>    - The setup functions no longer create vdevs if needed,
>      this must be done by the user when running the test.
>    - Patch added to standardise return values for skipped testcases.
> v2:
>    - Added macro in place of testcase/testsuite loops.
>    - Added more detail in the summary output.
>    - Moved testcase counts to the testsuite structure.
>    - Flattened testsuite structure to remove union.
>    - Added patch for fix of blockcipher test return value.
>    - Squashed release note into last patch.
> 
> 
> Ciara Power (7):
>    app/test: refactor of unit test suite runner
>    test: introduce parent testsuite format
>    test/crypto: refactor to use sub-testsuites
>    test/crypto: replace unsupported with skipped
>    test/crypto: move testsuite params to header file
>    test/crypto: fix return value on test skipped
>    test/crypto: dynamically build blockcipher suite
> 
>   app/test/test.c                        |  176 +-
>   app/test/test.h                        |   23 +-
>   app/test/test_cryptodev.c              | 2324 ++++++++++++++----------
>   app/test/test_cryptodev.h              |   32 +
>   app/test/test_cryptodev_asym.c         |  111 +-
>   app/test/test_cryptodev_blockcipher.c  |  423 ++++-
>   app/test/test_cryptodev_blockcipher.h  |   12 +-
>   app/test/test_ipsec.c                  |   32 +-
>   doc/guides/rel_notes/release_21_05.rst |    5 +
>   9 files changed, 2000 insertions(+), 1138 deletions(-)
> 

Series Acked-by: Declan Doherty <declan.doherty@intel.com>

  parent reply	other threads:[~2021-04-30 21:00 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-23 16:18 Ciara Power
2021-04-23 16:18 ` [dpdk-dev] [PATCH v3 1/7] app/test: refactor of unit test suite runner Ciara Power
2021-04-23 16:18 ` [dpdk-dev] [PATCH v3 2/7] test: introduce parent testsuite format Ciara Power
2021-04-23 16:32   ` Aaron Conole
2021-04-23 16:18 ` [dpdk-dev] [PATCH v3 3/7] test/crypto: refactor to use sub-testsuites Ciara Power
2021-04-30 20:59   ` Doherty, Declan
2021-04-23 16:18 ` [dpdk-dev] [PATCH v3 4/7] test/crypto: replace unsupported with skipped Ciara Power
2021-04-23 16:18 ` [dpdk-dev] [PATCH v3 5/7] test/crypto: move testsuite params to header file Ciara Power
2021-04-23 16:18 ` [dpdk-dev] [PATCH v3 6/7] test/crypto: fix return value on test skipped Ciara Power
2021-04-23 16:18 ` [dpdk-dev] [PATCH v3 7/7] test/crypto: dynamically build blockcipher suite Ciara Power
2021-04-25  9:24 ` [dpdk-dev] [PATCH v3 0/7] test: refactor crypto unit test framework Ruifeng Wang
2021-04-30 21:00 ` Doherty, Declan [this message]
2021-05-05 10:41   ` Hemant Agrawal
2021-05-05 12:20     ` Hemant Agrawal
2021-05-08 15:38 ` [dpdk-dev] [EXT] " Akhil Goyal

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=365ff71b-92fd-dd9d-9bea-0b579629c810@intel.com \
    --to=declan.doherty@intel.com \
    --cc=aconole@redhat.com \
    --cc=ajit.khaparde@broadcom.com \
    --cc=anoobj@marvell.com \
    --cc=asomalap@amd.com \
    --cc=ciara.power@intel.com \
    --cc=dev@dpdk.org \
    --cc=g.singh@nxp.com \
    --cc=gakhil@marvell.com \
    --cc=hemant.agrawal@nxp.com \
    --cc=ruifeng.wang@arm.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).