Re: [dpdk-dev] [PATCH v3 0/7] test: refactor crypto unit test framework

[Hemant Agrawal <hemant.agrawal@oss.nxp.com>]

Sorry for top posting.

Akhil,

     Will you please hold for a day on this patch? we are seeing few 
failures on dpaa2 with this patch, currently debugging them.


Regards,

Hemant

On 5/1/2021 2:30 AM, Doherty, Declan wrote:
>
>
> On 23/04/2021 5:18 PM, Ciara Power wrote:
>> The current crypto unit test framework is not granular enough to
>> accurately track unit test results. This is caused by one testcase
>> in a suite actually running multiple testcases, but only returning
>> one result.
>>   The approach taken in this patchset allows a test suite have a
>> list of sub-testsuites, and/or a list of testcases as previously used.
>> The unit test suite runner can then recursively iterate and run the
>> sub-testsuites, until it reaches a suite with testcases,
>> and it then runs each testcase as it had done previously.
>> In the case of a testsuite with both testcases and sub-testsuites,
>> the testcases are executed first before iterating through the
>> sub-testsuites.
>>   By allowing this further breakdown into sub-testsuites,
>> a refactor of the crypto unit tests solves the issue of inaccurate
>> reporting, as sub-testsuites can be used in place of the testcases
>> that had multiple testcases hidden on a sub level.
>> The blockcipher tests previously had these hidden testcases,
>> but are now sub-testsuites that are dynamically created and added to a
>> parent test suite, allowing for each testcase status to be reported
>> directly to the runner.
>> The cryptodev test suite is broken down into smaller suites that are
>> used as sub-testsuites, which allows for more flexibility choosing which
>> sub-testsuites should run for the current device.
>> The introduction of sub-testsuites also allows for more precise
>> setup/teardown functions, that can check the capabilities required to
>> run its testcases.
>>   For example, when running the cryptodev_aesni_mb_autotest,
>> the parent Cryptodev Test Suite is executed.
>> Various sub-testsuites are added to the parent test suite, such as
>> the static suites of testcases that were once in the 
>> cryptodev_testsuite,
>> and blockcipher suites.
>> The unit test runner can then run the Cryptodev parent test suite,
>> which in turn will run the sub-testsuites.
>>
>> The user is now required to create vdevs via EAL commandline args,
>> this is no longer done within the test app for crypto autotests.
>>
>> Documentation will need to be added at a later stage,
>> adding to the test document that isn't yet merged. [1]
>>
>> [1] 
>> https://patchwork.dpdk.org/project/dpdk/patch/20210309155757.615536-1-aconole@redhat.com/
>>
>>
>> v3:
>>    - Added support for a testsuite having both a list of testcases,
>>      and a list of sub-testsuites.
>>    - Replaced PMD based parent testsuites with a cryptodev testsuite
>>      used by all autotests, with the exception of scheduler autotest.
>>    - Setup functions were added for all sub-testsuites, within which
>>      required capability support is checked.
>>    - The setup functions no longer create vdevs if needed,
>>      this must be done by the user when running the test.
>>    - Patch added to standardise return values for skipped testcases.
>> v2:
>>    - Added macro in place of testcase/testsuite loops.
>>    - Added more detail in the summary output.
>>    - Moved testcase counts to the testsuite structure.
>>    - Flattened testsuite structure to remove union.
>>    - Added patch for fix of blockcipher test return value.
>>    - Squashed release note into last patch.
>>
>>
>> Ciara Power (7):
>>    app/test: refactor of unit test suite runner
>>    test: introduce parent testsuite format
>>    test/crypto: refactor to use sub-testsuites
>>    test/crypto: replace unsupported with skipped
>>    test/crypto: move testsuite params to header file
>>    test/crypto: fix return value on test skipped
>>    test/crypto: dynamically build blockcipher suite
>>
>>   app/test/test.c                        |  176 +-
>>   app/test/test.h                        |   23 +-
>>   app/test/test_cryptodev.c              | 2324 ++++++++++++++----------
>>   app/test/test_cryptodev.h              |   32 +
>>   app/test/test_cryptodev_asym.c         |  111 +-
>>   app/test/test_cryptodev_blockcipher.c  |  423 ++++-
>>   app/test/test_cryptodev_blockcipher.h  |   12 +-
>>   app/test/test_ipsec.c                  |   32 +-
>>   doc/guides/rel_notes/release_21_05.rst |    5 +
>>   9 files changed, 2000 insertions(+), 1138 deletions(-)
>>
>
> Series Acked-by: Declan Doherty <declan.doherty@intel.com>