Re: [dpdk-dev] [PATCH 0/6] test: refactor crypto unit test framework

["Power, Ciara" <ciara.power@intel.com>]

>-----Original Message-----
>From: Ruifeng Wang <Ruifeng.Wang@arm.com>
>Sent: Thursday 1 April 2021 04:14
>To: Power, Ciara <ciara.power@intel.com>; dev@dpdk.org
>Cc: Doherty, Declan <declan.doherty@intel.com>; nd <nd@arm.com>
>Subject: RE: [dpdk-dev] [PATCH 0/6] test: refactor crypto unit test framework
>
>> -----Original Message-----
>> From: dev <dev-bounces@dpdk.org> On Behalf Of Ciara Power
>> Sent: Tuesday, March 16, 2021 10:33 PM
>> To: dev@dpdk.org
>> Cc: declan.doherty@intel.com; Ciara Power <ciara.power@intel.com>
>> Subject: [dpdk-dev] [PATCH 0/6] test: refactor crypto unit test
>> framework
>>
>> The current crypto unit test framework is not granular enough to
>> accurately track unit test results. This is caused by one testcase in
>> a suite actually running multiple testcases, but only returning one result.
>>
>> The approach taken in this patchset allows a test suite have either a
>> list of sub-testsuites, or a list of testcases as previously used.
>> The unit test suite runner can then recursively iterate and run the
>> sub- testsuites, until it reaches a suite with testcases, and it then
>> runs each testcase as it had done previously.
>>
>> By allowing this further breakdown into sub-testsuites, a refactor of
>> the crypto unit tests solves the issue of inaccurate reporting, as
>> sub-testsuites can be used in place of the testcases that had multiple
>> testcases hidden on a sub level.
>> The blockcipher tests previously had these hidden testcases, but are
>> now sub-testsuites that are dynamically created and added to a parent
>> test suite, allowing for each testcase status to be reported directly to the
>runner.
>> The cryptodev test suite is broken down into smaller suites that are
>> used as sub-testsuites, which allows for more flexibility choosing
>> which sub- testsuites should run for the current device autotest.
>> The introduction of sub-testsuites also allows for more precise
>> setup/teardown functions, rather than general ones loaded with
>> conditions as was seen with the initial setup function used for all crypto
>testsuites.
>>
>> For example, when running the cryptodev_aesni_mb_autotest, the AESNI
>> MB parent test suite has its own setup function to initialise the
>> AESNI MB device.
>> Various sub-testsuites are added to the parent test suite, such as
>> some of the static suites that were once in the cryptodev_testsuite,
>> and blockcipher suites.
>> The unit test runner can then run the AESNI MB parent test suite,
>> which in turn will run the sub-testsuites.
>>
>> Documentation will be added in a later version of the patchset, adding
>> to the test document that isn't yet merged. [1]
>>
>> ---
>> [1]
>> https://patchwork.dpdk.org/project/dpdk/patch/20210309155757.615536-
>1-
>> aconole@redhat.com/
>>
>> Ciara Power (6):
>>   app/test: refactor of unit test suite runner
>>   test: introduce parent testsuite format
>>   test/crypto: refactor to use sub-testsuites
>>   test/crypto: move testsuite params to header file
>>   test/crypto: dynamically build blockcipher suite
>>   doc: add unit test suite change to release notes
>>
>>  app/test/test.c                        |  168 +-
>>  app/test/test.h                        |   22 +-
>>  app/test/test_bitratestats.c           |    4 +-
>>  app/test/test_compressdev.c            |    4 +-
>>  app/test/test_cryptodev.c              | 2020 ++++++++++++++++++------
>>  app/test/test_cryptodev.h              |   20 +
>>  app/test/test_cryptodev_asym.c         |  105 +-
>>  app/test/test_cryptodev_blockcipher.c  |  121 +-
>>  app/test/test_cryptodev_blockcipher.h  |   12 +-
>>  app/test/test_ethdev_link.c            |    4 +-
>>  app/test/test_event_crypto_adapter.c   |    4 +-
>>  app/test/test_event_eth_rx_adapter.c   |    8 +-
>>  app/test/test_event_eth_tx_adapter.c   |    4 +-
>>  app/test/test_event_timer_adapter.c    |    4 +-
>>  app/test/test_eventdev.c               |    4 +-
>>  app/test/test_fbarray.c                |    4 +-
>>  app/test/test_fib.c                    |    8 +-
>>  app/test/test_fib6.c                   |    8 +-
>>  app/test/test_graph.c                  |    4 +-
>>  app/test/test_graph_perf.c             |    4 +-
>>  app/test/test_ipfrag.c                 |    4 +-
>>  app/test/test_ipsec.c                  |   36 +-
>>  app/test/test_ipsec_sad.c              |    4 +-
>>  app/test/test_latencystats.c           |    4 +-
>>  app/test/test_link_bonding.c           |    4 +-
>>  app/test/test_link_bonding_mode4.c     |    4 +-
>>  app/test/test_link_bonding_rssconf.c   |    4 +-
>>  app/test/test_metrics.c                |    4 +-
>>  app/test/test_pmd_ring.c               |    4 +-
>>  app/test/test_reorder.c                |    4 +-
>>  app/test/test_rib.c                    |    8 +-
>>  app/test/test_rib6.c                   |    8 +-
>>  app/test/test_security.c               |    4 +-
>>  app/test/test_service_cores.c          |    4 +-
>>  app/test/test_trace.c                  |    4 +-
>>  doc/guides/rel_notes/release_21_05.rst |    5 +
>>  36 files changed, 1898 insertions(+), 739 deletions(-)
>>
>> --
>> 2.25.1
>
>Tested against armv8crypto PMD and the result looks good.
>Tested-by: Ruifeng Wang <ruifeng.wang@arm.com>

Thanks for testing this Ruifeng.

I have just sent a v2 of the patchset but didn't add the "Tested-by" tag, as it has quite a number of changes.
I don't think the changes should have affected armv8crypto PMD, but if you could test with the new version that would be really appreciated.
https://patchwork.dpdk.org/project/dpdk/list/?series=16085

Thanks!
Ciara