Re: [dpdk-dev] [PATCH 0/6] test: refactor crypto unit test framework

["Power, Ciara" <ciara.power@intel.com>]

Hi Aaron,


>-----Original Message-----
>From: Aaron Conole <aconole@redhat.com>
>Sent: Wednesday 31 March 2021 15:43
>To: Doherty, Declan <declan.doherty@intel.com>
>Cc: Power, Ciara <ciara.power@intel.com>; dev@dpdk.org
>Subject: Re: [PATCH 0/6] test: refactor crypto unit test framework
>
>"Doherty, Declan" <declan.doherty@intel.com> writes:
>
>> Hey Aaron,
>>
>> based on the work you've been doing on the unit test documentation we
>> would appreciate if you could take a look over this patchset and get
>> your thoughts. The primary drive here is to make it easier to get a
>> clear picture of what is being executed in the cryptodev testsuite, as
>> at the moment there are test suites masquerading as unit tests and the
>> output doesn't reflect the actual number of unit tests being executed.
>
>I sent some comments - thanks :)
>

Thanks for the review!
I have sent a v2 including your suggested changes incorporated.
https://patchwork.dpdk.org/project/dpdk/list/?series=16085


>> Thanks
>> Declan
>>
>> On 16/03/2021 2:32 PM, Ciara Power wrote:
>>> The current crypto unit test framework is not granular enough to
>>> accurately track unit test results. This is caused by one testcase in
>>> a suite actually running multiple testcases, but only returning one
>>> result.
>>>   The approach taken in this patchset allows a test suite have either
>>> a list of sub-testsuites, or a list of testcases as previously used.
>>> The unit test suite runner can then recursively iterate and run the
>>> sub-testsuites, until it reaches a suite with testcases, and it then
>>> runs each testcase as it had done previously.
>>>   By allowing this further breakdown into sub-testsuites, a refactor
>>> of the crypto unit tests solves the issue of inaccurate reporting, as
>>> sub-testsuites can be used in place of the testcases that had
>>> multiple testcases hidden on a sub level.
>>> The blockcipher tests previously had these hidden testcases, but are
>>> now sub-testsuites that are dynamically created and added to a parent
>>> test suite, allowing for each testcase status to be reported directly
>>> to the runner.
>>> The cryptodev test suite is broken down into smaller suites that are
>>> used as sub-testsuites, which allows for more flexibility choosing
>>> which sub-testsuites should run for the current device autotest.
>>> The introduction of sub-testsuites also allows for more precise
>>> setup/teardown functions, rather than general ones loaded with
>>> conditions as was seen with the initial setup function used for all
>>> crypto testsuites.
>>>   For example, when running the cryptodev_aesni_mb_autotest, the
>>> AESNI MB parent test suite has its own setup function to initialise
>>> the AESNI MB device.
>>> Various sub-testsuites are added to the parent test suite, such as
>>> some of the static suites that were once in the cryptodev_testsuite,
>>> and blockcipher suites.
>>> The unit test runner can then run the AESNI MB parent test suite,
>>> which in turn will run the sub-testsuites.
>>>   Documentation will be added in a later version of the patchset,
>>> adding to the test document that isn't yet merged. [1]
>>>
>>> ---
>>> [1]
>>> https://patchwork.dpdk.org/project/dpdk/patch/20210309155757.615536-
>1
>>> -aconole@redhat.com/
>>>
>>> Ciara Power (6):
>>>    app/test: refactor of unit test suite runner
>>>    test: introduce parent testsuite format
>>>    test/crypto: refactor to use sub-testsuites
>>>    test/crypto: move testsuite params to header file
>>>    test/crypto: dynamically build blockcipher suite
>>>    doc: add unit test suite change to release notes
>>>
>>>   app/test/test.c                        |  168 +-
>>>   app/test/test.h                        |   22 +-
>>>   app/test/test_bitratestats.c           |    4 +-
>>>   app/test/test_compressdev.c            |    4 +-
>>>   app/test/test_cryptodev.c              | 2020 ++++++++++++++++++------
>>>   app/test/test_cryptodev.h              |   20 +
>>>   app/test/test_cryptodev_asym.c         |  105 +-
>>>   app/test/test_cryptodev_blockcipher.c  |  121 +-
>>>   app/test/test_cryptodev_blockcipher.h  |   12 +-
>>>   app/test/test_ethdev_link.c            |    4 +-
>>>   app/test/test_event_crypto_adapter.c   |    4 +-
>>>   app/test/test_event_eth_rx_adapter.c   |    8 +-
>>>   app/test/test_event_eth_tx_adapter.c   |    4 +-
>>>   app/test/test_event_timer_adapter.c    |    4 +-
>>>   app/test/test_eventdev.c               |    4 +-
>>>   app/test/test_fbarray.c                |    4 +-
>>>   app/test/test_fib.c                    |    8 +-
>>>   app/test/test_fib6.c                   |    8 +-
>>>   app/test/test_graph.c                  |    4 +-
>>>   app/test/test_graph_perf.c             |    4 +-
>>>   app/test/test_ipfrag.c                 |    4 +-
>>>   app/test/test_ipsec.c                  |   36 +-
>>>   app/test/test_ipsec_sad.c              |    4 +-
>>>   app/test/test_latencystats.c           |    4 +-
>>>   app/test/test_link_bonding.c           |    4 +-
>>>   app/test/test_link_bonding_mode4.c     |    4 +-
>>>   app/test/test_link_bonding_rssconf.c   |    4 +-
>>>   app/test/test_metrics.c                |    4 +-
>>>   app/test/test_pmd_ring.c               |    4 +-
>>>   app/test/test_reorder.c                |    4 +-
>>>   app/test/test_rib.c                    |    8 +-
>>>   app/test/test_rib6.c                   |    8 +-
>>>   app/test/test_security.c               |    4 +-
>>>   app/test/test_service_cores.c          |    4 +-
>>>   app/test/test_trace.c                  |    4 +-
>>>   doc/guides/rel_notes/release_21_05.rst |    5 +
>>>   36 files changed, 1898 insertions(+), 739 deletions(-)
>>>