[dpdk-dev] dpdk pre-patch testing introduction

[dpdk-dev] dpdk pre-patch testing introduction

[Liu, Yong]

Dear all,
Our validation team is working on pre-patch testing to support contiguous integration in DPDK community.
I'd like to share the latest status and some details of our solution to you.

The main target of pre-patch testing is supplement  the health of each patch set as soon as possible.
It will help both submitter and maintainer to know the status of patch set before it's merged.
Some of you have received the email which was generated by this tool. Let me introduce it to you.

Example email subject:
    |SUCCESS| pw 8162 ixgbe: remove useless fields in tx_offload on TCP checksum request
    The first item is the overall status of this patch set. The second item is the patchwork ID and its commit log.

There are 4 status to report as the following:
    Malformed patch: Patch's format doesn't incorrect, not applied in latest branch
    SUCCESS : Pass all testing.
    ERROR   : fail in some testing, error information can be found in report
    Not run  : Failed to compile, not able to perform functional testing

Email content included more detailed information of this patch set, like build failure log and failed cases log.
If patch set modified files for document, the content will also include the status of "Doc Compilation".

So far, we only enabled one platform to run automatic testing.
This platform installed Fedora 20 with Linux 3.18.9 and Intel Niantic NIC.

Currently the patch testing only run unit test and basic function test on this platform.
It can make sure that new patch doesn't break original code and functions.
Due to coverage limitation, it can't verify the functionality of new patch.
If you want to verify your patch's new function, you need to check with tester and perform specific testing on it.

Since platform limitation, we are not able to verify regression test on other NIC, Linux kernel.
We plan to add more platforms and support different NIC and kernels in the future.

Best regards,
DPDK validation team

Re: [dpdk-dev] dpdk pre-patch testing introduction

[Thomas Monjalon]

2015-10-29 15:19, Liu, Yong:
> So far, we only enabled one platform to run automatic testing.
> This platform installed Fedora 20 with Linux 3.18.9 and Intel Niantic NIC.

Thanks for offering this service.

> Currently the patch testing only run unit test and basic function test on this platform.
> It can make sure that new patch doesn't break original code and functions.
> Due to coverage limitation, it can't verify the functionality of new patch.
> If you want to verify your patch's new function, you need to check with tester and perform specific testing on it.
> 
> Since platform limitation, we are not able to verify regression test on other NIC, Linux kernel.
> We plan to add more platforms and support different NIC and kernels in the future.

The mailing list for test report is open to others.
Submitters are white-listed.
Feel free to contribute with your own tests on your own platform.

Some work is ongoing to track the test results from patchwork.
https://lists.ozlabs.org/pipermail/patchwork/2015-July/001363.html
https://lists.ozlabs.org/pipermail/patchwork/2015-July/001369.html

Re: [dpdk-dev] dpdk pre-patch testing introduction

[Simon Kågström]

Hi Liu!

I think this is a really, really great addition!

On 2015-10-29 16:19, Liu, Yong wrote:

> Currently the patch testing only run unit test and basic function test on this platform.
> It can make sure that new patch doesn't break original code and functions.
> Due to coverage limitation, it can't verify the functionality of new patch.
> If you want to verify your patch's new function, you need to check with tester and perform specific testing on it.

Speaking of coverage, I do have a few suggestions:

1. Build and test both with and without optimizations (build with
EXTRA_CFLAGS="-O0"). While uncommon, I've seen code break in the
non-optimized case because the compiler simply removes some unused code
which would otherwise cause link errors.

2. I think it would be very good to collect code coverage for patches as
well. (Yes, this is pushing my own tool) I've written a code coverage
tool called kcov [1], which only needs debugging information (-g) in the
binary to collect code coverage.

It outputs results in various formats: lcov-style HTML, cobertura XML
and can post directly to coveralls.io. It's also simple to run:

   kcov --include-pattern=dpdk /tmp/kcov-output ./testpmd [args...]

To the mail report, I think it would be good to add information about
code coverage increase/decrease with the patch and ideally which lines
of the patch which were covered

   Coverage: 69.2% (+5 lines with the patch)
   Coverage for the patch: 5/7 lines

both of these should be fairly simple to parse out from the XML output.


I'd be happy to help you if think it's a good idea!

// Simon

[1] https://github.com/SimonKagstrom/kcov