[dpdk-ci] GA Baseline Performance Testing

[dpdk-ci] GA Baseline Performance Testing

[Jeremy Plsek]

Hello all,

It was requested that we add a delta test result of the latest GA to
the result table. I said that it could be accomplished relatively
easily during the meeting. After some thought and laying out a plan,
adding this result will take much longer than expected.

If this ends up being a requirement before the results become public,
it may take much longer before the results become public.

Below are the steps that will have to be taken to get this working.

1. The member supplied test harness (such as DTS) will have to update
the `--update-expected` argument that expects an arbitrary value (such
as `master` or `v18.05.1`). This would save the absolute numbers as
before, but with appropriate keys. This will allow deltas to be
generated for each of those keys.
2. With the change above, the test harness will have to output a table
of deltas for each key defined.
3. Add an option to delete said keys (such as `--delete-expected
v17.11.4`) so we don’t have to generate more tables than needed.
4. Update the database to expect the individual delta results to
contain these tags. Old results would be migrated to a `master` tag.
5. Update the dashboard to provide a table of results for each unique tag.
6. Update current jobs with the new test harness parameters (which
would use `--update-expected master`, for example).
7. Create a new script that finds the latest tag from dpdk-stable,
builds, and uploads to our database appropriately (which would use
`--update-expected v18.05.1`, for example). Then delete older
"expected" keys.
8. Create GA performance Jenkins Job to run this script.
9. Done? (Everything should be automatic at this point?)

Implementation specifics will be added to the policy document if
members would like to go through with this.

This may also help pave the way for comparing GA versions as well (as
was also requested). Currently, with the data we have, we can compare
the performance delta of master. (I made a quick-and-dirty script that
plotted master over time before the presentation at the Summit, but my
logic was slightly off, so I didn’t present it.)

If anyone has a better way of doing this, please feel free to provide
suggestions.

I estimate that this will take at least a month to add these delta results.

Thanks
-- 
Jeremy Plsek
UNH InterOperability Laboratory

Re: [dpdk-ci] GA Baseline Performance Testing

[Tu, Lijuan]

Hi Jeremy,

Agree with that each individual delta result has a tag (such as dpdk version), but I have some comments on the DTS part.
'--update-expected v18.05.1' and `--delete-expected v17.11.4` are not make sense, as DTS is a test suite, it carries out the test cases based on the DPDK and give out results, but it doesn't care about DPDK version.
Could you consider my idea:
	- Each GA release expected results should be saved. 
	- run every single test with the stored expected result.
	- update delta result in database.

Take v17.11.4 as an example,
Firstly run DTS based DPDK-v17.11.4 to get the expected numbers which stored in conf/nic_single_core.cfg, and save the file. 
Secondly in runtime, just replace the configure file with the stored one, and then run DTS.
Thirdly update the results to database under unique tag(maybe v17.11.4)

Keep listening to you.
Thanks
Lijuan

> -----Original Message-----
> From: ci [mailto:ci-bounces@dpdk.org] On Behalf Of Jeremy Plsek
> Sent: Friday, September 28, 2018 11:40 PM
> To: ci@dpdk.org
> Cc: dpdklab@iol.unh.edu
> Subject: [dpdk-ci] GA Baseline Performance Testing
> 
> Hello all,
> 
> It was requested that we add a delta test result of the latest GA to the result
> table. I said that it could be accomplished relatively easily during the meeting.
> After some thought and laying out a plan, adding this result will take much
> longer than expected.
> 
> If this ends up being a requirement before the results become public, it may
> take much longer before the results become public.
> 
> Below are the steps that will have to be taken to get this working.
> 
> 1. The member supplied test harness (such as DTS) will have to update the
> `--update-expected` argument that expects an arbitrary value (such as
> `master` or `v18.05.1`). This would save the absolute numbers as before, but
> with appropriate keys. This will allow deltas to be generated for each of
> those keys.
> 2. With the change above, the test harness will have to output a table of
> deltas for each key defined.
> 3. Add an option to delete said keys (such as `--delete-expected
> v17.11.4`) so we don’t have to generate more tables than needed.
> 4. Update the database to expect the individual delta results to contain these
> tags. Old results would be migrated to a `master` tag.
> 5. Update the dashboard to provide a table of results for each unique tag.
> 6. Update current jobs with the new test harness parameters (which would
> use `--update-expected master`, for example).
> 7. Create a new script that finds the latest tag from dpdk-stable, builds, and
> uploads to our database appropriately (which would use `--update-expected
> v18.05.1`, for example). Then delete older "expected" keys.
> 8. Create GA performance Jenkins Job to run this script.
> 9. Done? (Everything should be automatic at this point?)
> 
> Implementation specifics will be added to the policy document if members
> would like to go through with this.
> 
> This may also help pave the way for comparing GA versions as well (as was
> also requested). Currently, with the data we have, we can compare the
> performance delta of master. (I made a quick-and-dirty script that plotted
> master over time before the presentation at the Summit, but my logic was
> slightly off, so I didn’t present it.)
> 
> If anyone has a better way of doing this, please feel free to provide
> suggestions.
> 
> I estimate that this will take at least a month to add these delta results.
> 
> Thanks
> --
> Jeremy Plsek
> UNH InterOperability Laboratory