DPDK CI discussions
 help / color / Atom feed
From: Jeremy Plsek <jplsek@iol.unh.edu>
To: ci@dpdk.org
Cc: dpdklab@iol.unh.edu
Subject: [dpdk-ci] GA Baseline Performance Testing
Date: Fri, 28 Sep 2018 11:39:50 -0400
Message-ID: <CA+xUZB4XgFSRuV2qGW48sj__vJLFgaTzfA1T8COyxQ+xhzpDdA@mail.gmail.com> (raw)

Hello all,

It was requested that we add a delta test result of the latest GA to
the result table. I said that it could be accomplished relatively
easily during the meeting. After some thought and laying out a plan,
adding this result will take much longer than expected.

If this ends up being a requirement before the results become public,
it may take much longer before the results become public.

Below are the steps that will have to be taken to get this working.

1. The member supplied test harness (such as DTS) will have to update
the `--update-expected` argument that expects an arbitrary value (such
as `master` or `v18.05.1`). This would save the absolute numbers as
before, but with appropriate keys. This will allow deltas to be
generated for each of those keys.
2. With the change above, the test harness will have to output a table
of deltas for each key defined.
3. Add an option to delete said keys (such as `--delete-expected
v17.11.4`) so we don’t have to generate more tables than needed.
4. Update the database to expect the individual delta results to
contain these tags. Old results would be migrated to a `master` tag.
5. Update the dashboard to provide a table of results for each unique tag.
6. Update current jobs with the new test harness parameters (which
would use `--update-expected master`, for example).
7. Create a new script that finds the latest tag from dpdk-stable,
builds, and uploads to our database appropriately (which would use
`--update-expected v18.05.1`, for example). Then delete older
"expected" keys.
8. Create GA performance Jenkins Job to run this script.
9. Done? (Everything should be automatic at this point?)

Implementation specifics will be added to the policy document if
members would like to go through with this.

This may also help pave the way for comparing GA versions as well (as
was also requested). Currently, with the data we have, we can compare
the performance delta of master. (I made a quick-and-dirty script that
plotted master over time before the presentation at the Summit, but my
logic was slightly off, so I didn’t present it.)

If anyone has a better way of doing this, please feel free to provide
suggestions.

I estimate that this will take at least a month to add these delta results.

Thanks
-- 
Jeremy Plsek
UNH InterOperability Laboratory

             reply index

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-09-28 15:39 Jeremy Plsek [this message]
2018-09-29  2:21 ` Tu, Lijuan

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CA+xUZB4XgFSRuV2qGW48sj__vJLFgaTzfA1T8COyxQ+xhzpDdA@mail.gmail.com \
    --to=jplsek@iol.unh.edu \
    --cc=ci@dpdk.org \
    --cc=dpdklab@iol.unh.edu \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK CI discussions

Archives are clonable:
	git clone --mirror http://inbox.dpdk.org/ci/0 ci/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 ci ci/ http://inbox.dpdk.org/ci \
		ci@dpdk.org
	public-inbox-index ci


Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.ci


AGPL code for this site: git clone https://public-inbox.org/ public-inbox