DPDK CI discussions
 help / color / mirror / Atom feed
From: Thomas Monjalon <thomas@monjalon.net>
To: Patrick MacArthur <pmacarth@iol.unh.edu>
Cc: ci@dpdk.org, Bob Noseworthy <ren@iol.unh.edu>
Subject: Re: [dpdk-ci] [RFC] test lab database schema
Date: Tue, 28 Nov 2017 15:14:14 +0100	[thread overview]
Message-ID: <2135477.CoD8K78SAQ@xps> (raw)
In-Reply-To: <CABj6NQwtUWcZN2yN0dMozP=F0VfDVATEjnyYQ=ObmZuTbAe45g@mail.gmail.com>

Hi,

10/11/2017 00:53, Patrick MacArthur:
> I have been working on a database schema for storing performance
> results. I am attempting to make this generic enough to store whatever
> measurements we want.

Thanks for working on it.

> I have attached an entity-relationship diagram of the schema which
> should illustrate the starting point for the schema.

I will do some comments below.

[...]
> patch: I propose that for this schema patches will be stored on the
> filesystem content-addressable by the sha256 hash of the patch file.

I don't see the need to store the full diff.
You could store the patchwork id to retrieve it.

> patchset: "branch" refers to corresponding repo (master -> dpdk,
> dpdk-next-net -> dpdk/next-net, etc.) and will be NULL until the
> patchset is applied. Tarballs stored as files named after patchset id.

You should also be able to store measurements for a given state of a git tree.
It will be used for regular tests (like daily).

> patchset_result: Result entry for a patchset. A patchset passes if
> there is a result for every measurement which is either PASS or N/A.
> 
> environment: This should represent everything about where the test was
> run and a new environment needs to be created every time this changes
> (e.g., kernel or compiler update). I gathered the list of fields by
> looking at the existing performance reports on the DPDK website. This
> can be used for verification, to allow the test environment to be
> reproducible, and to ensure that all comparisons are within an
> identical setup.

Do we want a limited set of environment fields, or something flexible?
For instance, there is a field dts_config, but we could use other test tools.

> measurement: A single measurement which can be applied to any
> patchset. We can use values like (name: “BUILD”, higherIsBetter: TRUE,
> expectedValue: 1, deltaLimit: 0) to verify non-performance conditions,
> such as the build succeeding for the given environment.

I think we should store the normal values per-environment at different time.
I mean, the normal value will evolve with time and we should keep history of it.

> The primary keys for these tables are not shown; I will likely be
> implementing the database using the data modeling framework for
> whatever Web backend we wind up selecting, which will set up primary
> keys and table join relationships automatically.
> 
> Comments/suggestions? Is there anything that this schema does not cover?

As I said above, we need to cover tests based on git state, not patchset.

Thanks :)

  parent reply	other threads:[~2017-11-28 14:14 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-11-09 23:53 Patrick MacArthur
2017-11-10 12:33 ` Shepard Siegel
2017-11-28 14:14 ` Thomas Monjalon [this message]
2017-11-28 16:32 ` Gema Gomez
2017-11-28 17:13   ` Thomas Monjalon
2017-11-28 17:26     ` Gema Gomez

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=2135477.CoD8K78SAQ@xps \
    --to=thomas@monjalon.net \
    --cc=ci@dpdk.org \
    --cc=pmacarth@iol.unh.edu \
    --cc=ren@iol.unh.edu \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).