Re: [dpdk-ci] [RFC] test lab database schema

[Gema Gomez <gema.gomez-solano@linaro.org>]

Hi Patrick,

thanks for starting this work.

As per our discussion today, summarizing my feedback on an email:

- I would like to have some way of debugging issues when there are
performance regressions. Even if that is not a primary goal at this
stage, being able to do that in the future would be important for CI.
Specially when there are regressions on HW that the community / patch
writer won't have access to. Making performance counters available would
be useful.

- Make sure every server is distinguishable from each other in the
database. This is important when it comes to comparing performance
metrics as different servers from same vendor may vary in terms of
performance measurements.

- Would be nice to make the column names homogeneous (i.e. underscores
or camel casing, but not both), I vote for underscores.

- There is kernel_version on the database but no OS/distro/installed
packages. Is there a default distro that everybody is going to test on?
Having a list of packages/versions were installed on the server when the
tests run will be useful for traceability.

Having the UI as something to think about/implement later worries me.
Representing the data in a way that it is consumable by engineering is
not an easy task, and it is important. It will also affect the database
design. I'd like to contribute an example performance graph/drawings
from a previous life that may or may not be useful:

http://ci.ubuntu.com/bootspeed/arch/amd64/

Cheers,
Gema


On 09/11/17 23:53, Patrick MacArthur wrote:
> Hi, all,
> 
> I have been working on a database schema for storing performance
> results. I am attempting to make this generic enough to store whatever
> measurements we want.
> 
> I have attached an entity-relationship diagram of the schema which
> should illustrate the starting point for the schema.
> 
> As a side note, I have managed to get DTS to run some of the
> functional tests on a pair of nodes locally in a test setup while I
> wait for equipment to be ready to ship. I am still working on a setup
> to get it to run the performance tests so I can get some output to
> parse to begin working on pulling information into the database.
> 
> Some notes on the tables:
> 
> patch: I propose that for this schema patches will be stored on the
> filesystem content-addressable by the sha256 hash of the patch file.
> 
> patchset: "branch" refers to corresponding repo (master -> dpdk,
> dpdk-next-net -> dpdk/next-net, etc.) and will be NULL until the
> patchset is applied. Tarballs stored as files named after patchset id.
> 
> patchset_result: Result entry for a patchset. A patchset passes if
> there is a result for every measurement which is either PASS or N/A.
> 
> environment: This should represent everything about where the test was
> run and a new environment needs to be created every time this changes
> (e.g., kernel or compiler update). I gathered the list of fields by
> looking at the existing performance reports on the DPDK website. This
> can be used for verification, to allow the test environment to be
> reproducible, and to ensure that all comparisons are within an
> identical setup.
> 
> measurement: A single measurement which can be applied to any
> patchset. We can use values like (name: “BUILD”, higherIsBetter: TRUE,
> expectedValue: 1, deltaLimit: 0) to verify non-performance conditions,
> such as the build succeeding for the given environment.
> 
> The primary keys for these tables are not shown; I will likely be
> implementing the database using the data modeling framework for
> whatever Web backend we wind up selecting, which will set up primary
> keys and table join relationships automatically.
> 
> Comments/suggestions? Is there anything that this schema does not cover?
> 
> Thanks,
> Patrick
> 


-- 
Gema Gomez-Solano
Tech Lead, SDI
Linaro Ltd
IRC: gema@#linaro on irc.freenode.net