From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ot0-f172.google.com (mail-ot0-f172.google.com [74.125.82.172]) by dpdk.org (Postfix) with ESMTP id 7AA3F378B for ; Fri, 10 Nov 2017 13:33:06 +0100 (CET) Received: by mail-ot0-f172.google.com with SMTP id s88so8034106ota.4 for ; Fri, 10 Nov 2017 04:33:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=atomicrules-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=CyVNTjsif8GsBdMvsQJsKe1LnjPON9ZQZ8ticM7NLhA=; b=fLbUar7yj4GvA/Z7Er/F868UWLhnqAaO3JphYoaIo6VlNqZU1E/mwOYdFi8jFmPtt3 HF2E+TTx6LiumiX6sjnz0V80mvWpQL2y0WuA85cAPmM0hL26AFWkpezIBhxAT0WOUoua 0eqQM6HoQmi13xPbQrpMrTHMz91ICqaYs2y3C/CSpjK5KG80iSj7XPR605yRezKmmlXg rS4KqRFcGvMnItVfz+5yrJcEtGeVoFfCZhEhLna9BPh4fgTWXU1N4Ni/z+C3k9wIhAkY bUhteCsV3kCqRYK6MnZVhWH2Uy0S+/f/k6wK6JpfPzyioPnBGBaWhPOpdpwCraWfPhij lSqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=CyVNTjsif8GsBdMvsQJsKe1LnjPON9ZQZ8ticM7NLhA=; b=QnwMUz+l5T5N20wJfX8/B17jfMq+7TZLgSeRztfA8re2aNIC93PkkKCLDwb6wKiXkg idO6nXZb3V1t2hG97OYkvaZQTcKA1XchOIa6//T1SEGpLPf0hXT2e05EqwukBHfgDXft Uqk8pOtk/pujfF/VnwIlLm8MXD/cf1asV2k0x4pDZtHLGwa4pFV+OJXBbsSSDwesdN5F qG2sW/syapFhCarufwxBGII4vO4CT1ohQkxLnoTpg4dtaG/kwuMbAcxJv/d4gwuftF1G r+MtV9LvmqfQ5DbJEM8wmQskIZ7lSfhLAKVjIMQiq8eedQcJs6r/1f71eu/Oided/EE/ r8Sw== X-Gm-Message-State: AJaThX5JXB0RuOJuq1SgS5Rb9PQUPHOl3bV3M1mrQVF/0p3z2VTMvYP0 Ia3JHe6CBvzXXM0CnGDe905zKN4rS2u6cVrnByIXOQ== X-Google-Smtp-Source: AGs4zMZlzt0lWZVLVBj6OvGILCETWFUpoK/8RVq0bg5taIyGIZNbJCpghQTMZucfuT16j5FsPtStEzFFoTmYhShWEQA= X-Received: by 10.157.9.234 with SMTP id 39mr133879otz.323.1510317185634; Fri, 10 Nov 2017 04:33:05 -0800 (PST) MIME-Version: 1.0 Received: by 10.74.116.90 with HTTP; Fri, 10 Nov 2017 04:33:04 -0800 (PST) In-Reply-To: References: From: Shepard Siegel Date: Fri, 10 Nov 2017 07:33:04 -0500 Message-ID: To: Patrick MacArthur Cc: ci@dpdk.org, Bob Noseworthy Content-Type: multipart/alternative; boundary="001a113e3c0a592b25055da01f7b" Subject: Re: [dpdk-ci] [RFC] test lab database schema X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Nov 2017 12:33:06 -0000 --001a113e3c0a592b25055da01f7b Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Patrick, This is a great start. Thanks. Those of us supporting FPGA based NICs have the added challenge of almost a universe of different firmware/gateware - sometimes capable of being changed at runtime (e.g. partial reconfiguration). I think your schema as-is covers this; but I feel it would be better to additionally include a sha256 hash field for something like "Firmware Source ID". That is, every time we make new FPGA NIC Firmware (dozens of times per day), baked into the bitstream is the hash of the git commit that was used to produce that bitstream. This is enormously empowering, as it allows one later at run time, to determine exactly what the code was that created that image. -Shep Shepard Siegel, CTO atomicrules.com On Thu, Nov 9, 2017 at 6:53 PM, Patrick MacArthur wrote: > Hi, all, > > I have been working on a database schema for storing performance > results. I am attempting to make this generic enough to store whatever > measurements we want. > > I have attached an entity-relationship diagram of the schema which > should illustrate the starting point for the schema. > > As a side note, I have managed to get DTS to run some of the > functional tests on a pair of nodes locally in a test setup while I > wait for equipment to be ready to ship. I am still working on a setup > to get it to run the performance tests so I can get some output to > parse to begin working on pulling information into the database. > > Some notes on the tables: > > patch: I propose that for this schema patches will be stored on the > filesystem content-addressable by the sha256 hash of the patch file. > > patchset: "branch" refers to corresponding repo (master -> dpdk, > dpdk-next-net -> dpdk/next-net, etc.) and will be NULL until the > patchset is applied. Tarballs stored as files named after patchset id. > > patchset_result: Result entry for a patchset. A patchset passes if > there is a result for every measurement which is either PASS or N/A. > > environment: This should represent everything about where the test was > run and a new environment needs to be created every time this changes > (e.g., kernel or compiler update). I gathered the list of fields by > looking at the existing performance reports on the DPDK website. This > can be used for verification, to allow the test environment to be > reproducible, and to ensure that all comparisons are within an > identical setup. > > measurement: A single measurement which can be applied to any > patchset. We can use values like (name: =E2=80=9CBUILD=E2=80=9D, higherIs= Better: TRUE, > expectedValue: 1, deltaLimit: 0) to verify non-performance conditions, > such as the build succeeding for the given environment. > > The primary keys for these tables are not shown; I will likely be > implementing the database using the data modeling framework for > whatever Web backend we wind up selecting, which will set up primary > keys and table join relationships automatically. > > Comments/suggestions? Is there anything that this schema does not cover? > > Thanks, > Patrick > > -- > Patrick MacArthur > Research and Development, High Performance Networking and Storage > UNH InterOperability Laboratory > --001a113e3c0a592b25055da01f7b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Patrick,

This is a great = start. Thanks. Those of us supporting FPGA based NICs have the added challe= nge of almost a universe of different firmware/gateware - sometimes capable= of being changed at runtime (e.g. partial reconfiguration). I think your s= chema as-is covers this; but I feel it would be better to additionally incl= ude a sha256 hash field for something like "Firmware Source ID". = That is, every time we make new FPGA NIC Firmware (dozens of times per day)= , baked into the bitstream is the hash of the git commit that was used to p= roduce that bitstream. This is enormously empowering, as it allows one late= r at run time, to determine exactly what the code was that created that ima= ge.

-Shep

Shepard Siegel, CTO
atomicrules.com


On Thu, Nov 9, 2017 at 6:53 PM, Pa= trick MacArthur <pmacarth@iol.unh.edu> wrote:
Hi, all,

I have been working on a database schema for storing performance
results. I am attempting to make this generic enough to store whatever
measurements we want.

I have attached an entity-relationship diagram of the schema which
should illustrate the starting point for the schema.

As a side note, I have managed to get DTS to run some of the
functional tests on a pair of nodes locally in a test setup while I
wait for equipment to be ready to ship. I am still working on a setup
to get it to run the performance tests so I can get some output to
parse to begin working on pulling information into the database.

Some notes on the tables:

patch: I propose that for this schema patches will be stored on the
filesystem content-addressable by the sha256 hash of the patch file.

patchset: "branch" refers to corresponding repo (master -> dpd= k,
dpdk-next-net -> dpdk/next-net, etc.) and will be NULL until the
patchset is applied. Tarballs stored as files named after patchset id.

patchset_result: Result entry for a patchset. A patchset passes if
there is a result for every measurement which is either PASS or N/A.

environment: This should represent everything about where the test was
run and a new environment needs to be created every time this changes
(e.g., kernel or compiler update). I gathered the list of fields by
looking at the existing performance reports on the DPDK website. This
can be used for verification, to allow the test environment to be
reproducible, and to ensure that all comparisons are within an
identical setup.

measurement: A single measurement which can be applied to any
patchset. We can use values like (name: =E2=80=9CBUILD=E2=80=9D, higherIsBe= tter: TRUE,
expectedValue: 1, deltaLimit: 0) to verify non-performance conditions,
such as the build succeeding for the given environment.

The primary keys for these tables are not shown; I will likely be
implementing the database using the data modeling framework for
whatever Web backend we wind up selecting, which will set up primary
keys and table join relationships automatically.

Comments/suggestions? Is there anything that this schema does not cover?
Thanks,
Patrick

--
Patrick MacArthur
Research and Development, High Performance Networking and Storage
UNH InterOperability Laboratory

--001a113e3c0a592b25055da01f7b--