From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-f174.google.com (mail-wr0-f174.google.com [209.85.128.174]) by dpdk.org (Postfix) with ESMTP id 55FA3237 for ; Tue, 28 Nov 2017 17:32:39 +0100 (CET) Received: by mail-wr0-f174.google.com with SMTP id h1so614913wre.12 for ; Tue, 28 Nov 2017 08:32:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=h4sgAEzvaTpZKzQO7cgWtBxmftRopHZdcbIQMl8Mo1c=; b=HUSGPlBcIAIZTYumXLic8darw/4Bofd6DuxBbDeoFEybgEVgBxGJS9yeGo1SQ0n0R0 leGMxYI3oB8nqJlxcOor/i9se9XNWilb8e034mj7hfAzVRxsuXWw98fHRS33gTNlE0tj T3vSkrD6CNBKnqFXPq9MzCsNBTKA8Q6muyXO0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=h4sgAEzvaTpZKzQO7cgWtBxmftRopHZdcbIQMl8Mo1c=; b=TXxKH9sr9IEPXDoFPQBZ8NsgIr7oqPa0FwEzdhVUEbDPONBP0tmjtPA3AEgNu0B3cW PlsL+0QvJUShwSnN1v/THCIn+mNxQ4yQLXHMWqyEv8+SpBXaHPNitFReiaxXrPseOLko E5q602146EckWKkV6tOTsRXTMhtVdY20TdOnmCKuDbIHlQT9SZ9/bIhtJ3pqHbwNJ+Pu MZSGVXK+qBS5CppS3wth3GBQwMjI6p43VjBaKEnCSvL0nCCWn2RQppdwyRRQAlKSdc88 kFJAIlKNWUT+bPTm5568oap2CBJva0ez1G8FVK21uttMaECrb0YoCAPx/yJuwNqnMumY cnSA== X-Gm-Message-State: AJaThX60q09xcQjJPQTVCqZ5/cqp5rD7Nu7JoxeBqxq9xv5foTRSMuCU WyLTqw6UhqCLbtHMt0iBqJmbXi/Pexw= X-Google-Smtp-Source: AGs4zMbHWP8TFgm4chpJYhz3oa75RAMhIC/pJRUGAyDJx7nogYRJGVdlUtW0ZY/9tHIS0g2CEIq8Ag== X-Received: by 10.223.135.169 with SMTP id b38mr33576448wrb.278.1511886758659; Tue, 28 Nov 2017 08:32:38 -0800 (PST) Received: from [10.0.1.3] (host86-178-232-116.range86-178.btcentralplus.com. [86.178.232.116]) by smtp.gmail.com with ESMTPSA id b33sm17067985wrg.48.2017.11.28.08.32.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Nov 2017 08:32:37 -0800 (PST) To: ci@dpdk.org References: From: Gema Gomez Message-ID: Date: Tue, 28 Nov 2017 16:32:36 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-CA Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-ci] [RFC] test lab database schema X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2017 16:32:39 -0000 Hi Patrick, thanks for starting this work. As per our discussion today, summarizing my feedback on an email: - I would like to have some way of debugging issues when there are performance regressions. Even if that is not a primary goal at this stage, being able to do that in the future would be important for CI. Specially when there are regressions on HW that the community / patch writer won't have access to. Making performance counters available would be useful. - Make sure every server is distinguishable from each other in the database. This is important when it comes to comparing performance metrics as different servers from same vendor may vary in terms of performance measurements. - Would be nice to make the column names homogeneous (i.e. underscores or camel casing, but not both), I vote for underscores. - There is kernel_version on the database but no OS/distro/installed packages. Is there a default distro that everybody is going to test on? Having a list of packages/versions were installed on the server when the tests run will be useful for traceability. Having the UI as something to think about/implement later worries me. Representing the data in a way that it is consumable by engineering is not an easy task, and it is important. It will also affect the database design. I'd like to contribute an example performance graph/drawings from a previous life that may or may not be useful: http://ci.ubuntu.com/bootspeed/arch/amd64/ Cheers, Gema On 09/11/17 23:53, Patrick MacArthur wrote: > Hi, all, > > I have been working on a database schema for storing performance > results. I am attempting to make this generic enough to store whatever > measurements we want. > > I have attached an entity-relationship diagram of the schema which > should illustrate the starting point for the schema. > > As a side note, I have managed to get DTS to run some of the > functional tests on a pair of nodes locally in a test setup while I > wait for equipment to be ready to ship. I am still working on a setup > to get it to run the performance tests so I can get some output to > parse to begin working on pulling information into the database. > > Some notes on the tables: > > patch: I propose that for this schema patches will be stored on the > filesystem content-addressable by the sha256 hash of the patch file. > > patchset: "branch" refers to corresponding repo (master -> dpdk, > dpdk-next-net -> dpdk/next-net, etc.) and will be NULL until the > patchset is applied. Tarballs stored as files named after patchset id. > > patchset_result: Result entry for a patchset. A patchset passes if > there is a result for every measurement which is either PASS or N/A. > > environment: This should represent everything about where the test was > run and a new environment needs to be created every time this changes > (e.g., kernel or compiler update). I gathered the list of fields by > looking at the existing performance reports on the DPDK website. This > can be used for verification, to allow the test environment to be > reproducible, and to ensure that all comparisons are within an > identical setup. > > measurement: A single measurement which can be applied to any > patchset. We can use values like (name: “BUILD”, higherIsBetter: TRUE, > expectedValue: 1, deltaLimit: 0) to verify non-performance conditions, > such as the build succeeding for the given environment. > > The primary keys for these tables are not shown; I will likely be > implementing the database using the data modeling framework for > whatever Web backend we wind up selecting, which will set up primary > keys and table join relationships automatically. > > Comments/suggestions? Is there anything that this schema does not cover? > > Thanks, > Patrick > -- Gema Gomez-Solano Tech Lead, SDI Linaro Ltd IRC: gema@#linaro on irc.freenode.net