From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <pmacarth@iol.unh.edu>
Received: from mail-ua0-f181.google.com (mail-ua0-f181.google.com
 [209.85.217.181]) by dpdk.org (Postfix) with ESMTP id 0A64A1E35
 for <ci@dpdk.org>; Tue, 20 Mar 2018 14:56:13 +0100 (CET)
Received: by mail-ua0-f181.google.com with SMTP id x17so1053432uaj.12
 for <ci@dpdk.org>; Tue, 20 Mar 2018 06:56:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol;
 h=mime-version:from:date:message-id:subject:to;
 bh=oTuRzP2DDiJ9psvyjtyFVR0jUvhvEkkrzxq00LrG6jg=;
 b=jh2zQWmFCakluyoBfeSi74lFQBb5+ALxeFNFv+gvQktRJLI0SZVAFYWO6YO2Gevfq8
 G+N386pR000HRJhKdqCgOJ4UCXYb6hEa+mJerP4V39aZxkh7SsM/T1I1c/te1hdrdrDv
 zAWFKs418PYA0cdVvPNrM7crJMDbwE7blxSzg=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
 bh=oTuRzP2DDiJ9psvyjtyFVR0jUvhvEkkrzxq00LrG6jg=;
 b=Xo/sAU9BstKC2AbXRIyN7lgZtawiKoNspWgxGu/d6GDjuh9KPb+RxUJhyLhAtwRfFv
 poDxqVHYlh2f7ZmzWRv009uocjl7r31FW1hd+h9yBO2qPJP+lnz8FeFhRcxW1S+1iZy9
 NaA6JQpkCL0Eew+/hRpNkc/YaEDDmPh47ZEe8IUsGCIKbM7XJLhh8s610nRaDfB3YuoB
 7HTDQveGKLMJZmPYFA2y/TyySmPa+EpW/6ycww8qS1H2PI67gI/oQWhTEv0j4CWGM92Z
 a8NUIYb8LbJ5Q1n4fWg7DeEfIwVHqYqvhBrBGXwhIwRqrqGwV0HRld4JPscuwBNvg49S
 B2dQ==
X-Gm-Message-State: AElRT7G1vlrg0ylTp5G2x9VQqnzVSryaRmaZbgadSSCAL5TzSCu9sAFx
 5nqvZF+2cmHy6IpGxHv9+Br6fpYcs3Tf29BS3PZbeEsL
X-Google-Smtp-Source: AG47ELtH13ABSc4CwYnrupeVky30xvzeYl9pe/LVf0h3vYF85zz16UlnqNFGBpepX4oqKtJaP1XFBkXAV/5g29H8ltI=
X-Received: by 10.176.72.168 with SMTP id x37mr5172249uac.8.1521554172134;
 Tue, 20 Mar 2018 06:56:12 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.176.23.133 with HTTP; Tue, 20 Mar 2018 06:56:11 -0700 (PDT)
From: Patrick MacArthur <pmacarth@iol.unh.edu>
Date: Tue, 20 Mar 2018 09:56:11 -0400
Message-ID: <CABj6NQwxbF_dr3Gcyup-M_z4d5YOOD2TjVheFhWZkNHwK+D-KA@mail.gmail.com>
To: ci@dpdk.org
Content-Type: text/plain; charset="UTF-8"
Subject: [dpdk-ci] [v3] Expected results JSON format
X-BeenThere: ci@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK CI discussions <ci.dpdk.org>
List-Unsubscribe: <https://dpdk.org/ml/options/ci>,
 <mailto:ci-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/ci/>
List-Post: <mailto:ci@dpdk.org>
List-Help: <mailto:ci-request@dpdk.org?subject=help>
List-Subscribe: <https://dpdk.org/ml/listinfo/ci>,
 <mailto:ci-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Tue, 20 Mar 2018 13:56:13 -0000

Hi, all,

This is a slight revision based on my test implementation:

    {
      "results": [
        {
          "parameters": {
            "frame_size": 64,
            "txd/rxd": 1024
          },
          "throughput": {
            "result": "PASS",
            "delta": -0.452,
            "unit": "Mpps"
          }
        },
        /* ... */
      ]
    }

OR:

    {
      "results": [
        {
          "parameters": {
            "frame_size": 64,
            "txd/rxd": 1024
          },
         "throughput": {
            "result": "PASS",
            "actual": 45.783,
            "expected": 46.423,
            "unit": "Mpps"
          }
        },
        /* ... */
      ]
    }

The difference from the previous version that the input parameters are
now part of their own dictionary instead of loose within the "results"
dictionary.

Each entry in the results list is essentially a table row. The
parameters "frame_size" and "txd/rxd" are the input parameters for
each given measurement; what I gave here is just an example.

If the vendor script provides a delta, that delta is the only thing
that will be stored in the database for that test case. If the vendor
script provides actual and expected values, the expected value and the
computed delta will be stored in the database.

Either way, as discussed on the call, the results database API will
have access control to only allow access to data from the respective
vendor's users. Note that while we will endeavor to make our access
control as secure as possible, there is some inherent risk in any
database of a leak. Vendors should be aware of this potential risk and
weigh the advantage of having the absolute measurements accessible to
them against this potential risk.

Thanks,
Patrick

-- 
Patrick MacArthur
Research and Development, High Performance Networking and Storage
UNH InterOperability Laboratory