From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-vk0-f45.google.com (mail-vk0-f45.google.com [209.85.213.45]) by dpdk.org (Postfix) with ESMTP id 27DFB271 for ; Tue, 20 Feb 2018 18:02:32 +0100 (CET) Received: by mail-vk0-f45.google.com with SMTP id z190so6034607vkg.1 for ; Tue, 20 Feb 2018 09:02:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:from:date:message-id:subject:to:cc; bh=V4MO7QVbaTjV5wq82ojP9QGUF+HcghKWNzX6eC6JAR8=; b=MpjXTOQylNAyRSAj5O/DDLO0c6YyxWsL44B8h1AyTTCPsW0DHDoZYT6tIZBkp/059I MGBDDjjpqjHiPKimCF+8snpPG14sjNX+dYQPXHLJ7GbsPz377PswZ12SABFvn6RQzq1H HMQdgD6yPxFQ3TRtmuiE9dmYypZbrt3PnGMPs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=V4MO7QVbaTjV5wq82ojP9QGUF+HcghKWNzX6eC6JAR8=; b=RIELf5bZztWv1k5KL538bF+lFjbSJtU4S3IE3DImW67h6aI4IgphgO9D4huX459wkQ WL7mIKx2DpBUqLhcbf16XIQNO7HanHk+CaXIdc5aJV5jz+7wNpR9nlbnZ7DWQeivnFFG BTjHfcj+F/BgbBbGhMf5rAF/bGPLITjxGxsAjqDhl1lX6KqPjHqtoZsNMAQJuiJ9JwtU 8jXVhJ1W2KK7m2+zmFF/ruVJbbbt18Eu1s/0n3DC+c20vMmWIO9fbBP/dULLR7CEkaTH VS3m4Zd9HRduvpeO+DUQNfyjxwlY1KycO+rg1hGSDoWZGi38j4x2lvomWYJx41JW+Pmn ipPA== X-Gm-Message-State: APf1xPBVvR/XeZmuwURmPDB/Xg00tO5ad9gXFyaqqvina2Qkz0M0JPxH sx2j1QrkPppFVwc8UG7zBZvlUGTegiBEVl/TcXgQvbj/ X-Google-Smtp-Source: AH8x2254sjlcHQvX7vLgw5U+C22duXmgXRhaEEH/KRuF6ArIyFJXEC83dzEWz4/ZN1GpGHJ3TRkSyMd+BilCXnzCOwg= X-Received: by 10.31.157.151 with SMTP id g145mr241676vke.114.1519146152023; Tue, 20 Feb 2018 09:02:32 -0800 (PST) MIME-Version: 1.0 Received: by 10.176.23.133 with HTTP; Tue, 20 Feb 2018 09:02:31 -0800 (PST) From: Patrick MacArthur Date: Tue, 20 Feb 2018 12:02:31 -0500 Message-ID: To: ci@dpdk.org Cc: dpdklab@iol.unh.edu Content-Type: text/plain; charset="UTF-8" Subject: [dpdk-ci] [RFC v2] Expected results JSON format X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Feb 2018 17:02:33 -0000 Hi, all, Based on feedback and internal discussion that occurred after our meeting this morning, this is the format that I am now expecting for the results: { "environment": "https://dpdklab.iol.unh.edu/results/environments/9/", "results": [ { "frame_size": 64, "txd/rxd": 1024, "throughput": { "result": "PASS", "delta": -0.452, "unit": "Mpps" } }, /* ... */ ] } OR: { "environment": "https://dpdklab.iol.unh.edu/results/environments/9/", "results": [ { "frame_size": 64, "txd/rxd": 1024, "throughput": { "result": "PASS", "actual": 45.783, "expected": 46.423, "unit": "Mpps" } }, /* ... */ ] } The environment URL will be accessible via the CI_ENVIRONMENT_URL environment variable passed to your script via Jenkins; in the interest of making the results output self-describing, this should be echoed back in the JSON response. Each entry in the results list is essentially a table row. The parameters "frame_size" and "txd/rxd" are the input parameters for each given measurement; what I gave here is just an example. If the vendor script provides a delta, that delta is the only thing that will be stored in the database for that test case. If the vendor script provides actual and expected values, the expected value and the computed delta will be stored in the database. Either way, as discussed on the call, the results database API will have access control to only allow access to data from the respective vendor's users. Note that while we will endeavor to make our access control as secure as possible, there is some inherent risk in any database of a leak. Vendors should be aware of this potential risk and weigh the advantage of having the absolute measurements accessible to them against this potential risk. Thoughts/concerns? Thanks, Patrick -- Patrick MacArthur Research and Development, High Performance Networking and Storage UNH InterOperability Laboratory