DPDK CI discussions
 help / color / mirror / Atom feed
* [dpdk-ci] [RFC v2] Expected results JSON format
@ 2018-02-20 17:02 Patrick MacArthur
  2018-03-06 12:42 ` Shreyansh Jain
  0 siblings, 1 reply; 2+ messages in thread
From: Patrick MacArthur @ 2018-02-20 17:02 UTC (permalink / raw)
  To: ci; +Cc: dpdklab

Hi, all,

Based on feedback and internal discussion that occurred after our
meeting this morning, this is the format that I am now expecting for
the results:

    {
      "environment": "https://dpdklab.iol.unh.edu/results/environments/9/",
      "results": [
        {
          "frame_size": 64,
          "txd/rxd": 1024,
          "throughput": {
            "result": "PASS",
            "delta": -0.452,
            "unit": "Mpps"
          }
        },
        /* ... */
      ]
    }

OR:

    {
      "environment": "https://dpdklab.iol.unh.edu/results/environments/9/",
      "results": [
        {
          "frame_size": 64,
          "txd/rxd": 1024,
          "throughput": {
            "result": "PASS",
            "actual": 45.783,
            "expected": 46.423,
            "unit": "Mpps"
          }
        },
        /* ... */
      ]
    }

The environment URL will be accessible via the CI_ENVIRONMENT_URL
environment variable passed to your script via Jenkins; in the
interest of making the results output self-describing, this should be
echoed back in the JSON response.

Each entry in the results list is essentially a table row. The
parameters "frame_size" and "txd/rxd" are the input parameters for
each given measurement; what I gave here is just an example.

If the vendor script provides a delta, that delta is the only thing
that will be stored in the database for that test case. If the vendor
script provides actual and expected values, the expected value and the
computed delta will be stored in the database.

Either way, as discussed on the call, the results database API will
have access control to only allow access to data from the respective
vendor's users. Note that while we will endeavor to make our access
control as secure as possible, there is some inherent risk in any
database of a leak. Vendors should be aware of this potential risk and
weigh the advantage of having the absolute measurements accessible to
them against this potential risk.

Thoughts/concerns?

Thanks,
Patrick

-- 
Patrick MacArthur
Research and Development, High Performance Networking and Storage
UNH InterOperability Laboratory

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: [dpdk-ci] [RFC v2] Expected results JSON format
  2018-02-20 17:02 [dpdk-ci] [RFC v2] Expected results JSON format Patrick MacArthur
@ 2018-03-06 12:42 ` Shreyansh Jain
  0 siblings, 0 replies; 2+ messages in thread
From: Shreyansh Jain @ 2018-03-06 12:42 UTC (permalink / raw)
  To: Patrick MacArthur; +Cc: dpdklab, ci

Hello Patrick,

Apologies for unusually long silence even though I was the one who initiated this.

> -----Original Message-----
> From: ci [mailto:ci-bounces@dpdk.org] On Behalf Of Patrick MacArthur
> Sent: Tuesday, February 20, 2018 10:33 PM
> To: ci@dpdk.org
> Cc: dpdklab@iol.unh.edu
> Subject: [dpdk-ci] [RFC v2] Expected results JSON format
> 
> Hi, all,
> 
> Based on feedback and internal discussion that occurred after our
> meeting this morning, this is the format that I am now expecting for
> the results:
> 
>     {
>       "environment":
> "https://dpdklab.iol.unh.edu/results/environments/9/",
>       "results": [
>         {
>           "frame_size": 64,
>           "txd/rxd": 1024,
>           "throughput": {
>             "result": "PASS",
>             "delta": -0.452,
>             "unit": "Mpps"
>           }
>         },
>         /* ... */
>       ]
>     }
> 
> OR:
> 
>     {
>       "environment":
> "https://dpdklab.iol.unh.edu/results/environments/9/",
>       "results": [
>         {
>           "frame_size": 64,
>           "txd/rxd": 1024,
>           "throughput": {
>             "result": "PASS",
>             "actual": 45.783,
>             "expected": 46.423,
>             "unit": "Mpps"
>           }
>         },
>         /* ... */
>       ]
>     }
> 
> The environment URL will be accessible via the CI_ENVIRONMENT_URL
> environment variable passed to your script via Jenkins; in the
> interest of making the results output self-describing, this should be
> echoed back in the JSON response.
> 
> Each entry in the results list is essentially a table row. The
> parameters "frame_size" and "txd/rxd" are the input parameters for
> each given measurement; what I gave here is just an example.
> 
> If the vendor script provides a delta, that delta is the only thing
> that will be stored in the database for that test case. If the vendor
> script provides actual and expected values, the expected value and the
> computed delta will be stored in the database.

Should I assume both are options are applicable/available?

If, as also mentioned below, there is access control related to data for respective vendor, I think the second option (absolute values with 'actual', 'expected') is easier and cleaner to implement. If, we have to come to a choice of selecting one of the methods above.

> 
> Either way, as discussed on the call, the results database API will
> have access control to only allow access to data from the respective
> vendor's users. Note that while we will endeavor to make our access
> control as secure as possible, there is some inherent risk in any
> database of a leak. Vendors should be aware of this potential risk and
> weigh the advantage of having the absolute measurements accessible to
> them against this potential risk.

I agree and understand this risk, for NXP's perspective at least.

Thanks for highlighting this as well as for providing a way out of this 'delta' problem.

> 
> Thoughts/concerns?
> 
> Thanks,
> Patrick
> 
> --
> Patrick MacArthur
> Research and Development, High Performance Networking and Storage
> UNH InterOperability Laboratory

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2018-03-06 12:42 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-20 17:02 [dpdk-ci] [RFC v2] Expected results JSON format Patrick MacArthur
2018-03-06 12:42 ` Shreyansh Jain

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).