Re: [dpdk-ci] [RFC v2] Expected results JSON format

[Shreyansh Jain <shreyansh.jain@nxp.com>]

Hello Patrick,

Apologies for unusually long silence even though I was the one who initiated this.

> -----Original Message-----
> From: ci [mailto:ci-bounces@dpdk.org] On Behalf Of Patrick MacArthur
> Sent: Tuesday, February 20, 2018 10:33 PM
> To: ci@dpdk.org
> Cc: dpdklab@iol.unh.edu
> Subject: [dpdk-ci] [RFC v2] Expected results JSON format
> 
> Hi, all,
> 
> Based on feedback and internal discussion that occurred after our
> meeting this morning, this is the format that I am now expecting for
> the results:
> 
>     {
>       "environment":
> "https://dpdklab.iol.unh.edu/results/environments/9/",
>       "results": [
>         {
>           "frame_size": 64,
>           "txd/rxd": 1024,
>           "throughput": {
>             "result": "PASS",
>             "delta": -0.452,
>             "unit": "Mpps"
>           }
>         },
>         /* ... */
>       ]
>     }
> 
> OR:
> 
>     {
>       "environment":
> "https://dpdklab.iol.unh.edu/results/environments/9/",
>       "results": [
>         {
>           "frame_size": 64,
>           "txd/rxd": 1024,
>           "throughput": {
>             "result": "PASS",
>             "actual": 45.783,
>             "expected": 46.423,
>             "unit": "Mpps"
>           }
>         },
>         /* ... */
>       ]
>     }
> 
> The environment URL will be accessible via the CI_ENVIRONMENT_URL
> environment variable passed to your script via Jenkins; in the
> interest of making the results output self-describing, this should be
> echoed back in the JSON response.
> 
> Each entry in the results list is essentially a table row. The
> parameters "frame_size" and "txd/rxd" are the input parameters for
> each given measurement; what I gave here is just an example.
> 
> If the vendor script provides a delta, that delta is the only thing
> that will be stored in the database for that test case. If the vendor
> script provides actual and expected values, the expected value and the
> computed delta will be stored in the database.

Should I assume both are options are applicable/available?

If, as also mentioned below, there is access control related to data for respective vendor, I think the second option (absolute values with 'actual', 'expected') is easier and cleaner to implement. If, we have to come to a choice of selecting one of the methods above.

> 
> Either way, as discussed on the call, the results database API will
> have access control to only allow access to data from the respective
> vendor's users. Note that while we will endeavor to make our access
> control as secure as possible, there is some inherent risk in any
> database of a leak. Vendors should be aware of this potential risk and
> weigh the advantage of having the absolute measurements accessible to
> them against this potential risk.

I agree and understand this risk, for NXP's perspective at least.

Thanks for highlighting this as well as for providing a way out of this 'delta' problem.

> 
> Thoughts/concerns?
> 
> Thanks,
> Patrick
> 
> --
> Patrick MacArthur
> Research and Development, High Performance Networking and Storage
> UNH InterOperability Laboratory