From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ot1-f47.google.com (mail-ot1-f47.google.com [209.85.210.47]) by dpdk.org (Postfix) with ESMTP id 01D791B450 for ; Fri, 28 Sep 2018 17:40:27 +0200 (CEST) Received: by mail-ot1-f47.google.com with SMTP id h26-v6so6423657otl.9 for ; Fri, 28 Sep 2018 08:40:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:from:date:message-id:subject:to:cc :content-transfer-encoding; bh=R/CUROGXyu4ykyMcqkh6OokZPp8HTFCsrDcUKgXBfzE=; b=JoFkHYMzYMJEYzZt+T7TWBa4skkbt8Cgic+eJx/vJpmZecYyWX6SkCzYBTYz8QJg4s UwKmqP+As1FsFy3rWWuTnBPDl+5lt/KEOG/ZpzriYJm6kPjdhGtzbBcYE+wjAaJtx7YF ilqHAkx8+DcPzapiULWz6sG/gqG+yIW7BvlkM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc :content-transfer-encoding; bh=R/CUROGXyu4ykyMcqkh6OokZPp8HTFCsrDcUKgXBfzE=; b=pmATdn/uMdgSOT7IshjxKPB6TNyFHFGZEyN9+WvTL4huHx3pzoM5rr7//lrm8wNL/c c67l9Gv6MKf24I/9pqc/tikytUger63yDXyxLOYuhpRIpoN8zZBVU2PpiVbI3s+8nHZf yBiy+f7v+0D81SJnZ+6rrE5wneDwEPKu42ajhGz3vxK26MZweDUKyYa/1gM8SoZGdzfW A5GESAqKMpBKmqB3kvNL3Rw+rGAU+KScVo94GPHvI2pL/MXaMflnRNXSqQO5mIpbVIka bUjyHehOjUhQgJWxRBiLjn2jGIt9wL4v7jT9Um58DrOCq9Stni+6SdKN20Pa7iFwEy/S MR/w== X-Gm-Message-State: ABuFfogFE50fkmdtr7FwxZ6c78xtCpkIlj35kga+w4WWF0H/nhzzd9s0 Xm8+edmIJU8tTjKQxVb1gXqGGhtMAA/paUwMZsfj9rJOU+Y= X-Google-Smtp-Source: ACcGV61WDx0aFjc6M1trrlULRXMKSwr1fi4UOOER42sgF5fkitu8rYe6r5hK1rbFtAKBNMTswBJL/A8Bn5PhpOoaL2s= X-Received: by 2002:a9d:3098:: with SMTP id s24-v6mr9886192otc.24.1538149226752; Fri, 28 Sep 2018 08:40:26 -0700 (PDT) MIME-Version: 1.0 From: Jeremy Plsek Date: Fri, 28 Sep 2018 11:39:50 -0400 Message-ID: To: ci@dpdk.org Cc: dpdklab@iol.unh.edu Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: [dpdk-ci] GA Baseline Performance Testing X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Sep 2018 15:40:28 -0000 Hello all, It was requested that we add a delta test result of the latest GA to the result table. I said that it could be accomplished relatively easily during the meeting. After some thought and laying out a plan, adding this result will take much longer than expected. If this ends up being a requirement before the results become public, it may take much longer before the results become public. Below are the steps that will have to be taken to get this working. 1. The member supplied test harness (such as DTS) will have to update the `--update-expected` argument that expects an arbitrary value (such as `master` or `v18.05.1`). This would save the absolute numbers as before, but with appropriate keys. This will allow deltas to be generated for each of those keys. 2. With the change above, the test harness will have to output a table of deltas for each key defined. 3. Add an option to delete said keys (such as `--delete-expected v17.11.4`) so we don=E2=80=99t have to generate more tables than needed. 4. Update the database to expect the individual delta results to contain these tags. Old results would be migrated to a `master` tag. 5. Update the dashboard to provide a table of results for each unique tag. 6. Update current jobs with the new test harness parameters (which would use `--update-expected master`, for example). 7. Create a new script that finds the latest tag from dpdk-stable, builds, and uploads to our database appropriately (which would use `--update-expected v18.05.1`, for example). Then delete older "expected" keys. 8. Create GA performance Jenkins Job to run this script. 9. Done? (Everything should be automatic at this point?) Implementation specifics will be added to the policy document if members would like to go through with this. This may also help pave the way for comparing GA versions as well (as was also requested). Currently, with the data we have, we can compare the performance delta of master. (I made a quick-and-dirty script that plotted master over time before the presentation at the Summit, but my logic was slightly off, so I didn=E2=80=99t present it.) If anyone has a better way of doing this, please feel free to provide suggestions. I estimate that this will take at least a month to add these delta results. Thanks --=20 Jeremy Plsek UNH InterOperability Laboratory