Re: [dpdk-ci] [dpdk-marketing] Results are now public

[Jeremy Plsek <jplsek@iol.unh.edu>]

With help from Lincoln and Patrick, below is an initial draft of the blog post.



DPDK Community Lab Publishes Relative Performance Testing Results

The DPDK Community Lab is an open, independent testing resource for
the DPDK project. Its purpose is to perform automated testing on
incoming patch submissions, to ensure the performance and quality of
DPDK is maintained. Participation in the lab is open to all DPDK
project participants.

For some time now, the DPDK Community Lab has been gathering
performance deltas using the single-core packet I/O layer 2 throughput
test from DTS for each patch series submitted to DPDK compared to the
master branch. We are pleased to announce that the Lab has recently
been allowed to make these results public. These results are also now
published to Patchwork as they are automatically generated. These
results currently contain Mellanox and Intel devices, and the lab is
able to support hardware from any DPDK participants wishing to support
these testing efforts.

To view these results, you can go to DPDK Community Lab Dashboard via
the following link: https://lab.dpdk.org. The dashboard lists an
overview of all active patch series and their results. Detailed
results can be viewed by clicking on the patch series. If a patch
fails to merge into master, a build log will show to help identify any
issues. If a patch cleanly merges into master, performance delta
results will show for each participating member.

The Lab is hosted by the University of New Hampshire InterOperability
Laboratory, as a neutral, third party location. This provides a secure
environment for hosting equipment and generating unbiased results for
all participating vendors. Lab participants, i.e. companies hosting
equipment in the testing, can securely access their equipment through
a VPN, allowing for maintenance and performance tuning, as the DPDK
project progresses.

The Lab works by polling the Patchwork API. When new patches are
submitted, the CI server merges them with the master branch and
generates a tarball. Each participating system unpacks and installs
the DPDK tarball and then runs the performance testing against this
DPDK build. When all systems have finished testing, the CI gathers the
results into our internal database to be shown on the Dashboard, and
sends final reports to Patchwork to show up on the submitted patch.
This allows patch submitters to utilize Patchwork to view their
individual results, while also allowing anyone to quickly see an
overview of results on the Dashboard. The system provides maintainers
with positive confirmation of the stability and performance of the
overall project.

In the future, we plan to open the Lab to more testing scenarios, such
as performance testing of other features, beyond single-core packet
I/O layer 2 throughput, and possibly running Unit Tests for DPDK.
Additional features will be added to the Dashboard, such as showing
graphs of the performance changes of master over time.

If your company would like to be involved, email the Continuous
Integration group at ci@dpdk.org and dpdklab@iol.unh.edu.



Thanks!

On Fri, Nov 9, 2018 at 10:03 AM Jeremy Plsek <jplsek@iol.unh.edu> wrote:
>
> I can try to get it done later today, but I expect it to be more likely done on Monday.
>
> On Fri, Nov 9, 2018, 9:08 AM O'Driscoll, Tim <tim.odriscoll@intel.com wrote:
>>
>>
>>
>> > -----Original Message-----
>> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
>> > Sent: Friday, November 9, 2018 2:02 PM
>> > To: O'Driscoll, Tim <tim.odriscoll@intel.com>
>> > Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; Jeremy Plsek
>> > <jplsek@iol.unh.edu>; ci@dpdk.org; marketing@dpdk.org
>> > Subject: Re: [dpdk-marketing] [dpdk-ci] Results are now public
>> >
>> > 09/11/2018 14:58, O'Driscoll, Tim:
>> > > From: Thomas Monjalon
>> > > > 08/11/2018 14:01, Ferruh Yigit:
>> > > > > On 11/7/2018 7:30 AM, Thomas Monjalon wrote:
>> > > > > > 06/11/2018 16:33, Jeremy Plsek:
>> > > > > >> Hi all,
>> > > > > >>
>> > > > > >> This is just an update that all detailed results are now
>> > public.
>> > > > > >>
>> > > > > >> If there is anything out of place, feel free to let me know!
>> > > > > >
>> > > > > > It is really nice!
>> > > > > > Thank you
>> > > > >
>> > > > > Should we announce this in 'announce' or 'dev' mail lists? Many
>> > people
>> > > > not aware
>> > > > > of this.
>> > > >
>> > > > It deserves an announce.
>> > > > And even a blog post I guess.
>> > > > +Cc marketing team
>> > >
>> > > Agreed. I can do a quick post to announce@dpdk.org. I believe Jeremy
>> > is working on a blog post which can provide more detail.
>> >
>> > Should we wait to have a blog post and reference it in the announce?
>>
>> Good question. I think it depends how long the blog will take. If we can do it fairly quickly, then this is a good idea. If it's going to take a few weeks, then we should announce it now.
>>
>> Jeremy: When do you think the blog will be ready?
>>


--
Jeremy Plsek
UNH InterOperability Laboratory