DPDK CI discussions
 help / color / Atom feed
From: Jeremy Plsek <jplsek@iol.unh.edu>
To: Thomas Monjalon <thomas@monjalon.net>, Rami Rosen <ramirose@gmail.com>
Cc: ci@dpdk.org
Subject: Re: [dpdk-ci] Question about performance test
Date: Fri, 4 Jan 2019 11:15:49 -0500
Message-ID: <CA+xUZB7WjTQ7aiGmrVPVXaYV9ZNEeUcHwk-HFLOHb1JU02P2zA@mail.gmail.com> (raw)
In-Reply-To: <2631230.hnArF6gTu5@xps>

[-- Attachment #1: Type: text/plain, Size: 2787 bytes --]

>
> I do not see the actual results of the DTS perf test in the links you
> posted, only percentage of degradation or improvement, unless I miss
> something.
>

This was done on purpose as requested by the Members participating in the
effort. This CI is meant for the DPDK maintainers to make sure that a patch
does not introduce significant performance regressions on various hardware
platforms, or to show how a patch may improve the performance of DPDK on
these platforms (such as driver updates or something in the core of DPDK
itself). It is not to compare the performance between different devices.

I believe it can be helpful if the baseline of the actual results will also
> be shown
> to enable comparing to other vendors besides Intel and Mellanox.
>

There is not a single baseline for all devices. The baselines are generated
per device. Knowing the baseline would allow creating an absolute result,
which we are trying to avoid.

Do you plan to add such URL in the report sent to patchwork?
>

Created a ticket to add the URL to the emailed reports:
https://bugs.dpdk.org/show_bug.cgi?id=180

Instead of filtering based on the label, you could filter based on
> the paths of modified files.
> Note that such filter depends on the test you run,
> because you could also test the doc syntax in the CI.
>

Okay, so if a series only modified the doc folder, then don't include it
for performance tests. Later on, we can introduce syntax checking for
documentation when we introduce more unit testing / functional testing.
https://bugs.dpdk.org/show_bug.cgi?id=181

On Fri, Jan 4, 2019 at 10:34 AM Thomas Monjalon <thomas@monjalon.net> wrote:

> 04/01/2019 15:44, Jeremy Plsek:
> > Hi Rami,
> >
> > I'm the current maintainer of the DPDK Performance CI. I realize that the
> > performance results don't point to the website, so it's not obvious on
> > where to find this information. You can find an overview of these tests
> > here: https://lab.dpdk.org
> >
> > Most of this information can be either found on the detailed results of a
> > test (such as https://lab.dpdk.org/results/dashboard/patchsets/4157/)
> or on
> > the about page (https://lab.dpdk.org/results/dashboard/about/).
>
> Do you plan to add such URL in the report sent to patchwork?
>
> > We don't apply the doc folder when applying the series, in case a patch
> > included code unrelated to documentation. If others in the group feel
> that
> > it's still unnecessary to include "doc" labeled series, I can look into
> > filtering them out.
>
> Instead of filtering based on the label, you could filter based on
> the paths of modified files.
> Note that such filter depends on the test you run,
> because you could also test the doc syntax in the CI.
>
>
>

-- 
Jeremy Plsek
UNH InterOperability Laboratory

[-- Attachment #2: Type: text/html, Size: 4895 bytes --]

<div dir="ltr"><div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>I do not see the actual results of the DTS perf test in the links you 
posted, only percentage of degradation or improvement, unless I miss 
something.</div></blockquote><div><br></div><div>This was done on purpose as requested by the Members participating in the effort. This CI is meant for the DPDK maintainers to make sure that a patch does not introduce significant performance regressions on various hardware platforms, or to show how a patch may improve the performance of DPDK on these platforms (such as driver updates or something in the core of DPDK itself). It is not to compare the performance between different devices.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div> I believe it can be helpful if the baseline of the actual results will also be shown <div dir="auto">to enable comparing to other vendors besides Intel and Mellanox.</div></div></blockquote><div><br></div><div>There is not a single baseline for all devices. The baselines are generated per device. Knowing the baseline would allow creating an absolute result, which we are trying to avoid.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Do you plan to add such URL in the report sent to patchwork?<span class="m_1793572375996440537gmail-m_4722669703508491933gmail-im"><br></span></div></blockquote><div dir="ltr"><span class="m_1793572375996440537gmail-m_4722669703508491933gmail-im"><br></span></div><div dir="ltr">Created a ticket to add the URL to the emailed reports: <a href="https://bugs.dpdk.org/show_bug.cgi?id=180" target="_blank">https://bugs.dpdk.org/show_bug.cgi?id=180</a></div><div dir="ltr"><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Instead of filtering based on the label, you could filter based on<br>
the paths of modified files.<br>
Note that such filter depends on the test you run,<br>
because you could also test the doc syntax in the CI.</div></blockquote><div><br></div><div>Okay, so if a series only modified the doc folder, then don&#39;t include it for performance tests. Later on, we can introduce syntax checking for documentation when we introduce more unit testing / functional testing. <a href="https://bugs.dpdk.org/show_bug.cgi?id=181" target="_blank">https://bugs.dpdk.org/show_bug.cgi?id=181</a><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Jan 4, 2019 at 10:34 AM Thomas Monjalon &lt;<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">04/01/2019 15:44, Jeremy Plsek:<br>
&gt; Hi Rami,<br>
&gt; <br>
&gt; I&#39;m the current maintainer of the DPDK Performance CI. I realize that the<br>
&gt; performance results don&#39;t point to the website, so it&#39;s not obvious on<br>
&gt; where to find this information. You can find an overview of these tests<br>
&gt; here: <a href="https://lab.dpdk.org" rel="noreferrer" target="_blank">https://lab.dpdk.org</a><br>
&gt; <br>
&gt; Most of this information can be either found on the detailed results of a<br>
&gt; test (such as <a href="https://lab.dpdk.org/results/dashboard/patchsets/4157/" rel="noreferrer" target="_blank">https://lab.dpdk.org/results/dashboard/patchsets/4157/</a>) or on<br>
&gt; the about page (<a href="https://lab.dpdk.org/results/dashboard/about/" rel="noreferrer" target="_blank">https://lab.dpdk.org/results/dashboard/about/</a>).<br>
<br>
Do you plan to add such URL in the report sent to patchwork?<br>
<br>
&gt; We don&#39;t apply the doc folder when applying the series, in case a patch<br>
&gt; included code unrelated to documentation. If others in the group feel that<br>
&gt; it&#39;s still unnecessary to include &quot;doc&quot; labeled series, I can look into<br>
&gt; filtering them out.<br>
<br>
Instead of filtering based on the label, you could filter based on<br>
the paths of modified files.<br>
Note that such filter depends on the test you run,<br>
because you could also test the doc syntax in the CI.<br>
<br>
<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail-m_1793572375996440537gmail-m_4722669703508491933gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div></div>Jeremy Plsek<br></div><div>UNH InterOperability Laboratory<br></div></div></div></div></div></div></div></div></div></div>

  reply index

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-01-04 10:45 Rami Rosen
2019-01-04 14:44 ` Jeremy Plsek
2019-01-04 15:33   ` Rami Rosen
2019-01-04 15:34   ` Thomas Monjalon
2019-01-04 16:15     ` Jeremy Plsek [this message]
2019-01-04 16:19       ` Thomas Monjalon
2019-01-08  6:40   ` Rami Rosen
2019-01-08  7:06     ` Tu, Lijuan
2019-01-08  8:23       ` Rami Rosen

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CA+xUZB7WjTQ7aiGmrVPVXaYV9ZNEeUcHwk-HFLOHb1JU02P2zA@mail.gmail.com \
    --to=jplsek@iol.unh.edu \
    --cc=ci@dpdk.org \
    --cc=ramirose@gmail.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK CI discussions

Archives are clonable:
	git clone --mirror http://inbox.dpdk.org/ci/0 ci/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 ci ci/ http://inbox.dpdk.org/ci \
		ci@dpdk.org
	public-inbox-index ci


Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.ci


AGPL code for this site: git clone https://public-inbox.org/ public-inbox