[dpdk-ci] Question about performance test

[dpdk-ci] Question about performance test

[Rami Rosen]

[-- Attachment #1: Type: text/plain, Size: 1034 bytes --]

Hi,
I have a question about DPDK CI process and the tests done when a patch is
submitted to dpdk-dev mailing list.
In DPDK patch work I see these response messages from the DPDK CI for all
patches:

...
ci/intel-Performance-Testing success Performance Testing PASS
ci/mellanox-Performance-Testing success Performance Testing PASS
...

My question is (I hope and believe the info is available publicly) : which
tests are run in the ci, generating these messages?  is it done with IXIA
and DTS ? (DPDK test suite, https://doc.dpdk.org/dts/gsg/) ? are these
l2fwd/l3fwd performance tests? or more than that ? and on which
Intel/Mellanox nics ?  Are these merely performance tests, or also
functional tests ?

And BTW, I noticed that the CI runs a full performance cycle also for doc
patches (at least these messages are generated), which is a kind of
redundant (unless there is some filter which checks that if a patch only
affects modules under "doc", than such a cycle is not done but the messages
are still sent)

Regards,
Rami Rosen

[-- Attachment #2: Type: text/html, Size: 2628 bytes --]

Re: [dpdk-ci] Question about performance test

[Jeremy Plsek]

[-- Attachment #1: Type: text/plain, Size: 2259 bytes --]

Hi Rami,

I'm the current maintainer of the DPDK Performance CI. I realize that the
performance results don't point to the website, so it's not obvious on
where to find this information. You can find an overview of these tests
here: https://lab.dpdk.org

Most of this information can be either found on the detailed results of a
test (such as https://lab.dpdk.org/results/dashboard/patchsets/4157/) or on
the about page (https://lab.dpdk.org/results/dashboard/about/).

But to answer your questions:
At the moment, we only run performance tests. Specifically the
nic_single_core_perf_test from the DPDK Test Suite with the TRex traffic
generator.
The devices we are testing are currently the Intel 82599ES 10G, the Intel
XL710-QDA2 40G, the Mellanox ConnectX-5 100G, and the ConnectX-4 Lx 25G and
40G.

We don't apply the doc folder when applying the series, in case a patch
included code unrelated to documentation. If others in the group feel that
it's still unnecessary to include "doc" labeled series, I can look into
filtering them out.

Thanks.


On Fri, Jan 4, 2019 at 5:46 AM Rami Rosen <ramirose@gmail.com> wrote:

> Hi,
> I have a question about DPDK CI process and the tests done when a patch is
> submitted to dpdk-dev mailing list.
> In DPDK patch work I see these response messages from the DPDK CI for all
> patches:
>
> ...
> ci/intel-Performance-Testing success Performance Testing PASS
> ci/mellanox-Performance-Testing success Performance Testing PASS
> ...
>
> My question is (I hope and believe the info is available publicly) : which
> tests are run in the ci, generating these messages?  is it done with IXIA
> and DTS ? (DPDK test suite, https://doc.dpdk.org/dts/gsg/) ? are these
> l2fwd/l3fwd performance tests? or more than that ? and on which
> Intel/Mellanox nics ?  Are these merely performance tests, or also
> functional tests ?
>
> And BTW, I noticed that the CI runs a full performance cycle also for doc
> patches (at least these messages are generated), which is a kind of
> redundant (unless there is some filter which checks that if a patch only
> affects modules under "doc", than such a cycle is not done but the messages
> are still sent)
>
> Regards,
> Rami Rosen
>


-- 
Jeremy Plsek
UNH InterOperability Laboratory

[-- Attachment #2: Type: text/html, Size: 4921 bytes --]

Re: [dpdk-ci] Question about performance test

[Rami Rosen]

[-- Attachment #1: Type: text/plain, Size: 2937 bytes --]

Hi Jeremy,


Thanks for your quick reply, it fully clarifies the query I had posted.

I do not see the actual results of the DTS perf test in the links you
posted, only percentage of degradation or improvement, unless I miss
something. I believe it can be helpful if the baseline of the actual
results will also be shown
to enable comparing to other vendors besides Intel and Mellanox.

Regards,
Rami Rosen





בתאריך יום ו׳, 4 בינו׳ 2019, 16:44, מאת Jeremy Plsek <jplsek@iol.unh.edu>:

> Hi Rami,
>
> I'm the current maintainer of the DPDK Performance CI. I realize that the
> performance results don't point to the website, so it's not obvious on
> where to find this information. You can find an overview of these tests
> here: https://lab.dpdk.org
>
> Most of this information can be either found on the detailed results of a
> test (such as https://lab.dpdk.org/results/dashboard/patchsets/4157/) or
> on the about page (https://lab.dpdk.org/results/dashboard/about/).
>
> But to answer your questions:
> At the moment, we only run performance tests. Specifically the
> nic_single_core_perf_test from the DPDK Test Suite with the TRex traffic
> generator.
> The devices we are testing are currently the Intel 82599ES 10G, the Intel
> XL710-QDA2 40G, the Mellanox ConnectX-5 100G, and the ConnectX-4 Lx 25G and
> 40G.
>
> We don't apply the doc folder when applying the series, in case a patch
> included code unrelated to documentation. If others in the group feel that
> it's still unnecessary to include "doc" labeled series, I can look into
> filtering them out.
>
> Thanks.
>
>
> On Fri, Jan 4, 2019 at 5:46 AM Rami Rosen <ramirose@gmail.com> wrote:
>
>> Hi,
>> I have a question about DPDK CI process and the tests done when a patch
>> is submitted to dpdk-dev mailing list.
>> In DPDK patch work I see these response messages from the DPDK CI for all
>> patches:
>>
>> ...
>> ci/intel-Performance-Testing success Performance Testing PASS
>> ci/mellanox-Performance-Testing success Performance Testing PASS
>> ...
>>
>> My question is (I hope and believe the info is available publicly) :
>> which tests are run in the ci, generating these messages?  is it done with
>> IXIA and DTS ? (DPDK test suite, https://doc.dpdk.org/dts/gsg/) ? are
>> these l2fwd/l3fwd performance tests? or more than that ? and on which
>> Intel/Mellanox nics ?  Are these merely performance tests, or also
>> functional tests ?
>>
>> And BTW, I noticed that the CI runs a full performance cycle also for doc
>> patches (at least these messages are generated), which is a kind of
>> redundant (unless there is some filter which checks that if a patch only
>> affects modules under "doc", than such a cycle is not done but the messages
>> are still sent)
>>
>> Regards,
>> Rami Rosen
>>
>
>
> --
> Jeremy Plsek
> UNH InterOperability Laboratory
>
>

[-- Attachment #2: Type: text/html, Size: 6182 bytes --]

Re: [dpdk-ci] Question about performance test

[Thomas Monjalon]

04/01/2019 15:44, Jeremy Plsek:
> Hi Rami,
> 
> I'm the current maintainer of the DPDK Performance CI. I realize that the
> performance results don't point to the website, so it's not obvious on
> where to find this information. You can find an overview of these tests
> here: https://lab.dpdk.org
> 
> Most of this information can be either found on the detailed results of a
> test (such as https://lab.dpdk.org/results/dashboard/patchsets/4157/) or on
> the about page (https://lab.dpdk.org/results/dashboard/about/).

Do you plan to add such URL in the report sent to patchwork?

> We don't apply the doc folder when applying the series, in case a patch
> included code unrelated to documentation. If others in the group feel that
> it's still unnecessary to include "doc" labeled series, I can look into
> filtering them out.

Instead of filtering based on the label, you could filter based on
the paths of modified files.
Note that such filter depends on the test you run,
because you could also test the doc syntax in the CI.

Re: [dpdk-ci] Question about performance test

[Jeremy Plsek]

[-- Attachment #1: Type: text/plain, Size: 2787 bytes --]

>
> I do not see the actual results of the DTS perf test in the links you
> posted, only percentage of degradation or improvement, unless I miss
> something.
>

This was done on purpose as requested by the Members participating in the
effort. This CI is meant for the DPDK maintainers to make sure that a patch
does not introduce significant performance regressions on various hardware
platforms, or to show how a patch may improve the performance of DPDK on
these platforms (such as driver updates or something in the core of DPDK
itself). It is not to compare the performance between different devices.

I believe it can be helpful if the baseline of the actual results will also
> be shown
> to enable comparing to other vendors besides Intel and Mellanox.
>

There is not a single baseline for all devices. The baselines are generated
per device. Knowing the baseline would allow creating an absolute result,
which we are trying to avoid.

Do you plan to add such URL in the report sent to patchwork?
>

Created a ticket to add the URL to the emailed reports:
https://bugs.dpdk.org/show_bug.cgi?id=180

Instead of filtering based on the label, you could filter based on
> the paths of modified files.
> Note that such filter depends on the test you run,
> because you could also test the doc syntax in the CI.
>

Okay, so if a series only modified the doc folder, then don't include it
for performance tests. Later on, we can introduce syntax checking for
documentation when we introduce more unit testing / functional testing.
https://bugs.dpdk.org/show_bug.cgi?id=181

On Fri, Jan 4, 2019 at 10:34 AM Thomas Monjalon <thomas@monjalon.net> wrote:

> 04/01/2019 15:44, Jeremy Plsek:
> > Hi Rami,
> >
> > I'm the current maintainer of the DPDK Performance CI. I realize that the
> > performance results don't point to the website, so it's not obvious on
> > where to find this information. You can find an overview of these tests
> > here: https://lab.dpdk.org
> >
> > Most of this information can be either found on the detailed results of a
> > test (such as https://lab.dpdk.org/results/dashboard/patchsets/4157/)
> or on
> > the about page (https://lab.dpdk.org/results/dashboard/about/).
>
> Do you plan to add such URL in the report sent to patchwork?
>
> > We don't apply the doc folder when applying the series, in case a patch
> > included code unrelated to documentation. If others in the group feel
> that
> > it's still unnecessary to include "doc" labeled series, I can look into
> > filtering them out.
>
> Instead of filtering based on the label, you could filter based on
> the paths of modified files.
> Note that such filter depends on the test you run,
> because you could also test the doc syntax in the CI.
>
>
>

-- 
Jeremy Plsek
UNH InterOperability Laboratory

[-- Attachment #2: Type: text/html, Size: 4895 bytes --]

Re: [dpdk-ci] Question about performance test

[Thomas Monjalon]

04/01/2019 17:15, Jeremy Plsek:
> > Do you plan to add such URL in the report sent to patchwork?
> 
> Created a ticket to add the URL to the emailed reports:
> https://bugs.dpdk.org/show_bug.cgi?id=180
> 
> > Instead of filtering based on the label, you could filter based on
> > the paths of modified files.
> > Note that such filter depends on the test you run,
> > because you could also test the doc syntax in the CI.
> 
> Okay, so if a series only modified the doc folder, then don't include it
> for performance tests. Later on, we can introduce syntax checking for
> documentation when we introduce more unit testing / functional testing.
> https://bugs.dpdk.org/show_bug.cgi?id=181

Yes, thank you

Re: [dpdk-ci] Question about performance test

[Rami Rosen]

[-- Attachment #1: Type: text/plain, Size: 341 bytes --]

Hi Jeremy,
>At the moment, we only run performance tests. Specifically the
nic_single_core_perf_test from the DPDK Test Suite

I had looked at the latest DTS repo from today:

git grep nic_single_core_perf_test
gives nothing.

Can you be more specific about exactly which *.py module from under "tests"
folder is used ?

Regards,
Rami Rosen

[-- Attachment #2: Type: text/html, Size: 625 bytes --]

Re: [dpdk-ci] Question about performance test

[Tu, Lijuan]

[-- Attachment #1: Type: text/plain, Size: 921 bytes --]

Hi Rami,

nic_single_core_perf_test is in the next branch of dts, so it can’t be found in the master branch.
Since next branch is to rework packet generator module to support a software packet generator called TREX.
A software packet generator is easily to set up with little resources comparing with hardware packet generator, IXIA.



From: ci [mailto:ci-bounces@dpdk.org] On Behalf Of Rami Rosen
Sent: Tuesday, January 8, 2019 2:40 PM
To: Jeremy Plsek <jplsek@iol.unh.edu>
Cc: ci@dpdk.org
Subject: Re: [dpdk-ci] Question about performance test

Hi Jeremy,
>At the moment, we only run performance tests. Specifically the nic_single_core_perf_test from the DPDK Test Suite

I had looked at the latest DTS repo from today:

git grep nic_single_core_perf_test
gives nothing.

Can you be more specific about exactly which *.py module from under "tests" folder is used ?

Regards,
Rami Rosen



[-- Attachment #2: Type: text/html, Size: 5324 bytes --]

Re: [dpdk-ci] Question about performance test

[Rami Rosen]

[-- Attachment #1: Type: text/plain, Size: 1797 bytes --]

Hi Lijuan,
Thanks for the quick response, your replies clarifies everything now!

And Indeed:
git checkout next
Switched to branch 'next'

git grep nic_single_core_perf
doc/dts_gsg/trex.rst:This is supported for the ``nic_single_core_per
framework/etgen.py:        This function is set only for function se
framework/etgen.py:        Note that this function is only set for t
test_plans/nic_single_core_perf_test_plan.rst:    1) nic_single_core
test_plans/nic_single_core_perf_test_plan.rst:    3) nic_single_core
tests/TestSuite_nic_single_core_perf.py:    def test_nic_single_core

Regards,
Rami Rosen



On Tue, Jan 8, 2019 at 9:06 AM Tu, Lijuan <lijuan.tu@intel.com> wrote:

> Hi Rami,
>
>
>
> nic_single_core_perf_test is in the next branch of dts, so it can’t be
> found in the master branch.
>
> Since next branch is to rework packet generator module to support a
> software packet generator called TREX.
>
> A software packet generator is easily to set up with little resources
> comparing with hardware packet generator, IXIA.
>
>
>
>
>
>
>
> *From:* ci [mailto:ci-bounces@dpdk.org] *On Behalf Of *Rami Rosen
> *Sent:* Tuesday, January 8, 2019 2:40 PM
> *To:* Jeremy Plsek <jplsek@iol.unh.edu>
> *Cc:* ci@dpdk.org
> *Subject:* Re: [dpdk-ci] Question about performance test
>
>
>
> Hi Jeremy,
>
> >At the moment, we only run performance tests. Specifically the
> nic_single_core_perf_test from the DPDK Test Suite
>
>
>
> I had looked at the latest DTS repo from today:
>
>
>
> git grep nic_single_core_perf_test
>
> gives nothing.
>
>
>
> Can you be more specific about exactly which *.py module from under
> "tests" folder is used ?
>
>
>
> Regards,
>
> Rami Rosen
>
>
>
>
>


-- 
regards,
Rami Rosen

[-- Attachment #2: Type: text/html, Size: 5299 bytes --]