From: Brandon Lo <blo@iol.unh.edu>
To: "Morten Brørup" <mb@smartsharesystems.com>
Cc: Lincoln Lavoie <lylavoie@iol.unh.edu>,
Ferruh Yigit <ferruh.yigit@intel.com>,
Ali Alnubani <alialnu@nvidia.com>,
David Marchand <david.marchand@redhat.com>,
Olivier Matz <olivier.matz@6wind.com>,
"Chen, Zhaoyan" <zhaoyan.chen@intel.com>, dev <dev@dpdk.org>,
Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>,
"Ananyev, Konstantin" <konstantin.ananyev@intel.com>,
ajitkhaparde@gmail.com, dpdk stable <stable@dpdk.org>,
Ajit Khaparde <ajit.khaparde@broadcom.com>,
dpdklab <dpdklab@iol.unh.edu>
Subject: Re: [dpdk-dev] [dpdklab] RE: [dpdk-stable] [PATCH v4] mbuf: fix reset on mbuf free
Date: Mon, 25 Jan 2021 12:00:18 -0500 [thread overview]
Message-ID: <CAOeXdvbNe+Zhzk26XM_0Lw+wKt_s5UREj35XevWLUKLJp0LO2Q@mail.gmail.com> (raw)
In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35C61599@smartserver.smartshare.dk>
Hi,
This seems like a good task for us to do. I will see what it would
take in order to convert the difference into a decimal-formatted
percentage.
I have put this into bugzilla to keep track of this issue:
https://bugs.dpdk.org/show_bug.cgi?id=626
Thanks,
Brandon
On Sat, Jan 23, 2021 at 3:57 AM Morten Brørup <mb@smartsharesystems.com> wrote:
>
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Lincoln Lavoie
> > Sent: Thursday, January 21, 2021 5:35 PM
> > To: Morten Brørup
> >
> > Hi All,
> >
> > Trying to follow the specific conversation. It is correct, the lab
> > does
> > not list the specific throughput values achieved by the hardware, as
> > that
> > data can be sensitive to the hardware vendors, etc. The purpose of the
> > lab
> > is to check for degradations caused by patches, so the difference is
> > really
> > the important factor. The comparison is against a prior run on the
> > same
> > hardware, via the DPDK main branch, so any delta should be caused by
> > the
> > specific patch changes (excluding statistical "wiggle").
> >
>
> Thank you for clarifying, Lincoln.
>
> This sounds like a perfect solution to the meet the purpose.
>
> I request that you change the output to show the relative difference (i.e. percentage above/below baseline), instead of the absolute difference (i.e. number of packets per second).
>
> By showing a percentage, anyone reading the test report can understand if the difference is insignificant, or big enough to require further discussion before accepting the patch. Showing the difference in packets per second requires the reader of the test report to have prior knowledge about the expected performance.
>
> > If the group would prefer, we could calculate additional references if
> > desired (i.e. difference from the last official release, or a monthly
> > run
> > of the current, etc.). We just need the community to define their
> > needs,
> > and we can add this to the development queue.
> >
>
> I retract my suggestion about adding additional references. You clearly explained how your baselining works, and I think it fully serves the needs of a regression test.
>
> So please just put this small change in your development queue: Output the difference in percent with a few decimals after the comma, instead of the difference in packets per second.
>
> For readability, performance drops should be shown as negative values, and increases as positive, e.g.:
>
> Difference = (NewPPS - BaselinePPS) / BaselinePPS * 100.00 %.
>
>
> If we want to compare performance against official releases, current or older, it should go elsewhere, not in the continuous testing environment. E.g. the release notes could include a report showing differences from the last few official releases. But that is a task for another day.
>
>
> > Cheers,
> > Lincoln
> >
> >
> > On Thu, Jan 21, 2021 at 4:29 AM Morten Brørup
> > <mb@smartsharesystems.com>
> > wrote:
> >
> > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit
> > > > Sent: Thursday, January 21, 2021 10:19 AM
> > > >
> > > > On 1/15/2021 6:39 PM, Ali Alnubani wrote:
> > > > > Hi,
> > > > > Adding Ferruh and Zhaoyan,
> > > > >
> > > > >> Ali,
> > > > >>
> > > > >> You reported some performance regression, did you confirm it?
> > > > >> If I get no reply by monday, I'll proceed with this patch.
> > > > >
> > > > > Sure I'll confirm by Monday.
> > > > >
> > > > > Doesn't the regression also reproduce on the Lab's Intel servers?
> > > > > Even though the check iol-intel-Performance isn't failing, I can
> > see
> > > > that the throughput differences from expected for this patch are
> > less
> > > > than those of another patch that was tested only 20 minutes
> > earlier.
> > > > Both patches were applied to the same tree:
> > > > >
> > > > > https://mails.dpdk.org/archives/test-report/2021-
> > January/173927.html
> > > > >> | 64 | 512 | 1.571 |
> > > > >
> > > > > https://mails.dpdk.org/archives/test-report/2021-
> > January/173919.html
> > > > >> | 64 | 512 | 2.698 |
> > > > >
> > > > > Assuming that pw86457 doesn't have an effect on this test, it
> > looks
> > > > to me that this patch caused a regression in Intel hardware as
> > well.
> > > > >
> > > > > Can someone update the baseline's expected values for the Intel
> > NICs
> > > > and rerun the test on this patch?
> > > > >
> > > >
> > > > Zhaoyan said that the baseline is calculated dynamically,
> > > > what I understand is baseline set based on previous days
> > performance
> > > > result, so
> > > > it shouldn't require updating.
> > >
> > > That sounds smart!
> > >
> > > Perhaps another reference baseline could be added, for informational
> > > purposes only:
> > > Deviation from the performance of the last official release.
> > >
> > > >
> > > > But cc'ed the lab for more details.
> > >
> > >
> >
> > --
> > *Lincoln Lavoie*
> > Senior Engineer, Broadband Technologies
> > 21 Madbury Rd., Ste. 100, Durham, NH 03824
> > lylavoie@iol.unh.edu
> > https://www.iol.unh.edu
> > +1-603-674-2755 (m)
> > <https://www.iol.unh.edu>
>
--
Brandon Lo
UNH InterOperability Laboratory
21 Madbury Rd, Suite 100, Durham, NH 03824
blo@iol.unh.edu
www.iol.unh.edu
next prev parent reply other threads:[~2021-01-25 17:00 UTC|newest]
Thread overview: 74+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-04 17:00 [dpdk-dev] [PATCH] " Olivier Matz
2020-11-05 0:15 ` Ananyev, Konstantin
2020-11-05 7:46 ` Olivier Matz
2020-11-05 8:26 ` Andrew Rybchenko
2020-11-05 9:10 ` Olivier Matz
2020-11-05 11:34 ` Ananyev, Konstantin
2020-11-05 12:31 ` Olivier Matz
2020-11-05 13:14 ` Ananyev, Konstantin
2020-11-05 13:24 ` Olivier Matz
2020-11-05 13:55 ` Ananyev, Konstantin
2020-11-05 16:30 ` Morten Brørup
2020-11-05 23:55 ` Ananyev, Konstantin
2020-11-06 7:52 ` Morten Brørup
2020-11-06 8:20 ` Olivier Matz
2020-11-06 8:50 ` Morten Brørup
2020-11-06 10:04 ` Olivier Matz
2020-11-06 10:07 ` Morten Brørup
2020-11-06 11:53 ` Ananyev, Konstantin
2020-11-06 12:23 ` Morten Brørup
2020-11-08 14:16 ` Andrew Rybchenko
2020-11-08 14:19 ` Ananyev, Konstantin
2020-11-10 16:26 ` Olivier Matz
2020-11-05 8:33 ` Morten Brørup
2020-11-05 9:03 ` Olivier Matz
2020-11-05 9:09 ` Andrew Rybchenko
2020-11-08 7:25 ` Ali Alnubani
2020-12-18 12:52 ` [dpdk-dev] [PATCH v2] " Olivier Matz
2020-12-18 13:18 ` Morten Brørup
2020-12-18 23:33 ` Ajit Khaparde
2021-01-06 13:33 ` [dpdk-dev] [PATCH v3] " Olivier Matz
2021-01-10 9:28 ` Ali Alnubani
2021-01-11 13:14 ` Ananyev, Konstantin
2021-01-13 13:27 ` [dpdk-dev] [PATCH v4] " Olivier Matz
2021-01-15 13:59 ` [dpdk-dev] [dpdk-stable] " David Marchand
2021-01-15 18:39 ` Ali Alnubani
2021-01-18 17:52 ` Ali Alnubani
2021-01-19 8:32 ` Olivier Matz
2021-01-19 8:53 ` Morten Brørup
2021-01-19 12:00 ` Ferruh Yigit
2021-01-19 12:27 ` Morten Brørup
2021-01-19 14:03 ` Ferruh Yigit
2021-01-19 14:21 ` Morten Brørup
2021-01-21 9:15 ` Ferruh Yigit
2021-01-19 14:04 ` Slava Ovsiienko
2021-07-24 8:47 ` Thomas Monjalon
2021-07-30 12:36 ` Olivier Matz
2021-07-30 14:35 ` Morten Brørup
2021-07-30 14:54 ` Thomas Monjalon
2021-07-30 15:14 ` Olivier Matz
2021-07-30 15:23 ` Morten Brørup
2021-08-04 13:29 ` [dpdk-dev] [PATCH] doc: add known issue with mbuf segment Thomas Monjalon
2021-08-04 14:25 ` Ajit Khaparde
2021-08-05 6:08 ` Morten Brørup
2021-08-06 14:21 ` Thomas Monjalon
2021-08-06 14:24 ` Morten Brørup
2021-09-28 8:28 ` [dpdk-dev] [dpdk-stable] [PATCH v4] mbuf: fix reset on mbuf free Thomas Monjalon
2021-09-28 9:00 ` Slava Ovsiienko
2021-09-28 9:25 ` Ananyev, Konstantin
2021-09-28 9:39 ` Morten Brørup
2021-09-29 8:03 ` Ali Alnubani
2021-09-29 21:39 ` Olivier Matz
2021-09-30 13:29 ` Ali Alnubani
2021-10-21 8:26 ` Thomas Monjalon
2021-01-21 9:19 ` Ferruh Yigit
2021-01-21 9:29 ` Morten Brørup
2021-01-21 16:35 ` [dpdk-dev] [dpdklab] " Lincoln Lavoie
2021-01-23 8:57 ` Morten Brørup
2021-01-25 17:00 ` Brandon Lo [this message]
2021-01-25 18:42 ` Ferruh Yigit
2021-06-15 13:56 ` [dpdk-dev] " Morten Brørup
2021-09-29 21:37 ` [dpdk-dev] [PATCH v5] " Olivier Matz
2021-09-30 13:27 ` Ali Alnubani
2021-10-21 9:18 ` David Marchand
2022-07-28 14:06 ` CI performance test results might be misleading Morten Brørup
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAOeXdvbNe+Zhzk26XM_0Lw+wKt_s5UREj35XevWLUKLJp0LO2Q@mail.gmail.com \
--to=blo@iol.unh.edu \
--cc=ajit.khaparde@broadcom.com \
--cc=ajitkhaparde@gmail.com \
--cc=alialnu@nvidia.com \
--cc=andrew.rybchenko@oktetlabs.ru \
--cc=david.marchand@redhat.com \
--cc=dev@dpdk.org \
--cc=dpdklab@iol.unh.edu \
--cc=ferruh.yigit@intel.com \
--cc=konstantin.ananyev@intel.com \
--cc=lylavoie@iol.unh.edu \
--cc=mb@smartsharesystems.com \
--cc=olivier.matz@6wind.com \
--cc=stable@dpdk.org \
--cc=zhaoyan.chen@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).