Re: [dpdk-ci] [dpdklab] Re: Intel performance test is failing

[Brandon Lo <blo@iol.unh.edu>]

[-- Attachment #1: Type: text/plain, Size: 10502 bytes --]

Hi Zhaoyan,

To make changes to either Intel machine, please reboot using the command
"reboot_to_rw" as root to reboot the machine into read/write mode.
This command will also disable any testing on the machine.

To re-enable the machine, please run "reboot_to_ro" as root, and it will
save all of the changes that you've made and re-enable testing on the
machine.
I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" instead
of the normal "reboot" while you're making changes.

After you're done, please let me know. I'll have to manually run a test and
update the baseline using our internal CI.

Thank you

On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com>
wrote:

> Hi, Brandon,
>
>
>
> Please let me know how to make change to this reset machine.
> (ip/access...) and disable it.
>
>
>
> After that please help to change the baseline.
>
>
>
>
>
> *Regards,*
>
> *Zhaoyan Chen*
>
>
>
> *From:* Brandon Lo <blo@iol.unh.edu>
> *Sent:* Thursday, March 26, 2020 11:39 PM
> *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com>
> *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu;
> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <
> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu,
> Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang,
> XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <
> tim.odriscoll@intel.com>
> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing
>
>
>
> Hi Zhaoyan,
>
>
>
> Currently, we have a system in place that resets any changes made while
> testing is enabled for a machine.
>
> If you would like, I can disable testing and allow you to make permanent
> changes.
>
>
>
> I can also reset the baseline of Intel 10G test performance once you make
> these changes.
>
> Please let me know if you would like to make permanent changes on the
> Intel 10G so I can disable it for you.
>
>
>
> Thanks
>
>
>
> On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com>
> wrote:
>
> Thanks. Brandon.
>
>
>
> That’s good. We have made changed on 10G testbed.
>
>
>
> I monitored the several execution results; I found the results of 10G
> always has -0.9%~-1.x% gap against expected number. So it could lead to see
> sometime failures..+-1% I suggest adjusting the expected number. I don’t
> know where the expected number is from? as I know it a dynamic number?
> depends on baseline.. Please help to clarify, thanks.
>
>
>
>
>
> Thanks.
>
>
>
> *Regards,*
>
> *Zhaoyan Chen*
>
>
>
> *From:* Brandon Lo <blo@iol.unh.edu>
> *Sent:* Tuesday, March 24, 2020 9:31 PM
> *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com>
> *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu;
> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <
> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu,
> Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang,
> XuemingX <xuemingx.zhang@intel.com>
> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing
>
>
>
> Hi Zhaoyan,
>
>
>
> I have enabled the 10G Intel machine for testing.
>
> If you would like to make any more changes, please let me know so I can
> perform the necessary steps to prepare the machine for changes.
>
> Please feel free to let me know if you need anything.
>
>
>
> Thank you
>
>
>
> On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com>
> wrote:
>
> Hi, Brandon,
>
>
>
> For 10G, please enable it. our code is at original path
> */opt/test-harness/dts.*
>
>
>
> For 40G, please keep running. and see if any issue. But, anyway, we have
> modified the DTS code at /opt/test-harness/dts-new-suite. If we met same
> problem, then use this new DTS instead.
>
>
>
> Thanks a lot
>
>
>
> *Regards,*
>
> *Zhaoyan Chen*
>
>
>
> *From:* Brandon Lo <blo@iol.unh.edu>
> *Sent:* Saturday, March 21, 2020 1:49 AM
> *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com>
> *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu;
> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <
> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu,
> Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang,
> XuemingX <xuemingx.zhang@intel.com>
> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing
>
>
>
> Hi Zhaoyan,
>
>
>
> Currently, the 40G machine is stable enough to be put on production
> dashboard to run tests which may cause Trex to be killed.
>
> Should I disable the 40G Intel machine for you to make changes?
>
>
>
> Also, just for confirmation: on the 10G machine, is the folder that you
> are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r
> are you still using the one in the standard */opt/test-harness/dts*
> folder?
>
>
>
> If everything is ok, I will enable the 10G machine for production testing.
>
>
>
> Thank you very much
>
>
>
> On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com>
> wrote:
>
> Brandon,
>
>
>
> We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could
> you please help to recover them?
>
>
>
> But, for FVL(40G) testbed,  we met some problems, could you please help to
> check before recover it
>
>    - Sometime 1G hugepage will be changed to 2Mhugepage
>    automatically...we have to restart the system
>    - When we debugging on the testbed, found that Trex was killed by some
>    one(app)..
>
> Please help to check if any other program running on the testbed.
>
>
>
> Thanks a lot.
>
>
>
>
>
>
>
> *Regards,*
>
> *Zhaoyan Chen*
>
>
>
> *From:* Chen, Zhaoyan <zhaoyan.chen@intel.com>
> *Sent:* Wednesday, March 18, 2020 9:04 PM
> *To:* Brandon Lo <blo@iol.unh.edu>
> *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu;
> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <
> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu,
> Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com>
> *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing
>
>
>
> Brandon, we almost made a workaround.
>
>
>
> Maybe tomorrow, you could recover Intel’s testbed. I will let you know
> soon.
>
>
>
>
>
>
>
> *Regards,*
>
> *Zhaoyan Chen*
>
>
>
> *From:* Brandon Lo <blo@iol.unh.edu>
> *Sent:* Wednesday, March 18, 2020 3:34 AM
> *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com>
> *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu;
> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <
> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu,
> Qian Q <qian.q.xu@intel.com>
> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing
>
>
>
> Hi Zhaoyan,
>
>
>
> Have you finished making changes on the Intel machine?
>
> I will turn on the machine on March 3rd for testing if you do not have any
> issues with it.
>
> Please let me know if you need anything else.
>
>
>
> Thanks
>
>
>
> On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com>
> wrote:
>
> Hi, Brandon,
>
>
>
> Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex
> upgrading.
>
> So we are reviewing our DTS script, different Trex version, and CI calling
> procedure.
>
>
>
> Anyway, we are focusing on this task recently, any update will let you
> know.
>
>
>
> Thanks.
>
>
>
> *Regards,*
>
> *Zhaoyan Chen*
>
>
>
> *From:* Brandon Lo <blo@iol.unh.edu>
> *Sent:* Tuesday, March 10, 2020 10:46 PM
> *To:* David Marchand <david.marchand@redhat.com>
> *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.edu;
> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <
> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu,
> Qian Q <qian.q.xu@intel.com>
> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing
>
>
>
> Hi Zhaoyan,
>
>
>
> How is the current status of the Intel 82599ES?
>
> Were there any configuration changes made to fix performance issues?
>
>
>
> Thanks
>
>
>
> On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wrote:
>
> Hi David,
>
>
>
> This was just a weird issue with the packet generator not cleaning itself
> after a test fast enough before another test.
>
> I'll rerun the tests that were affected and keep an eye out to see if it's
> stable enough to be put back online.
>
>
>
> Thanks
>
>
>
> On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com>
> wrote:
>
> On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote:
> >
> > Hi David and Zhaoyan,
> >
> >
> > Yes, those results are related to the Intel machine; I have disabled
> testing for the Intel testbed.
> >
> > The 82599ES machine is now available for ssh and modifications.
>
> Any news about this?
>
> I received a failure on a patch of mine (changing macros in a ARM header).
> https://lab.dpdk.org/results/dashboard/patchsets/9900/
>
> But this time, it is with the 40G Intel nic test.
>
> --
> David Marchand
>
>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>
>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>
>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>
>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>
>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>
>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>


-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu

[-- Attachment #2: Type: text/html, Size: 38074 bytes --]