DPDK CI discussions
 help / color / mirror / Atom feed
From: Brandon Lo <blo@iol.unh.edu>
To: "Chen, Zhaoyan" <zhaoyan.chen@intel.com>
Cc: David Marchand <david.marchand@redhat.com>,
	"dpdklab@iol.unh.edu" <dpdklab@iol.unh.edu>,
	 Lincoln Lavoie <lylavoie@iol.unh.edu>,
	Thomas Monjalon <thomas@monjalon.net>,
	"ci@dpdk.org" <ci@dpdk.org>,  "Tu, Lijuan" <lijuan.tu@intel.com>,
	"Xu, Qian Q" <qian.q.xu@intel.com>,
	 "Ma, LihongX" <lihongx.ma@intel.com>,
	"Zhang, XuemingX" <xuemingx.zhang@intel.com>,
	 "O'Driscoll, Tim" <tim.odriscoll@intel.com>
Subject: Re: [dpdk-ci] [dpdklab] Re:  Intel performance test is failing
Date: Tue, 31 Mar 2020 09:42:00 -0400
Message-ID: <CAOeXdva7sXO6Mtg7WQRz8e7X6Cz0OfOu390Qrfuyv8uU7Hq=ow@mail.gmail.com> (raw)
In-Reply-To: <9DEEADBC57E43F4DA73B571777FECECA41F18C47@SHSMSX104.ccr.corp.intel.com>

[-- Attachment #1: Type: text/plain, Size: 10502 bytes --]

Hi Zhaoyan,

To make changes to either Intel machine, please reboot using the command
"reboot_to_rw" as root to reboot the machine into read/write mode.
This command will also disable any testing on the machine.

To re-enable the machine, please run "reboot_to_ro" as root, and it will
save all of the changes that you've made and re-enable testing on the
machine.
I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" instead
of the normal "reboot" while you're making changes.

After you're done, please let me know. I'll have to manually run a test and
update the baseline using our internal CI.

Thank you

On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com>
wrote:

> Hi, Brandon,
>
>
>
> Please let me know how to make change to this reset machine.
> (ip/access...) and disable it.
>
>
>
> After that please help to change the baseline.
>
>
>
>
>
> *Regards,*
>
> *Zhaoyan Chen*
>
>
>
> *From:* Brandon Lo <blo@iol.unh.edu>
> *Sent:* Thursday, March 26, 2020 11:39 PM
> *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com>
> *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu;
> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <
> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu,
> Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang,
> XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <
> tim.odriscoll@intel.com>
> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing
>
>
>
> Hi Zhaoyan,
>
>
>
> Currently, we have a system in place that resets any changes made while
> testing is enabled for a machine.
>
> If you would like, I can disable testing and allow you to make permanent
> changes.
>
>
>
> I can also reset the baseline of Intel 10G test performance once you make
> these changes.
>
> Please let me know if you would like to make permanent changes on the
> Intel 10G so I can disable it for you.
>
>
>
> Thanks
>
>
>
> On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com>
> wrote:
>
> Thanks. Brandon.
>
>
>
> That’s good. We have made changed on 10G testbed.
>
>
>
> I monitored the several execution results; I found the results of 10G
> always has -0.9%~-1.x% gap against expected number. So it could lead to see
> sometime failures..+-1% I suggest adjusting the expected number. I don’t
> know where the expected number is from? as I know it a dynamic number?
> depends on baseline.. Please help to clarify, thanks.
>
>
>
>
>
> Thanks.
>
>
>
> *Regards,*
>
> *Zhaoyan Chen*
>
>
>
> *From:* Brandon Lo <blo@iol.unh.edu>
> *Sent:* Tuesday, March 24, 2020 9:31 PM
> *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com>
> *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu;
> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <
> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu,
> Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang,
> XuemingX <xuemingx.zhang@intel.com>
> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing
>
>
>
> Hi Zhaoyan,
>
>
>
> I have enabled the 10G Intel machine for testing.
>
> If you would like to make any more changes, please let me know so I can
> perform the necessary steps to prepare the machine for changes.
>
> Please feel free to let me know if you need anything.
>
>
>
> Thank you
>
>
>
> On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com>
> wrote:
>
> Hi, Brandon,
>
>
>
> For 10G, please enable it. our code is at original path
> */opt/test-harness/dts.*
>
>
>
> For 40G, please keep running. and see if any issue. But, anyway, we have
> modified the DTS code at /opt/test-harness/dts-new-suite. If we met same
> problem, then use this new DTS instead.
>
>
>
> Thanks a lot
>
>
>
> *Regards,*
>
> *Zhaoyan Chen*
>
>
>
> *From:* Brandon Lo <blo@iol.unh.edu>
> *Sent:* Saturday, March 21, 2020 1:49 AM
> *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com>
> *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu;
> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <
> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu,
> Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang,
> XuemingX <xuemingx.zhang@intel.com>
> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing
>
>
>
> Hi Zhaoyan,
>
>
>
> Currently, the 40G machine is stable enough to be put on production
> dashboard to run tests which may cause Trex to be killed.
>
> Should I disable the 40G Intel machine for you to make changes?
>
>
>
> Also, just for confirmation: on the 10G machine, is the folder that you
> are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r
> are you still using the one in the standard */opt/test-harness/dts*
> folder?
>
>
>
> If everything is ok, I will enable the 10G machine for production testing.
>
>
>
> Thank you very much
>
>
>
> On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com>
> wrote:
>
> Brandon,
>
>
>
> We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could
> you please help to recover them?
>
>
>
> But, for FVL(40G) testbed,  we met some problems, could you please help to
> check before recover it
>
>    - Sometime 1G hugepage will be changed to 2Mhugepage
>    automatically...we have to restart the system
>    - When we debugging on the testbed, found that Trex was killed by some
>    one(app)..
>
> Please help to check if any other program running on the testbed.
>
>
>
> Thanks a lot.
>
>
>
>
>
>
>
> *Regards,*
>
> *Zhaoyan Chen*
>
>
>
> *From:* Chen, Zhaoyan <zhaoyan.chen@intel.com>
> *Sent:* Wednesday, March 18, 2020 9:04 PM
> *To:* Brandon Lo <blo@iol.unh.edu>
> *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu;
> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <
> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu,
> Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com>
> *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing
>
>
>
> Brandon, we almost made a workaround.
>
>
>
> Maybe tomorrow, you could recover Intel’s testbed. I will let you know
> soon.
>
>
>
>
>
>
>
> *Regards,*
>
> *Zhaoyan Chen*
>
>
>
> *From:* Brandon Lo <blo@iol.unh.edu>
> *Sent:* Wednesday, March 18, 2020 3:34 AM
> *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com>
> *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu;
> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <
> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu,
> Qian Q <qian.q.xu@intel.com>
> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing
>
>
>
> Hi Zhaoyan,
>
>
>
> Have you finished making changes on the Intel machine?
>
> I will turn on the machine on March 3rd for testing if you do not have any
> issues with it.
>
> Please let me know if you need anything else.
>
>
>
> Thanks
>
>
>
> On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com>
> wrote:
>
> Hi, Brandon,
>
>
>
> Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex
> upgrading.
>
> So we are reviewing our DTS script, different Trex version, and CI calling
> procedure.
>
>
>
> Anyway, we are focusing on this task recently, any update will let you
> know.
>
>
>
> Thanks.
>
>
>
> *Regards,*
>
> *Zhaoyan Chen*
>
>
>
> *From:* Brandon Lo <blo@iol.unh.edu>
> *Sent:* Tuesday, March 10, 2020 10:46 PM
> *To:* David Marchand <david.marchand@redhat.com>
> *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.edu;
> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <
> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu,
> Qian Q <qian.q.xu@intel.com>
> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing
>
>
>
> Hi Zhaoyan,
>
>
>
> How is the current status of the Intel 82599ES?
>
> Were there any configuration changes made to fix performance issues?
>
>
>
> Thanks
>
>
>
> On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wrote:
>
> Hi David,
>
>
>
> This was just a weird issue with the packet generator not cleaning itself
> after a test fast enough before another test.
>
> I'll rerun the tests that were affected and keep an eye out to see if it's
> stable enough to be put back online.
>
>
>
> Thanks
>
>
>
> On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com>
> wrote:
>
> On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote:
> >
> > Hi David and Zhaoyan,
> >
> >
> > Yes, those results are related to the Intel machine; I have disabled
> testing for the Intel testbed.
> >
> > The 82599ES machine is now available for ssh and modifications.
>
> Any news about this?
>
> I received a failure on a patch of mine (changing macros in a ARM header).
> https://lab.dpdk.org/results/dashboard/patchsets/9900/
>
> But this time, it is with the 40G Intel nic test.
>
> --
> David Marchand
>
>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>
>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>
>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>
>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>
>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>
>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>


-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu

[-- Attachment #2: Type: text/html, Size: 38074 bytes --]

  reply	other threads:[~2020-03-31 13:42 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-23 15:58 [dpdk-ci] " Thomas Monjalon
2020-01-23 16:25 ` Jeremy Plsek
2020-01-23 20:28   ` Lincoln Lavoie
2020-01-31  3:17     ` Chen, Zhaoyan
2020-01-31 14:48       ` Jeremy Plsek
2020-02-03  3:57         ` Chen, Zhaoyan
2020-03-03  9:21           ` David Marchand
2020-03-03 14:13             ` [dpdk-ci] [dpdklab] " Brandon Lo
2020-03-10  9:32               ` David Marchand
2020-03-10 13:11                 ` Brandon Lo
2020-03-10 14:46                   ` Brandon Lo
2020-03-11  2:13                     ` Chen, Zhaoyan
2020-03-17 19:34                       ` Brandon Lo
2020-03-18 13:04                         ` Chen, Zhaoyan
2020-03-20  1:35                           ` Chen, Zhaoyan
2020-03-20 17:48                             ` Brandon Lo
2020-03-23  1:58                               ` Chen, Zhaoyan
2020-03-24 13:30                                 ` Brandon Lo
2020-03-25  4:59                                   ` Chen, Zhaoyan
2020-03-26 15:39                                     ` Brandon Lo
2020-03-30  4:43                                       ` Chen, Zhaoyan
2020-03-31 13:42                                         ` Brandon Lo [this message]
2020-04-01  6:00                                           ` Ma, LihongX
2020-04-02 18:39                                             ` Brandon Lo
2020-04-03  1:14                                               ` Ma, LihongX
2020-04-15  9:39                                               ` Ma, LihongX
2020-04-15 19:30                                                 ` Brandon Lo
2020-04-16  3:40                                                   ` Ma, LihongX
2020-04-16 15:04                                                     ` Brandon Lo
2020-04-17  1:54                                                       ` Ma, LihongX
2020-04-17  7:00                                                       ` Ma, LihongX
2020-04-17 15:52                                                         ` Brandon Lo
2020-04-18  1:13                                                           ` Ma, LihongX
2020-04-28  3:39                                               ` Ma, LihongX
2020-04-28 16:51                                                 ` Brandon Lo
2020-04-29  5:30                                                   ` Ma, LihongX
2020-04-30 14:19                                                     ` Brandon Lo
2020-05-06 13:05                                                       ` Brandon Lo
2020-05-08  8:02                                                         ` Ma, LihongX
2020-05-12  5:41                                                         ` Ma, LihongX
2020-05-14 16:12                                                           ` Brandon Lo
2020-05-15  1:30                                                             ` Ma, LihongX

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAOeXdva7sXO6Mtg7WQRz8e7X6Cz0OfOu390Qrfuyv8uU7Hq=ow@mail.gmail.com' \
    --to=blo@iol.unh.edu \
    --cc=ci@dpdk.org \
    --cc=david.marchand@redhat.com \
    --cc=dpdklab@iol.unh.edu \
    --cc=lihongx.ma@intel.com \
    --cc=lijuan.tu@intel.com \
    --cc=lylavoie@iol.unh.edu \
    --cc=qian.q.xu@intel.com \
    --cc=thomas@monjalon.net \
    --cc=tim.odriscoll@intel.com \
    --cc=xuemingx.zhang@intel.com \
    --cc=zhaoyan.chen@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK CI discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/ci/0 ci/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 ci ci/ https://inbox.dpdk.org/ci \
		ci@dpdk.org
	public-inbox-index ci

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.ci


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git