From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id B43B3A057C for ; Thu, 26 Mar 2020 16:39:48 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7CF362C15; Thu, 26 Mar 2020 16:39:48 +0100 (CET) Received: from mail-io1-f46.google.com (mail-io1-f46.google.com [209.85.166.46]) by dpdk.org (Postfix) with ESMTP id E9FCD2BAE for ; Thu, 26 Mar 2020 16:39:45 +0100 (CET) Received: by mail-io1-f46.google.com with SMTP id m15so6497530iob.5 for ; Thu, 26 Mar 2020 08:39:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=UFhjAC+rfDuOQxqSwecNCY74Elh+sV+JPo2s+lnWf78=; b=H4Sz3yJSVaxNEapAdiyhOYgrI+DGfF2gEE4iBcZebii11WtHgE+oKwf6QFaWTiyCDd rT0zX1SDlFHhnmiiPba2bw5eD3kEs+SV3teGd2VCVeFFH0Jofnb1LkupspjKtue3oW1X wmgWxdKin8Jv/k6zPdsXg8XPKpuNzpaPNAKmE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=UFhjAC+rfDuOQxqSwecNCY74Elh+sV+JPo2s+lnWf78=; b=geLC7stqlXYdlytqPIPqwllLqgRbdr8zqT0jWdIt+MMBPtJqD8a98NDO2/7jTOq007 1JFm54ZOIZTtEGNU4VQw590kdJCrnu2zZcj/QDqkorpaAv6/a36cytYwFkmYYmjBz7nC EjIWxMFkQTrbbL0vKER9dskfa3t7WlAPOWk/g05kut8V3PrNUV81MmqdZnaOAukl1FhR 3Tg2NPfrAeILXv2NLwSQo9QTxLmm45s1oe0yJTiI3m6QtmmXdAjAqOsczPiiRHxuleah NVeQAhubD1Bc1U9mQQKUysG1qJ3BAVR4K0jJjVXgFp4LNYUG4laiRFYJeUxnFxPI0R3U hDjw== X-Gm-Message-State: ANhLgQ1XblDfak0GtMgGilcoSOV/qWdK7vdOUHYL6ZZzGFmLY+u7a0Fu 7RfTSIgR8K6Ou6WUzqYARO4dKeuuYVzrCuRXDX9llA== X-Google-Smtp-Source: ADFU+vsYB+wuK8XfyprfKwu64kocBL4/caSUrWAfSxr9XabTPAekLMd9uQWOjDldbnKwbhkhFR4ecKbR2qXSBGhwtGk= X-Received: by 2002:a5d:84d0:: with SMTP id z16mr8028787ior.88.1585237185124; Thu, 26 Mar 2020 08:39:45 -0700 (PDT) MIME-Version: 1.0 References: <1827408.xz2uEaWSZ7@xps> <9DEEADBC57E43F4DA73B571777FECECA41EAB066@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41EBB4AF@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41EFB9C5@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41F01113@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41F025A1@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41F03B43@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41F058DF@SHSMSX104.ccr.corp.intel.com> In-Reply-To: <9DEEADBC57E43F4DA73B571777FECECA41F058DF@SHSMSX104.ccr.corp.intel.com> From: Brandon Lo Date: Thu, 26 Mar 2020 11:39:08 -0400 Message-ID: To: "Chen, Zhaoyan" Cc: David Marchand , "dpdklab@iol.unh.edu" , Lincoln Lavoie , Thomas Monjalon , "ci@dpdk.org" , "Tu, Lijuan" , "Xu, Qian Q" , "Ma, LihongX" , "Zhang, XuemingX" , "O'Driscoll, Tim" Content-Type: multipart/alternative; boundary="0000000000004df3ed05a1c3cbb1" Subject: Re: [dpdk-ci] [dpdklab] Re: Intel performance test is failing X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" --0000000000004df3ed05a1c3cbb1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Zhaoyan, Currently, we have a system in place that resets any changes made while testing is enabled for a machine. If you would like, I can disable testing and allow you to make permanent changes. I can also reset the baseline of Intel 10G test performance once you make these changes. Please let me know if you would like to make permanent changes on the Intel 10G so I can disable it for you. Thanks On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan wrote: > Thanks. Brandon. > > > > That=E2=80=99s good. We have made changed on 10G testbed. > > > > I monitored the several execution results; I found the results of 10G > always has -0.9%~-1.x% gap against expected number. So it could lead to s= ee > sometime failures..+-1% I suggest adjusting the expected number. I don=E2= =80=99t > know where the expected number is from? as I know it a dynamic number? > depends on baseline.. Please help to clarify, thanks. > > > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Tuesday, March 24, 2020 9:31 PM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Ma, LihongX ; Zhang, > XuemingX > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > I have enabled the 10G Intel machine for testing. > > If you would like to make any more changes, please let me know so I can > perform the necessary steps to prepare the machine for changes. > > Please feel free to let me know if you need anything. > > > > Thank you > > > > On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan > wrote: > > Hi, Brandon, > > > > For 10G, please enable it. our code is at original path > */opt/test-harness/dts.* > > > > For 40G, please keep running. and see if any issue. But, anyway, we have > modified the DTS code at /opt/test-harness/dts-new-suite. If we met same > problem, then use this new DTS instead. > > > > Thanks a lot > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Saturday, March 21, 2020 1:49 AM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Ma, LihongX ; Zhang, > XuemingX > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, the 40G machine is stable enough to be put on production > dashboard to run tests which may cause Trex to be killed. > > Should I disable the 40G Intel machine for you to make changes? > > > > Also, just for confirmation: on the 10G machine, is the folder that you > are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r > are you still using the one in the standard */opt/test-harness/dts* > folder? > > > > If everything is ok, I will enable the 10G machine for production testing= . > > > > Thank you very much > > > > On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan > wrote: > > Brandon, > > > > We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Coul= d > you please help to recover them? > > > > But, for FVL(40G) testbed, we met some problems, could you please help t= o > check before recover it > > - Sometime 1G hugepage will be changed to 2Mhugepage > automatically...we have to restart the system > - When we debugging on the testbed, found that Trex was killed by some > one(app).. > > Please help to check if any other program running on the testbed. > > > > Thanks a lot. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Chen, Zhaoyan > *Sent:* Wednesday, March 18, 2020 9:04 PM > *To:* Brandon Lo > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Chen, Zhaoyan > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Brandon, we almost made a workaround. > > > > Maybe tomorrow, you could recover Intel=E2=80=99s testbed. I will let you= know > soon. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Wednesday, March 18, 2020 3:34 AM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Have you finished making changes on the Intel machine? > > I will turn on the machine on March 3rd for testing if you do not have an= y > issues with it. > > Please let me know if you need anything else. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan > wrote: > > Hi, Brandon, > > > > Yes, it=E2=80=99s a wired issue. And it also mixed our DTS upgrading and = Trex > upgrading. > > So we are reviewing our DTS script, different Trex version, and CI callin= g > procedure. > > > > Anyway, we are focusing on this task recently, any update will let you > know. > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Tuesday, March 10, 2020 10:46 PM > *To:* David Marchand > *Cc:* Chen, Zhaoyan ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > How is the current status of the Intel 82599ES? > > Were there any configuration changes made to fix performance issues? > > > > Thanks > > > > On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo wrote: > > Hi David, > > > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > > I'll rerun the tests that were affected and keep an eye out to see if it'= s > stable enough to be put back online. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand > wrote: > > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header)= . > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > --=20 Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu --0000000000004df3ed05a1c3cbb1 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Zhaoyan,

Currently, we have a system= in place that resets any changes made while testing is enabled for a machi= ne.
If you would like, I can disable testing and allow you to mak= e permanent changes.

I can also reset the baseline= of Intel 10G test performance once you make these changes.
Pleas= e let me know if you would like to make permanent changes on the Intel 10G = so I can disable it for you.

Thanks
On Wed, M= ar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> wrote:

Thanks. Brandon.

=C2=A0

That=E2=80=99s good. We have made changed= on 10G testbed.

=C2=A0

I monitored the several execution results= ; I found the results of 10G always has -0.9%~-1.x% gap against expected nu= mber. So it could lead to see sometime failures..+-1% I suggest adjusting the expected number. I don=E2=80=99t know where the ex= pected number is from? as I know it a dynamic number? depends on baseline..= Please help to clarify, thanks.

=C2=A0

=C2=A0

Thanks.

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Brandon Lo <blo@iol.unh.edu>
Sent: Tuesday, March 24, 2020 9:31 PM
To: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie = <lylavoie@iol.= unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q= <qian.q.xu@int= el.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Zhaoyan,

=C2=A0

I have enabled the 10G Intel machine for testing.=

If you would like to make any more changes, please l= et me know so I can perform the necessary steps to prepare the machine for = changes.

Please feel free to let me know if you need anything= .

=C2=A0

Thank you

=C2=A0

On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@inte= l.com> wrote:

Hi, Brandon,

=C2=A0

For 10G, please enable it. our code is at= original path /opt/test-harness/dts.

=C2=A0

For 40G, please keep running. and see if = any issue. But, anyway, we have modified the DTS code at /opt/test-harness/= dts-new-suite. If we met same problem, then use this new DTS instead.

=C2=A0

Thanks a lot=

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Brandon Lo <blo@iol.unh.edu>
Sent: Saturday, March 21, 2020 1:49 AM
To: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Zhaoyan,

=C2=A0

Currently, the 40G machine is stable enough to be pu= t on production dashboard to run tests which may cause Trex to be killed.

Should I disable the 40G Intel machine for you to ma= ke changes?

=C2=A0

Also, just for confirmation: on the 10G machine, is = the folder that you are using for the testing located in=C2=A0/opt/test-= harness/dts-2020-3-4, or are you still using the one in the standard=C2=A0/opt/test-harness/dts folder?

=C2=A0

If everything is ok, I will enable the 10G machine f= or production testing.

=C2=A0

Thank you very much

=C2=A0

On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@inte= l.com> wrote:

Brandon,

=C2=A0

We worked out a workaround on Intel testb= eds. NNT(10G) and FVL(40G). Could you please help to recover them?

=C2=A0

But, for FVL(40G) testbed, =C2=A0we met s= ome problems, could you please help to check before recover it

  • Sometime 1= G hugepage will be changed to 2Mhugepage automatically...we have to restart= the system
  • When we de= bugging on the testbed, found that Trex was killed by some one(app)..

Please help to check if any other program= running on the testbed.

=C2=A0

Thanks a lot.

=C2=A0

=C2=A0

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Sent: Wednesday, March 18, 2020 9:04 PM
To: Brandon Lo <blo@iol.unh.edu>
Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.ch= en@intel.com>
Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Brandon, we almost made a workaround.

=C2=A0

Maybe tomorrow, you could recover Intel= =E2=80=99s testbed. I will let you know soon.

=C2=A0

=C2=A0

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Brandon Lo <blo@iol.unh.edu>
Sent: Wednesday, March 18, 2020 3:34 AM
To: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Zhaoyan,

=C2=A0

Have you finished making changes on the Intel machin= e?

I will turn on the machine on March 3rd for testing = if you do not have any issues with it.

Please let me know if you need anything else.=

=C2=A0

Thanks

=C2=A0

On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <<= a href=3D"mailto:zhaoyan.chen@intel.com" target=3D"_blank">zhaoyan.chen@int= el.com> wrote:

Hi, Brandon,

=C2=A0

Yes, it=E2=80=99s a wired issue. And it a= lso mixed our DTS upgrading and Trex upgrading.

So we are reviewing our DTS script, diffe= rent Trex version, and CI calling procedure.

=C2=A0

Anyway, we are focusing on= this task recently, any update will let you know.

=C2=A0

Thanks.

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Brandon Lo <blo@iol.unh.edu>
Sent: Tuesday, March 10, 2020 10:46 PM
To: David Marchand <david.marchand@redhat.com>
Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Zhaoyan,

=C2=A0

How is the current status of the Intel 82599ES?

Were there any configuration changes made to fix per= formance issues?

=C2=A0

Thanks

=C2=A0

On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wro= te:

Hi David,

=C2=A0

This was just a weird issue with the packet generato= r not cleaning itself after a test fast enough before another test.<= u>

I'll rerun the tests that were affected and keep= an eye out to see if it's stable enough to be put back online.<= u>

=C2=A0

Thanks

=C2=A0

On Tue, Mar 10, 2020 at 5:33 AM David Marchand <<= a href=3D"mailto:david.marchand@redhat.com" target=3D"_blank">david.marchan= d@redhat.com> wrote:

On Tue, Mar 3, 2020 at = 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote:
>
> Hi David and Zhaoyan,
>
>
> Yes, those results are related to the Intel machine; I have disabled t= esting for the Intel testbed.
>
> The 82599ES machine is now available for ssh and modifications.

Any news about this?

I received a failure on a patch of mine (changing macros in a ARM header).<= br> https://lab.dpdk.org/results/dashboard/patchsets/9900/

But this time, it is with the 40G Intel nic test.

--
David Marchand


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

blo@iol.unh.edu

www.iol.unh.edu



--

Brandon Lo

UNH = InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

<= a href=3D"http://www.iol.unh.edu/" target=3D"_blank">www.iol.unh.edu

--0000000000004df3ed05a1c3cbb1--