Hi, The test iol-intel-Performance is failing on all patches for a couple of days. It seems to be due to a perf drop with a 10G device Can we disable this test until we understand what happens? We need also to understand how to avoid such failure, and how to monitor and disable it faster. Thanks
On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> Hi,
>
> The test iol-intel-Performance is failing on all patches
> for a couple of days.
> It seems to be due to a perf drop with a 10G device
>
> Can we disable this test until we understand what happens?
>
> We need also to understand how to avoid such failure,
> and how to monitor and disable it faster.
>
> Thanks
>
>
Hi Thomas,
I've disabled it for now until someone from the Intel team can look
into it (cc'd).
I've also seem problems where the results are completely off
(-15mpps), and I've been rerunning those when they happen. I'm hoping
those can be resolved too.
--
Jeremy Plsek
UNH InterOperability Laboratory
[-- Attachment #1: Type: text/plain, Size: 1157 bytes --] In the "off cases," is that specific to that hardware, or does it happen on all hardware variants from time-to-time? On Thu, Jan 23, 2020 at 11:25 AM Jeremy Plsek <jplsek@iol.unh.edu> wrote: > On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon <thomas@monjalon.net> > wrote: > > > > Hi, > > > > The test iol-intel-Performance is failing on all patches > > for a couple of days. > > It seems to be due to a perf drop with a 10G device > > > > Can we disable this test until we understand what happens? > > > > We need also to understand how to avoid such failure, > > and how to monitor and disable it faster. > > > > Thanks > > > > > Hi Thomas, > > I've disabled it for now until someone from the Intel team can look > into it (cc'd). > > I've also seem problems where the results are completely off > (-15mpps), and I've been rerunning those when they happen. I'm hoping > those can be resolved too. > > -- > Jeremy Plsek > UNH InterOperability Laboratory > -- *Lincoln Lavoie* Senior Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 lylavoie@iol.unh.edu https://www.iol.unh.edu +1-603-674-2755 (m) <https://www.iol.unh.edu/> [-- Attachment #2: Type: text/html, Size: 2304 bytes --]
[-- Attachment #1: Type: text/plain, Size: 1869 bytes --] Hi, Jeremy and Lincoln, Did you change anything on the Intel testbed? DTS code or testbed environment? And could you send me the detailed test logs? We will do a quick check. Thanks Regards, Zhaoyan Chen From: Lincoln Lavoie <lylavoie@iol.unh.edu> Sent: Friday, January 24, 2020 4:29 AM To: Jeremy Plsek <jplsek@iol.unh.edu> Cc: Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com>; Xu, Qian Q <qian.q.xu@intel.com> Subject: Re: [dpdk-ci] Intel performance test is failing In the "off cases," is that specific to that hardware, or does it happen on all hardware variants from time-to-time? On Thu, Jan 23, 2020 at 11:25 AM Jeremy Plsek <jplsek@iol.unh.edu<mailto:jplsek@iol.unh.edu>> wrote: On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>> wrote: > > Hi, > > The test iol-intel-Performance is failing on all patches > for a couple of days. > It seems to be due to a perf drop with a 10G device > > Can we disable this test until we understand what happens? > > We need also to understand how to avoid such failure, > and how to monitor and disable it faster. > > Thanks > > Hi Thomas, I've disabled it for now until someone from the Intel team can look into it (cc'd). I've also seem problems where the results are completely off (-15mpps), and I've been rerunning those when they happen. I'm hoping those can be resolved too. -- Jeremy Plsek UNH InterOperability Laboratory -- Lincoln Lavoie Senior Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu> https://www.iol.unh.edu +1-603-674-2755 (m) [http://homeautomation.lavoieholdings.com/_/rsrc/1390068882701/unh-iol-logo.png]<https://www.iol.unh.edu/> [-- Attachment #2: Type: text/html, Size: 7413 bytes --]
[-- Attachment #1: Type: text/plain, Size: 3285 bytes --] It's possible this is happening more frequently because of enabling overlayfs. We also changed how the baseline is retrieved because of the overlayfs system, but this issue was happening before this change. I think this started happening after a few months after upgrading to Ubuntu 18.04. Here are the recent results (login to view and download the artifacts): https://lab.dpdk.org/results/dashboard/patchsets/9344/#env-19 https://lab.dpdk.org/results/dashboard/patchsets/9343/#env-19 https://lab.dpdk.org/results/dashboard/patchsets/9340/#env-19 Here are some results that were "off" (see the "Run 1" tab): https://lab.dpdk.org/results/dashboard/patchsets/9346/#env-19 https://lab.dpdk.org/results/dashboard/patchsets/9331/#env-19 https://lab.dpdk.org/results/dashboard/patchsets/9321/#env-19 And here are some older tests where this occurred: https://lab.dpdk.org/results/dashboard/patchsets/8813/#env-19 https://lab.dpdk.org/results/dashboard/patchsets/8795/#env-19 I also found some tests where it happened on the 40g platform: https://lab.dpdk.org/results/dashboard/patchsets/9188/#env-18 https://lab.dpdk.org/results/dashboard/patchsets/8870/#env-18 (older, before enabling overlayfs): https://lab.dpdk.org/results/dashboard/patchsets/7665/#env-18 https://lab.dpdk.org/results/dashboard/patchsets/7653/#env-18 On Thu, Jan 30, 2020 at 10:17 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> wrote: > Hi, Jeremy and Lincoln, > > > > Did you change anything on the Intel testbed? DTS code or testbed > environment? > > And could you send me the detailed test logs? We will do a quick check. > > > > Thanks > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Lincoln Lavoie <lylavoie@iol.unh.edu> > *Sent:* Friday, January 24, 2020 4:29 AM > *To:* Jeremy Plsek <jplsek@iol.unh.edu> > *Cc:* Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan < > lijuan.tu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com>; Xu, Qian Q < > qian.q.xu@intel.com> > *Subject:* Re: [dpdk-ci] Intel performance test is failing > > > > In the "off cases," is that specific to that hardware, or does it happen > on all hardware variants from time-to-time? > > > > On Thu, Jan 23, 2020 at 11:25 AM Jeremy Plsek <jplsek@iol.unh.edu> wrote: > > On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon <thomas@monjalon.net> > wrote: > > > > Hi, > > > > The test iol-intel-Performance is failing on all patches > > for a couple of days. > > It seems to be due to a perf drop with a 10G device > > > > Can we disable this test until we understand what happens? > > > > We need also to understand how to avoid such failure, > > and how to monitor and disable it faster. > > > > Thanks > > > > > Hi Thomas, > > I've disabled it for now until someone from the Intel team can look > into it (cc'd). > > I've also seem problems where the results are completely off > (-15mpps), and I've been rerunning those when they happen. I'm hoping > those can be resolved too. > > -- > Jeremy Plsek > UNH InterOperability Laboratory > > > > > -- > > *Lincoln Lavoie* > > Senior Engineer, Broadband Technologies > > 21 Madbury Rd., Ste. 100, Durham, NH 03824 > > lylavoie@iol.unh.edu > > https://www.iol.unh.edu > > +1-603-674-2755 (m) > > <https://www.iol.unh.edu/> > -- Jeremy Plsek UNH InterOperability Laboratory [-- Attachment #2: Type: text/html, Size: 9225 bytes --]
[-- Attachment #1: Type: text/plain, Size: 4570 bytes --] Thanks Jeremy. I quickly checked per-patches results and logs for the failures. That seems not always failed case.. like fluctuation. For this case, could you help to check - if BIOS’s turbo boost is off, C0/C1 is off, C6 is off - if cores’ are isolated in kernel’s parameter - if other tasks are scheduled on the testbed when running performance test If all settings are good, we may consider it’s a regression or unstable performance issue. We will double check the performance by IXIA with latest DPDK master when we back to office (1 week later). Thanks. Any problem, please let me know. Regards, Zhaoyan Chen From: Jeremy Plsek <jplsek@iol.unh.edu> Sent: Friday, January 31, 2020 10:48 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com> Cc: Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com> Subject: Re: [dpdk-ci] Intel performance test is failing It's possible this is happening more frequently because of enabling overlayfs. We also changed how the baseline is retrieved because of the overlayfs system, but this issue was happening before this change. I think this started happening after a few months after upgrading to Ubuntu 18.04. Here are the recent results (login to view and download the artifacts): https://lab.dpdk.org/results/dashboard/patchsets/9344/#env-19 https://lab.dpdk.org/results/dashboard/patchsets/9343/#env-19 https://lab.dpdk.org/results/dashboard/patchsets/9340/#env-19 Here are some results that were "off" (see the "Run 1" tab): https://lab.dpdk.org/results/dashboard/patchsets/9346/#env-19 https://lab.dpdk.org/results/dashboard/patchsets/9331/#env-19 https://lab.dpdk.org/results/dashboard/patchsets/9321/#env-19 And here are some older tests where this occurred: https://lab.dpdk.org/results/dashboard/patchsets/8813/#env-19 https://lab.dpdk.org/results/dashboard/patchsets/8795/#env-19 I also found some tests where it happened on the 40g platform: https://lab.dpdk.org/results/dashboard/patchsets/9188/#env-18 https://lab.dpdk.org/results/dashboard/patchsets/8870/#env-18 (older, before enabling overlayfs): https://lab.dpdk.org/results/dashboard/patchsets/7665/#env-18 https://lab.dpdk.org/results/dashboard/patchsets/7653/#env-18 On Thu, Jan 30, 2020 at 10:17 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Jeremy and Lincoln, Did you change anything on the Intel testbed? DTS code or testbed environment? And could you send me the detailed test logs? We will do a quick check. Thanks Regards, Zhaoyan Chen From: Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>> Sent: Friday, January 24, 2020 4:29 AM To: Jeremy Plsek <jplsek@iol.unh.edu<mailto:jplsek@iol.unh.edu>> Cc: Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdk-ci] Intel performance test is failing In the "off cases," is that specific to that hardware, or does it happen on all hardware variants from time-to-time? On Thu, Jan 23, 2020 at 11:25 AM Jeremy Plsek <jplsek@iol.unh.edu<mailto:jplsek@iol.unh.edu>> wrote: On Thu, Jan 23, 2020 at 11:06 AM Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>> wrote: > > Hi, > > The test iol-intel-Performance is failing on all patches > for a couple of days. > It seems to be due to a perf drop with a 10G device > > Can we disable this test until we understand what happens? > > We need also to understand how to avoid such failure, > and how to monitor and disable it faster. > > Thanks > > Hi Thomas, I've disabled it for now until someone from the Intel team can look into it (cc'd). I've also seem problems where the results are completely off (-15mpps), and I've been rerunning those when they happen. I'm hoping those can be resolved too. -- Jeremy Plsek UNH InterOperability Laboratory -- Lincoln Lavoie Senior Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu> https://www.iol.unh.edu +1-603-674-2755 (m) [http://homeautomation.lavoieholdings.com/_/rsrc/1390068882701/unh-iol-logo.png]<https://www.iol.unh.edu/> -- Jeremy Plsek UNH InterOperability Laboratory [-- Attachment #2: Type: text/html, Size: 17542 bytes --]
[-- Attachment #1: Type: text/plain, Size: 1180 bytes --] On Mon, Feb 3, 2020 at 4:58 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> wrote: > Thanks Jeremy. > > > > I quickly checked per-patches results and logs for the failures. That > seems not always failed case.. like fluctuation. > > > > For this case, could you help to check > > - if BIOS’s turbo boost is off, C0/C1 is off, C6 is off > > - if cores’ are isolated in kernel’s parameter > > - if other tasks are scheduled on the testbed when running performance test > > > > If all settings are good, we may consider it’s a regression or unstable > performance issue. We will double check the performance by IXIA with latest > DPDK master when we back to office (1 week later). > (replaced jeremy @mail with the dpdklab alias). Is this issue linked to the failures I see? https://lab.dpdk.org/results/dashboard/patchsets/9799/ https://lab.dpdk.org/results/dashboard/patchsets/9800/ https://lab.dpdk.org/results/dashboard/patchsets/9804/ https://lab.dpdk.org/results/dashboard/patchsets/9809/ https://lab.dpdk.org/results/dashboard/patchsets/9814/ https://lab.dpdk.org/results/dashboard/patchsets/9820/ Thanks. -- David Marchand [-- Attachment #2: Type: text/html, Size: 3382 bytes --]
[-- Attachment #1: Type: text/plain, Size: 1691 bytes --] Hi David and Zhaoyan, > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. The 82599ES machine is now available for ssh and modifications. Thanks On Tue, Mar 3, 2020 at 4:22 AM David Marchand <david.marchand@redhat.com> wrote: > On Mon, Feb 3, 2020 at 4:58 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > >> Thanks Jeremy. >> >> >> >> I quickly checked per-patches results and logs for the failures. That >> seems not always failed case.. like fluctuation. >> >> >> >> For this case, could you help to check >> >> - if BIOS’s turbo boost is off, C0/C1 is off, C6 is off >> >> - if cores’ are isolated in kernel’s parameter >> >> - if other tasks are scheduled on the testbed when running performance >> test >> >> >> >> If all settings are good, we may consider it’s a regression or unstable >> performance issue. We will double check the performance by IXIA with latest >> DPDK master when we back to office (1 week later). >> > > (replaced jeremy @mail with the dpdklab alias). > > Is this issue linked to the failures I see? > > https://lab.dpdk.org/results/dashboard/patchsets/9799/ > https://lab.dpdk.org/results/dashboard/patchsets/9800/ > https://lab.dpdk.org/results/dashboard/patchsets/9804/ > https://lab.dpdk.org/results/dashboard/patchsets/9809/ > https://lab.dpdk.org/results/dashboard/patchsets/9814/ > https://lab.dpdk.org/results/dashboard/patchsets/9820/ > > > Thanks. > > -- > David Marchand > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu [-- Attachment #2: Type: text/html, Size: 5789 bytes --]
On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand
[-- Attachment #1: Type: text/plain, Size: 991 bytes --] Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com> wrote: > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header). > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu [-- Attachment #2: Type: text/html, Size: 2927 bytes --]
[-- Attachment #1: Type: text/plain, Size: 1388 bytes --] Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wrote: > Hi David, > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > I'll rerun the tests that were affected and keep an eye out to see if it's > stable enough to be put back online. > > Thanks > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com> > wrote: > >> On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote: >> > >> > Hi David and Zhaoyan, >> > >> > >> > Yes, those results are related to the Intel machine; I have disabled >> testing for the Intel testbed. >> > >> > The 82599ES machine is now available for ssh and modifications. >> >> Any news about this? >> >> I received a failure on a patch of mine (changing macros in a ARM header). >> https://lab.dpdk.org/results/dashboard/patchsets/9900/ >> >> But this time, it is with the 40G Intel nic test. >> >> -- >> David Marchand >> >> > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu [-- Attachment #2: Type: text/html, Size: 4932 bytes --]
[-- Attachment #1: Type: text/plain, Size: 2255 bytes --] Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #2: Type: text/html, Size: 9792 bytes --]
[-- Attachment #1: Type: text/plain, Size: 2799 bytes --] Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> wrote: > Hi, Brandon, > > > > Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex > upgrading. > > So we are reviewing our DTS script, different Trex version, and CI calling > procedure. > > > > Anyway, we are focusing on this task recently, any update will let you > know. > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 10, 2020 10:46 PM > *To:* David Marchand <david.marchand@redhat.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > How is the current status of the Intel 82599ES? > > Were there any configuration changes made to fix performance issues? > > > > Thanks > > > > On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wrote: > > Hi David, > > > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > > I'll rerun the tests that were affected and keep an eye out to see if it's > stable enough to be put back online. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com> > wrote: > > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header). > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu [-- Attachment #2: Type: text/html, Size: 10498 bytes --]
[-- Attachment #1: Type: text/plain, Size: 3604 bytes --] Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com> Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #2: Type: text/html, Size: 16868 bytes --]
[-- Attachment #1: Type: text/plain, Size: 4849 bytes --] Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it * Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system * When we debugging on the testbed, found that Trex was killed by some one(app).. Please help to check if any other program running on the testbed. Thanks a lot. Regards, Zhaoyan Chen From: Chen, Zhaoyan <zhaoyan.chen@intel.com> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <blo@iol.unh.edu> Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #2: Type: text/html, Size: 23291 bytes --]
[-- Attachment #1: Type: text/plain, Size: 5409 bytes --] Hi Zhaoyan, Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. Should I disable the 40G Intel machine for you to make changes? Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r are you still using the one in the standard */opt/test-harness/dts* folder? If everything is ok, I will enable the 10G machine for production testing. Thank you very much On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> wrote: > Brandon, > > > > We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could > you please help to recover them? > > > > But, for FVL(40G) testbed, we met some problems, could you please help to > check before recover it > > - Sometime 1G hugepage will be changed to 2Mhugepage > automatically...we have to restart the system > - When we debugging on the testbed, found that Trex was killed by some > one(app).. > > Please help to check if any other program running on the testbed. > > > > Thanks a lot. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Sent:* Wednesday, March 18, 2020 9:04 PM > *To:* Brandon Lo <blo@iol.unh.edu> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Brandon, we almost made a workaround. > > > > Maybe tomorrow, you could recover Intel’s testbed. I will let you know > soon. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Wednesday, March 18, 2020 3:34 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Have you finished making changes on the Intel machine? > > I will turn on the machine on March 3rd for testing if you do not have any > issues with it. > > Please let me know if you need anything else. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex > upgrading. > > So we are reviewing our DTS script, different Trex version, and CI calling > procedure. > > > > Anyway, we are focusing on this task recently, any update will let you > know. > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 10, 2020 10:46 PM > *To:* David Marchand <david.marchand@redhat.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > How is the current status of the Intel 82599ES? > > Were there any configuration changes made to fix performance issues? > > > > Thanks > > > > On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wrote: > > Hi David, > > > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > > I'll rerun the tests that were affected and keep an eye out to see if it's > stable enough to be put back online. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com> > wrote: > > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header). > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu [-- Attachment #2: Type: text/html, Size: 20192 bytes --]
[-- Attachment #1: Type: text/plain, Size: 6792 bytes --] Hi, Brandon, For 10G, please enable it. our code is at original path /opt/test-harness/dts. For 40G, please keep running. and see if any issue. But, anyway, we have modified the DTS code at /opt/test-harness/dts-new-suite. If we met same problem, then use this new DTS instead. Thanks a lot Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu> Sent: Saturday, March 21, 2020 1:49 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com> Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. Should I disable the 40G Intel machine for you to make changes? Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in /opt/test-harness/dts-2020-3-4, or are you still using the one in the standard /opt/test-harness/dts folder? If everything is ok, I will enable the 10G machine for production testing. Thank you very much On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it * Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system * When we debugging on the testbed, found that Trex was killed by some one(app).. Please help to check if any other program running on the testbed. Thanks a lot. Regards, Zhaoyan Chen From: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #2: Type: text/html, Size: 33765 bytes --]
[-- Attachment #1: Type: text/plain, Size: 6931 bytes --] Hi Zhaoyan, I have enabled the 10G Intel machine for testing. If you would like to make any more changes, please let me know so I can perform the necessary steps to prepare the machine for changes. Please feel free to let me know if you need anything. Thank you On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> wrote: > Hi, Brandon, > > > > For 10G, please enable it. our code is at original path > */opt/test-harness/dts.* > > > > For 40G, please keep running. and see if any issue. But, anyway, we have > modified the DTS code at /opt/test-harness/dts-new-suite. If we met same > problem, then use this new DTS instead. > > > > Thanks a lot > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Saturday, March 21, 2020 1:49 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, the 40G machine is stable enough to be put on production > dashboard to run tests which may cause Trex to be killed. > > Should I disable the 40G Intel machine for you to make changes? > > > > Also, just for confirmation: on the 10G machine, is the folder that you > are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r > are you still using the one in the standard */opt/test-harness/dts* > folder? > > > > If everything is ok, I will enable the 10G machine for production testing. > > > > Thank you very much > > > > On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Brandon, > > > > We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could > you please help to recover them? > > > > But, for FVL(40G) testbed, we met some problems, could you please help to > check before recover it > > - Sometime 1G hugepage will be changed to 2Mhugepage > automatically...we have to restart the system > - When we debugging on the testbed, found that Trex was killed by some > one(app).. > > Please help to check if any other program running on the testbed. > > > > Thanks a lot. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Sent:* Wednesday, March 18, 2020 9:04 PM > *To:* Brandon Lo <blo@iol.unh.edu> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Brandon, we almost made a workaround. > > > > Maybe tomorrow, you could recover Intel’s testbed. I will let you know > soon. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Wednesday, March 18, 2020 3:34 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Have you finished making changes on the Intel machine? > > I will turn on the machine on March 3rd for testing if you do not have any > issues with it. > > Please let me know if you need anything else. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex > upgrading. > > So we are reviewing our DTS script, different Trex version, and CI calling > procedure. > > > > Anyway, we are focusing on this task recently, any update will let you > know. > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 10, 2020 10:46 PM > *To:* David Marchand <david.marchand@redhat.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > How is the current status of the Intel 82599ES? > > Were there any configuration changes made to fix performance issues? > > > > Thanks > > > > On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wrote: > > Hi David, > > > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > > I'll rerun the tests that were affected and keep an eye out to see if it's > stable enough to be put back online. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com> > wrote: > > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header). > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu [-- Attachment #2: Type: text/html, Size: 25817 bytes --]
[-- Attachment #1: Type: text/plain, Size: 8641 bytes --] Thanks. Brandon. That’s good. We have made changed on 10G testbed. I monitored the several execution results; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjusting the expected number. I don’t know where the expected number is from? as I know it a dynamic number? depends on baseline.. Please help to clarify, thanks. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu> Sent: Tuesday, March 24, 2020 9:31 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com> Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, I have enabled the 10G Intel machine for testing. If you would like to make any more changes, please let me know so I can perform the necessary steps to prepare the machine for changes. Please feel free to let me know if you need anything. Thank you On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, For 10G, please enable it. our code is at original path /opt/test-harness/dts. For 40G, please keep running. and see if any issue. But, anyway, we have modified the DTS code at /opt/test-harness/dts-new-suite. If we met same problem, then use this new DTS instead. Thanks a lot Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Saturday, March 21, 2020 1:49 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. Should I disable the 40G Intel machine for you to make changes? Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in /opt/test-harness/dts-2020-3-4, or are you still using the one in the standard /opt/test-harness/dts folder? If everything is ok, I will enable the 10G machine for production testing. Thank you very much On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it * Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system * When we debugging on the testbed, found that Trex was killed by some one(app).. Please help to check if any other program running on the testbed. Thanks a lot. Regards, Zhaoyan Chen From: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #2: Type: text/html, Size: 39065 bytes --]
[-- Attachment #1: Type: text/plain, Size: 8721 bytes --] Hi Zhaoyan, Currently, we have a system in place that resets any changes made while testing is enabled for a machine. If you would like, I can disable testing and allow you to make permanent changes. I can also reset the baseline of Intel 10G test performance once you make these changes. Please let me know if you would like to make permanent changes on the Intel 10G so I can disable it for you. Thanks On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> wrote: > Thanks. Brandon. > > > > That’s good. We have made changed on 10G testbed. > > > > I monitored the several execution results; I found the results of 10G > always has -0.9%~-1.x% gap against expected number. So it could lead to see > sometime failures..+-1% I suggest adjusting the expected number. I don’t > know where the expected number is from? as I know it a dynamic number? > depends on baseline.. Please help to clarify, thanks. > > > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 24, 2020 9:31 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > I have enabled the 10G Intel machine for testing. > > If you would like to make any more changes, please let me know so I can > perform the necessary steps to prepare the machine for changes. > > Please feel free to let me know if you need anything. > > > > Thank you > > > > On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > For 10G, please enable it. our code is at original path > */opt/test-harness/dts.* > > > > For 40G, please keep running. and see if any issue. But, anyway, we have > modified the DTS code at /opt/test-harness/dts-new-suite. If we met same > problem, then use this new DTS instead. > > > > Thanks a lot > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Saturday, March 21, 2020 1:49 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, the 40G machine is stable enough to be put on production > dashboard to run tests which may cause Trex to be killed. > > Should I disable the 40G Intel machine for you to make changes? > > > > Also, just for confirmation: on the 10G machine, is the folder that you > are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r > are you still using the one in the standard */opt/test-harness/dts* > folder? > > > > If everything is ok, I will enable the 10G machine for production testing. > > > > Thank you very much > > > > On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Brandon, > > > > We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could > you please help to recover them? > > > > But, for FVL(40G) testbed, we met some problems, could you please help to > check before recover it > > - Sometime 1G hugepage will be changed to 2Mhugepage > automatically...we have to restart the system > - When we debugging on the testbed, found that Trex was killed by some > one(app).. > > Please help to check if any other program running on the testbed. > > > > Thanks a lot. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Sent:* Wednesday, March 18, 2020 9:04 PM > *To:* Brandon Lo <blo@iol.unh.edu> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Brandon, we almost made a workaround. > > > > Maybe tomorrow, you could recover Intel’s testbed. I will let you know > soon. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Wednesday, March 18, 2020 3:34 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Have you finished making changes on the Intel machine? > > I will turn on the machine on March 3rd for testing if you do not have any > issues with it. > > Please let me know if you need anything else. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex > upgrading. > > So we are reviewing our DTS script, different Trex version, and CI calling > procedure. > > > > Anyway, we are focusing on this task recently, any update will let you > know. > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 10, 2020 10:46 PM > *To:* David Marchand <david.marchand@redhat.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > How is the current status of the Intel 82599ES? > > Were there any configuration changes made to fix performance issues? > > > > Thanks > > > > On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wrote: > > Hi David, > > > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > > I'll rerun the tests that were affected and keep an eye out to see if it's > stable enough to be put back online. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com> > wrote: > > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header). > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu [-- Attachment #2: Type: text/html, Size: 31958 bytes --]
[-- Attachment #1: Type: text/plain, Size: 10408 bytes --] Hi, Brandon, Please let me know how to make change to this reset machine. (ip/access...) and disable it. After that please help to change the baseline. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu> Sent: Thursday, March 26, 2020 11:39 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com> Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, we have a system in place that resets any changes made while testing is enabled for a machine. If you would like, I can disable testing and allow you to make permanent changes. I can also reset the baseline of Intel 10G test performance once you make these changes. Please let me know if you would like to make permanent changes on the Intel 10G so I can disable it for you. Thanks On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Thanks. Brandon. That’s good. We have made changed on 10G testbed. I monitored the several execution results; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjusting the expected number. I don’t know where the expected number is from? as I know it a dynamic number? depends on baseline.. Please help to clarify, thanks. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 24, 2020 9:31 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, I have enabled the 10G Intel machine for testing. If you would like to make any more changes, please let me know so I can perform the necessary steps to prepare the machine for changes. Please feel free to let me know if you need anything. Thank you On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, For 10G, please enable it. our code is at original path /opt/test-harness/dts. For 40G, please keep running. and see if any issue. But, anyway, we have modified the DTS code at /opt/test-harness/dts-new-suite. If we met same problem, then use this new DTS instead. Thanks a lot Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Saturday, March 21, 2020 1:49 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. Should I disable the 40G Intel machine for you to make changes? Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in /opt/test-harness/dts-2020-3-4, or are you still using the one in the standard /opt/test-harness/dts folder? If everything is ok, I will enable the 10G machine for production testing. Thank you very much On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it * Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system * When we debugging on the testbed, found that Trex was killed by some one(app).. Please help to check if any other program running on the testbed. Thanks a lot. Regards, Zhaoyan Chen From: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #2: Type: text/html, Size: 46321 bytes --]
[-- Attachment #1: Type: text/plain, Size: 10502 bytes --] Hi Zhaoyan, To make changes to either Intel machine, please reboot using the command "reboot_to_rw" as root to reboot the machine into read/write mode. This command will also disable any testing on the machine. To re-enable the machine, please run "reboot_to_ro" as root, and it will save all of the changes that you've made and re-enable testing on the machine. I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" instead of the normal "reboot" while you're making changes. After you're done, please let me know. I'll have to manually run a test and update the baseline using our internal CI. Thank you On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> wrote: > Hi, Brandon, > > > > Please let me know how to make change to this reset machine. > (ip/access...) and disable it. > > > > After that please help to change the baseline. > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Thursday, March 26, 2020 11:39 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, we have a system in place that resets any changes made while > testing is enabled for a machine. > > If you would like, I can disable testing and allow you to make permanent > changes. > > > > I can also reset the baseline of Intel 10G test performance once you make > these changes. > > Please let me know if you would like to make permanent changes on the > Intel 10G so I can disable it for you. > > > > Thanks > > > > On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Thanks. Brandon. > > > > That’s good. We have made changed on 10G testbed. > > > > I monitored the several execution results; I found the results of 10G > always has -0.9%~-1.x% gap against expected number. So it could lead to see > sometime failures..+-1% I suggest adjusting the expected number. I don’t > know where the expected number is from? as I know it a dynamic number? > depends on baseline.. Please help to clarify, thanks. > > > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 24, 2020 9:31 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > I have enabled the 10G Intel machine for testing. > > If you would like to make any more changes, please let me know so I can > perform the necessary steps to prepare the machine for changes. > > Please feel free to let me know if you need anything. > > > > Thank you > > > > On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > For 10G, please enable it. our code is at original path > */opt/test-harness/dts.* > > > > For 40G, please keep running. and see if any issue. But, anyway, we have > modified the DTS code at /opt/test-harness/dts-new-suite. If we met same > problem, then use this new DTS instead. > > > > Thanks a lot > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Saturday, March 21, 2020 1:49 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, the 40G machine is stable enough to be put on production > dashboard to run tests which may cause Trex to be killed. > > Should I disable the 40G Intel machine for you to make changes? > > > > Also, just for confirmation: on the 10G machine, is the folder that you > are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r > are you still using the one in the standard */opt/test-harness/dts* > folder? > > > > If everything is ok, I will enable the 10G machine for production testing. > > > > Thank you very much > > > > On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Brandon, > > > > We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could > you please help to recover them? > > > > But, for FVL(40G) testbed, we met some problems, could you please help to > check before recover it > > - Sometime 1G hugepage will be changed to 2Mhugepage > automatically...we have to restart the system > - When we debugging on the testbed, found that Trex was killed by some > one(app).. > > Please help to check if any other program running on the testbed. > > > > Thanks a lot. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Sent:* Wednesday, March 18, 2020 9:04 PM > *To:* Brandon Lo <blo@iol.unh.edu> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Brandon, we almost made a workaround. > > > > Maybe tomorrow, you could recover Intel’s testbed. I will let you know > soon. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Wednesday, March 18, 2020 3:34 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Have you finished making changes on the Intel machine? > > I will turn on the machine on March 3rd for testing if you do not have any > issues with it. > > Please let me know if you need anything else. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex > upgrading. > > So we are reviewing our DTS script, different Trex version, and CI calling > procedure. > > > > Anyway, we are focusing on this task recently, any update will let you > know. > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 10, 2020 10:46 PM > *To:* David Marchand <david.marchand@redhat.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > How is the current status of the Intel 82599ES? > > Were there any configuration changes made to fix performance issues? > > > > Thanks > > > > On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wrote: > > Hi David, > > > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > > I'll rerun the tests that were affected and keep an eye out to see if it's > stable enough to be put back online. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com> > wrote: > > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header). > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu [-- Attachment #2: Type: text/html, Size: 38074 bytes --]
[-- Attachment #1: Type: text/plain, Size: 13093 bytes --] Hi, Brandon Thanks for you recommends, I have done the changes. As the throughput value of nic_single_core is proportional to the cpu frequency. I recommend you can change the baseline according to our report system. On the our 2.50GHz system, the baseline value as below: NNT: pkt_size trd/rxd expected_value 64 512 52.562 64 2048 41.439 FVL: pkt_size trd/rxd expected_value 64 512 59.608 64 2048 47.73 For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number should be NNT: pkt_size trd/rxd expected_value 64 512 52.562 / 2.5 * 2.1=44.152 64 2048 41.439 / 2.5 * 2.1=34.809 FVL: pkt_size trd/rxd expected_value 64 512 59.608 / 2.5 * 2.1=50.071 64 2048 47.73 / 2.5 * 2.1=40.093 Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu] Sent: Tuesday, March 31, 2020 9:42 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com> Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, To make changes to either Intel machine, please reboot using the command "reboot_to_rw" as root to reboot the machine into read/write mode. This command will also disable any testing on the machine. To re-enable the machine, please run "reboot_to_ro" as root, and it will save all of the changes that you've made and re-enable testing on the machine. I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" instead of the normal "reboot" while you're making changes. After you're done, please let me know. I'll have to manually run a test and update the baseline using our internal CI. Thank you On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Please let me know how to make change to this reset machine. (ip/access...) and disable it. After that please help to change the baseline. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Thursday, March 26, 2020 11:39 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, we have a system in place that resets any changes made while testing is enabled for a machine. If you would like, I can disable testing and allow you to make permanent changes. I can also reset the baseline of Intel 10G test performance once you make these changes. Please let me know if you would like to make permanent changes on the Intel 10G so I can disable it for you. Thanks On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Thanks. Brandon. That’s good. We have made changed on 10G testbed. I monitored the several execution results; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjusting the expected number. I don’t know where the expected number is from? as I know it a dynamic number? depends on baseline.. Please help to clarify, thanks. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 24, 2020 9:31 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, I have enabled the 10G Intel machine for testing. If you would like to make any more changes, please let me know so I can perform the necessary steps to prepare the machine for changes. Please feel free to let me know if you need anything. Thank you On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, For 10G, please enable it. our code is at original path /opt/test-harness/dts. For 40G, please keep running. and see if any issue. But, anyway, we have modified the DTS code at /opt/test-harness/dts-new-suite. If we met same problem, then use this new DTS instead. Thanks a lot Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Saturday, March 21, 2020 1:49 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. Should I disable the 40G Intel machine for you to make changes? Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in /opt/test-harness/dts-2020-3-4, or are you still using the one in the standard /opt/test-harness/dts folder? If everything is ok, I will enable the 10G machine for production testing. Thank you very much On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it * Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system * When we debugging on the testbed, found that Trex was killed by some one(app).. Please help to check if any other program running on the testbed. Thanks a lot. Regards, Zhaoyan Chen From: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #2: Type: text/html, Size: 70751 bytes --]
[-- Attachment #1: Type: text/plain, Size: 12806 bytes --] Hi Lihong, I have changed the baselines to reflect the new expected values. The performance tests should work as expected and pass. We will email again in the future if we come across any problems. Feel free to email us as well if you would like to make any other changes. Thank you for all your help On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com> wrote: > Hi, Brandon > > Thanks for you recommends, I have done the changes. > > As the throughput value of nic_single_core is proportional to the cpu > frequency. > > I recommend you can change the baseline according to our report system. > > > > On the our 2.50GHz system, the baseline value as below: > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 > > 64 > > 2048 > > 41.439 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 > > 64 > > 2048 > > 47.73 > > > > For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number > should be > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 / 2.5 * 2.1=44.152 > > 64 > > 2048 > > 41.439 / 2.5 * 2.1=34.809 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 / 2.5 * 2.1=50.071 > > 64 > > 2048 > > 47.73 / 2.5 * 2.1=40.093 > > > > > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Tuesday, March 31, 2020 9:42 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > To make changes to either Intel machine, please reboot using the command > "reboot_to_rw" as root to reboot the machine into read/write mode. > > This command will also disable any testing on the machine. > > > > To re-enable the machine, please run "reboot_to_ro" as root, and it will > save all of the changes that you've made and re-enable testing on the > machine. > > I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" > instead of the normal "reboot" while you're making changes. > > > > After you're done, please let me know. I'll have to manually run a test > and update the baseline using our internal CI. > > > > Thank you > > > > On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Please let me know how to make change to this reset machine. > (ip/access...) and disable it. > > > > After that please help to change the baseline. > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Thursday, March 26, 2020 11:39 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, we have a system in place that resets any changes made while > testing is enabled for a machine. > > If you would like, I can disable testing and allow you to make permanent > changes. > > > > I can also reset the baseline of Intel 10G test performance once you make > these changes. > > Please let me know if you would like to make permanent changes on the > Intel 10G so I can disable it for you. > > > > Thanks > > > > On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Thanks. Brandon. > > > > That’s good. We have made changed on 10G testbed. > > > > I monitored the several execution results; I found the results of 10G > always has -0.9%~-1.x% gap against expected number. So it could lead to see > sometime failures..+-1% I suggest adjusting the expected number. I don’t > know where the expected number is from? as I know it a dynamic number? > depends on baseline.. Please help to clarify, thanks. > > > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 24, 2020 9:31 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > I have enabled the 10G Intel machine for testing. > > If you would like to make any more changes, please let me know so I can > perform the necessary steps to prepare the machine for changes. > > Please feel free to let me know if you need anything. > > > > Thank you > > > > On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > For 10G, please enable it. our code is at original path > */opt/test-harness/dts.* > > > > For 40G, please keep running. and see if any issue. But, anyway, we have > modified the DTS code at /opt/test-harness/dts-new-suite. If we met same > problem, then use this new DTS instead. > > > > Thanks a lot > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Saturday, March 21, 2020 1:49 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, the 40G machine is stable enough to be put on production > dashboard to run tests which may cause Trex to be killed. > > Should I disable the 40G Intel machine for you to make changes? > > > > Also, just for confirmation: on the 10G machine, is the folder that you > are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r > are you still using the one in the standard */opt/test-harness/dts* > folder? > > > > If everything is ok, I will enable the 10G machine for production testing. > > > > Thank you very much > > > > On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Brandon, > > > > We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could > you please help to recover them? > > > > But, for FVL(40G) testbed, we met some problems, could you please help to > check before recover it > > - Sometime 1G hugepage will be changed to 2Mhugepage > automatically...we have to restart the system > - When we debugging on the testbed, found that Trex was killed by some > one(app).. > > Please help to check if any other program running on the testbed. > > > > Thanks a lot. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Sent:* Wednesday, March 18, 2020 9:04 PM > *To:* Brandon Lo <blo@iol.unh.edu> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Brandon, we almost made a workaround. > > > > Maybe tomorrow, you could recover Intel’s testbed. I will let you know > soon. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Wednesday, March 18, 2020 3:34 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Have you finished making changes on the Intel machine? > > I will turn on the machine on March 3rd for testing if you do not have any > issues with it. > > Please let me know if you need anything else. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex > upgrading. > > So we are reviewing our DTS script, different Trex version, and CI calling > procedure. > > > > Anyway, we are focusing on this task recently, any update will let you > know. > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 10, 2020 10:46 PM > *To:* David Marchand <david.marchand@redhat.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > How is the current status of the Intel 82599ES? > > Were there any configuration changes made to fix performance issues? > > > > Thanks > > > > On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wrote: > > Hi David, > > > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > > I'll rerun the tests that were affected and keep an eye out to see if it's > stable enough to be put back online. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com> > wrote: > > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header). > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu [-- Attachment #2: Type: text/html, Size: 58914 bytes --]
[-- Attachment #1: Type: text/plain, Size: 14644 bytes --] Got it. Thanks Brandon. Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu] Sent: Friday, April 3, 2020 2:39 AM To: Ma, LihongX <lihongx.ma@intel.com> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I have changed the baselines to reflect the new expected values. The performance tests should work as expected and pass. We will email again in the future if we come across any problems. Feel free to email us as well if you would like to make any other changes. Thank you for all your help On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon Thanks for you recommends, I have done the changes. As the throughput value of nic_single_core is proportional to the cpu frequency. I recommend you can change the baseline according to our report system. On the our 2.50GHz system, the baseline value as below: NNT: pkt_size trd/rxd expected_value 64 512 52.562 64 2048 41.439 FVL: pkt_size trd/rxd expected_value 64 512 59.608 64 2048 47.73 For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number should be NNT: pkt_size trd/rxd expected_value 64 512 52.562 / 2.5 * 2.1=44.152 64 2048 41.439 / 2.5 * 2.1=34.809 FVL: pkt_size trd/rxd expected_value 64 512 59.608 / 2.5 * 2.1=50.071 64 2048 47.73 / 2.5 * 2.1=40.093 Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Tuesday, March 31, 2020 9:42 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, To make changes to either Intel machine, please reboot using the command "reboot_to_rw" as root to reboot the machine into read/write mode. This command will also disable any testing on the machine. To re-enable the machine, please run "reboot_to_ro" as root, and it will save all of the changes that you've made and re-enable testing on the machine. I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" instead of the normal "reboot" while you're making changes. After you're done, please let me know. I'll have to manually run a test and update the baseline using our internal CI. Thank you On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Please let me know how to make change to this reset machine. (ip/access...) and disable it. After that please help to change the baseline. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Thursday, March 26, 2020 11:39 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, we have a system in place that resets any changes made while testing is enabled for a machine. If you would like, I can disable testing and allow you to make permanent changes. I can also reset the baseline of Intel 10G test performance once you make these changes. Please let me know if you would like to make permanent changes on the Intel 10G so I can disable it for you. Thanks On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Thanks. Brandon. That’s good. We have made changed on 10G testbed. I monitored the several execution results; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjusting the expected number. I don’t know where the expected number is from? as I know it a dynamic number? depends on baseline.. Please help to clarify, thanks. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 24, 2020 9:31 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, I have enabled the 10G Intel machine for testing. If you would like to make any more changes, please let me know so I can perform the necessary steps to prepare the machine for changes. Please feel free to let me know if you need anything. Thank you On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, For 10G, please enable it. our code is at original path /opt/test-harness/dts. For 40G, please keep running. and see if any issue. But, anyway, we have modified the DTS code at /opt/test-harness/dts-new-suite. If we met same problem, then use this new DTS instead. Thanks a lot Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Saturday, March 21, 2020 1:49 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. Should I disable the 40G Intel machine for you to make changes? Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in /opt/test-harness/dts-2020-3-4, or are you still using the one in the standard /opt/test-harness/dts folder? If everything is ok, I will enable the 10G machine for production testing. Thank you very much On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it * Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system * When we debugging on the testbed, found that Trex was killed by some one(app).. Please help to check if any other program running on the testbed. Thanks a lot. Regards, Zhaoyan Chen From: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #2: Type: text/html, Size: 77594 bytes --]
[-- Attachment #1.1: Type: text/plain, Size: 15136 bytes --] Hi, Brandon I find the grub configuration of isolation cpu miss the logic cores which at some thread. For example: If the server cpus layout as below, and want to config the isolation cpu from 1-20 The config of isolation should is ‘isolcpus=1-20,49-68’ [cid:image003.jpg@01D6134C.D6DDF140] So, I want to change the grub configuration of the isolation cpus. From ‘isolcpus=1-48’ change to ‘isolcpus=1-21,49-68,24-37,72-85’ Can you help me check which time is suite to do this change ? Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu] Sent: Friday, April 3, 2020 2:39 AM To: Ma, LihongX <lihongx.ma@intel.com> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I have changed the baselines to reflect the new expected values. The performance tests should work as expected and pass. We will email again in the future if we come across any problems. Feel free to email us as well if you would like to make any other changes. Thank you for all your help On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon Thanks for you recommends, I have done the changes. As the throughput value of nic_single_core is proportional to the cpu frequency. I recommend you can change the baseline according to our report system. On the our 2.50GHz system, the baseline value as below: NNT: pkt_size trd/rxd expected_value 64 512 52.562 64 2048 41.439 FVL: pkt_size trd/rxd expected_value 64 512 59.608 64 2048 47.73 For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number should be NNT: pkt_size trd/rxd expected_value 64 512 52.562 / 2.5 * 2.1=44.152 64 2048 41.439 / 2.5 * 2.1=34.809 FVL: pkt_size trd/rxd expected_value 64 512 59.608 / 2.5 * 2.1=50.071 64 2048 47.73 / 2.5 * 2.1=40.093 Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Tuesday, March 31, 2020 9:42 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, To make changes to either Intel machine, please reboot using the command "reboot_to_rw" as root to reboot the machine into read/write mode. This command will also disable any testing on the machine. To re-enable the machine, please run "reboot_to_ro" as root, and it will save all of the changes that you've made and re-enable testing on the machine. I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" instead of the normal "reboot" while you're making changes. After you're done, please let me know. I'll have to manually run a test and update the baseline using our internal CI. Thank you On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Please let me know how to make change to this reset machine. (ip/access...) and disable it. After that please help to change the baseline. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Thursday, March 26, 2020 11:39 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, we have a system in place that resets any changes made while testing is enabled for a machine. If you would like, I can disable testing and allow you to make permanent changes. I can also reset the baseline of Intel 10G test performance once you make these changes. Please let me know if you would like to make permanent changes on the Intel 10G so I can disable it for you. Thanks On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Thanks. Brandon. That’s good. We have made changed on 10G testbed. I monitored the several execution results; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjusting the expected number. I don’t know where the expected number is from? as I know it a dynamic number? depends on baseline.. Please help to clarify, thanks. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 24, 2020 9:31 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, I have enabled the 10G Intel machine for testing. If you would like to make any more changes, please let me know so I can perform the necessary steps to prepare the machine for changes. Please feel free to let me know if you need anything. Thank you On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, For 10G, please enable it. our code is at original path /opt/test-harness/dts. For 40G, please keep running. and see if any issue. But, anyway, we have modified the DTS code at /opt/test-harness/dts-new-suite. If we met same problem, then use this new DTS instead. Thanks a lot Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Saturday, March 21, 2020 1:49 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. Should I disable the 40G Intel machine for you to make changes? Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in /opt/test-harness/dts-2020-3-4, or are you still using the one in the standard /opt/test-harness/dts folder? If everything is ok, I will enable the 10G machine for production testing. Thank you very much On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it * Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system * When we debugging on the testbed, found that Trex was killed by some one(app).. Please help to check if any other program running on the testbed. Thanks a lot. Regards, Zhaoyan Chen From: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #1.2: Type: text/html, Size: 83861 bytes --] [-- Attachment #2: image003.jpg --] [-- Type: image/jpeg, Size: 58140 bytes --]
[-- Attachment #1.1: Type: text/plain, Size: 14752 bytes --] Hi Lihong, I think any time is fine to make these changes. Please use the 'reboot_to_rw' command once you have logged on, this will make sure your changes are saved. It will also disable testing momentarily while you make your changes. Once you're finished, you can reboot it with 'reboot_to_ro'. Please let me know when you're done so I can make sure everything is working as intended. We could also do this change for you if needed. Thanks, Brandon On Wed, Apr 15, 2020 at 5:39 AM Ma, LihongX <lihongx.ma@intel.com> wrote: > Hi, Brandon > > > > I find the grub configuration of isolation cpu miss the logic cores which > at some thread. > > > > For example: > > If the server cpus layout as below, and want to config the isolation cpu > from 1-20 > > The config of isolation should is ‘isolcpus=1-20,49-68’ > > > > > > So, I want to change the grub configuration of the isolation cpus. > > From ‘isolcpus=1-48’ change to ‘isolcpus=1-21,49-68,24-37,72-85’ > > > > Can you help me check which time is suite to do this change ? > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Friday, April 3, 2020 2:39 AM > *To:* Ma, LihongX <lihongx.ma@intel.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; > Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; > Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > I have changed the baselines to reflect the new expected values. > > The performance tests should work as expected and pass. > > > > We will email again in the future if we come across any problems. > > Feel free to email us as well if you would like to make any other changes. > > > > Thank you for all your help > > > > On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com> wrote: > > Hi, Brandon > > Thanks for you recommends, I have done the changes. > > As the throughput value of nic_single_core is proportional to the cpu > frequency. > > I recommend you can change the baseline according to our report system. > > > > On the our 2.50GHz system, the baseline value as below: > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 > > 64 > > 2048 > > 41.439 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 > > 64 > > 2048 > > 47.73 > > > > For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number > should be > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 / 2.5 * 2.1=44.152 > > 64 > > 2048 > > 41.439 / 2.5 * 2.1=34.809 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 / 2.5 * 2.1=50.071 > > 64 > > 2048 > > 47.73 / 2.5 * 2.1=40.093 > > > > > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Tuesday, March 31, 2020 9:42 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > To make changes to either Intel machine, please reboot using the command > "reboot_to_rw" as root to reboot the machine into read/write mode. > > This command will also disable any testing on the machine. > > > > To re-enable the machine, please run "reboot_to_ro" as root, and it will > save all of the changes that you've made and re-enable testing on the > machine. > > I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" > instead of the normal "reboot" while you're making changes. > > > > After you're done, please let me know. I'll have to manually run a test > and update the baseline using our internal CI. > > > > Thank you > > > > On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Please let me know how to make change to this reset machine. > (ip/access...) and disable it. > > > > After that please help to change the baseline. > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Thursday, March 26, 2020 11:39 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, we have a system in place that resets any changes made while > testing is enabled for a machine. > > If you would like, I can disable testing and allow you to make permanent > changes. > > > > I can also reset the baseline of Intel 10G test performance once you make > these changes. > > Please let me know if you would like to make permanent changes on the > Intel 10G so I can disable it for you. > > > > Thanks > > > > On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Thanks. Brandon. > > > > That’s good. We have made changed on 10G testbed. > > > > I monitored the several execution results; I found the results of 10G > always has -0.9%~-1.x% gap against expected number. So it could lead to see > sometime failures..+-1% I suggest adjusting the expected number. I don’t > know where the expected number is from? as I know it a dynamic number? > depends on baseline.. Please help to clarify, thanks. > > > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 24, 2020 9:31 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > I have enabled the 10G Intel machine for testing. > > If you would like to make any more changes, please let me know so I can > perform the necessary steps to prepare the machine for changes. > > Please feel free to let me know if you need anything. > > > > Thank you > > > > On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > For 10G, please enable it. our code is at original path > */opt/test-harness/dts.* > > > > For 40G, please keep running. and see if any issue. But, anyway, we have > modified the DTS code at /opt/test-harness/dts-new-suite. If we met same > problem, then use this new DTS instead. > > > > Thanks a lot > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Saturday, March 21, 2020 1:49 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, the 40G machine is stable enough to be put on production > dashboard to run tests which may cause Trex to be killed. > > Should I disable the 40G Intel machine for you to make changes? > > > > Also, just for confirmation: on the 10G machine, is the folder that you > are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r > are you still using the one in the standard */opt/test-harness/dts* > folder? > > > > If everything is ok, I will enable the 10G machine for production testing. > > > > Thank you very much > > > > On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Brandon, > > > > We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could > you please help to recover them? > > > > But, for FVL(40G) testbed, we met some problems, could you please help to > check before recover it > > - Sometime 1G hugepage will be changed to 2Mhugepage > automatically...we have to restart the system > - When we debugging on the testbed, found that Trex was killed by some > one(app).. > > Please help to check if any other program running on the testbed. > > > > Thanks a lot. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Sent:* Wednesday, March 18, 2020 9:04 PM > *To:* Brandon Lo <blo@iol.unh.edu> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Brandon, we almost made a workaround. > > > > Maybe tomorrow, you could recover Intel’s testbed. I will let you know > soon. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Wednesday, March 18, 2020 3:34 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Have you finished making changes on the Intel machine? > > I will turn on the machine on March 3rd for testing if you do not have any > issues with it. > > Please let me know if you need anything else. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex > upgrading. > > So we are reviewing our DTS script, different Trex version, and CI calling > procedure. > > > > Anyway, we are focusing on this task recently, any update will let you > know. > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 10, 2020 10:46 PM > *To:* David Marchand <david.marchand@redhat.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > How is the current status of the Intel 82599ES? > > Were there any configuration changes made to fix performance issues? > > > > Thanks > > > > On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wrote: > > Hi David, > > > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > > I'll rerun the tests that were affected and keep an eye out to see if it's > stable enough to be put back online. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com> > wrote: > > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header). > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu [-- Attachment #1.2: Type: text/html, Size: 67441 bytes --] [-- Attachment #2: image003.jpg --] [-- Type: image/jpeg, Size: 58140 bytes --]
[-- Attachment #1.1: Type: text/plain, Size: 17724 bytes --] Hi, Brandon I use the cmd ‘reboot_to_rw’ to restart server, and edit the file ‘/etc/default/grub’ to reset the isolation cpus, Changes the field ‘isolcpus=1-48 nohz_full=1-48 rcu_nocbs=1-48’ to ‘isolcpus=1-21,49-69,24-37,72-85 nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85’ Detail as below: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash hugepagesz=1G hugepages=40 default_hugepagesz=1G isolcpus=1-21,49-69,24-37,72-85 intel_iommu=on iommu=pt nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85 intel_pstate=disable nmi_watchdog=0 audit=0 nosoftlockup processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable mce=off tsc=reliable numa_balancing=disable" Then use cmd: update-grub to make the configuration effective. Then reboot server use ‘reboot_to_ro’, but I can’t find the change in cmdline ( cat /proc/cmdline). Can you help me to check it? Thanks Lihong From: Brandon Lo [mailto:blo@iol.unh.edu] Sent: Thursday, April 16, 2020 3:31 AM To: Ma, LihongX <lihongx.ma@intel.com> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I think any time is fine to make these changes. Please use the 'reboot_to_rw' command once you have logged on, this will make sure your changes are saved. It will also disable testing momentarily while you make your changes. Once you're finished, you can reboot it with 'reboot_to_ro'. Please let me know when you're done so I can make sure everything is working as intended. We could also do this change for you if needed. Thanks, Brandon On Wed, Apr 15, 2020 at 5:39 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon I find the grub configuration of isolation cpu miss the logic cores which at some thread. For example: If the server cpus layout as below, and want to config the isolation cpu from 1-20 The config of isolation should is ‘isolcpus=1-20,49-68’ [cid:1717f4fa8006917eb1] So, I want to change the grub configuration of the isolation cpus. From ‘isolcpus=1-48’ change to ‘isolcpus=1-21,49-68,24-37,72-85’ Can you help me check which time is suite to do this change ? Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Friday, April 3, 2020 2:39 AM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I have changed the baselines to reflect the new expected values. The performance tests should work as expected and pass. We will email again in the future if we come across any problems. Feel free to email us as well if you would like to make any other changes. Thank you for all your help On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon Thanks for you recommends, I have done the changes. As the throughput value of nic_single_core is proportional to the cpu frequency. I recommend you can change the baseline according to our report system. On the our 2.50GHz system, the baseline value as below: NNT: pkt_size trd/rxd expected_value 64 512 52.562 64 2048 41.439 FVL: pkt_size trd/rxd expected_value 64 512 59.608 64 2048 47.73 For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number should be NNT: pkt_size trd/rxd expected_value 64 512 52.562 / 2.5 * 2.1=44.152 64 2048 41.439 / 2.5 * 2.1=34.809 FVL: pkt_size trd/rxd expected_value 64 512 59.608 / 2.5 * 2.1=50.071 64 2048 47.73 / 2.5 * 2.1=40.093 Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Tuesday, March 31, 2020 9:42 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, To make changes to either Intel machine, please reboot using the command "reboot_to_rw" as root to reboot the machine into read/write mode. This command will also disable any testing on the machine. To re-enable the machine, please run "reboot_to_ro" as root, and it will save all of the changes that you've made and re-enable testing on the machine. I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" instead of the normal "reboot" while you're making changes. After you're done, please let me know. I'll have to manually run a test and update the baseline using our internal CI. Thank you On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Please let me know how to make change to this reset machine. (ip/access...) and disable it. After that please help to change the baseline. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Thursday, March 26, 2020 11:39 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, we have a system in place that resets any changes made while testing is enabled for a machine. If you would like, I can disable testing and allow you to make permanent changes. I can also reset the baseline of Intel 10G test performance once you make these changes. Please let me know if you would like to make permanent changes on the Intel 10G so I can disable it for you. Thanks On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Thanks. Brandon. That’s good. We have made changed on 10G testbed. I monitored the several execution results; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjusting the expected number. I don’t know where the expected number is from? as I know it a dynamic number? depends on baseline.. Please help to clarify, thanks. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 24, 2020 9:31 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, I have enabled the 10G Intel machine for testing. If you would like to make any more changes, please let me know so I can perform the necessary steps to prepare the machine for changes. Please feel free to let me know if you need anything. Thank you On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, For 10G, please enable it. our code is at original path /opt/test-harness/dts. For 40G, please keep running. and see if any issue. But, anyway, we have modified the DTS code at /opt/test-harness/dts-new-suite. If we met same problem, then use this new DTS instead. Thanks a lot Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Saturday, March 21, 2020 1:49 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. Should I disable the 40G Intel machine for you to make changes? Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in /opt/test-harness/dts-2020-3-4, or are you still using the one in the standard /opt/test-harness/dts folder? If everything is ok, I will enable the 10G machine for production testing. Thank you very much On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it * Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system * When we debugging on the testbed, found that Trex was killed by some one(app).. Please help to check if any other program running on the testbed. Thanks a lot. Regards, Zhaoyan Chen From: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #1.2: Type: text/html, Size: 95500 bytes --] [-- Attachment #2: image001.jpg --] [-- Type: image/jpeg, Size: 41815 bytes --]
[-- Attachment #1.1: Type: text/plain, Size: 17587 bytes --] Hi Lihong, I saw your changes and applied them manually. This involves me taking the machine down for maintenance on our backend, and then doing a normal `reboot`. Now when I do `cat /proc/cmdline` it shows: `BOOT_IMAGE=/boot/vmlinuz-4.15.0-55-generic root=UUID=3801030b-237d-428e-9e67-d81e12f16308 ro quiet splash hugepagesz=1G hugepages=40 default_hugepagesz=1G isolcpus=1-21,49-69,24-37,72-85 intel_iommu=on iommu=pt nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85 intel_pstate=disable nmi_watchdog=0 audit=0 nosoftlockup processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable mce=off tsc=reliable numa_balancing=disable vt.handoff=1` Which is what I think we wanted to achieve. I have now done the `reboot_to_ro` command and it should be working fine for testing. Thanks, Brandon On Wed, Apr 15, 2020 at 11:40 PM Ma, LihongX <lihongx.ma@intel.com> wrote: > Hi, Brandon > > > > I use the cmd ‘reboot_to_rw’ to restart server, and edit the file > ‘/etc/default/grub’ to reset the isolation cpus, > > > > Changes the field ‘isolcpus=1-48 nohz_full=1-48 rcu_nocbs=1-48’ to > > ‘isolcpus=1-21,49-69,24-37,72-85 nohz_full=1-21,49-69,24-37,72-85 > rcu_nocbs=1-21,49-69,24-37,72-85’ > > > > Detail as below: > > GRUB_CMDLINE_LINUX_DEFAULT="quiet splash hugepagesz=1G hugepages=40 > default_hugepagesz=1G isolcpus=1-21,49-69,24-37,72-85 intel_iommu=on > iommu=pt nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85 > intel_pstate=disable nmi_watchdog=0 audit=0 nosoftlockup > processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable mce=off > tsc=reliable numa_balancing=disable" > > > > Then use cmd: update-grub to make the configuration effective. > > Then reboot server use ‘reboot_to_ro’, but I can’t find the change in > cmdline ( cat /proc/cmdline). > > > > Can you help me to check it? > > > > Thanks > > Lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Thursday, April 16, 2020 3:31 AM > *To:* Ma, LihongX <lihongx.ma@intel.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; > Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; > Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > I think any time is fine to make these changes. > > > > Please use the 'reboot_to_rw' command once you have logged on, this will > make sure your changes are saved. > > It will also disable testing momentarily while you make your changes. > > > > Once you're finished, you can reboot it with 'reboot_to_ro'. > > Please let me know when you're done so I can make sure everything is > working as intended. > > We could also do this change for you if needed. > > > > Thanks, > > Brandon > > > > On Wed, Apr 15, 2020 at 5:39 AM Ma, LihongX <lihongx.ma@intel.com> wrote: > > Hi, Brandon > > > > I find the grub configuration of isolation cpu miss the logic cores which > at some thread. > > > > For example: > > If the server cpus layout as below, and want to config the isolation cpu > from 1-20 > > The config of isolation should is ‘isolcpus=1-20,49-68’ > > > > [image: cid:1717f4fa8006917eb1] > > > > So, I want to change the grub configuration of the isolation cpus. > > From ‘isolcpus=1-48’ change to ‘isolcpus=1-21,49-68,24-37,72-85’ > > > > Can you help me check which time is suite to do this change ? > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Friday, April 3, 2020 2:39 AM > *To:* Ma, LihongX <lihongx.ma@intel.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; > Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; > Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > I have changed the baselines to reflect the new expected values. > > The performance tests should work as expected and pass. > > > > We will email again in the future if we come across any problems. > > Feel free to email us as well if you would like to make any other changes. > > > > Thank you for all your help > > > > On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com> wrote: > > Hi, Brandon > > Thanks for you recommends, I have done the changes. > > As the throughput value of nic_single_core is proportional to the cpu > frequency. > > I recommend you can change the baseline according to our report system. > > > > On the our 2.50GHz system, the baseline value as below: > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 > > 64 > > 2048 > > 41.439 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 > > 64 > > 2048 > > 47.73 > > > > For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number > should be > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 / 2.5 * 2.1=44.152 > > 64 > > 2048 > > 41.439 / 2.5 * 2.1=34.809 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 / 2.5 * 2.1=50.071 > > 64 > > 2048 > > 47.73 / 2.5 * 2.1=40.093 > > > > > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Tuesday, March 31, 2020 9:42 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > To make changes to either Intel machine, please reboot using the command > "reboot_to_rw" as root to reboot the machine into read/write mode. > > This command will also disable any testing on the machine. > > > > To re-enable the machine, please run "reboot_to_ro" as root, and it will > save all of the changes that you've made and re-enable testing on the > machine. > > I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" > instead of the normal "reboot" while you're making changes. > > > > After you're done, please let me know. I'll have to manually run a test > and update the baseline using our internal CI. > > > > Thank you > > > > On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Please let me know how to make change to this reset machine. > (ip/access...) and disable it. > > > > After that please help to change the baseline. > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Thursday, March 26, 2020 11:39 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, we have a system in place that resets any changes made while > testing is enabled for a machine. > > If you would like, I can disable testing and allow you to make permanent > changes. > > > > I can also reset the baseline of Intel 10G test performance once you make > these changes. > > Please let me know if you would like to make permanent changes on the > Intel 10G so I can disable it for you. > > > > Thanks > > > > On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Thanks. Brandon. > > > > That’s good. We have made changed on 10G testbed. > > > > I monitored the several execution results; I found the results of 10G > always has -0.9%~-1.x% gap against expected number. So it could lead to see > sometime failures..+-1% I suggest adjusting the expected number. I don’t > know where the expected number is from? as I know it a dynamic number? > depends on baseline.. Please help to clarify, thanks. > > > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 24, 2020 9:31 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > I have enabled the 10G Intel machine for testing. > > If you would like to make any more changes, please let me know so I can > perform the necessary steps to prepare the machine for changes. > > Please feel free to let me know if you need anything. > > > > Thank you > > > > On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > For 10G, please enable it. our code is at original path > */opt/test-harness/dts.* > > > > For 40G, please keep running. and see if any issue. But, anyway, we have > modified the DTS code at /opt/test-harness/dts-new-suite. If we met same > problem, then use this new DTS instead. > > > > Thanks a lot > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Saturday, March 21, 2020 1:49 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, the 40G machine is stable enough to be put on production > dashboard to run tests which may cause Trex to be killed. > > Should I disable the 40G Intel machine for you to make changes? > > > > Also, just for confirmation: on the 10G machine, is the folder that you > are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r > are you still using the one in the standard */opt/test-harness/dts* > folder? > > > > If everything is ok, I will enable the 10G machine for production testing. > > > > Thank you very much > > > > On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Brandon, > > > > We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could > you please help to recover them? > > > > But, for FVL(40G) testbed, we met some problems, could you please help to > check before recover it > > - Sometime 1G hugepage will be changed to 2Mhugepage > automatically...we have to restart the system > - When we debugging on the testbed, found that Trex was killed by some > one(app).. > > Please help to check if any other program running on the testbed. > > > > Thanks a lot. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Sent:* Wednesday, March 18, 2020 9:04 PM > *To:* Brandon Lo <blo@iol.unh.edu> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Brandon, we almost made a workaround. > > > > Maybe tomorrow, you could recover Intel’s testbed. I will let you know > soon. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Wednesday, March 18, 2020 3:34 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Have you finished making changes on the Intel machine? > > I will turn on the machine on March 3rd for testing if you do not have any > issues with it. > > Please let me know if you need anything else. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex > upgrading. > > So we are reviewing our DTS script, different Trex version, and CI calling > procedure. > > > > Anyway, we are focusing on this task recently, any update will let you > know. > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 10, 2020 10:46 PM > *To:* David Marchand <david.marchand@redhat.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > How is the current status of the Intel 82599ES? > > Were there any configuration changes made to fix performance issues? > > > > Thanks > > > > On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wrote: > > Hi David, > > > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > > I'll rerun the tests that were affected and keep an eye out to see if it's > stable enough to be put back online. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com> > wrote: > > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header). > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu [-- Attachment #1.2: Type: text/html, Size: 77004 bytes --] [-- Attachment #2: image001.jpg --] [-- Type: image/jpeg, Size: 41815 bytes --]
[-- Attachment #1.1: Type: text/plain, Size: 19927 bytes --] Hi, Brandon Very thanks for your help, now the NNT server have changed as we wanted, and can you help me also change the cmdline on FVL server. Thanks Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu] Sent: Thursday, April 16, 2020 11:04 PM To: Ma, LihongX <lihongx.ma@intel.com> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I saw your changes and applied them manually. This involves me taking the machine down for maintenance on our backend, and then doing a normal `reboot`. Now when I do `cat /proc/cmdline` it shows: `BOOT_IMAGE=/boot/vmlinuz-4.15.0-55-generic root=UUID=3801030b-237d-428e-9e67-d81e12f16308 ro quiet splash hugepagesz=1G hugepages=40 default_hugepagesz=1G isolcpus=1-21,49-69,24-37,72-85 intel_iommu=on iommu=pt nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85 intel_pstate=disable nmi_watchdog=0 audit=0 nosoftlockup processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable mce=off tsc=reliable numa_balancing=disable vt.handoff=1` Which is what I think we wanted to achieve. I have now done the `reboot_to_ro` command and it should be working fine for testing. Thanks, Brandon On Wed, Apr 15, 2020 at 11:40 PM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon I use the cmd ‘reboot_to_rw’ to restart server, and edit the file ‘/etc/default/grub’ to reset the isolation cpus, Changes the field ‘isolcpus=1-48 nohz_full=1-48 rcu_nocbs=1-48’ to ‘isolcpus=1-21,49-69,24-37,72-85 nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85’ Detail as below: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash hugepagesz=1G hugepages=40 default_hugepagesz=1G isolcpus=1-21,49-69,24-37,72-85 intel_iommu=on iommu=pt nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85 intel_pstate=disable nmi_watchdog=0 audit=0 nosoftlockup processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable mce=off tsc=reliable numa_balancing=disable" Then use cmd: update-grub to make the configuration effective. Then reboot server use ‘reboot_to_ro’, but I can’t find the change in cmdline ( cat /proc/cmdline). Can you help me to check it? Thanks Lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Thursday, April 16, 2020 3:31 AM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I think any time is fine to make these changes. Please use the 'reboot_to_rw' command once you have logged on, this will make sure your changes are saved. It will also disable testing momentarily while you make your changes. Once you're finished, you can reboot it with 'reboot_to_ro'. Please let me know when you're done so I can make sure everything is working as intended. We could also do this change for you if needed. Thanks, Brandon On Wed, Apr 15, 2020 at 5:39 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon I find the grub configuration of isolation cpu miss the logic cores which at some thread. For example: If the server cpus layout as below, and want to config the isolation cpu from 1-20 The config of isolation should is ‘isolcpus=1-20,49-68’ [cid:1717f4fa8006917eb1] So, I want to change the grub configuration of the isolation cpus. From ‘isolcpus=1-48’ change to ‘isolcpus=1-21,49-68,24-37,72-85’ Can you help me check which time is suite to do this change ? Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Friday, April 3, 2020 2:39 AM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I have changed the baselines to reflect the new expected values. The performance tests should work as expected and pass. We will email again in the future if we come across any problems. Feel free to email us as well if you would like to make any other changes. Thank you for all your help On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon Thanks for you recommends, I have done the changes. As the throughput value of nic_single_core is proportional to the cpu frequency. I recommend you can change the baseline according to our report system. On the our 2.50GHz system, the baseline value as below: NNT: pkt_size trd/rxd expected_value 64 512 52.562 64 2048 41.439 FVL: pkt_size trd/rxd expected_value 64 512 59.608 64 2048 47.73 For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number should be NNT: pkt_size trd/rxd expected_value 64 512 52.562 / 2.5 * 2.1=44.152 64 2048 41.439 / 2.5 * 2.1=34.809 FVL: pkt_size trd/rxd expected_value 64 512 59.608 / 2.5 * 2.1=50.071 64 2048 47.73 / 2.5 * 2.1=40.093 Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Tuesday, March 31, 2020 9:42 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, To make changes to either Intel machine, please reboot using the command "reboot_to_rw" as root to reboot the machine into read/write mode. This command will also disable any testing on the machine. To re-enable the machine, please run "reboot_to_ro" as root, and it will save all of the changes that you've made and re-enable testing on the machine. I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" instead of the normal "reboot" while you're making changes. After you're done, please let me know. I'll have to manually run a test and update the baseline using our internal CI. Thank you On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Please let me know how to make change to this reset machine. (ip/access...) and disable it. After that please help to change the baseline. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Thursday, March 26, 2020 11:39 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, we have a system in place that resets any changes made while testing is enabled for a machine. If you would like, I can disable testing and allow you to make permanent changes. I can also reset the baseline of Intel 10G test performance once you make these changes. Please let me know if you would like to make permanent changes on the Intel 10G so I can disable it for you. Thanks On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Thanks. Brandon. That’s good. We have made changed on 10G testbed. I monitored the several execution results; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjusting the expected number. I don’t know where the expected number is from? as I know it a dynamic number? depends on baseline.. Please help to clarify, thanks. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 24, 2020 9:31 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, I have enabled the 10G Intel machine for testing. If you would like to make any more changes, please let me know so I can perform the necessary steps to prepare the machine for changes. Please feel free to let me know if you need anything. Thank you On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, For 10G, please enable it. our code is at original path /opt/test-harness/dts. For 40G, please keep running. and see if any issue. But, anyway, we have modified the DTS code at /opt/test-harness/dts-new-suite. If we met same problem, then use this new DTS instead. Thanks a lot Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Saturday, March 21, 2020 1:49 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. Should I disable the 40G Intel machine for you to make changes? Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in /opt/test-harness/dts-2020-3-4, or are you still using the one in the standard /opt/test-harness/dts folder? If everything is ok, I will enable the 10G machine for production testing. Thank you very much On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it * Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system * When we debugging on the testbed, found that Trex was killed by some one(app).. Please help to check if any other program running on the testbed. Thanks a lot. Regards, Zhaoyan Chen From: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #1.2: Type: text/html, Size: 102094 bytes --] [-- Attachment #2: image001.jpg --] [-- Type: image/jpeg, Size: 41815 bytes --]
[-- Attachment #1.1: Type: text/plain, Size: 19864 bytes --] Hi, Brandon I find the NNT server did not be called by the Jenkins, can you help to check? Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu] Sent: Thursday, April 16, 2020 11:04 PM To: Ma, LihongX <lihongx.ma@intel.com> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I saw your changes and applied them manually. This involves me taking the machine down for maintenance on our backend, and then doing a normal `reboot`. Now when I do `cat /proc/cmdline` it shows: `BOOT_IMAGE=/boot/vmlinuz-4.15.0-55-generic root=UUID=3801030b-237d-428e-9e67-d81e12f16308 ro quiet splash hugepagesz=1G hugepages=40 default_hugepagesz=1G isolcpus=1-21,49-69,24-37,72-85 intel_iommu=on iommu=pt nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85 intel_pstate=disable nmi_watchdog=0 audit=0 nosoftlockup processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable mce=off tsc=reliable numa_balancing=disable vt.handoff=1` Which is what I think we wanted to achieve. I have now done the `reboot_to_ro` command and it should be working fine for testing. Thanks, Brandon On Wed, Apr 15, 2020 at 11:40 PM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon I use the cmd ‘reboot_to_rw’ to restart server, and edit the file ‘/etc/default/grub’ to reset the isolation cpus, Changes the field ‘isolcpus=1-48 nohz_full=1-48 rcu_nocbs=1-48’ to ‘isolcpus=1-21,49-69,24-37,72-85 nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85’ Detail as below: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash hugepagesz=1G hugepages=40 default_hugepagesz=1G isolcpus=1-21,49-69,24-37,72-85 intel_iommu=on iommu=pt nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85 intel_pstate=disable nmi_watchdog=0 audit=0 nosoftlockup processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable mce=off tsc=reliable numa_balancing=disable" Then use cmd: update-grub to make the configuration effective. Then reboot server use ‘reboot_to_ro’, but I can’t find the change in cmdline ( cat /proc/cmdline). Can you help me to check it? Thanks Lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Thursday, April 16, 2020 3:31 AM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I think any time is fine to make these changes. Please use the 'reboot_to_rw' command once you have logged on, this will make sure your changes are saved. It will also disable testing momentarily while you make your changes. Once you're finished, you can reboot it with 'reboot_to_ro'. Please let me know when you're done so I can make sure everything is working as intended. We could also do this change for you if needed. Thanks, Brandon On Wed, Apr 15, 2020 at 5:39 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon I find the grub configuration of isolation cpu miss the logic cores which at some thread. For example: If the server cpus layout as below, and want to config the isolation cpu from 1-20 The config of isolation should is ‘isolcpus=1-20,49-68’ [cid:1717f4fa8006917eb1] So, I want to change the grub configuration of the isolation cpus. From ‘isolcpus=1-48’ change to ‘isolcpus=1-21,49-68,24-37,72-85’ Can you help me check which time is suite to do this change ? Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Friday, April 3, 2020 2:39 AM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I have changed the baselines to reflect the new expected values. The performance tests should work as expected and pass. We will email again in the future if we come across any problems. Feel free to email us as well if you would like to make any other changes. Thank you for all your help On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon Thanks for you recommends, I have done the changes. As the throughput value of nic_single_core is proportional to the cpu frequency. I recommend you can change the baseline according to our report system. On the our 2.50GHz system, the baseline value as below: NNT: pkt_size trd/rxd expected_value 64 512 52.562 64 2048 41.439 FVL: pkt_size trd/rxd expected_value 64 512 59.608 64 2048 47.73 For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number should be NNT: pkt_size trd/rxd expected_value 64 512 52.562 / 2.5 * 2.1=44.152 64 2048 41.439 / 2.5 * 2.1=34.809 FVL: pkt_size trd/rxd expected_value 64 512 59.608 / 2.5 * 2.1=50.071 64 2048 47.73 / 2.5 * 2.1=40.093 Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Tuesday, March 31, 2020 9:42 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, To make changes to either Intel machine, please reboot using the command "reboot_to_rw" as root to reboot the machine into read/write mode. This command will also disable any testing on the machine. To re-enable the machine, please run "reboot_to_ro" as root, and it will save all of the changes that you've made and re-enable testing on the machine. I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" instead of the normal "reboot" while you're making changes. After you're done, please let me know. I'll have to manually run a test and update the baseline using our internal CI. Thank you On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Please let me know how to make change to this reset machine. (ip/access...) and disable it. After that please help to change the baseline. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Thursday, March 26, 2020 11:39 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, we have a system in place that resets any changes made while testing is enabled for a machine. If you would like, I can disable testing and allow you to make permanent changes. I can also reset the baseline of Intel 10G test performance once you make these changes. Please let me know if you would like to make permanent changes on the Intel 10G so I can disable it for you. Thanks On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Thanks. Brandon. That’s good. We have made changed on 10G testbed. I monitored the several execution results; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjusting the expected number. I don’t know where the expected number is from? as I know it a dynamic number? depends on baseline.. Please help to clarify, thanks. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 24, 2020 9:31 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, I have enabled the 10G Intel machine for testing. If you would like to make any more changes, please let me know so I can perform the necessary steps to prepare the machine for changes. Please feel free to let me know if you need anything. Thank you On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, For 10G, please enable it. our code is at original path /opt/test-harness/dts. For 40G, please keep running. and see if any issue. But, anyway, we have modified the DTS code at /opt/test-harness/dts-new-suite. If we met same problem, then use this new DTS instead. Thanks a lot Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Saturday, March 21, 2020 1:49 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. Should I disable the 40G Intel machine for you to make changes? Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in /opt/test-harness/dts-2020-3-4, or are you still using the one in the standard /opt/test-harness/dts folder? If everything is ok, I will enable the 10G machine for production testing. Thank you very much On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it * Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system * When we debugging on the testbed, found that Trex was killed by some one(app).. Please help to check if any other program running on the testbed. Thanks a lot. Regards, Zhaoyan Chen From: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #1.2: Type: text/html, Size: 101763 bytes --] [-- Attachment #2: image001.jpg --] [-- Type: image/jpeg, Size: 41815 bytes --]
[-- Attachment #1.1: Type: text/plain, Size: 19017 bytes --] Hi Lihong, I have fixed the NNT server not being called by Jenkins. It should work fine now. We did run into an issue where it could not detect huge pages information, but rebooting it again fixed the issue. I have also done the same thing to the FVL server. Both of them should be called by Jenkins for testing. Thanks, Brandon On Fri, Apr 17, 2020 at 3:00 AM Ma, LihongX <lihongx.ma@intel.com> wrote: > Hi, Brandon > > > > I find the NNT server did not be called by the Jenkins, can you help to > check? > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Thursday, April 16, 2020 11:04 PM > *To:* Ma, LihongX <lihongx.ma@intel.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; > Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; > Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > I saw your changes and applied them manually. > > This involves me taking the machine down for maintenance on our backend, > and then doing a normal `reboot`. > > > > Now when I do `cat /proc/cmdline` it shows: > > `BOOT_IMAGE=/boot/vmlinuz-4.15.0-55-generic > root=UUID=3801030b-237d-428e-9e67-d81e12f16308 ro quiet splash > hugepagesz=1G hugepages=40 default_hugepagesz=1G > isolcpus=1-21,49-69,24-37,72-85 intel_iommu=on iommu=pt > nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85 > intel_pstate=disable nmi_watchdog=0 audit=0 nosoftlockup > processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable mce=off > tsc=reliable numa_balancing=disable vt.handoff=1` > > > > Which is what I think we wanted to achieve. > > I have now done the `reboot_to_ro` command and it should be working fine > for testing. > > > > Thanks, > > Brandon > > > > > > On Wed, Apr 15, 2020 at 11:40 PM Ma, LihongX <lihongx.ma@intel.com> wrote: > > Hi, Brandon > > > > I use the cmd ‘reboot_to_rw’ to restart server, and edit the file > ‘/etc/default/grub’ to reset the isolation cpus, > > > > Changes the field ‘isolcpus=1-48 nohz_full=1-48 rcu_nocbs=1-48’ to > > ‘isolcpus=1-21,49-69,24-37,72-85 nohz_full=1-21,49-69,24-37,72-85 > rcu_nocbs=1-21,49-69,24-37,72-85’ > > > > Detail as below: > > GRUB_CMDLINE_LINUX_DEFAULT="quiet splash hugepagesz=1G hugepages=40 > default_hugepagesz=1G isolcpus=1-21,49-69,24-37,72-85 intel_iommu=on > iommu=pt nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85 > intel_pstate=disable nmi_watchdog=0 audit=0 nosoftlockup > processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable mce=off > tsc=reliable numa_balancing=disable" > > > > Then use cmd: update-grub to make the configuration effective. > > Then reboot server use ‘reboot_to_ro’, but I can’t find the change in > cmdline ( cat /proc/cmdline). > > > > Can you help me to check it? > > > > Thanks > > Lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Thursday, April 16, 2020 3:31 AM > *To:* Ma, LihongX <lihongx.ma@intel.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; > Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; > Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > I think any time is fine to make these changes. > > > > Please use the 'reboot_to_rw' command once you have logged on, this will > make sure your changes are saved. > > It will also disable testing momentarily while you make your changes. > > > > Once you're finished, you can reboot it with 'reboot_to_ro'. > > Please let me know when you're done so I can make sure everything is > working as intended. > > We could also do this change for you if needed. > > > > Thanks, > > Brandon > > > > On Wed, Apr 15, 2020 at 5:39 AM Ma, LihongX <lihongx.ma@intel.com> wrote: > > Hi, Brandon > > > > I find the grub configuration of isolation cpu miss the logic cores which > at some thread. > > > > For example: > > If the server cpus layout as below, and want to config the isolation cpu > from 1-20 > > The config of isolation should is ‘isolcpus=1-20,49-68’ > > > > [image: cid:1717f4fa8006917eb1] > > > > So, I want to change the grub configuration of the isolation cpus. > > From ‘isolcpus=1-48’ change to ‘isolcpus=1-21,49-68,24-37,72-85’ > > > > Can you help me check which time is suite to do this change ? > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Friday, April 3, 2020 2:39 AM > *To:* Ma, LihongX <lihongx.ma@intel.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; > Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; > Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > I have changed the baselines to reflect the new expected values. > > The performance tests should work as expected and pass. > > > > We will email again in the future if we come across any problems. > > Feel free to email us as well if you would like to make any other changes. > > > > Thank you for all your help > > > > On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com> wrote: > > Hi, Brandon > > Thanks for you recommends, I have done the changes. > > As the throughput value of nic_single_core is proportional to the cpu > frequency. > > I recommend you can change the baseline according to our report system. > > > > On the our 2.50GHz system, the baseline value as below: > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 > > 64 > > 2048 > > 41.439 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 > > 64 > > 2048 > > 47.73 > > > > For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number > should be > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 / 2.5 * 2.1=44.152 > > 64 > > 2048 > > 41.439 / 2.5 * 2.1=34.809 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 / 2.5 * 2.1=50.071 > > 64 > > 2048 > > 47.73 / 2.5 * 2.1=40.093 > > > > > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Tuesday, March 31, 2020 9:42 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > To make changes to either Intel machine, please reboot using the command > "reboot_to_rw" as root to reboot the machine into read/write mode. > > This command will also disable any testing on the machine. > > > > To re-enable the machine, please run "reboot_to_ro" as root, and it will > save all of the changes that you've made and re-enable testing on the > machine. > > I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" > instead of the normal "reboot" while you're making changes. > > > > After you're done, please let me know. I'll have to manually run a test > and update the baseline using our internal CI. > > > > Thank you > > > > On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Please let me know how to make change to this reset machine. > (ip/access...) and disable it. > > > > After that please help to change the baseline. > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Thursday, March 26, 2020 11:39 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, we have a system in place that resets any changes made while > testing is enabled for a machine. > > If you would like, I can disable testing and allow you to make permanent > changes. > > > > I can also reset the baseline of Intel 10G test performance once you make > these changes. > > Please let me know if you would like to make permanent changes on the > Intel 10G so I can disable it for you. > > > > Thanks > > > > On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Thanks. Brandon. > > > > That’s good. We have made changed on 10G testbed. > > > > I monitored the several execution results; I found the results of 10G > always has -0.9%~-1.x% gap against expected number. So it could lead to see > sometime failures..+-1% I suggest adjusting the expected number. I don’t > know where the expected number is from? as I know it a dynamic number? > depends on baseline.. Please help to clarify, thanks. > > > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 24, 2020 9:31 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > I have enabled the 10G Intel machine for testing. > > If you would like to make any more changes, please let me know so I can > perform the necessary steps to prepare the machine for changes. > > Please feel free to let me know if you need anything. > > > > Thank you > > > > On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > For 10G, please enable it. our code is at original path > */opt/test-harness/dts.* > > > > For 40G, please keep running. and see if any issue. But, anyway, we have > modified the DTS code at /opt/test-harness/dts-new-suite. If we met same > problem, then use this new DTS instead. > > > > Thanks a lot > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Saturday, March 21, 2020 1:49 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, the 40G machine is stable enough to be put on production > dashboard to run tests which may cause Trex to be killed. > > Should I disable the 40G Intel machine for you to make changes? > > > > Also, just for confirmation: on the 10G machine, is the folder that you > are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r > are you still using the one in the standard */opt/test-harness/dts* > folder? > > > > If everything is ok, I will enable the 10G machine for production testing. > > > > Thank you very much > > > > On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Brandon, > > > > We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could > you please help to recover them? > > > > But, for FVL(40G) testbed, we met some problems, could you please help to > check before recover it > > - Sometime 1G hugepage will be changed to 2Mhugepage > automatically...we have to restart the system > - When we debugging on the testbed, found that Trex was killed by some > one(app).. > > Please help to check if any other program running on the testbed. > > > > Thanks a lot. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Sent:* Wednesday, March 18, 2020 9:04 PM > *To:* Brandon Lo <blo@iol.unh.edu> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Brandon, we almost made a workaround. > > > > Maybe tomorrow, you could recover Intel’s testbed. I will let you know > soon. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Wednesday, March 18, 2020 3:34 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Have you finished making changes on the Intel machine? > > I will turn on the machine on March 3rd for testing if you do not have any > issues with it. > > Please let me know if you need anything else. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex > upgrading. > > So we are reviewing our DTS script, different Trex version, and CI calling > procedure. > > > > Anyway, we are focusing on this task recently, any update will let you > know. > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 10, 2020 10:46 PM > *To:* David Marchand <david.marchand@redhat.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > How is the current status of the Intel 82599ES? > > Were there any configuration changes made to fix performance issues? > > > > Thanks > > > > On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wrote: > > Hi David, > > > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > > I'll rerun the tests that were affected and keep an eye out to see if it's > stable enough to be put back online. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com> > wrote: > > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header). > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu [-- Attachment #1.2: Type: text/html, Size: 82351 bytes --] [-- Attachment #2: image001.jpg --] [-- Type: image/jpeg, Size: 41815 bytes --]
[-- Attachment #1.1: Type: text/plain, Size: 21479 bytes --] Yes, they are work fine now, very thanks Brandon’s help. Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu] Sent: Friday, April 17, 2020 11:53 PM To: Ma, LihongX <lihongx.ma@intel.com> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I have fixed the NNT server not being called by Jenkins. It should work fine now. We did run into an issue where it could not detect huge pages information, but rebooting it again fixed the issue. I have also done the same thing to the FVL server. Both of them should be called by Jenkins for testing. Thanks, Brandon On Fri, Apr 17, 2020 at 3:00 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon I find the NNT server did not be called by the Jenkins, can you help to check? Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Thursday, April 16, 2020 11:04 PM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I saw your changes and applied them manually. This involves me taking the machine down for maintenance on our backend, and then doing a normal `reboot`. Now when I do `cat /proc/cmdline` it shows: `BOOT_IMAGE=/boot/vmlinuz-4.15.0-55-generic root=UUID=3801030b-237d-428e-9e67-d81e12f16308 ro quiet splash hugepagesz=1G hugepages=40 default_hugepagesz=1G isolcpus=1-21,49-69,24-37,72-85 intel_iommu=on iommu=pt nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85 intel_pstate=disable nmi_watchdog=0 audit=0 nosoftlockup processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable mce=off tsc=reliable numa_balancing=disable vt.handoff=1` Which is what I think we wanted to achieve. I have now done the `reboot_to_ro` command and it should be working fine for testing. Thanks, Brandon On Wed, Apr 15, 2020 at 11:40 PM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon I use the cmd ‘reboot_to_rw’ to restart server, and edit the file ‘/etc/default/grub’ to reset the isolation cpus, Changes the field ‘isolcpus=1-48 nohz_full=1-48 rcu_nocbs=1-48’ to ‘isolcpus=1-21,49-69,24-37,72-85 nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85’ Detail as below: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash hugepagesz=1G hugepages=40 default_hugepagesz=1G isolcpus=1-21,49-69,24-37,72-85 intel_iommu=on iommu=pt nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85 intel_pstate=disable nmi_watchdog=0 audit=0 nosoftlockup processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable mce=off tsc=reliable numa_balancing=disable" Then use cmd: update-grub to make the configuration effective. Then reboot server use ‘reboot_to_ro’, but I can’t find the change in cmdline ( cat /proc/cmdline). Can you help me to check it? Thanks Lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Thursday, April 16, 2020 3:31 AM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I think any time is fine to make these changes. Please use the 'reboot_to_rw' command once you have logged on, this will make sure your changes are saved. It will also disable testing momentarily while you make your changes. Once you're finished, you can reboot it with 'reboot_to_ro'. Please let me know when you're done so I can make sure everything is working as intended. We could also do this change for you if needed. Thanks, Brandon On Wed, Apr 15, 2020 at 5:39 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon I find the grub configuration of isolation cpu miss the logic cores which at some thread. For example: If the server cpus layout as below, and want to config the isolation cpu from 1-20 The config of isolation should is ‘isolcpus=1-20,49-68’ [cid:1717f4fa8006917eb1] So, I want to change the grub configuration of the isolation cpus. From ‘isolcpus=1-48’ change to ‘isolcpus=1-21,49-68,24-37,72-85’ Can you help me check which time is suite to do this change ? Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Friday, April 3, 2020 2:39 AM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I have changed the baselines to reflect the new expected values. The performance tests should work as expected and pass. We will email again in the future if we come across any problems. Feel free to email us as well if you would like to make any other changes. Thank you for all your help On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon Thanks for you recommends, I have done the changes. As the throughput value of nic_single_core is proportional to the cpu frequency. I recommend you can change the baseline according to our report system. On the our 2.50GHz system, the baseline value as below: NNT: pkt_size trd/rxd expected_value 64 512 52.562 64 2048 41.439 FVL: pkt_size trd/rxd expected_value 64 512 59.608 64 2048 47.73 For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number should be NNT: pkt_size trd/rxd expected_value 64 512 52.562 / 2.5 * 2.1=44.152 64 2048 41.439 / 2.5 * 2.1=34.809 FVL: pkt_size trd/rxd expected_value 64 512 59.608 / 2.5 * 2.1=50.071 64 2048 47.73 / 2.5 * 2.1=40.093 Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Tuesday, March 31, 2020 9:42 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, To make changes to either Intel machine, please reboot using the command "reboot_to_rw" as root to reboot the machine into read/write mode. This command will also disable any testing on the machine. To re-enable the machine, please run "reboot_to_ro" as root, and it will save all of the changes that you've made and re-enable testing on the machine. I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" instead of the normal "reboot" while you're making changes. After you're done, please let me know. I'll have to manually run a test and update the baseline using our internal CI. Thank you On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Please let me know how to make change to this reset machine. (ip/access...) and disable it. After that please help to change the baseline. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Thursday, March 26, 2020 11:39 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, we have a system in place that resets any changes made while testing is enabled for a machine. If you would like, I can disable testing and allow you to make permanent changes. I can also reset the baseline of Intel 10G test performance once you make these changes. Please let me know if you would like to make permanent changes on the Intel 10G so I can disable it for you. Thanks On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Thanks. Brandon. That’s good. We have made changed on 10G testbed. I monitored the several execution results; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjusting the expected number. I don’t know where the expected number is from? as I know it a dynamic number? depends on baseline.. Please help to clarify, thanks. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 24, 2020 9:31 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, I have enabled the 10G Intel machine for testing. If you would like to make any more changes, please let me know so I can perform the necessary steps to prepare the machine for changes. Please feel free to let me know if you need anything. Thank you On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, For 10G, please enable it. our code is at original path /opt/test-harness/dts. For 40G, please keep running. and see if any issue. But, anyway, we have modified the DTS code at /opt/test-harness/dts-new-suite. If we met same problem, then use this new DTS instead. Thanks a lot Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Saturday, March 21, 2020 1:49 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. Should I disable the 40G Intel machine for you to make changes? Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in /opt/test-harness/dts-2020-3-4, or are you still using the one in the standard /opt/test-harness/dts folder? If everything is ok, I will enable the 10G machine for production testing. Thank you very much On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it * Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system * When we debugging on the testbed, found that Trex was killed by some one(app).. Please help to check if any other program running on the testbed. Thanks a lot. Regards, Zhaoyan Chen From: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #1.2: Type: text/html, Size: 108194 bytes --] [-- Attachment #2: image001.jpg --] [-- Type: image/jpeg, Size: 41815 bytes --]
[-- Attachment #1: Type: text/plain, Size: 14780 bytes --] Hi, Brandon I find the baseline of NNT have changed as expected, but FVL still same as before. Can you help to check it and change the baseline as expected ? Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu] Sent: Friday, April 3, 2020 2:39 AM To: Ma, LihongX <lihongx.ma@intel.com> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I have changed the baselines to reflect the new expected values. The performance tests should work as expected and pass. We will email again in the future if we come across any problems. Feel free to email us as well if you would like to make any other changes. Thank you for all your help On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon Thanks for you recommends, I have done the changes. As the throughput value of nic_single_core is proportional to the cpu frequency. I recommend you can change the baseline according to our report system. On the our 2.50GHz system, the baseline value as below: NNT: pkt_size trd/rxd expected_value 64 512 52.562 64 2048 41.439 FVL: pkt_size trd/rxd expected_value 64 512 59.608 64 2048 47.73 For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number should be NNT: pkt_size trd/rxd expected_value 64 512 52.562 / 2.5 * 2.1=44.152 64 2048 41.439 / 2.5 * 2.1=34.809 FVL: pkt_size trd/rxd expected_value 64 512 59.608 / 2.5 * 2.1=50.071 64 2048 47.73 / 2.5 * 2.1=40.093 Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Tuesday, March 31, 2020 9:42 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, To make changes to either Intel machine, please reboot using the command "reboot_to_rw" as root to reboot the machine into read/write mode. This command will also disable any testing on the machine. To re-enable the machine, please run "reboot_to_ro" as root, and it will save all of the changes that you've made and re-enable testing on the machine. I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" instead of the normal "reboot" while you're making changes. After you're done, please let me know. I'll have to manually run a test and update the baseline using our internal CI. Thank you On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Please let me know how to make change to this reset machine. (ip/access...) and disable it. After that please help to change the baseline. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Thursday, March 26, 2020 11:39 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, we have a system in place that resets any changes made while testing is enabled for a machine. If you would like, I can disable testing and allow you to make permanent changes. I can also reset the baseline of Intel 10G test performance once you make these changes. Please let me know if you would like to make permanent changes on the Intel 10G so I can disable it for you. Thanks On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Thanks. Brandon. That’s good. We have made changed on 10G testbed. I monitored the several execution results; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjusting the expected number. I don’t know where the expected number is from? as I know it a dynamic number? depends on baseline.. Please help to clarify, thanks. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 24, 2020 9:31 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, I have enabled the 10G Intel machine for testing. If you would like to make any more changes, please let me know so I can perform the necessary steps to prepare the machine for changes. Please feel free to let me know if you need anything. Thank you On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, For 10G, please enable it. our code is at original path /opt/test-harness/dts. For 40G, please keep running. and see if any issue. But, anyway, we have modified the DTS code at /opt/test-harness/dts-new-suite. If we met same problem, then use this new DTS instead. Thanks a lot Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Saturday, March 21, 2020 1:49 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. Should I disable the 40G Intel machine for you to make changes? Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in /opt/test-harness/dts-2020-3-4, or are you still using the one in the standard /opt/test-harness/dts folder? If everything is ok, I will enable the 10G machine for production testing. Thank you very much On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it * Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system * When we debugging on the testbed, found that Trex was killed by some one(app).. Please help to check if any other program running on the testbed. Thanks a lot. Regards, Zhaoyan Chen From: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #2: Type: text/html, Size: 81936 bytes --]
[-- Attachment #1: Type: text/plain, Size: 14137 bytes --] Hi Lihong, Sorry about that, I have reset the baseline to the values you sent in the previous email. I'll look to rerun tests that have failed due to the incorrect baseline. Thanks for letting me know, Brandon On Mon, Apr 27, 2020 at 11:39 PM Ma, LihongX <lihongx.ma@intel.com> wrote: > Hi, Brandon > > I find the baseline of NNT have changed as expected, but FVL still same as > before. > > Can you help to check it and change the baseline as expected ? > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Friday, April 3, 2020 2:39 AM > *To:* Ma, LihongX <lihongx.ma@intel.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; > Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; > Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > I have changed the baselines to reflect the new expected values. > > The performance tests should work as expected and pass. > > > > We will email again in the future if we come across any problems. > > Feel free to email us as well if you would like to make any other changes. > > > > Thank you for all your help > > > > On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com> wrote: > > Hi, Brandon > > Thanks for you recommends, I have done the changes. > > As the throughput value of nic_single_core is proportional to the cpu > frequency. > > I recommend you can change the baseline according to our report system. > > > > On the our 2.50GHz system, the baseline value as below: > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 > > 64 > > 2048 > > 41.439 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 > > 64 > > 2048 > > 47.73 > > > > For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number > should be > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 / 2.5 * 2.1=44.152 > > 64 > > 2048 > > 41.439 / 2.5 * 2.1=34.809 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 / 2.5 * 2.1=50.071 > > 64 > > 2048 > > 47.73 / 2.5 * 2.1=40.093 > > > > > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Tuesday, March 31, 2020 9:42 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > To make changes to either Intel machine, please reboot using the command > "reboot_to_rw" as root to reboot the machine into read/write mode. > > This command will also disable any testing on the machine. > > > > To re-enable the machine, please run "reboot_to_ro" as root, and it will > save all of the changes that you've made and re-enable testing on the > machine. > > I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" > instead of the normal "reboot" while you're making changes. > > > > After you're done, please let me know. I'll have to manually run a test > and update the baseline using our internal CI. > > > > Thank you > > > > On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Please let me know how to make change to this reset machine. > (ip/access...) and disable it. > > > > After that please help to change the baseline. > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Thursday, March 26, 2020 11:39 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, we have a system in place that resets any changes made while > testing is enabled for a machine. > > If you would like, I can disable testing and allow you to make permanent > changes. > > > > I can also reset the baseline of Intel 10G test performance once you make > these changes. > > Please let me know if you would like to make permanent changes on the > Intel 10G so I can disable it for you. > > > > Thanks > > > > On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Thanks. Brandon. > > > > That’s good. We have made changed on 10G testbed. > > > > I monitored the several execution results; I found the results of 10G > always has -0.9%~-1.x% gap against expected number. So it could lead to see > sometime failures..+-1% I suggest adjusting the expected number. I don’t > know where the expected number is from? as I know it a dynamic number? > depends on baseline.. Please help to clarify, thanks. > > > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 24, 2020 9:31 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > I have enabled the 10G Intel machine for testing. > > If you would like to make any more changes, please let me know so I can > perform the necessary steps to prepare the machine for changes. > > Please feel free to let me know if you need anything. > > > > Thank you > > > > On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > For 10G, please enable it. our code is at original path > */opt/test-harness/dts.* > > > > For 40G, please keep running. and see if any issue. But, anyway, we have > modified the DTS code at /opt/test-harness/dts-new-suite. If we met same > problem, then use this new DTS instead. > > > > Thanks a lot > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Saturday, March 21, 2020 1:49 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, the 40G machine is stable enough to be put on production > dashboard to run tests which may cause Trex to be killed. > > Should I disable the 40G Intel machine for you to make changes? > > > > Also, just for confirmation: on the 10G machine, is the folder that you > are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r > are you still using the one in the standard */opt/test-harness/dts* > folder? > > > > If everything is ok, I will enable the 10G machine for production testing. > > > > Thank you very much > > > > On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Brandon, > > > > We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could > you please help to recover them? > > > > But, for FVL(40G) testbed, we met some problems, could you please help to > check before recover it > > - Sometime 1G hugepage will be changed to 2Mhugepage > automatically...we have to restart the system > - When we debugging on the testbed, found that Trex was killed by some > one(app).. > > Please help to check if any other program running on the testbed. > > > > Thanks a lot. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Sent:* Wednesday, March 18, 2020 9:04 PM > *To:* Brandon Lo <blo@iol.unh.edu> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Brandon, we almost made a workaround. > > > > Maybe tomorrow, you could recover Intel’s testbed. I will let you know > soon. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Wednesday, March 18, 2020 3:34 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Have you finished making changes on the Intel machine? > > I will turn on the machine on March 3rd for testing if you do not have any > issues with it. > > Please let me know if you need anything else. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex > upgrading. > > So we are reviewing our DTS script, different Trex version, and CI calling > procedure. > > > > Anyway, we are focusing on this task recently, any update will let you > know. > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 10, 2020 10:46 PM > *To:* David Marchand <david.marchand@redhat.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > How is the current status of the Intel 82599ES? > > Were there any configuration changes made to fix performance issues? > > > > Thanks > > > > On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wrote: > > Hi David, > > > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > > I'll rerun the tests that were affected and keep an eye out to see if it's > stable enough to be put back online. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com> > wrote: > > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header). > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu [-- Attachment #2: Type: text/html, Size: 65457 bytes --]
[-- Attachment #1.1: Type: text/plain, Size: 16513 bytes --] Hi, Brandon I checked the new result of FVL, find the expected value also not changed. From the log find the expected value also is: [cid:image001.png@01D61E29.70A1C270] Can you help to double check it ? Is there any different between FVL and NNT ? Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu] Sent: Wednesday, April 29, 2020 12:52 AM To: Ma, LihongX <lihongx.ma@intel.com> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com>; Lin, Xueqin <xueqin.lin@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, Sorry about that, I have reset the baseline to the values you sent in the previous email. I'll look to rerun tests that have failed due to the incorrect baseline. Thanks for letting me know, Brandon On Mon, Apr 27, 2020 at 11:39 PM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon I find the baseline of NNT have changed as expected, but FVL still same as before. Can you help to check it and change the baseline as expected ? Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Friday, April 3, 2020 2:39 AM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I have changed the baselines to reflect the new expected values. The performance tests should work as expected and pass. We will email again in the future if we come across any problems. Feel free to email us as well if you would like to make any other changes. Thank you for all your help On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon Thanks for you recommends, I have done the changes. As the throughput value of nic_single_core is proportional to the cpu frequency. I recommend you can change the baseline according to our report system. On the our 2.50GHz system, the baseline value as below: NNT: pkt_size trd/rxd expected_value 64 512 52.562 64 2048 41.439 FVL: pkt_size trd/rxd expected_value 64 512 59.608 64 2048 47.73 For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number should be NNT: pkt_size trd/rxd expected_value 64 512 52.562 / 2.5 * 2.1=44.152 64 2048 41.439 / 2.5 * 2.1=34.809 FVL: pkt_size trd/rxd expected_value 64 512 59.608 / 2.5 * 2.1=50.071 64 2048 47.73 / 2.5 * 2.1=40.093 Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Tuesday, March 31, 2020 9:42 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, To make changes to either Intel machine, please reboot using the command "reboot_to_rw" as root to reboot the machine into read/write mode. This command will also disable any testing on the machine. To re-enable the machine, please run "reboot_to_ro" as root, and it will save all of the changes that you've made and re-enable testing on the machine. I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" instead of the normal "reboot" while you're making changes. After you're done, please let me know. I'll have to manually run a test and update the baseline using our internal CI. Thank you On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Please let me know how to make change to this reset machine. (ip/access...) and disable it. After that please help to change the baseline. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Thursday, March 26, 2020 11:39 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, we have a system in place that resets any changes made while testing is enabled for a machine. If you would like, I can disable testing and allow you to make permanent changes. I can also reset the baseline of Intel 10G test performance once you make these changes. Please let me know if you would like to make permanent changes on the Intel 10G so I can disable it for you. Thanks On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Thanks. Brandon. That’s good. We have made changed on 10G testbed. I monitored the several execution results; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjusting the expected number. I don’t know where the expected number is from? as I know it a dynamic number? depends on baseline.. Please help to clarify, thanks. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 24, 2020 9:31 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, I have enabled the 10G Intel machine for testing. If you would like to make any more changes, please let me know so I can perform the necessary steps to prepare the machine for changes. Please feel free to let me know if you need anything. Thank you On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, For 10G, please enable it. our code is at original path /opt/test-harness/dts. For 40G, please keep running. and see if any issue. But, anyway, we have modified the DTS code at /opt/test-harness/dts-new-suite. If we met same problem, then use this new DTS instead. Thanks a lot Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Saturday, March 21, 2020 1:49 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. Should I disable the 40G Intel machine for you to make changes? Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in /opt/test-harness/dts-2020-3-4, or are you still using the one in the standard /opt/test-harness/dts folder? If everything is ok, I will enable the 10G machine for production testing. Thank you very much On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it * Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system * When we debugging on the testbed, found that Trex was killed by some one(app).. Please help to check if any other program running on the testbed. Thanks a lot. Regards, Zhaoyan Chen From: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #1.2: Type: text/html, Size: 87012 bytes --] [-- Attachment #2: image001.png --] [-- Type: image/png, Size: 4767 bytes --]
[-- Attachment #1.1: Type: text/plain, Size: 15690 bytes --] Hi Lihong, The expected value was reset by one of our internal scripts. I believe that I have resolved this issue for the future by ensuring that the baseline that you sent me will not be overwritten automatically. I will continue to monitor this expected throughput in case of any issues. Thanks for your patience, Brandon On Wed, Apr 29, 2020 at 1:30 AM Ma, LihongX <lihongx.ma@intel.com> wrote: > Hi, Brandon > > > > I checked the new result of FVL, find the expected value also not changed. > > From the log find the expected value also is: > > > > Can you help to double check it ? Is there any different between FVL and > NNT ? > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Wednesday, April 29, 2020 12:52 AM > *To:* Ma, LihongX <lihongx.ma@intel.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; > Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; > Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com>; Lin, Xueqin <xueqin.lin@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > Sorry about that, I have reset the baseline to the values you sent in the > previous email. > > I'll look to rerun tests that have failed due to the incorrect baseline. > > > > Thanks for letting me know, > > Brandon > > > > On Mon, Apr 27, 2020 at 11:39 PM Ma, LihongX <lihongx.ma@intel.com> wrote: > > Hi, Brandon > > I find the baseline of NNT have changed as expected, but FVL still same as > before. > > Can you help to check it and change the baseline as expected ? > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Friday, April 3, 2020 2:39 AM > *To:* Ma, LihongX <lihongx.ma@intel.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; > Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; > Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > I have changed the baselines to reflect the new expected values. > > The performance tests should work as expected and pass. > > > > We will email again in the future if we come across any problems. > > Feel free to email us as well if you would like to make any other changes. > > > > Thank you for all your help > > > > On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com> wrote: > > Hi, Brandon > > Thanks for you recommends, I have done the changes. > > As the throughput value of nic_single_core is proportional to the cpu > frequency. > > I recommend you can change the baseline according to our report system. > > > > On the our 2.50GHz system, the baseline value as below: > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 > > 64 > > 2048 > > 41.439 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 > > 64 > > 2048 > > 47.73 > > > > For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number > should be > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 / 2.5 * 2.1=44.152 > > 64 > > 2048 > > 41.439 / 2.5 * 2.1=34.809 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 / 2.5 * 2.1=50.071 > > 64 > > 2048 > > 47.73 / 2.5 * 2.1=40.093 > > > > > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Tuesday, March 31, 2020 9:42 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > To make changes to either Intel machine, please reboot using the command > "reboot_to_rw" as root to reboot the machine into read/write mode. > > This command will also disable any testing on the machine. > > > > To re-enable the machine, please run "reboot_to_ro" as root, and it will > save all of the changes that you've made and re-enable testing on the > machine. > > I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" > instead of the normal "reboot" while you're making changes. > > > > After you're done, please let me know. I'll have to manually run a test > and update the baseline using our internal CI. > > > > Thank you > > > > On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Please let me know how to make change to this reset machine. > (ip/access...) and disable it. > > > > After that please help to change the baseline. > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Thursday, March 26, 2020 11:39 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, we have a system in place that resets any changes made while > testing is enabled for a machine. > > If you would like, I can disable testing and allow you to make permanent > changes. > > > > I can also reset the baseline of Intel 10G test performance once you make > these changes. > > Please let me know if you would like to make permanent changes on the > Intel 10G so I can disable it for you. > > > > Thanks > > > > On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Thanks. Brandon. > > > > That’s good. We have made changed on 10G testbed. > > > > I monitored the several execution results; I found the results of 10G > always has -0.9%~-1.x% gap against expected number. So it could lead to see > sometime failures..+-1% I suggest adjusting the expected number. I don’t > know where the expected number is from? as I know it a dynamic number? > depends on baseline.. Please help to clarify, thanks. > > > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 24, 2020 9:31 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > I have enabled the 10G Intel machine for testing. > > If you would like to make any more changes, please let me know so I can > perform the necessary steps to prepare the machine for changes. > > Please feel free to let me know if you need anything. > > > > Thank you > > > > On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > For 10G, please enable it. our code is at original path > */opt/test-harness/dts.* > > > > For 40G, please keep running. and see if any issue. But, anyway, we have > modified the DTS code at /opt/test-harness/dts-new-suite. If we met same > problem, then use this new DTS instead. > > > > Thanks a lot > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Saturday, March 21, 2020 1:49 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, the 40G machine is stable enough to be put on production > dashboard to run tests which may cause Trex to be killed. > > Should I disable the 40G Intel machine for you to make changes? > > > > Also, just for confirmation: on the 10G machine, is the folder that you > are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r > are you still using the one in the standard */opt/test-harness/dts* > folder? > > > > If everything is ok, I will enable the 10G machine for production testing. > > > > Thank you very much > > > > On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Brandon, > > > > We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could > you please help to recover them? > > > > But, for FVL(40G) testbed, we met some problems, could you please help to > check before recover it > > - Sometime 1G hugepage will be changed to 2Mhugepage > automatically...we have to restart the system > - When we debugging on the testbed, found that Trex was killed by some > one(app).. > > Please help to check if any other program running on the testbed. > > > > Thanks a lot. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Sent:* Wednesday, March 18, 2020 9:04 PM > *To:* Brandon Lo <blo@iol.unh.edu> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Brandon, we almost made a workaround. > > > > Maybe tomorrow, you could recover Intel’s testbed. I will let you know > soon. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Wednesday, March 18, 2020 3:34 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Have you finished making changes on the Intel machine? > > I will turn on the machine on March 3rd for testing if you do not have any > issues with it. > > Please let me know if you need anything else. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex > upgrading. > > So we are reviewing our DTS script, different Trex version, and CI calling > procedure. > > > > Anyway, we are focusing on this task recently, any update will let you > know. > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 10, 2020 10:46 PM > *To:* David Marchand <david.marchand@redhat.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > How is the current status of the Intel 82599ES? > > Were there any configuration changes made to fix performance issues? > > > > Thanks > > > > On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wrote: > > Hi David, > > > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > > I'll rerun the tests that were affected and keep an eye out to see if it's > stable enough to be put back online. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com> > wrote: > > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header). > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu [-- Attachment #1.2: Type: text/html, Size: 71056 bytes --] [-- Attachment #2: image001.png --] [-- Type: image/png, Size: 4767 bytes --]
[-- Attachment #1.1: Type: text/plain, Size: 17038 bytes --] Hi Lihong, Just a further update: we have noticed that there is another internal script that is used to calculate baselines that will pull a newer baseline if it is found. We are looking to solve the issues that we are having with baselines and will get back to you. Thanks for your patience, Brandon On Thu, Apr 30, 2020 at 10:19 AM Brandon Lo <blo@iol.unh.edu> wrote: > Hi Lihong, > > The expected value was reset by one of our internal scripts. > I believe that I have resolved this issue for the future by ensuring that > the baseline that you sent me will not be overwritten automatically. > > I will continue to monitor this expected throughput in case of any issues. > > Thanks for your patience, > Brandon > > On Wed, Apr 29, 2020 at 1:30 AM Ma, LihongX <lihongx.ma@intel.com> wrote: > >> Hi, Brandon >> >> >> >> I checked the new result of FVL, find the expected value also not changed. >> >> From the log find the expected value also is: >> >> >> >> Can you help to double check it ? Is there any different between FVL and >> NNT ? >> >> >> >> Regards, >> >> Ma,lihong >> >> >> >> *From:* Brandon Lo [mailto:blo@iol.unh.edu] >> *Sent:* Wednesday, April 29, 2020 12:52 AM >> *To:* Ma, LihongX <lihongx.ma@intel.com> >> *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand < >> david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < >> lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; >> Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; >> Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < >> tim.odriscoll@intel.com>; Lin, Xueqin <xueqin.lin@intel.com> >> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing >> >> >> >> Hi Lihong, >> >> >> >> Sorry about that, I have reset the baseline to the values you sent in the >> previous email. >> >> I'll look to rerun tests that have failed due to the incorrect baseline. >> >> >> >> Thanks for letting me know, >> >> Brandon >> >> >> >> On Mon, Apr 27, 2020 at 11:39 PM Ma, LihongX <lihongx.ma@intel.com> >> wrote: >> >> Hi, Brandon >> >> I find the baseline of NNT have changed as expected, but FVL still same >> as before. >> >> Can you help to check it and change the baseline as expected ? >> >> >> >> Regards, >> >> Ma,lihong >> >> >> >> *From:* Brandon Lo [mailto:blo@iol.unh.edu] >> *Sent:* Friday, April 3, 2020 2:39 AM >> *To:* Ma, LihongX <lihongx.ma@intel.com> >> *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand < >> david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < >> lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; >> Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; >> Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < >> tim.odriscoll@intel.com> >> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing >> >> >> >> Hi Lihong, >> >> >> >> I have changed the baselines to reflect the new expected values. >> >> The performance tests should work as expected and pass. >> >> >> >> We will email again in the future if we come across any problems. >> >> Feel free to email us as well if you would like to make any other changes. >> >> >> >> Thank you for all your help >> >> >> >> On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com> wrote: >> >> Hi, Brandon >> >> Thanks for you recommends, I have done the changes. >> >> As the throughput value of nic_single_core is proportional to the cpu >> frequency. >> >> I recommend you can change the baseline according to our report system. >> >> >> >> On the our 2.50GHz system, the baseline value as below: >> >> NNT: >> >> *pkt_size* >> >> *trd/rxd* >> >> *expected_value* >> >> 64 >> >> 512 >> >> 52.562 >> >> 64 >> >> 2048 >> >> 41.439 >> >> >> >> FVL: >> >> *pkt_size* >> >> *trd/rxd* >> >> *expected_value* >> >> 64 >> >> 512 >> >> 59.608 >> >> 64 >> >> 2048 >> >> 47.73 >> >> >> >> For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number >> should be >> >> NNT: >> >> *pkt_size* >> >> *trd/rxd* >> >> *expected_value* >> >> 64 >> >> 512 >> >> 52.562 / 2.5 * 2.1=44.152 >> >> 64 >> >> 2048 >> >> 41.439 / 2.5 * 2.1=34.809 >> >> >> >> FVL: >> >> *pkt_size* >> >> *trd/rxd* >> >> *expected_value* >> >> 64 >> >> 512 >> >> 59.608 / 2.5 * 2.1=50.071 >> >> 64 >> >> 2048 >> >> 47.73 / 2.5 * 2.1=40.093 >> >> >> >> >> >> >> >> Regards, >> >> Ma,lihong >> >> >> >> *From:* Brandon Lo [mailto:blo@iol.unh.edu] >> *Sent:* Tuesday, March 31, 2020 9:42 PM >> *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> >> *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; >> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < >> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, >> Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, >> XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < >> tim.odriscoll@intel.com> >> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing >> >> >> >> Hi Zhaoyan, >> >> >> >> To make changes to either Intel machine, please reboot using the command >> "reboot_to_rw" as root to reboot the machine into read/write mode. >> >> This command will also disable any testing on the machine. >> >> >> >> To re-enable the machine, please run "reboot_to_ro" as root, and it will >> save all of the changes that you've made and re-enable testing on the >> machine. >> >> I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" >> instead of the normal "reboot" while you're making changes. >> >> >> >> After you're done, please let me know. I'll have to manually run a test >> and update the baseline using our internal CI. >> >> >> >> Thank you >> >> >> >> On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> >> wrote: >> >> Hi, Brandon, >> >> >> >> Please let me know how to make change to this reset machine. >> (ip/access...) and disable it. >> >> >> >> After that please help to change the baseline. >> >> >> >> >> >> *Regards,* >> >> *Zhaoyan Chen* >> >> >> >> *From:* Brandon Lo <blo@iol.unh.edu> >> *Sent:* Thursday, March 26, 2020 11:39 PM >> *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> >> *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; >> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < >> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, >> Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, >> XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < >> tim.odriscoll@intel.com> >> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing >> >> >> >> Hi Zhaoyan, >> >> >> >> Currently, we have a system in place that resets any changes made while >> testing is enabled for a machine. >> >> If you would like, I can disable testing and allow you to make permanent >> changes. >> >> >> >> I can also reset the baseline of Intel 10G test performance once you make >> these changes. >> >> Please let me know if you would like to make permanent changes on the >> Intel 10G so I can disable it for you. >> >> >> >> Thanks >> >> >> >> On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> >> wrote: >> >> Thanks. Brandon. >> >> >> >> That’s good. We have made changed on 10G testbed. >> >> >> >> I monitored the several execution results; I found the results of 10G >> always has -0.9%~-1.x% gap against expected number. So it could lead to see >> sometime failures..+-1% I suggest adjusting the expected number. I don’t >> know where the expected number is from? as I know it a dynamic number? >> depends on baseline.. Please help to clarify, thanks. >> >> >> >> >> >> Thanks. >> >> >> >> *Regards,* >> >> *Zhaoyan Chen* >> >> >> >> *From:* Brandon Lo <blo@iol.unh.edu> >> *Sent:* Tuesday, March 24, 2020 9:31 PM >> *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> >> *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; >> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < >> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, >> Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, >> XuemingX <xuemingx.zhang@intel.com> >> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing >> >> >> >> Hi Zhaoyan, >> >> >> >> I have enabled the 10G Intel machine for testing. >> >> If you would like to make any more changes, please let me know so I can >> perform the necessary steps to prepare the machine for changes. >> >> Please feel free to let me know if you need anything. >> >> >> >> Thank you >> >> >> >> On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> >> wrote: >> >> Hi, Brandon, >> >> >> >> For 10G, please enable it. our code is at original path >> */opt/test-harness/dts.* >> >> >> >> For 40G, please keep running. and see if any issue. But, anyway, we have >> modified the DTS code at /opt/test-harness/dts-new-suite. If we met same >> problem, then use this new DTS instead. >> >> >> >> Thanks a lot >> >> >> >> *Regards,* >> >> *Zhaoyan Chen* >> >> >> >> *From:* Brandon Lo <blo@iol.unh.edu> >> *Sent:* Saturday, March 21, 2020 1:49 AM >> *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> >> *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; >> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < >> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, >> Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, >> XuemingX <xuemingx.zhang@intel.com> >> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing >> >> >> >> Hi Zhaoyan, >> >> >> >> Currently, the 40G machine is stable enough to be put on production >> dashboard to run tests which may cause Trex to be killed. >> >> Should I disable the 40G Intel machine for you to make changes? >> >> >> >> Also, just for confirmation: on the 10G machine, is the folder that you >> are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r >> are you still using the one in the standard */opt/test-harness/dts* >> folder? >> >> >> >> If everything is ok, I will enable the 10G machine for production testing. >> >> >> >> Thank you very much >> >> >> >> On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> >> wrote: >> >> Brandon, >> >> >> >> We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). >> Could you please help to recover them? >> >> >> >> But, for FVL(40G) testbed, we met some problems, could you please help >> to check before recover it >> >> - Sometime 1G hugepage will be changed to 2Mhugepage >> automatically...we have to restart the system >> - When we debugging on the testbed, found that Trex was killed by >> some one(app).. >> >> Please help to check if any other program running on the testbed. >> >> >> >> Thanks a lot. >> >> >> >> >> >> >> >> *Regards,* >> >> *Zhaoyan Chen* >> >> >> >> *From:* Chen, Zhaoyan <zhaoyan.chen@intel.com> >> *Sent:* Wednesday, March 18, 2020 9:04 PM >> *To:* Brandon Lo <blo@iol.unh.edu> >> *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; >> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < >> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, >> Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com> >> *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing >> >> >> >> Brandon, we almost made a workaround. >> >> >> >> Maybe tomorrow, you could recover Intel’s testbed. I will let you know >> soon. >> >> >> >> >> >> >> >> *Regards,* >> >> *Zhaoyan Chen* >> >> >> >> *From:* Brandon Lo <blo@iol.unh.edu> >> *Sent:* Wednesday, March 18, 2020 3:34 AM >> *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> >> *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; >> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < >> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, >> Qian Q <qian.q.xu@intel.com> >> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing >> >> >> >> Hi Zhaoyan, >> >> >> >> Have you finished making changes on the Intel machine? >> >> I will turn on the machine on March 3rd for testing if you do not have >> any issues with it. >> >> Please let me know if you need anything else. >> >> >> >> Thanks >> >> >> >> On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> >> wrote: >> >> Hi, Brandon, >> >> >> >> Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex >> upgrading. >> >> So we are reviewing our DTS script, different Trex version, and CI >> calling procedure. >> >> >> >> Anyway, we are focusing on this task recently, any update will let you >> know. >> >> >> >> Thanks. >> >> >> >> *Regards,* >> >> *Zhaoyan Chen* >> >> >> >> *From:* Brandon Lo <blo@iol.unh.edu> >> *Sent:* Tuesday, March 10, 2020 10:46 PM >> *To:* David Marchand <david.marchand@redhat.com> >> *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.edu; >> Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < >> thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, >> Qian Q <qian.q.xu@intel.com> >> *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing >> >> >> >> Hi Zhaoyan, >> >> >> >> How is the current status of the Intel 82599ES? >> >> Were there any configuration changes made to fix performance issues? >> >> >> >> Thanks >> >> >> >> On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wrote: >> >> Hi David, >> >> >> >> This was just a weird issue with the packet generator not cleaning itself >> after a test fast enough before another test. >> >> I'll rerun the tests that were affected and keep an eye out to see if >> it's stable enough to be put back online. >> >> >> >> Thanks >> >> >> >> On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com> >> wrote: >> >> On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote: >> > >> > Hi David and Zhaoyan, >> > >> > >> > Yes, those results are related to the Intel machine; I have disabled >> testing for the Intel testbed. >> > >> > The 82599ES machine is now available for ssh and modifications. >> >> Any news about this? >> >> I received a failure on a patch of mine (changing macros in a ARM header). >> https://lab.dpdk.org/results/dashboard/patchsets/9900/ >> >> But this time, it is with the 40G Intel nic test. >> >> -- >> David Marchand >> >> >> >> >> -- >> >> Brandon Lo >> >> UNH InterOperability Laboratory >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824 >> >> blo@iol.unh.edu >> >> www.iol.unh.edu >> >> >> >> >> -- >> >> Brandon Lo >> >> UNH InterOperability Laboratory >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824 >> >> blo@iol.unh.edu >> >> www.iol.unh.edu >> >> >> >> >> -- >> >> Brandon Lo >> >> UNH InterOperability Laboratory >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824 >> >> blo@iol.unh.edu >> >> www.iol.unh.edu >> >> >> >> >> -- >> >> Brandon Lo >> >> UNH InterOperability Laboratory >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824 >> >> blo@iol.unh.edu >> >> www.iol.unh.edu >> >> >> >> >> -- >> >> Brandon Lo >> >> UNH InterOperability Laboratory >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824 >> >> blo@iol.unh.edu >> >> www.iol.unh.edu >> >> >> >> >> -- >> >> Brandon Lo >> >> UNH InterOperability Laboratory >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824 >> >> blo@iol.unh.edu >> >> www.iol.unh.edu >> >> >> >> >> -- >> >> Brandon Lo >> >> UNH InterOperability Laboratory >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824 >> >> blo@iol.unh.edu >> >> www.iol.unh.edu >> >> >> >> >> -- >> >> Brandon Lo >> >> UNH InterOperability Laboratory >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824 >> >> blo@iol.unh.edu >> >> www.iol.unh.edu >> >> >> >> >> -- >> >> Brandon Lo >> >> UNH InterOperability Laboratory >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824 >> >> blo@iol.unh.edu >> >> www.iol.unh.edu >> > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu [-- Attachment #1.2: Type: text/html, Size: 73215 bytes --] [-- Attachment #2: image001.png --] [-- Type: image/png, Size: 4767 bytes --]
[-- Attachment #1.1: Type: text/plain, Size: 18759 bytes --] Thanks Brandon, wait your reply. Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu] Sent: Wednesday, May 6, 2020 9:05 PM To: Ma, LihongX <lihongx.ma@intel.com> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com>; Lin, Xueqin <xueqin.lin@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, Just a further update: we have noticed that there is another internal script that is used to calculate baselines that will pull a newer baseline if it is found. We are looking to solve the issues that we are having with baselines and will get back to you. Thanks for your patience, Brandon On Thu, Apr 30, 2020 at 10:19 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi Lihong, The expected value was reset by one of our internal scripts. I believe that I have resolved this issue for the future by ensuring that the baseline that you sent me will not be overwritten automatically. I will continue to monitor this expected throughput in case of any issues. Thanks for your patience, Brandon On Wed, Apr 29, 2020 at 1:30 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon I checked the new result of FVL, find the expected value also not changed. From the log find the expected value also is: [cid:image001.png@01D62552.0A1802A0] Can you help to double check it ? Is there any different between FVL and NNT ? Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Wednesday, April 29, 2020 12:52 AM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>>; Lin, Xueqin <xueqin.lin@intel.com<mailto:xueqin.lin@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, Sorry about that, I have reset the baseline to the values you sent in the previous email. I'll look to rerun tests that have failed due to the incorrect baseline. Thanks for letting me know, Brandon On Mon, Apr 27, 2020 at 11:39 PM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon I find the baseline of NNT have changed as expected, but FVL still same as before. Can you help to check it and change the baseline as expected ? Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Friday, April 3, 2020 2:39 AM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I have changed the baselines to reflect the new expected values. The performance tests should work as expected and pass. We will email again in the future if we come across any problems. Feel free to email us as well if you would like to make any other changes. Thank you for all your help On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon Thanks for you recommends, I have done the changes. As the throughput value of nic_single_core is proportional to the cpu frequency. I recommend you can change the baseline according to our report system. On the our 2.50GHz system, the baseline value as below: NNT: pkt_size trd/rxd expected_value 64 512 52.562 64 2048 41.439 FVL: pkt_size trd/rxd expected_value 64 512 59.608 64 2048 47.73 For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number should be NNT: pkt_size trd/rxd expected_value 64 512 52.562 / 2.5 * 2.1=44.152 64 2048 41.439 / 2.5 * 2.1=34.809 FVL: pkt_size trd/rxd expected_value 64 512 59.608 / 2.5 * 2.1=50.071 64 2048 47.73 / 2.5 * 2.1=40.093 Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Tuesday, March 31, 2020 9:42 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, To make changes to either Intel machine, please reboot using the command "reboot_to_rw" as root to reboot the machine into read/write mode. This command will also disable any testing on the machine. To re-enable the machine, please run "reboot_to_ro" as root, and it will save all of the changes that you've made and re-enable testing on the machine. I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" instead of the normal "reboot" while you're making changes. After you're done, please let me know. I'll have to manually run a test and update the baseline using our internal CI. Thank you On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Please let me know how to make change to this reset machine. (ip/access...) and disable it. After that please help to change the baseline. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Thursday, March 26, 2020 11:39 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, we have a system in place that resets any changes made while testing is enabled for a machine. If you would like, I can disable testing and allow you to make permanent changes. I can also reset the baseline of Intel 10G test performance once you make these changes. Please let me know if you would like to make permanent changes on the Intel 10G so I can disable it for you. Thanks On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Thanks. Brandon. That’s good. We have made changed on 10G testbed. I monitored the several execution results; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjusting the expected number. I don’t know where the expected number is from? as I know it a dynamic number? depends on baseline.. Please help to clarify, thanks. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 24, 2020 9:31 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, I have enabled the 10G Intel machine for testing. If you would like to make any more changes, please let me know so I can perform the necessary steps to prepare the machine for changes. Please feel free to let me know if you need anything. Thank you On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, For 10G, please enable it. our code is at original path /opt/test-harness/dts. For 40G, please keep running. and see if any issue. But, anyway, we have modified the DTS code at /opt/test-harness/dts-new-suite. If we met same problem, then use this new DTS instead. Thanks a lot Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Saturday, March 21, 2020 1:49 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. Should I disable the 40G Intel machine for you to make changes? Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in /opt/test-harness/dts-2020-3-4, or are you still using the one in the standard /opt/test-harness/dts folder? If everything is ok, I will enable the 10G machine for production testing. Thank you very much On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it * Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system * When we debugging on the testbed, found that Trex was killed by some one(app).. Please help to check if any other program running on the testbed. Thanks a lot. Regards, Zhaoyan Chen From: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #1.2: Type: text/html, Size: 95596 bytes --] [-- Attachment #2: image001.png --] [-- Type: image/png, Size: 4767 bytes --]
[-- Attachment #1.1: Type: text/plain, Size: 19921 bytes --] Hi, Brandon Thanks for your help to check the issues of baseline, and I find the FVL env is ok, the expected value has been changed to the value we wanted. But the baseline of NNT has changed, can you help to check it ? Thanks Regards, Ma,lihong From: Ma, LihongX Sent: Friday, May 8, 2020 4:02 PM To: 'Brandon Lo' <blo@iol.unh.edu> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com>; Lin, Xueqin <xueqin.lin@intel.com> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Thanks Brandon, wait your reply. Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu] Sent: Wednesday, May 6, 2020 9:05 PM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>>; Lin, Xueqin <xueqin.lin@intel.com<mailto:xueqin.lin@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, Just a further update: we have noticed that there is another internal script that is used to calculate baselines that will pull a newer baseline if it is found. We are looking to solve the issues that we are having with baselines and will get back to you. Thanks for your patience, Brandon On Thu, Apr 30, 2020 at 10:19 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi Lihong, The expected value was reset by one of our internal scripts. I believe that I have resolved this issue for the future by ensuring that the baseline that you sent me will not be overwritten automatically. I will continue to monitor this expected throughput in case of any issues. Thanks for your patience, Brandon On Wed, Apr 29, 2020 at 1:30 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon I checked the new result of FVL, find the expected value also not changed. From the log find the expected value also is: [cid:image001.png@01D62862.A8915090] Can you help to double check it ? Is there any different between FVL and NNT ? Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Wednesday, April 29, 2020 12:52 AM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>>; Lin, Xueqin <xueqin.lin@intel.com<mailto:xueqin.lin@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, Sorry about that, I have reset the baseline to the values you sent in the previous email. I'll look to rerun tests that have failed due to the incorrect baseline. Thanks for letting me know, Brandon On Mon, Apr 27, 2020 at 11:39 PM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon I find the baseline of NNT have changed as expected, but FVL still same as before. Can you help to check it and change the baseline as expected ? Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Friday, April 3, 2020 2:39 AM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I have changed the baselines to reflect the new expected values. The performance tests should work as expected and pass. We will email again in the future if we come across any problems. Feel free to email us as well if you would like to make any other changes. Thank you for all your help On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon Thanks for you recommends, I have done the changes. As the throughput value of nic_single_core is proportional to the cpu frequency. I recommend you can change the baseline according to our report system. On the our 2.50GHz system, the baseline value as below: NNT: pkt_size trd/rxd expected_value 64 512 52.562 64 2048 41.439 FVL: pkt_size trd/rxd expected_value 64 512 59.608 64 2048 47.73 For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number should be NNT: pkt_size trd/rxd expected_value 64 512 52.562 / 2.5 * 2.1=44.152 64 2048 41.439 / 2.5 * 2.1=34.809 FVL: pkt_size trd/rxd expected_value 64 512 59.608 / 2.5 * 2.1=50.071 64 2048 47.73 / 2.5 * 2.1=40.093 Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Tuesday, March 31, 2020 9:42 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, To make changes to either Intel machine, please reboot using the command "reboot_to_rw" as root to reboot the machine into read/write mode. This command will also disable any testing on the machine. To re-enable the machine, please run "reboot_to_ro" as root, and it will save all of the changes that you've made and re-enable testing on the machine. I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" instead of the normal "reboot" while you're making changes. After you're done, please let me know. I'll have to manually run a test and update the baseline using our internal CI. Thank you On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Please let me know how to make change to this reset machine. (ip/access...) and disable it. After that please help to change the baseline. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Thursday, March 26, 2020 11:39 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, we have a system in place that resets any changes made while testing is enabled for a machine. If you would like, I can disable testing and allow you to make permanent changes. I can also reset the baseline of Intel 10G test performance once you make these changes. Please let me know if you would like to make permanent changes on the Intel 10G so I can disable it for you. Thanks On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Thanks. Brandon. That’s good. We have made changed on 10G testbed. I monitored the several execution results; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjusting the expected number. I don’t know where the expected number is from? as I know it a dynamic number? depends on baseline.. Please help to clarify, thanks. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 24, 2020 9:31 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, I have enabled the 10G Intel machine for testing. If you would like to make any more changes, please let me know so I can perform the necessary steps to prepare the machine for changes. Please feel free to let me know if you need anything. Thank you On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, For 10G, please enable it. our code is at original path /opt/test-harness/dts. For 40G, please keep running. and see if any issue. But, anyway, we have modified the DTS code at /opt/test-harness/dts-new-suite. If we met same problem, then use this new DTS instead. Thanks a lot Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Saturday, March 21, 2020 1:49 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. Should I disable the 40G Intel machine for you to make changes? Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in /opt/test-harness/dts-2020-3-4, or are you still using the one in the standard /opt/test-harness/dts folder? If everything is ok, I will enable the 10G machine for production testing. Thank you very much On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it * Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system * When we debugging on the testbed, found that Trex was killed by some one(app).. Please help to check if any other program running on the testbed. Thanks a lot. Regards, Zhaoyan Chen From: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #1.2: Type: text/html, Size: 100596 bytes --] [-- Attachment #2: image001.png --] [-- Type: image/png, Size: 4767 bytes --]
[-- Attachment #1.1: Type: text/plain, Size: 18369 bytes --] Hi Lihong, I've applied the same fix to the Intel 10G machine. It should be pulling the correct baseline values from now on. Thanks, Brandon On Tue, May 12, 2020 at 1:41 AM Ma, LihongX <lihongx.ma@intel.com> wrote: > Hi, Brandon > > > > Thanks for your help to check the issues of baseline, and I find the FVL > env is ok, the expected value has been changed to the value we wanted. > > But the baseline of NNT has changed, can you help to check it ? > > > > Thanks > > Regards, > > Ma,lihong > > > > *From:* Ma, LihongX > *Sent:* Friday, May 8, 2020 4:02 PM > *To:* 'Brandon Lo' <blo@iol.unh.edu> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; > Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; > Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com>; Lin, Xueqin <xueqin.lin@intel.com> > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Thanks Brandon, wait your reply. > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu <blo@iol.unh.edu>] > *Sent:* Wednesday, May 6, 2020 9:05 PM > *To:* Ma, LihongX <lihongx.ma@intel.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; > Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; > Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com>; Lin, Xueqin <xueqin.lin@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > Just a further update: we have noticed that there is another internal > script that is used to calculate baselines that will pull a newer baseline > if it is found. > > We are looking to solve the issues that we are having with baselines and > will get back to you. > > > > Thanks for your patience, > > Brandon > > > > On Thu, Apr 30, 2020 at 10:19 AM Brandon Lo <blo@iol.unh.edu> wrote: > > Hi Lihong, > > > > The expected value was reset by one of our internal scripts. > > I believe that I have resolved this issue for the future by ensuring that > the baseline that you sent me will not be overwritten automatically. > > > > I will continue to monitor this expected throughput in case of any issues. > > > > Thanks for your patience, > > Brandon > > > > On Wed, Apr 29, 2020 at 1:30 AM Ma, LihongX <lihongx.ma@intel.com> wrote: > > Hi, Brandon > > > > I checked the new result of FVL, find the expected value also not changed. > > From the log find the expected value also is: > > > > Can you help to double check it ? Is there any different between FVL and > NNT ? > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Wednesday, April 29, 2020 12:52 AM > *To:* Ma, LihongX <lihongx.ma@intel.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; > Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; > Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com>; Lin, Xueqin <xueqin.lin@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > Sorry about that, I have reset the baseline to the values you sent in the > previous email. > > I'll look to rerun tests that have failed due to the incorrect baseline. > > > > Thanks for letting me know, > > Brandon > > > > On Mon, Apr 27, 2020 at 11:39 PM Ma, LihongX <lihongx.ma@intel.com> wrote: > > Hi, Brandon > > I find the baseline of NNT have changed as expected, but FVL still same as > before. > > Can you help to check it and change the baseline as expected ? > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Friday, April 3, 2020 2:39 AM > *To:* Ma, LihongX <lihongx.ma@intel.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; > Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; > Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > I have changed the baselines to reflect the new expected values. > > The performance tests should work as expected and pass. > > > > We will email again in the future if we come across any problems. > > Feel free to email us as well if you would like to make any other changes. > > > > Thank you for all your help > > > > On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com> wrote: > > Hi, Brandon > > Thanks for you recommends, I have done the changes. > > As the throughput value of nic_single_core is proportional to the cpu > frequency. > > I recommend you can change the baseline according to our report system. > > > > On the our 2.50GHz system, the baseline value as below: > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 > > 64 > > 2048 > > 41.439 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 > > 64 > > 2048 > > 47.73 > > > > For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number > should be > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 / 2.5 * 2.1=44.152 > > 64 > > 2048 > > 41.439 / 2.5 * 2.1=34.809 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 / 2.5 * 2.1=50.071 > > 64 > > 2048 > > 47.73 / 2.5 * 2.1=40.093 > > > > > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Tuesday, March 31, 2020 9:42 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > To make changes to either Intel machine, please reboot using the command > "reboot_to_rw" as root to reboot the machine into read/write mode. > > This command will also disable any testing on the machine. > > > > To re-enable the machine, please run "reboot_to_ro" as root, and it will > save all of the changes that you've made and re-enable testing on the > machine. > > I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" > instead of the normal "reboot" while you're making changes. > > > > After you're done, please let me know. I'll have to manually run a test > and update the baseline using our internal CI. > > > > Thank you > > > > On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Please let me know how to make change to this reset machine. > (ip/access...) and disable it. > > > > After that please help to change the baseline. > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Thursday, March 26, 2020 11:39 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, we have a system in place that resets any changes made while > testing is enabled for a machine. > > If you would like, I can disable testing and allow you to make permanent > changes. > > > > I can also reset the baseline of Intel 10G test performance once you make > these changes. > > Please let me know if you would like to make permanent changes on the > Intel 10G so I can disable it for you. > > > > Thanks > > > > On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Thanks. Brandon. > > > > That’s good. We have made changed on 10G testbed. > > > > I monitored the several execution results; I found the results of 10G > always has -0.9%~-1.x% gap against expected number. So it could lead to see > sometime failures..+-1% I suggest adjusting the expected number. I don’t > know where the expected number is from? as I know it a dynamic number? > depends on baseline.. Please help to clarify, thanks. > > > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 24, 2020 9:31 PM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > I have enabled the 10G Intel machine for testing. > > If you would like to make any more changes, please let me know so I can > perform the necessary steps to prepare the machine for changes. > > Please feel free to let me know if you need anything. > > > > Thank you > > > > On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > For 10G, please enable it. our code is at original path > */opt/test-harness/dts.* > > > > For 40G, please keep running. and see if any issue. But, anyway, we have > modified the DTS code at /opt/test-harness/dts-new-suite. If we met same > problem, then use this new DTS instead. > > > > Thanks a lot > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Saturday, March 21, 2020 1:49 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, > XuemingX <xuemingx.zhang@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, the 40G machine is stable enough to be put on production > dashboard to run tests which may cause Trex to be killed. > > Should I disable the 40G Intel machine for you to make changes? > > > > Also, just for confirmation: on the 10G machine, is the folder that you > are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r > are you still using the one in the standard */opt/test-harness/dts* > folder? > > > > If everything is ok, I will enable the 10G machine for production testing. > > > > Thank you very much > > > > On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Brandon, > > > > We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could > you please help to recover them? > > > > But, for FVL(40G) testbed, we met some problems, could you please help to > check before recover it > > - Sometime 1G hugepage will be changed to 2Mhugepage > automatically...we have to restart the system > - When we debugging on the testbed, found that Trex was killed by some > one(app).. > > Please help to check if any other program running on the testbed. > > > > Thanks a lot. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Sent:* Wednesday, March 18, 2020 9:04 PM > *To:* Brandon Lo <blo@iol.unh.edu> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Brandon, we almost made a workaround. > > > > Maybe tomorrow, you could recover Intel’s testbed. I will let you know > soon. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Wednesday, March 18, 2020 3:34 AM > *To:* Chen, Zhaoyan <zhaoyan.chen@intel.com> > *Cc:* David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Have you finished making changes on the Intel machine? > > I will turn on the machine on March 3rd for testing if you do not have any > issues with it. > > Please let me know if you need anything else. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com> > wrote: > > Hi, Brandon, > > > > Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex > upgrading. > > So we are reviewing our DTS script, different Trex version, and CI calling > procedure. > > > > Anyway, we are focusing on this task recently, any update will let you > know. > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo <blo@iol.unh.edu> > *Sent:* Tuesday, March 10, 2020 10:46 PM > *To:* David Marchand <david.marchand@redhat.com> > *Cc:* Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.edu; > Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, > Qian Q <qian.q.xu@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > How is the current status of the Intel 82599ES? > > Were there any configuration changes made to fix performance issues? > > > > Thanks > > > > On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wrote: > > Hi David, > > > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > > I'll rerun the tests that were affected and keep an eye out to see if it's > stable enough to be put back online. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com> > wrote: > > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header). > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu [-- Attachment #1.2: Type: text/html, Size: 80881 bytes --] [-- Attachment #2: image001.png --] [-- Type: image/png, Size: 4767 bytes --]
[-- Attachment #1.1: Type: text/plain, Size: 21397 bytes --] Thanks Brandon Now the baseline values are corrected both on NNT and FVL. Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu] Sent: Friday, May 15, 2020 12:13 AM To: Ma, LihongX <lihongx.ma@intel.com> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com>; Lin, Xueqin <xueqin.lin@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I've applied the same fix to the Intel 10G machine. It should be pulling the correct baseline values from now on. Thanks, Brandon On Tue, May 12, 2020 at 1:41 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon Thanks for your help to check the issues of baseline, and I find the FVL env is ok, the expected value has been changed to the value we wanted. But the baseline of NNT has changed, can you help to check it ? Thanks Regards, Ma,lihong From: Ma, LihongX Sent: Friday, May 8, 2020 4:02 PM To: 'Brandon Lo' <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>>; Lin, Xueqin <xueqin.lin@intel.com<mailto:xueqin.lin@intel.com>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Thanks Brandon, wait your reply. Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu] Sent: Wednesday, May 6, 2020 9:05 PM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>>; Lin, Xueqin <xueqin.lin@intel.com<mailto:xueqin.lin@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, Just a further update: we have noticed that there is another internal script that is used to calculate baselines that will pull a newer baseline if it is found. We are looking to solve the issues that we are having with baselines and will get back to you. Thanks for your patience, Brandon On Thu, Apr 30, 2020 at 10:19 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi Lihong, The expected value was reset by one of our internal scripts. I believe that I have resolved this issue for the future by ensuring that the baseline that you sent me will not be overwritten automatically. I will continue to monitor this expected throughput in case of any issues. Thanks for your patience, Brandon On Wed, Apr 29, 2020 at 1:30 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon I checked the new result of FVL, find the expected value also not changed. From the log find the expected value also is: [cid:image001.png@01D62A9B.7B8F1380] Can you help to double check it ? Is there any different between FVL and NNT ? Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Wednesday, April 29, 2020 12:52 AM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>>; Lin, Xueqin <xueqin.lin@intel.com<mailto:xueqin.lin@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, Sorry about that, I have reset the baseline to the values you sent in the previous email. I'll look to rerun tests that have failed due to the incorrect baseline. Thanks for letting me know, Brandon On Mon, Apr 27, 2020 at 11:39 PM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon I find the baseline of NNT have changed as expected, but FVL still same as before. Can you help to check it and change the baseline as expected ? Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Friday, April 3, 2020 2:39 AM To: Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Lihong, I have changed the baselines to reflect the new expected values. The performance tests should work as expected and pass. We will email again in the future if we come across any problems. Feel free to email us as well if you would like to make any other changes. Thank you for all your help On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>> wrote: Hi, Brandon Thanks for you recommends, I have done the changes. As the throughput value of nic_single_core is proportional to the cpu frequency. I recommend you can change the baseline according to our report system. On the our 2.50GHz system, the baseline value as below: NNT: pkt_size trd/rxd expected_value 64 512 52.562 64 2048 41.439 FVL: pkt_size trd/rxd expected_value 64 512 59.608 64 2048 47.73 For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number should be NNT: pkt_size trd/rxd expected_value 64 512 52.562 / 2.5 * 2.1=44.152 64 2048 41.439 / 2.5 * 2.1=34.809 FVL: pkt_size trd/rxd expected_value 64 512 59.608 / 2.5 * 2.1=50.071 64 2048 47.73 / 2.5 * 2.1=40.093 Regards, Ma,lihong From: Brandon Lo [mailto:blo@iol.unh.edu<mailto:blo@iol.unh.edu>] Sent: Tuesday, March 31, 2020 9:42 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, To make changes to either Intel machine, please reboot using the command "reboot_to_rw" as root to reboot the machine into read/write mode. This command will also disable any testing on the machine. To re-enable the machine, please run "reboot_to_ro" as root, and it will save all of the changes that you've made and re-enable testing on the machine. I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" instead of the normal "reboot" while you're making changes. After you're done, please let me know. I'll have to manually run a test and update the baseline using our internal CI. Thank you On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Please let me know how to make change to this reset machine. (ip/access...) and disable it. After that please help to change the baseline. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Thursday, March 26, 2020 11:39 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>>; O'Driscoll, Tim <tim.odriscoll@intel.com<mailto:tim.odriscoll@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, we have a system in place that resets any changes made while testing is enabled for a machine. If you would like, I can disable testing and allow you to make permanent changes. I can also reset the baseline of Intel 10G test performance once you make these changes. Please let me know if you would like to make permanent changes on the Intel 10G so I can disable it for you. Thanks On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Thanks. Brandon. That’s good. We have made changed on 10G testbed. I monitored the several execution results; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjusting the expected number. I don’t know where the expected number is from? as I know it a dynamic number? depends on baseline.. Please help to clarify, thanks. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 24, 2020 9:31 PM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, I have enabled the 10G Intel machine for testing. If you would like to make any more changes, please let me know so I can perform the necessary steps to prepare the machine for changes. Please feel free to let me know if you need anything. Thank you On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, For 10G, please enable it. our code is at original path /opt/test-harness/dts. For 40G, please keep running. and see if any issue. But, anyway, we have modified the DTS code at /opt/test-harness/dts-new-suite. If we met same problem, then use this new DTS instead. Thanks a lot Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Saturday, March 21, 2020 1:49 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Ma, LihongX <lihongx.ma@intel.com<mailto:lihongx.ma@intel.com>>; Zhang, XuemingX <xuemingx.zhang@intel.com<mailto:xuemingx.zhang@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Currently, the 40G machine is stable enough to be put on production dashboard to run tests which may cause Trex to be killed. Should I disable the 40G Intel machine for you to make changes? Also, just for confirmation: on the 10G machine, is the folder that you are using for the testing located in /opt/test-harness/dts-2020-3-4, or are you still using the one in the standard /opt/test-harness/dts folder? If everything is ok, I will enable the 10G machine for production testing. Thank you very much On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Brandon, We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could you please help to recover them? But, for FVL(40G) testbed, we met some problems, could you please help to check before recover it * Sometime 1G hugepage will be changed to 2Mhugepage automatically...we have to restart the system * When we debugging on the testbed, found that Trex was killed by some one(app).. Please help to check if any other program running on the testbed. Thanks a lot. Regards, Zhaoyan Chen From: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Sent: Wednesday, March 18, 2020 9:04 PM To: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>>; Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Brandon, we almost made a workaround. Maybe tomorrow, you could recover Intel’s testbed. I will let you know soon. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Wednesday, March 18, 2020 3:34 AM To: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> Cc: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, Have you finished making changes on the Intel machine? I will turn on the machine on March 3rd for testing if you do not have any issues with it. Please let me know if you need anything else. Thanks On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>> wrote: Hi, Brandon, Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex upgrading. So we are reviewing our DTS script, different Trex version, and CI calling procedure. Anyway, we are focusing on this task recently, any update will let you know. Thanks. Regards, Zhaoyan Chen From: Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> Sent: Tuesday, March 10, 2020 10:46 PM To: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com<mailto:zhaoyan.chen@intel.com>>; dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>; Lincoln Lavoie <lylavoie@iol.unh.edu<mailto:lylavoie@iol.unh.edu>>; Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>; ci@dpdk.org<mailto:ci@dpdk.org>; Tu, Lijuan <lijuan.tu@intel.com<mailto:lijuan.tu@intel.com>>; Xu, Qian Q <qian.q.xu@intel.com<mailto:qian.q.xu@intel.com>> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing Hi Zhaoyan, How is the current status of the Intel 82599ES? Were there any configuration changes made to fix performance issues? Thanks On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: Hi David, This was just a weird issue with the packet generator not cleaning itself after a test fast enough before another test. I'll rerun the tests that were affected and keep an eye out to see if it's stable enough to be put back online. Thanks On Tue, Mar 10, 2020 at 5:33 AM David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo <blo@iol.unh.edu<mailto:blo@iol.unh.edu>> wrote: > > Hi David and Zhaoyan, > > > Yes, those results are related to the Intel machine; I have disabled testing for the Intel testbed. > > The 82599ES machine is now available for ssh and modifications. Any news about this? I received a failure on a patch of mine (changing macros in a ARM header). https://lab.dpdk.org/results/dashboard/patchsets/9900/ But this time, it is with the 40G Intel nic test. -- David Marchand -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu<mailto:blo@iol.unh.edu> www.iol.unh.edu<http://www.iol.unh.edu/> [-- Attachment #1.2: Type: text/html, Size: 105970 bytes --] [-- Attachment #2: image001.png --] [-- Type: image/png, Size: 4767 bytes --]