Hi Lihong, I saw your changes and applied them manually. This involves me taking the machine down for maintenance on our backend, and then doing a normal `reboot`. Now when I do `cat /proc/cmdline` it shows: `BOOT_IMAGE=/boot/vmlinuz-4.15.0-55-generic root=UUID=3801030b-237d-428e-9e67-d81e12f16308 ro quiet splash hugepagesz=1G hugepages=40 default_hugepagesz=1G isolcpus=1-21,49-69,24-37,72-85 intel_iommu=on iommu=pt nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85 intel_pstate=disable nmi_watchdog=0 audit=0 nosoftlockup processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable mce=off tsc=reliable numa_balancing=disable vt.handoff=1` Which is what I think we wanted to achieve. I have now done the `reboot_to_ro` command and it should be working fine for testing. Thanks, Brandon On Wed, Apr 15, 2020 at 11:40 PM Ma, LihongX wrote: > Hi, Brandon > > > > I use the cmd ‘reboot_to_rw’ to restart server, and edit the file > ‘/etc/default/grub’ to reset the isolation cpus, > > > > Changes the field ‘isolcpus=1-48 nohz_full=1-48 rcu_nocbs=1-48’ to > > ‘isolcpus=1-21,49-69,24-37,72-85 nohz_full=1-21,49-69,24-37,72-85 > rcu_nocbs=1-21,49-69,24-37,72-85’ > > > > Detail as below: > > GRUB_CMDLINE_LINUX_DEFAULT="quiet splash hugepagesz=1G hugepages=40 > default_hugepagesz=1G isolcpus=1-21,49-69,24-37,72-85 intel_iommu=on > iommu=pt nohz_full=1-21,49-69,24-37,72-85 rcu_nocbs=1-21,49-69,24-37,72-85 > intel_pstate=disable nmi_watchdog=0 audit=0 nosoftlockup > processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable mce=off > tsc=reliable numa_balancing=disable" > > > > Then use cmd: update-grub to make the configuration effective. > > Then reboot server use ‘reboot_to_ro’, but I can’t find the change in > cmdline ( cat /proc/cmdline). > > > > Can you help me to check it? > > > > Thanks > > Lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Thursday, April 16, 2020 3:31 AM > *To:* Ma, LihongX > *Cc:* Chen, Zhaoyan ; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon ; ci@dpdk.org; > Tu, Lijuan ; Xu, Qian Q ; > Zhang, XuemingX ; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > I think any time is fine to make these changes. > > > > Please use the 'reboot_to_rw' command once you have logged on, this will > make sure your changes are saved. > > It will also disable testing momentarily while you make your changes. > > > > Once you're finished, you can reboot it with 'reboot_to_ro'. > > Please let me know when you're done so I can make sure everything is > working as intended. > > We could also do this change for you if needed. > > > > Thanks, > > Brandon > > > > On Wed, Apr 15, 2020 at 5:39 AM Ma, LihongX wrote: > > Hi, Brandon > > > > I find the grub configuration of isolation cpu miss the logic cores which > at some thread. > > > > For example: > > If the server cpus layout as below, and want to config the isolation cpu > from 1-20 > > The config of isolation should is ‘isolcpus=1-20,49-68’ > > > > [image: cid:1717f4fa8006917eb1] > > > > So, I want to change the grub configuration of the isolation cpus. > > From ‘isolcpus=1-48’ change to ‘isolcpus=1-21,49-68,24-37,72-85’ > > > > Can you help me check which time is suite to do this change ? > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Friday, April 3, 2020 2:39 AM > *To:* Ma, LihongX > *Cc:* Chen, Zhaoyan ; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon ; ci@dpdk.org; > Tu, Lijuan ; Xu, Qian Q ; > Zhang, XuemingX ; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > I have changed the baselines to reflect the new expected values. > > The performance tests should work as expected and pass. > > > > We will email again in the future if we come across any problems. > > Feel free to email us as well if you would like to make any other changes. > > > > Thank you for all your help > > > > On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX wrote: > > Hi, Brandon > > Thanks for you recommends, I have done the changes. > > As the throughput value of nic_single_core is proportional to the cpu > frequency. > > I recommend you can change the baseline according to our report system. > > > > On the our 2.50GHz system, the baseline value as below: > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 > > 64 > > 2048 > > 41.439 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 > > 64 > > 2048 > > 47.73 > > > > For the testbed in UNH, it’s a 2.1Ghz CPU server, so the expected number > should be > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 / 2.5 * 2.1=44.152 > > 64 > > 2048 > > 41.439 / 2.5 * 2.1=34.809 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 / 2.5 * 2.1=50.071 > > 64 > > 2048 > > 47.73 / 2.5 * 2.1=40.093 > > > > > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Tuesday, March 31, 2020 9:42 PM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Ma, LihongX ; Zhang, > XuemingX ; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > To make changes to either Intel machine, please reboot using the command > "reboot_to_rw" as root to reboot the machine into read/write mode. > > This command will also disable any testing on the machine. > > > > To re-enable the machine, please run "reboot_to_ro" as root, and it will > save all of the changes that you've made and re-enable testing on the > machine. > > I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" > instead of the normal "reboot" while you're making changes. > > > > After you're done, please let me know. I'll have to manually run a test > and update the baseline using our internal CI. > > > > Thank you > > > > On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan > wrote: > > Hi, Brandon, > > > > Please let me know how to make change to this reset machine. > (ip/access...) and disable it. > > > > After that please help to change the baseline. > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Thursday, March 26, 2020 11:39 PM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Ma, LihongX ; Zhang, > XuemingX ; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, we have a system in place that resets any changes made while > testing is enabled for a machine. > > If you would like, I can disable testing and allow you to make permanent > changes. > > > > I can also reset the baseline of Intel 10G test performance once you make > these changes. > > Please let me know if you would like to make permanent changes on the > Intel 10G so I can disable it for you. > > > > Thanks > > > > On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan > wrote: > > Thanks. Brandon. > > > > That’s good. We have made changed on 10G testbed. > > > > I monitored the several execution results; I found the results of 10G > always has -0.9%~-1.x% gap against expected number. So it could lead to see > sometime failures..+-1% I suggest adjusting the expected number. I don’t > know where the expected number is from? as I know it a dynamic number? > depends on baseline.. Please help to clarify, thanks. > > > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Tuesday, March 24, 2020 9:31 PM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Ma, LihongX ; Zhang, > XuemingX > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > I have enabled the 10G Intel machine for testing. > > If you would like to make any more changes, please let me know so I can > perform the necessary steps to prepare the machine for changes. > > Please feel free to let me know if you need anything. > > > > Thank you > > > > On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan > wrote: > > Hi, Brandon, > > > > For 10G, please enable it. our code is at original path > */opt/test-harness/dts.* > > > > For 40G, please keep running. and see if any issue. But, anyway, we have > modified the DTS code at /opt/test-harness/dts-new-suite. If we met same > problem, then use this new DTS instead. > > > > Thanks a lot > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Saturday, March 21, 2020 1:49 AM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Ma, LihongX ; Zhang, > XuemingX > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, the 40G machine is stable enough to be put on production > dashboard to run tests which may cause Trex to be killed. > > Should I disable the 40G Intel machine for you to make changes? > > > > Also, just for confirmation: on the 10G machine, is the folder that you > are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r > are you still using the one in the standard */opt/test-harness/dts* > folder? > > > > If everything is ok, I will enable the 10G machine for production testing. > > > > Thank you very much > > > > On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan > wrote: > > Brandon, > > > > We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Could > you please help to recover them? > > > > But, for FVL(40G) testbed, we met some problems, could you please help to > check before recover it > > - Sometime 1G hugepage will be changed to 2Mhugepage > automatically...we have to restart the system > - When we debugging on the testbed, found that Trex was killed by some > one(app).. > > Please help to check if any other program running on the testbed. > > > > Thanks a lot. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Chen, Zhaoyan > *Sent:* Wednesday, March 18, 2020 9:04 PM > *To:* Brandon Lo > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Chen, Zhaoyan > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Brandon, we almost made a workaround. > > > > Maybe tomorrow, you could recover Intel’s testbed. I will let you know > soon. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Wednesday, March 18, 2020 3:34 AM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Have you finished making changes on the Intel machine? > > I will turn on the machine on March 3rd for testing if you do not have any > issues with it. > > Please let me know if you need anything else. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan > wrote: > > Hi, Brandon, > > > > Yes, it’s a wired issue. And it also mixed our DTS upgrading and Trex > upgrading. > > So we are reviewing our DTS script, different Trex version, and CI calling > procedure. > > > > Anyway, we are focusing on this task recently, any update will let you > know. > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Tuesday, March 10, 2020 10:46 PM > *To:* David Marchand > *Cc:* Chen, Zhaoyan ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > How is the current status of the Intel 82599ES? > > Were there any configuration changes made to fix performance issues? > > > > Thanks > > > > On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo wrote: > > Hi David, > > > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > > I'll rerun the tests that were affected and keep an eye out to see if it's > stable enough to be put back online. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand > wrote: > > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header). > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu