From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id E8401A0352 for ; Thu, 14 May 2020 18:13:36 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id C5B931D9A9; Thu, 14 May 2020 18:13:36 +0200 (CEST) Received: from mail-io1-f44.google.com (mail-io1-f44.google.com [209.85.166.44]) by dpdk.org (Postfix) with ESMTP id 8F91E1BE9E for ; Thu, 14 May 2020 18:13:34 +0200 (CEST) Received: by mail-io1-f44.google.com with SMTP id e18so3000833iog.9 for ; Thu, 14 May 2020 09:13:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=uoHBASHc5HOJP+u6qhDV6ZD9kW300eIAqZCwAHyazKM=; b=Id1qNjZWPunBe6qALr1cpoYBMXhz1zMHqP2TJVFLkngcQQMMZbZGZdh7jiyivAtRYL N9cvudzvTAo4HrGN1U1LJw3jdgPs7tMkTef5x8WS/Q68bnuDhfB/bjsXpE0y5efrcWn5 YcQKdzm88OTkkjzJ3UnTY6W5+SXZul8fwQiF4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=uoHBASHc5HOJP+u6qhDV6ZD9kW300eIAqZCwAHyazKM=; b=rkRHkjB4pFfhFBtNx+V0p/R83hxEzfyXWuo0UthIAsYAudXt+3jQqsQMvWqrv0dgII biogVwYPEGtEOUAEVWsPU1d1+/w6/KwhGUlLKNuDokZh9Hg+auFgWg9x6H0Khoij+BF8 GD8cGLQ+I49QfbVn8M4RxK30dd5zv9lPs4YRAaGKO0iAUkC0StbNJLDyI55IbWQy75dw Q3Od4q04g7kx+Zv9Old3ZsUy1kl2Y5cNJ66joHE0Y6NV4qlyGQQs4BAljTaNOcF8aUyw 6a7sF8/xROK59YzzXEwi/Ja6/xtGvq8X6vZN5rynx7Sb1Y5+HnVyHClRnExBtXF5Qzh8 UBEQ== X-Gm-Message-State: AOAM533HY4I9idZhnW0nHa69v3KymmQNz98mkKlk/wjaiQ4vZmp8PdTg DPv8ZZ9uiJvDmPr1jTYZPxJYdYMvpfm8cnxVzMJJBg== X-Google-Smtp-Source: ABdhPJyuW8wFqeZnTdJC1mwk3pJwu0TBpwYrUqEF556ZeZPnfnPqQAKxUgcI4ImA+1XIwNkzK203NP8Dk0koiKPhSQ8= X-Received: by 2002:a6b:5813:: with SMTP id m19mr4678809iob.88.1589472813691; Thu, 14 May 2020 09:13:33 -0700 (PDT) MIME-Version: 1.0 References: <1827408.xz2uEaWSZ7@xps> <9DEEADBC57E43F4DA73B571777FECECA41EAB066@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41EBB4AF@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41EFB9C5@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41F01113@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41F025A1@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41F03B43@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41F058DF@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41F18C47@SHSMSX104.ccr.corp.intel.com> <08e9c34dce60404abb862f35dabd4198@intel.com> <539a8463a02848e399f2340e4f5be692@intel.com> In-Reply-To: From: Brandon Lo Date: Thu, 14 May 2020 12:12:56 -0400 Message-ID: To: "Ma, LihongX" Cc: "Chen, Zhaoyan" , David Marchand , "dpdklab@iol.unh.edu" , Lincoln Lavoie , Thomas Monjalon , "ci@dpdk.org" , "Tu, Lijuan" , "Xu, Qian Q" , "Zhang, XuemingX" , "O'Driscoll, Tim" , "Lin, Xueqin" Content-Type: multipart/related; boundary="00000000000070f99c05a59dface" Subject: Re: [dpdk-ci] [dpdklab] Re: Intel performance test is failing X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" --00000000000070f99c05a59dface Content-Type: multipart/alternative; boundary="00000000000070f99b05a59dfacd" --00000000000070f99b05a59dfacd Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Lihong, I've applied the same fix to the Intel 10G machine. It should be pulling the correct baseline values from now on. Thanks, Brandon On Tue, May 12, 2020 at 1:41 AM Ma, LihongX wrote: > Hi, Brandon > > > > Thanks for your help to check the issues of baseline, and I find the FVL > env is ok, the expected value has been changed to the value we wanted. > > But the baseline of NNT has changed, can you help to check it ? > > > > Thanks > > Regards, > > Ma,lihong > > > > *From:* Ma, LihongX > *Sent:* Friday, May 8, 2020 4:02 PM > *To:* 'Brandon Lo' > *Cc:* Chen, Zhaoyan ; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon ; ci@dpdk.org= ; > Tu, Lijuan ; Xu, Qian Q ; > Zhang, XuemingX ; O'Driscoll, Tim < > tim.odriscoll@intel.com>; Lin, Xueqin > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Thanks Brandon, wait your reply. > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu ] > *Sent:* Wednesday, May 6, 2020 9:05 PM > *To:* Ma, LihongX > *Cc:* Chen, Zhaoyan ; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon ; ci@dpdk.org= ; > Tu, Lijuan ; Xu, Qian Q ; > Zhang, XuemingX ; O'Driscoll, Tim < > tim.odriscoll@intel.com>; Lin, Xueqin > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > Just a further update: we have noticed that there is another internal > script that is used to calculate baselines that will pull a newer baselin= e > if it is found. > > We are looking to solve the issues that we are having with baselines and > will get back to you. > > > > Thanks for your patience, > > Brandon > > > > On Thu, Apr 30, 2020 at 10:19 AM Brandon Lo wrote: > > Hi Lihong, > > > > The expected value was reset by one of our internal scripts. > > I believe that I have resolved this issue for the future by ensuring that > the baseline that you sent me will not be overwritten automatically. > > > > I will continue to monitor this expected throughput in case of any issues= . > > > > Thanks for your patience, > > Brandon > > > > On Wed, Apr 29, 2020 at 1:30 AM Ma, LihongX wrote: > > Hi, Brandon > > > > I checked the new result of FVL, find the expected value also not changed= . > > From the log find the expected value also is: > > > > Can you help to double check it ? Is there any different between FVL and > NNT ? > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Wednesday, April 29, 2020 12:52 AM > *To:* Ma, LihongX > *Cc:* Chen, Zhaoyan ; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon ; ci@dpdk.org= ; > Tu, Lijuan ; Xu, Qian Q ; > Zhang, XuemingX ; O'Driscoll, Tim < > tim.odriscoll@intel.com>; Lin, Xueqin > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > Sorry about that, I have reset the baseline to the values you sent in the > previous email. > > I'll look to rerun tests that have failed due to the incorrect baseline. > > > > Thanks for letting me know, > > Brandon > > > > On Mon, Apr 27, 2020 at 11:39 PM Ma, LihongX wrote= : > > Hi, Brandon > > I find the baseline of NNT have changed as expected, but FVL still same a= s > before. > > Can you help to check it and change the baseline as expected ? > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Friday, April 3, 2020 2:39 AM > *To:* Ma, LihongX > *Cc:* Chen, Zhaoyan ; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon ; ci@dpdk.org= ; > Tu, Lijuan ; Xu, Qian Q ; > Zhang, XuemingX ; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > I have changed the baselines to reflect the new expected values. > > The performance tests should work as expected and pass. > > > > We will email again in the future if we come across any problems. > > Feel free to email us as well if you would like to make any other changes= . > > > > Thank you for all your help > > > > On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX wrote: > > Hi, Brandon > > Thanks for you recommends, I have done the changes. > > As the throughput value of nic_single_core is proportional to the cpu > frequency. > > I recommend you can change the baseline according to our report system. > > > > On the our 2.50GHz system, the baseline value as below: > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 > > 64 > > 2048 > > 41.439 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 > > 64 > > 2048 > > 47.73 > > > > For the testbed in UNH, it=E2=80=99s a 2.1Ghz CPU server, so the expected= number > should be > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 / 2.5 * 2.1=3D44.152 > > 64 > > 2048 > > 41.439 / 2.5 * 2.1=3D34.809 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 / 2.5 * 2.1=3D50.071 > > 64 > > 2048 > > 47.73 / 2.5 * 2.1=3D40.093 > > > > > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Tuesday, March 31, 2020 9:42 PM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Ma, LihongX ; Zhang, > XuemingX ; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > To make changes to either Intel machine, please reboot using the command > "reboot_to_rw" as root to reboot the machine into read/write mode. > > This command will also disable any testing on the machine. > > > > To re-enable the machine, please run "reboot_to_ro" as root, and it will > save all of the changes that you've made and re-enable testing on the > machine. > > I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" > instead of the normal "reboot" while you're making changes. > > > > After you're done, please let me know. I'll have to manually run a test > and update the baseline using our internal CI. > > > > Thank you > > > > On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan > wrote: > > Hi, Brandon, > > > > Please let me know how to make change to this reset machine. > (ip/access...) and disable it. > > > > After that please help to change the baseline. > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Thursday, March 26, 2020 11:39 PM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Ma, LihongX ; Zhang, > XuemingX ; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, we have a system in place that resets any changes made while > testing is enabled for a machine. > > If you would like, I can disable testing and allow you to make permanent > changes. > > > > I can also reset the baseline of Intel 10G test performance once you make > these changes. > > Please let me know if you would like to make permanent changes on the > Intel 10G so I can disable it for you. > > > > Thanks > > > > On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan > wrote: > > Thanks. Brandon. > > > > That=E2=80=99s good. We have made changed on 10G testbed. > > > > I monitored the several execution results; I found the results of 10G > always has -0.9%~-1.x% gap against expected number. So it could lead to s= ee > sometime failures..+-1% I suggest adjusting the expected number. I don=E2= =80=99t > know where the expected number is from? as I know it a dynamic number? > depends on baseline.. Please help to clarify, thanks. > > > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Tuesday, March 24, 2020 9:31 PM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Ma, LihongX ; Zhang, > XuemingX > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > I have enabled the 10G Intel machine for testing. > > If you would like to make any more changes, please let me know so I can > perform the necessary steps to prepare the machine for changes. > > Please feel free to let me know if you need anything. > > > > Thank you > > > > On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan > wrote: > > Hi, Brandon, > > > > For 10G, please enable it. our code is at original path > */opt/test-harness/dts.* > > > > For 40G, please keep running. and see if any issue. But, anyway, we have > modified the DTS code at /opt/test-harness/dts-new-suite. If we met same > problem, then use this new DTS instead. > > > > Thanks a lot > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Saturday, March 21, 2020 1:49 AM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Ma, LihongX ; Zhang, > XuemingX > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, the 40G machine is stable enough to be put on production > dashboard to run tests which may cause Trex to be killed. > > Should I disable the 40G Intel machine for you to make changes? > > > > Also, just for confirmation: on the 10G machine, is the folder that you > are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r > are you still using the one in the standard */opt/test-harness/dts* > folder? > > > > If everything is ok, I will enable the 10G machine for production testing= . > > > > Thank you very much > > > > On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan > wrote: > > Brandon, > > > > We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Coul= d > you please help to recover them? > > > > But, for FVL(40G) testbed, we met some problems, could you please help t= o > check before recover it > > - Sometime 1G hugepage will be changed to 2Mhugepage > automatically...we have to restart the system > - When we debugging on the testbed, found that Trex was killed by some > one(app).. > > Please help to check if any other program running on the testbed. > > > > Thanks a lot. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Chen, Zhaoyan > *Sent:* Wednesday, March 18, 2020 9:04 PM > *To:* Brandon Lo > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Chen, Zhaoyan > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Brandon, we almost made a workaround. > > > > Maybe tomorrow, you could recover Intel=E2=80=99s testbed. I will let you= know > soon. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Wednesday, March 18, 2020 3:34 AM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Have you finished making changes on the Intel machine? > > I will turn on the machine on March 3rd for testing if you do not have an= y > issues with it. > > Please let me know if you need anything else. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan > wrote: > > Hi, Brandon, > > > > Yes, it=E2=80=99s a wired issue. And it also mixed our DTS upgrading and = Trex > upgrading. > > So we are reviewing our DTS script, different Trex version, and CI callin= g > procedure. > > > > Anyway, we are focusing on this task recently, any update will let you > know. > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Tuesday, March 10, 2020 10:46 PM > *To:* David Marchand > *Cc:* Chen, Zhaoyan ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > How is the current status of the Intel 82599ES? > > Were there any configuration changes made to fix performance issues? > > > > Thanks > > > > On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo wrote: > > Hi David, > > > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > > I'll rerun the tests that were affected and keep an eye out to see if it'= s > stable enough to be put back online. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand > wrote: > > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header)= . > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > --=20 Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu --00000000000070f99b05a59dfacd Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Lihong,

I've applied the same fi= x to the Intel 10G machine.
It should be pulling the correct base= line values from now on.

Thanks,
Brandon=

On Tue, May 12, 2020 at 1:41 AM Ma, LihongX <lihongx.ma@intel.com> wrote:

Hi, Brandon

=C2=A0

Thanks for your help to check the issues of baseline, and I= find the FVL env is ok, the expected value has been changed to the value w= e wanted.

But the baseline of NNT has changed, can you help to check = it ?

=C2=A0

Thanks

Regards,

Ma,lihong

=C2=A0

From: Ma, LihongX
Sent: Friday, May 8, 2020 4:02 PM
To: 'Brandon Lo' <blo@iol.unh.edu>
Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.= com>; dpdkl= ab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; T= u, Lijuan <liju= an.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.c= om>; Lin, Xueqin <xueqin.lin@intel.com>
Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Thanks Brandon, wait your reply.

=C2=A0

Regards,

Ma,lihong

=C2=A0

From: Brandon Lo [mailto:blo@iol.unh.edu]
Sent: Wednesday, May 6, 2020 9:05 PM
To: Ma, LihongX <lihongx.ma@intel.com>
Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.= com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com<= /a>>; O'Driscoll, Tim <tim.odriscoll@intel.com>; Lin, Xueqin <xueqin.lin@intel.com> Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Lihong,

=C2=A0

Just a further update: we have noticed that there is= another internal script that is used to calculate baselines that will pull= a newer baseline if it is found.

We are looking to solve the issues that we are havin= g with baselines and will get back to you.

=C2=A0

Thanks for your patience,

Brandon

=C2=A0

On Thu, Apr 30, 2020 at 10:19 AM Brandon Lo <blo@iol.unh.edu> wr= ote:

Hi Lihong,

=C2=A0

The expected value was reset by one of our internal = scripts.

I believe that I have resolved this issue for the fu= ture by ensuring that the baseline that you sent me will not be overwritten= automatically.

=C2=A0

I will continue to monitor this expected throughput = in case of any issues.

=C2=A0

Thanks for your patience,

Brandon

=C2=A0

On Wed, Apr 29, 2020 at 1:30 AM Ma, LihongX <lihongx.ma@intel.com<= /a>> wrote:

Hi, Brandon

=C2=A0

I checked the new result of FVL, find the expected value al= so not changed.

From the log find the expected value also is:=

=C2=A0

Can you help to double check it ? Is there any different be= tween FVL and NNT ?

=C2=A0

Regards,

Ma,lihong

=C2=A0

From: Brandon Lo [mailto:blo@iol.unh.edu]
Sent: Wednesday, April 29, 2020 12:52 AM
To: Ma, LihongX <lihongx.ma@intel.com>
Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.= com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx= .zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com>; Lin, X= ueqin <xueqin.= lin@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Lihong,

=C2=A0

Sorry about that, I have reset the baseline to the v= alues you sent in the previous email.

I'll look to rerun tests that have failed due to= the incorrect baseline.

=C2=A0

Thanks for letting me know,

Brandon

=C2=A0

On Mon, Apr 27, 2020 at 11:39 PM Ma, LihongX <lihongx.ma@intel.com= > wrote:

Hi, Brandon

I find the baseline of NNT have changed as e= xpected, but FVL still same as before.

Can you help to check it and change the base= line as expected ?

=C2=A0

Regards,

Ma,lihong

=C2=A0

From: Brandon Lo [mailto:blo@iol.unh.edu]
Sent: Friday, April 3, 2020 2:39 AM
To: Ma, LihongX <lihongx.ma@intel.com>
Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.= com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx= .zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Lihong,

=C2=A0

I have changed the baselines to reflect the new expe= cted values.

The=C2=A0performance tests should work as expected a= nd=C2=A0pass.

=C2=A0

We will email again in the future if we come across = any problems.

Feel free to email us as well if you would like to m= ake any other changes.

=C2=A0

Thank you for all your help

=C2=A0

On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com> wrote:

Hi, Brandon

Thanks for you recommends, I have done the c= hanges.

As the throughput =C2=A0value of nic_single_= core is proportional to the cpu frequency.

I recommend you can change the baseline acco= rding to our report system.

=C2=A0

On the our 2.50GHz system, the baseline valu= e as below:

NNT:

pkt_size

trd/rxd

expected_value

6= 4

5= 12

5= 2.562

6= 4

2= 048

4= 1.439

=C2=A0

FVL:

pkt_size

trd/rxd

expected_value

6= 4

5= 12

5= 9.608

6= 4

2= 048

4= 7.73

=C2=A0

For the testbed in UNH, it= =E2=80=99s a 2.1Ghz CPU server, so the expected number should be

NNT:<= /p>

pkt_size

trd/rxd

expected_value

6= 4

5= 12

5= 2.562 / 2.5 * 2.1=3D44.152

6= 4

2= 048

4= 1.439 / 2.5 * 2.1=3D34.809

=C2=A0

FVL:<= /p>

pkt_size

trd/rxd

expected_value

6= 4

5= 12

5= 9.608 / 2.5 * 2.1=3D50.071

6= 4

2= 048

4= 7.73 / 2.5 * 2.1=3D40.093

=C2=A0

=C2=A0

=C2=A0

Regards,

Ma,lihong

=C2=A0

From: Brandon Lo [mailto:blo@iol.unh.edu]
Sent: Tuesday, March 31, 2020 9:42 PM
To: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Linco= ln Lavoie <lylavoie@= iol.unh.edu>; Thomas Monjalon <t= homas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com= >; Xu, Qian Q <qian.q.x= u@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odrisco= ll@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Zhaoyan,

=C2=A0

To make changes to either Intel machine, please rebo= ot using the command "reboot_to_rw" as root to reboot the machine= into read/write mode.

This command will also disable any testing on the ma= chine.

=C2=A0

To re-enable the machine, please run "reboot_to= _ro" as root, and it will save all of the changes that you've made= and re-enable testing on the machine.

I recommend rebooting using either "reboot_to_r= w" or "reboot_to_ro" instead of the normal "reboot"= ; while you're making changes.

=C2=A0

After you're done, please let me know. I'll = have to manually run a test and update=C2=A0the baseline using our internal= CI.

=C2=A0

Thank you

=C2=A0

On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <<= a href=3D"mailto:zhaoyan.chen@intel.com" target=3D"_blank">zhaoyan.chen@int= el.com> wrote:

Hi, Brandon,

=C2=A0

Please let me know how to make change to = this reset machine. (ip/access...) and disable it.

=C2=A0

After that please help to change the base= line.

=C2=A0

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Brandon Lo <blo@iol.unh.edu>
Sent: Thursday, March 26, 2020 11:39 PM
To: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.c= om>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Zhaoyan,

=C2=A0

Currently, we have a system in place that resets any= changes made while testing is enabled for a machine.

If you would like, I can disable testing and allow y= ou to make permanent changes.

=C2=A0

I can also reset the baseline of Intel 10G test perf= ormance once you make these changes.

Please let me know if you would like to make permane= nt changes on the Intel 10G so I can disable it for you.

=C2=A0

Thanks

=C2=A0

On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <<= a href=3D"mailto:zhaoyan.chen@intel.com" target=3D"_blank">zhaoyan.chen@int= el.com> wrote:

Thanks. Brandon.

=C2=A0

That=E2=80=99s good. We have made changed= on 10G testbed.

=C2=A0

I monitored the several execution results= ; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjustin= g the expected number. I don=E2=80=99t know where the expected number is fr= om? as I know it a dynamic number? depends on baseline.. Please help to cla= rify, thanks.

=C2=A0

=C2=A0

Thanks.

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Brandon Lo <blo@iol.unh.edu>
Sent: Tuesday, March 24, 2020 9:31 PM
To: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Zhaoyan,

=C2=A0

I have enabled the 10G Intel machine for testing.=

If you would like to make any more changes, please l= et me know so I can perform the necessary steps to prepare the machine for = changes.

Please feel free to let me know if you need anything= .

=C2=A0

Thank you

=C2=A0

On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@inte= l.com> wrote:

Hi, Brandon,

=C2=A0

For 10G, please enable it. our code is at= original path /opt/test-harness/dts.

=C2=A0

For 40G, please keep running. and see if = any issue. But, anyway, we have modified the DTS code at /opt/test-harness/= dts-new-suite. If we met same problem, then use this new DTS instead.

=C2=A0

Thanks a lot=

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Brandon Lo <blo@iol.unh.edu>
Sent: Saturday, March 21, 2020 1:49 AM
To: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Zhaoyan,

=C2=A0

Currently, the 40G machine is stable enough to be pu= t on production dashboard to run tests which may cause Trex to be killed.

Should I disable the 40G Intel machine for you to ma= ke changes?

=C2=A0

Also, just for confirmation: on the 10G machine, is = the folder that you are using for the testing located in=C2=A0/opt/test-= harness/dts-2020-3-4, or are you still using the one in the standard=C2=A0/opt/test-harness/dts folder?

=C2=A0

If everything is ok, I will enable the 10G machine f= or production testing.

=C2=A0

Thank you very much

=C2=A0

On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@inte= l.com> wrote:

Brandon,

=C2=A0

We worked out a workaround on Intel testb= eds. NNT(10G) and FVL(40G). Could you please help to recover them?

=C2=A0

But, for FVL(40G) testbed, =C2=A0we met s= ome problems, could you please help to check before recover it

  • Sometime 1= G hugepage will be changed to 2Mhugepage automatically...we have to restart= the system
  • When we de= bugging on the testbed, found that Trex was killed by some one(app)..

Please help to check if any other program= running on the testbed.

=C2=A0

Thanks a lot.

=C2=A0

=C2=A0

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Sent: Wednesday, March 18, 2020 9:04 PM
To: Brandon Lo <blo@iol.unh.edu>
Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.ch= en@intel.com>
Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Brandon, we almost made a workaround.

=C2=A0

Maybe tomorrow, you could recover Intel= =E2=80=99s testbed. I will let you know soon.

=C2=A0

=C2=A0

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Brandon Lo <blo@iol.unh.edu>
Sent: Wednesday, March 18, 2020 3:34 AM
To: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Zhaoyan,

=C2=A0

Have you finished making changes on the Intel machin= e?

I will turn on the machine on March 3rd for testing = if you do not have any issues with it.

Please let me know if you need anything else.=

=C2=A0

Thanks

=C2=A0

On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <<= a href=3D"mailto:zhaoyan.chen@intel.com" target=3D"_blank">zhaoyan.chen@int= el.com> wrote:

Hi, Brandon,

=C2=A0

Yes, it=E2=80=99s a wired issue. And it a= lso mixed our DTS upgrading and Trex upgrading.

So we are reviewing our DTS script, diffe= rent Trex version, and CI calling procedure.

=C2=A0

Anyway, we are focusing on= this task recently, any update will let you know.

=C2=A0

Thanks.

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Brandon Lo <blo@iol.unh.edu>
Sent: Tuesday, March 10, 2020 10:46 PM
To: David Marchand <david.marchand@redhat.com>
Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Zhaoyan,

=C2=A0

How is the current status of the Intel 82599ES?

Were there any configuration changes made to fix per= formance issues?

=C2=A0

Thanks

=C2=A0

On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wro= te:

Hi David,

=C2=A0

This was just a weird issue with the packet generato= r not cleaning itself after a test fast enough before another test.<= u>

I'll rerun the tests that were affected and keep= an eye out to see if it's stable enough to be put back online.<= u>

=C2=A0

Thanks

=C2=A0

On Tue, Mar 10, 2020 at 5:33 AM David Marchand <<= a href=3D"mailto:david.marchand@redhat.com" target=3D"_blank">david.marchan= d@redhat.com> wrote:

On Tue, Mar 3, 2020 at = 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote:
>
> Hi David and Zhaoyan,
>
>
> Yes, those results are related to the Intel machine; I have disabled t= esting for the Intel testbed.
>
> The 82599ES machine is now available for ssh and modifications.

Any news about this?

I received a failure on a patch of mine (changing macros in a ARM header).<= br> https://lab.dpdk.org/results/dashboard/patchsets/9900/

But this time, it is with the 40G Intel nic test.

--
David Marchand


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

blo@iol.unh.edu

www.iol.unh.edu



--

Brandon Lo

UNH = InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

<= a href=3D"http://www.iol.unh.edu/" target=3D"_blank">www.iol.unh.edu

--00000000000070f99b05a59dfacd-- --00000000000070f99c05a59dface Content-Type: image/png; name="image001.png" Content-Disposition: inline; filename="image001.png" Content-Transfer-Encoding: base64 Content-ID: <17213f553974cff311> X-Attachment-Id: 17213f553974cff311 iVBORw0KGgoAAAANSUhEUgAAAQ8AAACSCAIAAADUwthAAAAAAXNSR0IArs4c6QAAEllJREFUeF7t Xc9LHEseL91l/wGdMa7uYU8j66gvPJ8nNzCgCO8FBHl7UJAlObi5zCUeksvDlbCQHJKLlxB4eQdh 3EMQBHmLOMFFPImPJOqAc9eXODPZy572tt/qnu6u7q6erunumXTbn2YOOvOtqm99vvWp+ta360fP n/44zPAAASCggECvggxEgAAQ4AiALWgHQEAVAbBFFSnIAQGwBW0ACKgi0Ltxf0JVFnJAIB0IDN8t SnmBsSUd9kcto0AAbIkCReSRDgTAlnTYGbWMAgGwJQoUkUc6EOh5u/2jVdP6UfHpPv1Ls5xHhX4p Amdba69O+C+QAT4OBBLdNqbury+PepDe4EXP2yffFV9/SEfXgFoCASUE+EiQPXTzAp6YEnwQAgKE ANiCZgAEVBEAW1SRghwQAFvQBoCAKgI92N+iChXkUo8AxpbUNwEAoIwA2KIMFQRTjwDYkvomAACU EQBblKGCYOoRAFtS3wQAgDICYIsyVBBMPQJgS+qbAABQRgBsUYYKgqlHAGxJfRMAAMoI/Oafq3/+ 17trZXkIyhGgNd7/+Nt3386N/a5yXP1vdCgNzf7w97/+Za7wh//8+5dfo8u2ozklUWc7INya0/9z 8yLo/pbJpY3FnAfmjfKLjZ2rjhokVOa072eu9vLJ7sdQuUgST6w8v3PdRt1JfmHMQ4nagaXh/OP1 gXJzE17UOncqvy7rHK1No97fclIqrq5tVhgZlf4QPttnncL/RuZLPUsTvXKd0d5DHUkCFk8MEcC8 JYZGgUoxReC3EelFTsXo+9XSMfvwatXatyzsda5u8l8F36OyTTs5DQH+6+Dj9ZmMoY72q0M3Gtwt AaZn6H7s7k1le5MtmH6XlcPog42ClVZ0e/RvVcqy7eTmCkeEJWO/1hruvISDENy+rlnr5k+G/i5J uwtt7qTnxRk/SbbXu8wh4kPyP9/SD3JwFuels5Ld1fRRt2l48/RsFb8J7MELzqKny87xyl/oh2OY DXH8XJg20KTwYYbVc9maxRCOZrZ5pIaWSmsNos00KG3GNuwtfqlbxUEGPx9XqSxuJKfCjbMMa2fe YjNfC0effhpnjSwzYOSITV8bx4kIuehWOBx4uDCgT3s4SszsVjQ0xF6G17QpKVjHMUdyOfGOVNq/ 9Ua2ZtvI7q+zkt15t+WnD1fdz6btMYWqfI+9cfMirCeWLTzYeL6+4T1bvdzdeHY+svF4Vk6VZi1y 7PylOJgcv16jVD/cHdR/H757x0YV+oomTi+OBhaXpiwcJlYW+2gaoB9Joz+UT7tzAJWyCE0ivEth 5jVlb89WMmmLKvTr1f5PB42xcemRvP0zDxfY1lrT0nx6aYzAk0u8A7INyOQIrJ3mH6xMtqHg1H0H wSiT7euM5IQgBZ197N6GWl0RDcsWY5bfanJvEob3N+KoYtWwcfrOGaG6fHfB8nld5Ot8/9mp62Ca q/3Teu4r09KTo2P1C3csjgjT1uCpUNbEtwUmUXj3sHMRjrOyNTi3bhjO8daQnhrP1c7P3Wl/Ofci nrScia9Gq3vOcOKHnw8krqOCzj527woF2igkLFuEoj5/9I4aE2HKbHqGHXk0XI+0mYw5dIwt0gjm /NA0ZuCWMf7c6mO1WhtV9xb1LSuSUjqTSePaIzA+mGXXn0IHzYeyA1Hq7W/3KEsLnVdUbKHhuDni k5dielCmevqoIrpkds37BodkVanXzXm8GV21B6ytcePy02eWzYYGhGfgW1YkpXQ5k481q2cJXvRV LdI32f52D65qB1JGxRaball7qzUdMMccRkjTP367OUSYXw7fHmGG57BTrnq46UIeJ5WzzMi8m3U0 030uTm98UFQoixwPJlGYJlcdsFBUWR6fVrOGZyvmKfc8PUv98L6SmzPmk4YUuabyk039lPexu1/y bv/eEbYIlaCAiW2u0iSMs/lWWf6BeGOGHkmz3LaTEgWCzVCBaSTKXBjHeCOeeVgUCUOhEi10Zos1 U3w2WyhY4QFOJyEfhbKoFjQ5FodQTWHWuXlLBO2C6lWbdlxLondkYlzEMY1xn997/Hr7umALDMw/ 5jGxQBr62Z0xX330cn1sGkg5d6KgEeRWK18Y00O9WqDT8I1c71vM2LwWSdxbLTHxIFrj5FlRY7fl JDNaW6GMyfKhPO2H3kpe3aiUZXsnwwuqNVeyyF4WedrLjaSos/hr83vxhVLz/YbXmdRufByn/bpf NJGejnqR/+x6l2J7qUWZ/MS+t45CVdNZax5KdlfQh6Pra1N1ynhFkIOyRb1kX0kDNdmrRt/EEIgF Al4Lq1opF2O7d+p9SyxsBSWAQFcQ6G3rXURXVEIhSUNgculRgZX3bs5NDTQvlfLiC59VqeJGJ63t pEJf+zq6tvdoJNTuX5gtqWhZqORNQaDTEeSbghPqAQRwfwvaABBQRwBjizpWkEw7AmBL2lsA6q+O ANiijhUk044A2JL2FoD6qyMAtqhjBcm0I9DrWJSadjxQfyDAd7YXpbzA2ILWAQRUEQBbVJGCHBCQ nezq2CLiBElbFMTEvStuGCEDfLzIFcu2YT9SQr4BYWg26DnI6GeAwM1FwGu7Ts/b7R+tWnvsNLy5 sKBmQKCJgGNZtA0XgxfwxExYYukhwOP1pHOk9oInhm4TCARDIOobKYJpgVRAIMkIIIKcZOtB9+4i ALZ0F2+UlmQEsNM4ydaD7t1FAGNLd/FGaUlGAGxJsvWge3cRAFu6izdKSzICYEuSrQfdu4sA2NJd vFFakhEAW5JsPejeXQTAlu7ijdKSjADYkmTrQffuIgC2dBdvlJZkBMLtBpPtsvS6gZpQ0rcQSO+j 8sHQazuneAuXioxejE2y7fPhk2xu6K6EgPdusCff8Vvvgj1tXe/EhUdYvZ+dv2z/0hi6um30vXFt sq6sq0oqMvo1cQJDtGvfWjDcDYx5GYMjlXWdXVs36QVDHqk6icCXX7E/NTedrRzuBbzR3rph3ESJ LuOt1cTsVGTYztO14urGjrn756T07KDhf2GyYBvKoVxntXpjbGZW+Lp5ry+NnMF7n062AOQdHoFu zVuGZudGG1FeHzU0O55pnL772AoCFZmgEF6fX9TEC8cnR8dY9awSNDukSwICXWLL/PI0O3hj9eih oRm+PZKtX7TOUEWGBb4F7tP+XqV/Zm5Cr8r8TK52cPDeqBe/efw5fZbonnHy3LS/tc/9pjwJqsiY OPHz4MxMKFtyIB+LI1toQJGBAgK9dmdGIYVLZFAwpHiFvCU4uTSTqe7tthwH2iz599l+ZnPDJOlb ydAkSm98i338wCf7tmxFXY73jmqjd+aHeNjAMdAdv14rvjgiN3GOpkm17eIquX/8s8kWzFauIqNr Qrx6lL8wMyluseXFnKKSEAuGgJQX4ceW3IxgyNP8Azdh9H430gu+J74aZWenrYMTLWWu9p/ozZfG h4d8BAjyXO2f1vvHbw/yQaxyKKNcjkIa4jSGGELXz9shUpFhtq7hpFTcqgZRGGnCIRCOLXqbe7pv 6rDzdPu6ULA1vg4MLEybJLw/aVl1FRnGePM96FsO6tXslKvZwoNHhX4P6kpmVpfvLlg+L6juL0NK 8kHJ8sTWV1hJhD1cG0BqVQTCsUVeSt8gOSfG04GBhU2N51il0nqwUpHRdbzcPTwT5+uq0GlyJyWK j7H60Ss5dT9/lPp4mYzQoajIcFZbntjqNltclzu9bSkP4TYR6ARbRBUmBjKMel+zX6S3k/xfYbLb psJcfDBLjolPKFpFJkDR7iQ8JC2MrnYBW8dh/VSvC1RXkXEUS7FyGsO/51MmPF1EIARb+ETZ5fFz /0fsLMmuYqdI8wT+Ll/6RsII+xR9GoFKXFhFxkB5+O6dMb/wWlCL8FmNIy1Nctj5ufCln4wU56AK IV0YBEKw5Wr/J/L4n6+vTBoKNN+Ll4JN6Olto5aRpPWINVSJC3vKaKEw0YfRX8Cfla2pVxg0XWmr LG8bSHlx+Qv7UgYVmdyyvWOaur8wJo8rRKo+MrMj0LNV/Kb9dShCHsprrswFIzyxbG2IsXLEZ+EW 5TPut3amtYxNE1bdtC+o8WkhWo9gyphr3uyn6Gp5GsuCmLY6rvk4TppWl3FEjbG4ppNMpqZ4j71x 8yI0WzqpdLLzVllEpyKTbBQSqb0XW0J4YonEAUoDgeAI9IZyw4KXi5RAIL4IXO5uSHmBsyo7YjPH ZSDSLT0qMh1RDpkGRQBsCYoc0qUPAcxb0mdz1DgoAmBLUOSQLn0IgC3pszlqHBQBsCUockiXPgTA lvTZHDUOigDYEhQ5pEsfAmBL+myOGgdFoDfkVpOg5SIdEIgvAnzziGwLFsaW+NoMmsUNAbAlbhaB PvFFAGyJr22gWdwQAFviZhHoE18Eet5u/2hpZ+zssw7AdmluHpUNGalVgQ/BksS24VgSbjOuwYtw N1LEtxeAZkAgOAJf/oz94LojJRCIBwKYt8TDDtAiCQiALUmwEnSMBwJgSzzsAC2SgAB2GifBStAx HghgbImHHaBFEhAAW5JgJegYDwTAlnjYAVokAQGwJQlWgo7xQABsiYcdoEUSEABbkmAl6BgPBMCW eNgBWiQBAbAlCVaCjvFAAGyJhx2gRRIQiGzFvr49QHqYPBPvD3NcjtUmRsK1Xh53eineVaZf8YUb ttrEPyXiXiv2I2ILb6MjrN7P3HfcNS+jXNPvyNZI1ebldYaJOFXYUfMCYZ4tc1yCp3FJuIjPXrSe jcG3avmgbyZ7KL0vtnWbMBlrbvzS5a0tUCBhwlnV2f0tU3PT2crhnuRS7omVxZzYquje981KbjnA DeCTSxZVyBgnpSJdxWjPh1/GvbqxY15Rf1J6dtAYG58QbafJ0Kf0S1CLUg7lOqvVG2Mzs0IeE98W +C2zXjc2By0N6WKEQBTzlqHZudFGee+DrFofritH+qhiPsenVZbNtovB8K2+mu3ibCJM5Wx0dKrd jCKSvz6/qGVGrMvK+dXn1bNKRLkjm1giEAFb5pen2cEbq0e313PntfNybWr3rCYZhgLh0zc45J1u culRgXnQOFBpYqJP+3uV/pm55sA1P5OrHRy8NwTI4dx4Tp8lIjN5btrf2kcYDFVkzAL5eXBmJpQt OZmPxZEtdHWQgQICvbWQDZccpEx1b/ejQlmayNDsvcLnzdfSgahVHpefPmfzeZvEUHaA9Q8MulLR JEpvWIt95ReCY6aqoqrc8d5RbfQOH16GZsczjdN3FgjkcBZfHFGXMEdTqdq25vvxzyZbMFu5ioyu CvHqUf7CzIS7oMIt5KrqQq4dBKS8CDu26H3qsaIeWjDg9EVJVV7M9qRUZtNm38w7Zh5XkBV8tf9E b5rU9z/kvXunnqv903r/+O3B4dsjNG2Tja45CnuIgQRiyLPzkR/uihRXkaH5kDAa8zlbtVOVQr7e CIRjS1sDS5MqwTt7Pr3OLugOiRZY2zhtaVreNA/6ljvpseyUq9nCg0eF/rNT6WhpG3B0ZS/fXTDb IOkvw0MjNChZntj6Cis1Y4No3F1EIBRb2hhYQlNFx8SIaPGgljZANa5b+oCXu4dn4lw8cmRpxKPx re6MZBjlfP5oBujEojMZYcRTkWHcbTPcueLqNltctw9QkVcMGUoQCMOWiYEMo57V7PPo7ST/1x0d jogqTvX5bMGjqXXR1pzAT52RDKN8jyBEvS74oioyjvp8eLW6fV343orIdbG+aS4qDFvIZmKHR/ME 2dsGit7wuYq/A2aEfYqKjYBicddbPlOg4bt3xuoXXvG6zhuez2ocpdAkh9lC4X4yPGjRydlX51G4 MSWEYYsCCPxtuiswRV/K3k5+nedv9xiTtB5XSRMrz9fHz1/a3uRooTDRP9Ffrp+VvTp+Bf3DilRZ 3jbYcpXyF09sIUQVmdyynTBT9xfG5HGFsBojfQsEeraK39iNFwQuYfkWExdfeZ6HK1stZggLS1dE XfSVXcbjWHVifm/ThEmW2NgFzHTKi3Hsapjr4uyn6Gq5cf8zs7daYtoKuubjqLi6jCNqjMU1Qdqp ahpqivfYGzcvomGLqhapkjOY0CpcriKTKtDiUVkvtnTYE4tH5aEFEIgEgd7wblgkeiATIBAfBC53 N6S8wFmVHbGR4zIQ6bYfFZmOKIdMgyIAtgRFDunShwDmLemzOWocFAGwJShySJc+BMCW9NkcNQ6K ANgSFDmkSx8CYEv6bI4aB0UAbAmKHNKlDwGwJX02R42DItArXQ4cNDekAwI3AQG+eUS2TB5jy02w LurQHQTAlu7gjFJuAgL/BxcD9eNUmfWQAAAAAElFTkSuQmCC --00000000000070f99c05a59dface--