From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2DC8FA00C5 for ; Thu, 30 Apr 2020 16:19:49 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0AB531DB23; Thu, 30 Apr 2020 16:19:49 +0200 (CEST) Received: from mail-il1-f180.google.com (mail-il1-f180.google.com [209.85.166.180]) by dpdk.org (Postfix) with ESMTP id EEE501DB22 for ; Thu, 30 Apr 2020 16:19:46 +0200 (CEST) Received: by mail-il1-f180.google.com with SMTP id c16so1486549ilr.3 for ; Thu, 30 Apr 2020 07:19:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=qTiyB+gL/c3GOSwU2piqzB6xG+kuBx2s6N8wGp/XxjI=; b=HfxsVrwmj6s6tOqNMD+m6UmI/3ymcgNsKXNcluKwlkv23eUEik5i2hjsUQzu2NrDKK 8Y9XPA17cevPq7KK1/ewl1o/H2xUoCxjvmAFF/vmgVBwzvnP2xqCYiYHbDH/nELGJ7Wl Ptt7hN3jraMt5WAQzg5qOcmp9F+NSpg9Xw/TQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=qTiyB+gL/c3GOSwU2piqzB6xG+kuBx2s6N8wGp/XxjI=; b=LVk+jkbH2HhTcdObiU5etwQDrGDNEJq6jrett8RxsQa3zm27JF7PosP6BMPYa644f4 KQTgH41Vy8ReA0A2ZD4/3YUeUMR91SOvrkwHURX2Q7MyZ4MeuxhcbwsmeiRb81x7jytZ uKRw0oqLy7bmlYf5RAchCirwhm+Pbdg+8nDzhE4g+UnEk0RAlRcXgvHY9glzgvPfwDg+ fuAoRhuqiEymf4RNbTy+elmbLpqhRr6yQpVxrwF2X9uOOzh+k0L7CR5vvJL+xjB4Y+sY o+DWua2SqvW8ysaAiOaDp0LKr1vqZqZiwcaVvRisBnJJmOQR4RCx1OTdENnILn0jdCY6 mRww== X-Gm-Message-State: AGi0PuaD6Hd5NM6GqxOS1E5xAqEqB0J4Yr7D/KVLg9ZUVtdHGr8OyvGx RYgJ7z7+mVHQnVuAym+/GDx9e/9TTB5/Dz5vntSaZQ== X-Google-Smtp-Source: APiQypJDd2DjP+k+vcEo9wSeSwYHQ/yJ/Mrt6r8re9njebx0mk6EibaJFkBsQVSf5mup66xYbVQjJlS9En/Ge7VAtR4= X-Received: by 2002:a92:cece:: with SMTP id z14mr2130022ilq.147.1588256385892; Thu, 30 Apr 2020 07:19:45 -0700 (PDT) MIME-Version: 1.0 References: <1827408.xz2uEaWSZ7@xps> <9DEEADBC57E43F4DA73B571777FECECA41EAB066@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41EBB4AF@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41EFB9C5@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41F01113@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41F025A1@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41F03B43@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41F058DF@SHSMSX104.ccr.corp.intel.com> <9DEEADBC57E43F4DA73B571777FECECA41F18C47@SHSMSX104.ccr.corp.intel.com> <08e9c34dce60404abb862f35dabd4198@intel.com> <539a8463a02848e399f2340e4f5be692@intel.com> In-Reply-To: <539a8463a02848e399f2340e4f5be692@intel.com> From: Brandon Lo Date: Thu, 30 Apr 2020 10:19:07 -0400 Message-ID: To: "Ma, LihongX" Cc: "Chen, Zhaoyan" , David Marchand , "dpdklab@iol.unh.edu" , Lincoln Lavoie , Thomas Monjalon , "ci@dpdk.org" , "Tu, Lijuan" , "Xu, Qian Q" , "Zhang, XuemingX" , "O'Driscoll, Tim" , "Lin, Xueqin" Content-Type: multipart/related; boundary="000000000000b28b9c05a482c192" Subject: Re: [dpdk-ci] [dpdklab] Re: Intel performance test is failing X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" --000000000000b28b9c05a482c192 Content-Type: multipart/alternative; boundary="000000000000b28b9b05a482c191" --000000000000b28b9b05a482c191 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Lihong, The expected value was reset by one of our internal scripts. I believe that I have resolved this issue for the future by ensuring that the baseline that you sent me will not be overwritten automatically. I will continue to monitor this expected throughput in case of any issues. Thanks for your patience, Brandon On Wed, Apr 29, 2020 at 1:30 AM Ma, LihongX wrote: > Hi, Brandon > > > > I checked the new result of FVL, find the expected value also not changed= . > > From the log find the expected value also is: > > > > Can you help to double check it ? Is there any different between FVL and > NNT ? > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Wednesday, April 29, 2020 12:52 AM > *To:* Ma, LihongX > *Cc:* Chen, Zhaoyan ; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon ; ci@dpdk.org= ; > Tu, Lijuan ; Xu, Qian Q ; > Zhang, XuemingX ; O'Driscoll, Tim < > tim.odriscoll@intel.com>; Lin, Xueqin > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > Sorry about that, I have reset the baseline to the values you sent in the > previous email. > > I'll look to rerun tests that have failed due to the incorrect baseline. > > > > Thanks for letting me know, > > Brandon > > > > On Mon, Apr 27, 2020 at 11:39 PM Ma, LihongX wrote= : > > Hi, Brandon > > I find the baseline of NNT have changed as expected, but FVL still same a= s > before. > > Can you help to check it and change the baseline as expected ? > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Friday, April 3, 2020 2:39 AM > *To:* Ma, LihongX > *Cc:* Chen, Zhaoyan ; David Marchand < > david.marchand@redhat.com>; dpdklab@iol.unh.edu; Lincoln Lavoie < > lylavoie@iol.unh.edu>; Thomas Monjalon ; ci@dpdk.org= ; > Tu, Lijuan ; Xu, Qian Q ; > Zhang, XuemingX ; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Lihong, > > > > I have changed the baselines to reflect the new expected values. > > The performance tests should work as expected and pass. > > > > We will email again in the future if we come across any problems. > > Feel free to email us as well if you would like to make any other changes= . > > > > Thank you for all your help > > > > On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX wrote: > > Hi, Brandon > > Thanks for you recommends, I have done the changes. > > As the throughput value of nic_single_core is proportional to the cpu > frequency. > > I recommend you can change the baseline according to our report system. > > > > On the our 2.50GHz system, the baseline value as below: > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 > > 64 > > 2048 > > 41.439 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 > > 64 > > 2048 > > 47.73 > > > > For the testbed in UNH, it=E2=80=99s a 2.1Ghz CPU server, so the expected= number > should be > > NNT: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 52.562 / 2.5 * 2.1=3D44.152 > > 64 > > 2048 > > 41.439 / 2.5 * 2.1=3D34.809 > > > > FVL: > > *pkt_size* > > *trd/rxd* > > *expected_value* > > 64 > > 512 > > 59.608 / 2.5 * 2.1=3D50.071 > > 64 > > 2048 > > 47.73 / 2.5 * 2.1=3D40.093 > > > > > > > > Regards, > > Ma,lihong > > > > *From:* Brandon Lo [mailto:blo@iol.unh.edu] > *Sent:* Tuesday, March 31, 2020 9:42 PM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Ma, LihongX ; Zhang, > XuemingX ; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > To make changes to either Intel machine, please reboot using the command > "reboot_to_rw" as root to reboot the machine into read/write mode. > > This command will also disable any testing on the machine. > > > > To re-enable the machine, please run "reboot_to_ro" as root, and it will > save all of the changes that you've made and re-enable testing on the > machine. > > I recommend rebooting using either "reboot_to_rw" or "reboot_to_ro" > instead of the normal "reboot" while you're making changes. > > > > After you're done, please let me know. I'll have to manually run a test > and update the baseline using our internal CI. > > > > Thank you > > > > On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan > wrote: > > Hi, Brandon, > > > > Please let me know how to make change to this reset machine. > (ip/access...) and disable it. > > > > After that please help to change the baseline. > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Thursday, March 26, 2020 11:39 PM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Ma, LihongX ; Zhang, > XuemingX ; O'Driscoll, Tim < > tim.odriscoll@intel.com> > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, we have a system in place that resets any changes made while > testing is enabled for a machine. > > If you would like, I can disable testing and allow you to make permanent > changes. > > > > I can also reset the baseline of Intel 10G test performance once you make > these changes. > > Please let me know if you would like to make permanent changes on the > Intel 10G so I can disable it for you. > > > > Thanks > > > > On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan > wrote: > > Thanks. Brandon. > > > > That=E2=80=99s good. We have made changed on 10G testbed. > > > > I monitored the several execution results; I found the results of 10G > always has -0.9%~-1.x% gap against expected number. So it could lead to s= ee > sometime failures..+-1% I suggest adjusting the expected number. I don=E2= =80=99t > know where the expected number is from? as I know it a dynamic number? > depends on baseline.. Please help to clarify, thanks. > > > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Tuesday, March 24, 2020 9:31 PM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Ma, LihongX ; Zhang, > XuemingX > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > I have enabled the 10G Intel machine for testing. > > If you would like to make any more changes, please let me know so I can > perform the necessary steps to prepare the machine for changes. > > Please feel free to let me know if you need anything. > > > > Thank you > > > > On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan > wrote: > > Hi, Brandon, > > > > For 10G, please enable it. our code is at original path > */opt/test-harness/dts.* > > > > For 40G, please keep running. and see if any issue. But, anyway, we have > modified the DTS code at /opt/test-harness/dts-new-suite. If we met same > problem, then use this new DTS instead. > > > > Thanks a lot > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Saturday, March 21, 2020 1:49 AM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Ma, LihongX ; Zhang, > XuemingX > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Currently, the 40G machine is stable enough to be put on production > dashboard to run tests which may cause Trex to be killed. > > Should I disable the 40G Intel machine for you to make changes? > > > > Also, just for confirmation: on the 10G machine, is the folder that you > are using for the testing located in */opt/test-harness/dts-2020-3-4, o*r > are you still using the one in the standard */opt/test-harness/dts* > folder? > > > > If everything is ok, I will enable the 10G machine for production testing= . > > > > Thank you very much > > > > On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan > wrote: > > Brandon, > > > > We worked out a workaround on Intel testbeds. NNT(10G) and FVL(40G). Coul= d > you please help to recover them? > > > > But, for FVL(40G) testbed, we met some problems, could you please help t= o > check before recover it > > - Sometime 1G hugepage will be changed to 2Mhugepage > automatically...we have to restart the system > - When we debugging on the testbed, found that Trex was killed by some > one(app).. > > Please help to check if any other program running on the testbed. > > > > Thanks a lot. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Chen, Zhaoyan > *Sent:* Wednesday, March 18, 2020 9:04 PM > *To:* Brandon Lo > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q ; Chen, Zhaoyan > *Subject:* RE: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Brandon, we almost made a workaround. > > > > Maybe tomorrow, you could recover Intel=E2=80=99s testbed. I will let you= know > soon. > > > > > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Wednesday, March 18, 2020 3:34 AM > *To:* Chen, Zhaoyan > *Cc:* David Marchand ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > Have you finished making changes on the Intel machine? > > I will turn on the machine on March 3rd for testing if you do not have an= y > issues with it. > > Please let me know if you need anything else. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan > wrote: > > Hi, Brandon, > > > > Yes, it=E2=80=99s a wired issue. And it also mixed our DTS upgrading and = Trex > upgrading. > > So we are reviewing our DTS script, different Trex version, and CI callin= g > procedure. > > > > Anyway, we are focusing on this task recently, any update will let you > know. > > > > Thanks. > > > > *Regards,* > > *Zhaoyan Chen* > > > > *From:* Brandon Lo > *Sent:* Tuesday, March 10, 2020 10:46 PM > *To:* David Marchand > *Cc:* Chen, Zhaoyan ; dpdklab@iol.unh.edu; > Lincoln Lavoie ; Thomas Monjalon < > thomas@monjalon.net>; ci@dpdk.org; Tu, Lijuan ; Xu, > Qian Q > *Subject:* Re: [dpdklab] Re: [dpdk-ci] Intel performance test is failing > > > > Hi Zhaoyan, > > > > How is the current status of the Intel 82599ES? > > Were there any configuration changes made to fix performance issues? > > > > Thanks > > > > On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo wrote: > > Hi David, > > > > This was just a weird issue with the packet generator not cleaning itself > after a test fast enough before another test. > > I'll rerun the tests that were affected and keep an eye out to see if it'= s > stable enough to be put back online. > > > > Thanks > > > > On Tue, Mar 10, 2020 at 5:33 AM David Marchand > wrote: > > On Tue, Mar 3, 2020 at 3:14 PM Brandon Lo wrote: > > > > Hi David and Zhaoyan, > > > > > > Yes, those results are related to the Intel machine; I have disabled > testing for the Intel testbed. > > > > The 82599ES machine is now available for ssh and modifications. > > Any news about this? > > I received a failure on a patch of mine (changing macros in a ARM header)= . > https://lab.dpdk.org/results/dashboard/patchsets/9900/ > > But this time, it is with the 40G Intel nic test. > > -- > David Marchand > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu > --=20 Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu --000000000000b28b9b05a482c191 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Lihong,

The expected value was reset= by one of our internal scripts.
I believe that I have resolved t= his issue for the future by ensuring that the baseline that you sent me wil= l not be overwritten automatically.

I will continu= e to monitor this expected throughput in case of any issues.

=
Thanks for your patience,
Brandon

On Wed, Apr 29,= 2020 at 1:30 AM Ma, LihongX <li= hongx.ma@intel.com> wrote:

Hi, Brandon

=C2=A0

I checked the new result of FVL, find the expected value al= so not changed.

From the log find the expected value also is:=

=C2=A0

Can you help to double check it ? Is there any different be= tween FVL and NNT ?

=C2=A0

Regards,

Ma,lihong

=C2=A0

From: Brandon Lo [mailto:blo@iol.unh.edu]
Sent: Wednesday, April 29, 2020 12:52 AM
To: Ma, LihongX <lihongx.ma@intel.com>
Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.= com>; dpdkl= ab@iol.unh.edu; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; T= u, Lijuan <liju= an.tu@intel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.c= om>; Lin, Xueqin <xueqin.lin@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Lihong,

=C2=A0

Sorry about that, I have reset the baseline to the v= alues you sent in the previous email.

I'll look to rerun tests that have failed due to= the incorrect baseline.

=C2=A0

Thanks for letting me know,

Brandon

=C2=A0

On Mon, Apr 27, 2020 at 11:39 PM Ma, LihongX <lihongx.ma@intel.com= > wrote:

Hi, Brandon

I find the baseline of NNT have changed as e= xpected, but FVL still same as before.

Can you help to check it and change the base= line as expected ?

=C2=A0

Regards,

Ma,lihong

=C2=A0

From: Brandon Lo [mailto:blo@iol.unh.edu]
Sent: Friday, April 3, 2020 2:39 AM
To: Ma, LihongX <lihongx.ma@intel.com>
Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; David Marchand <david.marchand@redhat.= com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Zhang, XuemingX <xuemingx= .zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Lihong,

=C2=A0

I have changed the baselines to reflect the new expe= cted values.

The=C2=A0performance tests should work as expected a= nd=C2=A0pass.

=C2=A0

We will email again in the future if we come across = any problems.

Feel free to email us as well if you would like to m= ake any other changes.

=C2=A0

Thank you for all your help

=C2=A0

On Wed, Apr 1, 2020 at 2:00 AM Ma, LihongX <lihongx.ma@intel.com> wrote:

Hi, Brandon

Thanks for you recommends, I have done the c= hanges.

As the throughput =C2=A0value of nic_single_= core is proportional to the cpu frequency.

I recommend you can change the baseline acco= rding to our report system.

=C2=A0

On the our 2.50GHz system, the baseline valu= e as below:

NNT:

pkt_size

trd/rxd

expected_value

6= 4

5= 12

5= 2.562

6= 4

2= 048

4= 1.439

=C2=A0

FVL:

pkt_size

trd/rxd

expected_value

6= 4

5= 12

5= 9.608

6= 4

2= 048

4= 7.73

=C2=A0

For the testbed in UNH, it= =E2=80=99s a 2.1Ghz CPU server, so the expected number should be

NNT:<= /p>

pkt_size

trd/rxd

expected_value

6= 4

5= 12

5= 2.562 / 2.5 * 2.1=3D44.152

6= 4

2= 048

4= 1.439 / 2.5 * 2.1=3D34.809

=C2=A0

FVL:<= /p>

pkt_size

trd/rxd

expected_value

6= 4

5= 12

5= 9.608 / 2.5 * 2.1=3D50.071

6= 4

2= 048

4= 7.73 / 2.5 * 2.1=3D40.093

=C2=A0

=C2=A0

=C2=A0

Regards,

Ma,lihong

=C2=A0

From: Brandon Lo [mailto:blo@iol.unh.edu]
Sent: Tuesday, March 31, 2020 9:42 PM
To: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.edu; Linco= ln Lavoie <lylavoie@= iol.unh.edu>; Thomas Monjalon <t= homas@monjalon.net>; ci@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com= >; Xu, Qian Q <qian.q.x= u@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odrisco= ll@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Zhaoyan,

=C2=A0

To make changes to either Intel machine, please rebo= ot using the command "reboot_to_rw" as root to reboot the machine= into read/write mode.

This command will also disable any testing on the ma= chine.

=C2=A0

To re-enable the machine, please run "reboot_to= _ro" as root, and it will save all of the changes that you've made= and re-enable testing on the machine.

I recommend rebooting using either "reboot_to_r= w" or "reboot_to_ro" instead of the normal "reboot"= ; while you're making changes.

=C2=A0

After you're done, please let me know. I'll = have to manually run a test and update=C2=A0the baseline using our internal= CI.

=C2=A0

Thank you

=C2=A0

On Mon, Mar 30, 2020 at 12:43 AM Chen, Zhaoyan <<= a href=3D"mailto:zhaoyan.chen@intel.com" target=3D"_blank">zhaoyan.chen@int= el.com> wrote:

Hi, Brandon,

=C2=A0

Please let me know how to make change to = this reset machine. (ip/access...) and disable it.

=C2=A0

After that please help to change the base= line.

=C2=A0

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Brandon Lo <blo@iol.unh.edu>
Sent: Thursday, March 26, 2020 11:39 PM
To: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.c= om>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Zhaoyan,

=C2=A0

Currently, we have a system in place that resets any= changes made while testing is enabled for a machine.

If you would like, I can disable testing and allow y= ou to make permanent changes.

=C2=A0

I can also reset the baseline of Intel 10G test perf= ormance once you make these changes.

Please let me know if you would like to make permane= nt changes on the Intel 10G so I can disable it for you.

=C2=A0

Thanks

=C2=A0

On Wed, Mar 25, 2020 at 12:59 AM Chen, Zhaoyan <<= a href=3D"mailto:zhaoyan.chen@intel.com" target=3D"_blank">zhaoyan.chen@int= el.com> wrote:

Thanks. Brandon.

=C2=A0

That=E2=80=99s good. We have made changed= on 10G testbed.

=C2=A0

I monitored the several execution results= ; I found the results of 10G always has -0.9%~-1.x% gap against expected number. So it could lead to see sometime failures..+-1% I suggest adjustin= g the expected number. I don=E2=80=99t know where the expected number is fr= om? as I know it a dynamic number? depends on baseline.. Please help to cla= rify, thanks.

=C2=A0

=C2=A0

Thanks.

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Brandon Lo <blo@iol.unh.edu>
Sent: Tuesday, March 24, 2020 9:31 PM
To: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Zhaoyan,

=C2=A0

I have enabled the 10G Intel machine for testing.=

If you would like to make any more changes, please l= et me know so I can perform the necessary steps to prepare the machine for = changes.

Please feel free to let me know if you need anything= .

=C2=A0

Thank you

=C2=A0

On Sun, Mar 22, 2020 at 9:58 PM Chen, Zhaoyan <zhaoyan.chen@inte= l.com> wrote:

Hi, Brandon,

=C2=A0

For 10G, please enable it. our code is at= original path /opt/test-harness/dts.

=C2=A0

For 40G, please keep running. and see if = any issue. But, anyway, we have modified the DTS code at /opt/test-harness/= dts-new-suite. If we met same problem, then use this new DTS instead.

=C2=A0

Thanks a lot=

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Brandon Lo <blo@iol.unh.edu>
Sent: Saturday, March 21, 2020 1:49 AM
To: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Ma, LihongX <lihongx.ma@intel.com>; Zhang, XuemingX <xuemingx.zhang@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Zhaoyan,

=C2=A0

Currently, the 40G machine is stable enough to be pu= t on production dashboard to run tests which may cause Trex to be killed.

Should I disable the 40G Intel machine for you to ma= ke changes?

=C2=A0

Also, just for confirmation: on the 10G machine, is = the folder that you are using for the testing located in=C2=A0/opt/test-= harness/dts-2020-3-4, or are you still using the one in the standard=C2=A0/opt/test-harness/dts folder?

=C2=A0

If everything is ok, I will enable the 10G machine f= or production testing.

=C2=A0

Thank you very much

=C2=A0

On Thu, Mar 19, 2020 at 9:36 PM Chen, Zhaoyan <zhaoyan.chen@inte= l.com> wrote:

Brandon,

=C2=A0

We worked out a workaround on Intel testb= eds. NNT(10G) and FVL(40G). Could you please help to recover them?

=C2=A0

But, for FVL(40G) testbed, =C2=A0we met s= ome problems, could you please help to check before recover it

  • Sometime 1= G hugepage will be changed to 2Mhugepage automatically...we have to restart= the system
  • When we de= bugging on the testbed, found that Trex was killed by some one(app)..

Please help to check if any other program= running on the testbed.

=C2=A0

Thanks a lot.

=C2=A0

=C2=A0

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Sent: Wednesday, March 18, 2020 9:04 PM
To: Brandon Lo <blo@iol.unh.edu>
Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>; Chen, Zhaoyan <zhaoyan.ch= en@intel.com>
Subject: RE: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Brandon, we almost made a workaround.

=C2=A0

Maybe tomorrow, you could recover Intel= =E2=80=99s testbed. I will let you know soon.

=C2=A0

=C2=A0

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Brandon Lo <blo@iol.unh.edu>
Sent: Wednesday, March 18, 2020 3:34 AM
To: Chen, Zhaoyan <zhaoyan.chen@intel.com>
Cc: David Marchand <david.marchand@redhat.com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Zhaoyan,

=C2=A0

Have you finished making changes on the Intel machin= e?

I will turn on the machine on March 3rd for testing = if you do not have any issues with it.

Please let me know if you need anything else.=

=C2=A0

Thanks

=C2=A0

On Tue, Mar 10, 2020 at 10:13 PM Chen, Zhaoyan <<= a href=3D"mailto:zhaoyan.chen@intel.com" target=3D"_blank">zhaoyan.chen@int= el.com> wrote:

Hi, Brandon,

=C2=A0

Yes, it=E2=80=99s a wired issue. And it a= lso mixed our DTS upgrading and Trex upgrading.

So we are reviewing our DTS script, diffe= rent Trex version, and CI calling procedure.

=C2=A0

Anyway, we are focusing on= this task recently, any update will let you know.

=C2=A0

Thanks.

=C2=A0

Regards,

Zhaoyan Chen<= /p>

=C2=A0

From: Brandon Lo <blo@iol.unh.edu>
Sent: Tuesday, March 10, 2020 10:46 PM
To: David Marchand <david.marchand@redhat.com>
Cc: Chen, Zhaoyan <zhaoyan.chen@intel.com>; dpdklab@iol.unh.ed= u; Lincoln Lavoie <lylavoie@iol.unh.edu>; Thomas Monjalon <thomas@monjalon.net>; ci@dpdk.org; Tu, Lijua= n <lijuan.tu@in= tel.com>; Xu, Qian Q <qian.q.xu@intel.com>
Subject: Re: [dpdklab] Re: [dpdk-ci] Intel performance test is faili= ng

=C2=A0

Hi Zhaoyan,

=C2=A0

How is the current status of the Intel 82599ES?

Were there any configuration changes made to fix per= formance issues?

=C2=A0

Thanks

=C2=A0

On Tue, Mar 10, 2020 at 9:11 AM Brandon Lo <blo@iol.unh.edu> wro= te:

Hi David,

=C2=A0

This was just a weird issue with the packet generato= r not cleaning itself after a test fast enough before another test.<= u>

I'll rerun the tests that were affected and keep= an eye out to see if it's stable enough to be put back online.<= u>

=C2=A0

Thanks

=C2=A0

On Tue, Mar 10, 2020 at 5:33 AM David Marchand <<= a href=3D"mailto:david.marchand@redhat.com" target=3D"_blank">david.marchan= d@redhat.com> wrote:

On Tue, Mar 3, 2020 at = 3:14 PM Brandon Lo <blo@iol.unh.edu> wrote:
>
> Hi David and Zhaoyan,
>
>
> Yes, those results are related to the Intel machine; I have disabled t= esting for the Intel testbed.
>
> The 82599ES machine is now available for ssh and modifications.

Any news about this?

I received a failure on a patch of mine (changing macros in a ARM header).<= br> https://lab.dpdk.org/results/dashboard/patchsets/9900/

But this time, it is with the 40G Intel nic test.

--
David Marchand


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

= blo@iol.unh.edu

www.iol.unh.edu


=C2=A0

--

Brandon Lo

UNH InterOperability Laboratory=

21 Madbury Rd, Suite 100, Durham, NH 0382= 4

blo@iol.unh.edu

www.iol.unh.edu



--

Brandon Lo

UNH = InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

<= a href=3D"http://www.iol.unh.edu/" target=3D"_blank">www.iol.unh.edu

--000000000000b28b9b05a482c191-- --000000000000b28b9c05a482c192 Content-Type: image/png; name="image001.png" Content-Disposition: inline; filename="image001.png" Content-Transfer-Encoding: base64 Content-ID: <171cb6d77a54cff311> X-Attachment-Id: 171cb6d77a54cff311 iVBORw0KGgoAAAANSUhEUgAAAQ8AAACSCAIAAADUwthAAAAAAXNSR0IArs4c6QAAEllJREFUeF7t Xc9LHEseL91l/wGdMa7uYU8j66gvPJ8nNzCgCO8FBHl7UJAlObi5zCUeksvDlbCQHJKLlxB4eQdh 3EMQBHmLOMFFPImPJOqAc9eXODPZy572tt/qnu6u7q6erunumXTbn2YOOvOtqm99vvWp+ta360fP n/44zPAAASCggECvggxEgAAQ4AiALWgHQEAVAbBFFSnIAQGwBW0ACKgi0Ltxf0JVFnJAIB0IDN8t SnmBsSUd9kcto0AAbIkCReSRDgTAlnTYGbWMAgGwJQoUkUc6EOh5u/2jVdP6UfHpPv1Ls5xHhX4p Amdba69O+C+QAT4OBBLdNqbury+PepDe4EXP2yffFV9/SEfXgFoCASUE+EiQPXTzAp6YEnwQAgKE ANiCZgAEVBEAW1SRghwQAFvQBoCAKgI92N+iChXkUo8AxpbUNwEAoIwA2KIMFQRTjwDYkvomAACU EQBblKGCYOoRAFtS3wQAgDICYIsyVBBMPQJgS+qbAABQRgBsUYYKgqlHAGxJfRMAAMoI/Oafq3/+ 17trZXkIyhGgNd7/+Nt3386N/a5yXP1vdCgNzf7w97/+Za7wh//8+5dfo8u2ozklUWc7INya0/9z 8yLo/pbJpY3FnAfmjfKLjZ2rjhokVOa072eu9vLJ7sdQuUgST6w8v3PdRt1JfmHMQ4nagaXh/OP1 gXJzE17UOncqvy7rHK1No97fclIqrq5tVhgZlf4QPttnncL/RuZLPUsTvXKd0d5DHUkCFk8MEcC8 JYZGgUoxReC3EelFTsXo+9XSMfvwatXatyzsda5u8l8F36OyTTs5DQH+6+Dj9ZmMoY72q0M3Gtwt AaZn6H7s7k1le5MtmH6XlcPog42ClVZ0e/RvVcqy7eTmCkeEJWO/1hruvISDENy+rlnr5k+G/i5J uwtt7qTnxRk/SbbXu8wh4kPyP9/SD3JwFuels5Ld1fRRt2l48/RsFb8J7MELzqKny87xyl/oh2OY DXH8XJg20KTwYYbVc9maxRCOZrZ5pIaWSmsNos00KG3GNuwtfqlbxUEGPx9XqSxuJKfCjbMMa2fe YjNfC0effhpnjSwzYOSITV8bx4kIuehWOBx4uDCgT3s4SszsVjQ0xF6G17QpKVjHMUdyOfGOVNq/ 9Ua2ZtvI7q+zkt15t+WnD1fdz6btMYWqfI+9cfMirCeWLTzYeL6+4T1bvdzdeHY+svF4Vk6VZi1y 7PylOJgcv16jVD/cHdR/H757x0YV+oomTi+OBhaXpiwcJlYW+2gaoB9Joz+UT7tzAJWyCE0ivEth 5jVlb89WMmmLKvTr1f5PB42xcemRvP0zDxfY1lrT0nx6aYzAk0u8A7INyOQIrJ3mH6xMtqHg1H0H wSiT7euM5IQgBZ197N6GWl0RDcsWY5bfanJvEob3N+KoYtWwcfrOGaG6fHfB8nld5Ot8/9mp62Ca q/3Teu4r09KTo2P1C3csjgjT1uCpUNbEtwUmUXj3sHMRjrOyNTi3bhjO8daQnhrP1c7P3Wl/Ofci nrScia9Gq3vOcOKHnw8krqOCzj527woF2igkLFuEoj5/9I4aE2HKbHqGHXk0XI+0mYw5dIwt0gjm /NA0ZuCWMf7c6mO1WhtV9xb1LSuSUjqTSePaIzA+mGXXn0IHzYeyA1Hq7W/3KEsLnVdUbKHhuDni k5dielCmevqoIrpkds37BodkVanXzXm8GV21B6ytcePy02eWzYYGhGfgW1YkpXQ5k481q2cJXvRV LdI32f52D65qB1JGxRaball7qzUdMMccRkjTP367OUSYXw7fHmGG57BTrnq46UIeJ5WzzMi8m3U0 030uTm98UFQoixwPJlGYJlcdsFBUWR6fVrOGZyvmKfc8PUv98L6SmzPmk4YUuabyk039lPexu1/y bv/eEbYIlaCAiW2u0iSMs/lWWf6BeGOGHkmz3LaTEgWCzVCBaSTKXBjHeCOeeVgUCUOhEi10Zos1 U3w2WyhY4QFOJyEfhbKoFjQ5FodQTWHWuXlLBO2C6lWbdlxLondkYlzEMY1xn997/Hr7umALDMw/ 5jGxQBr62Z0xX330cn1sGkg5d6KgEeRWK18Y00O9WqDT8I1c71vM2LwWSdxbLTHxIFrj5FlRY7fl JDNaW6GMyfKhPO2H3kpe3aiUZXsnwwuqNVeyyF4WedrLjaSos/hr83vxhVLz/YbXmdRufByn/bpf NJGejnqR/+x6l2J7qUWZ/MS+t45CVdNZax5KdlfQh6Pra1N1ynhFkIOyRb1kX0kDNdmrRt/EEIgF Al4Lq1opF2O7d+p9SyxsBSWAQFcQ6G3rXURXVEIhSUNgculRgZX3bs5NDTQvlfLiC59VqeJGJ63t pEJf+zq6tvdoJNTuX5gtqWhZqORNQaDTEeSbghPqAQRwfwvaABBQRwBjizpWkEw7AmBL2lsA6q+O ANiijhUk044A2JL2FoD6qyMAtqhjBcm0I9DrWJSadjxQfyDAd7YXpbzA2ILWAQRUEQBbVJGCHBCQ nezq2CLiBElbFMTEvStuGCEDfLzIFcu2YT9SQr4BYWg26DnI6GeAwM1FwGu7Ts/b7R+tWnvsNLy5 sKBmQKCJgGNZtA0XgxfwxExYYukhwOP1pHOk9oInhm4TCARDIOobKYJpgVRAIMkIIIKcZOtB9+4i ALZ0F2+UlmQEsNM4ydaD7t1FAGNLd/FGaUlGAGxJsvWge3cRAFu6izdKSzICYEuSrQfdu4sA2NJd vFFakhEAW5JsPejeXQTAlu7ijdKSjADYkmTrQffuIgC2dBdvlJZkBMLtBpPtsvS6gZpQ0rcQSO+j 8sHQazuneAuXioxejE2y7fPhk2xu6K6EgPdusCff8Vvvgj1tXe/EhUdYvZ+dv2z/0hi6um30vXFt sq6sq0oqMvo1cQJDtGvfWjDcDYx5GYMjlXWdXVs36QVDHqk6icCXX7E/NTedrRzuBbzR3rph3ESJ LuOt1cTsVGTYztO14urGjrn756T07KDhf2GyYBvKoVxntXpjbGZW+Lp5ry+NnMF7n062AOQdHoFu zVuGZudGG1FeHzU0O55pnL772AoCFZmgEF6fX9TEC8cnR8dY9awSNDukSwICXWLL/PI0O3hj9eih oRm+PZKtX7TOUEWGBb4F7tP+XqV/Zm5Cr8r8TK52cPDeqBe/efw5fZbonnHy3LS/tc/9pjwJqsiY OPHz4MxMKFtyIB+LI1toQJGBAgK9dmdGIYVLZFAwpHiFvCU4uTSTqe7tthwH2iz599l+ZnPDJOlb ydAkSm98i338wCf7tmxFXY73jmqjd+aHeNjAMdAdv14rvjgiN3GOpkm17eIquX/8s8kWzFauIqNr Qrx6lL8wMyluseXFnKKSEAuGgJQX4ceW3IxgyNP8Azdh9H430gu+J74aZWenrYMTLWWu9p/ozZfG h4d8BAjyXO2f1vvHbw/yQaxyKKNcjkIa4jSGGELXz9shUpFhtq7hpFTcqgZRGGnCIRCOLXqbe7pv 6rDzdPu6ULA1vg4MLEybJLw/aVl1FRnGePM96FsO6tXslKvZwoNHhX4P6kpmVpfvLlg+L6juL0NK 8kHJ8sTWV1hJhD1cG0BqVQTCsUVeSt8gOSfG04GBhU2N51il0nqwUpHRdbzcPTwT5+uq0GlyJyWK j7H60Ss5dT9/lPp4mYzQoajIcFZbntjqNltclzu9bSkP4TYR6ARbRBUmBjKMel+zX6S3k/xfYbLb psJcfDBLjolPKFpFJkDR7iQ8JC2MrnYBW8dh/VSvC1RXkXEUS7FyGsO/51MmPF1EIARb+ETZ5fFz /0fsLMmuYqdI8wT+Ll/6RsII+xR9GoFKXFhFxkB5+O6dMb/wWlCL8FmNIy1Nctj5ufCln4wU56AK IV0YBEKw5Wr/J/L4n6+vTBoKNN+Ll4JN6Olto5aRpPWINVSJC3vKaKEw0YfRX8Cfla2pVxg0XWmr LG8bSHlx+Qv7UgYVmdyyvWOaur8wJo8rRKo+MrMj0LNV/Kb9dShCHsprrswFIzyxbG2IsXLEZ+EW 5TPut3amtYxNE1bdtC+o8WkhWo9gyphr3uyn6Gp5GsuCmLY6rvk4TppWl3FEjbG4ppNMpqZ4j71x 8yI0WzqpdLLzVllEpyKTbBQSqb0XW0J4YonEAUoDgeAI9IZyw4KXi5RAIL4IXO5uSHmBsyo7YjPH ZSDSLT0qMh1RDpkGRQBsCYoc0qUPAcxb0mdz1DgoAmBLUOSQLn0IgC3pszlqHBQBsCUockiXPgTA lvTZHDUOigDYEhQ5pEsfAmBL+myOGgdFoDfkVpOg5SIdEIgvAnzziGwLFsaW+NoMmsUNAbAlbhaB PvFFAGyJr22gWdwQAFviZhHoE18Eet5u/2hpZ+zssw7AdmluHpUNGalVgQ/BksS24VgSbjOuwYtw N1LEtxeAZkAgOAJf/oz94LojJRCIBwKYt8TDDtAiCQiALUmwEnSMBwJgSzzsAC2SgAB2GifBStAx HghgbImHHaBFEhAAW5JgJegYDwTAlnjYAVokAQGwJQlWgo7xQABsiYcdoEUSEABbkmAl6BgPBMCW eNgBWiQBAbAlCVaCjvFAAGyJhx2gRRIQiGzFvr49QHqYPBPvD3NcjtUmRsK1Xh53eineVaZf8YUb ttrEPyXiXiv2I2ILb6MjrN7P3HfcNS+jXNPvyNZI1ebldYaJOFXYUfMCYZ4tc1yCp3FJuIjPXrSe jcG3avmgbyZ7KL0vtnWbMBlrbvzS5a0tUCBhwlnV2f0tU3PT2crhnuRS7omVxZzYquje981KbjnA DeCTSxZVyBgnpSJdxWjPh1/GvbqxY15Rf1J6dtAYG58QbafJ0Kf0S1CLUg7lOqvVG2Mzs0IeE98W +C2zXjc2By0N6WKEQBTzlqHZudFGee+DrFofritH+qhiPsenVZbNtovB8K2+mu3ibCJM5Wx0dKrd jCKSvz6/qGVGrMvK+dXn1bNKRLkjm1giEAFb5pen2cEbq0e313PntfNybWr3rCYZhgLh0zc45J1u culRgXnQOFBpYqJP+3uV/pm55sA1P5OrHRy8NwTI4dx4Tp8lIjN5btrf2kcYDFVkzAL5eXBmJpQt OZmPxZEtdHWQgQICvbWQDZccpEx1b/ejQlmayNDsvcLnzdfSgahVHpefPmfzeZvEUHaA9Q8MulLR JEpvWIt95ReCY6aqoqrc8d5RbfQOH16GZsczjdN3FgjkcBZfHFGXMEdTqdq25vvxzyZbMFu5ioyu CvHqUf7CzIS7oMIt5KrqQq4dBKS8CDu26H3qsaIeWjDg9EVJVV7M9qRUZtNm38w7Zh5XkBV8tf9E b5rU9z/kvXunnqv903r/+O3B4dsjNG2Tja45CnuIgQRiyLPzkR/uihRXkaH5kDAa8zlbtVOVQr7e CIRjS1sDS5MqwTt7Pr3OLugOiRZY2zhtaVreNA/6ljvpseyUq9nCg0eF/rNT6WhpG3B0ZS/fXTDb IOkvw0MjNChZntj6Cis1Y4No3F1EIBRb2hhYQlNFx8SIaPGgljZANa5b+oCXu4dn4lw8cmRpxKPx re6MZBjlfP5oBujEojMZYcRTkWHcbTPcueLqNltctw9QkVcMGUoQCMOWiYEMo57V7PPo7ST/1x0d jogqTvX5bMGjqXXR1pzAT52RDKN8jyBEvS74oioyjvp8eLW6fV343orIdbG+aS4qDFvIZmKHR/ME 2dsGit7wuYq/A2aEfYqKjYBicddbPlOg4bt3xuoXXvG6zhuez2ocpdAkh9lC4X4yPGjRydlX51G4 MSWEYYsCCPxtuiswRV/K3k5+nedv9xiTtB5XSRMrz9fHz1/a3uRooTDRP9Ffrp+VvTp+Bf3DilRZ 3jbYcpXyF09sIUQVmdyynTBT9xfG5HGFsBojfQsEeraK39iNFwQuYfkWExdfeZ6HK1stZggLS1dE XfSVXcbjWHVifm/ThEmW2NgFzHTKi3Hsapjr4uyn6Gq5cf8zs7daYtoKuubjqLi6jCNqjMU1Qdqp ahpqivfYGzcvomGLqhapkjOY0CpcriKTKtDiUVkvtnTYE4tH5aEFEIgEgd7wblgkeiATIBAfBC53 N6S8wFmVHbGR4zIQ6bYfFZmOKIdMgyIAtgRFDunShwDmLemzOWocFAGwJShySJc+BMCW9NkcNQ6K ANgSFDmkSx8CYEv6bI4aB0UAbAmKHNKlDwGwJX02R42DItArXQ4cNDekAwI3AQG+eUS2TB5jy02w LurQHQTAlu7gjFJuAgL/BxcD9eNUmfWQAAAAAElFTkSuQmCC --000000000000b28b9c05a482c192--