From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D3EA9A034F; Tue, 30 Mar 2021 16:27:39 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C22B2140E18; Tue, 30 Mar 2021 16:27:39 +0200 (CEST) Received: from mail-lj1-f178.google.com (mail-lj1-f178.google.com [209.85.208.178]) by mails.dpdk.org (Postfix) with ESMTP id 072FA40691 for ; Tue, 30 Mar 2021 16:27:39 +0200 (CEST) Received: by mail-lj1-f178.google.com with SMTP id a1so20133002ljp.2 for ; Tue, 30 Mar 2021 07:27:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=rp31Ui6H3p3E5QFUatUZYdJOvOqsGYh5zG1TUcyvMJg=; b=bnu6t7PUGRLioMrI0XFe46xH0HN6TIP+d71dhh8A8Ycb/QDvNv1/6fqEtfzjlwHEg1 RUXFCo2L4nhQcdXE65ur0OOkfA5i6tJqdhJTWm70K3ZnCwtWdQx7+fv2WoRCB6e8+inW p1DF0NpeRWbJ26rsGEz9jTmEJLEUm/K6lILj8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=rp31Ui6H3p3E5QFUatUZYdJOvOqsGYh5zG1TUcyvMJg=; b=ZFX75kvNZi1Bd34AM/iyP1PmkEOKoaBeBiIl8zhjrsQ/qXZLUxLc3zr1jbmvUuz1+F /4YNVvdVfslzrJYpCzEePSlGIHdIq8JBox8q5bGLCfOHD4I17ZvebWAhGzR++lLuxmpX 5J8ek575bHVcMoFSmFPGKQcJYI6NUaj9KyvLAD6UBZ3QrV6kf21/GXINBij8XnJ7qw9Q OfwKxCOWNAn3CXzaozP/HmpMr+EGWL0Mv7bjCBYMePcPN3yUNsfBElkQFvB1v4DksHZq hD46E13U7p20gwQtPHI4yjZfQlm2jJlzL69AjS4QFeNj3NGIZBfh3D2qEF6i9qU3r56U zqkQ== X-Gm-Message-State: AOAM531wrTSqKepxSO++6wYhdcGCDpQ/qM1xkQn+s3EMVJ4u6RGiFg5E E1Wc7d/NeVGlXvJLX9eIuWMt2ZqBnqGNqG7+fMb6JQ== X-Google-Smtp-Source: ABdhPJz4e9kjPE540jfFljTB4xznKkO2KtjuI2PKtI35w6f/ydRx0rIwx/yuNVJvx0wXH4DG70KL50xDc2vftLxp7s8= X-Received: by 2002:a2e:140b:: with SMTP id u11mr21657222ljd.125.1617114458519; Tue, 30 Mar 2021 07:27:38 -0700 (PDT) MIME-Version: 1.0 References: <20210311104800.243752-1-lijuan.tu@intel.com> <0e4279f9e86442ed81df0416e9e19ffd@intel.com> <7af980019bc64098a093202578d96288@intel.com> <2388e0c11ce248aea86e00c9687b359f@intel.com> In-Reply-To: <2388e0c11ce248aea86e00c9687b359f@intel.com> From: David Liu Date: Tue, 30 Mar 2021 10:27:02 -0400 Message-ID: To: "Tu, Lijuan" Cc: Owen Hilyard , Ali Alnubani , "dts@dpdk.org" , "dpdklab@iol.unh.edu" Content-Type: multipart/alternative; boundary="000000000000dc8f4605bec1ccd7" Subject: Re: [dts] [dpdklab] Re: [PATCH] tests/nic_single_core_perf: modify accepted_tolerance to percentage X-BeenThere: dts@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: test suite reviews and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dts-bounces@dpdk.org Sender: "dts" --000000000000dc8f4605bec1ccd7 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Lijuan, Just a quick update. We had tested the patch on all our bare-metal machines. And they all work as expected. Best, David On Thu, Mar 25, 2021 at 11:34 PM Tu, Lijuan wrote: > Hi Owen, > > > > I =E2=80=98d like to add 2 more changes in V2: > > > > 1, test_interval will be renamed to throughput_stat_sample_interval for > better understanding. > > 2, accepted_tolerance will be a float to algin with delta in json, both > format are {:.2f} > > > > > > *From:* dts *On Behalf Of * Tu, Lijuan > *Sent:* 2021=E5=B9=B43=E6=9C=8825=E6=97=A5 15:29 > *To:* Owen Hilyard > *Cc:* David Liu ; Ali Alnubani ; > dts@dpdk.org; dpdklab@iol.unh.edu > *Subject:* Re: [dts] [dpdklab] Re: [PATCH] tests/nic_single_core_perf: > modify accepted_tolerance to percentage > > > > sure, I prefer to use a float too. > > > > *From:* Owen Hilyard > *Sent:* 2021=E5=B9=B43=E6=9C=8824=E6=97=A5 1:24 > *To:* Tu, Lijuan > *Cc:* David Liu ; Ali Alnubani ; > dts@dpdk.org; dpdklab@iol.unh.edu > *Subject:* Re: [dpdklab] Re: [dts] [PATCH] tests/nic_single_core_perf: > modify accepted_tolerance to percentage > > > > Hello Lijuan, > > > > Would it be possible to have the delta be a float in the json > representation? For example, a 20% delta would be encoded as "delta": .2 = in > json. Currently, most of the community lab's infrastructure expects to ma= ke > pass/fail determinations on a numeric type, so having the delta as a floa= t > would reduce the required downtime to change everything over to percentag= es > since we shouldn't need to patch all of our services to expect a string. > > > > > > Thanks, > > Owen Hilyard > > > > On Tue, Mar 23, 2021 at 10:53 AM David Liu wrote: > > Thank you for Ali and Lijuan. > > We had tested the patch on the Arm machines setup. Other than the value > issues, the test is running as expected. > > > > On Tue, Mar 23, 2021 at 4:48 AM Tu, Lijuan wrote: > > > > > -----Original Message----- > > From: Ali Alnubani > > Sent: 2021=E5=B9=B43=E6=9C=8823=E6=97=A5 16:44 > > To: Tu, Lijuan > > Cc: dts@dpdk.org; dpdklab@iol.unh.edu > > Subject: RE: [dts] [PATCH] tests/nic_single_core_perf: modify > > accepted_tolerance to percentage > > > > > -----Original Message----- > > > From: dts On Behalf Of Lijuan Tu > > > Sent: Thursday, March 11, 2021 12:48 PM > > > To: dpdklab@iol.unh.edu > > > Cc: dts@dpdk.org; Lijuan Tu > > > Subject: [dts] [PATCH] tests/nic_single_core_perf: modify > > > accepted_tolerance to percentage > > > > > > * accepted_tolerance in conf/nic_single_core_perf.cfg are changed to > > > percentage, e.g. > > > accepted_tolerance =3D '1%%' > > > if not set, it will be set to 2% by default. > > > * delat in json are chagned to percentage too, e.g. > > > 'delta': '1.5%' > > > * expected_throughput will be forced to 1Mpps for compute delat in > > > percentage > > > * add new conf test_interval in conf/nic_single_core_perf.cfg, which > > > indicates interval of getting throughput statistics in second, e.g. > > > test_interval =3D 10 > > > if not set, it will be set to 5 seconds by default. > > > * collect better statistics , get multiple throughput statistics > > > during the test run, and compute the average as the final results. > > > > > > Signed-off-by: Lijuan Tu > > > --- > > > > > +accepted_tolerance =3D '1%%' > > > > > + diff =3D > > > + (self.throughput[fwd_config][frame_size][nb_desc] - > > > + > > > self.expected_throughput[fwd_config][frame_size][nb_desc])/ \ > > > + > > > self.expected_throughput[fwd_config][frame_size][nb_desc] > > > > The value of `diff` here is still a decimal, and needs to be multiplied > by a 100 > > before it can be compared to `accepted_tolerance`. > > > > I suggest splitting this patch into multiple ones. One for making the > percentage > > change and one for taking the mean for multiple readings since they are > > unrelated changes. > > I also see other unrelated changes not explained in the commit message. > > Really appreciate for your review, I will accept your advices. > > > > > Thanks, > > Ali > > --000000000000dc8f4605bec1ccd7 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Lijuan,

Just a quick = update.

We had tested the patch on all our bare-metal mac= hines.=C2=A0

And they all work as expected.

Best,
D= avid



On Thu, Mar 25, 2021 at 11:34 PM Tu= , Lijuan <lijuan.tu@intel.com= > wrote:

Hi Owen,

=C2=A0

I =E2=80=98d like to add 2 more changes in V2:

=C2=A0

1,=C2=A0 test_interval = =C2=A0will be renamed to throughput_stat_sample_interval for better underst= anding.

2, =C2=A0accepted_tolera= nce will be a float to algin with delta in json, both format are {:.2f}

=C2=A0

=C2=A0

From: dts <dts-bounces@dpdk.org> On Behalf Of Tu, Lijuan
Sent: 2021=E5=B9=B43=E6=9C=8825= =E6=97=A5 15:29
To: Owen Hilyard <ohilyard@iol.unh.edu>
Cc: David Liu <dliu@iol.unh.edu>; Ali Alnubani <alialnu@nvidia.com>; dts@dpdk.org; dpdklab@iol.unh.edu
Subject: Re: [dts] [dpdklab] Re: [PATCH] tests/nic_single_core_perf:= modify accepted_tolerance to percentage

=C2=A0

sure, I prefer to use a float too.

=C2=A0

From: Owen Hilyard <ohilyard@iol.unh.edu>
Sent: 2021=E5=B9=B43=E6=9C=8824= =E6=97=A5 1:24
To: Tu, Lijuan <lijuan.tu@intel.com>
Cc: David Liu <dliu@iol.unh.edu>; Ali Alnubani <alialnu@nvidia.com>; dts@dpdk.org; dpdklab@iol.unh.edu
Subject: Re: [dpdklab] Re: [dts] [PATCH] tests/nic_single_core_perf:= modify accepted_tolerance to percentage

=C2=A0

Hello Lijuan,

=C2=A0

Would it be possible to have the delta be a float in= the json representation? For example, a 20% delta would be encoded as &quo= t;delta": .2 in json. Currently, most of the community lab's=C2=A0= infrastructure=C2=A0expects to make pass/fail determinations on a numeric type, so having the delta as a float would reduce the require= d downtime to change everything over to percentages since we shouldn't = need to patch all of our services to expect a string.=C2=A0

=C2=A0

=C2=A0

Thanks,

Owen Hilyard

=C2=A0

On Tue, Mar 23, 2021 at 10:53 AM David Liu <dliu@iol.unh.edu> w= rote:

Thank you for Ali and Lijuan.

We had tested the patch on the Arm machines setup. Other than the value iss= ues, the test is running=C2=A0as expected.

=C2=A0

On Tue, Mar 23, 2021 at 4:48 AM Tu, Lijuan <lijuan.tu@intel.com= > wrote:



> -----Original Message-----
> From: Ali Alnubani <alialnu@nvidia.com>
> Sent: 2021=E5=B9= =B43=E6=9C=8823=E6=97=A5 = 16:44
> To: Tu, Lijuan <lijuan.tu@intel.com>
> Cc: dts@dpdk.org= ; dpdklab@iol.unh.edu
> Subject: RE: [dts] [PATCH] tests/nic_single_core_perf: modify
> accepted_tolerance to percentage
>
> > -----Original Message-----
> > From: dts <dts-bounces@dpdk.org> On Behalf Of Lijuan Tu
> > Sent: Thursday, March 11, 2021 12:48 PM
> > To: dpdk= lab@iol.unh.edu
> > Cc: dts@dpdk.or= g; Lijuan Tu <lijuan.tu@intel.com>
> > Subject: [dts] [PATCH] tests/nic_single_core_perf: modify
> > accepted_tolerance to percentage
> >
> > * accepted_tolerance in conf/nic_single_core_perf.cfg are changed= to
> > percentage, e.g.
> >=C2=A0 =C2=A0 =C2=A0accepted_tolerance =3D '1%%'
> >=C2=A0 =C2=A0 =C2=A0if not set, it will be set to 2% by default. > > * delat in json are chagned to percentage too, e.g.
> >=C2=A0 =C2=A0 =C2=A0'delta': '1.5%'
> > * expected_throughput will be forced to 1Mpps for compute delat i= n
> > percentage
> > * add new conf test_interval in conf/nic_single_core_perf.cfg, wh= ich
> > indicates interval of getting throughput statistics in second, e.= g.
> >=C2=A0 =C2=A0 =C2=A0test_interval =3D 10
> >=C2=A0 =C2=A0 =C2=A0if not set, it will be set to 5 seconds by def= ault.
> > * collect better statistics , get multiple throughput statistics<= br> > > during the test run, and compute the average as the final results= .
> >
> > Signed-off-by: Lijuan Tu <lijuan.tu@intel.com>
> > ---
> <snipped>
> > +accepted_tolerance =3D '1%%'
> <snipped>
> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 diff =3D
> > + (self.throughput[fwd_config][frame_size][nb_desc] -
> > +
> > self.expected_throughput[fwd_config][frame_size][nb_desc])/ \
> > +
> > self.expected_throughput[fwd_config][frame_size][nb_desc]
>
> The value of `diff` here is still a decimal, and needs to be multiplie= d by a 100
> before it can be compared to `accepted_tolerance`.
>
> I suggest splitting this patch into multiple ones. One for making the = percentage
> change and one for taking the mean for multiple readings since they ar= e
> unrelated changes.
> I also see other unrelated changes not explained in the commit message= .

Really appreciate for your review, I will accept your advices.

>
> Thanks,
> Ali

--000000000000dc8f4605bec1ccd7--