RE: rte_service unit test failing randomly

RE: rte_service unit test failing randomly

[Morten Brørup]

+CC: DTS mailing list.

> From: David Marchand [mailto:david.marchand@redhat.com]
> Sent: Thursday, 6 October 2022 09.05
> 
> On Thu, Oct 6, 2022 at 8:53 AM Morten Brørup <mb@smartsharesystems.com>
> wrote:

[...]

> > Forgive me, if I am sidetracking a bit here... The issue discussed
> seems to be related to some threads waiting for other threads, and my
> question is not directly related to that.
> >
> > I have been wondering how accurate the tests really are. Where can I
> see what is being done to ensure that the EAL worker threads are fully
> isolated, and never interrupted by the O/S scheduler or similar?
> >
> > For reference, the max packet rate at 40 Gbit/s is 59.52 M pkt/s. If
> a NIC is configured with 4096 Rx descriptors, packet loss will occur
> after ca. 70 us (microseconds!) if not servicing the ingress queue when
> receiving at max packet rate.
> >
> > I recently posted some code for monitoring the O/S noise in EAL
> worker threads [1]. What should I do if I want to run that code in the
> automated test environment? It would be for informational purposes
> only, i.e. I would manually look at the test output to see the result.
> >
> > I would write a test application that simply starts the O/S noise
> monitor thread as an isolated EAL worker thread, the main thread would
> then wait for 10 minutes (or some other duration), dump the result to
> the standard output, and exit the application.
> >
> > [1]:
> http://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35D87352@smarts
> erver.smartshare.dk/
> 
> Maybe we could do some hack, like finding a test in the current CI
> that matches the test requirement: number of cores, ports, setup
> params etc... (retrieving the output could be another challenge).
> But if you think this is something we should have on the long run, I'd
> suggest writing a new DTS test.

It would be useful having in the long run - it could catch sudden anomalies in the test environment.

Where can I find documentation on how to add a new test case to DTS?

-Morten

RE: [dpdklab] RE: rte_service unit test failing randomly

[Morten Brørup]

From: Lincoln Lavoie [mailto:lylavoie@iol.unh.edu] 
Sent: Thursday, 6 October 2022 13.07
> On Thu, Oct 6, 2022 at 5:49 AM Morten Brørup <mb@smartsharesystems.com> wrote:
> > From: Mattias Rönnblom [mailto:mattias.ronnblom@ericsson.com]
> > Sent: Thursday, 6 October 2022 10.59
> > 
> > On 2022-10-06 10:18, Morten Brørup wrote:
> > >> From: Mattias Rönnblom [mailto:mattias.ronnblom@ericsson.com]
> > >> Sent: Thursday, 6 October 2022 09.51
> > >>
> > >> On 2022-10-06 08:53, Morten Brørup wrote:
> > >
> > > [...]
> > >
> > >>> I have been wondering how accurate the tests really are. Where can
> > I
> > >> see what is being done to ensure that the EAL worker threads are
> > fully
> > >> isolated, and never interrupted by the O/S scheduler or similar?

[...]

> Back to the topic of unit testing, I think we need to consider their purpose and where we expect them to run.  Unit tests are run in automated environments, across multiple CI systems, i.e. UNH-IOL Community Lab, GitHub, etc.  Those environments are typically virtualized and I don't think the unit tests should require turning down to the level of CPU clock ticks.  Those tests are likely better suited to dedicated performance environments, where the complete host is tightly controlled, for the purpose of repeatable and deterministic results on things like packet throughput, etc.

Excellent point, Lincoln. Verifying the performance of the surrounding runtime environment - i.e. the host running the tests - is not unit test material, it belongs with the performance tests.

So, how do I add a test case to the performance test suite? Is there a guide, a reference example, or any other documentation I can look at for inspiration?

NB: This is a good example of yesterday's techboard meeting discussion... I'm a DTS newbie wanting to contribute with a new test case, how do I get started?

If no such documentation exists, could someone please point at a couple of files representing what needs to be added?

RE: [dpdklab] RE: rte_service unit test failing randomly

[Honnappa Nagarahalli]

<snip>

> > >
> > > On 2022-10-06 10:18, Morten Brørup wrote:
> > > >> From: Mattias Rönnblom [mailto:mattias.ronnblom@ericsson.com]
> > > >> Sent: Thursday, 6 October 2022 09.51
> > > >>
> > > >> On 2022-10-06 08:53, Morten Brørup wrote:
> > > >
> > > > [...]
> > > >
> > > >>> I have been wondering how accurate the tests really are. Where can
> > > I
> > > >> see what is being done to ensure that the EAL worker threads are
> > > fully
> > > >> isolated, and never interrupted by the O/S scheduler or similar?
> 
> [...]
> 
> > Back to the topic of unit testing, I think we need to consider their purpose and
> where we expect them to run.  Unit tests are run in automated environments,
> across multiple CI systems, i.e. UNH-IOL Community Lab, GitHub, etc.  Those
> environments are typically virtualized and I don't think the unit tests should
> require turning down to the level of CPU clock ticks.  Those tests are likely better
> suited to dedicated performance environments, where the complete host is
> tightly controlled, for the purpose of repeatable and deterministic results on
> things like packet throughput, etc.
> 
> Excellent point, Lincoln. Verifying the performance of the surrounding runtime
> environment - i.e. the host running the tests - is not unit test material, it belongs
> with the performance tests.
IIRC, the unit tests were separated into perf tests and non-perf tests. The perf tests were meant to be run on bare metal systems in the lab.

> 
> So, how do I add a test case to the performance test suite? Is there a guide, a
> reference example, or any other documentation I can look at for inspiration?
> 
> NB: This is a good example of yesterday's techboard meeting discussion... I'm a
> DTS newbie wanting to contribute with a new test case, how do I get started?
> 
> If no such documentation exists, could someone please point at a couple of files
> representing what needs to be added?
In the current DTS, you can look at some documentation at doc/dts_gsg/usr_guide/intro.rst and look at hello world test case. You can also look at tests/TestSuite_hello_world.py and test_plans/hello_world_test_plan.rst. 

>