From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E3B5543C0C; Fri, 1 Mar 2024 17:11:36 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CEC67402EA; Fri, 1 Mar 2024 17:11:36 +0100 (CET) Received: from mail-oo1-f52.google.com (mail-oo1-f52.google.com [209.85.161.52]) by mails.dpdk.org (Postfix) with ESMTP id 982D8402E0 for ; Fri, 1 Mar 2024 17:11:34 +0100 (CET) Received: by mail-oo1-f52.google.com with SMTP id 006d021491bc7-5a04fb5e689so1068148eaf.1 for ; Fri, 01 Mar 2024 08:11:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1709309494; x=1709914294; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=QIgb52jeGVaCgdlbMnRLtXSYANU2FWbHiclFlrn4EBM=; b=O0VzKTcJ+omQ0r5B7ZD1r/AyT8uTlq6raOovyenEYGLoP2DKK/RZLwonQVbxE12bnR rcT7RPSDtRG4+NfJBfe1XvDlUHrIgbtO+oL8PjC/t6nTmtZ/e0goeQos4ymcVcY+DHrG sWLV3HoUeBgMxfkbdIE0hFvhRx79pN9P5X3Qk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709309494; x=1709914294; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=QIgb52jeGVaCgdlbMnRLtXSYANU2FWbHiclFlrn4EBM=; b=dqvKPACAljpXBQ8wxrsYmuYvBy9m7G4AyFlnSOo4K78dJ7rULjKZXz+x9guHMP+B3e JsJLggnEfpvuI2DvMmhRvLkmeEHsyfYjFnoVd1J43P3HS0T8XGP8l7+lEfWj1kDyRFTM 88UYpo0s6vlbFmO4epmWa491W2+JXI1JlTK1U7hc7+eyhhX2wTLPowtKscMCs6k1T8yQ neB39tQqzJj/x9m0JlPEwAVMMYHr+hyEg7f2HAIL9QwLGyI7EGqhpttOyFzhwwAi0Qhd 4CkHwr5JHJ7LDua1iIgyVeI03LF+UeXSfg9VzghU9uteSsN+vSXq+5TKGj8nQb/ECqeO aqQg== X-Forwarded-Encrypted: i=1; AJvYcCUcOu3CJnYmt8LbaiBLySaY+Mms30wvakdM/5Xdbz6NKbs5Hv+KXN0LvgnQHUdoRvp0wrcd2U3ueDEgDGQ= X-Gm-Message-State: AOJu0YyaT5kg7aifHdc9BOZA98phlpL2GtYzSms+VcGJjpqdhKFhG5gG l5QSDUG5hfB+YpTRgTgFd8UbztM5x+3uyq3VY3XPyKdDSBGGUDDhUir4h4vlciHsmkl2LVqlxJj rxEqUnj20ZpUO2lQZw1fV05Y+5O2OJIP+wBKupQ== X-Google-Smtp-Source: AGHT+IEt1/Gzr951qQAvCYXgk/XzL9ZyKz1+590gBKhruJ5fG2MpOVXt4acX8JxR4jSZPmpt2jcWQPzMpJi0qD85qPI= X-Received: by 2002:a4a:9213:0:b0:5a0:eda2:8a62 with SMTP id f19-20020a4a9213000000b005a0eda28a62mr1848654ooh.9.1709309493907; Fri, 01 Mar 2024 08:11:33 -0800 (PST) MIME-Version: 1.0 References: <20231220103331.60888-1-juraj.linkes@pantheon.tech> <20240301105522.79870-1-juraj.linkes@pantheon.tech> In-Reply-To: <20240301105522.79870-1-juraj.linkes@pantheon.tech> From: Patrick Robb Date: Fri, 1 Mar 2024 11:11:23 -0500 Message-ID: Subject: Re: [PATCH v4 0/7] test case blocking and logging To: =?UTF-8?Q?Juraj_Linke=C5=A1?= Cc: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, jspewock@iol.unh.edu, paul.szczepanek@arm.com, Luca.Vizzarro@arm.com, npratte@iol.unh.edu, dev@dpdk.org Content-Type: multipart/alternative; boundary="0000000000003227e006129ba1af" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --0000000000003227e006129ba1af Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable The Community CI Testing Lab had an infra failure this morning and some patches including yours were affected with false failures. The issue is now resolved and we are rerunning the tests in question for all patches submitted today. On Fri, Mar 1, 2024 at 5:55=E2=80=AFAM Juraj Linke=C5=A1 wrote: > We currently don't record test case results that couldn't be executed > because of a previous failure, such as when a test suite setup failed, > resulting in no executed test cases. > > In order to record the test cases that couldn't be executed, we must > know the lists of test suites and test cases ahead of the actual test > suite execution, as an error could occur before we even start executing > test suites. > > In addition, the patch series contains two refactors and one > improvement. > > The first refactor is closely related. The dts.py was renamed to > runner.py and given a clear purpose - running the test suites and all > other orchestration needed to run test suites. The logic for this was > not all in the original dts.py module and it was brought there. The > runner is also responsible for recording results, which is the blocked > test cases are recorded. > > The other refactor, logging, is related to the first refactor. The > logging module was simplified while extending capabilities. Each test > suite logs into its own log file in addition to the main log file which > the runner must handle (as it knows when we start executing particular > test suites). The runner also handles the switching between execution > stages for the purposes of logging. > > The one aforementioned improvement is in unifying how we specify test > cases in the conf.yaml file and in the environment variable/command line > argument. > > v2: > Rebase and update of the whole patch. There are changes in all parts of > the code, mainly improving the design and logic. > Also added the last patch which improves test suite specification on the > cmdline. > > v3: > Improved variables in _get_test_suite_class along with docstring. > Fixed smoke test suite not being added into the list of test suites to > be executed. > > v4: > Added the checking of CamelCase convention when discovering test cases. > Also added additional test stages. The stages were split into setup and > teardown. > > Juraj Linke=C5=A1 (7): > dts: convert dts.py methods to class > dts: move test suite execution logic to DTSRunner > dts: filter test suites in executions > dts: reorganize test result > dts: block all test cases when earlier setup fails > dts: refactor logging configuration > dts: improve test suite and case filtering > > doc/guides/tools/dts.rst | 14 +- > dts/framework/config/__init__.py | 36 +- > dts/framework/config/conf_yaml_schema.json | 2 +- > dts/framework/dts.py | 338 --------- > dts/framework/logger.py | 246 +++--- > dts/framework/remote_session/__init__.py | 6 +- > .../interactive_remote_session.py | 6 +- > .../remote_session/interactive_shell.py | 6 +- > .../remote_session/remote_session.py | 8 +- > dts/framework/runner.py | 711 ++++++++++++++++++ > dts/framework/settings.py | 188 +++-- > dts/framework/test_result.py | 565 ++++++++------ > dts/framework/test_suite.py | 239 +----- > dts/framework/testbed_model/node.py | 11 +- > dts/framework/testbed_model/os_session.py | 7 +- > .../traffic_generator/traffic_generator.py | 6 +- > dts/main.py | 9 +- > dts/pyproject.toml | 3 + > dts/tests/TestSuite_os_udp.py | 2 +- > dts/tests/TestSuite_smoke_tests.py | 2 +- > 20 files changed, 1375 insertions(+), 1030 deletions(-) > delete mode 100644 dts/framework/dts.py > create mode 100644 dts/framework/runner.py > > -- > 2.34.1 > > --0000000000003227e006129ba1af Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
The Community CI Testing Lab had an infra= failure this morning and some patches including yours were affected with f= alse failures. The issue is now resolved and we are rerunning the tests in = question for all patches submitted today.=C2=A0

On Fri, Mar 1, 2024 at 5= :55=E2=80=AFAM Juraj Linke=C5=A1 <juraj.linkes@pantheon.tech> wrote:<= br>
We currently don= 't record test case results that couldn't be executed
because of a previous failure, such as when a test suite setup failed,
resulting in no executed test cases.

In order to record the test cases that couldn't be executed, we must know the lists of test suites and test cases ahead of the actual test
suite execution, as an error could occur before we even start executing
test suites.

In addition, the patch series contains two refactors and one
improvement.

The first refactor is closely related. The dts.py was renamed to
runner.py and given a clear purpose - running the test suites and all
other orchestration needed to run test suites. The logic for this was
not all in the original dts.py module and it was brought there. The
runner is also responsible for recording results, which is the blocked
test cases are recorded.

The other refactor, logging, is related to the first refactor. The
logging module was simplified while extending capabilities. Each test
suite logs into its own log file in addition to the main log file which
the runner must handle (as it knows when we start executing particular
test suites). The runner also handles the switching between execution
stages for the purposes of logging.

The one aforementioned improvement is in unifying how we specify test
cases in the conf.yaml file and in the environment variable/command line argument.

v2:
Rebase and update of the whole patch. There are changes in all parts of
the code, mainly improving the design and logic.
Also added the last patch which improves test suite specification on the cmdline.

v3:
Improved variables in _get_test_suite_class along with docstring.
Fixed smoke test suite not being added into the list of test suites to
be executed.

v4:
Added the checking of CamelCase convention when discovering test cases.
Also added additional test stages. The stages were split into setup and
teardown.

Juraj Linke=C5=A1 (7):
=C2=A0 dts: convert dts.py methods to class
=C2=A0 dts: move test suite execution logic to DTSRunner
=C2=A0 dts: filter test suites in executions
=C2=A0 dts: reorganize test result
=C2=A0 dts: block all test cases when earlier setup fails
=C2=A0 dts: refactor logging configuration
=C2=A0 dts: improve test suite and case filtering

=C2=A0doc/guides/tools/dts.rst=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 14 +-
=C2=A0dts/framework/config/__init__.py=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 |=C2=A0 36 +-
=C2=A0dts/framework/config/conf_yaml_schema.json=C2=A0 =C2=A0 |=C2=A0 =C2= =A02 +-
=C2=A0dts/framework/dts.py=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | 338 ---------
=C2=A0dts/framework/logger.py=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| 246 +++---
=C2=A0dts/framework/remote_session/__init__.py=C2=A0 =C2=A0 =C2=A0 |=C2=A0 = =C2=A06 +-
=C2=A0.../interactive_remote_session.py=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0|=C2=A0 =C2=A06 +-
=C2=A0.../remote_session/interactive_shell.py=C2=A0 =C2=A0 =C2=A0 =C2=A0|= =C2=A0 =C2=A06 +-
=C2=A0.../remote_session/remote_session.py=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 |=C2=A0 =C2=A08 +-
=C2=A0dts/framework/runner.py=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| 711 ++++++++++++++++++
=C2=A0dts/framework/settings.py=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| 188 +++--
=C2=A0dts/framework/test_result.py=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 | 565 ++++++++------
=C2=A0dts/framework/test_suite.py=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0| 239 +-----
=C2=A0dts/framework/testbed_model/node.py=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0|=C2=A0 11 +-
=C2=A0dts/framework/testbed_model/os_session.py=C2=A0 =C2=A0 =C2=A0|=C2=A0 = =C2=A07 +-
=C2=A0.../traffic_generator/traffic_generator.py=C2=A0 =C2=A0 |=C2=A0 =C2= =A06 +-
=C2=A0dts/main.py=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2= =A0 =C2=A09 +-
=C2=A0dts/pyproject.toml=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A03 +
=C2=A0dts/tests/TestSuite_os_udp.py=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A02 +-
=C2=A0dts/tests/TestSuite_smoke_tests.py=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 |=C2=A0 =C2=A02 +-
=C2=A020 files changed, 1375 insertions(+), 1030 deletions(-)
=C2=A0delete mode 100644 dts/framework/dts.py
=C2=A0create mode 100644 dts/framework/runner.py

--
2.34.1



--0000000000003227e006129ba1af--