From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A489243723; Wed, 20 Dec 2023 11:33:38 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8F27E40291; Wed, 20 Dec 2023 11:33:38 +0100 (CET) Received: from mail-lj1-f178.google.com (mail-lj1-f178.google.com [209.85.208.178]) by mails.dpdk.org (Postfix) with ESMTP id DBFA540266 for ; Wed, 20 Dec 2023 11:33:36 +0100 (CET) Received: by mail-lj1-f178.google.com with SMTP id 38308e7fff4ca-2cc6eecd319so45384981fa.1 for ; Wed, 20 Dec 2023 02:33:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1703068415; x=1703673215; darn=dpdk.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=yLo6edQ4OEzPM1Nhwb8C+V/MCWoRe7BhDKUVsWDLsuI=; b=TBXjlOUtiNuXLVvwHD2LLbi7o3BkiiXIfF3Z2gGQ7YT26gGiX5AI+Q4TX402VEtlRm 2uMyooJLu3sloIbQYd1Nzw+0HMKEoscM5XBSG2/8NkO3cYykxHJafH9aP26CY8IyZco4 CciqhuEPKT+gPuktIS4GCTE8Yu92jhsflsP0l6NYUvKQQ4ySqa6F8BIyvW535McR2mDl JZ5fOxBapYtqptLSVJNu7UyvgqDagwrlSe7zaspcpjX5PKgJ04coc/fT7XWqW8Ne4e2k l1SBrhMxoFeT7Vpyg6XYJRYD2+K7U64q8jD6iQD1dS5V4EFAormCol6TJB/Bw5hY40Ss 9y9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703068415; x=1703673215; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=yLo6edQ4OEzPM1Nhwb8C+V/MCWoRe7BhDKUVsWDLsuI=; b=vkI9M//AfzEOX9rypqE9phivs/bWrz5gF5TxqCMwpUBnBdlPwQgU1he5J0Tbz2VwBN nWhoxmNAb1CX/tywrLk6Lj0bHFBm1nmw6o4qRSEaBqhv/8KiGJCb4mzSu66eVFj+qVg7 MIDBe3tXgidxnotaRnnF8WgnhB2LbAwQskDpIh4wu2qQf5fnpOOp/IpzdKY9QJ25taPa sGgoA3VwKDg6smqCt346QbETvoEjIkXm1nInjmFeTcbyHwzdCFfdTozS88NT9QU7h6sg Qmvj4/7boQ4ZmO0uyz0+jYhHInUDhXypPeiw7WtTsaldOmotPPwO+AJOJOcbLkzU1ldG nhsA== X-Gm-Message-State: AOJu0YzO6fkuSSRXkoFJrQHQIs5keHSl2c29SfdybufiQCtZvBmvYUJi QOLAfs/REoFL1ygB1gjBngh/iA== X-Google-Smtp-Source: AGHT+IES9XSmt4mbeDokCoWduU32HKCf5Tl1bkt5hdR42e8QFfRyfIVU2Muj3jwA4TT4jB5YQcbT2A== X-Received: by 2002:a05:651c:1315:b0:2cc:8e11:be79 with SMTP id u21-20020a05651c131500b002cc8e11be79mr359899lja.64.1703068415424; Wed, 20 Dec 2023 02:33:35 -0800 (PST) Received: from jlinkes-PT-Latitude-5530.pantheon.local (81.89.53.154.host.vnet.sk. [81.89.53.154]) by smtp.gmail.com with ESMTPSA id bd18-20020a056402207200b00542db304680sm12588981edb.63.2023.12.20.02.33.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Dec 2023 02:33:33 -0800 (PST) From: =?UTF-8?q?Juraj=20Linke=C5=A1?= To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, jspewock@iol.unh.edu, probb@iol.unh.edu, paul.szczepanek@arm.com, yoan.picchi@foss.arm.com, Luca.Vizzarro@arm.com Cc: dev@dpdk.org, =?UTF-8?q?Juraj=20Linke=C5=A1?= Subject: [RFC PATCH v1 0/5] test case blocking and logging Date: Wed, 20 Dec 2023 11:33:26 +0100 Message-Id: <20231220103331.60888-1-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org We currently don't store test cases that couldn't be executed because of a previous failure, such as when a test suite setup failed, resulting in no executed test cases. In order to record the test cases that couldn't be executed, we must know the lists of test suites and test cases ahead of the actual test suite execution, as an error could occur before we even start executing test suites. In addition, the patch series contains two refactors. The first refactor is closely related. The dts.py was renamed to runner.py and given a clear purpose - running the test suites and all other orchestration needed to run test suites. The logic for this was not all in the original dts.py module and it was brought there. The runner is also responsible for recording results, which is the blocked test cases are recorded. The other refactor, logging, is related to the first refactor. The logging module was simplified while extending capabilities. Each test suite logs into its own log file in addition to the main log file which the runner must handle (as it knows when we start executing particular test suites). The runner also handles the switching between execution stages for the purposes of logging. Juraj Linkeš (5): dts: convert dts.py methods to class dts: move test suite execution logic to DTSRunner dts: process test suites at the beginning of run dts: block all testcases when earlier setup fails dts: refactor logging configuration dts/framework/config/__init__.py | 8 +- dts/framework/dts.py | 228 -------- dts/framework/logger.py | 162 +++--- dts/framework/remote_session/__init__.py | 4 +- dts/framework/remote_session/os_session.py | 6 +- .../remote_session/remote/__init__.py | 7 +- .../remote/interactive_remote_session.py | 7 +- .../remote/interactive_shell.py | 7 +- .../remote_session/remote/remote_session.py | 8 +- .../remote_session/remote/ssh_session.py | 5 +- dts/framework/runner.py | 499 ++++++++++++++++++ dts/framework/test_result.py | 365 +++++++------ dts/framework/test_suite.py | 193 +------ dts/framework/testbed_model/node.py | 10 +- dts/framework/testbed_model/scapy.py | 6 +- .../testbed_model/traffic_generator.py | 5 +- dts/main.py | 12 +- dts/tests/TestSuite_smoke_tests.py | 2 +- 18 files changed, 830 insertions(+), 704 deletions(-) delete mode 100644 dts/framework/dts.py create mode 100644 dts/framework/runner.py -- 2.34.1