From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B50CA43A73; Tue, 6 Feb 2024 15:57:19 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A4DCB42686; Tue, 6 Feb 2024 15:57:19 +0100 (CET) Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) by mails.dpdk.org (Postfix) with ESMTP id 5485C41611 for ; Tue, 6 Feb 2024 15:57:18 +0100 (CET) Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-a2f22bfb4e6so751546766b.0 for ; Tue, 06 Feb 2024 06:57:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1707231438; x=1707836238; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2pzkO9Heft0aBxsalVeUF3waLc2hz7IHO6KxL3HpTBQ=; b=NhqRkzFzOJU5bD4j06xIdok0ETXmtJ4RREsKWBgE4zwKGKYfSS9d3pFJhdMvt9miT8 7ERKCzBM2DKJkyK1Y1Pr+gj6uWXqQF0m2GQP1aF0En26KQmDnlRWmQIKrf0oTWnl5JQ5 wwsoY8agpKKqgZ/V1adXZi7N9qcBp2FutHvBC+Tcec4bdc4FL49C0TP9xMGan4KRMldA 4ikTe/Mk11iibCHlJpJPlO4O6JOuIu/qNq4WEZH6wAWWTEa2Bj427yEaH8cUTU8fYizF y3yzAKS+w3pG4SINDA3mgSJado1t+89Rr38qP8GCSTI1/Pr6oNqY7xETDxZJqEugcd9/ Uyjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707231438; x=1707836238; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2pzkO9Heft0aBxsalVeUF3waLc2hz7IHO6KxL3HpTBQ=; b=cOEBvwe2/y6i7t9UIMcZ+IfAhmHxMB37XCt22Hz36EfoXgO21YHOTk8pTrJIlPt+z/ fKHfv8Q8KXhFwpupws2wvu1/0IueQ82HXosezQNkLUf/4XRyDBUQJsPG+Aazpbm8hub3 W7PV8xE94wQe1tSOGSFRicNMZQw3lzD7tOqRr5DZwCkDGmPzfXZK/6FxOqXSMTrPgXx1 sadfNjR/xrP/LK2dIRVlYgTJxsomjLUMhcBaBPagbv8PRQxKUE6bExu3jPx7aniJpIaf eJFgQr1K0bbwRoQzvGJSBuPNk/NXvbKtLEFTkC083/DN2Qr860wxoK69E8oO/LpSg19H AUUg== X-Gm-Message-State: AOJu0YybkXu9T7DDyecuhI/1Kmolh8mXCFf0ygYzYiiRS2XyVfEzNDKh Hv0MOpxTfQo7MvU2obmfahNCrmpCu+1kDPWoivlvaKzXDSg9w4TaMn/LnoPAFRE= X-Google-Smtp-Source: AGHT+IHDl4co34MOnZMv7EmDwhneb6Ms52tgZDaRm7x2P+lpnY4eykdaHC0s2vqcFlciIdDfyypvMA== X-Received: by 2002:a17:906:3b8b:b0:a37:6303:18b8 with SMTP id u11-20020a1709063b8b00b00a37630318b8mr1905579ejf.54.1707231437910; Tue, 06 Feb 2024 06:57:17 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCWgyFuRJ0gKBP3AjIR4UEWwW9bI56SFM/gR3Ap9qeTBNfA8CGoLYn7Lxv3VUTMgfFWgyh9QkIiC+zpPvgPzZJ8utIZTnDbHrW5OhGui+9pSQy6kkIpNpyS31OBc8uVpDwH/Wn9vXkkjwK/ppszEg0mNx+RboUWfaAYKjCKFPEGeKvfgcNUfd2SSZPpCmScbYD3/vG20B4hsB+zpwfLGRxyckTWgxt/eBJQKtCjs/atmTzfjj6su6heeb7cCbpH4wmym Received: from localhost.localdomain ([84.245.120.62]) by smtp.gmail.com with ESMTPSA id lg25-20020a170907181900b00a36c5b01ef3sm1220786ejc.225.2024.02.06.06.57.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 06:57:17 -0800 (PST) From: =?UTF-8?q?Juraj=20Linke=C5=A1?= To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, jspewock@iol.unh.edu, probb@iol.unh.edu, paul.szczepanek@arm.com, Luca.Vizzarro@arm.com Cc: dev@dpdk.org, =?UTF-8?q?Juraj=20Linke=C5=A1?= Subject: [PATCH v2 0/7] test case blocking and logging Date: Tue, 6 Feb 2024 15:57:09 +0100 Message-Id: <20240206145716.71435-1-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231220103331.60888-1-juraj.linkes@pantheon.tech> References: <20231220103331.60888-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org We currently don't record test case results that couldn't be executed because of a previous failure, such as when a test suite setup failed, resulting in no executed test cases. In order to record the test cases that couldn't be executed, we must know the lists of test suites and test cases ahead of the actual test suite execution, as an error could occur before we even start executing test suites. In addition, the patch series contains two refactors and one improvement. The first refactor is closely related. The dts.py was renamed to runner.py and given a clear purpose - running the test suites and all other orchestration needed to run test suites. The logic for this was not all in the original dts.py module and it was brought there. The runner is also responsible for recording results, which is the blocked test cases are recorded. The other refactor, logging, is related to the first refactor. The logging module was simplified while extending capabilities. Each test suite logs into its own log file in addition to the main log file which the runner must handle (as it knows when we start executing particular test suites). The runner also handles the switching between execution stages for the purposes of logging. The one aforementioned improvement is in unifying how we specify test cases in the conf.yaml file and in the environment variable/command line argument. v2: Rebase and update of the whole patch. There are changes in all parts of the code, mainly improving the design and logic. Also added the last patch which improves test suite specification on the cmdline. Juraj Linkeš (7): dts: convert dts.py methods to class dts: move test suite execution logic to DTSRunner dts: filter test suites in executions dts: reorganize test result dts: block all test cases when earlier setup fails dts: refactor logging configuration dts: improve test suite and case filtering doc/guides/tools/dts.rst | 14 +- dts/framework/config/__init__.py | 36 +- dts/framework/config/conf_yaml_schema.json | 2 +- dts/framework/dts.py | 338 --------- dts/framework/logger.py | 235 +++--- dts/framework/remote_session/__init__.py | 6 +- .../interactive_remote_session.py | 6 +- .../remote_session/interactive_shell.py | 6 +- .../remote_session/remote_session.py | 8 +- dts/framework/runner.py | 701 ++++++++++++++++++ dts/framework/settings.py | 188 +++-- dts/framework/test_result.py | 565 ++++++++------ dts/framework/test_suite.py | 239 +----- dts/framework/testbed_model/node.py | 11 +- dts/framework/testbed_model/os_session.py | 7 +- .../traffic_generator/traffic_generator.py | 6 +- dts/main.py | 9 +- dts/pyproject.toml | 3 + dts/tests/TestSuite_smoke_tests.py | 2 +- 19 files changed, 1354 insertions(+), 1028 deletions(-) delete mode 100644 dts/framework/dts.py create mode 100644 dts/framework/runner.py -- 2.34.1