From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4AD4145A1C; Tue, 24 Sep 2024 10:20:16 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id ECD6F40295; Tue, 24 Sep 2024 10:20:15 +0200 (CEST) Received: from mail-lj1-f178.google.com (mail-lj1-f178.google.com [209.85.208.178]) by mails.dpdk.org (Postfix) with ESMTP id 1FA7C4028E for ; Tue, 24 Sep 2024 10:20:15 +0200 (CEST) Received: by mail-lj1-f178.google.com with SMTP id 38308e7fff4ca-2f75e5f3debso47370911fa.1 for ; Tue, 24 Sep 2024 01:20:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1727166014; x=1727770814; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Lr8LPFAnmRkndXqgAJP+NGwfSM89gbdlRWYcs6e+apk=; b=pl0MIi6VhwqOQu2643e/KSL/PsEsnBaYtulJJWiTfE+zhDow3OzbAANhCnhiNGfoYI FdEn2r1ka0cZOrydPiwg5hwc/Jn5+Hboe416PS7cAfr70lqw3nXcHsHGFvLtEYQ4AS9m /jJZc5WeT9jTs19hUO3RhMCqQuslFIsJ+MZYjxJPrpgnb4LU84x69dZn/YHp9Cx+ITiR jHHyyVzL/9yjUBdv6E3yrBmBpqRRXBOH2ipmkGeuCxMG10mw7LVIPuxX1H7BMQzgzHvE sTCx7NqmyAavWlV1ZjqwtyW03ZYSwi2ltaXCXcR8gODD87fvh7n0xRB/N2wWKqS/5L+r YBHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727166014; x=1727770814; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Lr8LPFAnmRkndXqgAJP+NGwfSM89gbdlRWYcs6e+apk=; b=kByUl/n52kNEvpoWBRw4onSsNgQF9tXBwdKjUCCASLtsOM3qzxtfkdFVdi7Rk+iPIa Zzu9k6VNK1yNF/loFDiY9v6gh/eIzCcbQqGA8oZaen5+nidTEWeHa3vH7H0E3lBuTtSq IaN7cdayDzJF2oPzH0muPETIDSXLnE+7O1LSgHFbMdkGJPeT0P7qnTDHrUprn+sPE0ys exDnbDqJn8J900PpRDAMWKnWMfA8eecPQKXsc8wMP+quNrqx3aE2ziNYIIBPqI46+sqe Quz32DihtA3C1Nzx9inxlXB1jgHvpA1SoXvsAd69MXPpoSnXkRiKB2u2VCS4Px7yW9f6 vJ8Q== X-Gm-Message-State: AOJu0YwxXoSZIz+Nt2vW2Qhv6z+BQeMKApGo1ymukAPerYao6k9eXkCO oWjApQCMtlLc0VJJZ4ZadVPRrZEqEvzyBptYdG6DjwpkTdoDrzn2kGP7ql0OtzU= X-Google-Smtp-Source: AGHT+IFWU9LhBulkMHIDGGpIA6RwH4548Oa4Gxv5N+ISYrH4jrU6e4EuQCyKXFErgjAhEP7WGVibUA== X-Received: by 2002:a2e:a983:0:b0:2f7:5900:1a2e with SMTP id 38308e7fff4ca-2f7cb2cfe60mr73752571fa.3.1727166014283; Tue, 24 Sep 2024 01:20:14 -0700 (PDT) Received: from jlinkes-PT-Latitude-5530.pantheon.local (81.89.53.154.host.vnet.sk. [81.89.53.154]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5c5cf497295sm490997a12.29.2024.09.24.01.20.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Sep 2024 01:20:13 -0700 (PDT) From: =?UTF-8?q?Juraj=20Linke=C5=A1?= To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, paul.szczepanek@arm.com, Luca.Vizzarro@arm.com, alex.chapman@arm.com, probb@iol.unh.edu, jspewock@iol.unh.edu, npratte@iol.unh.edu, dmarx@iol.unh.edu Cc: dev@dpdk.org, =?UTF-8?q?Juraj=20Linke=C5=A1?= Subject: [PATCH v4 00/11] dts: add test skipping based on capabilities Date: Tue, 24 Sep 2024 10:20:08 +0200 Message-ID: <20240924082008.14763-1-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240301155416.96960-1-juraj.linkes@pantheon.tech> References: <20240301155416.96960-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add an automated way to gather available capabilities of the tested hardware and skip test suites or cases which require capabilities that are not available. This is done through two decorators: 1. The first marks a test suite method as test case. This populates the default attributes of each test case. 2. The seconds adds the required capabilities to a test suite or case, using the attributes from 1). Two types of capabilities are added: 1. NIC capabilities. These are gathered once DPDK has been built because we use testpmd for this. It's possible to add a function that will add configuration before assessing capabilities associated with the function. This is because some capabilities return different status with different configuration present. 2. The topology capability. Each test case is marked as requiring a default topology. The required topology of a test case (or a whole test suite) may be change with the second decorator. This is how it all works: 1. The required capabilities are first all gathered from all test suites and test cases. 2. The list of required capabilities is divided into supported and unsupported capabilities. In this step, the probing of hardware and/or anything else that needs to happen to gauge whether a capability is supported is done. 3. Each test suite and test case is then marked to be skipped if any of their required capabilities are not supported. v4: Rebased on next-dts. Adjusted the decorator for setting and reverting 9000 MTU to be parametrized. Added references to DPDK libraries and testpmd code to testpmd command parsing methods. Updated capability docstrings. Moved the capability decorator to NicCapability. Juraj Linkeš (11): dts: add the aenum dependency dts: add test case decorators dts: add mechanism to skip test cases or suites dts: add support for simpler topologies dts: add basic capability support dts: add NIC capability support dts: add NIC capabilities from show rxq info dts: add topology capability doc: add DTS capability doc sources dts: add Rx offload capabilities dts: add NIC capabilities from show port info .../framework.testbed_model.capability.rst | 6 + doc/api/dts/framework.testbed_model.rst | 2 + .../dts/framework.testbed_model.topology.rst | 6 + dts/framework/remote_session/testpmd_shell.py | 461 +++++++++++++++- dts/framework/runner.py | 155 +++--- dts/framework/test_result.py | 119 ++-- dts/framework/test_suite.py | 161 +++++- dts/framework/testbed_model/capability.py | 507 ++++++++++++++++++ dts/framework/testbed_model/node.py | 2 +- dts/framework/testbed_model/port.py | 4 +- dts/framework/testbed_model/topology.py | 127 +++++ dts/poetry.lock | 14 +- dts/pyproject.toml | 1 + dts/tests/TestSuite_hello_world.py | 10 +- dts/tests/TestSuite_os_udp.py | 3 +- dts/tests/TestSuite_pmd_buffer_scatter.py | 14 +- dts/tests/TestSuite_smoke_tests.py | 8 +- 17 files changed, 1447 insertions(+), 153 deletions(-) create mode 100644 doc/api/dts/framework.testbed_model.capability.rst create mode 100644 doc/api/dts/framework.testbed_model.topology.rst create mode 100644 dts/framework/testbed_model/capability.py create mode 100644 dts/framework/testbed_model/topology.py -- 2.43.0