From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 451BC45830; Wed, 21 Aug 2024 16:53:27 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D86FD410FA; Wed, 21 Aug 2024 16:53:21 +0200 (CEST) Received: from mail-lj1-f179.google.com (mail-lj1-f179.google.com [209.85.208.179]) by mails.dpdk.org (Postfix) with ESMTP id 9585F400EF for ; Wed, 21 Aug 2024 16:53:18 +0200 (CEST) Received: by mail-lj1-f179.google.com with SMTP id 38308e7fff4ca-2f029e9c9cfso42846951fa.2 for ; Wed, 21 Aug 2024 07:53:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1724251998; x=1724856798; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NcXjYMbZaFYTGldsEqOC5XHvxVWGQz8lo1z2ZfJh+rM=; b=aUve/UGgOsNVjtBkRgaa1IorjMiruG0xMpkvGZ2AR2JW52sIpHxy49hD+4fvMaXe4Y RbB2sufGf1OShIFIkVpZsMIUSfcICTDyUisOfvpO5Bb2hAmbICX1BLnLQgOF1J8y7LCi GNnJN/abssdmWjdmpMehdDJZi3IcgiyOS0/ldsutIIp5sIOqvtienlHKtgrE8UuJT4Ai 6Dmsol4VCpi7zp+B5c4/+JM5+bJVltgwQLCUlgSN9QDkzQc9wJodZgmFk6qwl3o97xQr GORgRGvAVlZCZ4MYRnorirAE2/rqAs2TKsSI6Sv/YhuM6KI1ixa+X0aAQZjjdUH3Bvpr /rjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724251998; x=1724856798; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NcXjYMbZaFYTGldsEqOC5XHvxVWGQz8lo1z2ZfJh+rM=; b=rkZWhyBC4Pp8sYMwEg3qyJC/rpNILHtzP081MkTKEspEiA2Rf6qJuvJDG2uYskFJ8l qaHaLuTJ7CnWBH5DcHZ5eyw0J1Uz0NukhleU1uk0xcgU3mAtDOHXJlV9XjSG1+w53m5o wLfbQ6ky7ZPTlFt74x/6RRpWUiCcaTRuvGSP5Ed2STkjZ2/ESoBbTPgKKZ05TV90Xi50 uvvTr/0uqTZiYXkgp/qb/tBU9hkCE50UGJwiHzIgI23XJAKoppJ9PLgr/3MZLTw2Wfg3 0xI28CZMQDCfIwvuzpsOSi1RJ0jHMtebA+hoK6MktEUEnt8YxJIiDskEfTq95mwy+HRs /F2Q== X-Gm-Message-State: AOJu0YyfwpzW8AgsEFPzDNewXVYnrvQMje5u1w/km95etURpuuUn7D2L QNQvTjMHQipZ0VF4gYvr8ImAxBD23BZmyVkLGz6m40dFsdXR/aSuXnHm4WVJgHY= X-Google-Smtp-Source: AGHT+IE8NkCIGxmdEuaIMAlQrmqwopEqkBDXndwe1p5KNbv1rOnFypWCSqrrwNtOHqsjDYokAounEA== X-Received: by 2002:a2e:86cc:0:b0:2f3:f256:aeec with SMTP id 38308e7fff4ca-2f3f887b17cmr18279601fa.19.1724251997732; Wed, 21 Aug 2024 07:53:17 -0700 (PDT) Received: from localhost.localdomain ([84.245.121.236]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8680c4725csm52434866b.91.2024.08.21.07.53.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 07:53:17 -0700 (PDT) From: =?UTF-8?q?Juraj=20Linke=C5=A1?= To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, jspewock@iol.unh.edu, probb@iol.unh.edu, paul.szczepanek@arm.com, Luca.Vizzarro@arm.com, npratte@iol.unh.edu, dmarx@iol.unh.edu, alex.chapman@arm.com Cc: dev@dpdk.org, =?UTF-8?q?Juraj=20Linke=C5=A1?= Subject: [PATCH v3 00/12] dts: add test skipping based on capabilities Date: Wed, 21 Aug 2024 16:53:03 +0200 Message-Id: <20240821145315.97974-1-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240301155416.96960-1-juraj.linkes@pantheon.tech> References: <20240301155416.96960-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add an automated way to gather available capabilities of the tested hardware and skip test suites or cases which require capabilities that are not available. This is done through two decorators: 1. The first marks a test suite method as test case. This populates the default attributes of each test case. 2. The seconds adds the required capabilities to a test suite or case, using the attributes from 1). Two types of capabilities are added: 1. NIC capabilities. These are gathered once DPDK has been built because we use testpmd for this. It's possible to add a function that will add configuration before assessing capabilities associated with the function. This is because some capabilities return different status with different configuration present. 2. The topology capability. Each test case is marked as requiring a default topology. The required topology of a test case (or a whole test suite) may be change with the second decorator. This is how it all works: 1. The required capabilities are first all gathered from all test suites and test cases. 2. The list of required capabilities is divided into supported and unsupported capabilities. In this step, the probing of hardware and/or anything else that needs to happen to gauge whether a capability is supported is done. 3. Each test suite and test case is then marked to be skipped if any of their required capabilities are not supported. Depends-on: patch-142276 ("dts: add methods for modifying MTU to testpmd shell") Juraj Linkeš (12): dts: fix default device error handling mode dts: add the aenum dependency dts: add test case decorators dts: add mechanism to skip test cases or suites dts: add support for simpler topologies dst: add basic capability support dts: add testpmd port information caching dts: add NIC capability support dts: add topology capability doc: add DTS capability doc sources dts: add Rx offload capabilities dts: add NIC capabilities from show port info .../framework.testbed_model.capability.rst | 6 + doc/api/dts/framework.testbed_model.rst | 2 + .../dts/framework.testbed_model.topology.rst | 6 + dts/framework/remote_session/testpmd_shell.py | 461 +++++++++++++++- dts/framework/runner.py | 155 +++--- dts/framework/test_result.py | 120 +++-- dts/framework/test_suite.py | 161 +++++- dts/framework/testbed_model/capability.py | 491 ++++++++++++++++++ dts/framework/testbed_model/node.py | 2 +- dts/framework/testbed_model/port.py | 4 +- dts/framework/testbed_model/topology.py | 128 +++++ dts/poetry.lock | 14 +- dts/pyproject.toml | 1 + dts/tests/TestSuite_hello_world.py | 10 +- dts/tests/TestSuite_os_udp.py | 3 +- dts/tests/TestSuite_pmd_buffer_scatter.py | 14 +- dts/tests/TestSuite_smoke_tests.py | 8 +- 17 files changed, 1429 insertions(+), 157 deletions(-) create mode 100644 doc/api/dts/framework.testbed_model.capability.rst create mode 100644 doc/api/dts/framework.testbed_model.topology.rst create mode 100644 dts/framework/testbed_model/capability.py create mode 100644 dts/framework/testbed_model/topology.py -- 2.34.1