From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 52104A0524 for ; Fri, 5 Feb 2021 21:37:32 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4D2FC40682; Fri, 5 Feb 2021 21:37:32 +0100 (CET) Received: from mail-ot1-f98.google.com (mail-ot1-f98.google.com [209.85.210.98]) by mails.dpdk.org (Postfix) with ESMTP id 8F1BB4067B for ; Fri, 5 Feb 2021 21:37:30 +0100 (CET) Received: by mail-ot1-f98.google.com with SMTP id v1so8127704ott.10 for ; Fri, 05 Feb 2021 12:37:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rvqcpMlc6f08JH77yz44k3okogLWU3+2djo0FrhykZY=; b=NkVatmjIbKD8kPrITjoXfQp4gWZ94xIq+Ey/gTNetVxE3TiblFIlg30GnzO+PUnSlP 3Cv5PRh/DZK3W8JIeyZk6GuR8q4vTUBqL//f5i/O+mDSVS4Lb3ysXg/0DjTIYIxe6HQs ZT6jIMZycnsV+a/TqM0EtBlNpaJQXn6j1E5OQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rvqcpMlc6f08JH77yz44k3okogLWU3+2djo0FrhykZY=; b=m/M7Ja6+UErzxe0lDth6pgJS7waCLmzy+sqoHUVH9d4A0gFKsFOOzCDnNQNO6W/qbG BQUsI+4/uFLw/pFzG9SV9CF+9eO1a5kEFEtxbNTlhxodKg4ntZ7bjPu+B/P0jdTwefqS fVyePHTSYYlV7EEdnBA9gMBg1iNFY4tKOobBBD7o0d8LKdIDDoF/SS1TNcvFffpmi5hy O6ul0F/WyUTbr/CFo/jQNY8eDPk81r1AHLKB9YOCgsvMf/rqMm92wEXDKCJKrQdNATQh epSAKNMDTIbxaer4xqRQ3pIpht8MZimVeU1KCNXqILkAFZCZJgJP+AZKtgKwghnoNXYO X/ZA== X-Gm-Message-State: AOAM532WNQhtzExpTDsO2GuVBNbU4voi1kOhV+xmj/fQeHYfnEG4fRFn KDMmn2bYXtXEViyKxQ1lOQu5AqYjWo4sXflobaw5yv/BA+VozdUV+GCKxoF2aoGUe1fdQtH8yNy I2ZPS68FTVvl25d2KGTisecbLCx2aB5rtsUGuUyfKbWhOcesxThtfuGP8nX/O+GQn+/njVpHw X-Google-Smtp-Source: ABdhPJwlTw7X7vrccxOC+5OPbGPanuqPCvtehdhUcUQoNkj7FuKsGDxLk3eId47hqI6r34ogecFy4jFj85xK X-Received: by 2002:a9d:7699:: with SMTP id j25mr4639879otl.202.1612557449792; Fri, 05 Feb 2021 12:37:29 -0800 (PST) Received: from postal.iol.unh.edu (postal.iol.unh.edu. [132.177.123.84]) by smtp-relay.gmail.com with ESMTPS id s7sm2223717otd.10.2021.02.05.12.37.29 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 05 Feb 2021 12:37:29 -0800 (PST) X-Relaying-Domain: iol.unh.edu Received: from iol.unh.edu (unknown [IPv6:2606:4100:3880:1254::1033]) by postal.iol.unh.edu (Postfix) with ESMTP id 390006052472; Fri, 5 Feb 2021 15:37:29 -0500 (EST) From: ohilyard@iol.unh.edu To: ci@dpdk.org Cc: thomas@monjalon.net, Owen Hilyard Date: Fri, 5 Feb 2021 15:37:25 -0500 Message-Id: <20210205203725.143804-2-ohilyard@iol.unh.edu> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210205203725.143804-1-ohilyard@iol.unh.edu> References: <20210205203725.143804-1-ohilyard@iol.unh.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: [dpdk-ci] [PATCH 2/2] patch-tagging: Added execution file creator X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" From: Owen Hilyard The new config file contains mappings from a tag to a list of tests to be run. This will be where the community can configure which test suites should run for which tag. Additional tags can be created freely, since the script just performs a dictionary lookup. The script uses the provided execution file for 2 things. First, uses it to provide all of the other fields in all of the configs. This is to allow preserving configuration information for dts. The otehr way the scripts are used is that the lists of test suites already there is treated as a filter. This was done to allow a CI implimentation to only run test suites that they have verified are functional on their hardware. The resulting execution file will have a "test_suites" field which is the lexographically sorted intersection between this "filter" list and the tests that are configured to run for the provided tags. The script will also append ":func=true:perf=false" or ":func=false:perf=true" onto the parameters as appropriate. Signed-off-by: Owen Hilyard --- config/tests_for_tag.cfg | 33 ++++++ tools/create_new_execution_file_from_tags.py | 113 +++++++++++++++++++ 2 files changed, 146 insertions(+) create mode 100644 config/tests_for_tag.cfg create mode 100755 tools/create_new_execution_file_from_tags.py diff --git a/config/tests_for_tag.cfg b/config/tests_for_tag.cfg new file mode 100644 index 0000000..01c497d --- /dev/null +++ b/config/tests_for_tag.cfg @@ -0,0 +1,33 @@ +[functional] +core = dynamic_config, + link_status_interrupt, + mac_filter, + mtu_update, + scatter, + stats_checks, + stats_checks, + unit_tests_mbuf +driver = dynamic_config, + link_status_interrupt, + mac_filter, + mtu_update, + scatter, + stats_checks, + stats_checks, + unit_tests_mbuf +application = dynamic_config, + link_status_interrupt, + mac_filter, + mtu_update, + scatter, + stats_checks, + stats_checks, + unit_tests_mbuf +; Nothing in documentation +documentation = + +[performance] +core = nic_single_core_perf +driver = nic_single_core_perf +application = nic_single_core_perf +documentation = diff --git a/tools/create_new_execution_file_from_tags.py b/tools/create_new_execution_file_from_tags.py new file mode 100755 index 0000000..af24615 --- /dev/null +++ b/tools/create_new_execution_file_from_tags.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python3 + +# BSD LICENSE +# +# Copyright(c) 2020 Intel Corporation. All rights reserved. +# Copyright © 2018[, 2019] The University of New Hampshire. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +import sys +from enum import Enum + +import itertools +from configparser import ConfigParser +from typing import List, Dict, Set +import argparse + + +def parse_comma_delimited_list_from_string(mod_str: str) -> List[str]: + return list(map(str.strip, mod_str.split(','))) + + +def map_tags_to_tests(tags: List[str], test_map: Dict[str, List[str]]) -> Set[str]: + """ + Returns a list that is the union of all of the map lookups. + """ + try: + return set( + filter(lambda test: test != '', set(itertools.chain.from_iterable(map(lambda tag: test_map[tag], tags))))) + except KeyError as e: + print(f'Tag {e} is not present in tests_for_tag.cfg') + exit(1) + +class TestingType(Enum): + functional = 'functional' + performance = 'performance' + + def __str__(self): + return self.value + + +if __name__ == '__main__': + parser = argparse.ArgumentParser( + description='Take a template execution file and add the relevant tests' + ' for the given tags to it, creating a new file.') + parser.add_argument('config_file_path', help='The path to tests_for_tag.cfg', default='config/tests_for_tag.cfg') + parser.add_argument('template_execution_file', help='The path to the execution file to use as a template') + parser.add_argument('output_path', help='The path to the output execution file') + parser.add_argument('testing_type', type=TestingType, choices=list(TestingType), + help='What type of testing to create an execution file for') + parser.add_argument('tags', metavar='tag', type=str, nargs='+', help='The tags to create an execution file for.') + + args = parser.parse_args() + + tag_to_test_map_parser = ConfigParser() + tag_to_test_map_parser.read(args.config_file_path) + + template_execution_file_parser = ConfigParser() + template_execution_file_parser.read(args.template_execution_file) + + test_map = {key: parse_comma_delimited_list_from_string(value.strip()) for key, value in + tag_to_test_map_parser[str(args.testing_type)].items()} + + tests = map_tags_to_tests(args.tags, test_map) + + try: + output_file = open(args.output_path, 'x') + except FileExistsError: + output_file = open(args.output_path, 'w') + + for execution_plan in template_execution_file_parser: + # The DEFAULT section is always present and contains top-level items, so it needs to be ignored + if execution_plan != 'DEFAULT': + test_allowlist = parse_comma_delimited_list_from_string(template_execution_file_parser[execution_plan]['test_suites']) + tests_to_run = list(set(test_allowlist).intersection(tests)) + tests_to_run.sort() + template_execution_file_parser[execution_plan]['test_suites'] = ", ".join(tests_to_run) + + if args.testing_type == TestingType.functional: + template_execution_file_parser[execution_plan]['parameters'] += ':func=true:perf=false' + elif args.testing_type == TestingType.performance: + template_execution_file_parser[execution_plan]['parameters'] += ':func=false:perf=true' + else: + # This code should be unreachable, since this is checked at the top of the file + print("Fatal error: testing type is neither performance nor functional", file=sys.stderr) + exit(1) + + template_execution_file_parser.write(output_file) + -- 2.27.0