From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C64CDA0032; Tue, 13 Sep 2022 19:19:54 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5F6A94021D; Tue, 13 Sep 2022 19:19:54 +0200 (CEST) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id 1FD2040151 for ; Tue, 13 Sep 2022 19:19:51 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663089592; x=1694625592; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=BdRsTgLTd67vEidH4RwMUnG95OtFSS2+RUS7wu7S06Y=; b=gAYPMEFGEV6EmFEBUPEbF69vls5jB6F63Ct5SAF7cChVaC/U0SDtK3d2 Swk9uKyVVEiOkjxWks9szc9RzVUShKQ0y7Bjf56N7W4iWh2gGJuiMVYZF m6xYk4ycIm7bY/TpWy1yRYo7+tcBcZMY/THop73J3HoD8NIEfuQzTHald jg9HpqkU5sZB/Z1voOR0MFniibp6BgEnmuV6JI0YdYSwg3nDBXBCR1YID 6uoMw0bKHzyjc+NwdpwydCOsfRlbQT6Sc1u3GgQD+AjMy7ETxI1OSLkqh 9VnlYqE2RIpzCm2ijH5vODHsIgiGaHjvVAUkzeNSl9K+KHQXzL7gOUtsu A==; X-IronPort-AV: E=McAfee;i="6500,9779,10469"; a="296933909" X-IronPort-AV: E=Sophos;i="5.93,313,1654585200"; d="scan'208";a="296933909" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Sep 2022 10:19:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,313,1654585200"; d="scan'208";a="594028361" Received: from orsmsx601.amr.corp.intel.com ([10.22.229.14]) by orsmga006.jf.intel.com with ESMTP; 13 Sep 2022 10:19:50 -0700 Received: from orsmsx601.amr.corp.intel.com (10.22.229.14) by ORSMSX601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Tue, 13 Sep 2022 10:19:50 -0700 Received: from ORSEDG602.ED.cps.intel.com (10.7.248.7) by orsmsx601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31 via Frontend Transport; Tue, 13 Sep 2022 10:19:50 -0700 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (104.47.58.168) by edgegateway.intel.com (134.134.137.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2375.31; Tue, 13 Sep 2022 10:19:49 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TApaCL2nbI1fbH38LtpZhYKAh3f3qhy43cObSXDrA16Kp7z04tRoKobs9JeKaqHqRQQjWJ0pqeTVMwoi+uzsJAPXKOlsoBloXu7lWVmcAeUsI7kK8NuRoz8jK0J76bVv5tRaPRLK8kd0TMRarhSCNrtPRsIbVxYBrHAdMkpvXBkfui18fdfIphpxhEn5rcEhBEuVDjigOL9t1xjKORHWcPEPlHvLMPUWNFvj8tQn4z3gZ6zfK+e4EdoQV9lUQ5u3f4DwAUDY2dAkUJYRp3oDYWd97FcvLIF1wB6Tsvkj5XbhzBpH6vT7M2rkXWPijvUD4YcwS/VSQ/SyMTYXOOFQ7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JSylfFhs1wOt9zBDAf3llU6BQkpJrQpnELHCrN8Ba/8=; b=UF8jZTAkB8Fgp4GVCKGzsmAmAXJmDH0lylFZVQNwO3PvTxIVmPCSocnb5VAVBPs+Jq/qK4r2vgRQ1AuFiOZ8eN9cEUkcDA1WCV29eWfcVPo2zSmGBayEkxndF78yJ2z4ZXJoswVpRVsrCR7k2OqUM3YKTvI7b30uWuFMmBeWyGl1ZuT6IXB6CbucclnBunvDkrVrRL5IsccrrXcKe8jDvPcWTR6S7M3LEOAjDiPdZa2OBUhilkynszAMEDMxjvVmtAlS/3PdF20piEeM2kDP8CCmqe3g4qCwPRe3R7HmRDOpUnxz4qxOe4BOvxPWDn8npZwirHpuQf9AwlR2D7M7Zg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from MWHPR11MB1629.namprd11.prod.outlook.com (2603:10b6:301:d::21) by PH7PR11MB7027.namprd11.prod.outlook.com (2603:10b6:510:20a::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5612.22; Tue, 13 Sep 2022 17:19:48 +0000 Received: from MWHPR11MB1629.namprd11.prod.outlook.com ([fe80::13c:8120:d994:16d2]) by MWHPR11MB1629.namprd11.prod.outlook.com ([fe80::13c:8120:d994:16d2%6]) with mapi id 15.20.5612.022; Tue, 13 Sep 2022 17:19:45 +0000 Date: Tue, 13 Sep 2022 18:19:34 +0100 From: Bruce Richardson To: Juraj =?utf-8?Q?Linke=C5=A1?= CC: , , , , , , Subject: Re: [PATCH v4 6/9] dts: add config parser module Message-ID: References: <20220728100044.1318484-1-juraj.linkes@pantheon.tech> <20220729105550.1382664-1-juraj.linkes@pantheon.tech> <20220729105550.1382664-7-juraj.linkes@pantheon.tech> Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20220729105550.1382664-7-juraj.linkes@pantheon.tech> X-ClientProxiedBy: LO4P265CA0117.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:2c3::20) To MWHPR11MB1629.namprd11.prod.outlook.com (2603:10b6:301:d::21) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MWHPR11MB1629:EE_|PH7PR11MB7027:EE_ X-MS-Office365-Filtering-Correlation-Id: 0e20a098-f833-4c89-c95a-08da95ac2732 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Byz88hfRhVHAUQSnlQW+Pttno7iFD1hh2cLKhTHvghMt1/s+uItFcsCBQD4vX4FD66X8wwC+HgKrlsahKcK6CQnYt0lyhf6l0NS9PsK0eULrnI1HaOhc34zDyL+l/8oHfYfa3EmH8GqI+1Z127XTw5F7UYqhQgzCSbRPRCRIA71SVT8bEjwCEpvQQc/pw1T6bq+V2ZAaiOqxLHVbok4w77RkCGAy7t0thnNVyy0tP7ZBccrYtzf9iaVJzIEI3XWnZTpG2CnYO9tlfoMTManuUr/zVdMdRkQI0PG3hScQjY20beRapanIFkv0GjWABGQoQ0Ar6vJ+T7VmBUWMPWmPB80Va2VoLlWfQwVUQZKkACqD+jjZ4WnDe1xE49wLqmb69kR4mWb0l1WeJsLz8UmiRix00UNcm5NniVKvE1CPF902Tzk23teUwv4b0Jkvk+4I5HKxzqLDHud9i+jSLOEZsGOpKEVz6ZvcRMwPcA+E31g9w0Z/G1Y4RIeorQj3j+axLlm8hLwQgBC9DcSGaA3Di3omGfSr7eawa66y/FOJooECJ/UJJIeo4zhnMIcDfesCej4JxFterE1THImqj8lIP8Ln1MvklEsUvJyMniDBVVnxQeH8wIrh8sWPmnu/8MFVHUrTZUmrCv1e4Q7P0dEPE6hjkskiT3YEo5wziwGcgYnti2GaIvQobySvSjqyEpLKnuV50ZRgmLusatjvDu7MW75GCr2AtMm0xJenfcXexoAIPdRPOd3XG3dQKdzGYFb2 X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MWHPR11MB1629.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(136003)(396003)(346002)(366004)(376002)(39860400002)(451199015)(44832011)(66556008)(82960400001)(38100700002)(5660300002)(66476007)(478600001)(186003)(8676002)(2906002)(316002)(4326008)(83380400001)(6666004)(26005)(41300700001)(6916009)(6512007)(8936002)(86362001)(6506007)(6486002)(66946007)(67856001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UThBMlhwNFI0eDR2ODA0MkNaOHB1dnBoTzk0UGljdHB3MGI3ZkZoL0wzMHVU?= =?utf-8?B?YXlJRHllZmE4TUNaOG1iSk1tdXp5MXBacVJQMmNTamhMUjRRM1ZhaXVGZ2Fi?= =?utf-8?B?NXd3WUZDWm93cUZSRFV4bU5CcVZTZHY0Yk1DZVZHSUZlZHZzalVmVXNCNDJ1?= =?utf-8?B?Skl0blQrM042V25NdUg4em5OY20yUlhvMFRUM1F0bmkvTGU5MEJrN3Zaem5k?= =?utf-8?B?UHBESzViSDBzeEhYVzAwbHJhK0E5VEQ1UHdHSDd4M2xpRnVOMWpld3VOeUkz?= =?utf-8?B?UTQ0K203MTZxUWpCeW5Hcmc3a3R5M25vVnJJR1NwM2w4RFgzaGJTemsxYnhs?= =?utf-8?B?QURJWFRnUXJHVDZ5SlVHWS9EV1hXM0RuUGR2aTAzYnl2Z3BCNURVNVhGVEp6?= =?utf-8?B?WVF5YW95bjJhQmI0a1pZbVRBMVBvN3g5SXRuVkVic0NnSWRRTHVjRlVrNTJF?= =?utf-8?B?OVVTdU85K3hjQTMwZUkwTWpmU0w0SlZlSUM2WmZBOTNSb0hyRysxakt5VUk2?= =?utf-8?B?OG0zWENVVVRQalJ5UWh5U3pOaDQ2d0pTSmMvMExkM0xTN095UisxTGxiTWx1?= =?utf-8?B?eThibWRnTTBheXpKVGVMVXp2UHFnQ1gzdXp2dUxJMTNZeUZWNVVhYksrdlRC?= =?utf-8?B?WkpjSkJzdDhMdVdQSUswK0p5RUJ4YWM2dUNOQ3pvTGZQVFRDK1o3eXNIandV?= =?utf-8?B?R05HREJvWHpmbzl0OFNydzNJNkM5QWVpWTdYQVpoMEJydjNsR2t5S0lGU3d1?= =?utf-8?B?RTZtSTZ3RXcyWHRqZVN3ZFRKOVJSaFdTVVFZYXdUdmFoaXJrZmlLdUhDOElB?= =?utf-8?B?ZWpzWHljWnNBa2ljbHlzSngxMWlpVFEycDNnNnBnSi93NnlpMWVGbHRpbFVV?= =?utf-8?B?amtrMDl4RlZ5amhVWTVSa3NpNXlhUFNxckRKbFNIUkFYTEJzenpwSEdxeUtW?= =?utf-8?B?ckEwYTlUeXkvSi9XcjlQc3VuYWFjbkJqbWl3RVNnMlVGaUUrMnB4RVR1K2Vt?= =?utf-8?B?OXIzbEFQUEMwWS8zTzJXbzZhS1p2bnhQN1VFMmM1V3JlR3VqUlFpcDJVd1l3?= =?utf-8?B?VThiNW9qeUcwVnBCN3BHaFFlVCtEcHI2alhRaDBPR1FJTTVPeWdjakRjTFpT?= =?utf-8?B?TDhRNkd1WmN2WDJIU1lqQ1hjRnVwY1hLRFZPa2FzS2x2ckpLTWRzNWU2VURx?= =?utf-8?B?cXVmemcvS1c0dkdhMlpOV2hWTnNjVHhZZG1pWENwc0dXbUs2SldGK2RxWXQz?= =?utf-8?B?Wm1Wa0JxMndyUDNrdHpsekFtK2ZuaXRYZXFJeGJERG9heVowcG9kTk1pam8x?= =?utf-8?B?UzhyMmMzRWNXSlVLQkJIMC9mQ2RyWW1VSFdVb2x0c21Dd1lKU0lPdFROekZ5?= =?utf-8?B?ck9WU2dSekxwZmpCN2RlZjFTOFRqcUNOZ1BBUnN3dVZpNGNva2NIRXFsbTNx?= =?utf-8?B?c05iK1M3RnJULzhqSzAwOVdUYzIvL3FUTU03SkFUbG9jalBkSXQ2M1M3L3J0?= =?utf-8?B?YnB1Y25JQ3V1QXRDOTlNM3FFV3V2cUEyd2QzQldmRmV1Sngvc0psT3Z0M05s?= =?utf-8?B?c1NEM0lBaFFWRHdKTkJKbmhoWm9GdTFKS2tySHBKbzZOVXNHbUYwWlRabmwx?= =?utf-8?B?bFhoZmJ1SkdheEZ3a3VnUGZ5cEV4ejZ3SlYxL0VsMHZtcXZNNEdqcTdMUDcr?= =?utf-8?B?RkJmdEpGT1RNZmJzMTlTRXArVUZLTkRCRzRsU1JIQVVjMUdFeTArZEE2UFlo?= =?utf-8?B?Y05KdUpydEx3TUMzT0NLSUZyRXp4ellqTjl2OFVCSTVadjV1ZGtSQ3N2b0lx?= =?utf-8?B?Q1l1QkdjUk94WWFvZ0FrL21vN0laVDR6YWxQUGw2M2NmR0tweHNKellaZlE4?= =?utf-8?B?SlRhRzVJbUpVYmNpWS84WU1meXZEUTM2ekl6NUFobFdGdGduUE94dTNiVC9O?= =?utf-8?B?WCtIT0VTUWd2RFBjSlIrNUpJenNSS3pGdGJKbHdmRHJnTjdXcVdxTTAzNnA2?= =?utf-8?B?b2ZTb0xmOHU4azVqWHVUbTZ2QUZHZDZEazVXb2RoNHdUZkU0Unoyb1hiR1Ro?= =?utf-8?B?OGE0NjQ3T0dJeWJsenRRR0tYcmQybnYrYTdJQmlUZy9WY2JUVVB5TW1vQXJl?= =?utf-8?B?MmNUdUpDdWRtK3RGU0ZIT1JXazNMV054emx1bWtEdVZId0oxbFExdFp6dkhi?= =?utf-8?B?MlE9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: 0e20a098-f833-4c89-c95a-08da95ac2732 X-MS-Exchange-CrossTenant-AuthSource: MWHPR11MB1629.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Sep 2022 17:19:45.4961 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MdofWCqHe91qA72+XuskZ+twobSaPU/48LIZM6lsG29IA5YcHaAHf0X1EF9cJDKQbMMNzIdGxwpYUx10CDuXSRWvNCY7Hvqd6KupilyU5M0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB7027 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Fri, Jul 29, 2022 at 10:55:47AM +0000, Juraj Linkeš wrote: > From: Owen Hilyard > > The configuration is split into two parts, one defining the parameters > of the test run and the other defining the topology to be used. > > The format of the configuration is YAML. It is validated according to a > json schema which also servers as detailed documentation of the various s/servers/serves/ > configuration fields. This means that the complete set of allowed values > are tied to the schema as a source of truth. This enables making changes > to parts of DTS that interface with config files without a high risk of > breaking someone's configuration. > > This configuration system uses immutable objects to represent the > configuration, making IDE/LSP autocomplete work properly. > > There are two ways to specify the configuration file path, an > environment variable or a command line argument, applied in that order. > > Signed-off-by: Owen Hilyard > Signed-off-by: Juraj Linkeš > --- > dts/conf.yaml | 7 ++ > dts/framework/config/__init__.py | 99 ++++++++++++++++++++++ > dts/framework/config/conf_yaml_schema.json | 73 ++++++++++++++++ > dts/framework/exception.py | 14 +++ > dts/framework/settings.py | 65 ++++++++++++++ > 5 files changed, 258 insertions(+) > create mode 100644 dts/conf.yaml > create mode 100644 dts/framework/config/__init__.py > create mode 100644 dts/framework/config/conf_yaml_schema.json > create mode 100644 dts/framework/settings.py > > diff --git a/dts/conf.yaml b/dts/conf.yaml > new file mode 100644 > index 0000000000..cb12ea3d0f > --- /dev/null > +++ b/dts/conf.yaml > @@ -0,0 +1,7 @@ > +executions: > + - system_under_test: "SUT 1" > +nodes: > + - name: "SUT 1" > + hostname: "SUT IP address or hostname" > + user: root > + password: "Leave blank to use SSH keys" > diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py > new file mode 100644 > index 0000000000..a0fdffcd77 > --- /dev/null > +++ b/dts/framework/config/__init__.py > @@ -0,0 +1,99 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright(c) 2010-2021 Intel Corporation > +# Copyright(c) 2022 University of New Hampshire > +# > + > +""" > +Generic port and topology nodes configuration file load function > +""" > +import json > +import os.path > +import pathlib > +from dataclasses import dataclass > +from typing import Any, Optional > + > +import warlock > +import yaml > + > +from framework.settings import SETTINGS > + > + > +# Slots enables some optimizations, by pre-allocating space for the defined > +# attributes in the underlying data structure. > +# > +# Frozen makes the object immutable. This enables further optimizations, > +# and makes it thread safe should we every want to move in that direction. > +@dataclass(slots=True, frozen=True) > +class NodeConfiguration: > + name: str > + hostname: str > + user: str > + password: Optional[str] > + > + @staticmethod > + def from_dict(d: dict) -> "NodeConfiguration": > + return NodeConfiguration( > + name=d["name"], > + hostname=d["hostname"], > + user=d["user"], > + password=d.get("password"), > + ) > + Out of curiosity, what is the reason for having a static "from_dict" method rather than just a regular constructor function that takes a dict as parameter? > + > +@dataclass(slots=True, frozen=True) > +class ExecutionConfiguration: > + system_under_test: NodeConfiguration > + Minor comment: seems strange having only a single member variable in this class, effectively duplicating the class above. > + @staticmethod > + def from_dict(d: dict, node_map: dict) -> "ExecutionConfiguration": from reading the code it appears that node_map is a dict of NodeConfiguration objects, right? Might be worth adding that to the definition for clarity, and also the specific type of the dict "d" (if it has one) > + sut_name = d["system_under_test"] > + assert sut_name in node_map, f"Unknown SUT {sut_name} in execution {d}" > + > + return ExecutionConfiguration( > + system_under_test=node_map[sut_name], > + ) > + > + > +@dataclass(slots=True, frozen=True) > +class Configuration: > + executions: list[ExecutionConfiguration] > + > + @staticmethod > + def from_dict(d: dict) -> "Configuration": > + nodes: list[NodeConfiguration] = list( > + map(NodeConfiguration.from_dict, d["nodes"]) So "d" is a dict of dicts? > + ) > + assert len(nodes) > 0, "There must be a node to test" > + > + node_map = {node.name: node for node in nodes} > + assert len(nodes) == len(node_map), "Duplicate node names are not allowed" > + > + executions: list[ExecutionConfiguration] = list( > + map( > + ExecutionConfiguration.from_dict, d["executions"], [node_map for _ in d] > + ) > + ) > + > + return Configuration(executions=executions) > + > + > +def load_config() -> Configuration: > + """ > + Loads the configuration file and the configuration file schema, > + validates the configuration file, and creates a configuration object. > + """ > + with open(SETTINGS.config_file_path, "r") as f: > + config_data = yaml.safe_load(f) > + > + schema_path = os.path.join( > + pathlib.Path(__file__).parent.resolve(), "conf_yaml_schema.json" > + ) > + > + with open(schema_path, "r") as f: > + schema = json.load(f) > + config: dict[str, Any] = warlock.model_factory(schema, name="_Config")(config_data) > + config_obj: Configuration = Configuration.from_dict(dict(config)) > + return config_obj > + > + > +CONFIGURATION = load_config()