From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 10DE745849; Fri, 23 Aug 2024 12:30:25 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 108EE402E6; Fri, 23 Aug 2024 12:30:23 +0200 (CEST) Received: from mail-ed1-f47.google.com (mail-ed1-f47.google.com [209.85.208.47]) by mails.dpdk.org (Postfix) with ESMTP id 36899402CA for ; Fri, 23 Aug 2024 12:30:21 +0200 (CEST) Received: by mail-ed1-f47.google.com with SMTP id 4fb4d7f45d1cf-5bed72ff2f2so2351992a12.2 for ; Fri, 23 Aug 2024 03:30:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1724409021; x=1725013821; darn=dpdk.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=wt4YR3pRfcW1pv/NnMt18mExOVLWxJdYORzc5o9G/Ek=; b=PLfWm+NFWXL2SCKtFswt2FUv6Xxkgxz7izxOV6ZiaoAvdGIznArBX+UQRfyScXCS3t y640wJjdhMHHIsf/QmEvYj3GpmGUEol3z6PbY+4UjaKPhPTpF/ZLMIKtGxwOMUkfjG77 47DfzNRB1tlTQpon9FpN2kt6zSBo9GrIZOYukNKFiHTMTqo5O+lvjF7xp/Aeivk9aOt7 HgkQoiTcialdZuOD2tDfQPrCjInRqa5Ix09cW6iC7fdy77yn2r0ptlmAFno6BMuVxez2 mMGOPSFJWHnaEBnu8fXKMcbzVke7tuznHWZ97+d79ZZ2+rarDX95CqnK6qM1oYdapWsb BiAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724409021; x=1725013821; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wt4YR3pRfcW1pv/NnMt18mExOVLWxJdYORzc5o9G/Ek=; b=ZyypVq1RxIItejCr+uM8CkpJ5YMmVuV53q4CWOUr/5t0Sx3L06zy6HPO2MwzurrwLA 7kh/ftquLNOp1fkF7ZUgT7WnoIhiZSgvTsfBRCS72W5Vo6n+z5FBgMjned5zKPnFZy59 utiuTr5fujiRRE2mmriPl9YuPCqZWm8WDF3OKMaDxHNYmtikzcrkdRAOlK035xu65h3D r2tgu3mGL1PUhfwDZmDVlgFSyetBio8YQyelCB4qVxqFSlJb94tI54le/27p8SeDC0wb Z6y+DU5B45+sVmNMZdZErosUmkLW0NWi4PzkChi+jv91FhwbFoL4ngcBHjBARIONl+zE sPSg== X-Forwarded-Encrypted: i=1; AJvYcCXgaDXXQ+ZnbhoEykJj6cxUO/a9J8Al+kR03eBWUMW5JAKBaXGk85kaUSg1HLHKgFpUo2c=@dpdk.org X-Gm-Message-State: AOJu0Ywuka3xnuJJ5OfoMEzZeN7/J/qEgk4uuU0XUEMTXn0dl95gAqnC dUbiJdwhpQqYj/0XMGy7vBZ7KoMMgGHKVXr9/Got/MLPg/f1F7Efq+Na+ul7qCQ= X-Google-Smtp-Source: AGHT+IGdaaGFlcAKbbphQfdhX1k1kI5OYxLymf601gcLjFE8mWsOtsXKL6j/VbYsAeMtHsbmWbdUuw== X-Received: by 2002:a17:906:d264:b0:a86:a1cd:5a8c with SMTP id a640c23a62f3a-a86a52bb759mr116871766b.22.1724409020716; Fri, 23 Aug 2024 03:30:20 -0700 (PDT) Received: from [192.168.200.22] ([84.245.121.107]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a868f2e6795sm241606366b.92.2024.08.23.03.30.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 23 Aug 2024 03:30:20 -0700 (PDT) Message-ID: <1c74fadc-aa35-4c6d-afcb-7cea14bba771@pantheon.tech> Date: Fri, 23 Aug 2024 12:30:19 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2 2/5] dts: add random generation seed setting To: Luca Vizzarro , dev@dpdk.org Cc: Jeremy Spewock , Honnappa Nagarahalli , Paul Szczepanek , Alex Chapman References: <20240806121417.2567708-1-Luca.Vizzarro@arm.com> <20240806124642.2580828-1-luca.vizzarro@arm.com> <20240806124642.2580828-3-luca.vizzarro@arm.com> Content-Language: en-US From: =?UTF-8?Q?Juraj_Linke=C5=A1?= In-Reply-To: <20240806124642.2580828-3-luca.vizzarro@arm.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This is a great idea. I originally didn't think the config file option would be very useful, but thinking a bit more about it, when debugging and trying to find a solution the devs need to run with the same seed over and over, so it's going to have a place. Reviewed-by: Juraj Linkeš On 6. 8. 2024 14:46, Luca Vizzarro wrote: > When introducing pseudo-random generation in the test runs we need to > ensure that these can be reproduced by setting a pre-defined seed. > This commits adds the ability to set one or allow for one to be > generated and reported back to the user. > > Signed-off-by: Luca Vizzarro > Reviewed-by: Paul Szczepanek > Reviewed-by: Alex Chapman > --- > doc/guides/tools/dts.rst | 5 +++++ > dts/framework/config/__init__.py | 4 ++++ > dts/framework/config/conf_yaml_schema.json | 4 ++++ > dts/framework/config/types.py | 2 ++ > dts/framework/runner.py | 8 ++++++++ > dts/framework/settings.py | 17 +++++++++++++++++ > 6 files changed, 40 insertions(+) > > diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst > index 515b15e4d8..9b5ea9779c 100644 > --- a/doc/guides/tools/dts.rst > +++ b/doc/guides/tools/dts.rst > @@ -251,6 +251,8 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet > ... | DTS_TEST_SUITES='suite, suite case, ...' (default: []) > --re-run N_TIMES, --re_run N_TIMES > [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0) > + --random-seed NUMBER [DTS_RANDOM_SEED] The seed to use with the pseudo-random generator. If not specified, the configuration value is > + used instead. If that's also not specified, a random seed is generated. (default: None) > > > The brackets contain the names of environment variables that set the same thing. > @@ -548,6 +550,9 @@ involved in the testing. These can be defined with the following mappings: > +----------------------------+---------------+---------------------------------------------------+ > | ``traffic_generator_node`` | Node name for the traffic generator node. | > +----------------------------+-------------------------------------------------------------------+ > + | ``random_seed`` | (*optional*) *int* – Allows you to set a seed for pseudo-random | > + | | generation. | > + +----------------------------+-------------------------------------------------------------------+ > > ``nodes`` > `sequence `_ listing > diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py > index df60a5030e..269d9ec318 100644 > --- a/dts/framework/config/__init__.py > +++ b/dts/framework/config/__init__.py > @@ -445,6 +445,7 @@ class TestRunConfiguration: > system_under_test_node: The SUT node to use in this test run. > traffic_generator_node: The TG node to use in this test run. > vdevs: The names of virtual devices to test. > + random_seed: The seed to use for pseudo-random generation. > """ > > build_targets: list[BuildTargetConfiguration] > @@ -455,6 +456,7 @@ class TestRunConfiguration: > system_under_test_node: SutNodeConfiguration > traffic_generator_node: TGNodeConfiguration > vdevs: list[str] > + random_seed: int | None > > @classmethod > def from_dict( > @@ -497,6 +499,7 @@ def from_dict( > vdevs = ( > d["system_under_test_node"]["vdevs"] if "vdevs" in d["system_under_test_node"] else [] > ) > + random_seed = d.get("random_seed", None) > return cls( > build_targets=build_targets, > perf=d["perf"], > @@ -506,6 +509,7 @@ def from_dict( > system_under_test_node=system_under_test_node, > traffic_generator_node=traffic_generator_node, > vdevs=vdevs, > + random_seed=random_seed, > ) > > def copy_and_modify(self, **kwargs) -> Self: > diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json > index f02a310bb5..df390e8ae2 100644 > --- a/dts/framework/config/conf_yaml_schema.json > +++ b/dts/framework/config/conf_yaml_schema.json > @@ -379,6 +379,10 @@ > }, > "traffic_generator_node": { > "$ref": "#/definitions/node_name" > + }, > + "random_seed": { > + "type": "integer", > + "description": "Optional field. Allows you to set a seed for pseudo-random generation." > } > }, > "additionalProperties": false, > diff --git a/dts/framework/config/types.py b/dts/framework/config/types.py > index cf16556403..ce7b784ac8 100644 > --- a/dts/framework/config/types.py > +++ b/dts/framework/config/types.py > @@ -121,6 +121,8 @@ class TestRunConfigDict(TypedDict): > system_under_test_node: TestRunSUTConfigDict > #: > traffic_generator_node: str > + #: > + random_seed: int > > > class ConfigurationDict(TypedDict): > diff --git a/dts/framework/runner.py b/dts/framework/runner.py > index 6b6f6a05f5..34b1dad5c4 100644 > --- a/dts/framework/runner.py > +++ b/dts/framework/runner.py > @@ -20,6 +20,7 @@ > import importlib > import inspect > import os > +import random > import re > import sys > from pathlib import Path > @@ -147,6 +148,7 @@ def run(self) -> None: > self._logger.info( > f"Running test run with SUT '{test_run_config.system_under_test_node.name}'." > ) > + self._init_random_seed(test_run_config) > test_run_result = self._result.add_test_run(test_run_config) > # we don't want to modify the original config, so create a copy > test_run_test_suites = list( > @@ -723,3 +725,9 @@ def _exit_dts(self) -> None: > self._logger.info("DTS execution has ended.") > > sys.exit(self._result.get_return_code()) > + > + def _init_random_seed(self, conf: TestRunConfiguration) -> None: > + """Initialize the random seed to use for the test run.""" > + seed = SETTINGS.random_seed or conf.random_seed or random.randrange(0xFFFF_FFFF) > + self._logger.info(f"Initializing test run with random seed {seed}") > + random.seed(seed) > diff --git a/dts/framework/settings.py b/dts/framework/settings.py > index f6303066d4..7744e37f54 100644 > --- a/dts/framework/settings.py > +++ b/dts/framework/settings.py > @@ -66,6 +66,12 @@ > > Re-run each test case this many times in case of a failure. > > +.. option:: --random-seed > +.. envvar:: DTS_RANDOM_SEED > + > + The seed to use with the pseudo-random generator. If not specified, the configuration value is > + used instead. If that's also not specified, a random seed is generated. > + > The module provides one key module-level variable: > > Attributes: > @@ -115,6 +121,8 @@ class Settings: > test_suites: list[TestSuiteConfig] = field(default_factory=list) > #: > re_run: int = 0 > + #: > + random_seed: int | None = None > > > SETTINGS: Settings = Settings() > @@ -375,6 +383,15 @@ def _get_parser() -> _DTSArgumentParser: > ) > _add_env_var_to_action(action, "RERUN") > > + action = parser.add_argument( > + "--random-seed", > + type=int, > + help="The seed to use with the pseudo-random generator. If not specified, the configuration" > + " value is used instead. If that's also not specified, a random seed is generated.", > + metavar="NUMBER", > + ) > + _add_env_var_to_action(action) > + > return parser > >