From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A89A54339B; Wed, 22 Nov 2023 12:18:38 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 482FD4028C; Wed, 22 Nov 2023 12:18:38 +0100 (CET) Received: from mail-ed1-f42.google.com (mail-ed1-f42.google.com [209.85.208.42]) by mails.dpdk.org (Postfix) with ESMTP id 81A4540273 for ; Wed, 22 Nov 2023 12:18:37 +0100 (CET) Received: by mail-ed1-f42.google.com with SMTP id 4fb4d7f45d1cf-549070a04baso2199761a12.3 for ; Wed, 22 Nov 2023 03:18:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1700651917; x=1701256717; darn=dpdk.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=H89pdyIl/ImlrIQRm4Bs/hnV3skMIJYae9X9cglb4tU=; b=Lgf0TodKSvvSf5LURNseSxvlmN41kr4aqjNJjqqP7NOSNDpXtTONVB8y/htxA0KSaP TKRT/NQRCkieZOyD44D3jdrJrvIRrAtGoIPnVUvIG7caf2Qnou1b+V3bhxb6QP6XZTQg r5ubs4gN78x/But6ZsqfUm/Ewmcu6afAEgz2DmDTp9hiJBVVz/YAMA/CwtMFBi01ed8y IwFGicwXiUoZK2AOI9WajjLNzXIlWRd/u7vTdcPN06iKKAsZ8qKzl+V9VpjK19OGU8iK GBHBBmacQ+vjdcwHOWrUgiDViUnEf7AaaCdHvwB8jGLp1/3VCQ9TwqNDeQonCVNAHr0w tP9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700651917; x=1701256717; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H89pdyIl/ImlrIQRm4Bs/hnV3skMIJYae9X9cglb4tU=; b=gk6jRHb7ltJDcCK1+VN0PB1OLWGOsokt4EctoBBsC9vJ5ouUL0IWkye3uWyUr3rNol y9hPVCDNu+DuY8HyVMbnJIvDd12l1i/VCLrONdNE5TsXMPsbkbIfthuKn2l/uGGc3GrS fs51arDj0lAw8v+7lbBXwH4H978h2CdONlhYFXCOLjay8GwMPKFp5JPGx+MN3YzfRCNf rUfku8cV5dJ3jwyE8Pudu7RJuas4POJiXsYmlpwcpZdTdik5lGLXxw2OUcFWHUWpquZE O0JTH3z6BQHVER3gH3H6YTluOpNthDpZ3a1XrRrRibB15XEO0OLExN3NgVmJTBbuD7II stdw== X-Gm-Message-State: AOJu0YwcA0vkDgYYannQ8UaOJ2qv/4remhPm4pEJ/fgb6DQXWROL6EeI 2llMWPvvAMSG0rEIfvy53BMqHO5MAc3lkPkBny6N/g== X-Google-Smtp-Source: AGHT+IE+k+XJbtsse3IvtWksGE0O0pIRIGZULxLM3H2vAqpUUiLoEiVuJt5pmtuET6EimTM/TBELr6O1b6IzdWZHcKw= X-Received: by 2002:a17:906:f6d3:b0:a01:bc90:736d with SMTP id jo19-20020a170906f6d300b00a01bc90736dmr1358825ejb.40.1700651917055; Wed, 22 Nov 2023 03:18:37 -0800 (PST) MIME-Version: 1.0 References: <20231108125324.191005-23-juraj.linkes@pantheon.tech> <20231115130959.39420-1-juraj.linkes@pantheon.tech> <20231115130959.39420-15-juraj.linkes@pantheon.tech> <927d9e84-5583-4516-8a0e-bff52470406a@foss.arm.com> In-Reply-To: <927d9e84-5583-4516-8a0e-bff52470406a@foss.arm.com> From: =?UTF-8?Q?Juraj_Linke=C5=A1?= Date: Wed, 22 Nov 2023 12:18:26 +0100 Message-ID: Subject: Re: [PATCH v7 14/21] dts: cpu docstring update To: Yoan Picchi Cc: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, jspewock@iol.unh.edu, probb@iol.unh.edu, paul.szczepanek@arm.com, dev@dpdk.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Tue, Nov 21, 2023 at 6:45=E2=80=AFPM Yoan Picchi wrote: > > On 11/15/23 13:09, Juraj Linke=C5=A1 wrote: > > Format according to the Google format and PEP257, with slight > > deviations. > > > > Signed-off-by: Juraj Linke=C5=A1 > > --- > > dts/framework/testbed_model/cpu.py | 196 +++++++++++++++++++++-------= - > > 1 file changed, 144 insertions(+), 52 deletions(-) > > > > diff --git a/dts/framework/testbed_model/cpu.py b/dts/framework/testbed= _model/cpu.py > > index 8fe785dfe4..4edeb4a7c2 100644 > > --- a/dts/framework/testbed_model/cpu.py > > +++ b/dts/framework/testbed_model/cpu.py > > @@ -1,6 +1,22 @@ > > # SPDX-License-Identifier: BSD-3-Clause > > # Copyright(c) 2023 PANTHEON.tech s.r.o. > > > > +"""CPU core representation and filtering. > > + > > +This module provides a unified representation of logical CPU cores alo= ng > > +with filtering capabilities. > > + > > +When symmetric multiprocessing (SMP or multithreading) is enabled on a= server, > > +the physical CPU cores are split into logical CPU cores with different= IDs. > > + > > +:class:`LogicalCoreCountFilter` filters by the number of logical cores= . It's possible to specify > > +the socket from which to filter the number of logical cores. It's also= possible to not use all > > +logical CPU cores from each physical core (e.g. only the first logical= core of each physical core). > > + > > +:class:`LogicalCoreListFilter` filters by logical core IDs. This mostl= y checks that > > +the logical cores are actually present on the server. > > +""" > > + > > import dataclasses > > from abc import ABC, abstractmethod > > from collections.abc import Iterable, ValuesView > > @@ -11,9 +27,17 @@ > > > > @dataclass(slots=3DTrue, frozen=3DTrue) > > class LogicalCore(object): > > - """ > > - Representation of a CPU core. A physical core is represented in OS > > - by multiple logical cores (lcores) if CPU multithreading is enable= d. > > + """Representation of a logical CPU core. > > + > > + A physical core is represented in OS by multiple logical cores (lc= ores) > > + if CPU multithreading is enabled. When multithreading is disabled,= their IDs are the same. > > + > > + Attributes: > > + lcore: The logical core ID of a CPU core. It's the same as `co= re` with > > + disabled multithreading. > > + core: The physical core ID of a CPU core. > > + socket: The physical socket ID where the CPU resides. > > + node: The NUMA node ID where the CPU resides. > > """ > > > > lcore: int > > @@ -22,27 +46,36 @@ class LogicalCore(object): > > node: int > > > > def __int__(self) -> int: > > + """The CPU is best represented by the logical core, as that's = what we configure in EAL.""" > > return self.lcore > > > > > > class LogicalCoreList(object): > > - """ > > - Convert these options into a list of logical core ids. > > - lcore_list=3D[LogicalCore1, LogicalCore2] - a list of LogicalCores > > - lcore_list=3D[0,1,2,3] - a list of int indices > > - lcore_list=3D['0','1','2-3'] - a list of str indices; ranges are s= upported > > - lcore_list=3D'0,1,2-3' - a comma delimited str of indices; ranges = are supported > > - > > - The class creates a unified format used across the framework and a= llows > > - the user to use either a str representation (using str(instance) o= r directly > > - in f-strings) or a list representation (by accessing instance.lcor= e_list). > > - Empty lcore_list is allowed. > > + r"""A unified way to store :class:`LogicalCore`\s. > > + > > + Create a unified format used across the framework and allow the us= er to use > > + either a :class:`str` representation (using ``str(instance)`` or d= irectly in f-strings) > > + or a :class:`list` representation (by accessing the `lcore_list` p= roperty, > > + which stores logical core IDs). > > """ > > > > _lcore_list: list[int] > > _lcore_str: str > > > > def __init__(self, lcore_list: list[int] | list[str] | list[Logic= alCore] | str): > > + """Process `lcore_list`, then sort. > > + > > + There are four supported logical core list formats:: > > + > > + lcore_list=3D[LogicalCore1, LogicalCore2] # a list of Log= icalCores > > + lcore_list=3D[0,1,2,3] # a list of int indices > > + lcore_list=3D['0','1','2-3'] # a list of str indices; ran= ges are supported > > + lcore_list=3D'0,1,2-3' # a comma delimited str of i= ndices; ranges are supported > > + > > + Args: > > + lcore_list: Various ways to represent multiple logical cor= es. > > + Empty `lcore_list` is allowed. > > + """ > > self._lcore_list =3D [] > > if isinstance(lcore_list, str): > > lcore_list =3D lcore_list.split(",") > > @@ -60,6 +93,7 @@ def __init__(self, lcore_list: list[int] | list[str] = | list[LogicalCore] | str): > > > > @property > > def lcore_list(self) -> list[int]: > > + """The logical core IDs.""" > > return self._lcore_list > > > > def _get_consecutive_lcores_range(self, lcore_ids_list: list[int]= ) -> list[str]: > > @@ -89,28 +123,30 @@ def _get_consecutive_lcores_range(self, lcore_ids_= list: list[int]) -> list[str]: > > return formatted_core_list > > > > def __str__(self) -> str: > > + """The consecutive ranges of logical core IDs.""" > > return self._lcore_str > > > > > > @dataclasses.dataclass(slots=3DTrue, frozen=3DTrue) > > class LogicalCoreCount(object): > > - """ > > - Define the number of logical cores to use. > > - If sockets is not None, socket_count is ignored. > > - """ > > + """Define the number of logical cores per physical cores per socke= ts.""" > > > > + #: Use this many logical cores per each physical core. > > lcores_per_core: int =3D 1 > > + #: Use this many physical cores per each socket. > > cores_per_socket: int =3D 2 > > + #: Use this many sockets. > > socket_count: int =3D 1 > > + #: Use exactly these sockets. This takes precedence over `socket_c= ount`, > > + #: so when `sockets` is not :data:`None`, `socket_count` is ignore= d. > > sockets: list[int] | None =3D None > > > > > > class LogicalCoreFilter(ABC): > > - """ > > - Filter according to the input filter specifier. Each filter needs = to be > > - implemented in a derived class. > > - This class only implements operations common to all filters, such = as sorting > > - the list to be filtered beforehand. > > + """Common filtering class. > > + > > + Each filter needs to be implemented in a subclass. This base class= sorts the list of cores > > + and defines the filtering method, which must be implemented by sub= classes. > > """ > > > > _filter_specifier: LogicalCoreCount | LogicalCoreList > > @@ -122,6 +158,17 @@ def __init__( > > filter_specifier: LogicalCoreCount | LogicalCoreList, > > ascending: bool =3D True, > > ): > > + """Filter according to the input filter specifier. > > + > > + The input `lcore_list` is copied and sorted by physical core b= efore filtering. > > + The list is copied so that the original is left intact. > > + > > + Args: > > + lcore_list: The logical CPU cores to filter. > > + filter_specifier: Filter cores from `lcore_list` according= to this filter. > > + ascending: Sort cores in ascending order (lowest to highes= t IDs). If data:`False`, > > + sort in descending order. > > + """ > > self._filter_specifier =3D filter_specifier > > > > # sorting by core is needed in case hyperthreading is enabled > > @@ -132,31 +179,45 @@ def __init__( > > > > @abstractmethod > > def filter(self) -> list[LogicalCore]: > > - """ > > - Use self._filter_specifier to filter self._lcores_to_filter > > - and return the list of filtered LogicalCores. > > - self._lcores_to_filter is a sorted copy of the original list, > > - so it may be modified. > > + r"""Filter the cores. > > + > > + Use `self._filter_specifier` to filter `self._lcores_to_filter= ` and return > > + the filtered :class:`LogicalCore`\s. > > + `self._lcores_to_filter` is a sorted copy of the original list= , so it may be modified. > > + > > + Returns: > > + The filtered cores. > > """ > > > > > > class LogicalCoreCountFilter(LogicalCoreFilter): > > - """ > > + """Filter cores by specified counts. > > + > > Filter the input list of LogicalCores according to specified rule= s: > > - Use cores from the specified number of sockets or from the specifi= ed socket ids. > > - If sockets is specified, it takes precedence over socket_count. > > - From each of those sockets, use only cores_per_socket of cores. > > - And for each core, use lcores_per_core of logical cores. Hyperthea= ding > > - must be enabled for this to take effect. > > - If ascending is True, use cores with the lowest numerical id first > > - and continue in ascending order. If False, start with the highest > > - id and continue in descending order. This ordering affects which > > - sockets to consider first as well. > > + > > + * The input `filter_specifier` is :class:`LogicalCoreCount`, > > + * Use cores from the specified number of sockets or from the s= pecified socket ids, > > + * If `sockets` is specified, it takes precedence over `socket_= count`, > > + * From each of those sockets, use only `cores_per_socket` of c= ores, > > + * And for each core, use `lcores_per_core` of logical cores. H= ypertheading > > + must be enabled for this to take effect. > > """ > > > > _filter_specifier: LogicalCoreCount > > > > def filter(self) -> list[LogicalCore]: > > + """Filter the cores according to :class:`LogicalCoreCount`. > > + > > + Start by filtering the allowed sockets. The cores matching the= allowed socket are returned. > > allowed socket*s* > Ack. > > + The cores of each socket are stored in separate lists. > > + > > + Then filter the allowed physical cores from those lists of cor= es per socket. When filtering > > + physical cores, store the desired number of logical cores per = physical core which then > > + together constitute the final filtered list. > > + > > + Returns: > > + The filtered cores. > > + """ > > sockets_to_filter =3D self._filter_sockets(self._lcores_to_fi= lter) > > filtered_lcores =3D [] > > for socket_to_filter in sockets_to_filter: > > @@ -166,24 +227,37 @@ def filter(self) -> list[LogicalCore]: > > def _filter_sockets( > > self, lcores_to_filter: Iterable[LogicalCore] > > ) -> ValuesView[list[LogicalCore]]: > > - """ > > - Remove all lcores that don't match the specified socket(s). > > - If self._filter_specifier.sockets is not None, keep lcores fro= m those sockets, > > - otherwise keep lcores from the first > > - self._filter_specifier.socket_count sockets. > > + """Filter a list of cores per each allowed socket. > > + > > + The sockets may be specified in two ways, either a number or a= specific list of sockets. > > + In case of a specific list, we just need to return the cores f= rom those sockets. > > + If filtering a number of cores, we need to go through all core= s and note which sockets > > + appear and only filter from the first n that appear. > > + > > + Args: > > + lcores_to_filter: The cores to filter. These must be sorte= d by the physical core. > > + > > + Returns: > > + A list of lists of logical CPU cores. Each list contains c= ores from one socket. > > """ > > allowed_sockets: set[int] =3D set() > > socket_count =3D self._filter_specifier.socket_count > > if self._filter_specifier.sockets: > > + # when sockets in filter is specified, the sockets are alr= eady set > > socket_count =3D len(self._filter_specifier.sockets) > > allowed_sockets =3D set(self._filter_specifier.sockets) > > > > + # filter socket_count sockets from all sockets by checking the= socket of each CPU > > filtered_lcores: dict[int, list[LogicalCore]] =3D {} > > for lcore in lcores_to_filter: > > if not self._filter_specifier.sockets: > > + # this is when sockets is not set, so we do the actual= filtering > > + # when it is set, allowed_sockets is already defined a= nd can't be changed > > if len(allowed_sockets) < socket_count: > > + # allowed_sockets is a set, so adding an existing = socket won't re-add it > > allowed_sockets.add(lcore.socket) > > if lcore.socket in allowed_sockets: > > + # separate sockets per socket; this makes it easier in= further processing > > socket*s* per socket ? > Good catch, this should be "separate lcores into sockets". > > if lcore.socket in filtered_lcores: > > filtered_lcores[lcore.socket].append(lcore) > > else: > > @@ -200,12 +274,13 @@ def _filter_sockets( > > def _filter_cores_from_socket( > > self, lcores_to_filter: Iterable[LogicalCore] > > ) -> list[LogicalCore]: > > - """ > > - Keep only the first self._filter_specifier.cores_per_socket co= res. > > - In multithreaded environments, keep only > > - the first self._filter_specifier.lcores_per_core lcores of tho= se cores. > > - """ > > + """Filter a list of cores from the given socket. > > + > > + Go through the cores and note how many logical cores per physi= cal core have been filtered. > > > > + Returns: > > + The filtered logical CPU cores. > > + """ > > # no need to use ordered dict, from Python3.7 the dict > > # insertion order is preserved (LIFO). > > lcore_count_per_core_map: dict[int, int] =3D {} > > @@ -248,15 +323,21 @@ def _filter_cores_from_socket( > > > > > > class LogicalCoreListFilter(LogicalCoreFilter): > > - """ > > - Filter the input list of Logical Cores according to the input list= of > > - lcore indices. > > - An empty LogicalCoreList won't filter anything. > > + """Filter the logical CPU cores by logical CPU core IDs. > > + > > + This is a simple filter that looks at logical CPU IDs and only fil= ter those that match. > > + > > + The input filter is :class:`LogicalCoreList`. An empty LogicalCore= List won't filter anything. > > """ > > > > _filter_specifier: LogicalCoreList > > > > def filter(self) -> list[LogicalCore]: > > + """Filter based on logical CPU core ID. > > + > > + Return: > > + The filtered logical CPU cores. > > + """ > > if not len(self._filter_specifier.lcore_list): > > return self._lcores_to_filter > > > > @@ -279,6 +360,17 @@ def lcore_filter( > > filter_specifier: LogicalCoreCount | LogicalCoreList, > > ascending: bool, > > ) -> LogicalCoreFilter: > > + """Factory for using the right filter with `filter_specifier`. > > + > > + Args: > > + core_list: The logical CPU cores to filter. > > + filter_specifier: The filter to use. > > + ascending: Sort cores in ascending order (lowest to highest ID= s). If :data:`False`, > > + sort in descending order. > > + > > + Returns: > > + The filter matching `filter_specifier`. > > + """ > > if isinstance(filter_specifier, LogicalCoreList): > > return LogicalCoreListFilter(core_list, filter_specifier, asc= ending) > > elif isinstance(filter_specifier, LogicalCoreCount): >