DPDK patches and discussions
 help / color / mirror / Atom feed
From: Tomasz Duszynski <tduszynski@marvell.com>
To: Tomasz Duszynski <tduszynski@marvell.com>
Cc: "Ruifeng.Wang@arm.com" <Ruifeng.Wang@arm.com>,
	"bruce.richardson@intel.com" <bruce.richardson@intel.com>,
	"david.marchand@redhat.com" <david.marchand@redhat.com>,
	"dev@dpdk.org" <dev@dpdk.org>, Jerin Jacob <jerinj@marvell.com>,
	"konstantin.v.ananyev@yandex.ru" <konstantin.v.ananyev@yandex.ru>,
	"mattias.ronnblom@ericsson.com" <mattias.ronnblom@ericsson.com>,
	"mb@smartsharesystems.com" <mb@smartsharesystems.com>,
	"roretzla@linux.microsoft.com" <roretzla@linux.microsoft.com>,
	"stephen@networkplumber.org" <stephen@networkplumber.org>,
	"thomas@monjalon.net" <thomas@monjalon.net>,
	"zhoumin@loongson.cn" <zhoumin@loongson.cn>
Subject: RE: [PATCH v15 0/4] add support for self monitoring
Date: Tue, 5 Nov 2024 04:04:32 +0000	[thread overview]
Message-ID: <DM4PR18MB436808478A9AF295D4092E35D2522@DM4PR18MB4368.namprd18.prod.outlook.com> (raw)
In-Reply-To: <20241025085414.3412068-1-tduszynski@marvell.com>

>-----Original Message-----
>From: Tomasz Duszynski <tduszynski@marvell.com>
>Sent: Friday, October 25, 2024 10:54 AM
>To: Tomasz Duszynski <tduszynski@marvell.com>
>Cc: Ruifeng.Wang@arm.com; bruce.richardson@intel.com; david.marchand@redhat.com; dev@dpdk.org;
>Jerin Jacob <jerinj@marvell.com>; konstantin.v.ananyev@yandex.ru; mattias.ronnblom@ericsson.com;
>mb@smartsharesystems.com; roretzla@linux.microsoft.com; stephen@networkplumber.org;
>thomas@monjalon.net; zhoumin@loongson.cn
>Subject: [PATCH v15 0/4] add support for self monitoring
>
>This series adds self monitoring support i.e allows to configure and read performance measurement
>unit (PMU) counters in runtime without using perf utility. This has certain advantages when
>application runs on isolated cores running dedicated tasks.
>
>Events can be read directly using rte_pmu_read() or using dedicated tracepoint
>rte_eal_trace_pmu_read(). The latter will cause events to be stored inside CTF file.
>
>By design, all enabled events are grouped together and the same group is attached to lcores that
>use self monitoring funtionality.
>
>Events are enabled by names, which need to be read from standard location under sysfs i.e
>
>/sys/bus/event_source/devices/PMU/events
>
>where PMU is a core pmu i.e one measuring cpu events. As of today raw events are not supported.
>
>v15:
>- add some basic logs related to API failures
>- get rid of thread-local-storage
>- do not support MT-safety (which was buggy anyway) in management APIs (rte_pmu_init(),
>  rte_pmu_fini(), rte_pmu_add_{event,events_by_pattern}() as it impacts rte_pmu_read()
>  performance because more logic needs to be incorporated to handle all corner cases
>- improve documentation slightly
>- various other improvements here and there
>v14:
>- replace __atomic_xxx with rte_atomic_xxx
>- rebase to dpdk/main since that's a new feature
>v13:
>- allow init/fini calling from multiple contexts
>- get rid of conditional compilation and return erors in case APIs are
>  used on unsupported OS
>v12:
>- rebase old series
>- slightly refresh existing documentation
>- make sure compiler won't optimize accesses to certain variables during
>  event readout
>- drop previous Acked-by to respin a fresh review cycle
>v11:
>- skip fast test in case init fails
>v10:
>- check permissions before using counters
>- do not use internal symbols in exported functions
>- address review comments
>v9:
>- fix 'maybe-uninitialized' warning reported by CI
>v8:
>- just rebase series
>v7:
>- use per-lcore event group instead of global table index by lcore-id
>- don't add pmu_autotest to fast tests because due to lack of suported
>  on every arch
>v6:
>- move codebase to the separate library
>- address review comments
>v5:
>- address review comments
>- fix sign extension while reading pmu on x86
>- fix regex mentioned in doc
>- various minor changes/improvements here and there
>v4:
>- fix freeing mem detected by debug_autotest
>v3:
>- fix shared build
>v2:
>- fix problems reported by test build infra
>
>Tomasz Duszynski (4):
>  lib: add generic support for reading PMU events
>  pmu: support reading ARM PMU events in runtime
>  pmu: support reading Intel x86_64 PMU events in runtime
>  eal: add PMU support to tracing library
>
> MAINTAINERS                              |   5 +
> app/test/meson.build                     |   1 +
> app/test/test_pmu.c                      |  55 +++
> app/test/test_trace_perf.c               |  10 +
> doc/api/doxy-api-index.md                |   3 +-
> doc/api/doxy-api.conf.in                 |   1 +
> doc/guides/prog_guide/profile_app.rst    |  34 ++
> doc/guides/prog_guide/trace_lib.rst      |  32 ++
> doc/guides/rel_notes/release_24_11.rst   |   7 +
> lib/eal/common/eal_common_trace.c        |   5 +-
> lib/eal/common/eal_common_trace_pmu.c    |  38 ++
> lib/eal/common/eal_common_trace_points.c |   5 +
> lib/eal/common/eal_trace.h               |   4 +
> lib/eal/common/meson.build               |   1 +
> lib/eal/include/rte_eal_trace.h          |  11 +
> lib/eal/meson.build                      |   3 +
> lib/eal/version.map                      |   1 +
> lib/meson.build                          |   1 +
> lib/pmu/meson.build                      |  22 +
> lib/pmu/pmu_arm64.c                      |  94 ++++
> lib/pmu/pmu_private.h                    |  32 ++
> lib/pmu/rte_pmu.c                        | 536 +++++++++++++++++++++++
> lib/pmu/rte_pmu.h                        | 229 ++++++++++
> lib/pmu/rte_pmu_pmc_arm64.h              |  30 ++
> lib/pmu/rte_pmu_pmc_x86_64.h             |  24 +
> lib/pmu/version.map                      |  14 +
> 26 files changed, 1196 insertions(+), 2 deletions(-)  create mode 100644 app/test/test_pmu.c
>create mode 100644 lib/eal/common/eal_common_trace_pmu.c
> create mode 100644 lib/pmu/meson.build
> create mode 100644 lib/pmu/pmu_arm64.c
> create mode 100644 lib/pmu/pmu_private.h  create mode 100644 lib/pmu/rte_pmu.c  create mode 100644
>lib/pmu/rte_pmu.h  create mode 100644 lib/pmu/rte_pmu_pmc_arm64.h  create mode 100644
>lib/pmu/rte_pmu_pmc_x86_64.h  create mode 100644 lib/pmu/version.map
>
>--
>2.34.1

Hi, 

Did anyone interested have a time to look into latest series?   

  parent reply	other threads:[~2024-11-05  4:04 UTC|newest]

Thread overview: 187+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-11-11  9:43 [PATCH " Tomasz Duszynski
2022-11-11  9:43 ` [PATCH 1/4] eal: add generic support for reading PMU events Tomasz Duszynski
2022-12-15  8:33   ` Mattias Rönnblom
2022-11-11  9:43 ` [PATCH 2/4] eal/arm: support reading ARM PMU events in runtime Tomasz Duszynski
2022-11-11  9:43 ` [PATCH 3/4] eal/x86: support reading Intel " Tomasz Duszynski
2022-11-11  9:43 ` [PATCH 4/4] eal: add PMU support to tracing library Tomasz Duszynski
2022-11-21 12:11 ` [PATCH v2 0/4] add support for self monitoring Tomasz Duszynski
2022-11-21 12:11   ` [PATCH v2 1/4] eal: add generic support for reading PMU events Tomasz Duszynski
2022-11-21 12:11   ` [PATCH v2 2/4] eal/arm: support reading ARM PMU events in runtime Tomasz Duszynski
2022-11-21 12:11   ` [PATCH v2 3/4] eal/x86: support reading Intel " Tomasz Duszynski
2022-11-21 12:11   ` [PATCH v2 4/4] eal: add PMU support to tracing library Tomasz Duszynski
2022-11-29  9:28   ` [PATCH v3 0/4] add support for self monitoring Tomasz Duszynski
2022-11-29  9:28     ` [PATCH v3 1/4] eal: add generic support for reading PMU events Tomasz Duszynski
2022-11-30  8:32       ` zhoumin
2022-12-13  8:05         ` [EXT] " Tomasz Duszynski
2022-11-29  9:28     ` [PATCH v3 2/4] eal/arm: support reading ARM PMU events in runtime Tomasz Duszynski
2022-11-29  9:28     ` [PATCH v3 3/4] eal/x86: support reading Intel " Tomasz Duszynski
2022-11-29  9:28     ` [PATCH v3 4/4] eal: add PMU support to tracing library Tomasz Duszynski
2022-11-29 10:42     ` [PATCH v3 0/4] add support for self monitoring Morten Brørup
2022-12-13  8:23       ` Tomasz Duszynski
2022-12-13 10:43     ` [PATCH v4 " Tomasz Duszynski
2022-12-13 10:43       ` [PATCH v4 1/4] eal: add generic support for reading PMU events Tomasz Duszynski
2022-12-13 11:52         ` Morten Brørup
2022-12-14  9:38           ` Tomasz Duszynski
2022-12-14 10:41             ` Morten Brørup
2022-12-15  8:22               ` Morten Brørup
2022-12-16  7:33                 ` Morten Brørup
2023-01-05 21:14               ` Tomasz Duszynski
2023-01-05 22:07                 ` Morten Brørup
2023-01-08 15:41                   ` Tomasz Duszynski
2023-01-08 16:30                     ` Morten Brørup
2022-12-15  8:46         ` Mattias Rönnblom
2023-01-04 15:47           ` Tomasz Duszynski
2023-01-09  7:37         ` Ruifeng Wang
2023-01-09 15:40           ` Tomasz Duszynski
2022-12-13 10:43       ` [PATCH v4 2/4] eal/arm: support reading ARM PMU events in runtime Tomasz Duszynski
2022-12-13 10:43       ` [PATCH v4 3/4] eal/x86: support reading Intel " Tomasz Duszynski
2022-12-13 10:43       ` [PATCH v4 4/4] eal: add PMU support to tracing library Tomasz Duszynski
2023-01-10 23:46       ` [PATCH v5 0/4] add support for self monitoring Tomasz Duszynski
2023-01-10 23:46         ` [PATCH v5 1/4] eal: add generic support for reading PMU events Tomasz Duszynski
2023-01-11  9:05           ` Morten Brørup
2023-01-11 16:20             ` Tomasz Duszynski
2023-01-11 16:54               ` Morten Brørup
2023-01-10 23:46         ` [PATCH v5 2/4] eal/arm: support reading ARM PMU events in runtime Tomasz Duszynski
2023-01-10 23:46         ` [PATCH v5 3/4] eal/x86: support reading Intel " Tomasz Duszynski
2023-01-10 23:46         ` [PATCH v5 4/4] eal: add PMU support to tracing library Tomasz Duszynski
2023-01-11  0:32         ` [PATCH v5 0/4] add support for self monitoring Tyler Retzlaff
2023-01-11  9:31           ` Morten Brørup
2023-01-11 14:24             ` Tomasz Duszynski
2023-01-11 14:32               ` Bruce Richardson
2023-01-11  9:39           ` [EXT] " Tomasz Duszynski
2023-01-11 21:05             ` Tyler Retzlaff
2023-01-13  7:44               ` Tomasz Duszynski
2023-01-13 19:22                 ` Tyler Retzlaff
2023-01-14  9:53                   ` Morten Brørup
2023-01-19 23:39         ` [PATCH v6 " Tomasz Duszynski
2023-01-19 23:39           ` [PATCH v6 1/4] lib: add generic support for reading PMU events Tomasz Duszynski
2023-01-20  9:46             ` Morten Brørup
2023-01-26  9:40               ` Tomasz Duszynski
2023-01-26 12:29                 ` Morten Brørup
2023-01-26 12:59                   ` Bruce Richardson
2023-01-26 15:28                     ` [EXT] " Tomasz Duszynski
2023-02-02 14:27                       ` Morten Brørup
2023-01-26 15:17                   ` Tomasz Duszynski
2023-01-20 18:29             ` Tyler Retzlaff
2023-01-26  9:05               ` [EXT] " Tomasz Duszynski
2023-01-19 23:39           ` [PATCH v6 2/4] pmu: support reading ARM PMU events in runtime Tomasz Duszynski
2023-01-19 23:39           ` [PATCH v6 3/4] pmu: support reading Intel x86_64 " Tomasz Duszynski
2023-01-19 23:39           ` [PATCH v6 4/4] eal: add PMU support to tracing library Tomasz Duszynski
2023-02-01 13:17           ` [PATCH v7 0/4] add support for self monitoring Tomasz Duszynski
2023-02-01 13:17             ` [PATCH v7 1/4] lib: add generic support for reading PMU events Tomasz Duszynski
2023-02-01 13:17             ` [PATCH v7 2/4] pmu: support reading ARM PMU events in runtime Tomasz Duszynski
2023-02-01 13:17             ` [PATCH v7 3/4] pmu: support reading Intel x86_64 " Tomasz Duszynski
2023-02-01 13:17             ` [PATCH v7 4/4] eal: add PMU support to tracing library Tomasz Duszynski
2023-02-01 13:51             ` [PATCH v7 0/4] add support for self monitoring Morten Brørup
2023-02-02  7:54               ` Tomasz Duszynski
2023-02-02  9:43             ` [PATCH v8 " Tomasz Duszynski
2023-02-02  9:43               ` [PATCH v8 1/4] lib: add generic support for reading PMU events Tomasz Duszynski
2023-02-02 10:32                 ` Ruifeng Wang
2023-02-02  9:43               ` [PATCH v8 2/4] pmu: support reading ARM PMU events in runtime Tomasz Duszynski
2023-02-02  9:43               ` [PATCH v8 3/4] pmu: support reading Intel x86_64 " Tomasz Duszynski
2023-02-02  9:43               ` [PATCH v8 4/4] eal: add PMU support to tracing library Tomasz Duszynski
2023-02-02 12:49               ` [PATCH v9 0/4] add support for self monitoring Tomasz Duszynski
2023-02-02 12:49                 ` [PATCH v9 1/4] lib: add generic support for reading PMU events Tomasz Duszynski
2023-02-06 11:02                   ` David Marchand
2023-02-09 11:09                     ` [EXT] " Tomasz Duszynski
2023-02-02 12:49                 ` [PATCH v9 2/4] pmu: support reading ARM PMU events in runtime Tomasz Duszynski
2023-02-02 12:49                 ` [PATCH v9 3/4] pmu: support reading Intel x86_64 " Tomasz Duszynski
2023-02-02 12:49                 ` [PATCH v9 4/4] eal: add PMU support to tracing library Tomasz Duszynski
2023-02-13 11:31                 ` [PATCH v10 0/4] add support for self monitoring Tomasz Duszynski
2023-02-13 11:31                   ` [PATCH v10 1/4] lib: add generic support for reading PMU events Tomasz Duszynski
2023-02-16  7:39                     ` Ruifeng Wang
2023-02-16 14:44                       ` Tomasz Duszynski
2023-02-13 11:31                   ` [PATCH v10 2/4] pmu: support reading ARM PMU events in runtime Tomasz Duszynski
2023-02-16  7:41                     ` Ruifeng Wang
2023-02-13 11:31                   ` [PATCH v10 3/4] pmu: support reading Intel x86_64 " Tomasz Duszynski
2023-02-13 11:31                   ` [PATCH v10 4/4] eal: add PMU support to tracing library Tomasz Duszynski
2023-02-16 17:54                   ` [PATCH v11 0/4] add support for self monitoring Tomasz Duszynski
2023-02-16 17:54                     ` [PATCH v11 1/4] lib: add generic support for reading PMU events Tomasz Duszynski
2023-02-16 23:50                       ` Konstantin Ananyev
2023-02-17  8:49                         ` [EXT] " Tomasz Duszynski
2023-02-17 10:14                           ` Konstantin Ananyev
2023-02-19 14:23                             ` Tomasz Duszynski
2023-02-20 14:31                               ` Konstantin Ananyev
2023-02-20 16:59                                 ` Tomasz Duszynski
2023-02-20 17:21                                   ` Konstantin Ananyev
2023-02-20 20:42                                     ` Tomasz Duszynski
2023-02-21  0:48                                       ` Konstantin Ananyev
2023-02-27  8:12                                         ` Tomasz Duszynski
2023-02-28 11:35                                           ` Konstantin Ananyev
2023-02-21 12:15                           ` Konstantin Ananyev
2023-02-21  2:17                       ` Konstantin Ananyev
2023-02-27  9:19                         ` [EXT] " Tomasz Duszynski
2023-02-27 20:53                           ` Konstantin Ananyev
2023-02-28  8:25                             ` Morten Brørup
2023-02-28 12:04                               ` Konstantin Ananyev
2023-02-28 13:15                                 ` Morten Brørup
2023-02-28 16:22                                 ` Morten Brørup
2023-03-05 16:30                                   ` Konstantin Ananyev
2023-02-28  9:57                             ` Tomasz Duszynski
2023-02-28 11:58                               ` Konstantin Ananyev
2023-02-16 17:55                     ` [PATCH v11 2/4] pmu: support reading ARM PMU events in runtime Tomasz Duszynski
2023-02-16 17:55                     ` [PATCH v11 3/4] pmu: support reading Intel x86_64 " Tomasz Duszynski
2023-02-16 17:55                     ` [PATCH v11 4/4] eal: add PMU support to tracing library Tomasz Duszynski
2023-02-16 18:03                     ` [PATCH v11 0/4] add support for self monitoring Ruifeng Wang
2023-05-04  8:02                     ` David Marchand
2023-07-31 12:33                       ` Thomas Monjalon
2023-08-07  8:11                         ` [EXT] " Tomasz Duszynski
2023-09-21  8:26                           ` David Marchand
2024-09-02 14:48                             ` Morten Brørup
2024-09-05  3:49                               ` Tomasz Duszynski
2024-09-27 22:06                     ` [PATCH v12 " Tomasz Duszynski
2024-09-27 22:06                       ` [PATCH v12 1/4] lib: add generic support for reading PMU events Tomasz Duszynski
2024-10-06 14:30                         ` Morten Brørup
2024-10-09  9:17                           ` Tomasz Duszynski
2024-10-07  6:59                         ` Jerin Jacob
2024-10-09  7:50                           ` [EXTERNAL] " Tomasz Duszynski
2024-09-27 22:06                       ` [PATCH v12 2/4] pmu: support reading ARM PMU events in runtime Tomasz Duszynski
2024-09-27 22:06                       ` [PATCH v12 3/4] pmu: support reading Intel x86_64 " Tomasz Duszynski
2024-09-27 22:06                       ` [PATCH v12 4/4] eal: add PMU support to tracing library Tomasz Duszynski
2024-10-04 11:00                       ` [PATCH v12 0/4] add support for self monitoring David Marchand
2024-10-09 12:45                         ` [EXTERNAL] " Tomasz Duszynski
2024-10-09 11:23                       ` [PATCH v13 " Tomasz Duszynski
2024-10-09 11:23                         ` [PATCH v13 1/4] lib: add generic support for reading PMU events Tomasz Duszynski
2024-10-09 11:23                         ` [PATCH v13 2/4] pmu: support reading ARM PMU events in runtime Tomasz Duszynski
2024-10-09 11:23                         ` [PATCH v13 3/4] pmu: support reading Intel x86_64 " Tomasz Duszynski
2024-10-09 11:23                         ` [PATCH v13 4/4] eal: add PMU support to tracing library Tomasz Duszynski
2024-10-09 12:50                           ` Morten Brørup
2024-10-09 17:56                             ` Stephen Hemminger
2024-10-10  7:24                               ` [EXTERNAL] " Tomasz Duszynski
2024-10-10 12:48                                 ` Morten Brørup
2024-10-11  9:49                         ` [PATCH v14 0/4] add support for self monitoring Tomasz Duszynski
2024-10-11  9:49                           ` [PATCH v14 1/4] lib: add generic support for reading PMU events Tomasz Duszynski
2024-10-11 11:56                             ` Konstantin Ananyev
2024-10-11 14:19                               ` Stephen Hemminger
2024-10-15  9:14                                 ` [EXTERNAL] " Tomasz Duszynski
2024-10-15  9:08                               ` Tomasz Duszynski
2024-10-16  8:49                                 ` Konstantin Ananyev
2024-10-17  7:11                                   ` Tomasz Duszynski
2024-10-11  9:49                           ` [PATCH v14 2/4] pmu: support reading ARM PMU events in runtime Tomasz Duszynski
2024-10-11  9:49                           ` [PATCH v14 3/4] pmu: support reading Intel x86_64 " Tomasz Duszynski
2024-10-11  9:49                           ` [PATCH v14 4/4] eal: add PMU support to tracing library Tomasz Duszynski
2024-10-11 13:29                             ` David Marchand
2024-10-15  9:18                               ` [EXTERNAL] " Tomasz Duszynski
2024-10-25  8:54                           ` [PATCH v15 0/4] add support for self monitoring Tomasz Duszynski
2024-10-25  8:54                             ` [PATCH v15 1/4] lib: add generic support for reading PMU events Tomasz Duszynski
2024-11-05 10:58                               ` Konstantin Ananyev
2024-10-25  8:54                             ` [PATCH v15 2/4] pmu: support reading ARM PMU events in runtime Tomasz Duszynski
2024-10-25  8:54                             ` [PATCH v15 3/4] pmu: support reading Intel x86_64 " Tomasz Duszynski
2024-10-25  8:54                             ` [PATCH v15 4/4] eal: add PMU support to tracing library Tomasz Duszynski
2024-10-25 11:02                               ` Jerin Jacob
2024-10-28 10:32                                 ` [EXTERNAL] " Tomasz Duszynski
2024-11-05  7:41                                   ` Morten Brørup
2024-11-08 10:36                                     ` Tomasz Duszynski
2024-11-05 11:04                               ` Konstantin Ananyev
2024-11-08 11:44                                 ` Tomasz Duszynski
2024-11-05  4:04                             ` Tomasz Duszynski [this message]
2023-01-25 10:33         ` [PATCH 0/2] add platform bus Tomasz Duszynski
2023-01-25 10:33           ` [PATCH 1/2] lib: add helper to read strings from sysfs files Tomasz Duszynski
2023-01-25 10:39             ` Thomas Monjalon
2023-01-25 16:16               ` Tyler Retzlaff
2023-01-26  8:30                 ` [EXT] " Tomasz Duszynski
2023-01-26 17:21                   ` Tyler Retzlaff
2023-01-26  8:35               ` Tomasz Duszynski
2023-01-25 10:33           ` [PATCH 2/2] bus: add platform bus Tomasz Duszynski
2023-01-25 10:41           ` [PATCH 0/2] " Tomasz Duszynski
2023-02-16 20:56         ` [PATCH v5 0/4] add support for self monitoring Liang Ma

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DM4PR18MB436808478A9AF295D4092E35D2522@DM4PR18MB4368.namprd18.prod.outlook.com \
    --to=tduszynski@marvell.com \
    --cc=Ruifeng.Wang@arm.com \
    --cc=bruce.richardson@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=jerinj@marvell.com \
    --cc=konstantin.v.ananyev@yandex.ru \
    --cc=mattias.ronnblom@ericsson.com \
    --cc=mb@smartsharesystems.com \
    --cc=roretzla@linux.microsoft.com \
    --cc=stephen@networkplumber.org \
    --cc=thomas@monjalon.net \
    --cc=zhoumin@loongson.cn \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).