From: Amit Prakash Shukla <amitprakashs@marvell.com>
To: Anatoly Burakov <anatoly.burakov@intel.com>
Cc: <dev@dpdk.org>, <jerinj@marvell.com>, <david.marchand@redhat.com>,
<bruce.richardson@intel.com>, <ciara.power@intel.com>,
<dmitry.kozliuk@gmail.com>,
Amit Prakash Shukla <amitprakashs@marvell.com>
Subject: [PATCH v6] mem: telemetry support for memseg and element information
Date: Tue, 25 Oct 2022 17:21:38 +0530 [thread overview]
Message-ID: <20221025115138.4071745-1-amitprakashs@marvell.com> (raw)
In-Reply-To: <20220929114313.1346972-1-amitprakashs@marvell.com>
Changes adds telemetry support to display memory occupancy
in memseg and the information of the elements allocated from
a memseg based on arguments provided by user. This patch
adds following endpoints:
1. /eal/memseg_lists
The command displays the memseg list from which the memory
has been allocated.
Example:
--> /eal/memseg_lists
{"/eal/memseg_lists": [0, 1]}
2. /eal/memseg_list_info,<memseg-list-id>
The command outputs the memsegs, from which the memory is
allocated, for the memseg_list given as input.
Example:
--> /eal/memseg_list_info,0
{"/eal/memseg_list_info": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, \
12, 13, 14, 15]}
3. /eal/memseg_info,<memseg-list-id>,<memseg-id>
The command outputs the memseg information based on the
memseg-list and the memseg-id given as input.
Example:
--> /eal/memseg_info,0,10
{"/eal/memseg_info": {"Memseg_list_index": 0, \
"Memseg_index": 10, "Start_addr": "0x101600000", \
"End_addr": "0x101800000", "Size": 2097152, "Hugepage_size": 2097152, \
"Socket_id": 0, "flags": 0}}
--> /eal/memseg_info,0,15
{"/eal/memseg_info": {"Memseg_list_index": 0, "Memseg_index": 15, \
"Start_addr": "0x102000000", "End_addr": "0x102200000", \
"Size": 2097152, "Hugepage_size": 2097152, "Socket_id": 0, "flags": 0}}
4. /eal/mem_element_list,<heap-id>,<memseg-list-id>,<memseg-id>
The command outputs number of elements in a memseg based
on the heap-id, memseg-list-id and memseg-id given as input.
Example:
--> /eal/mem_element_list,0,0,10
{"/eal/mem_element_list": {"Element_count": 52}}
--> /eal/mem_element_list,0,1,15
{"/eal/mem_element_list": {"Element_count": 52}}
5. /eal/mem_element_info,<heap-id>,<memseg-list-id>,<memseg-id>, \
<elem-start-id>,<elem-end-id>
The command outputs element information like element start
address, end address, to which memseg it belongs, element
state, element size. User can give a range of elements to be
printed.
Example:
--> /eal/mem_element_info,0,0,15,1,2
{"/eal/mem_element_info": {"element_1": {"msl_id": 0, "ms_id": 15, \
"memseg_start_addr": "0x102000000", "memseg_end_addr": "0x102200000", \
"element_start_addr": "0x102000b00", "element_end_addr": "0x102003380", \
"element_size": 10368, "element_state": "Busy"}, "element_2": \
{"msl_id": 0, "ms_id": 15, "memseg_start_addr": "0x102000000", \
"memseg_end_addr": "0x102200000", "element_start_addr": "0x102003380", \
"element_end_addr": "0x102005c00", "element_size": 10368, \
"element_state": "Busy"}, "Element_count": 2}}
Signed-off-by: Amit Prakash Shukla <amitprakashs@marvell.com>
---
v2:
- Fixed compilation error related int-to-pointer-cast
- Changes for code review suggestions
v3:
- Commit message changes
- Renaming end-points
- Changing input parameters to comma-seperated
- Reverting telemetry output buffer size
v4:
- Patch-2 adds telemetry support to display system memory
v5:
- Removed command help related changes
v6:
- Changes for code review suggestions
- Dropped Patch-2 from the series
lib/eal/common/eal_common_memory.c | 387 ++++++++++++++++++++++++++++-
1 file changed, 382 insertions(+), 5 deletions(-)
diff --git a/lib/eal/common/eal_common_memory.c b/lib/eal/common/eal_common_memory.c
index 688dc615d7..f43dbf2f12 100644
--- a/lib/eal/common/eal_common_memory.c
+++ b/lib/eal/common/eal_common_memory.c
@@ -3,6 +3,7 @@
*/
#include <errno.h>
+#include <ctype.h>
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
@@ -26,6 +27,7 @@
#include "eal_memcfg.h"
#include "eal_options.h"
#include "malloc_heap.h"
+#include "malloc_elem.h"
/*
* Try to mmap *size bytes in /dev/zero. If it is successful, return the
@@ -1113,11 +1115,17 @@ rte_eal_memory_init(void)
}
#ifndef RTE_EXEC_ENV_WINDOWS
-#define EAL_MEMZONE_LIST_REQ "/eal/memzone_list"
-#define EAL_MEMZONE_INFO_REQ "/eal/memzone_info"
-#define EAL_HEAP_LIST_REQ "/eal/heap_list"
-#define EAL_HEAP_INFO_REQ "/eal/heap_info"
-#define ADDR_STR 15
+#define EAL_MEMZONE_LIST_REQ "/eal/memzone_list"
+#define EAL_MEMZONE_INFO_REQ "/eal/memzone_info"
+#define EAL_HEAP_LIST_REQ "/eal/heap_list"
+#define EAL_HEAP_INFO_REQ "/eal/heap_info"
+#define EAL_MEMSEG_LISTS_REQ "/eal/memseg_lists"
+#define EAL_MEMSEG_LIST_INFO_REQ "/eal/memseg_list_info"
+#define EAL_MEMSEG_INFO_REQ "/eal/memseg_info"
+#define EAL_ELEMENT_LIST_REQ "/eal/mem_element_list"
+#define EAL_ELEMENT_INFO_REQ "/eal/mem_element_info"
+#define ADDR_STR 15
+
/* Telemetry callback handler to return heap stats for requested heap id. */
static int
@@ -1265,6 +1273,358 @@ handle_eal_memzone_list_request(const char *cmd __rte_unused,
return 0;
}
+/* n_vals is the number of params to be parsed. */
+static int
+parse_params(const char *params, uint32_t *vals, size_t n_vals)
+{
+ char dlim[2] = ",";
+ char *params_args;
+ size_t count = 0;
+ char *token;
+
+ if (vals == NULL || params == NULL || strlen(params) == 0)
+ return -1;
+
+ /* strtok expects char * and param is const char *. Hence on using
+ * params as "const char *" compiler throws warning.
+ */
+ params_args = strdup(params);
+ if (params_args == NULL)
+ return -1;
+
+ token = strtok(params_args, dlim);
+ while (token && isdigit(*token) && count < n_vals) {
+ vals[count++] = strtoul(token, NULL, 10);
+ token = strtok(NULL, dlim);
+ }
+
+ free(params_args);
+
+ if (count < n_vals)
+ return -1;
+
+ return 0;
+}
+
+static int
+handle_eal_memseg_lists_request(const char *cmd __rte_unused,
+ const char *params __rte_unused,
+ struct rte_tel_data *d)
+{
+ struct rte_mem_config *mcfg;
+ int i;
+
+ rte_tel_data_start_array(d, RTE_TEL_INT_VAL);
+
+ rte_mcfg_mem_read_lock();
+ mcfg = rte_eal_get_configuration()->mem_config;
+
+ for (i = 0; i < RTE_MAX_MEMSEG_LISTS; i++) {
+ struct rte_memseg_list *msl = &mcfg->memsegs[i];
+ if (msl->memseg_arr.count == 0)
+ continue;
+
+ rte_tel_data_add_array_int(d, i);
+ }
+ rte_mcfg_mem_read_unlock();
+
+ return 0;
+}
+
+static int
+handle_eal_memseg_list_info_request(const char *cmd __rte_unused,
+ const char *params, struct rte_tel_data *d)
+{
+ struct rte_mem_config *mcfg;
+ struct rte_memseg_list *msl;
+ struct rte_fbarray *arr;
+ uint32_t ms_list_idx;
+ int ms_idx;
+ uint32_t vals[1]; /* size of an array == num params to be parsed. */
+
+ if (parse_params(params, vals, RTE_DIM(vals)) < 0)
+ return -1;
+
+ ms_list_idx = vals[0];
+ if (ms_list_idx >= RTE_MAX_MEMSEG_LISTS)
+ return -1;
+
+ rte_tel_data_start_array(d, RTE_TEL_INT_VAL);
+
+ rte_mcfg_mem_read_lock();
+ mcfg = rte_eal_get_configuration()->mem_config;
+ msl = &mcfg->memsegs[ms_list_idx];
+ if (msl->memseg_arr.count == 0)
+ goto done;
+
+ arr = &msl->memseg_arr;
+
+ ms_idx = rte_fbarray_find_next_used(arr, 0);
+ while (ms_idx >= 0) {
+ rte_tel_data_add_array_int(d, ms_idx);
+ ms_idx = rte_fbarray_find_next_used(arr, ms_idx + 1);
+ }
+
+done:
+ rte_mcfg_mem_read_unlock();
+
+ return 0;
+}
+
+static int
+handle_eal_memseg_info_request(const char *cmd __rte_unused,
+ const char *params, struct rte_tel_data *d)
+{
+ struct rte_mem_config *mcfg;
+ uint64_t ms_start_addr, ms_end_addr, ms_size, hugepage_size;
+ struct rte_memseg_list *msl;
+ const struct rte_memseg *ms;
+ struct rte_fbarray *arr;
+ char addr[ADDR_STR];
+ uint32_t ms_list_idx = 0;
+ uint32_t ms_idx = 0;
+ int32_t ms_socket_id;
+ uint32_t ms_flags;
+ uint32_t vals[2]; /* size of an array == num params to be parsed. */
+
+ if (parse_params(params, vals, RTE_DIM(vals)) < 0)
+ return -1;
+
+ ms_list_idx = vals[0];
+ if (ms_list_idx >= RTE_MAX_MEMSEG_LISTS)
+ return -1;
+
+ ms_idx = vals[1];
+
+ rte_mcfg_mem_read_lock();
+
+ mcfg = rte_eal_get_configuration()->mem_config;
+ msl = &mcfg->memsegs[ms_list_idx];
+ if (msl->memseg_arr.count == 0) {
+ rte_mcfg_mem_read_unlock();
+ return -1;
+ }
+
+ arr = &msl->memseg_arr;
+ ms = rte_fbarray_get(arr, ms_idx);
+ if (ms == NULL) {
+ rte_mcfg_mem_read_unlock();
+ RTE_LOG(DEBUG, EAL, "Error fetching requested memseg.\n");
+ return -1;
+ }
+
+ ms_start_addr = ms->addr_64;
+ ms_end_addr = (uint64_t)RTE_PTR_ADD(ms_start_addr, ms->len);
+ ms_size = ms->len;
+ hugepage_size = ms->hugepage_sz;
+ ms_socket_id = ms->socket_id;
+ ms_flags = ms->flags;
+
+ rte_mcfg_mem_read_unlock();
+
+ rte_tel_data_start_dict(d);
+ rte_tel_data_add_dict_int(d, "Memseg_list_index", ms_list_idx);
+ rte_tel_data_add_dict_int(d, "Memseg_index", ms_idx);
+ snprintf(addr, ADDR_STR, "0x%"PRIx64, ms_start_addr);
+ rte_tel_data_add_dict_string(d, "Start_addr", addr);
+ snprintf(addr, ADDR_STR, "0x%"PRIx64, ms_end_addr);
+ rte_tel_data_add_dict_string(d, "End_addr", addr);
+ rte_tel_data_add_dict_u64(d, "Size", ms_size);
+ rte_tel_data_add_dict_u64(d, "Hugepage_size", hugepage_size);
+ rte_tel_data_add_dict_int(d, "Socket_id", ms_socket_id);
+ rte_tel_data_add_dict_int(d, "flags", ms_flags);
+
+ return 0;
+}
+
+static int
+handle_eal_element_list_request(const char *cmd __rte_unused,
+ const char *params, struct rte_tel_data *d)
+{
+ struct rte_mem_config *mcfg;
+ struct rte_memseg_list *msl;
+ const struct rte_memseg *ms;
+ struct malloc_elem *elem;
+ struct malloc_heap *heap;
+ uint64_t ms_start_addr, ms_end_addr;
+ uint64_t elem_start_addr, elem_end_addr;
+ uint32_t ms_list_idx = 0;
+ uint32_t heap_id = 0;
+ uint32_t ms_idx = 0;
+ int elem_count = 0;
+ uint32_t vals[3]; /* size of an array == num params to be parsed. */
+
+ if (parse_params(params, vals, RTE_DIM(vals)) < 0)
+ return -1;
+
+ heap_id = vals[0];
+ if (heap_id >= RTE_MAX_HEAPS)
+ return -1;
+
+ ms_list_idx = vals[1];
+ if (ms_list_idx >= RTE_MAX_MEMSEG_LISTS)
+ return -1;
+
+ ms_idx = vals[2];
+
+ rte_mcfg_mem_read_lock();
+
+ mcfg = rte_eal_get_configuration()->mem_config;
+ msl = &mcfg->memsegs[ms_list_idx];
+ ms = rte_fbarray_get(&msl->memseg_arr, ms_idx);
+ if (ms == NULL) {
+ rte_mcfg_mem_read_unlock();
+ RTE_LOG(DEBUG, EAL, "Error fetching requested memseg.\n");
+ return -1;
+ }
+
+ ms_start_addr = ms->addr_64;
+ ms_end_addr = (uint64_t)RTE_PTR_ADD(ms_start_addr, ms->len);
+ rte_mcfg_mem_read_unlock();
+
+ rte_tel_data_start_dict(d);
+
+ heap = &mcfg->malloc_heaps[heap_id];
+ rte_spinlock_lock(&heap->lock);
+
+ elem = heap->first;
+ while (elem) {
+ elem_start_addr = (uint64_t)elem;
+ elem_end_addr =
+ (uint64_t)RTE_PTR_ADD(elem_start_addr, elem->size);
+
+ if ((uint64_t)elem_start_addr >= ms_start_addr &&
+ (uint64_t)elem_end_addr <= ms_end_addr)
+ elem_count++;
+ elem = elem->next;
+ }
+
+ rte_spinlock_unlock(&heap->lock);
+
+ rte_tel_data_add_dict_int(d, "Element_count", elem_count);
+
+ return 0;
+}
+
+static int
+handle_eal_element_info_request(const char *cmd __rte_unused,
+ const char *params, struct rte_tel_data *d)
+{
+ struct rte_mem_config *mcfg;
+ struct rte_memseg_list *msl;
+ const struct rte_memseg *ms;
+ struct malloc_elem *elem;
+ struct malloc_heap *heap;
+ struct rte_tel_data *c;
+ uint64_t ms_start_addr, ms_end_addr;
+ uint64_t elem_start_addr, elem_end_addr;
+ uint32_t ms_list_idx = 0;
+ uint32_t heap_id = 0;
+ uint32_t ms_idx = 0;
+ uint32_t start_elem = 0, end_elem = 0;
+ uint32_t count = 0, elem_count = 0;
+ char str[ADDR_STR];
+ uint32_t vals[5]; /* size of an array == num params to be parsed. */
+
+ if (parse_params(params, vals, RTE_DIM(vals)) < 0)
+ return -1;
+
+ heap_id = vals[0];
+ if (heap_id >= RTE_MAX_HEAPS)
+ return -1;
+
+ ms_list_idx = vals[1];
+ if (ms_list_idx >= RTE_MAX_MEMSEG_LISTS)
+ return -1;
+
+ ms_idx = vals[2];
+ start_elem = vals[3];
+ end_elem = vals[4];
+
+ if (end_elem < start_elem)
+ return -1;
+
+ rte_mcfg_mem_read_lock();
+
+ mcfg = rte_eal_get_configuration()->mem_config;
+ msl = &mcfg->memsegs[ms_list_idx];
+ ms = rte_fbarray_get(&msl->memseg_arr, ms_idx);
+ if (ms == NULL) {
+ rte_mcfg_mem_read_unlock();
+ RTE_LOG(DEBUG, EAL, "Error fetching requested memseg.\n");
+ return -1;
+ }
+
+ ms_start_addr = ms->addr_64;
+ ms_end_addr = (uint64_t)RTE_PTR_ADD(ms_start_addr, ms->len);
+
+ rte_mcfg_mem_read_unlock();
+
+ rte_tel_data_start_dict(d);
+
+ heap = &mcfg->malloc_heaps[heap_id];
+ rte_spinlock_lock(&heap->lock);
+
+ elem = heap->first;
+ while (elem) {
+ elem_start_addr = (uint64_t)elem;
+ elem_end_addr =
+ (uint64_t)RTE_PTR_ADD(elem_start_addr, elem->size);
+
+ if (elem_start_addr < ms_start_addr ||
+ elem_end_addr > ms_end_addr) {
+ elem = elem->next;
+ continue;
+ }
+
+ if (count < start_elem) {
+ elem = elem->next;
+ count++;
+ continue;
+ }
+
+ c = rte_tel_data_alloc();
+ if (c == NULL)
+ break;
+
+ rte_tel_data_start_dict(c);
+ rte_tel_data_add_dict_int(c, "msl_id", ms_list_idx);
+ rte_tel_data_add_dict_int(c, "ms_id", ms_idx);
+ snprintf(str, ADDR_STR, "0x%"PRIx64, ms_start_addr);
+ rte_tel_data_add_dict_string(c, "memseg_start_addr", str);
+ snprintf(str, ADDR_STR, "0x%"PRIx64, ms_end_addr);
+ rte_tel_data_add_dict_string(c, "memseg_end_addr", str);
+ snprintf(str, ADDR_STR, "0x%"PRIx64, elem_start_addr);
+ rte_tel_data_add_dict_string(c, "element_start_addr", str);
+ snprintf(str, ADDR_STR, "0x%"PRIx64, elem_end_addr);
+ rte_tel_data_add_dict_string(c, "element_end_addr", str);
+ rte_tel_data_add_dict_int(c, "element_size", elem->size);
+ snprintf(str, ADDR_STR, "%s", elem->state == 0 ? "Free" :
+ elem->state == 1 ? "Busy" : elem->state == 2 ?
+ "Pad" : "Error");
+ rte_tel_data_add_dict_string(c, "element_state", str);
+
+ snprintf(str, ADDR_STR, "%s_%u", "element", count);
+ if (rte_tel_data_add_dict_container(d, str, c, 0) != 0) {
+ rte_tel_data_free(c);
+ break;
+ }
+
+ elem_count++;
+ count++;
+ if (count > end_elem)
+ break;
+
+ elem = elem->next;
+ }
+
+ rte_spinlock_unlock(&heap->lock);
+
+ rte_tel_data_add_dict_int(d, "Element_count", elem_count);
+
+ return 0;
+}
+
RTE_INIT(memory_telemetry)
{
rte_telemetry_register_cmd(
@@ -1279,5 +1639,22 @@ RTE_INIT(memory_telemetry)
rte_telemetry_register_cmd(
EAL_HEAP_INFO_REQ, handle_eal_heap_info_request,
"Returns malloc heap stats. Parameters: int heap_id");
+ rte_telemetry_register_cmd(
+ EAL_MEMSEG_LISTS_REQ,
+ handle_eal_memseg_lists_request,
+ "Returns array of memseg list IDs. Takes no parameters");
+ rte_telemetry_register_cmd(
+ EAL_MEMSEG_LIST_INFO_REQ,
+ handle_eal_memseg_list_info_request,
+ "Returns memseg list info. Parameters: int memseg_list_id");
+ rte_telemetry_register_cmd(
+ EAL_MEMSEG_INFO_REQ, handle_eal_memseg_info_request,
+ "Returns memseg info. Parameter: int memseg_list_id,int memseg_id");
+ rte_telemetry_register_cmd(EAL_ELEMENT_LIST_REQ,
+ handle_eal_element_list_request,
+ "Returns array of heap element IDs. Parameters: int heap_id, int memseg_list_id, int memseg_id");
+ rte_telemetry_register_cmd(EAL_ELEMENT_INFO_REQ,
+ handle_eal_element_info_request,
+ "Returns element info. Parameters: int heap_id, int memseg_list_id, int memseg_id, int start_elem_id, int end_elem_id");
}
#endif
--
2.25.1
next prev parent reply other threads:[~2022-10-25 11:51 UTC|newest]
Thread overview: 42+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-05-19 6:30 [PATCH] " Amit Prakash Shukla
2022-05-19 12:42 ` David Marchand
2022-05-19 18:57 ` [PATCH v2] " Amit Prakash Shukla
2022-05-23 11:14 ` Bruce Richardson
2022-05-23 13:35 ` [EXT] " Amit Prakash Shukla
2022-05-23 13:43 ` Bruce Richardson
2022-05-24 10:30 ` Amit Prakash Shukla
2022-05-24 10:33 ` [PATCH v3] " Amit Prakash Shukla
2022-05-25 10:33 ` [PATCH v4 1/2] " Amit Prakash Shukla
2022-05-25 10:33 ` [PATCH v4 2/2] mem: telemetry support for system memory information Amit Prakash Shukla
2022-06-30 5:54 ` Amit Prakash Shukla
2022-07-21 11:21 ` Amit Prakash Shukla
2022-06-14 12:50 ` [PATCH v4 1/2] mem: telemetry support for memseg and element information Amit Prakash Shukla
2022-06-30 5:52 ` Amit Prakash Shukla
2022-07-21 11:20 ` Amit Prakash Shukla
2022-09-29 8:29 ` David Marchand
2022-09-29 11:30 ` [EXT] " Amit Prakash Shukla
2022-09-29 11:43 ` [PATCH v5 " Amit Prakash Shukla
2022-09-29 11:43 ` [PATCH v5 2/2] mem: telemetry support for system memory information Amit Prakash Shukla
2022-10-07 19:46 ` David Marchand
2022-10-11 7:10 ` [EXT] " Amit Prakash Shukla
2022-10-20 19:18 ` Dmitry Kozlyuk
2022-10-20 19:50 ` Stephen Hemminger
2022-10-06 7:07 ` [PATCH v5 1/2] mem: telemetry support for memseg and element information Amit Prakash Shukla
2022-10-07 19:52 ` David Marchand
2022-10-07 19:48 ` David Marchand
2022-10-11 7:22 ` [EXT] " Amit Prakash Shukla
2022-10-20 11:40 ` Dmitry Kozlyuk
2022-10-21 19:26 ` [EXT] " Amit Prakash Shukla
2022-10-21 20:07 ` Dmitry Kozlyuk
2022-10-25 7:25 ` Amit Prakash Shukla
2022-10-25 11:51 ` Amit Prakash Shukla [this message]
2022-10-25 13:02 ` [PATCH v7] " Amit Prakash Shukla
2022-12-06 11:46 ` Amit Prakash Shukla
2023-01-30 10:18 ` Amit Prakash Shukla
2023-02-20 11:10 ` Thomas Monjalon
2023-02-28 7:30 ` [EXT] " Amit Prakash Shukla
2023-05-15 11:51 ` Amit Prakash Shukla
2023-05-16 10:47 ` Burakov, Anatoly
2023-05-17 9:08 ` [EXT] " Amit Prakash Shukla
2023-05-17 9:21 ` [PATCH v8] " Amit Prakash Shukla
2023-06-07 20:40 ` David Marchand
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20221025115138.4071745-1-amitprakashs@marvell.com \
--to=amitprakashs@marvell.com \
--cc=anatoly.burakov@intel.com \
--cc=bruce.richardson@intel.com \
--cc=ciara.power@intel.com \
--cc=david.marchand@redhat.com \
--cc=dev@dpdk.org \
--cc=dmitry.kozliuk@gmail.com \
--cc=jerinj@marvell.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).