From: Jianfeng Tan <jianfeng.tan@intel.com>
To: dev@dpdk.org
Cc: konstantin.ananyev@intel.com, jingjing.wu@intel.com,
Jianfeng Tan <jianfeng.tan@intel.com>,
Zhe Tao <zhe.tao@intel.com>
Subject: [dpdk-dev] [PATCH v5 3/3] app/testpmd: support tunneled TSO in csum fwd engine
Date: Mon, 26 Sep 2016 13:48:34 +0000 [thread overview]
Message-ID: <1474897714-28037-1-git-send-email-jianfeng.tan@intel.com> (raw)
In-Reply-To: <1467752375-25984-1-git-send-email-zhe.tao@intel.com>
Add a new command "tunnel_tso set <tso_segsz> <port>" to enable
segmentation offload and set MSS to tso_segsz. Another command,
"tunnel_tso show <port>" is added to show tunneled packet MSS.
Result 0 means tunnel_tso is disabled.
The original commands, "tso set <tso_segsz> <port>" and "tso show
<port>" are only reponsible for non-tunneled packets. And the new
commands are for tunneled packets.
Below conditions are needed to make it work:
a. tunnel TSO is supported by the NIC;
b. "csum parse_tunnel" must be set so that tunneled pkts are
recognized;
c. for tunneled pkts with outer L3 is IPv4, "csum set outer-ip"
must be set to hw, because after tso, total_len of outer IP
header is changed, and the checksum of outer IP header calculated
by sw should be wrong; that is not necessary for IPv6 tunneled
pkts because there's no checksum field to be filled anymore.
Suggested-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Signed-off-by: Zhe Tao <zhe.tao@intel.com>
Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
---
v5:
-- Instead of reuse original tso command, add a new command for
tunneled tso;
-- Fix a implicit conversion from long -> int bug, as the parameter
of process_outer_cksums() in previous version.
app/test-pmd/cmdline.c | 132 +++++++++++++++++++++++++++++++++++++++++++++---
app/test-pmd/csumonly.c | 69 ++++++++++++++++++-------
app/test-pmd/testpmd.h | 3 +-
3 files changed, 179 insertions(+), 25 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 17d238f..a1da8b8 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -3472,7 +3472,7 @@ cmdline_parse_inst_t cmd_csum_tunnel = {
},
};
-/* *** ENABLE HARDWARE SEGMENTATION IN TX PACKETS *** */
+/* *** ENABLE HARDWARE SEGMENTATION IN TX NON-TUNNELED PACKETS *** */
struct cmd_tso_set_result {
cmdline_fixed_string_t tso;
cmdline_fixed_string_t mode;
@@ -3495,9 +3495,9 @@ cmd_tso_set_parsed(void *parsed_result,
ports[res->port_id].tso_segsz = res->tso_segsz;
if (ports[res->port_id].tso_segsz == 0)
- printf("TSO is disabled\n");
+ printf("TSO for non-tunneled packets is disabled\n");
else
- printf("TSO segment size is %d\n",
+ printf("TSO segment size for non-tunneled packets is %d\n",
ports[res->port_id].tso_segsz);
/* display warnings if configuration is not supported by the NIC */
@@ -3525,8 +3525,8 @@ cmdline_parse_token_num_t cmd_tso_set_portid =
cmdline_parse_inst_t cmd_tso_set = {
.f = cmd_tso_set_parsed,
.data = NULL,
- .help_str = "Set TSO segment size for csum engine (0 to disable): "
- "tso set <tso_segsz> <port>",
+ .help_str = "Set TSO segment size of non-tunneled packets "
+ "for csum engine (0 to disable): tso set <tso_segsz> <port>",
.tokens = {
(void *)&cmd_tso_set_tso,
(void *)&cmd_tso_set_mode,
@@ -3544,8 +3544,8 @@ cmdline_parse_token_string_t cmd_tso_show_mode =
cmdline_parse_inst_t cmd_tso_show = {
.f = cmd_tso_set_parsed,
.data = NULL,
- .help_str = "Show TSO segment size for csum engine: "
- "tso show <port>",
+ .help_str = "Show TSO segment size of non-tunneled packets "
+ "for csum engine: tso show <port>",
.tokens = {
(void *)&cmd_tso_set_tso,
(void *)&cmd_tso_show_mode,
@@ -3554,6 +3554,122 @@ cmdline_parse_inst_t cmd_tso_show = {
},
};
+/* *** ENABLE HARDWARE SEGMENTATION IN TX TUNNELED PACKETS *** */
+struct cmd_tunnel_tso_set_result {
+ cmdline_fixed_string_t tso;
+ cmdline_fixed_string_t mode;
+ uint16_t tso_segsz;
+ uint8_t port_id;
+};
+
+static void
+check_tunnel_tso_nic_support(uint8_t port_id)
+{
+ struct rte_eth_dev_info dev_info;
+
+ rte_eth_dev_info_get(port_id, &dev_info);
+ if (!(dev_info.tx_offload_capa & DEV_TX_OFFLOAD_VXLAN_TNL_TSO))
+ printf("Warning: TSO enabled but VXLAN TUNNEL TSO not "
+ "supported by port %d\n", port_id);
+ if (!(dev_info.tx_offload_capa & DEV_TX_OFFLOAD_GRE_TNL_TSO))
+ printf("Warning: TSO enabled but GRE TUNNEL TSO not "
+ "supported by port %d\n", port_id);
+ if (!(dev_info.tx_offload_capa & DEV_TX_OFFLOAD_IPIP_TNL_TSO))
+ printf("Warning: TSO enabled but IPIP TUNNEL TSO not "
+ "supported by port %d\n", port_id);
+ if (!(dev_info.tx_offload_capa & DEV_TX_OFFLOAD_GENEVE_TNL_TSO))
+ printf("Warning: TSO enabled but GENEVE TUNNEL TSO not "
+ "supported by port %d\n", port_id);
+}
+
+static void
+cmd_tunnel_tso_set_parsed(void *parsed_result,
+ __attribute__((unused)) struct cmdline *cl,
+ __attribute__((unused)) void *data)
+{
+ struct cmd_tunnel_tso_set_result *res = parsed_result;
+
+ if (port_id_is_invalid(res->port_id, ENABLED_WARN))
+ return;
+
+ if (!strcmp(res->mode, "set"))
+ ports[res->port_id].tunnel_tso_segsz = res->tso_segsz;
+
+ if (ports[res->port_id].tunnel_tso_segsz == 0)
+ printf("TSO for tunneled packets is disabled\n");
+ else {
+ printf("TSO segment size for tunneled packets is %d\n",
+ ports[res->port_id].tunnel_tso_segsz);
+
+ /* Below conditions are needed to make it work:
+ * (1) tunnel TSO is supported by the NIC;
+ * (2) "csum parse_tunnel" must be set so that tunneled pkts
+ * are recognized;
+ * (3) for tunneled pkts with outer L3 of IPv4,
+ * "csum set outer-ip" must be set to hw, because after tso,
+ * total_len of outer IP header is changed, and the checksum
+ * of outer IP header calculated by sw should be wrong; that
+ * is not necessary for IPv6 tunneled pkts because there's no
+ * checksum in IP header anymore.
+ */
+ check_tunnel_tso_nic_support(res->port_id);
+
+ if (!(ports[res->port_id].tx_ol_flags &
+ TESTPMD_TX_OFFLOAD_PARSE_TUNNEL))
+ printf("Warning: csum parse_tunnel must be set "
+ "so that tunneled packets are recognized\n");
+ if (!(ports[res->port_id].tx_ol_flags &
+ TESTPMD_TX_OFFLOAD_OUTER_IP_CKSUM))
+ printf("Warning: csum set outer-ip must be set to hw "
+ "if outer L3 is IPv4; not necessary for IPv6\n");
+ }
+}
+
+cmdline_parse_token_string_t cmd_tunnel_tso_set_tso =
+ TOKEN_STRING_INITIALIZER(struct cmd_tunnel_tso_set_result,
+ tso, "tunnel_tso");
+cmdline_parse_token_string_t cmd_tunnel_tso_set_mode =
+ TOKEN_STRING_INITIALIZER(struct cmd_tunnel_tso_set_result,
+ mode, "set");
+cmdline_parse_token_num_t cmd_tunnel_tso_set_tso_segsz =
+ TOKEN_NUM_INITIALIZER(struct cmd_tunnel_tso_set_result,
+ tso_segsz, UINT16);
+cmdline_parse_token_num_t cmd_tunnel_tso_set_portid =
+ TOKEN_NUM_INITIALIZER(struct cmd_tunnel_tso_set_result,
+ port_id, UINT8);
+
+cmdline_parse_inst_t cmd_tunnel_tso_set = {
+ .f = cmd_tunnel_tso_set_parsed,
+ .data = NULL,
+ .help_str = "Set TSO segment size of tunneled packets for csum engine "
+ "(0 to disable): tunnel_tso set <tso_segsz> <port>",
+ .tokens = {
+ (void *)&cmd_tunnel_tso_set_tso,
+ (void *)&cmd_tunnel_tso_set_mode,
+ (void *)&cmd_tunnel_tso_set_tso_segsz,
+ (void *)&cmd_tunnel_tso_set_portid,
+ NULL,
+ },
+};
+
+cmdline_parse_token_string_t cmd_tunnel_tso_show_mode =
+ TOKEN_STRING_INITIALIZER(struct cmd_tunnel_tso_set_result,
+ mode, "show");
+
+
+cmdline_parse_inst_t cmd_tunnel_tso_show = {
+ .f = cmd_tunnel_tso_set_parsed,
+ .data = NULL,
+ .help_str = "Show TSO segment size of tunneled packets "
+ "for csum engine: tunnel_tso show <port>",
+ .tokens = {
+ (void *)&cmd_tunnel_tso_set_tso,
+ (void *)&cmd_tunnel_tso_show_mode,
+ (void *)&cmd_tunnel_tso_set_portid,
+ NULL,
+ },
+};
+
/* *** ENABLE/DISABLE FLUSH ON RX STREAMS *** */
struct cmd_set_flush_rx {
cmdline_fixed_string_t set;
@@ -10646,6 +10762,8 @@ cmdline_parse_ctx_t main_ctx[] = {
(cmdline_parse_inst_t *)&cmd_csum_tunnel,
(cmdline_parse_inst_t *)&cmd_tso_set,
(cmdline_parse_inst_t *)&cmd_tso_show,
+ (cmdline_parse_inst_t *)&cmd_tunnel_tso_set,
+ (cmdline_parse_inst_t *)&cmd_tunnel_tso_show,
(cmdline_parse_inst_t *)&cmd_link_flow_control_set,
(cmdline_parse_inst_t *)&cmd_link_flow_control_set_rx,
(cmdline_parse_inst_t *)&cmd_link_flow_control_set_tx,
diff --git a/app/test-pmd/csumonly.c b/app/test-pmd/csumonly.c
index 21cb78f..4fe038d 100644
--- a/app/test-pmd/csumonly.c
+++ b/app/test-pmd/csumonly.c
@@ -101,6 +101,7 @@ struct testpmd_offload_info {
uint16_t outer_l3_len;
uint8_t outer_l4_proto;
uint16_t tso_segsz;
+ uint16_t tunnel_tso_segsz;
};
/* simplified GRE header */
@@ -349,7 +350,9 @@ process_inner_cksums(void *l3_hdr, const struct testpmd_offload_info *info,
ipv4_hdr->hdr_checksum = 0;
ol_flags |= PKT_TX_IPV4;
- if (info->tso_segsz != 0 && info->l4_proto == IPPROTO_TCP) {
+ if (info->l4_proto == IPPROTO_TCP &&
+ ((info->is_tunnel && info->tunnel_tso_segsz != 0) ||
+ (!info->is_tunnel && info->tso_segsz != 0))) {
ol_flags |= PKT_TX_IP_CKSUM;
} else {
if (testpmd_ol_flags & TESTPMD_TX_OFFLOAD_IP_CKSUM)
@@ -381,7 +384,8 @@ process_inner_cksums(void *l3_hdr, const struct testpmd_offload_info *info,
} else if (info->l4_proto == IPPROTO_TCP) {
tcp_hdr = (struct tcp_hdr *)((char *)l3_hdr + info->l3_len);
tcp_hdr->cksum = 0;
- if (info->tso_segsz != 0) {
+ if ((info->is_tunnel && info->tunnel_tso_segsz != 0) ||
+ (!info->is_tunnel && info->tso_segsz != 0)) {
ol_flags |= PKT_TX_TCP_SEG;
tcp_hdr->cksum = get_psd_sum(l3_hdr, info->ethertype,
ol_flags);
@@ -411,12 +415,10 @@ process_inner_cksums(void *l3_hdr, const struct testpmd_offload_info *info,
return ol_flags;
}
-/* Calculate the checksum of outer header (only vxlan is supported,
- * meaning IP + UDP). The caller already checked that it's a vxlan
- * packet */
+/* Calculate the checksum of outer header */
static uint64_t
process_outer_cksums(void *outer_l3_hdr, struct testpmd_offload_info *info,
- uint16_t testpmd_ol_flags)
+ uint16_t testpmd_ol_flags, int tso_enabled)
{
struct ipv4_hdr *ipv4_hdr = outer_l3_hdr;
struct ipv6_hdr *ipv6_hdr = outer_l3_hdr;
@@ -437,10 +439,20 @@ process_outer_cksums(void *outer_l3_hdr, struct testpmd_offload_info *info,
if (info->outer_l4_proto != IPPROTO_UDP)
return ol_flags;
- /* outer UDP checksum is always done in software as we have no
- * hardware supporting it today, and no API for it. */
-
udp_hdr = (struct udp_hdr *)((char *)outer_l3_hdr + info->outer_l3_len);
+
+ /* outer UDP checksum is done in software as we have no hardware
+ * supporting it today, and no API for it. In the other side, for
+ * UDP tunneling, like VXLAN or Geneve, outer UDP checksum can be
+ * set to zero.
+ *
+ * If a packet will be TSOed into small packets by NIC, we cannot
+ * set/calculate a non-zero checksum, because it will be a wrong
+ * value after the packet be split into several small packets.
+ */
+ if (tso_enabled)
+ udp_hdr->dgram_cksum = 0;
+
/* do not recalculate udp cksum if it was 0 */
if (udp_hdr->dgram_cksum != 0) {
udp_hdr->dgram_cksum = 0;
@@ -674,6 +686,7 @@ pkt_burst_checksum_forward(struct fwd_stream *fs)
testpmd_ol_flags = txp->tx_ol_flags;
memset(&info, 0, sizeof(info));
info.tso_segsz = txp->tso_segsz;
+ info.tunnel_tso_segsz = txp->tunnel_tso_segsz;
for (i = 0; i < nb_rx; i++) {
if (likely(i < nb_rx - 1))
@@ -703,18 +716,27 @@ pkt_burst_checksum_forward(struct fwd_stream *fs)
if (testpmd_ol_flags & TESTPMD_TX_OFFLOAD_PARSE_TUNNEL) {
if (info.l4_proto == IPPROTO_UDP) {
struct udp_hdr *udp_hdr;
+
udp_hdr = (struct udp_hdr *)((char *)l3_hdr +
info.l3_len);
parse_vxlan(udp_hdr, &info, m->packet_type);
+ if (info.is_tunnel)
+ ol_flags |= PKT_TX_TUNNEL_VXLAN;
} else if (info.l4_proto == IPPROTO_GRE) {
struct simple_gre_hdr *gre_hdr;
+
gre_hdr = (struct simple_gre_hdr *)
((char *)l3_hdr + info.l3_len);
parse_gre(gre_hdr, &info);
+ if (info.is_tunnel)
+ ol_flags |= PKT_TX_TUNNEL_GRE;
} else if (info.l4_proto == IPPROTO_IPIP) {
void *encap_ip_hdr;
+
encap_ip_hdr = (char *)l3_hdr + info.l3_len;
parse_encap_ip(encap_ip_hdr, &info);
+ if (info.is_tunnel)
+ ol_flags |= PKT_TX_TUNNEL_IPIP;
}
}
@@ -744,18 +766,21 @@ pkt_burst_checksum_forward(struct fwd_stream *fs)
* processed in hardware. */
if (info.is_tunnel == 1) {
ol_flags |= process_outer_cksums(outer_l3_hdr, &info,
- testpmd_ol_flags);
+ testpmd_ol_flags,
+ !!(ol_flags & PKT_TX_TCP_SEG));
}
/* step 4: fill the mbuf meta data (flags and header lengths) */
if (info.is_tunnel == 1) {
- if (testpmd_ol_flags & TESTPMD_TX_OFFLOAD_OUTER_IP_CKSUM) {
+ if (info.tunnel_tso_segsz ||
+ testpmd_ol_flags & TESTPMD_TX_OFFLOAD_OUTER_IP_CKSUM) {
m->outer_l2_len = info.outer_l2_len;
m->outer_l3_len = info.outer_l3_len;
m->l2_len = info.l2_len;
m->l3_len = info.l3_len;
m->l4_len = info.l4_len;
+ m->tso_segsz = info.tunnel_tso_segsz;
}
else {
/* if there is a outer UDP cksum
@@ -775,8 +800,8 @@ pkt_burst_checksum_forward(struct fwd_stream *fs)
m->l2_len = info.l2_len;
m->l3_len = info.l3_len;
m->l4_len = info.l4_len;
+ m->tso_segsz = info.tso_segsz;
}
- m->tso_segsz = info.tso_segsz;
m->ol_flags = ol_flags;
/* Do split & copy for the packet. */
@@ -805,6 +830,10 @@ pkt_burst_checksum_forward(struct fwd_stream *fs)
{ PKT_TX_OUTER_IPV4, PKT_TX_OUTER_IPV4 },
{ PKT_TX_OUTER_IPV6, PKT_TX_OUTER_IPV6 },
{ PKT_TX_TCP_SEG, PKT_TX_TCP_SEG },
+ { PKT_TX_TUNNEL_VXLAN, PKT_TX_TUNNEL_MASK },
+ { PKT_TX_TUNNEL_GRE, PKT_TX_TUNNEL_MASK },
+ { PKT_TX_TUNNEL_IPIP, PKT_TX_TUNNEL_MASK },
+ { PKT_TX_TUNNEL_GENEVE, PKT_TX_TUNNEL_MASK },
};
unsigned j;
const char *name;
@@ -831,11 +860,17 @@ pkt_burst_checksum_forward(struct fwd_stream *fs)
printf("tx: m->l2_len=%d m->l3_len=%d "
"m->l4_len=%d\n",
m->l2_len, m->l3_len, m->l4_len);
- if ((info.is_tunnel == 1) &&
- (testpmd_ol_flags & TESTPMD_TX_OFFLOAD_OUTER_IP_CKSUM))
- printf("tx: m->outer_l2_len=%d m->outer_l3_len=%d\n",
- m->outer_l2_len, m->outer_l3_len);
- if (info.tso_segsz != 0)
+ if (info.is_tunnel == 1) {
+ if (testpmd_ol_flags &
+ TESTPMD_TX_OFFLOAD_OUTER_IP_CKSUM)
+ printf("tx: m->outer_l2_len=%d "
+ "m->outer_l3_len=%d\n",
+ m->outer_l2_len,
+ m->outer_l3_len);
+ if (info.tunnel_tso_segsz != 0)
+ printf("tx: m->tso_segsz=%d\n",
+ m->tso_segsz);
+ } else if (info.tso_segsz != 0)
printf("tx: m->tso_segsz=%d\n", m->tso_segsz);
printf("tx: flags=");
for (j = 0; j < sizeof(tx_flags)/sizeof(*tx_flags); j++) {
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 2b281cc..881d283 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -152,7 +152,8 @@ struct rte_port {
struct fwd_stream *tx_stream; /**< Port TX stream, if unique */
unsigned int socket_id; /**< For NUMA support */
uint16_t tx_ol_flags;/**< TX Offload Flags (TESTPMD_TX_OFFLOAD...). */
- uint16_t tso_segsz; /**< MSS for segmentation offload. */
+ uint16_t tso_segsz; /**< Segmentation offload MSS for non-tunneled packets. */
+ uint16_t tunnel_tso_segsz; /**< Segmentation offload MSS for tunneled pkts. */
uint16_t tx_vlan_id;/**< The tag ID */
uint16_t tx_vlan_id_outer;/**< The outer tag ID */
void *fwd_ctx; /**< Forwarding mode context */
--
2.7.4
next prev parent reply other threads:[~2016-09-26 13:48 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-07-05 20:59 [dpdk-dev] [PATCH v1] i40: fix the VXLAN TSO issue Zhe Tao
2016-07-06 5:38 ` Wu, Jingjing
2016-07-07 4:27 ` [dpdk-dev] [PATCH v2] " Zhe Tao
2016-07-07 10:01 ` Ananyev, Konstantin
2016-07-07 10:50 ` Ananyev, Konstantin
2016-07-07 12:24 ` Ananyev, Konstantin
2016-07-15 15:40 ` Bruce Richardson
2016-07-18 2:57 ` Zhe Tao
2016-07-18 11:56 ` [dpdk-dev] [PATCH v3] " Zhe Tao
2016-07-19 10:29 ` Ananyev, Konstantin
2016-07-26 12:22 ` Tan, Jianfeng
2016-07-29 7:11 ` Tan, Jianfeng
2016-07-29 8:45 ` Ananyev, Konstantin
2016-07-29 10:11 ` Tan, Jianfeng
2016-10-10 3:58 ` [dpdk-dev] [PATCH v2] " Wu, Jingjing
2016-10-10 4:14 ` Yuanhan Liu
2016-08-01 3:56 ` [dpdk-dev] [PATCH v4 0/3] Add TSO on tunneling packet Jianfeng Tan
2016-08-01 3:56 ` [dpdk-dev] [PATCH v4 1/3] mbuf: add Tx side tunneling type Jianfeng Tan
2016-08-01 3:56 ` [dpdk-dev] [PATCH v4 2/3] net/i40e: add TSO support on tunneling packet Jianfeng Tan
2016-08-01 3:56 ` [dpdk-dev] [PATCH v4 3/3] app/testpmd: fix Tx offload " Jianfeng Tan
2016-09-19 12:09 ` Ananyev, Konstantin
2016-09-21 12:36 ` Tan, Jianfeng
2016-09-21 15:47 ` Ananyev, Konstantin
2016-09-22 1:29 ` Tan, Jianfeng
2016-09-22 9:15 ` Ananyev, Konstantin
[not found] ` <ED26CBA2FAD1BF48A8719AEF02201E364E5E09BC@SHSMSX103.ccr.corp.intel.com>
[not found] ` <2601191342CEEE43887BDE71AB97725836BA2698@irsmsx105.ger.corp.intel.com>
2016-09-27 17:29 ` [dpdk-dev] [PATCH v4 0/3] Add TSO " Ananyev, Konstantin
2016-09-27 17:52 ` Tan, Jianfeng
2016-09-27 19:47 ` Thomas Monjalon
2016-10-09 21:27 ` Thomas Monjalon
2016-09-26 13:48 ` Jianfeng Tan [this message]
2016-09-27 17:25 ` [dpdk-dev] [PATCH v5 3/3] app/testpmd: support tunneled TSO in csum fwd engine Ananyev, Konstantin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1474897714-28037-1-git-send-email-jianfeng.tan@intel.com \
--to=jianfeng.tan@intel.com \
--cc=dev@dpdk.org \
--cc=jingjing.wu@intel.com \
--cc=konstantin.ananyev@intel.com \
--cc=zhe.tao@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).