* Re: [dpdk-dev] [PATCH] net/ixgbe: fixed port can not link up in FreeBSD
2019-12-12 11:17 [dpdk-dev] [PATCH] net/ixgbe: fixed port can not link up in FreeBSD Lunyuan Cui
@ 2019-12-12 11:15 ` Ananyev, Konstantin
2019-12-13 2:36 ` [dpdk-dev] [PATCH v2] " Lunyuan Cui
2019-12-16 2:24 ` [dpdk-dev] [PATCH v3] net/ixgbe: fix " Lunyuan Cui
2 siblings, 0 replies; 7+ messages in thread
From: Ananyev, Konstantin @ 2019-12-12 11:15 UTC (permalink / raw)
To: Cui, LunyuanX, dev; +Cc: Lu, Wenzhuo, Yang, Qiming, Cui, LunyuanX, stable
Hi,
>
> In FreeBSD environment, nic_uio drivers do not support interrupts,
> rte_intr_callback_register() will fail to register interrupts.
> We can not make link status to change from down to up by interrupt
> callback. So we need to wait for the controller to acquire link
> when ports start. Through multiple tests, 5s should be enough.
>
> Fixes: b9bd0f09fa15 ("ethdev: fix link status query")
> Cc: stable@dpdk.org
>
> Signed-off-by: Lunyuan Cui <lunyuanx.cui@intel.com>
> ---
> drivers/net/ixgbe/ixgbe_ethdev.c | 22 ++++++++++++++++++++++
> 1 file changed, 22 insertions(+)
>
> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
> index 2c6fd0f13..e33b5483c 100644
> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> @@ -2555,6 +2555,9 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
> IXGBE_DEV_PRIVATE_TO_TM_CONF(dev->data->dev_private);
> struct ixgbe_macsec_setting *macsec_setting =
> IXGBE_DEV_PRIVATE_TO_MACSEC_SETTING(dev->data->dev_private);
> +#ifdef RTE_EXEC_ENV_FREEBSD
> + int i;
> +#endif
>
> PMD_INIT_FUNC_TRACE();
>
> @@ -2801,6 +2804,25 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
> "please call hierarchy_commit() "
> "before starting the port");
>
> +#ifdef RTE_EXEC_ENV_FREEBSD
> + /*
> + * In freebsd environment, nic_uio drivers do not support interrupts,
> + * rte_intr_callback_register() will fail to register interrupts.
> + * We can not make link status to change from down to up by interrupt
> + * callback. So we need to wait for the controller to acquire link
> + * when ports start.
> + */
> + for (i = 0; i < 25; i++) {
> + /* If link up, just jump out */
> + err = ixgbe_check_link(hw, &speed, &link_up, 0);
> + if (err)
> + goto error;
> + if (link_up)
> + break;
> + msec_delay(200);
> + }
> +#endif
Might be better to put it into a separate function to keep start() code clean.
Konstantin
> +
> /*
> * Update link status right before return, because it may
> * start link configuration process in a separate thread.
> --
> 2.17.1
^ permalink raw reply [flat|nested] 7+ messages in thread* [dpdk-dev] [PATCH v2] net/ixgbe: fixed port can not link up in FreeBSD
2019-12-12 11:17 [dpdk-dev] [PATCH] net/ixgbe: fixed port can not link up in FreeBSD Lunyuan Cui
2019-12-12 11:15 ` Ananyev, Konstantin
@ 2019-12-13 2:36 ` Lunyuan Cui
2019-12-13 14:32 ` Ananyev, Konstantin
2019-12-16 2:24 ` [dpdk-dev] [PATCH v3] net/ixgbe: fix " Lunyuan Cui
2 siblings, 1 reply; 7+ messages in thread
From: Lunyuan Cui @ 2019-12-13 2:36 UTC (permalink / raw)
To: dev; +Cc: Wenzhuo Lu, Qiming Yang, Ananyev, Konstantin, Lunyuan Cui, stable
In FreeBSD environment, nic_uio drivers do not support interrupts,
rte_intr_callback_register() will fail to register interrupts.
We can not make link status to change from down to up by interrupt
callback. So we need to wait for the controller to acquire link
when ports start. Through multiple tests, 5s should be enough.
Fixes: b9bd0f09fa15 ("ethdev: fix link status query")
Cc: stable@dpdk.org
Signed-off-by: Lunyuan Cui <lunyuanx.cui@intel.com>
---
v2 changes:
* Put waiting into a separate function to keep start() code clean.
---
drivers/net/ixgbe/ixgbe_ethdev.c | 32 ++++++++++++++++++++++++++++++++
drivers/net/ixgbe/ixgbe_ethdev.h | 3 +++
2 files changed, 35 insertions(+)
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 2c6fd0f13..fba666186 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -2801,6 +2801,13 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
"please call hierarchy_commit() "
"before starting the port");
+#ifdef RTE_EXEC_ENV_FREEBSD
+ /* wait for the controller to acquire link */
+ err = ixgbe_wait_for_link_up(hw);
+ if (err)
+ goto error;
+#endif
+
/*
* Update link status right before return, because it may
* start link configuration process in a separate thread.
@@ -4114,6 +4121,31 @@ ixgbe_dev_setup_link_alarm_handler(void *param)
intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
}
+#ifdef RTE_EXEC_ENV_FREEBSD
+/*
+ * In freebsd environment, nic_uio drivers do not support interrupts,
+ * rte_intr_callback_register() will fail to register interrupts.
+ * We can not make link status to change from down to up by interrupt
+ * callback. So we need to wait for the controller to acquire link
+ * when ports start.
+ * It returns 0 on link up.
+ */
+int32_t ixgbe_wait_for_link_up(struct ixgbe_hw *hw)
+{
+ int err, i, link_up = 0;
+ uint32_t speed = 0;
+ for (i = 0; i < 25; i++) {
+ err = ixgbe_check_link(hw, &speed, &link_up, 0);
+ if (err)
+ return err;
+ if (link_up)
+ return 0;
+ msec_delay(200);
+ }
+ return 0;
+}
+#endif
+
/* return 0 means link status changed, -1 means not changed */
int
ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
index 76a1b9d18..9a1d8c54c 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.h
+++ b/drivers/net/ixgbe/ixgbe_ethdev.h
@@ -700,6 +700,9 @@ int
ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
int wait_to_complete, int vf);
+#ifdef RTE_EXEC_ENV_FREEBSD
+int32_t ixgbe_wait_for_link_up(struct ixgbe_hw *hw);
+#endif
/*
* misc function prototypes
*/
--
2.17.1
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [dpdk-dev] [PATCH v2] net/ixgbe: fixed port can not link up in FreeBSD
2019-12-13 2:36 ` [dpdk-dev] [PATCH v2] " Lunyuan Cui
@ 2019-12-13 14:32 ` Ananyev, Konstantin
0 siblings, 0 replies; 7+ messages in thread
From: Ananyev, Konstantin @ 2019-12-13 14:32 UTC (permalink / raw)
To: Cui, LunyuanX, dev; +Cc: Lu, Wenzhuo, Yang, Qiming, Ananyev, stable
Few small nits below.
s/fixed/fix/
> In FreeBSD environment, nic_uio drivers do not support interrupts,
> rte_intr_callback_register() will fail to register interrupts.
> We can not make link status to change from down to up by interrupt
> callback. So we need to wait for the controller to acquire link
> when ports start. Through multiple tests, 5s should be enough.
>
> Fixes: b9bd0f09fa15 ("ethdev: fix link status query")
> Cc: stable@dpdk.org
>
> Signed-off-by: Lunyuan Cui <lunyuanx.cui@intel.com>
> ---
>
> v2 changes:
> * Put waiting into a separate function to keep start() code clean.
> ---
> drivers/net/ixgbe/ixgbe_ethdev.c | 32 ++++++++++++++++++++++++++++++++
> drivers/net/ixgbe/ixgbe_ethdev.h | 3 +++
> 2 files changed, 35 insertions(+)
>
> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
> index 2c6fd0f13..fba666186 100644
> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> @@ -2801,6 +2801,13 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
> "please call hierarchy_commit() "
> "before starting the port");
>
> +#ifdef RTE_EXEC_ENV_FREEBSD
> + /* wait for the controller to acquire link */
> + err = ixgbe_wait_for_link_up(hw);
> + if (err)
> + goto error;
> +#endif
You can hide ifdef inside the function, see below.
> +
> /*
> * Update link status right before return, because it may
> * start link configuration process in a separate thread.
> @@ -4114,6 +4121,31 @@ ixgbe_dev_setup_link_alarm_handler(void *param)
> intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
> }
>
> +#ifdef RTE_EXEC_ENV_FREEBSD
> +/*
> + * In freebsd environment, nic_uio drivers do not support interrupts,
> + * rte_intr_callback_register() will fail to register interrupts.
> + * We can not make link status to change from down to up by interrupt
> + * callback. So we need to wait for the controller to acquire link
> + * when ports start.
> + * It returns 0 on link up.
> + */
> +int32_t ixgbe_wait_for_link_up(struct ixgbe_hw *hw)
This function can be static, plus pls try to follow dpdk codying style:
static int
ixgbe_wait_for_link_up(struct ixgbe_hw *hw)
{
....
You can put #ifdef inside the function:
#ifdef RTE_EXEC_ENV_FREEBSD
<do something meaningfull here>
#else
RTE_SET_USED(hw);
#endif
Or even:
#ifdef RTE_EXEC_ENV_FREEBSD
const uint32_t nb_iter = 25;
#else
const uint32_t nb_iter = 0;
#endif
...
for (i = 0; i < nb_iter; i++) {
...
> +{
> + int err, i, link_up = 0;
> + uint32_t speed = 0;
Empty line here would help readability.
> + for (i = 0; i < 25; i++) {
> + err = ixgbe_check_link(hw, &speed, &link_up, 0);
> + if (err)
> + return err;
> + if (link_up)
> + return 0;
> + msec_delay(200);
> + }
> + return 0;
> +}
> +#endif
> +
> /* return 0 means link status changed, -1 means not changed */
> int
> ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
> index 76a1b9d18..9a1d8c54c 100644
> --- a/drivers/net/ixgbe/ixgbe_ethdev.h
> +++ b/drivers/net/ixgbe/ixgbe_ethdev.h
> @@ -700,6 +700,9 @@ int
> ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
> int wait_to_complete, int vf);
>
> +#ifdef RTE_EXEC_ENV_FREEBSD
> +int32_t ixgbe_wait_for_link_up(struct ixgbe_hw *hw);
> +#endif
Pls see above, that function can be static.
> /*
> * misc function prototypes
> */
> --
> 2.17.1
^ permalink raw reply [flat|nested] 7+ messages in thread
* [dpdk-dev] [PATCH v3] net/ixgbe: fix port can not link up in FreeBSD
2019-12-12 11:17 [dpdk-dev] [PATCH] net/ixgbe: fixed port can not link up in FreeBSD Lunyuan Cui
2019-12-12 11:15 ` Ananyev, Konstantin
2019-12-13 2:36 ` [dpdk-dev] [PATCH v2] " Lunyuan Cui
@ 2019-12-16 2:24 ` Lunyuan Cui
2019-12-18 2:47 ` Ye Xiaolong
2019-12-18 10:21 ` Ananyev, Konstantin
2 siblings, 2 replies; 7+ messages in thread
From: Lunyuan Cui @ 2019-12-16 2:24 UTC (permalink / raw)
To: dev; +Cc: Wenzhuo Lu, Qiming Yang, Ananyev, Konstantin, Lunyuan Cui, stable
In FreeBSD environment, nic_uio drivers do not support interrupts,
rte_intr_callback_register() will fail to register interrupts.
We can not make link status to change from down to up by interrupt
callback. So we need to wait for the controller to acquire link
when ports start. Through multiple tests, 5s should be enough.
Fixes: b9bd0f09fa15 ("ethdev: fix link status query")
Cc: stable@dpdk.org
Signed-off-by: Lunyuan Cui <lunyuanx.cui@intel.com>
---
v3 changes:
* Hide ifdef inside the function
v2 changes:
* Put waiting into a separate function to keep start() code clean.
---
drivers/net/ixgbe/ixgbe_ethdev.c | 36 ++++++++++++++++++++++++++++++++
1 file changed, 36 insertions(+)
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 2c6fd0f13..d9b0c5b02 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -378,6 +378,7 @@ static int ixgbe_dev_udp_tunnel_port_del(struct rte_eth_dev *dev,
struct rte_eth_udp_tunnel *udp_tunnel);
static int ixgbe_filter_restore(struct rte_eth_dev *dev);
static void ixgbe_l2_tunnel_conf(struct rte_eth_dev *dev);
+static int ixgbe_wait_for_link_up(struct ixgbe_hw *hw);
/*
* Define VF Stats MACRO for Non "cleared on read" register
@@ -2801,6 +2802,11 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
"please call hierarchy_commit() "
"before starting the port");
+ /* wait for the controller to acquire link */
+ err = ixgbe_wait_for_link_up(hw);
+ if (err)
+ goto error;
+
/*
* Update link status right before return, because it may
* start link configuration process in a separate thread.
@@ -4114,6 +4120,36 @@ ixgbe_dev_setup_link_alarm_handler(void *param)
intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
}
+/*
+ * In freebsd environment, nic_uio drivers do not support interrupts,
+ * rte_intr_callback_register() will fail to register interrupts.
+ * We can not make link status to change from down to up by interrupt
+ * callback. So we need to wait for the controller to acquire link
+ * when ports start.
+ * It returns 0 on link up.
+ */
+static int
+ixgbe_wait_for_link_up(struct ixgbe_hw *hw)
+{
+#ifdef RTE_EXEC_ENV_FREEBSD
+ const int nb_iter = 25;
+#else
+ const int nb_iter = 0;
+#endif
+ int err, i, link_up = 0;
+ uint32_t speed = 0;
+
+ for (i = 0; i < nb_iter; i++) {
+ err = ixgbe_check_link(hw, &speed, &link_up, 0);
+ if (err)
+ return err;
+ if (link_up)
+ return 0;
+ msec_delay(200);
+ }
+ return 0;
+}
+
/* return 0 means link status changed, -1 means not changed */
int
ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
--
2.17.1
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [dpdk-dev] [PATCH v3] net/ixgbe: fix port can not link up in FreeBSD
2019-12-16 2:24 ` [dpdk-dev] [PATCH v3] net/ixgbe: fix " Lunyuan Cui
@ 2019-12-18 2:47 ` Ye Xiaolong
2019-12-18 10:21 ` Ananyev, Konstantin
1 sibling, 0 replies; 7+ messages in thread
From: Ye Xiaolong @ 2019-12-18 2:47 UTC (permalink / raw)
To: Lunyuan Cui; +Cc: dev, Wenzhuo Lu, Qiming Yang, Ananyev, Konstantin, stable
On 12/16, Lunyuan Cui wrote:
>In FreeBSD environment, nic_uio drivers do not support interrupts,
>rte_intr_callback_register() will fail to register interrupts.
>We can not make link status to change from down to up by interrupt
>callback. So we need to wait for the controller to acquire link
>when ports start. Through multiple tests, 5s should be enough.
>
>Fixes: b9bd0f09fa15 ("ethdev: fix link status query")
>Cc: stable@dpdk.org
>
>Signed-off-by: Lunyuan Cui <lunyuanx.cui@intel.com>
>---
>
>v3 changes:
>* Hide ifdef inside the function
>
>v2 changes:
>* Put waiting into a separate function to keep start() code clean.
>---
> drivers/net/ixgbe/ixgbe_ethdev.c | 36 ++++++++++++++++++++++++++++++++
> 1 file changed, 36 insertions(+)
>
>diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
>index 2c6fd0f13..d9b0c5b02 100644
>--- a/drivers/net/ixgbe/ixgbe_ethdev.c
>+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
>@@ -378,6 +378,7 @@ static int ixgbe_dev_udp_tunnel_port_del(struct rte_eth_dev *dev,
> struct rte_eth_udp_tunnel *udp_tunnel);
> static int ixgbe_filter_restore(struct rte_eth_dev *dev);
> static void ixgbe_l2_tunnel_conf(struct rte_eth_dev *dev);
>+static int ixgbe_wait_for_link_up(struct ixgbe_hw *hw);
>
> /*
> * Define VF Stats MACRO for Non "cleared on read" register
>@@ -2801,6 +2802,11 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
> "please call hierarchy_commit() "
> "before starting the port");
>
>+ /* wait for the controller to acquire link */
>+ err = ixgbe_wait_for_link_up(hw);
>+ if (err)
>+ goto error;
>+
> /*
> * Update link status right before return, because it may
> * start link configuration process in a separate thread.
>@@ -4114,6 +4120,36 @@ ixgbe_dev_setup_link_alarm_handler(void *param)
> intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
> }
>
>+/*
>+ * In freebsd environment, nic_uio drivers do not support interrupts,
>+ * rte_intr_callback_register() will fail to register interrupts.
>+ * We can not make link status to change from down to up by interrupt
>+ * callback. So we need to wait for the controller to acquire link
>+ * when ports start.
>+ * It returns 0 on link up.
>+ */
>+static int
>+ixgbe_wait_for_link_up(struct ixgbe_hw *hw)
>+{
>+#ifdef RTE_EXEC_ENV_FREEBSD
>+ const int nb_iter = 25;
>+#else
>+ const int nb_iter = 0;
>+#endif
>+ int err, i, link_up = 0;
>+ uint32_t speed = 0;
>+
>+ for (i = 0; i < nb_iter; i++) {
>+ err = ixgbe_check_link(hw, &speed, &link_up, 0);
>+ if (err)
>+ return err;
>+ if (link_up)
>+ return 0;
>+ msec_delay(200);
>+ }
>+ return 0;
>+}
>+
> /* return 0 means link status changed, -1 means not changed */
> int
> ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
>--
>2.17.1
>
Acked-by: Xiaolong Ye <xiaolong.ye@intel.com>
Applied to dpdk-next-net-intel, Thanks.
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [dpdk-dev] [PATCH v3] net/ixgbe: fix port can not link up in FreeBSD
2019-12-16 2:24 ` [dpdk-dev] [PATCH v3] net/ixgbe: fix " Lunyuan Cui
2019-12-18 2:47 ` Ye Xiaolong
@ 2019-12-18 10:21 ` Ananyev, Konstantin
1 sibling, 0 replies; 7+ messages in thread
From: Ananyev, Konstantin @ 2019-12-18 10:21 UTC (permalink / raw)
To: Cui, LunyuanX, dev; +Cc: Lu, Wenzhuo, Yang, Qiming, Ananyev, stable
> In FreeBSD environment, nic_uio drivers do not support interrupts,
> rte_intr_callback_register() will fail to register interrupts.
> We can not make link status to change from down to up by interrupt
> callback. So we need to wait for the controller to acquire link
> when ports start. Through multiple tests, 5s should be enough.
>
> Fixes: b9bd0f09fa15 ("ethdev: fix link status query")
> Cc: stable@dpdk.org
>
> Signed-off-by: Lunyuan Cui <lunyuanx.cui@intel.com>
> ---
>
> v3 changes:
> * Hide ifdef inside the function
>
> v2 changes:
> * Put waiting into a separate function to keep start() code clean.
> ---
> drivers/net/ixgbe/ixgbe_ethdev.c | 36 ++++++++++++++++++++++++++++++++
> 1 file changed, 36 insertions(+)
>
> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
> index 2c6fd0f13..d9b0c5b02 100644
> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> @@ -378,6 +378,7 @@ static int ixgbe_dev_udp_tunnel_port_del(struct rte_eth_dev *dev,
> struct rte_eth_udp_tunnel *udp_tunnel);
> static int ixgbe_filter_restore(struct rte_eth_dev *dev);
> static void ixgbe_l2_tunnel_conf(struct rte_eth_dev *dev);
> +static int ixgbe_wait_for_link_up(struct ixgbe_hw *hw);
>
> /*
> * Define VF Stats MACRO for Non "cleared on read" register
> @@ -2801,6 +2802,11 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
> "please call hierarchy_commit() "
> "before starting the port");
>
> + /* wait for the controller to acquire link */
> + err = ixgbe_wait_for_link_up(hw);
> + if (err)
> + goto error;
> +
> /*
> * Update link status right before return, because it may
> * start link configuration process in a separate thread.
> @@ -4114,6 +4120,36 @@ ixgbe_dev_setup_link_alarm_handler(void *param)
> intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
> }
>
> +/*
> + * In freebsd environment, nic_uio drivers do not support interrupts,
> + * rte_intr_callback_register() will fail to register interrupts.
> + * We can not make link status to change from down to up by interrupt
> + * callback. So we need to wait for the controller to acquire link
> + * when ports start.
> + * It returns 0 on link up.
> + */
> +static int
> +ixgbe_wait_for_link_up(struct ixgbe_hw *hw)
> +{
> +#ifdef RTE_EXEC_ENV_FREEBSD
> + const int nb_iter = 25;
> +#else
> + const int nb_iter = 0;
> +#endif
> + int err, i, link_up = 0;
> + uint32_t speed = 0;
> +
> + for (i = 0; i < nb_iter; i++) {
> + err = ixgbe_check_link(hw, &speed, &link_up, 0);
> + if (err)
> + return err;
> + if (link_up)
> + return 0;
> + msec_delay(200);
> + }
> + return 0;
> +}
> +
> /* return 0 means link status changed, -1 means not changed */
> int
> ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
> --
Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> 2.17.1
^ permalink raw reply [flat|nested] 7+ messages in thread