[PATCH] lib/telemetry:fix telemetry conns leak in case of socket write fail

[PATCH] lib/telemetry:fix telemetry conns leak in case of socket write fail

[sunshaowei01]

Telemetry can only create 10 conns by default, each of which is processed
by a thread.

When a thread fails to write using socket, the thread will end directly
without reducing the total number of conns.

This will result in the machine running for a long time, and if there are
10 failures, the telemetry will be unavailable

Signed-off-by: sunshaowei01 <1819846787@qq.com>
---
 lib/telemetry/telemetry.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/telemetry/telemetry.c b/lib/telemetry/telemetry.c
index 31e2391867..0b00c04090 100644
--- a/lib/telemetry/telemetry.c
+++ b/lib/telemetry/telemetry.c
@@ -378,8 +378,8 @@ client_handler(void *sock_id)
 			"{\"version\":\"%s\",\"pid\":%d,\"max_output_len\":%d}",
 			telemetry_version, getpid(), MAX_OUTPUT_LEN);
 	if (write(s, info_str, strlen(info_str)) < 0) {
-		close(s);
-		return NULL;
+		TMTY_LOG_LINE(ERR, "Socket write base info to client failed");
+		goto exit;
 	}
 
 	/* receive data is not null terminated */
@@ -404,6 +404,7 @@ client_handler(void *sock_id)
 
 		bytes = read(s, buffer, sizeof(buffer) - 1);
 	}
+exit:
 	close(s);
 	rte_atomic_fetch_sub_explicit(&v2_clients, 1, rte_memory_order_relaxed);
 	return NULL;
-- 
2.37.1 (Apple Git-137.1)

Re: [PATCH] lib/telemetry:fix telemetry conns leak in case of socket write fail

[Bruce Richardson]

On Fri, Jan 19, 2024 at 07:40:00PM +0800, sunshaowei01 wrote:
> Telemetry can only create 10 conns by default, each of which is processed
> by a thread.
> 
> When a thread fails to write using socket, the thread will end directly
> without reducing the total number of conns.
> 
> This will result in the machine running for a long time, and if there are
> 10 failures, the telemetry will be unavailable
> 
> Signed-off-by: sunshaowei01 <1819846787@qq.com>
> ---

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

RE: [PATCH] lib/telemetry:fix telemetry conns leak in case of socket write fail

[Power, Ciara]

> -----Original Message-----
> From: sunshaowei01 <1819846787@qq.com>
> Sent: Friday, January 19, 2024 11:40 AM
> To: dev@dpdk.org
> Cc: Power, Ciara <ciara.power@intel.com>
> Subject: [PATCH] lib/telemetry:fix telemetry conns leak in case of socket write
> fail
> 
> Telemetry can only create 10 conns by default, each of which is processed by a
> thread.
> 
> When a thread fails to write using socket, the thread will end directly without
> reducing the total number of conns.
> 
> This will result in the machine running for a long time, and if there are
> 10 failures, the telemetry will be unavailable
> 
> Signed-off-by: sunshaowei01 <1819846787@qq.com>
> ---
>  lib/telemetry/telemetry.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/telemetry/telemetry.c b/lib/telemetry/telemetry.c index
> 31e2391867..0b00c04090 100644
> --- a/lib/telemetry/telemetry.c
> +++ b/lib/telemetry/telemetry.c
> @@ -378,8 +378,8 @@ client_handler(void *sock_id)
> 
> 	"{\"version\":\"%s\",\"pid\":%d,\"max_output_len\":%d}",
>  			telemetry_version, getpid(), MAX_OUTPUT_LEN);
>  	if (write(s, info_str, strlen(info_str)) < 0) {
> -		close(s);
> -		return NULL;
> +		TMTY_LOG_LINE(ERR, "Socket write base info to client failed");
> +		goto exit;
>  	}
> 
>  	/* receive data is not null terminated */ @@ -404,6 +404,7 @@
> client_handler(void *sock_id)
> 
>  		bytes = read(s, buffer, sizeof(buffer) - 1);
>  	}
> +exit:
>  	close(s);
>  	rte_atomic_fetch_sub_explicit(&v2_clients, 1,
> rte_memory_order_relaxed);
>  	return NULL;
> --
> 2.37.1 (Apple Git-137.1)

Thanks for fixing this.

Acked-by: Ciara Power <ciara.power@intel.com>

Re: [PATCH] lib/telemetry:fix telemetry conns leak in case of socket write fail

[David Marchand]

Hello,

On Fri, Jan 19, 2024 at 12:40 PM sunshaowei01 <1819846787@qq.com> wrote:
>
> Telemetry can only create 10 conns by default, each of which is processed
> by a thread.
>
> When a thread fails to write using socket, the thread will end directly
> without reducing the total number of conns.
>
> This will result in the machine running for a long time, and if there are
> 10 failures, the telemetry will be unavailable
>

Thanks for the patch, do you know which commit first triggered the issue?
This is needed so we add a Fixes: tag in the commitlog for backporting
the fix in stable releases.
See https://doc.dpdk.org/guides/contributing/patches.html#patch-fix-related-issues

> Signed-off-by: sunshaowei01 <1819846787@qq.com>

We need your full name in the SoB tag.
Like, for example in my case, it would be David Marchand
<david.marchand@redhat.com>.


-- 
David Marchand

=?gb18030?B?u9i4tKO6IFtQQVRDSF0gbGliL3RlbGVtZXRyeTpmaXggdGVsZW1ldHJ5IGNvbm5zIGxlYWsgaW4gY2FzZSBvZiBzb2NrZXQgd3JpdGUgZmFpbA==?=

[=?gb18030?B?MTgxOTg0Njc4Nw==?=]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="gb18030", Size: 1530 bytes --]

I have modified my  commitlog and resubmitted the patch, but I seem to have forgotten to add a [v2] flag to the patch. Do I need to resubmit the patch again?




------------------ Ô­Ê¼Óʼþ ------------------
·¢¼þÈË: "David Marchand"<david.marchand@redhat.com&gt;; 
·¢ËÍʱ¼ä: 2024Äê1ÔÂ19ÈÕ(ÐÇÆÚÎå) ÍíÉÏ11:54
ÊÕ¼þÈË: " "<1819846787@qq.com&gt;; 
³­ËÍ: "dev"<dev@dpdk.org&gt;; "ciara.power"<ciara.power@intel.com&gt;; "Bruce Richardson"<bruce.richardson@intel.com&gt;; 
Ö÷Ìâ: Re: [PATCH] lib/telemetry:fix telemetry conns leak in case of socket write fail





On Fri, Jan 19, 2024 at 12:406§2PM sunshaowei01 <1819846787@qq.com&gt; wrote:
&gt;
&gt; Telemetry can only create 10 conns by default, each of which is processed
&gt; by a thread.
&gt;
&gt; When a thread fails to write using socket, the thread will end directly
&gt; without reducing the total number of conns.
&gt;
&gt; This will result in the machine running for a long time, and if there are
&gt; 10 failures, the telemetry will be unavailable
&gt;

Thanks for the patch, do you know which commit first triggered the issue?
This is needed so we add a Fixes: tag in the commitlog for backporting
the fix in stable releases.
See https://doc.dpdk.org/guides/contributing/patches.html#patch-fix-related-issues

&gt; Signed-off-by: sunshaowei01 <1819846787@qq.com&gt;

We need your full name in the SoB tag.
Like, for example in my case, it would be David Marchand
<david.marchand@redhat.com&gt;.


-- 
David Marchand

[-- Attachment #2: Type: text/html, Size: 2515 bytes --]

[PATCH] [v2]lib/telemetry:fix telemetry conns leak in case of socket write fail

[Shaowei Sun]

Telemetry can only create 10 conns by default, each of which is processed
by a thread.

When a thread fails to write using socket, the thread will end directly
without reducing the total number of conns.

This will result in the machine running for a long time, and if there are
10 failures, the telemetry will be unavailable

Fixes: 6dd571fd07c3 ("telemetry: introduce new functionality")

Signed-off-by: Shaowei Sun <1819846787@qq.com>
---
 lib/telemetry/telemetry.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/telemetry/telemetry.c b/lib/telemetry/telemetry.c
index 31e2391867..0b00c04090 100644
--- a/lib/telemetry/telemetry.c
+++ b/lib/telemetry/telemetry.c
@@ -378,8 +378,8 @@ client_handler(void *sock_id)
 			"{\"version\":\"%s\",\"pid\":%d,\"max_output_len\":%d}",
 			telemetry_version, getpid(), MAX_OUTPUT_LEN);
 	if (write(s, info_str, strlen(info_str)) < 0) {
-		close(s);
-		return NULL;
+		TMTY_LOG_LINE(ERR, "Socket write base info to client failed");
+		goto exit;
 	}
 
 	/* receive data is not null terminated */
@@ -404,6 +404,7 @@ client_handler(void *sock_id)
 
 		bytes = read(s, buffer, sizeof(buffer) - 1);
 	}
+exit:
 	close(s);
 	rte_atomic_fetch_sub_explicit(&v2_clients, 1, rte_memory_order_relaxed);
 	return NULL;
-- 
2.37.1 (Apple Git-137.1)

Re: [PATCH] [v2]lib/telemetry:fix telemetry conns leak in case of socket write fail

[fengchengwen]

Acked-by: Chengwen Feng <fengchengwen@huawei.com>

On 2024/1/20 16:58, Shaowei Sun wrote:
> Telemetry can only create 10 conns by default, each of which is processed
> by a thread.
> 
> When a thread fails to write using socket, the thread will end directly
> without reducing the total number of conns.
> 
> This will result in the machine running for a long time, and if there are
> 10 failures, the telemetry will be unavailable
> 
> Fixes: 6dd571fd07c3 ("telemetry: introduce new functionality")
> 
> Signed-off-by: Shaowei Sun <1819846787@qq.com>
> ---
>  lib/telemetry/telemetry.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/telemetry/telemetry.c b/lib/telemetry/telemetry.c
> index 31e2391867..0b00c04090 100644
> --- a/lib/telemetry/telemetry.c
> +++ b/lib/telemetry/telemetry.c
> @@ -378,8 +378,8 @@ client_handler(void *sock_id)
>  			"{\"version\":\"%s\",\"pid\":%d,\"max_output_len\":%d}",
>  			telemetry_version, getpid(), MAX_OUTPUT_LEN);
>  	if (write(s, info_str, strlen(info_str)) < 0) {
> -		close(s);
> -		return NULL;
> +		TMTY_LOG_LINE(ERR, "Socket write base info to client failed");
> +		goto exit;
>  	}
>  
>  	/* receive data is not null terminated */
> @@ -404,6 +404,7 @@ client_handler(void *sock_id)
>  
>  		bytes = read(s, buffer, sizeof(buffer) - 1);
>  	}
> +exit:
>  	close(s);
>  	rte_atomic_fetch_sub_explicit(&v2_clients, 1, rte_memory_order_relaxed);
>  	return NULL;
> 

Re: 回复： [PATCH] lib/telemetry:fix telemetry conns leak in case of socket write fail

[Bruce Richardson]

On Sat, Jan 20, 2024 at 12:18:38PM +0800, 1819846787 wrote:
>    I have modified my  commitlog and resubmitted the patch, but I seem to
>    have forgotten to add a [v2] flag to the patch. Do I need to resubmit
>    the patch again?
> 

It's better if the v2 is added, but it's probably not necessary in this
case. However, I see now you have resubmitted as a v2 anyway, which is
fine. Please mark older versions of the patch as "superceded" in patchwork
(patches.dpdk.org site - you'll need an account created with the email
address used to submit your patches. That will allow you to update the
status of your own patches yourself)

Also, one other tip, when submitting a new version of a previously reviewed
patch, unless there are major changes, you can keep any previously added
acks from reviewers. For your v2, for example, after your signed-off-by you
could have also added the "Acked-by: Ciara Power ... " line.

Hope these tips help in the future! Thanks for the contribution.

/Bruce

>    ------------------ ԭʼʼ� ------------------
> 
>    ����: "David Marchand"<david.marchand@redhat.com>;
>    ��ʱ�: 2024119() 11:54
>    ռ�: " "<1819846787@qq.com>;
>    ��: "dev"<dev@dpdk.org>; "ciara.power"<ciara.power@intel.com>; "Bruce
>    Richardson"<bruce.richardson@intel.com>;
>    : Re: [PATCH] lib/telemetry:fix telemetry conns leak in case of socket
>    write fail
> 
>    Hello,
>    On Fri, Jan 19, 2024 at 12:40�6�2PM sunshaowei01 <[1]1819846787@qq.com>
>    wrote:
>    >
>    > Telemetry can only create 10 conns by default, each of which is
>    processed
>    > by a thread.
>    >
>    > When a thread fails to write using socket, the thread will end
>    directly
>    > without reducing the total number of conns.
>    >
>    > This will result in the machine running for a long time, and if there
>    are
>    > 10 failures, the telemetry will be unavailable
>    >
>    Thanks for the patch, do you know which commit first triggered the
>    issue?
>    This is needed so we add a Fixes: tag in the commitlog for backporting
>    the fix in stable releases.
>    See
>    [2]https://doc.dpdk.org/guides/contributing/patches.html#patch-fix-rela
>    ted-issues
>    > Signed-off-by: sunshaowei01 <[3]1819846787@qq.com>
>    We need your full name in the SoB tag.
>    Like, for example in my case, it would be David Marchand
>    <[4]david.marchand@redhat.com>.
>    --
>    David Marchand
> 
> References
> 
>    1. mailto:1819846787@qq.com
>    2. https://doc.dpdk.org/guides/contributing/patches.html#patch-fix-related-issues
>    3. mailto:1819846787@qq.com
>    4. mailto:david.marchand@redhat.com

[PATCH] [v3]lib/telemetry:fix telemetry conns leak in case of socket write fail

[Shaowei Sun]

Telemetry can only create 10 conns by default, each of which is processed
by a thread.

When a thread fails to write using socket, the thread will end directly
without reducing the total number of conns.

This will result in the machine running for a long time, and if there are
10 failures, the telemetry will be unavailable

Fixes: 6dd571fd07c3 ("telemetry: introduce new functionality")

Signed-off-by: Shaowei Sun <1819846787@qq.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Ciara Power <ciara.power@intel.com>
Acked-by: Chengwen Feng <fengchengwen@huawei.com>
---
 lib/telemetry/telemetry.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/telemetry/telemetry.c b/lib/telemetry/telemetry.c
index 31e2391867..0b00c04090 100644
--- a/lib/telemetry/telemetry.c
+++ b/lib/telemetry/telemetry.c
@@ -378,8 +378,8 @@ client_handler(void *sock_id)
 			"{\"version\":\"%s\",\"pid\":%d,\"max_output_len\":%d}",
 			telemetry_version, getpid(), MAX_OUTPUT_LEN);
 	if (write(s, info_str, strlen(info_str)) < 0) {
-		close(s);
-		return NULL;
+		TMTY_LOG_LINE(ERR, "Socket write base info to client failed");
+		goto exit;
 	}
 
 	/* receive data is not null terminated */
@@ -404,6 +404,7 @@ client_handler(void *sock_id)
 
 		bytes = read(s, buffer, sizeof(buffer) - 1);
 	}
+exit:
 	close(s);
 	rte_atomic_fetch_sub_explicit(&v2_clients, 1, rte_memory_order_relaxed);
 	return NULL;
-- 
2.37.1 (Apple Git-137.1)

Re: [PATCH] [v3]lib/telemetry:fix telemetry conns leak in case of socket write fail

[David Marchand]

On Tue, Jan 30, 2024 at 2:57 AM Shaowei Sun <1819846787@qq.com> wrote:
>
> Telemetry can only create 10 conns by default, each of which is processed
> by a thread.
>
> When a thread fails to write using socket, the thread will end directly
> without reducing the total number of conns.
>
> This will result in the machine running for a long time, and if there are
> 10 failures, the telemetry will be unavailable
>
> Fixes: 6dd571fd07c3 ("telemetry: introduce new functionality")
>
> Signed-off-by: Shaowei Sun <1819846787@qq.com>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Ciara Power <ciara.power@intel.com>
> Acked-by: Chengwen Feng <fengchengwen@huawei.com>

Thanks for the fix.
As far as I can see, the limiting of the number of connections (which
here results in a DoS on the telemetry socket) was added in commit
2a7d0b872f79 ("telemetry: add upper limit on connections").

If you confirm this is indeed this commit that introduced the issue, I
will fix the Fixes: tag myself when applying.


-- 
David Marchand

=?gb18030?B?u9i4tKO6IFtQQVRDSF0gW3YzXWxpYi90ZWxlbWV0cnk6Zml4IHRlbGVtZXRyeSBjb25ucyBsZWFrIGluIGNhc2Ugb2Ygc29ja2V0IHdyaXRlIGZhaWw=?=

[=?gb18030?B?IFNoYW9XZWkgU3Vu?=]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="gb18030", Size: 1738 bytes --]

Yes, you are correct, it should be the commit 2a7d0b872f79 that introduced the issue. Thank you for the correction. 


 
1819846787@qq.com



 




------------------ Ô­Ê¼Óʼþ ------------------
·¢¼þÈË: "David Marchand"<david.marchand@redhat.com&gt;; 
·¢ËÍʱ¼ä: 2024Äê2ÔÂ1ÈÕ(ÐÇÆÚËÄ) ÍíÉÏ9:14
ÊÕ¼þÈË: " ShaoWei Sun"<1819846787@qq.com&gt;; "ciara.power"<ciara.power@intel.com&gt;; 
³­ËÍ: "dev"<dev@dpdk.org&gt;; "Bruce Richardson"<bruce.richardson@intel.com&gt;; "Chengwen Feng"<fengchengwen@huawei.com&gt;; 
Ö÷Ìâ: Re: [PATCH] [v3]lib/telemetry:fix telemetry conns leak in case of socket write fail



1819846787@qq.com&gt; wrote:
&gt;
&gt; Telemetry can only create 10 conns by default, each of which is processed
&gt; by a thread.
&gt;
&gt; When a thread fails to write using socket, the thread will end directly
&gt; without reducing the total number of conns.
&gt;
&gt; This will result in the machine running for a long time, and if there are
&gt; 10 failures, the telemetry will be unavailable
&gt;
&gt; Fixes: 6dd571fd07c3 ("telemetry: introduce new functionality")
&gt;
&gt; Signed-off-by: Shaowei Sun <1819846787@qq.com&gt;
&gt; Acked-by: Bruce Richardson <bruce.richardson@intel.com&gt;
&gt; Acked-by: Ciara Power <ciara.power@intel.com&gt;
&gt; Acked-by: Chengwen Feng <fengchengwen@huawei.com&gt;

Thanks for the fix.
As far as I can see, the limiting of the number of connections (which
here results in a DoS on the telemetry socket) was added in commit
2a7d0b872f79 ("telemetry: add upper limit on connections").

If you confirm this is indeed this commit that introduced the issue, I
will fix the Fixes: tag myself when applying.


-- 
David Marchand

[-- Attachment #2: Type: text/html, Size: 4599 bytes --]

Re: [PATCH] [v3]lib/telemetry:fix telemetry conns leak in case of socket write fail

[David Marchand]

On Tue, Jan 30, 2024 at 2:57 AM Shaowei Sun <1819846787@qq.com> wrote:
>
> Telemetry can only create 10 conns by default, each of which is processed
> by a thread.
>
> When a thread fails to write using socket, the thread will end directly
> without reducing the total number of conns.
>
> This will result in the machine running for a long time, and if there are
> 10 failures, the telemetry will be unavailable
>
Fixes: 2a7d0b872f79 ("telemetry: add upper limit on connections")
Cc: stable@dpdk.org
>
> Signed-off-by: Shaowei Sun <1819846787@qq.com>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Ciara Power <ciara.power@intel.com>
> Acked-by: Chengwen Feng <fengchengwen@huawei.com>

Applied, thanks for the fix!


-- 
David Marchand

[PATCH] lib/telemetry:fix telemetry conns leak in case of socket write fail

[Shaowei Sun]

Telemetry can only create 10 conns by default, each of which is processed
by a thread.

When a thread fails to write using socket, the thread will end directly
without reducing the total number of conns.

This will result in the machine running for a long time, and if there are
10 failures, the telemetry will be unavailable

Fixes: 6dd571fd07c3 ("telemetry: introduce new functionality")

Signed-off-by: Shaowei Sun <1819846787@qq.com>
---
 lib/telemetry/telemetry.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/telemetry/telemetry.c b/lib/telemetry/telemetry.c
index 31e2391867..0b00c04090 100644
--- a/lib/telemetry/telemetry.c
+++ b/lib/telemetry/telemetry.c
@@ -378,8 +378,8 @@ client_handler(void *sock_id)
 			"{\"version\":\"%s\",\"pid\":%d,\"max_output_len\":%d}",
 			telemetry_version, getpid(), MAX_OUTPUT_LEN);
 	if (write(s, info_str, strlen(info_str)) < 0) {
-		close(s);
-		return NULL;
+		TMTY_LOG_LINE(ERR, "Socket write base info to client failed");
+		goto exit;
 	}
 
 	/* receive data is not null terminated */
@@ -404,6 +404,7 @@ client_handler(void *sock_id)
 
 		bytes = read(s, buffer, sizeof(buffer) - 1);
 	}
+exit:
 	close(s);
 	rte_atomic_fetch_sub_explicit(&v2_clients, 1, rte_memory_order_relaxed);
 	return NULL;
-- 
2.37.1 (Apple Git-137.1)

[PATCH] lib/telemetry:fix telemetry conns leak in case of socket write fail

[sunshaowei01]

Telemetry can only create 10 conns by default, each of which is processed
by a thread.

When a thread fails to write using socket, the thread will end directly
without reducing the total number of conns.

This will result in the machine running for a long time, and if there are
10 failures, the telemetry will be unavailable

Fixes: 6dd571fd07c3 ("telemetry: introduce new functionality")

Signed-off-by: Shaowei Sun <1819846787@qq.com>
---
 lib/telemetry/telemetry.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/telemetry/telemetry.c b/lib/telemetry/telemetry.c
index 31e2391867..0b00c04090 100644
--- a/lib/telemetry/telemetry.c
+++ b/lib/telemetry/telemetry.c
@@ -378,8 +378,8 @@ client_handler(void *sock_id)
 			"{\"version\":\"%s\",\"pid\":%d,\"max_output_len\":%d}",
 			telemetry_version, getpid(), MAX_OUTPUT_LEN);
 	if (write(s, info_str, strlen(info_str)) < 0) {
-		close(s);
-		return NULL;
+		TMTY_LOG_LINE(ERR, "Socket write base info to client failed");
+		goto exit;
 	}
 
 	/* receive data is not null terminated */
@@ -404,6 +404,7 @@ client_handler(void *sock_id)
 
 		bytes = read(s, buffer, sizeof(buffer) - 1);
 	}
+exit:
 	close(s);
 	rte_atomic_fetch_sub_explicit(&v2_clients, 1, rte_memory_order_relaxed);
 	return NULL;
-- 
2.37.1 (Apple Git-137.1)