DPDK patches and discussions
 help / color / mirror / Atom feed
* [PATCH] vhost: Fix the crash caused by accessing the released memory
@ 2024-06-19 12:27 zhaoxinxin
  2024-06-24  9:20 ` Maxime Coquelin
  0 siblings, 1 reply; 10+ messages in thread
From: zhaoxinxin @ 2024-06-19 12:27 UTC (permalink / raw)
  To: dev; +Cc: zhaoxinxin

The rte_vhost_driver_unregister() vhost_user_read_cb()
vhost_user_client_reconnect() can be called at the same time by 3 threads.
when memory of vsocket is freed in rte_vhost_driver_unregister(),
then vhost_user_read_cb() maybe add vsocket to reconn_list,
the invalid memory of vsocket is accessed in vhost_user_client_reconnect().

The core trace is:
Program terminated with signal 11, Segmentation fault.
The fix is to perform a delete operation again after releasing the memory

Signed-off-by: zhaoxinxin <15957197901@163.com>
---
 lib/vhost/socket.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c
index a75728a2e4..01946096c4 100644
--- a/lib/vhost/socket.c
+++ b/lib/vhost/socket.c
@@ -1121,6 +1121,8 @@ rte_vhost_driver_unregister(const char *path)
 		if (vsocket->is_server) {
 			close(vsocket->socket_fd);
 			unlink(path);
+		} else if (vsocket->reconnect) {
+			vhost_user_remove_reconnect(vsocket);
 		}
 
 		pthread_mutex_destroy(&vsocket->conn_mutex);
-- 
2.45.2


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] vhost: Fix the crash caused by accessing the released memory
  2024-06-19 12:27 [PATCH] vhost: Fix the crash caused by accessing the released memory zhaoxinxin
@ 2024-06-24  9:20 ` Maxime Coquelin
  2024-06-25  1:56   ` 15957197901
  2024-12-03  6:49   ` 15957197901
  0 siblings, 2 replies; 10+ messages in thread
From: Maxime Coquelin @ 2024-06-24  9:20 UTC (permalink / raw)
  To: zhaoxinxin, dev

Hi,

On 6/19/24 14:27, zhaoxinxin wrote:
> The rte_vhost_driver_unregister() vhost_user_read_cb()
> vhost_user_client_reconnect() can be called at the same time by 3 threads.
> when memory of vsocket is freed in rte_vhost_driver_unregister(),
> then vhost_user_read_cb() maybe add vsocket to reconn_list,
> the invalid memory of vsocket is accessed in vhost_user_client_reconnect().

It is not clear to me why 3 threads are calling
rte_vhost_driver_unregister() at the same time, isn't it an application
issue?

> The core trace is:
> Program terminated with signal 11, Segmentation fault.
> The fix is to perform a delete operation again after releasing the memory
>

We need a Fixes tag and Cc stable@dpdk.org so that it is backported.

> Signed-off-by: zhaoxinxin <15957197901@163.com>

The format is Firstname Lastname <email>


> ---
>   lib/vhost/socket.c | 2 ++
>   1 file changed, 2 insertions(+)
> 
> diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c
> index a75728a2e4..01946096c4 100644
> --- a/lib/vhost/socket.c
> +++ b/lib/vhost/socket.c
> @@ -1121,6 +1121,8 @@ rte_vhost_driver_unregister(const char *path)
>   		if (vsocket->is_server) {
>   			close(vsocket->socket_fd);
>   			unlink(path);
> +		} else if (vsocket->reconnect) {
> +			vhost_user_remove_reconnect(vsocket);
>   		}
>   
>   		pthread_mutex_destroy(&vsocket->conn_mutex);


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re:Re: [PATCH] vhost: Fix the crash caused by accessing the released memory
  2024-06-24  9:20 ` Maxime Coquelin
@ 2024-06-25  1:56   ` 15957197901
  2024-12-03  6:49   ` 15957197901
  1 sibling, 0 replies; 10+ messages in thread
From: 15957197901 @ 2024-06-25  1:56 UTC (permalink / raw)
  To: Maxime Coquelin; +Cc: dev

[-- Attachment #1: Type: text/plain, Size: 2507 bytes --]

Hi Maxime,

The scenario where I encountered coredump was ovs-dpdk,

similar to patch: https://github.com/DPDK/dpdk/commit/52d874dc67055a943867456d3e5c730168bfba18.

Only one thread called rte_vhost_driver_unregister(), but at the same time, 

two other threads called vhost_user_read_cb() and vhost_user_client_reconnect() respectively.




The specific reasons for coredump are as follows:

vhostuser port is created as client.

Thread 1 calls rte_vhost_driver_unregister() to remove the vsocket of reconn from the reconn list.

then “vhost-events” thread calls vhost_user_read_cb() to add the vsocket of reconn back to the reconn list.
At this time, after thread 1 releases the vsocket memory, the socket of vhostuser reconnects successfully, 
"vhost_reconn" thread will access the released memory.
Therefore, The fix is to perform a delete operation again after releasing the memory.




>We need a Fixes tag and Cc stable@dpdk.org so that it is backported.
>The format is Firstname Lastname <email>
I will modify and resubmit the patch, thank you.






At 2024-06-24 17:20:00, "Maxime Coquelin" <maxime.coquelin@redhat.com> wrote: >Hi, > >On 6/19/24 14:27, zhaoxinxin wrote: >> The rte_vhost_driver_unregister() vhost_user_read_cb() >> vhost_user_client_reconnect() can be called at the same time by 3 threads. >> when memory of vsocket is freed in rte_vhost_driver_unregister(), >> then vhost_user_read_cb() maybe add vsocket to reconn_list, >> the invalid memory of vsocket is accessed in vhost_user_client_reconnect(). > >It is not clear to me why 3 threads are calling >rte_vhost_driver_unregister() at the same time, isn't it an application >issue? > >> The core trace is: >> Program terminated with signal 11, Segmentation fault. >> The fix is to perform a delete operation again after releasing the memory >> > >We need a Fixes tag and Cc stable@dpdk.org so that it is backported. > >> Signed-off-by: zhaoxinxin <15957197901@163.com> > >The format is Firstname Lastname <email> > > >> --- >> lib/vhost/socket.c | 2 ++ >> 1 file changed, 2 insertions(+) >> >> diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c >> index a75728a2e4..01946096c4 100644 >> --- a/lib/vhost/socket.c >> +++ b/lib/vhost/socket.c >> @@ -1121,6 +1121,8 @@ rte_vhost_driver_unregister(const char *path) >> if (vsocket->is_server) { >> close(vsocket->socket_fd); >> unlink(path); >> + } else if (vsocket->reconnect) { >> + vhost_user_remove_reconnect(vsocket); >> } >> >> pthread_mutex_destroy(&vsocket->conn_mutex);

[-- Attachment #2: Type: text/html, Size: 3689 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re:Re: [PATCH] vhost: Fix the crash caused by accessing the released memory
  2024-06-24  9:20 ` Maxime Coquelin
  2024-06-25  1:56   ` 15957197901
@ 2024-12-03  6:49   ` 15957197901
  1 sibling, 0 replies; 10+ messages in thread
From: 15957197901 @ 2024-12-03  6:49 UTC (permalink / raw)
  To: Maxime Coquelin; +Cc: dev

[-- Attachment #1: Type: text/plain, Size: 2539 bytes --]

Hello Maxime Coquelin,




The scenario where I encountered coredump was ovs-dpdk,

similar to patch: https://github.com/DPDK/dpdk/commit/52d874dc67055a943867456d3e5c730168bfba18.

Only one thread called rte_vhost_driver_unregister(), but at the same time, 

two other threads called vhost_user_read_cb() and vhost_user_client_reconnect() respectively.




The specific reasons for coredump are as follows:

vhostuser port is created as client.

Thread 1 calls rte_vhost_driver_unregister() to remove the vsocket of reconn from the reconn list.

then “vhost-events” thread calls vhost_user_read_cb() to add the vsocket of reconn back to the reconn list.
At this time, after thread 1 releases the vsocket memory, the socket of vhostuser reconnects successfully, 
"vhost_reconn" thread will access the released memory.
Therefore, The fix is to perform a delete operation again after releasing the memory.



I have resubmitted the patch, please review it again.

https://patches.dpdk.org/project/dpdk/patch/20240625093149.63247-1-15957197901@163.com/







At 2024-06-24 17:20:00, "Maxime Coquelin" <maxime.coquelin@redhat.com> wrote:
>Hi,
>
>On 6/19/24 14:27, zhaoxinxin wrote:
>> The rte_vhost_driver_unregister() vhost_user_read_cb()
>> vhost_user_client_reconnect() can be called at the same time by 3 threads.
>> when memory of vsocket is freed in rte_vhost_driver_unregister(),
>> then vhost_user_read_cb() maybe add vsocket to reconn_list,
>> the invalid memory of vsocket is accessed in vhost_user_client_reconnect().
>
>It is not clear to me why 3 threads are calling
>rte_vhost_driver_unregister() at the same time, isn't it an application
>issue?
>
>> The core trace is:
>> Program terminated with signal 11, Segmentation fault.
>> The fix is to perform a delete operation again after releasing the memory
>>
>
>We need a Fixes tag and Cc stable@dpdk.org so that it is backported.
>
>> Signed-off-by: zhaoxinxin <15957197901@163.com>
>
>The format is Firstname Lastname <email>
>
>
>> ---
>>   lib/vhost/socket.c | 2 ++
>>   1 file changed, 2 insertions(+)
>> 
>> diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c
>> index a75728a2e4..01946096c4 100644
>> --- a/lib/vhost/socket.c
>> +++ b/lib/vhost/socket.c
>> @@ -1121,6 +1121,8 @@ rte_vhost_driver_unregister(const char *path)
>>   		if (vsocket->is_server) {
>>   			close(vsocket->socket_fd);
>>   			unlink(path);
>> +		} else if (vsocket->reconnect) {
>> +			vhost_user_remove_reconnect(vsocket);
>>   		}
>>   
>>   		pthread_mutex_destroy(&vsocket->conn_mutex);

[-- Attachment #2: Type: text/html, Size: 4764 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] vhost: Fix the crash caused by accessing the released memory
  2024-06-25  2:13 zhaoxinxin
@ 2024-06-25 13:51 ` Patrick Robb
  0 siblings, 0 replies; 10+ messages in thread
From: Patrick Robb @ 2024-06-25 13:51 UTC (permalink / raw)
  To: zhaoxinxin; +Cc: dev, stable

rerunning the failing performance test in CI testing as it was a false failure.

On Tue, Jun 25, 2024 at 2:39 AM zhaoxinxin <15957197901@163.com> wrote:
>
> The rte_vhost_driver_unregister() vhost_user_read_cb()
> vhost_user_client_reconnect() can be called at the same time by 3 threads.
> when memory of vsocket is freed in rte_vhost_driver_unregister(),
> then vhost_user_read_cb() maybe add vsocket to reconn_list,
> the invalid memory of vsocket is accessed in vhost_user_client_reconnect().
> It's a bug for vhost as client.
>
> E.g., vhostuser port is created as client.
> Thread 1 calls rte_vhost_driver_unregister() to remove
> the vsocket of reconn from the reconn list.
> Then “vhost-events” thread calls vhost_user_read_cb() to
> add the vsocket of reconn back to the reconn list.
> At this time, after thread 1 releases the vsocket memory,
> the socket of vhostuser reconnects successfully,
> "vhost_reconn" thread will access the released memory.
>
> The core trace is:
> Program terminated with signal 11, Segmentation fault.
> The fix is to perform a delete operation again after releasing the memory
>
> Fixes: 451dc0f ("vhost: fix crash on port deletion")
> Cc: stable@dpdk.org
>
> Signed-off-by: Xinxin Zhao <15957197901@163.com>
> ---
>  lib/vhost/socket.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c
> index a75728a2e4..01946096c4 100644
> --- a/lib/vhost/socket.c
> +++ b/lib/vhost/socket.c
> @@ -1121,6 +1121,8 @@ rte_vhost_driver_unregister(const char *path)
>                 if (vsocket->is_server) {
>                         close(vsocket->socket_fd);
>                         unlink(path);
> +               } else if (vsocket->reconnect) {
> +                       vhost_user_remove_reconnect(vsocket);
>                 }
>
>                 pthread_mutex_destroy(&vsocket->conn_mutex);
> --
> 2.45.2
>

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH] vhost: Fix the crash caused by accessing the released memory
@ 2024-06-25  9:31 Xinxin Zhao
  0 siblings, 0 replies; 10+ messages in thread
From: Xinxin Zhao @ 2024-06-25  9:31 UTC (permalink / raw)
  To: dev; +Cc: Xinxin Zhao, stable

The rte_vhost_driver_unregister() vhost_user_read_cb()
vhost_user_client_reconnect() can be called at the same time by 3 threads.
when memory of vsocket is freed in rte_vhost_driver_unregister(),
then vhost_user_read_cb() maybe add vsocket to reconn_list,
the invalid memory of vsocket is accessed in vhost_user_client_reconnect().
It's a bug for vhost as client.

E.g., vhostuser port is created as client.
Thread 1 calls rte_vhost_driver_unregister() to remove
the vsocket of reconn from the reconn list.
Then “vhost-events” thread calls vhost_user_read_cb() to
add the vsocket of reconn back to the reconn list.
At this time, after thread 1 releases the vsocket memory,
the socket of vhostuser reconnects successfully,
"vhost_reconn" thread will access the released memory.

The core trace is:
Program terminated with signal 11, Segmentation fault.
The fix is to perform a delete operation again after releasing the memory

Fixes: 451dc0f ("vhost: fix crash on port deletion")
Cc: stable@dpdk.org

Signed-off-by: Xinxin Zhao <15957197901@163.com>
---
 lib/vhost/socket.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c
index a75728a2e4..01946096c4 100644
--- a/lib/vhost/socket.c
+++ b/lib/vhost/socket.c
@@ -1121,6 +1121,8 @@ rte_vhost_driver_unregister(const char *path)
 		if (vsocket->is_server) {
 			close(vsocket->socket_fd);
 			unlink(path);
+		} else if (vsocket->reconnect) {
+			vhost_user_remove_reconnect(vsocket);
 		}
 
 		pthread_mutex_destroy(&vsocket->conn_mutex);
-- 
2.45.2


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH] vhost: Fix the crash caused by accessing the released memory
@ 2024-06-25  2:13 zhaoxinxin
  2024-06-25 13:51 ` Patrick Robb
  0 siblings, 1 reply; 10+ messages in thread
From: zhaoxinxin @ 2024-06-25  2:13 UTC (permalink / raw)
  To: dev; +Cc: zhaoxinxin, stable

The rte_vhost_driver_unregister() vhost_user_read_cb()
vhost_user_client_reconnect() can be called at the same time by 3 threads.
when memory of vsocket is freed in rte_vhost_driver_unregister(),
then vhost_user_read_cb() maybe add vsocket to reconn_list,
the invalid memory of vsocket is accessed in vhost_user_client_reconnect().
It's a bug for vhost as client.

E.g., vhostuser port is created as client.
Thread 1 calls rte_vhost_driver_unregister() to remove
the vsocket of reconn from the reconn list.
Then “vhost-events” thread calls vhost_user_read_cb() to
add the vsocket of reconn back to the reconn list.
At this time, after thread 1 releases the vsocket memory,
the socket of vhostuser reconnects successfully,
"vhost_reconn" thread will access the released memory.

The core trace is:
Program terminated with signal 11, Segmentation fault.
The fix is to perform a delete operation again after releasing the memory

Fixes: 451dc0f ("vhost: fix crash on port deletion")
Cc: stable@dpdk.org

Signed-off-by: Xinxin Zhao <15957197901@163.com>
---
 lib/vhost/socket.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c
index a75728a2e4..01946096c4 100644
--- a/lib/vhost/socket.c
+++ b/lib/vhost/socket.c
@@ -1121,6 +1121,8 @@ rte_vhost_driver_unregister(const char *path)
 		if (vsocket->is_server) {
 			close(vsocket->socket_fd);
 			unlink(path);
+		} else if (vsocket->reconnect) {
+			vhost_user_remove_reconnect(vsocket);
 		}
 
 		pthread_mutex_destroy(&vsocket->conn_mutex);
-- 
2.45.2


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH] vhost: Fix the crash caused by accessing the released memory
@ 2024-06-19  8:39 zhaoxinxin
  0 siblings, 0 replies; 10+ messages in thread
From: zhaoxinxin @ 2024-06-19  8:39 UTC (permalink / raw)
  To: dev; +Cc: zhaoxinxin

The rte_vhost_driver_unregister() vhost_user_read_cb()
vhost_user_client_reconnect() can be called at the same time by 3 threads.
when memory of vsocket is freed in rte_vhost_driver_unregister(),
then vhost_user_read_cb() maybe add vsocket to reconn_list,
the invalid memory of vsocket is accessed in vhost_user_client_reconnect().

The core trace is:
    Program terminated with signal 11, Segmentation fault.
    The fix is to perform a delete operation again after releasing the memory

Signed-off-by: zhaoxinxin <15957197901@163.com>
---
 lib/vhost/socket.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c
index a75728a2e4..01946096c4 100644
--- a/lib/vhost/socket.c
+++ b/lib/vhost/socket.c
@@ -1121,6 +1121,8 @@ rte_vhost_driver_unregister(const char *path)
 		if (vsocket->is_server) {
 			close(vsocket->socket_fd);
 			unlink(path);
+		} else if (vsocket->reconnect) {
+			vhost_user_remove_reconnect(vsocket);
 		}
 
 		pthread_mutex_destroy(&vsocket->conn_mutex);
-- 
2.45.2


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH] vhost: Fix the crash caused by accessing the released memory
@ 2024-06-18  5:59 zhaoxinxin
  0 siblings, 0 replies; 10+ messages in thread
From: zhaoxinxin @ 2024-06-18  5:59 UTC (permalink / raw)
  To: dev; +Cc: zhaoxinxin

The rte_vhost_driver_unregister() vhost_user_read_cb() vhost_user_client_reconnect() can be called at the same time by 3 threads.
when memory of vsocket is freed in rte_vhost_driver_unregister(),
then vhost_user_read_cb() maybe add vsocket to reconn_list,
the invalid memory of vsocket is accessed in vhost_user_client_reconnect().

The core trace is:
    Program terminated with signal 11, Segmentation fault.
    The fix is to perform a delete operation again after releasing the memory

Signed-off-by: zhaoxinxin <15957197901@163.com>
---
 lib/vhost/socket.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c
index a75728a2e4..01946096c4 100644
--- a/lib/vhost/socket.c
+++ b/lib/vhost/socket.c
@@ -1121,6 +1121,8 @@ rte_vhost_driver_unregister(const char *path)
 		if (vsocket->is_server) {
 			close(vsocket->socket_fd);
 			unlink(path);
+		} else if (vsocket->reconnect) {
+			vhost_user_remove_reconnect(vsocket);
 		}
 
 		pthread_mutex_destroy(&vsocket->conn_mutex);
-- 
2.45.2


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH] vhost: Fix the crash caused by accessing the released memory
@ 2024-06-17  3:35 zhaoxinxin
  0 siblings, 0 replies; 10+ messages in thread
From: zhaoxinxin @ 2024-06-17  3:35 UTC (permalink / raw)
  To: dev; +Cc: zhaoxinxin

The rte_vhost_driver_unregister() 、vhost_user_read_cb()、vhost_user_client_reconnect() can be called at the same time by 3 threads.
when memory of vsocket is freed in rte_vhost_driver_unregister(), then vhost_user_read_cb() maybe add vsocket to reconn_list, the invalid memory of vsocket is accessed in vhost_user_client_reconnect().
The core trace is:
    Program terminated with signal 11, Segmentation fault.
    The fix is to perform a delete operation again after releasing the memory

Signed-off-by: zhaoxinxin <15957197901@163.com>
---
 lib/vhost/socket.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c
index a75728a2e4..01946096c4 100644
--- a/lib/vhost/socket.c
+++ b/lib/vhost/socket.c
@@ -1121,6 +1121,8 @@ rte_vhost_driver_unregister(const char *path)
 		if (vsocket->is_server) {
 			close(vsocket->socket_fd);
 			unlink(path);
+		} else if (vsocket->reconnect) {
+			vhost_user_remove_reconnect(vsocket);
 		}
 
 		pthread_mutex_destroy(&vsocket->conn_mutex);
-- 
2.45.2


^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2024-12-04  9:51 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-06-19 12:27 [PATCH] vhost: Fix the crash caused by accessing the released memory zhaoxinxin
2024-06-24  9:20 ` Maxime Coquelin
2024-06-25  1:56   ` 15957197901
2024-12-03  6:49   ` 15957197901
  -- strict thread matches above, loose matches on Subject: below --
2024-06-25  9:31 Xinxin Zhao
2024-06-25  2:13 zhaoxinxin
2024-06-25 13:51 ` Patrick Robb
2024-06-19  8:39 zhaoxinxin
2024-06-18  5:59 zhaoxinxin
2024-06-17  3:35 zhaoxinxin

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).