[dpdk-dev] [PATCH] eal: fix use-after-free issue on thread creation

[dpdk-dev] [PATCH] eal: fix use-after-free issue on thread creation

[Jianfeng Tan]

After below commit, we encounter some strange issue:
  1) Dead lock as described here:
     http://dpdk.org/ml/archives/dev/2018-April/099806.html
  2) SIGSEGV issue when starting a testpmd in VM.

Considering below commit changes to use dynamic memory instead of
stack for memory barrier, we doubt it's caused by use-after-free.

Fixes: 3d09a6e26d8b ("eal: fix threads block on barrier")

Reported-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reported-by: Lei Yao <lei.a.yao@intel.com>
Suggested-by: Stephen Hemminger <stephen@networkplumber.org>
Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
---
 lib/librte_eal/common/eal_common_thread.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_thread.c b/lib/librte_eal/common/eal_common_thread.c
index de69452..56ffd2e 100644
--- a/lib/librte_eal/common/eal_common_thread.c
+++ b/lib/librte_eal/common/eal_common_thread.c
@@ -149,11 +149,16 @@ struct rte_thread_ctrl_params {
 
 static void *rte_thread_init(void *arg)
 {
+	int ret;
 	struct rte_thread_ctrl_params *params = arg;
 	void *(*start_routine)(void *) = params->start_routine;
 	void *routine_arg = params->arg;
 
-	pthread_barrier_wait(&params->configured);
+	ret = pthread_barrier_wait(&params->configured);
+	if (ret == PTHREAD_BARRIER_SERIAL_THREAD) {
+		pthread_barrier_destroy(&params->configured);
+		free(params);
+	}
 
 	return start_routine(routine_arg);
 }
@@ -204,8 +209,11 @@ rte_ctrl_thread_create(pthread_t *thread, const char *name,
 	if (ret < 0)
 		goto fail;
 
-	pthread_barrier_wait(&params->configured);
-	free(params);
+	ret = pthread_barrier_wait(&params->configured);
+	if (ret == PTHREAD_BARRIER_SERIAL_THREAD) {
+		pthread_barrier_destroy(&params->configured);
+		free(params);
+	}
 
 	return 0;
 
-- 
2.7.4

[dpdk-dev] [PATCH v2] eal: fix use-after-free issue on thread creation

[Jianfeng Tan]

After below commit, we encounter some strange issue:
  1) Dead lock as described here:
     http://dpdk.org/ml/archives/dev/2018-April/099806.html
  2) SIGSEGV issue when starting a testpmd in VM.

Considering below commit changes to use dynamic memory instead of
stack for memory barrier, we doubt it's caused by use-after-free.

Fixes: 3d09a6e26d8b ("eal: fix threads block on barrier")

Reported-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reported-by: Lei Yao <lei.a.yao@intel.com>
Suggested-by: Stephen Hemminger <stephen@networkplumber.org>
Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
---
 v1->v2:
 - Destroy barrier if failure happens.
 lib/librte_eal/common/eal_common_thread.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_thread.c b/lib/librte_eal/common/eal_common_thread.c
index de69452..5f0c61f 100644
--- a/lib/librte_eal/common/eal_common_thread.c
+++ b/lib/librte_eal/common/eal_common_thread.c
@@ -149,11 +149,16 @@ struct rte_thread_ctrl_params {
 
 static void *rte_thread_init(void *arg)
 {
+	int ret;
 	struct rte_thread_ctrl_params *params = arg;
 	void *(*start_routine)(void *) = params->start_routine;
 	void *routine_arg = params->arg;
 
-	pthread_barrier_wait(&params->configured);
+	ret = pthread_barrier_wait(&params->configured);
+	if (ret == PTHREAD_BARRIER_SERIAL_THREAD) {
+		pthread_barrier_destroy(&params->configured);
+		free(params);
+	}
 
 	return start_routine(routine_arg);
 }
@@ -204,12 +209,16 @@ rte_ctrl_thread_create(pthread_t *thread, const char *name,
 	if (ret < 0)
 		goto fail;
 
-	pthread_barrier_wait(&params->configured);
-	free(params);
+	ret = pthread_barrier_wait(&params->configured);
+	if (ret == PTHREAD_BARRIER_SERIAL_THREAD) {
+		pthread_barrier_destroy(&params->configured);
+		free(params);
+	}
 
 	return 0;
 
 fail:
+	pthread_barrier_destroy(&params->configured);
 	pthread_cancel(*thread);
 	pthread_join(*thread, NULL);
 	free(params);
-- 
2.7.4

Re: [dpdk-dev] [PATCH v2] eal: fix use-after-free issue on thread creation

[Olivier Matz]

Hi Jianfeng,

On Wed, May 02, 2018 at 10:17:50AM +0000, Jianfeng Tan wrote:
> After below commit, we encounter some strange issue:
>   1) Dead lock as described here:
>      http://dpdk.org/ml/archives/dev/2018-April/099806.html
>   2) SIGSEGV issue when starting a testpmd in VM.
> 
> Considering below commit changes to use dynamic memory instead of
> stack for memory barrier, we doubt it's caused by use-after-free.
> 
> Fixes: 3d09a6e26d8b ("eal: fix threads block on barrier")
> 
> Reported-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> Reported-by: Lei Yao <lei.a.yao@intel.com>
> Suggested-by: Stephen Hemminger <stephen@networkplumber.org>
> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> ---
>  v1->v2:
>  - Destroy barrier if failure happens.
>  lib/librte_eal/common/eal_common_thread.c | 15 ++++++++++++---
>  1 file changed, 12 insertions(+), 3 deletions(-)
> 
> diff --git a/lib/librte_eal/common/eal_common_thread.c b/lib/librte_eal/common/eal_common_thread.c
> index de69452..5f0c61f 100644
> --- a/lib/librte_eal/common/eal_common_thread.c
> +++ b/lib/librte_eal/common/eal_common_thread.c
> @@ -149,11 +149,16 @@ struct rte_thread_ctrl_params {
>  
>  static void *rte_thread_init(void *arg)
>  {
> +	int ret;
>  	struct rte_thread_ctrl_params *params = arg;
>  	void *(*start_routine)(void *) = params->start_routine;
>  	void *routine_arg = params->arg;
>  
> -	pthread_barrier_wait(&params->configured);
> +	ret = pthread_barrier_wait(&params->configured);
> +	if (ret == PTHREAD_BARRIER_SERIAL_THREAD) {
> +		pthread_barrier_destroy(&params->configured);
> +		free(params);
> +	}
>  
>  	return start_routine(routine_arg);
>  }
> @@ -204,12 +209,16 @@ rte_ctrl_thread_create(pthread_t *thread, const char *name,
>  	if (ret < 0)
>  		goto fail;
>  
> -	pthread_barrier_wait(&params->configured);
> -	free(params);
> +	ret = pthread_barrier_wait(&params->configured);
> +	if (ret == PTHREAD_BARRIER_SERIAL_THREAD) {
> +		pthread_barrier_destroy(&params->configured);
> +		free(params);
> +	}
>  
>  	return 0;
>  
>  fail:
> +	pthread_barrier_destroy(&params->configured);

I think we should have the same code than above in the fail case:

	ret = pthread_barrier_wait(&params->configured);
	if (ret == PTHREAD_BARRIER_SERIAL_THREAD) {
		pthread_barrier_destroy(&params->configured);
		free(params);
	}

Else, the child will wait forever on the barrier on failure.

This can be tested with this standalone program:
https://www.droids-corp.org/~zer0/hidden/ctrl_thread.c

gcc -W -Wall -Werror -Wextra -pthread ctrl_thread.c
./a.out -> fail

gcc -W -Wall -Werror -Wextra -pthread -DFIX ctrl_thread.c
./a.out -> ok



>  	pthread_cancel(*thread);
>  	pthread_join(*thread, NULL);
>  	free(params);
> -- 
> 2.7.4
> 

[dpdk-dev] [PATCH v3] eal: fix use-after-free issue on thread creation

[Jianfeng Tan]

After below commit, we encounter some strange issue:
  1) Dead lock as described here:
     http://dpdk.org/ml/archives/dev/2018-April/099806.html
  2) SIGSEGV issue when starting a testpmd in VM.

Considering below commit changes to use dynamic memory instead of
stack for memory barrier, we doubt it's caused by use-after-free.

Fixes: 3d09a6e26d8b ("eal: fix threads block on barrier")

Reported-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reported-by: Lei Yao <lei.a.yao@intel.com>
Suggested-by: Stephen Hemminger <stephen@networkplumber.org>
Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
Suggested-by: Olivier Matz <olivier.matz@6wind.com>
---
 v1->v2:
 - Destroy barrier if failure happens.
 v2->v3:
 - Add proper failure handling.
 lib/librte_eal/common/eal_common_thread.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_thread.c b/lib/librte_eal/common/eal_common_thread.c
index de69452..cb0855d 100644
--- a/lib/librte_eal/common/eal_common_thread.c
+++ b/lib/librte_eal/common/eal_common_thread.c
@@ -149,11 +149,16 @@ struct rte_thread_ctrl_params {
 
 static void *rte_thread_init(void *arg)
 {
+	int ret;
 	struct rte_thread_ctrl_params *params = arg;
 	void *(*start_routine)(void *) = params->start_routine;
 	void *routine_arg = params->arg;
 
-	pthread_barrier_wait(&params->configured);
+	ret = pthread_barrier_wait(&params->configured);
+	if (ret == PTHREAD_BARRIER_SERIAL_THREAD) {
+		pthread_barrier_destroy(&params->configured);
+		free(params);
+	}
 
 	return start_routine(routine_arg);
 }
@@ -204,14 +209,21 @@ rte_ctrl_thread_create(pthread_t *thread, const char *name,
 	if (ret < 0)
 		goto fail;
 
-	pthread_barrier_wait(&params->configured);
-	free(params);
+	ret = pthread_barrier_wait(&params->configured);
+	if (ret == PTHREAD_BARRIER_SERIAL_THREAD) {
+		pthread_barrier_destroy(&params->configured);
+		free(params);
+	}
 
 	return 0;
 
 fail:
+	if (PTHREAD_BARRIER_SERIAL_THREAD ==
+	    pthread_barrier_wait(&params->configured)) {
+		pthread_barrier_destroy(&params->configured);
+		free(params);
+	}
 	pthread_cancel(*thread);
 	pthread_join(*thread, NULL);
-	free(params);
 	return ret;
 }
-- 
2.7.4

Re: [dpdk-dev] [PATCH v2] eal: fix use-after-free issue on thread creation

[Tan, Jianfeng]

Hi Olivier,


On 5/2/2018 7:24 PM, Olivier Matz wrote:
> Hi Jianfeng,
>
> On Wed, May 02, 2018 at 10:17:50AM +0000, Jianfeng Tan wrote:
>> After below commit, we encounter some strange issue:
>>    1) Dead lock as described here:
>>       http://dpdk.org/ml/archives/dev/2018-April/099806.html
>>    2) SIGSEGV issue when starting a testpmd in VM.
>>
>> Considering below commit changes to use dynamic memory instead of
>> stack for memory barrier, we doubt it's caused by use-after-free.
>>
>> Fixes: 3d09a6e26d8b ("eal: fix threads block on barrier")
>>
>> Reported-by: Maxime Coquelin <maxime.coquelin@redhat.com>
>> Reported-by: Lei Yao <lei.a.yao@intel.com>
>> Suggested-by: Stephen Hemminger <stephen@networkplumber.org>
>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
>> ---
>>   v1->v2:
>>   - Destroy barrier if failure happens.
>>   lib/librte_eal/common/eal_common_thread.c | 15 ++++++++++++---
>>   1 file changed, 12 insertions(+), 3 deletions(-)
>>
>> diff --git a/lib/librte_eal/common/eal_common_thread.c b/lib/librte_eal/common/eal_common_thread.c
>> index de69452..5f0c61f 100644
>> --- a/lib/librte_eal/common/eal_common_thread.c
>> +++ b/lib/librte_eal/common/eal_common_thread.c
>> @@ -149,11 +149,16 @@ struct rte_thread_ctrl_params {
>>   
>>   static void *rte_thread_init(void *arg)
>>   {
>> +	int ret;
>>   	struct rte_thread_ctrl_params *params = arg;
>>   	void *(*start_routine)(void *) = params->start_routine;
>>   	void *routine_arg = params->arg;
>>   
>> -	pthread_barrier_wait(&params->configured);
>> +	ret = pthread_barrier_wait(&params->configured);
>> +	if (ret == PTHREAD_BARRIER_SERIAL_THREAD) {
>> +		pthread_barrier_destroy(&params->configured);
>> +		free(params);
>> +	}
>>   
>>   	return start_routine(routine_arg);
>>   }
>> @@ -204,12 +209,16 @@ rte_ctrl_thread_create(pthread_t *thread, const char *name,
>>   	if (ret < 0)
>>   		goto fail;
>>   
>> -	pthread_barrier_wait(&params->configured);
>> -	free(params);
>> +	ret = pthread_barrier_wait(&params->configured);
>> +	if (ret == PTHREAD_BARRIER_SERIAL_THREAD) {
>> +		pthread_barrier_destroy(&params->configured);
>> +		free(params);
>> +	}
>>   
>>   	return 0;
>>   
>>   fail:
>> +	pthread_barrier_destroy(&params->configured);
> I think we should have the same code than above in the fail case:
>
> 	ret = pthread_barrier_wait(&params->configured);
> 	if (ret == PTHREAD_BARRIER_SERIAL_THREAD) {
> 		pthread_barrier_destroy(&params->configured);
> 		free(params);
> 	}
>
> Else, the child will wait forever on the barrier on failure.

Make sense. I send v3 patch according to your advice, except that not 
polluting ret.

Thanks,
Jianfeng

>
> This can be tested with this standalone program:
> https://www.droids-corp.org/~zer0/hidden/ctrl_thread.c
>
> gcc -W -Wall -Werror -Wextra -pthread ctrl_thread.c
> ./a.out -> fail
>
> gcc -W -Wall -Werror -Wextra -pthread -DFIX ctrl_thread.c
> ./a.out -> ok
>
>
>
>>   	pthread_cancel(*thread);
>>   	pthread_join(*thread, NULL);
>>   	free(params);
>> -- 
>> 2.7.4
>>

Re: [dpdk-dev] [PATCH v3] eal: fix use-after-free issue on thread creation

[Olivier Matz]

On Wed, May 02, 2018 at 01:52:14PM +0000, Jianfeng Tan wrote:
> After below commit, we encounter some strange issue:
>   1) Dead lock as described here:
>      http://dpdk.org/ml/archives/dev/2018-April/099806.html
>   2) SIGSEGV issue when starting a testpmd in VM.
> 
> Considering below commit changes to use dynamic memory instead of
> stack for memory barrier, we doubt it's caused by use-after-free.
> 
> Fixes: 3d09a6e26d8b ("eal: fix threads block on barrier")
> 
> Reported-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> Reported-by: Lei Yao <lei.a.yao@intel.com>
> Suggested-by: Stephen Hemminger <stephen@networkplumber.org>
> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> Suggested-by: Olivier Matz <olivier.matz@6wind.com>

Reviewed-by: Olivier Matz <olivier.matz@6wind.com>

Thank you for fixing my stuff :)

Re: [dpdk-dev] [PATCH v3] eal: fix use-after-free issue on thread creation

[Thomas Monjalon]

02/05/2018 16:56, Olivier Matz:
> On Wed, May 02, 2018 at 01:52:14PM +0000, Jianfeng Tan wrote:
> > After below commit, we encounter some strange issue:
> >   1) Dead lock as described here:
> >      http://dpdk.org/ml/archives/dev/2018-April/099806.html
> >   2) SIGSEGV issue when starting a testpmd in VM.
> > 
> > Considering below commit changes to use dynamic memory instead of
> > stack for memory barrier, we doubt it's caused by use-after-free.
> > 
> > Fixes: 3d09a6e26d8b ("eal: fix threads block on barrier")
> > 
> > Reported-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> > Reported-by: Lei Yao <lei.a.yao@intel.com>
> > Suggested-by: Stephen Hemminger <stephen@networkplumber.org>
> > Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> > Suggested-by: Olivier Matz <olivier.matz@6wind.com>
> 
> Reviewed-by: Olivier Matz <olivier.matz@6wind.com>
> 
> Thank you for fixing my stuff :)

Applied, thanks