[PATCH 3/4] timer: fix rte_timer_stop_all

[PATCH 3/4] timer: fix rte_timer_stop_all

[Naga Harish K S V]

there is a possibility of deadlock in this api,
as same spinlock is tried to be acquired in nested manner.

This patch removes the acquisition of nested locking.

Signed-off-by: Naga Harish K S V <s.v.naga.harish.k@intel.com>
---
 lib/timer/rte_timer.c | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)

diff --git a/lib/timer/rte_timer.c b/lib/timer/rte_timer.c
index 9994813d0d..cfbc8cb028 100644
--- a/lib/timer/rte_timer.c
+++ b/lib/timer/rte_timer.c
@@ -987,21 +987,16 @@ rte_timer_stop_all(uint32_t timer_data_id, unsigned int *walk_lcores,
 		walk_lcore = walk_lcores[i];
 		priv_timer = &timer_data->priv_timer[walk_lcore];
 
-		rte_spinlock_lock(&priv_timer->list_lock);
-
 		for (tim = priv_timer->pending_head.sl_next[0];
 		     tim != NULL;
 		     tim = next_tim) {
 			next_tim = tim->sl_next[0];
 
-			/* Call timer_stop with lock held */
-			__rte_timer_stop(tim, 1, timer_data);
+			__rte_timer_stop(tim, 0, timer_data);
 
 			if (f)
 				f(tim, f_arg);
 		}
-
-		rte_spinlock_unlock(&priv_timer->list_lock);
 	}
 
 	return 0;
-- 
2.25.1

Re: [PATCH 3/4] timer: fix rte_timer_stop_all

[Stephen Hemminger]

On Wed,  3 Aug 2022 11:26:51 -0500
Naga Harish K S V <s.v.naga.harish.k@intel.com> wrote:

> there is a possibility of deadlock in this api,
> as same spinlock is tried to be acquired in nested manner.
> 
> This patch removes the acquisition of nested locking.
> 
> Signed-off-by: Naga Harish K S V <s.v.naga.harish.k@intel.com>

The wording in this commit message is a little confusing, what is
the exact lock conflict? 

After your patch, there are no longer any callers for __rte_timer_stop()
with the local_is_locked flag. Please resubmit and remove all the
the local_is_locked from __rte_timer_stop().

It looks like the test suite for rte_timer is not exercising all
the exposed API's. That's a problem

[PATCH v2 3/4] timer: fix function to stop all timers

[Naga Harish K S V]

There is a possibility of deadlock in this API,
as same spinlock is tried to be acquired in nested manner.

In timer_del function, if the previous owner and current owner lcore
are different, the lock is tried to be acquired even though the same
lock is already acquired by the caller of timer_del function.

This patch removes the acquisition of nested locking.

Fixes: 821c51267bcd63a ("timer: add function to stop all timers in a list")
Cc: stable@dpdk.org

Signed-off-by: Naga Harish K S V <s.v.naga.harish.k@intel.com>
---
 lib/timer/rte_timer.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/lib/timer/rte_timer.c b/lib/timer/rte_timer.c
index 9994813d0d..85d67573eb 100644
--- a/lib/timer/rte_timer.c
+++ b/lib/timer/rte_timer.c
@@ -580,7 +580,7 @@ rte_timer_reset_sync(struct rte_timer *tim, uint64_t ticks,
 }
 
 static int
-__rte_timer_stop(struct rte_timer *tim, int local_is_locked,
+__rte_timer_stop(struct rte_timer *tim,
 		 struct rte_timer_data *timer_data)
 {
 	union rte_timer_status prev_status, status;
@@ -602,7 +602,7 @@ __rte_timer_stop(struct rte_timer *tim, int local_is_locked,
 
 	/* remove it from list */
 	if (prev_status.state == RTE_TIMER_PENDING) {
-		timer_del(tim, prev_status, local_is_locked, priv_timer);
+		timer_del(tim, prev_status, 0, priv_timer);
 		__TIMER_STAT_ADD(priv_timer, pending, -1);
 	}
 
@@ -631,7 +631,7 @@ rte_timer_alt_stop(uint32_t timer_data_id, struct rte_timer *tim)
 
 	TIMER_DATA_VALID_GET_OR_ERR_RET(timer_data_id, timer_data, -EINVAL);
 
-	return __rte_timer_stop(tim, 0, timer_data);
+	return __rte_timer_stop(tim, timer_data);
 }
 
 /* loop until rte_timer_stop() succeed */
@@ -987,21 +987,16 @@ rte_timer_stop_all(uint32_t timer_data_id, unsigned int *walk_lcores,
 		walk_lcore = walk_lcores[i];
 		priv_timer = &timer_data->priv_timer[walk_lcore];
 
-		rte_spinlock_lock(&priv_timer->list_lock);
-
 		for (tim = priv_timer->pending_head.sl_next[0];
 		     tim != NULL;
 		     tim = next_tim) {
 			next_tim = tim->sl_next[0];
 
-			/* Call timer_stop with lock held */
-			__rte_timer_stop(tim, 1, timer_data);
+			__rte_timer_stop(tim, timer_data);
 
 			if (f)
 				f(tim, f_arg);
 		}
-
-		rte_spinlock_unlock(&priv_timer->list_lock);
 	}
 
 	return 0;
-- 
2.25.1

RE: [PATCH 3/4] timer: fix rte_timer_stop_all

[Naga Harish K, S V]

Hi,
  V2 of the patch is submitted with suggested changes in __rte_timer_stop() function.

-Harish

> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Wednesday, August 3, 2022 11:19 PM
> To: Naga Harish K, S V <s.v.naga.harish.k@intel.com>
> Cc: Carrillo, Erik G <erik.g.carrillo@intel.com>; dev@dpdk.org
> Subject: Re: [PATCH 3/4] timer: fix rte_timer_stop_all
> 
> On Wed,  3 Aug 2022 11:26:51 -0500
> Naga Harish K S V <s.v.naga.harish.k@intel.com> wrote:
> 
> > there is a possibility of deadlock in this api, as same spinlock is
> > tried to be acquired in nested manner.
> >
> > This patch removes the acquisition of nested locking.
> >
> > Signed-off-by: Naga Harish K S V <s.v.naga.harish.k@intel.com>
> 
> The wording in this commit message is a little confusing, what is the exact
> lock conflict?
> 
> After your patch, there are no longer any callers for c with
> the local_is_locked flag. Please resubmit and remove all the the
> local_is_locked from __rte_timer_stop().
> 
> It looks like the test suite for rte_timer is not exercising all the exposed API's.
> That's a problem

RE: [PATCH v2 3/4] timer: fix function to stop all timers

[Carrillo, Erik G]

Hi Harish,

> -----Original Message-----
> From: Naga Harish K, S V <s.v.naga.harish.k@intel.com>
> Sent: Wednesday, August 10, 2022 2:10 AM
> To: Carrillo, Erik G <erik.g.carrillo@intel.com>
> Cc: dev@dpdk.org; stable@dpdk.org
> Subject: [PATCH v2 3/4] timer: fix function to stop all timers
> 
> There is a possibility of deadlock in this API, as same spinlock is tried to be
> acquired in nested manner.
> 
> In timer_del function, if the previous owner and current owner lcore are

It might be clearer to say something like:

 "If the lcore that is stopping the timer is different from the lcore that owns the timer, the timer list lock is acquired in timer_del(), even if local_is_locked is true.  Because the same lock was already acquired in rte_timer_stop_all(), the thread will hang."
  
Thanks,
Erik

> different, the lock is tried to be acquired even though the same lock is
> already acquired by the caller of timer_del function.
> 
> This patch removes the acquisition of nested locking.
> 
> Fixes: 821c51267bcd63a ("timer: add function to stop all timers in a list")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Naga Harish K S V <s.v.naga.harish.k@intel.com>
> ---

Re: [PATCH v2 3/4] timer: fix function to stop all timers

[Stephen Hemminger]

On Wed, 10 Aug 2022 19:29:36 +0000
"Carrillo, Erik G" <erik.g.carrillo@intel.com> wrote:

> Hi Harish,
> 
> > -----Original Message-----
> > From: Naga Harish K, S V <s.v.naga.harish.k@intel.com>
> > Sent: Wednesday, August 10, 2022 2:10 AM
> > To: Carrillo, Erik G <erik.g.carrillo@intel.com>
> > Cc: dev@dpdk.org; stable@dpdk.org
> > Subject: [PATCH v2 3/4] timer: fix function to stop all timers
> > 
> > There is a possibility of deadlock in this API, as same spinlock is tried to be
> > acquired in nested manner.
> > 
> > In timer_del function, if the previous owner and current owner lcore are  
> 
> It might be clearer to say something like:
> 
>  "If the lcore that is stopping the timer is different from the lcore that owns the timer, the timer list lock is acquired in timer_del(), even if local_is_locked is true.  Because the same lock was already acquired in rte_timer_stop_all(), the thread will hang."
>   

Yes, the timer owner flag acts like a lock and this is AB BA deadlock

[PATCH v3 3/4] timer: fix function to stop all timers

[Naga Harish K S V]

There is a possibility of deadlock in this API,
as same spinlock is tried to be acquired in nested manner.

If the lcore that is stopping the timer is different from the lcore
that owns the timer, the timer list lock is acquired in timer_del(),
even if local_is_locked is true. Because the same lock was already
acquired in rte_timer_stop_all(), the thread will hang.

This patch removes the acquisition of nested lock.

Fixes: 821c51267bcd63a ("timer: add function to stop all timers in a list")
Cc: stable@dpdk.org

Signed-off-by: Naga Harish K S V <s.v.naga.harish.k@intel.com>
---
 lib/timer/rte_timer.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/lib/timer/rte_timer.c b/lib/timer/rte_timer.c
index 9994813d0d..85d67573eb 100644
--- a/lib/timer/rte_timer.c
+++ b/lib/timer/rte_timer.c
@@ -580,7 +580,7 @@ rte_timer_reset_sync(struct rte_timer *tim, uint64_t ticks,
 }
 
 static int
-__rte_timer_stop(struct rte_timer *tim, int local_is_locked,
+__rte_timer_stop(struct rte_timer *tim,
 		 struct rte_timer_data *timer_data)
 {
 	union rte_timer_status prev_status, status;
@@ -602,7 +602,7 @@ __rte_timer_stop(struct rte_timer *tim, int local_is_locked,
 
 	/* remove it from list */
 	if (prev_status.state == RTE_TIMER_PENDING) {
-		timer_del(tim, prev_status, local_is_locked, priv_timer);
+		timer_del(tim, prev_status, 0, priv_timer);
 		__TIMER_STAT_ADD(priv_timer, pending, -1);
 	}
 
@@ -631,7 +631,7 @@ rte_timer_alt_stop(uint32_t timer_data_id, struct rte_timer *tim)
 
 	TIMER_DATA_VALID_GET_OR_ERR_RET(timer_data_id, timer_data, -EINVAL);
 
-	return __rte_timer_stop(tim, 0, timer_data);
+	return __rte_timer_stop(tim, timer_data);
 }
 
 /* loop until rte_timer_stop() succeed */
@@ -987,21 +987,16 @@ rte_timer_stop_all(uint32_t timer_data_id, unsigned int *walk_lcores,
 		walk_lcore = walk_lcores[i];
 		priv_timer = &timer_data->priv_timer[walk_lcore];
 
-		rte_spinlock_lock(&priv_timer->list_lock);
-
 		for (tim = priv_timer->pending_head.sl_next[0];
 		     tim != NULL;
 		     tim = next_tim) {
 			next_tim = tim->sl_next[0];
 
-			/* Call timer_stop with lock held */
-			__rte_timer_stop(tim, 1, timer_data);
+			__rte_timer_stop(tim, timer_data);
 
 			if (f)
 				f(tim, f_arg);
 		}
-
-		rte_spinlock_unlock(&priv_timer->list_lock);
 	}
 
 	return 0;
-- 
2.25.1

RE: [PATCH v2 3/4] timer: fix function to stop all timers

[Naga Harish K, S V]

Hi Gabe,

> -----Original Message-----
> From: Carrillo, Erik G <erik.g.carrillo@intel.com>
> Sent: Thursday, August 11, 2022 1:00 AM
> To: Naga Harish K, S V <s.v.naga.harish.k@intel.com>
> Cc: dev@dpdk.org; stable@dpdk.org
> Subject: RE: [PATCH v2 3/4] timer: fix function to stop all timers
> 
> Hi Harish,
> 
> > -----Original Message-----
> > From: Naga Harish K, S V <s.v.naga.harish.k@intel.com>
> > Sent: Wednesday, August 10, 2022 2:10 AM
> > To: Carrillo, Erik G <erik.g.carrillo@intel.com>
> > Cc: dev@dpdk.org; stable@dpdk.org
> > Subject: [PATCH v2 3/4] timer: fix function to stop all timers
> >
> > There is a possibility of deadlock in this API, as same spinlock is
> > tried to be acquired in nested manner.
> >
> > In timer_del function, if the previous owner and current owner lcore
> > are
> 
> It might be clearer to say something like:
> 
>  "If the lcore that is stopping the timer is different from the lcore that owns
> the timer, the timer list lock is acquired in timer_del(), even if local_is_locked
> is true.  Because the same lock was already acquired in rte_timer_stop_all(),
> the thread will hang."
> 

Incorporated the commit message in v3 version of the patch

> Thanks,
> Erik
> 
> > different, the lock is tried to be acquired even though the same lock
> > is already acquired by the caller of timer_del function.
> >
> > This patch removes the acquisition of nested locking.
> >
> > Fixes: 821c51267bcd63a ("timer: add function to stop all timers in a
> > list")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Naga Harish K S V <s.v.naga.harish.k@intel.com>
> > ---

[PATCH v4 3/4] timer: fix function to stop all timers

[Naga Harish K S V]

There is a possibility of deadlock in this API,
as same spinlock is tried to be acquired in nested manner.

If the lcore that is stopping the timer is different from the lcore
that owns the timer, the timer list lock is acquired in timer_del(),
even if local_is_locked is true. Because the same lock was already
acquired in rte_timer_stop_all(), the thread will hang.

This patch removes the acquisition of nested lock.

Fixes: 821c51267bcd63a ("timer: add function to stop all timers in a list")
Cc: stable@dpdk.org

Signed-off-by: Naga Harish K S V <s.v.naga.harish.k@intel.com>
---
 lib/timer/rte_timer.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/lib/timer/rte_timer.c b/lib/timer/rte_timer.c
index 9994813d0d..85d67573eb 100644
--- a/lib/timer/rte_timer.c
+++ b/lib/timer/rte_timer.c
@@ -580,7 +580,7 @@ rte_timer_reset_sync(struct rte_timer *tim, uint64_t ticks,
 }
 
 static int
-__rte_timer_stop(struct rte_timer *tim, int local_is_locked,
+__rte_timer_stop(struct rte_timer *tim,
 		 struct rte_timer_data *timer_data)
 {
 	union rte_timer_status prev_status, status;
@@ -602,7 +602,7 @@ __rte_timer_stop(struct rte_timer *tim, int local_is_locked,
 
 	/* remove it from list */
 	if (prev_status.state == RTE_TIMER_PENDING) {
-		timer_del(tim, prev_status, local_is_locked, priv_timer);
+		timer_del(tim, prev_status, 0, priv_timer);
 		__TIMER_STAT_ADD(priv_timer, pending, -1);
 	}
 
@@ -631,7 +631,7 @@ rte_timer_alt_stop(uint32_t timer_data_id, struct rte_timer *tim)
 
 	TIMER_DATA_VALID_GET_OR_ERR_RET(timer_data_id, timer_data, -EINVAL);
 
-	return __rte_timer_stop(tim, 0, timer_data);
+	return __rte_timer_stop(tim, timer_data);
 }
 
 /* loop until rte_timer_stop() succeed */
@@ -987,21 +987,16 @@ rte_timer_stop_all(uint32_t timer_data_id, unsigned int *walk_lcores,
 		walk_lcore = walk_lcores[i];
 		priv_timer = &timer_data->priv_timer[walk_lcore];
 
-		rte_spinlock_lock(&priv_timer->list_lock);
-
 		for (tim = priv_timer->pending_head.sl_next[0];
 		     tim != NULL;
 		     tim = next_tim) {
 			next_tim = tim->sl_next[0];
 
-			/* Call timer_stop with lock held */
-			__rte_timer_stop(tim, 1, timer_data);
+			__rte_timer_stop(tim, timer_data);
 
 			if (f)
 				f(tim, f_arg);
 		}
-
-		rte_spinlock_unlock(&priv_timer->list_lock);
 	}
 
 	return 0;
-- 
2.25.1

RE: [PATCH v4 3/4] timer: fix function to stop all timers

[Carrillo, Erik G]

> -----Original Message-----
> From: Naga Harish K, S V <s.v.naga.harish.k@intel.com>
> Sent: Friday, August 12, 2022 11:08 AM
> To: Carrillo, Erik G <erik.g.carrillo@intel.com>
> Cc: dev@dpdk.org; stable@dpdk.org
> Subject: [PATCH v4 3/4] timer: fix function to stop all timers
> 
> There is a possibility of deadlock in this API, as same spinlock is tried to be
> acquired in nested manner.
> 
> If the lcore that is stopping the timer is different from the lcore that owns the
> timer, the timer list lock is acquired in timer_del(), even if local_is_locked is
> true. Because the same lock was already acquired in rte_timer_stop_all(),
> the thread will hang.
> 
> This patch removes the acquisition of nested lock.
> 
> Fixes: 821c51267bcd63a ("timer: add function to stop all timers in a list")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Naga Harish K S V <s.v.naga.harish.k@intel.com>
Acked-by: Erik Gabriel Carrillo <erik.g.carrillo@intel.com>