[dpdk-dev] [PATCH] bus/vdev: reduce scope of device list lock

[dpdk-dev] [PATCH] bus/vdev: reduce scope of device list lock

[Thomas Monjalon]

The lock vdev_device_list_lock was taken before calling
"remove" function for the device.
So it prevents to remove sub-devices (as in failsafe) inside
its own "remove" function, because of a deadlock.

The lock is now only protecting the device list inside
the bus driver.

Fixes: 35f462839b69 ("bus/vdev: add lock on device list")

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
 drivers/bus/vdev/vdev.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 099b9ff85..2fbc86806 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -293,25 +293,23 @@ rte_vdev_uninit(const char *name)
 	if (name == NULL)
 		return -EINVAL;
 
-	rte_spinlock_lock(&vdev_device_list_lock);
-
 	dev = find_vdev(name);
 	if (!dev) {
 		ret = -ENOENT;
-		goto unlock;
+		return ret;
 	}
 
 	ret = vdev_remove_driver(dev);
 	if (ret)
-		goto unlock;
+		return ret;
 
+	rte_spinlock_lock(&vdev_device_list_lock);
 	TAILQ_REMOVE(&vdev_device_list, dev, next);
 	devargs = dev->device.devargs;
 	rte_devargs_remove(devargs->bus->name, devargs->name);
 	free(dev);
-
-unlock:
 	rte_spinlock_unlock(&vdev_device_list_lock);
+
 	return ret;
 }
 
-- 
2.16.2

[dpdk-dev] [PATCH v2] bus/vdev: fix scope of device list lock

[Thomas Monjalon]

The lock vdev_device_list_lock was taken before calling
"remove" function for the device.
So it prevents to remove sub-devices (as in failsafe) inside
its own "remove" function, because of a deadlock.

The lock is now only protecting the device list inside
the bus driver.

Fixes: 35f462839b69 ("bus/vdev: add lock on device list")

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
v2: reduce scope more by moving unlock
---
 drivers/bus/vdev/vdev.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 099b9ff85..470cff46c 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -293,25 +293,24 @@ rte_vdev_uninit(const char *name)
 	if (name == NULL)
 		return -EINVAL;
 
-	rte_spinlock_lock(&vdev_device_list_lock);
-
 	dev = find_vdev(name);
 	if (!dev) {
 		ret = -ENOENT;
-		goto unlock;
+		return ret;
 	}
 
 	ret = vdev_remove_driver(dev);
 	if (ret)
-		goto unlock;
+		return ret;
 
+	rte_spinlock_lock(&vdev_device_list_lock);
 	TAILQ_REMOVE(&vdev_device_list, dev, next);
+	rte_spinlock_unlock(&vdev_device_list_lock);
+
 	devargs = dev->device.devargs;
 	rte_devargs_remove(devargs->bus->name, devargs->name);
 	free(dev);
 
-unlock:
-	rte_spinlock_unlock(&vdev_device_list_lock);
 	return ret;
 }
 
-- 
2.16.2

Re: [dpdk-dev] [PATCH v2] bus/vdev: fix scope of device list lock

[Matan Azrad]

From: Thomas Monjalon
> The lock vdev_device_list_lock was taken before calling "remove" function for
> the device.
> So it prevents to remove sub-devices (as in failsafe) inside its own "remove"
> function, because of a deadlock.
> 
> The lock is now only protecting the device list inside the bus driver.
> 
> Fixes: 35f462839b69 ("bus/vdev: add lock on device list")
> 
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Tested-by: Matan Azrad <matan@mellanox.com>

Re: [dpdk-dev] [PATCH] bus/vdev: reduce scope of device list lock

[Burakov, Anatoly]

On 21-May-18 5:11 PM, Thomas Monjalon wrote:
> The lock vdev_device_list_lock was taken before calling
> "remove" function for the device.
> So it prevents to remove sub-devices (as in failsafe) inside
> its own "remove" function, because of a deadlock.
> 
> The lock is now only protecting the device list inside
> the bus driver.
> 
> Fixes: 35f462839b69 ("bus/vdev: add lock on device list")
> 
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
>   drivers/bus/vdev/vdev.c | 10 ++++------
>   1 file changed, 4 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
> index 099b9ff85..2fbc86806 100644
> --- a/drivers/bus/vdev/vdev.c
> +++ b/drivers/bus/vdev/vdev.c
> @@ -293,25 +293,23 @@ rte_vdev_uninit(const char *name)
>   	if (name == NULL)
>   		return -EINVAL;
>   
> -	rte_spinlock_lock(&vdev_device_list_lock);
> -
>   	dev = find_vdev(name);
>   	if (!dev) {
>   		ret = -ENOENT;
> -		goto unlock;
> +		return ret;
>   	}

Without that lock, all of this would be racy - find_dev would iterate a 
tailq that might change under its feet, and tailq_remove may be called 
with a pointer that has already been removed.

How about changing the lock to a recursive lock? Failsafe would be 
removing devices from within the same thread, correct?

>   
>   	ret = vdev_remove_driver(dev);
>   	if (ret)
> -		goto unlock;
> +		return ret;
>   
> +	rte_spinlock_lock(&vdev_device_list_lock);
>   	TAILQ_REMOVE(&vdev_device_list, dev, next);
>   	devargs = dev->device.devargs;
>   	rte_devargs_remove(devargs->bus->name, devargs->name);
>   	free(dev);
> -
> -unlock:
>   	rte_spinlock_unlock(&vdev_device_list_lock);
> +
>   	return ret;
>   }
>   
> 


-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v2] bus/vdev: fix scope of device list lock

[Gaëtan Rivet]

On Mon, May 21, 2018 at 05:28:52PM +0000, Matan Azrad wrote:
> 
> 
> From: Thomas Monjalon
> > The lock vdev_device_list_lock was taken before calling "remove" function for
> > the device.
> > So it prevents to remove sub-devices (as in failsafe) inside its own "remove"
> > function, because of a deadlock.
> > 
> > The lock is now only protecting the device list inside the bus driver.
> > 
> > Fixes: 35f462839b69 ("bus/vdev: add lock on device list")
> > 
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> Tested-by: Matan Azrad <matan@mellanox.com>

If these locks were necessary, they would be missing as well for
rte_devargs. Jianfeng inquired about it, I think it should be followed
upon.

Restricting the scope of the lock here could maybe re-introduce the bug
that motivated their introduction in the first place, as the
devargs_remove() is not in the critical section anymore.

However, this is an rte_devargs issue, not a vdev bus one, so
the fix makes sense and I'd like to have it ASAP for failsafe.
Without a vdev bus maintainer left:

Acked-by: Gaetan Rivet <gaetan.rivet@6wind.com>

-- 
Gaëtan Rivet
6WIND

Re: [dpdk-dev] [PATCH] bus/vdev: reduce scope of device list lock

[Thomas Monjalon]

22/05/2018 11:05, Burakov, Anatoly:
> On 21-May-18 5:11 PM, Thomas Monjalon wrote:
> > The lock vdev_device_list_lock was taken before calling
> > "remove" function for the device.
> > So it prevents to remove sub-devices (as in failsafe) inside
> > its own "remove" function, because of a deadlock.
> > 
> > The lock is now only protecting the device list inside
> > the bus driver.
> > 
> > Fixes: 35f462839b69 ("bus/vdev: add lock on device list")
> > 
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> 
> Without that lock, all of this would be racy - find_dev would iterate a 
> tailq that might change under its feet, and tailq_remove may be called 
> with a pointer that has already been removed.
> 
> How about changing the lock to a recursive lock? Failsafe would be 
> removing devices from within the same thread, correct?

Yes it could work.
I will give it a try.

[dpdk-dev] [PATCH v3] bus/vdev: replace device list lock by a recursive one

[Thomas Monjalon]

A device like failsafe can manage sub-devices.
When removing such device, it removes its sub-devices
and try to take the same vdev_device_list_lock.
It was causing a deadlock because the lock was not recursive.

Fixes: 35f462839b69 ("bus/vdev: add lock on device list")

Suggested-by: Anatoly Burakov <anatoly.burakov@intel.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
v3: try recursive lock
WARNING: not yet tested!
---
 drivers/bus/vdev/vdev.c | 26 ++++++++++++++------------
 1 file changed, 14 insertions(+), 12 deletions(-)

diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 099b9ff85..6139dd551 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -36,7 +36,9 @@ TAILQ_HEAD(vdev_device_list, rte_vdev_device);
 
 static struct vdev_device_list vdev_device_list =
 	TAILQ_HEAD_INITIALIZER(vdev_device_list);
-static rte_spinlock_t vdev_device_list_lock = RTE_SPINLOCK_INITIALIZER;
+/* The lock needs to be recursive because a vdev can manage another vdev. */
+static rte_spinlock_recursive_t vdev_device_list_lock =
+	RTE_SPINLOCK_RECURSIVE_INITIALIZER;
 
 struct vdev_driver_list vdev_driver_list =
 	TAILQ_HEAD_INITIALIZER(vdev_driver_list);
@@ -249,7 +251,7 @@ rte_vdev_init(const char *name, const char *args)
 	struct rte_devargs *devargs;
 	int ret;
 
-	rte_spinlock_lock(&vdev_device_list_lock);
+	rte_spinlock_recursive_lock(&vdev_device_list_lock);
 	ret = insert_vdev(name, args, &dev);
 	if (ret == 0) {
 		ret = vdev_probe_all_drivers(dev);
@@ -263,7 +265,7 @@ rte_vdev_init(const char *name, const char *args)
 			free(dev);
 		}
 	}
-	rte_spinlock_unlock(&vdev_device_list_lock);
+	rte_spinlock_recursive_unlock(&vdev_device_list_lock);
 	return ret;
 }
 
@@ -293,7 +295,7 @@ rte_vdev_uninit(const char *name)
 	if (name == NULL)
 		return -EINVAL;
 
-	rte_spinlock_lock(&vdev_device_list_lock);
+	rte_spinlock_recursive_lock(&vdev_device_list_lock);
 
 	dev = find_vdev(name);
 	if (!dev) {
@@ -311,7 +313,7 @@ rte_vdev_uninit(const char *name)
 	free(dev);
 
 unlock:
-	rte_spinlock_unlock(&vdev_device_list_lock);
+	rte_spinlock_recursive_unlock(&vdev_device_list_lock);
 	return ret;
 }
 
@@ -355,7 +357,7 @@ vdev_action(const struct rte_mp_msg *mp_msg, const void *peer)
 		ou->num = 1;
 		num = 0;
 
-		rte_spinlock_lock(&vdev_device_list_lock);
+		rte_spinlock_recursive_lock(&vdev_device_list_lock);
 		TAILQ_FOREACH(dev, &vdev_device_list, next) {
 			devname = rte_vdev_device_name(dev);
 			if (strlen(devname) == 0) {
@@ -369,7 +371,7 @@ vdev_action(const struct rte_mp_msg *mp_msg, const void *peer)
 					 devname, strerror(rte_errno));
 			num++;
 		}
-		rte_spinlock_unlock(&vdev_device_list_lock);
+		rte_spinlock_recursive_unlock(&vdev_device_list_lock);
 
 		ou->type = VDEV_SCAN_REP;
 		ou->num = num;
@@ -445,10 +447,10 @@ vdev_scan(void)
 		if (!dev)
 			return -1;
 
-		rte_spinlock_lock(&vdev_device_list_lock);
+		rte_spinlock_recursive_lock(&vdev_device_list_lock);
 
 		if (find_vdev(devargs->name)) {
-			rte_spinlock_unlock(&vdev_device_list_lock);
+			rte_spinlock_recursive_unlock(&vdev_device_list_lock);
 			free(dev);
 			continue;
 		}
@@ -459,7 +461,7 @@ vdev_scan(void)
 
 		TAILQ_INSERT_TAIL(&vdev_device_list, dev, next);
 
-		rte_spinlock_unlock(&vdev_device_list_lock);
+		rte_spinlock_recursive_unlock(&vdev_device_list_lock);
 	}
 
 	return 0;
@@ -498,7 +500,7 @@ vdev_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
 	const struct rte_vdev_device *vstart;
 	struct rte_vdev_device *dev;
 
-	rte_spinlock_lock(&vdev_device_list_lock);
+	rte_spinlock_recursive_lock(&vdev_device_list_lock);
 	if (start != NULL) {
 		vstart = RTE_DEV_TO_VDEV_CONST(start);
 		dev = TAILQ_NEXT(vstart, next);
@@ -510,7 +512,7 @@ vdev_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
 			break;
 		dev = TAILQ_NEXT(dev, next);
 	}
-	rte_spinlock_unlock(&vdev_device_list_lock);
+	rte_spinlock_recursive_unlock(&vdev_device_list_lock);
 
 	return dev ? &dev->device : NULL;
 }
-- 
2.16.2

Re: [dpdk-dev] [PATCH v3] bus/vdev: replace device list lock by a recursive one

[Matan Azrad]

From: Thomas Monjalon
> A device like failsafe can manage sub-devices.
> When removing such device, it removes its sub-devices and try to take the
> same vdev_device_list_lock.
> It was causing a deadlock because the lock was not recursive.
> 
> Fixes: 35f462839b69 ("bus/vdev: add lock on device list")
> 
> Suggested-by: Anatoly Burakov <anatoly.burakov@intel.com>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Tested-by: Matan Azrad <matan@mellanox.com>
> ---
> v3: try recursive lock
> WARNING: not yet tested!
> ---
>  drivers/bus/vdev/vdev.c | 26 ++++++++++++++------------
>  1 file changed, 14 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c index
> 099b9ff85..6139dd551 100644
> --- a/drivers/bus/vdev/vdev.c
> +++ b/drivers/bus/vdev/vdev.c
> @@ -36,7 +36,9 @@ TAILQ_HEAD(vdev_device_list, rte_vdev_device);
> 
>  static struct vdev_device_list vdev_device_list =
>  	TAILQ_HEAD_INITIALIZER(vdev_device_list);
> -static rte_spinlock_t vdev_device_list_lock = RTE_SPINLOCK_INITIALIZER;
> +/* The lock needs to be recursive because a vdev can manage another
> +vdev. */ static rte_spinlock_recursive_t vdev_device_list_lock =
> +	RTE_SPINLOCK_RECURSIVE_INITIALIZER;
> 
>  struct vdev_driver_list vdev_driver_list =
>  	TAILQ_HEAD_INITIALIZER(vdev_driver_list);
> @@ -249,7 +251,7 @@ rte_vdev_init(const char *name, const char *args)
>  	struct rte_devargs *devargs;
>  	int ret;
> 
> -	rte_spinlock_lock(&vdev_device_list_lock);
> +	rte_spinlock_recursive_lock(&vdev_device_list_lock);
>  	ret = insert_vdev(name, args, &dev);
>  	if (ret == 0) {
>  		ret = vdev_probe_all_drivers(dev);
> @@ -263,7 +265,7 @@ rte_vdev_init(const char *name, const char *args)
>  			free(dev);
>  		}
>  	}
> -	rte_spinlock_unlock(&vdev_device_list_lock);
> +	rte_spinlock_recursive_unlock(&vdev_device_list_lock);
>  	return ret;
>  }
> 
> @@ -293,7 +295,7 @@ rte_vdev_uninit(const char *name)
>  	if (name == NULL)
>  		return -EINVAL;
> 
> -	rte_spinlock_lock(&vdev_device_list_lock);
> +	rte_spinlock_recursive_lock(&vdev_device_list_lock);
> 
>  	dev = find_vdev(name);
>  	if (!dev) {
> @@ -311,7 +313,7 @@ rte_vdev_uninit(const char *name)
>  	free(dev);
> 
>  unlock:
> -	rte_spinlock_unlock(&vdev_device_list_lock);
> +	rte_spinlock_recursive_unlock(&vdev_device_list_lock);
>  	return ret;
>  }
> 
> @@ -355,7 +357,7 @@ vdev_action(const struct rte_mp_msg *mp_msg, const
> void *peer)
>  		ou->num = 1;
>  		num = 0;
> 
> -		rte_spinlock_lock(&vdev_device_list_lock);
> +		rte_spinlock_recursive_lock(&vdev_device_list_lock);
>  		TAILQ_FOREACH(dev, &vdev_device_list, next) {
>  			devname = rte_vdev_device_name(dev);
>  			if (strlen(devname) == 0) {
> @@ -369,7 +371,7 @@ vdev_action(const struct rte_mp_msg *mp_msg, const
> void *peer)
>  					 devname, strerror(rte_errno));
>  			num++;
>  		}
> -		rte_spinlock_unlock(&vdev_device_list_lock);
> +		rte_spinlock_recursive_unlock(&vdev_device_list_lock);
> 
>  		ou->type = VDEV_SCAN_REP;
>  		ou->num = num;
> @@ -445,10 +447,10 @@ vdev_scan(void)
>  		if (!dev)
>  			return -1;
> 
> -		rte_spinlock_lock(&vdev_device_list_lock);
> +		rte_spinlock_recursive_lock(&vdev_device_list_lock);
> 
>  		if (find_vdev(devargs->name)) {
> -			rte_spinlock_unlock(&vdev_device_list_lock);
> +
> 	rte_spinlock_recursive_unlock(&vdev_device_list_lock);
>  			free(dev);
>  			continue;
>  		}
> @@ -459,7 +461,7 @@ vdev_scan(void)
> 
>  		TAILQ_INSERT_TAIL(&vdev_device_list, dev, next);
> 
> -		rte_spinlock_unlock(&vdev_device_list_lock);
> +		rte_spinlock_recursive_unlock(&vdev_device_list_lock);
>  	}
> 
>  	return 0;
> @@ -498,7 +500,7 @@ vdev_find_device(const struct rte_device *start,
> rte_dev_cmp_t cmp,
>  	const struct rte_vdev_device *vstart;
>  	struct rte_vdev_device *dev;
> 
> -	rte_spinlock_lock(&vdev_device_list_lock);
> +	rte_spinlock_recursive_lock(&vdev_device_list_lock);
>  	if (start != NULL) {
>  		vstart = RTE_DEV_TO_VDEV_CONST(start);
>  		dev = TAILQ_NEXT(vstart, next);
> @@ -510,7 +512,7 @@ vdev_find_device(const struct rte_device *start,
> rte_dev_cmp_t cmp,
>  			break;
>  		dev = TAILQ_NEXT(dev, next);
>  	}
> -	rte_spinlock_unlock(&vdev_device_list_lock);
> +	rte_spinlock_recursive_unlock(&vdev_device_list_lock);
> 
>  	return dev ? &dev->device : NULL;
>  }
> --
> 2.16.2

Re: [dpdk-dev] [PATCH v3] bus/vdev: replace device list lock by a recursive one

[Burakov, Anatoly]

On 22-May-18 12:37 PM, Thomas Monjalon wrote:
> A device like failsafe can manage sub-devices.
> When removing such device, it removes its sub-devices
> and try to take the same vdev_device_list_lock.
> It was causing a deadlock because the lock was not recursive.
> 
> Fixes: 35f462839b69 ("bus/vdev: add lock on device list")
> 
> Suggested-by: Anatoly Burakov <anatoly.burakov@intel.com>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
> v3: try recursive lock
> WARNING: not yet tested!
> ---

LGTM

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v3] bus/vdev: replace device list lock by a recursive one

[Thomas Monjalon]

22/05/2018 15:34, Burakov, Anatoly:
> On 22-May-18 12:37 PM, Thomas Monjalon wrote:
> > A device like failsafe can manage sub-devices.
> > When removing such device, it removes its sub-devices
> > and try to take the same vdev_device_list_lock.
> > It was causing a deadlock because the lock was not recursive.
> > 
> > Fixes: 35f462839b69 ("bus/vdev: add lock on device list")
> > 
> > Suggested-by: Anatoly Burakov <anatoly.burakov@intel.com>
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > ---
> > v3: try recursive lock
> > WARNING: not yet tested!
> > ---
> 
> LGTM
> 
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

Tested-by: Matan Azrad <matan@mellanox.com>

Applied