[dpdk-dev] [PATCH] igb_uio: allow multi-process access

[dpdk-dev] [PATCH] igb_uio: allow multi-process access

[Xiao Wang]

In some case, one device are accessed by different processes via
different BARs, so one uio device may be opened by more than one
process, for this case we just need to enable interrupt once, and
pci_clear_master only when the last process closed.

Fixes: 5f6ff30dc507 ("igb_uio: fix interrupt enablement after FLR in VM")

Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
---
 lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
index a3a98c1..c239d98 100644
--- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
+++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
@@ -45,6 +45,7 @@ struct rte_uio_pci_dev {
 	struct uio_info info;
 	struct pci_dev *pdev;
 	enum rte_intr_mode mode;
+	uint32_t ref_cnt;
 };
 
 static char *intr_mode;
@@ -336,6 +337,9 @@ struct rte_uio_pci_dev {
 	struct pci_dev *dev = udev->pdev;
 	int err;
 
+	if (++(udev->ref_cnt) > 1)
+		return 0;
+
 	/* set bus master, which was cleared by the reset function */
 	pci_set_master(dev);
 
@@ -354,6 +358,9 @@ struct rte_uio_pci_dev {
 	struct rte_uio_pci_dev *udev = info->priv;
 	struct pci_dev *dev = udev->pdev;
 
+	if (--(udev->ref_cnt) > 0)
+		return 0;
+
 	/* disable interrupts */
 	igbuio_pci_disable_interrupts(udev);
 
-- 
1.8.3.1

Re: [dpdk-dev] [PATCH] igb_uio: allow multi-process access

[Stephen Hemminger]

On Fri,  8 Dec 2017 17:57:33 -0800
Xiao Wang <xiao.w.wang@intel.com> wrote:

> In some case, one device are accessed by different processes via
> different BARs, so one uio device may be opened by more than one
> process, for this case we just need to enable interrupt once, and
> pci_clear_master only when the last process closed.
> 
> Fixes: 5f6ff30dc507 ("igb_uio: fix interrupt enablement after FLR in VM")


Yes, this makes sense.

> 
> Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
> ---
>  lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
> index a3a98c1..c239d98 100644
> --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
> +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
> @@ -45,6 +45,7 @@ struct rte_uio_pci_dev {
>  	struct uio_info info;
>  	struct pci_dev *pdev;
>  	enum rte_intr_mode mode;
> +	uint32_t ref_cnt;

Simple unsigned reference count is not SMP safe on all architectures.
In kernel it is recommended to use refcount_t and associated API's.
Note: refcount_t was introduced in last 2 years and some DPDK users
still have ancient kernels.

>  };
>  
>  static char *intr_mode;
> @@ -336,6 +337,9 @@ struct rte_uio_pci_dev {
>  	struct pci_dev *dev = udev->pdev;
>  	int err;
>  
> +	if (++(udev->ref_cnt) > 1)
> +		return 0;

Do not use (unnecessary) parenthesis. C precedence order is well defined.


>  	/* set bus master, which was cleared by the reset function */
>  	pci_set_master(dev);
>  
> @@ -354,6 +358,9 @@ struct rte_uio_pci_dev {
>  	struct rte_uio_pci_dev *udev = info->priv;
>  	struct pci_dev *dev = udev->pdev;
>  
> +	if (--(udev->ref_cnt) > 0)
> +		return 0;
> +
>  	/* disable interrupts */
>  	igbuio_pci_disable_interrupts(udev);
>  

Re: [dpdk-dev] [PATCH] igb_uio: allow multi-process access

[Wang, Xiao W]

Hi,

> -----Original Message-----
> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Tuesday, December 12, 2017 9:39 AM
> To: Wang, Xiao W <xiao.w.wang@intel.com>
> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] igb_uio: allow multi-process access
> 
> On Fri,  8 Dec 2017 17:57:33 -0800
> Xiao Wang <xiao.w.wang@intel.com> wrote:
> 
> > In some case, one device are accessed by different processes via
> > different BARs, so one uio device may be opened by more than one
> > process, for this case we just need to enable interrupt once, and
> > pci_clear_master only when the last process closed.
> >
> > Fixes: 5f6ff30dc507 ("igb_uio: fix interrupt enablement after FLR in VM")
> 
> 
> Yes, this makes sense.
> 
> >
> > Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
> > ---
> >  lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> >
> > diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
> b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
> > index a3a98c1..c239d98 100644
> > --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
> > +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
> > @@ -45,6 +45,7 @@ struct rte_uio_pci_dev {
> >  	struct uio_info info;
> >  	struct pci_dev *pdev;
> >  	enum rte_intr_mode mode;
> > +	uint32_t ref_cnt;
> 
> Simple unsigned reference count is not SMP safe on all architectures.
> In kernel it is recommended to use refcount_t and associated API's.
> Note: refcount_t was introduced in last 2 years and some DPDK users
> still have ancient kernels.

I think atomic_t associated API will be enough, without worry about kernel version.

> 
> >  };
> >
> >  static char *intr_mode;
> > @@ -336,6 +337,9 @@ struct rte_uio_pci_dev {
> >  	struct pci_dev *dev = udev->pdev;
> >  	int err;
> >
> > +	if (++(udev->ref_cnt) > 1)
> > +		return 0;
> 
> Do not use (unnecessary) parenthesis. C precedence order is well defined.

Agree. Will change it in v2.

Thanks for your comments,
Xiao

Re: [dpdk-dev] [PATCH v2] igb_uio: allow multi-process access

[Tiwei Bie]

Hi,

On Tue, Dec 19, 2017 at 07:42:20AM -0800, Xiao Wang wrote:
> @@ -336,6 +337,10 @@ struct rte_uio_pci_dev {
>  	struct pci_dev *dev = udev->pdev;
>  	int err;
>  
> +	atomic_inc(&udev->refcnt);
> +	if (atomic_read(&udev->refcnt) > 1)

The "inc and read" should be atomic. Otherwise below
sequence may happen:

P1: atomic_inc(&udev->refcnt);
P2: atomic_inc(&udev->refcnt);
P1: if (atomic_read(&udev->refcnt) > 1)
P2: if (atomic_read(&udev->refcnt) > 1)

> +		return 0;
> +
>  	/* set bus master, which was cleared by the reset function */
>  	pci_set_master(dev);
>  
> @@ -354,6 +359,10 @@ struct rte_uio_pci_dev {
>  	struct rte_uio_pci_dev *udev = info->priv;
>  	struct pci_dev *dev = udev->pdev;
>  
> +	atomic_dec(&udev->refcnt);
> +	if (atomic_read(&udev->refcnt) > 0)
> +		return 0;

Ditto.

Best regards,
Tiwei Bie

> +
>  	/* disable interrupts */
>  	igbuio_pci_disable_interrupts(udev);
>  
> -- 
> 1.8.3.1
> 

Re: [dpdk-dev] [PATCH v2] igb_uio: allow multi-process access

[Wang, Xiao W]

Thanks for pointing it out. Fix it in v3.

BRs,
Xiao

> -----Original Message-----
> From: Bie, Tiwei
> Sent: Tuesday, December 19, 2017 5:05 PM
> To: Wang, Xiao W <xiao.w.wang@intel.com>
> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; dev@dpdk.org;
> stephen@networkplumber.org
> Subject: Re: [dpdk-dev] [PATCH v2] igb_uio: allow multi-process access
> 
> Hi,
> 
> On Tue, Dec 19, 2017 at 07:42:20AM -0800, Xiao Wang wrote:
> > @@ -336,6 +337,10 @@ struct rte_uio_pci_dev {
> >  	struct pci_dev *dev = udev->pdev;
> >  	int err;
> >
> > +	atomic_inc(&udev->refcnt);
> > +	if (atomic_read(&udev->refcnt) > 1)
> 
> The "inc and read" should be atomic. Otherwise below
> sequence may happen:
> 
> P1: atomic_inc(&udev->refcnt);
> P2: atomic_inc(&udev->refcnt);
> P1: if (atomic_read(&udev->refcnt) > 1)
> P2: if (atomic_read(&udev->refcnt) > 1)
> 
> > +		return 0;
> > +
> >  	/* set bus master, which was cleared by the reset function */
> >  	pci_set_master(dev);
> >
> > @@ -354,6 +359,10 @@ struct rte_uio_pci_dev {
> >  	struct rte_uio_pci_dev *udev = info->priv;
> >  	struct pci_dev *dev = udev->pdev;
> >
> > +	atomic_dec(&udev->refcnt);
> > +	if (atomic_read(&udev->refcnt) > 0)
> > +		return 0;
> 
> Ditto.
> 
> Best regards,
> Tiwei Bie
> 
> > +
> >  	/* disable interrupts */
> >  	igbuio_pci_disable_interrupts(udev);
> >
> > --
> > 1.8.3.1
> >

Re: [dpdk-dev] [PATCH v2] igb_uio: allow multi-process access

[Stephen Hemminger]

On Tue, 19 Dec 2017 07:42:20 -0800
Xiao Wang <xiao.w.wang@intel.com> wrote:

> +	atomic_dec(&udev->refcnt);
> +	if (atomic_read(&udev->refcnt) > 0)
> +		return 0;
> +

The point of using atomic functions is to do atomic operations.
You need to use atomic_dec_and_test here.

[dpdk-dev] [PATCH v2] igb_uio: allow multi-process access

[Xiao Wang]

In some case, one device are accessed by different processes via
different BARs, so one uio device may be opened by more than one
process, for this case we just need to enable interrupt once, and
pci_clear_master only when the last process closed.

Fixes: 5f6ff30dc507 ("igb_uio: fix interrupt enablement after FLR in VM")

Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
---
v2:
- Make uio reference counter operation atomic.
---
 lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
index a3a98c1..0c42cc4 100644
--- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
+++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
@@ -45,6 +45,7 @@ struct rte_uio_pci_dev {
 	struct uio_info info;
 	struct pci_dev *pdev;
 	enum rte_intr_mode mode;
+	atomic_t refcnt;
 };
 
 static char *intr_mode;
@@ -336,6 +337,10 @@ struct rte_uio_pci_dev {
 	struct pci_dev *dev = udev->pdev;
 	int err;
 
+	atomic_inc(&udev->refcnt);
+	if (atomic_read(&udev->refcnt) > 1)
+		return 0;
+
 	/* set bus master, which was cleared by the reset function */
 	pci_set_master(dev);
 
@@ -354,6 +359,10 @@ struct rte_uio_pci_dev {
 	struct rte_uio_pci_dev *udev = info->priv;
 	struct pci_dev *dev = udev->pdev;
 
+	atomic_dec(&udev->refcnt);
+	if (atomic_read(&udev->refcnt) > 0)
+		return 0;
+
 	/* disable interrupts */
 	igbuio_pci_disable_interrupts(udev);
 
-- 
1.8.3.1

[dpdk-dev] [PATCH v3] igb_uio: allow multi-process access

[Xiao Wang]

In some case, one device are accessed by different processes via
different BARs, so one uio device may be opened by more than one
process, for this case we just need to enable interrupt once, and
pci_clear_master only when the last process closed.

Fixes: 5f6ff30dc507 ("igb_uio: fix interrupt enablement after FLR in VM")

Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
---
v3:
- Use atomic_inc_return instead of atomic_inc and atomic_read.
- Ditto for atomic_dec.

v2:
- Make uio reference counter operation atomic.
---
 lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
index a3a98c1..8f40c01 100644
--- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
+++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
@@ -45,6 +45,7 @@ struct rte_uio_pci_dev {
 	struct uio_info info;
 	struct pci_dev *pdev;
 	enum rte_intr_mode mode;
+	atomic_t refcnt;
 };
 
 static char *intr_mode;
@@ -336,6 +337,9 @@ struct rte_uio_pci_dev {
 	struct pci_dev *dev = udev->pdev;
 	int err;
 
+	if (atomic_inc_return(&udev->refcnt) > 1)
+		return 0;
+
 	/* set bus master, which was cleared by the reset function */
 	pci_set_master(dev);
 
@@ -354,6 +358,9 @@ struct rte_uio_pci_dev {
 	struct rte_uio_pci_dev *udev = info->priv;
 	struct pci_dev *dev = udev->pdev;
 
+	if (atomic_dec_return(&udev->refcnt) > 0)
+		return 0;
+
 	/* disable interrupts */
 	igbuio_pci_disable_interrupts(udev);
 
-- 
1.8.3.1

Re: [dpdk-dev] [PATCH v3] igb_uio: allow multi-process access

[Tiwei Bie]

On Tue, Dec 19, 2017 at 09:58:13AM -0800, Xiao Wang wrote:
[...]
>  static char *intr_mode;
> @@ -336,6 +337,9 @@ struct rte_uio_pci_dev {
>  	struct pci_dev *dev = udev->pdev;
>  	int err;
>  
> +	if (atomic_inc_return(&udev->refcnt) > 1)
> +		return 0;
> +

Hmm, ideally, you also need to make sure that other calls
won't return until the setup is actually done.

Best regards,
Tiwei Bie

>  	/* set bus master, which was cleared by the reset function */
>  	pci_set_master(dev);
>  
> @@ -354,6 +358,9 @@ struct rte_uio_pci_dev {
>  	struct rte_uio_pci_dev *udev = info->priv;
>  	struct pci_dev *dev = udev->pdev;
>  
> +	if (atomic_dec_return(&udev->refcnt) > 0)
> +		return 0;
> +
>  	/* disable interrupts */
>  	igbuio_pci_disable_interrupts(udev);
>  
> -- 
> 1.8.3.1
> 

[dpdk-dev] [PATCH v4] igb_uio: allow multi-process access

[Xiao Wang]

In some case, one device are accessed by different processes via
different BARs, so one uio device may be opened by more than one
process, for this case we just need to enable interrupt once, and
pci_clear_master only when the last process closed.

Fixes: 5f6ff30dc507 ("igb_uio: fix interrupt enablement after FLR in VM")

Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
---
v4:
- Make open/release function atomic, to ensure that other call
  won't return until the setup is really done.

v3:
- Use atomic_inc_return instead of atomic_inc and atomic_read.
- Ditto for atomic_dec.

v2:
- Make uio reference counter operation atomic.
---
 lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
index a3a98c1..1826adb 100644
--- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
+++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
@@ -45,6 +45,8 @@ struct rte_uio_pci_dev {
 	struct uio_info info;
 	struct pci_dev *pdev;
 	enum rte_intr_mode mode;
+	struct mutex lock;
+	int refcnt;
 };
 
 static char *intr_mode;
@@ -336,11 +338,18 @@ struct rte_uio_pci_dev {
 	struct pci_dev *dev = udev->pdev;
 	int err;
 
+	mutex_lock(&udev->lock);
+	if (++udev->refcnt > 1) {
+		mutex_unlock(&udev->lock);
+		return 0;
+	}
+
 	/* set bus master, which was cleared by the reset function */
 	pci_set_master(dev);
 
 	/* enable interrupts */
 	err = igbuio_pci_enable_interrupts(udev);
+	mutex_unlock(&udev->lock);
 	if (err) {
 		dev_err(&dev->dev, "Enable interrupt fails\n");
 		return err;
@@ -354,12 +363,19 @@ struct rte_uio_pci_dev {
 	struct rte_uio_pci_dev *udev = info->priv;
 	struct pci_dev *dev = udev->pdev;
 
+	mutex_lock(&udev->lock);
+	if (--udev->refcnt > 0) {
+		mutex_unlock(&udev->lock);
+		return 0;
+	}
+
 	/* disable interrupts */
 	igbuio_pci_disable_interrupts(udev);
 
 	/* stop the device from further DMA */
 	pci_clear_master(dev);
 
+	mutex_unlock(&udev->lock);
 	return 0;
 }
 
@@ -480,6 +496,7 @@ struct rte_uio_pci_dev {
 	if (!udev)
 		return -ENOMEM;
 
+	mutex_init(&udev->lock);
 	/*
 	 * enable device: ask low-level code to enable I/O and
 	 * memory
@@ -570,6 +587,7 @@ struct rte_uio_pci_dev {
 {
 	struct rte_uio_pci_dev *udev = pci_get_drvdata(dev);
 
+	mutex_destroy(&udev->lock);
 	sysfs_remove_group(&dev->dev.kobj, &dev_attr_grp);
 	uio_unregister_device(&udev->info);
 	igbuio_pci_release_iomem(&udev->info);
-- 
1.8.3.1

Re: [dpdk-dev] [PATCH v4] igb_uio: allow multi-process access

[Ferruh Yigit]

On 1/1/2018 10:00 PM, Xiao Wang wrote:
> In some case, one device are accessed by different processes via
> different BARs, so one uio device may be opened by more than one
> process, for this case we just need to enable interrupt once, and
> pci_clear_master only when the last process closed.
> 
> Fixes: 5f6ff30dc507 ("igb_uio: fix interrupt enablement after FLR in VM")
> 
> Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>

Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>

Re: [dpdk-dev] [PATCH v4] igb_uio: allow multi-process access

[Thomas Monjalon]

16/01/2018 20:43, Ferruh Yigit:
> On 1/1/2018 10:00 PM, Xiao Wang wrote:
> > In some case, one device are accessed by different processes via
> > different BARs, so one uio device may be opened by more than one
> > process, for this case we just need to enable interrupt once, and
> > pci_clear_master only when the last process closed.
> > 
> > Fixes: 5f6ff30dc507 ("igb_uio: fix interrupt enablement after FLR in VM")

Cc: stable@dpdk.org

> > Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
> 
> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>

Applied, thanks

Re: [dpdk-dev] [PATCH] igb_uio: allow multi-process access

[Stephen Hemminger]

On Fri,  8 Dec 2017 17:57:33 -0800
Xiao Wang <xiao.w.wang@intel.com> wrote:

> In some case, one device are accessed by different processes via
> different BARs, so one uio device may be opened by more than one
> process, for this case we just need to enable interrupt once, and
> pci_clear_master only when the last process closed.
> 
> Fixes: 5f6ff30dc507 ("igb_uio: fix interrupt enablement after FLR in VM")
> 
> Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
> ---
>  lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
> index a3a98c1..c239d98 100644
> --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
> +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
> @@ -45,6 +45,7 @@ struct rte_uio_pci_dev {
>  	struct uio_info info;
>  	struct pci_dev *pdev;
>  	enum rte_intr_mode mode;
> +	uint32_t ref_cnt;
>  };
>  
>  static char *intr_mode;
> @@ -336,6 +337,9 @@ struct rte_uio_pci_dev {
>  	struct pci_dev *dev = udev->pdev;
>  	int err;
>  
> +	if (++(udev->ref_cnt) > 1)

Minor nit: Parenthesis is unnecessary here.

> +		return 0;
> +
>  	/* set bus master, which was cleared by the reset function */
>  	pci_set_master(dev);
>  
> @@ -354,6 +358,9 @@ struct rte_uio_pci_dev {
>  	struct rte_uio_pci_dev *udev = info->priv;
>  	struct pci_dev *dev = udev->pdev;
>  
> +	if (--(udev->ref_cnt) > 0)
> +		return 0;
> +
>  	/* disable interrupts */
>  	igbuio_pci_disable_interrupts(udev);
>  

You might consider using new reference counting (refcnt_t) in kernel which
protects against accidental under/overflow.