[dpdk-stable] [PATCH] igb_uio: prevent reset for a list of devices

[dpdk-stable] [PATCH] igb_uio: prevent reset for a list of devices

[Ferruh Yigit]

Some devices are having problem on device reset that happens during DPDK
application exit [1].

Create a static list of devices and exclude them from device reset.

[1]
http://dpdk.org/ml/archives/dev/2017-November/080927.html

Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of device file")
Cc: stable@dpdk.org

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: Jianfeng Tan <jianfeng.tan@intel.com>
Cc: Jingjing Wu <jingjing.wu@intel.com>
Cc: Shijith Thotton <shijith.thotton@caviumnetworks.com>
Cc: Gregory Etelson <gregory@weka.io>
Cc: Harish Patil <harish.patil@cavium.com>
Cc: George Prekas <george.prekas@epfl.ch>
Cc: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>
Cc: Rasesh Mody <rasesh.mody@cavium.com>
Cc: Lee Roberts <lee.roberts@hpe.com>

This is alternative approach to
http://dpdk.org/dev/patchwork/patch/31144/
---
 lib/librte_eal/linuxapp/igb_uio/compat.h  | 19 ++++++++++++++++++-
 lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 11 ++++++++++-
 2 files changed, 28 insertions(+), 2 deletions(-)

diff --git a/lib/librte_eal/linuxapp/igb_uio/compat.h b/lib/librte_eal/linuxapp/igb_uio/compat.h
index 30508f35c..5d7223124 100644
--- a/lib/librte_eal/linuxapp/igb_uio/compat.h
+++ b/lib/librte_eal/linuxapp/igb_uio/compat.h
@@ -133,4 +133,21 @@ static bool pci_check_and_mask_intx(struct pci_dev *pdev)
 #define HAVE_PCI_MSI_MASK_IRQ 1
 #endif
 
-
+#define BROADCOM_PCI_VENDOR_ID 0x14E4
+static const struct pci_device_id no_reset_pci_tbl[] = {
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x168a) }, /* 57800 */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x164f) }, /* 57711 */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x168e) }, /* 57810 */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x163d) }, /* 57811 */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x168d) }, /* 57840_OBS */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16a1) }, /* 57840_4_10 */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16a2) }, /* 57840_2_20 */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16ae) }, /* 57810_MF */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x163e) }, /* 57811_MF */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16a4) }, /* 57840_MF */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16a9) }, /* 57800_VF */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16af) }, /* 57810_VF */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x163f) }, /* 57811_VF */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16ad) }, /* 57840_VF */
+	{ 0 },
+};
diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
index fd320d87d..b0d92b51e 100644
--- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
+++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
@@ -348,6 +348,14 @@ igbuio_pci_open(struct uio_info *info, struct inode *inode)
 	return 0;
 }
 
+static int is_device_excluded_from_reset(struct pci_dev *pdev)
+{
+	if (pci_match_id(no_reset_pci_tbl, pdev))
+		return 1;
+
+	return 0;
+}
+
 static int
 igbuio_pci_release(struct uio_info *info, struct inode *inode)
 {
@@ -360,7 +368,8 @@ igbuio_pci_release(struct uio_info *info, struct inode *inode)
 	/* stop the device from further DMA */
 	pci_clear_master(dev);
 
-	pci_reset_function(dev);
+	if (!is_device_excluded_from_reset(dev))
+		pci_reset_function(dev);
 
 	return 0;
 }
-- 
2.13.6

Re: [dpdk-stable] [PATCH] igb_uio: prevent reset for a list of devices

[Mody, Rasesh]

> From: Ferruh Yigit [mailto:ferruh.yigit@intel.com]
> Sent: Friday, November 03, 2017 3:38 PM
> 
> Some devices are having problem on device reset that happens during DPDK
> application exit [1].
> 
> Create a static list of devices and exclude them from device reset.
> 
> [1]
> http://dpdk.org/ml/archives/dev/2017-November/080927.html
> 
> Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of device
> file")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> Cc: Jianfeng Tan <jianfeng.tan@intel.com>
> Cc: Jingjing Wu <jingjing.wu@intel.com>
> Cc: Shijith Thotton <shijith.thotton@caviumnetworks.com>
> Cc: Gregory Etelson <gregory@weka.io>
> Cc: Harish Patil <harish.patil@cavium.com>
> Cc: George Prekas <george.prekas@epfl.ch>
> Cc: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>
> Cc: Rasesh Mody <rasesh.mody@cavium.com>
> Cc: Lee Roberts <lee.roberts@hpe.com>
> 
> This is alternative approach to
> http://dpdk.org/dev/patchwork/patch/31144/

This patch is working.

Thanks!
-Rasesh

[dpdk-stable] [PATCH v2] igb_uio: prevent reset for a list of devices

[Ferruh Yigit]

Some devices are having problem on device reset that happens during DPDK
application exit [1].

Create a static list of devices and exclude them from device reset.

[1]
http://dpdk.org/ml/archives/dev/2017-November/080927.html

Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of device file")
Cc: stable@dpdk.org

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: Jianfeng Tan <jianfeng.tan@intel.com>
Cc: Jingjing Wu <jingjing.wu@intel.com>
Cc: Shijith Thotton <shijith.thotton@caviumnetworks.com>
Cc: Gregory Etelson <gregory@weka.io>
Cc: Harish Patil <harish.patil@cavium.com>
Cc: George Prekas <george.prekas@epfl.ch>
Cc: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>
Cc: Rasesh Mody <rasesh.mody@cavium.com>
Cc: Lee Roberts <lee.roberts@hpe.com>
Cc: Stephen Hemminger <stephen@networkplumber.org>

This is alternative approach to
http://dpdk.org/dev/patchwork/patch/31144/

v2:
* more concise function, no change in functionality
---
 lib/librte_eal/linuxapp/igb_uio/compat.h  | 19 ++++++++++++++++++-
 lib/librte_eal/linuxapp/igb_uio/igb_uio.c |  8 +++++++-
 2 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/lib/librte_eal/linuxapp/igb_uio/compat.h b/lib/librte_eal/linuxapp/igb_uio/compat.h
index 30508f35c..5d7223124 100644
--- a/lib/librte_eal/linuxapp/igb_uio/compat.h
+++ b/lib/librte_eal/linuxapp/igb_uio/compat.h
@@ -133,4 +133,21 @@ static bool pci_check_and_mask_intx(struct pci_dev *pdev)
 #define HAVE_PCI_MSI_MASK_IRQ 1
 #endif
 
-
+#define BROADCOM_PCI_VENDOR_ID 0x14E4
+static const struct pci_device_id no_reset_pci_tbl[] = {
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x168a) }, /* 57800 */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x164f) }, /* 57711 */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x168e) }, /* 57810 */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x163d) }, /* 57811 */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x168d) }, /* 57840_OBS */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16a1) }, /* 57840_4_10 */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16a2) }, /* 57840_2_20 */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16ae) }, /* 57810_MF */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x163e) }, /* 57811_MF */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16a4) }, /* 57840_MF */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16a9) }, /* 57800_VF */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16af) }, /* 57810_VF */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x163f) }, /* 57811_VF */
+	{ PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16ad) }, /* 57840_VF */
+	{ 0 },
+};
diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
index fd320d87d..037e02267 100644
--- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
+++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
@@ -348,6 +348,11 @@ igbuio_pci_open(struct uio_info *info, struct inode *inode)
 	return 0;
 }
 
+static bool is_device_excluded_from_reset(struct pci_dev *pdev)
+{
+	return !!pci_match_id(no_reset_pci_tbl, pdev);
+}
+
 static int
 igbuio_pci_release(struct uio_info *info, struct inode *inode)
 {
@@ -360,7 +365,8 @@ igbuio_pci_release(struct uio_info *info, struct inode *inode)
 	/* stop the device from further DMA */
 	pci_clear_master(dev);
 
-	pci_reset_function(dev);
+	if (!is_device_excluded_from_reset(dev))
+		pci_reset_function(dev);
 
 	return 0;
 }
-- 
2.13.6

Re: [dpdk-stable] [PATCH v2] igb_uio: prevent reset for a list of devices

[Thomas Monjalon]

06/11/2017 19:48, Ferruh Yigit:
> Some devices are having problem on device reset that happens during DPDK
> application exit [1].
> 
> Create a static list of devices and exclude them from device reset.
> 
> [1]
> http://dpdk.org/ml/archives/dev/2017-November/080927.html
> 
> Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of device file")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>

Applied, thanks

An option may be required to disable this exception
which may be a security hole.

Re: [dpdk-stable] [dpdk-dev] [PATCH v2] igb_uio: prevent reset for a list of devices

[Chas Williams]

We still have an issue with this and PCI pass-through.  If a guest is
restarted while using PCI pass-through and igb_uio issues a
pci_reset_function(), this causes the host to crash.

On Mon, Nov 6, 2017 at 6:55 PM, Thomas Monjalon <thomas@monjalon.net> wrote:

> 06/11/2017 19:48, Ferruh Yigit:
> > Some devices are having problem on device reset that happens during DPDK
> > application exit [1].
> >
> > Create a static list of devices and exclude them from device reset.
> >
> > [1]
> > http://dpdk.org/ml/archives/dev/2017-November/080927.html
> >
> > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of
> device file")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>
> Applied, thanks
>
> An option may be required to disable this exception
> which may be a security hole.
>
>

Re: [dpdk-stable] [dpdk-dev] [PATCH v2] igb_uio: prevent reset for a list of devices

[Stephen Hemminger]

On Nov 7, 2017 20:50, "Chas Williams" <3chas3@gmail.com> wrote:

We still have an issue with this and PCI pass-through.  If a guest is
restarted while using PCI pass-through and igb_uio issues a
pci_reset_function(), this causes the host to crash.

On Mon, Nov 6, 2017 at 6:55 PM, Thomas Monjalon <thomas@monjalon.net> wrote:

> 06/11/2017 19:48, Ferruh Yigit:
> > Some devices are having problem on device reset that happens during DPDK
> > application exit [1].
> >
> > Create a static list of devices and exclude them from device reset.
> >
> > [1]
> > http://dpdk.org/ml/archives/dev/2017-November/080927.html
> >
> > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of
> device file")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>
> Applied, thanks
>
> An option may be required to disable this exception
> which may be a security hole.
>
>

Which host. Anything guest can do to crash host is a high severity big in
the host

Re: [dpdk-stable] [dpdk-dev] [PATCH v2] igb_uio: prevent reset for a list of devices

[Chas Williams]

Regardless if the issue is actually in the host kernel, I cannot fix all
the hypervisors so I must attempt to be well behaved as a guest.

On Tue, Nov 7, 2017 at 8:13 AM, Stephen Hemminger <
stephen@networkplumber.org> wrote:

>
>
> On Nov 7, 2017 20:50, "Chas Williams" <3chas3@gmail.com> wrote:
>
> We still have an issue with this and PCI pass-through.  If a guest is
> restarted while using PCI pass-through and igb_uio issues a
> pci_reset_function(), this causes the host to crash.
>
> On Mon, Nov 6, 2017 at 6:55 PM, Thomas Monjalon <thomas@monjalon.net>
> wrote:
>
>> 06/11/2017 19:48, Ferruh Yigit:
>> > Some devices are having problem on device reset that happens during DPDK
>> > application exit [1].
>> >
>> > Create a static list of devices and exclude them from device reset.
>> >
>> > [1]
>> > http://dpdk.org/ml/archives/dev/2017-November/080927.html
>> >
>> > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of
>> device file")
>> > Cc: stable@dpdk.org
>> >
>> > Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>
>> Applied, thanks
>>
>> An option may be required to disable this exception
>> which may be a security hole.
>>
>>
>
> Which host. Anything guest can do to crash host is a high severity big in
> the host
>

[dpdk-stable] ugb_uio: r3.8xlarge bind failure

[Gregory Etelson]

Hello,

There are some AWS R3.8XLARGE instances 
that fail to bind Intel 10G VFs with igb_uio [c05cb4f939082]. 
System dmeg log show this backtrace:

igb_uio: probe of 0000:00:05.0 failed with error -16
IRQ handler type mismatch for IRQ 0
current handler: timer
Pid: 3619, comm: bash Not tainted 2.6.32-642.15.1.el6.x86_64 #1
Call Trace:
 [<ffffffff810f49e2>] ? __setup_irq+0x382/0x3c0
 [<ffffffffa03202a0>] ? uio_interrupt+0x0/0x48 [uio]
 [<ffffffff810f51e3>] ? request_threaded_irq+0x133/0x230
 [<ffffffffa0320193>] ? __uio_register_device+0x553/0x610 [uio]
 [<ffffffffa032698f>] ? igbuio_pci_probe+0x290/0x47a [igb_uio]
 [<ffffffff8129d00a>] ? kobject_get+0x1a/0x30
 [<ffffffff812c04f7>] ? local_pci_probe+0x17/0x20
 [<ffffffff812c16e1>] ? pci_device_probe+0x101/0x120
 [<ffffffff81382152>] ? driver_sysfs_add+0x62/0x90
 [<ffffffff813823fa>] ? driver_probe_device+0xaa/0x3a0
 [<ffffffff8138153a>] ? driver_bind+0xca/0x110
 [<ffffffff813805dc>] ? drv_attr_store+0x2c/0x30
 [<ffffffff812171c5>] ? sysfs_write_file+0xe5/0x170
 [<ffffffff81199e48>] ? vfs_write+0xb8/0x1a0
 [<ffffffff8119b336>] ? fget_light_pos+0x16/0x50
 [<ffffffff8119a981>] ? sys_write+0x51/0xb0

The VFs can be returned back to kernel ixgbevf driver with no faults. 

The instances can bind VFs with igb_uio[b58eedfc7dd57]

I could not find yet why some R3.8XLARGE instances can bind IXGBE VFs with 
igb_uio while other fail

lspci -vvv -s 0000:00:05.0
00:05.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller 
Virtual Function (rev 01)
        Physical Slot: 5
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64
        Region 0: Memory at f3010000 (64-bit, prefetchable) [size=16K]
        Region 3: Memory at f3014000 (64-bit, prefetchable) [size=16K]
        Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
                Vector table: BAR=3 offset=00000000
                PBA: BAR=3 offset=00002000
        Kernel driver in use: ixgbevf
        Kernel modules: ixgbevf

Regards,
Gregory

Re: [dpdk-stable] ugb_uio: r3.8xlarge bind failure

[Ferruh Yigit]

On 11/9/2017 9:20 AM, Gregory Etelson wrote:
> Hello,
> 
> There are some AWS R3.8XLARGE instances 
> that fail to bind Intel 10G VFs with igb_uio [c05cb4f939082]. 

Hi Gregory,

Will you dig this issue more? Please keep us updated.

> System dmeg log show this backtrace:
> 
> igb_uio: probe of 0000:00:05.0 failed with error -16
> IRQ handler type mismatch for IRQ 0
> current handler: timer
> Pid: 3619, comm: bash Not tainted 2.6.32-642.15.1.el6.x86_64 #1
> Call Trace:
>  [<ffffffff810f49e2>] ? __setup_irq+0x382/0x3c0
>  [<ffffffffa03202a0>] ? uio_interrupt+0x0/0x48 [uio]
>  [<ffffffff810f51e3>] ? request_threaded_irq+0x133/0x230
>  [<ffffffffa0320193>] ? __uio_register_device+0x553/0x610 [uio]
>  [<ffffffffa032698f>] ? igbuio_pci_probe+0x290/0x47a [igb_uio]
>  [<ffffffff8129d00a>] ? kobject_get+0x1a/0x30
>  [<ffffffff812c04f7>] ? local_pci_probe+0x17/0x20
>  [<ffffffff812c16e1>] ? pci_device_probe+0x101/0x120
>  [<ffffffff81382152>] ? driver_sysfs_add+0x62/0x90
>  [<ffffffff813823fa>] ? driver_probe_device+0xaa/0x3a0
>  [<ffffffff8138153a>] ? driver_bind+0xca/0x110
>  [<ffffffff813805dc>] ? drv_attr_store+0x2c/0x30
>  [<ffffffff812171c5>] ? sysfs_write_file+0xe5/0x170
>  [<ffffffff81199e48>] ? vfs_write+0xb8/0x1a0
>  [<ffffffff8119b336>] ? fget_light_pos+0x16/0x50
>  [<ffffffff8119a981>] ? sys_write+0x51/0xb0
> 
> The VFs can be returned back to kernel ixgbevf driver with no faults. 
> 
> The instances can bind VFs with igb_uio[b58eedfc7dd57]
> 
> I could not find yet why some R3.8XLARGE instances can bind IXGBE VFs with 
> igb_uio while other fail
> 
> lspci -vvv -s 0000:00:05.0
> 00:05.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller 
> Virtual Function (rev 01)
>         Physical Slot: 5
>         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx+
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 64
>         Region 0: Memory at f3010000 (64-bit, prefetchable) [size=16K]
>         Region 3: Memory at f3014000 (64-bit, prefetchable) [size=16K]
>         Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
>                 Vector table: BAR=3 offset=00000000
>                 PBA: BAR=3 offset=00002000
>         Kernel driver in use: ixgbevf
>         Kernel modules: ixgbevf
> 
> Regards,
> Gregory
> 

Re: [dpdk-stable] ugb_uio: r3.8xlarge bind failure

[Ferruh Yigit]

On 11/9/2017 5:42 PM, Ferruh Yigit wrote:
> On 11/9/2017 9:20 AM, Gregory Etelson wrote:
>> Hello,
>>
>> There are some AWS R3.8XLARGE instances 
>> that fail to bind Intel 10G VFs with igb_uio [c05cb4f939082]. 
> 
> Hi Gregory,
> 
> Will you dig this issue more? Please keep us updated.
> 
>> System dmeg log show this backtrace:
>>
>> igb_uio: probe of 0000:00:05.0 failed with error -16
>> IRQ handler type mismatch for IRQ 0
>> current handler: timer
>> Pid: 3619, comm: bash Not tainted 2.6.32-642.15.1.el6.x86_64 #1
>> Call Trace:
>>  [<ffffffff810f49e2>] ? __setup_irq+0x382/0x3c0
>>  [<ffffffffa03202a0>] ? uio_interrupt+0x0/0x48 [uio]
>>  [<ffffffff810f51e3>] ? request_threaded_irq+0x133/0x230
>>  [<ffffffffa0320193>] ? __uio_register_device+0x553/0x610 [uio]
>>  [<ffffffffa032698f>] ? igbuio_pci_probe+0x290/0x47a [igb_uio]

Here igb_uio probe() calls request_irq(), this behavior changed in latest code.

Can you please double check that you are using latest code?

Thanks,
ferruh

>>  [<ffffffff8129d00a>] ? kobject_get+0x1a/0x30
>>  [<ffffffff812c04f7>] ? local_pci_probe+0x17/0x20
>>  [<ffffffff812c16e1>] ? pci_device_probe+0x101/0x120
>>  [<ffffffff81382152>] ? driver_sysfs_add+0x62/0x90
>>  [<ffffffff813823fa>] ? driver_probe_device+0xaa/0x3a0
>>  [<ffffffff8138153a>] ? driver_bind+0xca/0x110
>>  [<ffffffff813805dc>] ? drv_attr_store+0x2c/0x30
>>  [<ffffffff812171c5>] ? sysfs_write_file+0xe5/0x170
>>  [<ffffffff81199e48>] ? vfs_write+0xb8/0x1a0
>>  [<ffffffff8119b336>] ? fget_light_pos+0x16/0x50
>>  [<ffffffff8119a981>] ? sys_write+0x51/0xb0
>>
>> The VFs can be returned back to kernel ixgbevf driver with no faults. 
>>
>> The instances can bind VFs with igb_uio[b58eedfc7dd57]
>>
>> I could not find yet why some R3.8XLARGE instances can bind IXGBE VFs with 
>> igb_uio while other fail
>>
>> lspci -vvv -s 0000:00:05.0
>> 00:05.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller 
>> Virtual Function (rev 01)
>>         Physical Slot: 5
>>         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
>> Stepping- SERR- FastB2B- DisINTx+
>>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>         Latency: 64
>>         Region 0: Memory at f3010000 (64-bit, prefetchable) [size=16K]
>>         Region 3: Memory at f3014000 (64-bit, prefetchable) [size=16K]
>>         Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
>>                 Vector table: BAR=3 offset=00000000
>>                 PBA: BAR=3 offset=00002000
>>         Kernel driver in use: ixgbevf
>>         Kernel modules: ixgbevf
>>
>> Regards,
>> Gregory
>>
> 

Re: [dpdk-stable] ugb_uio: r3.8xlarge bind failure

[Gregory Etelson]

It looks like igb_uio bind failed on servers running CentOS-6.x
Servers with CentOS-7.3 Ubuntu-14, Ubuntu-16 and AWS-1703 (Amazon Linux)
had no bind issues

Regards,
Gregory

On Fri, Nov 10, 2017 at 3:42 AM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 11/9/2017 9:20 AM, Gregory Etelson wrote:
> > Hello,
> >
> > There are some AWS R3.8XLARGE instances
> > that fail to bind Intel 10G VFs with igb_uio [c05cb4f939082].
>
> Hi Gregory,
>
> Will you dig this issue more? Please keep us updated.
>
> > System dmeg log show this backtrace:
> >
> > igb_uio: probe of 0000:00:05.0 failed with error -16
> > IRQ handler type mismatch for IRQ 0
> > current handler: timer
> > Pid: 3619, comm: bash Not tainted 2.6.32-642.15.1.el6.x86_64 #1
> > Call Trace:
> >  [<ffffffff810f49e2>] ? __setup_irq+0x382/0x3c0
> >  [<ffffffffa03202a0>] ? uio_interrupt+0x0/0x48 [uio]
> >  [<ffffffff810f51e3>] ? request_threaded_irq+0x133/0x230
> >  [<ffffffffa0320193>] ? __uio_register_device+0x553/0x610 [uio]
> >  [<ffffffffa032698f>] ? igbuio_pci_probe+0x290/0x47a [igb_uio]
> >  [<ffffffff8129d00a>] ? kobject_get+0x1a/0x30
> >  [<ffffffff812c04f7>] ? local_pci_probe+0x17/0x20
> >  [<ffffffff812c16e1>] ? pci_device_probe+0x101/0x120
> >  [<ffffffff81382152>] ? driver_sysfs_add+0x62/0x90
> >  [<ffffffff813823fa>] ? driver_probe_device+0xaa/0x3a0
> >  [<ffffffff8138153a>] ? driver_bind+0xca/0x110
> >  [<ffffffff813805dc>] ? drv_attr_store+0x2c/0x30
> >  [<ffffffff812171c5>] ? sysfs_write_file+0xe5/0x170
> >  [<ffffffff81199e48>] ? vfs_write+0xb8/0x1a0
> >  [<ffffffff8119b336>] ? fget_light_pos+0x16/0x50
> >  [<ffffffff8119a981>] ? sys_write+0x51/0xb0
> >
> > The VFs can be returned back to kernel ixgbevf driver with no faults.
> >
> > The instances can bind VFs with igb_uio[b58eedfc7dd57]
> >
> > I could not find yet why some R3.8XLARGE instances can bind IXGBE VFs
> with
> > igb_uio while other fail
> >
> > lspci -vvv -s 0000:00:05.0
> > 00:05.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller
> > Virtual Function (rev 01)
> >         Physical Slot: 5
> >         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr-
> > Stepping- SERR- FastB2B- DisINTx+
> >         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> > <TAbort- <MAbort- >SERR- <PERR- INTx-
> >         Latency: 64
> >         Region 0: Memory at f3010000 (64-bit, prefetchable) [size=16K]
> >         Region 3: Memory at f3014000 (64-bit, prefetchable) [size=16K]
> >         Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
> >                 Vector table: BAR=3 offset=00000000
> >                 PBA: BAR=3 offset=00002000
> >         Kernel driver in use: ixgbevf
> >         Kernel modules: ixgbevf
> >
> > Regards,
> > Gregory
> >
>
>

Re: [dpdk-stable] ugb_uio: r3.8xlarge bind failure

[Ferruh Yigit]

On 11/9/2017 10:36 PM, Gregory Etelson wrote:
> It looks like igb_uio bind failed on servers running CentOS-6.x

Hi Gregory,

Below backtrace seems coming from old code, can you please confirm that you are
using latest igb_uio?

And what is the kernel version in that boxes?

Thanks,
ferruh

> Servers with CentOS-7.3 Ubuntu-14, Ubuntu-16 and AWS-1703 (Amazon Linux)
> had no bind issues
> 
> Regards,
> Gregory
> 
> On Fri, Nov 10, 2017 at 3:42 AM, Ferruh Yigit <ferruh.yigit@intel.com
> <mailto:ferruh.yigit@intel.com>> wrote:
> 
>     On 11/9/2017 9:20 AM, Gregory Etelson wrote:
>     > Hello,
>     >
>     > There are some AWS R3.8XLARGE instances
>     > that fail to bind Intel 10G VFs with igb_uio [c05cb4f939082].
> 
>     Hi Gregory,
> 
>     Will you dig this issue more? Please keep us updated.
> 
>     > System dmeg log show this backtrace:
>     >
>     > igb_uio: probe of 0000:00:05.0 failed with error -16
>     > IRQ handler type mismatch for IRQ 0
>     > current handler: timer
>     > Pid: 3619, comm: bash Not tainted 2.6.32-642.15.1.el6.x86_64 #1
>     > Call Trace:
>     >  [<ffffffff810f49e2>] ? __setup_irq+0x382/0x3c0
>     >  [<ffffffffa03202a0>] ? uio_interrupt+0x0/0x48 [uio]
>     >  [<ffffffff810f51e3>] ? request_threaded_irq+0x133/0x230
>     >  [<ffffffffa0320193>] ? __uio_register_device+0x553/0x610 [uio]
>     >  [<ffffffffa032698f>] ? igbuio_pci_probe+0x290/0x47a [igb_uio]
>     >  [<ffffffff8129d00a>] ? kobject_get+0x1a/0x30
>     >  [<ffffffff812c04f7>] ? local_pci_probe+0x17/0x20
>     >  [<ffffffff812c16e1>] ? pci_device_probe+0x101/0x120
>     >  [<ffffffff81382152>] ? driver_sysfs_add+0x62/0x90
>     >  [<ffffffff813823fa>] ? driver_probe_device+0xaa/0x3a0
>     >  [<ffffffff8138153a>] ? driver_bind+0xca/0x110
>     >  [<ffffffff813805dc>] ? drv_attr_store+0x2c/0x30
>     >  [<ffffffff812171c5>] ? sysfs_write_file+0xe5/0x170
>     >  [<ffffffff81199e48>] ? vfs_write+0xb8/0x1a0
>     >  [<ffffffff8119b336>] ? fget_light_pos+0x16/0x50
>     >  [<ffffffff8119a981>] ? sys_write+0x51/0xb0
>     >
>     > The VFs can be returned back to kernel ixgbevf driver with no faults.
>     >
>     > The instances can bind VFs with igb_uio[b58eedfc7dd57]
>     >
>     > I could not find yet why some R3.8XLARGE instances can bind IXGBE VFs with
>     > igb_uio while other fail
>     >
>     > lspci -vvv -s 0000:00:05.0
>     > 00:05.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller
>     > Virtual Function (rev 01)
>     >         Physical Slot: 5
>     >         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>     > Stepping- SERR- FastB2B- DisINTx+
>     >         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>     > <TAbort- <MAbort- >SERR- <PERR- INTx-
>     >         Latency: 64
>     >         Region 0: Memory at f3010000 (64-bit, prefetchable) [size=16K]
>     >         Region 3: Memory at f3014000 (64-bit, prefetchable) [size=16K]
>     >         Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
>     >                 Vector table: BAR=3 offset=00000000
>     >                 PBA: BAR=3 offset=00002000
>     >         Kernel driver in use: ixgbevf
>     >         Kernel modules: ixgbevf
>     >
>     > Regards,
>     > Gregory
>     >
> 
> 

Re: [dpdk-stable] ugb_uio: r3.8xlarge bind failure

[Gregory Etelson]

Hello Ferruh,

re-checked igb_uio from dpdk master branch
0384f21dffc9081d1ae30f0a6e49926bfc4be85d
OS: CentOS release 6.7 (Final), 2.6.32-573.26.1.el6.x86_64

igb_uio: Use MSIX interrupt by default
ixgbevf: eth1: ixgbevf_remove: Remove complete
IRQ handler type mismatch for IRQ 0
current handler: timer
Pid: 7995, comm: python Not tainted 2.6.32-573.26.1.el6.x86_64 #1
Call Trace:
 [<ffffffff810eee02>] ? __setup_irq+0x382/0x3c0
 [<ffffffffa00212a0>] ? uio_interrupt+0x0/0x48 [uio]
 [<ffffffff810ef603>] ? request_threaded_irq+0x133/0x230
 [<ffffffffa0021193>] ? __uio_register_device+0x553/0x610 [uio]
 [<ffffffffa003797f>] ? igbuio_pci_probe+0x290/0x47a [igb_uio]
 [<ffffffff81292c7a>] ? kobject_get+0x1a/0x30
 [<ffffffff812b5e37>] ? local_pci_probe+0x17/0x20
 [<ffffffff812b7021>] ? pci_device_probe+0x101/0x120
 [<ffffffff813748e2>] ? driver_sysfs_add+0x62/0x90
 [<ffffffff81374b8a>] ? driver_probe_device+0xaa/0x3a0
 [<ffffffff81374f2b>] ? __driver_attach+0xab/0xb0
 [<ffffffff81374e80>] ? __driver_attach+0x0/0xb0
 [<ffffffff81373d74>] ? bus_for_each_dev+0x64/0x90
 [<ffffffff8137481e>] ? driver_attach+0x1e/0x20
 [<ffffffff812b73c7>] ? pci_add_dynid+0xc7/0xf0
 [<ffffffff812b74c2>] ? store_new_id+0xd2/0x110
 [<ffffffff81372d6c>] ? drv_attr_store+0x2c/0x30
 [<ffffffff8120eb15>] ? sysfs_write_file+0xe5/0x170
 [<ffffffff81192208>] ? vfs_write+0xb8/0x1a0
 [<ffffffff811936f6>] ? fget_light_pos+0x16/0x50
 [<ffffffff81192d41>] ? sys_write+0x51/0xb0
 [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b
igb_uio: probe of 0000:00:04.0 failed with error -16
IRQ handler type mismatch for IRQ 0

Regards,
Gregory

On Wed, Nov 15, 2017 at 5:44 PM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 11/9/2017 10:36 PM, Gregory Etelson wrote:
> > It looks like igb_uio bind failed on servers running CentOS-6.x
>
> Hi Gregory,
>
> Below backtrace seems coming from old code, can you please confirm that
> you are
> using latest igb_uio?
>
> And what is the kernel version in that boxes?
>
> Thanks,
> ferruh
>
> > Servers with CentOS-7.3 Ubuntu-14, Ubuntu-16 and AWS-1703 (Amazon Linux)
> > had no bind issues
> >
> > Regards,
> > Gregory
> >
> > On Fri, Nov 10, 2017 at 3:42 AM, Ferruh Yigit <ferruh.yigit@intel.com
> > <mailto:ferruh.yigit@intel.com>> wrote:
> >
> >     On 11/9/2017 9:20 AM, Gregory Etelson wrote:
> >     > Hello,
> >     >
> >     > There are some AWS R3.8XLARGE instances
> >     > that fail to bind Intel 10G VFs with igb_uio [c05cb4f939082].
> >
> >     Hi Gregory,
> >
> >     Will you dig this issue more? Please keep us updated.
> >
> >     > System dmeg log show this backtrace:
> >     >
> >     > igb_uio: probe of 0000:00:05.0 failed with error -16
> >     > IRQ handler type mismatch for IRQ 0
> >     > current handler: timer
> >     > Pid: 3619, comm: bash Not tainted 2.6.32-642.15.1.el6.x86_64 #1
> >     > Call Trace:
> >     >  [<ffffffff810f49e2>] ? __setup_irq+0x382/0x3c0
> >     >  [<ffffffffa03202a0>] ? uio_interrupt+0x0/0x48 [uio]
> >     >  [<ffffffff810f51e3>] ? request_threaded_irq+0x133/0x230
> >     >  [<ffffffffa0320193>] ? __uio_register_device+0x553/0x610 [uio]
> >     >  [<ffffffffa032698f>] ? igbuio_pci_probe+0x290/0x47a [igb_uio]
> >     >  [<ffffffff8129d00a>] ? kobject_get+0x1a/0x30
> >     >  [<ffffffff812c04f7>] ? local_pci_probe+0x17/0x20
> >     >  [<ffffffff812c16e1>] ? pci_device_probe+0x101/0x120
> >     >  [<ffffffff81382152>] ? driver_sysfs_add+0x62/0x90
> >     >  [<ffffffff813823fa>] ? driver_probe_device+0xaa/0x3a0
> >     >  [<ffffffff8138153a>] ? driver_bind+0xca/0x110
> >     >  [<ffffffff813805dc>] ? drv_attr_store+0x2c/0x30
> >     >  [<ffffffff812171c5>] ? sysfs_write_file+0xe5/0x170
> >     >  [<ffffffff81199e48>] ? vfs_write+0xb8/0x1a0
> >     >  [<ffffffff8119b336>] ? fget_light_pos+0x16/0x50
> >     >  [<ffffffff8119a981>] ? sys_write+0x51/0xb0
> >     >
> >     > The VFs can be returned back to kernel ixgbevf driver with no
> faults.
> >     >
> >     > The instances can bind VFs with igb_uio[b58eedfc7dd57]
> >     >
> >     > I could not find yet why some R3.8XLARGE instances can bind IXGBE
> VFs with
> >     > igb_uio while other fail
> >     >
> >     > lspci -vvv -s 0000:00:05.0
> >     > 00:05.0 Ethernet controller: Intel Corporation 82599 Ethernet
> Controller
> >     > Virtual Function (rev 01)
> >     >         Physical Slot: 5
> >     >         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV-
> VGASnoop- ParErr-
> >     > Stepping- SERR- FastB2B- DisINTx+
> >     >         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast
> >TAbort-
> >     > <TAbort- <MAbort- >SERR- <PERR- INTx-
> >     >         Latency: 64
> >     >         Region 0: Memory at f3010000 (64-bit, prefetchable)
> [size=16K]
> >     >         Region 3: Memory at f3014000 (64-bit, prefetchable)
> [size=16K]
> >     >         Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
> >     >                 Vector table: BAR=3 offset=00000000
> >     >                 PBA: BAR=3 offset=00002000
> >     >         Kernel driver in use: ixgbevf
> >     >         Kernel modules: ixgbevf
> >     >
> >     > Regards,
> >     > Gregory
> >     >
> >
> >
>
>