[dpdk-dev] Issues of MSIx interrupt enable with user space driver

[dpdk-dev] Issues of MSIx interrupt enable with user space driver

[James (Fei) Liu-SSI]

Hi All,
    We are porting driver from kernel into user space driver with DPDK for one of our devices. Polling thread works fine without any issue. However, we want to switch from polling to interrupt based handler. While we enable interrupt with VFIO MSIx, we always get below errors after issuing one command:
  
[  465.464186] dmar: DRHD: handling fault status reg 2
[  465.464194] dmar: INTR-REMAP: Request device [[02:00.0] fault index 27
[  465.464194] INTR-REMAP:[fault reason 34] Present field in the IRTE entry is clear

Here are steps we follow to enable interrupt:
1. --vfio-intr msix   was passed in as parameter for rte_eal_init
2. rte_intr_enable was called during the drive probe stage.
3. rte_intr_callback_register was called after interrupt was enabled



Is there anyone in the forum who worked on the interrupt handler to share with us about your comments? thanks for your help in advance.

Regards,
James
   

Re: [dpdk-dev] Issues of MSIx interrupt enable with user space driver

[Zhou, Danny]

The V6 "Interrupt PMD" patch has been sent out to DPDK.org 2 months ago, and V7 is under development with changes 
to include more generic APIs that could support all kinds of interrupts (e.g. HW Rx interrupt and SW ring interrupt). 

Stay tuned. 

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of James (Fei) Liu-SSI
> Sent: Wednesday, April 29, 2015 5:58 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] Issues of MSIx interrupt enable with user space driver
> 
> Hi All,
>     We are porting driver from kernel into user space driver with DPDK for one of our devices. Polling thread works fine without
> any issue. However, we want to switch from polling to interrupt based handler. While we enable interrupt with VFIO MSIx, we
> always get below errors after issuing one command:
> 
> [  465.464186] dmar: DRHD: handling fault status reg 2
> [  465.464194] dmar: INTR-REMAP: Request device [[02:00.0] fault index 27
> [  465.464194] INTR-REMAP:[fault reason 34] Present field in the IRTE entry is clear
> 
> Here are steps we follow to enable interrupt:
> 1. --vfio-intr msix   was passed in as parameter for rte_eal_init
> 2. rte_intr_enable was called during the drive probe stage.
> 3. rte_intr_callback_register was called after interrupt was enabled
> 
> 
> 
> Is there anyone in the forum who worked on the interrupt handler to share with us about your comments? thanks for your help
> in advance.
> 
> Regards,
> James
> 

Re: [dpdk-dev] Issues of MSIx interrupt enable with user space driver

[James (Fei) Liu-SSI]

Hi Danny,

  Thanks  a lot for your promptly answer. I am going to check your patch soon. In the mean time, Have you guys ever try DPDK with PCIe storage device?

 Regards,
 James

-----Original Message-----
From: Zhou, Danny [mailto:danny.zhou@intel.com] 
Sent: Tuesday, April 28, 2015 4:19 PM
To: James (Fei) Liu-SSI; dev@dpdk.org
Cc: Liang, Cunming
Subject: RE: [dpdk-dev] Issues of MSIx interrupt enable with user space driver

The V6 "Interrupt PMD" patch has been sent out to DPDK.org 2 months ago, and V7 is under development with changes to include more generic APIs that could support all kinds of interrupts (e.g. HW Rx interrupt and SW ring interrupt). 

Stay tuned. 

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of James (Fei) 
> Liu-SSI
> Sent: Wednesday, April 29, 2015 5:58 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] Issues of MSIx interrupt enable with user space 
> driver
> 
> Hi All,
>     We are porting driver from kernel into user space driver with DPDK 
> for one of our devices. Polling thread works fine without any issue. 
> However, we want to switch from polling to interrupt based handler. While we enable interrupt with VFIO MSIx, we always get below errors after issuing one command:
> 
> [  465.464186] dmar: DRHD: handling fault status reg 2 [  465.464194] 
> dmar: INTR-REMAP: Request device [[02:00.0] fault index 27 [  
> 465.464194] INTR-REMAP:[fault reason 34] Present field in the IRTE 
> entry is clear
> 
> Here are steps we follow to enable interrupt:
> 1. --vfio-intr msix   was passed in as parameter for rte_eal_init
> 2. rte_intr_enable was called during the drive probe stage.
> 3. rte_intr_callback_register was called after interrupt was enabled
> 
> 
> 
> Is there anyone in the forum who worked on the interrupt handler to 
> share with us about your comments? thanks for your help in advance.
> 
> Regards,
> James
> 

Re: [dpdk-dev] Issues of MSIx interrupt enable with user space driver

[Zhou, Danny]

DPDK is aimed to drive PCIe NIC device for high performance packet processing only. While we do have a Waikiki Beach project that
integrates user space DPDK PMD and NVMe poll mode iSCSI target and NVMe backend together to improve performance of storage.

> -----Original Message-----
> From: James (Fei) Liu-SSI [mailto:james.liu@ssi.samsung.com]
> Sent: Thursday, April 30, 2015 2:26 AM
> To: Zhou, Danny; dev@dpdk.org
> Cc: Liang, Cunming
> Subject: RE: [dpdk-dev] Issues of MSIx interrupt enable with user space driver
> 
> Hi Danny,
> 
>   Thanks  a lot for your promptly answer. I am going to check your patch soon. In the mean time, Have you guys ever try DPDK
> with PCIe storage device?
> 
>  Regards,
>  James
> 
> -----Original Message-----
> From: Zhou, Danny [mailto:danny.zhou@intel.com]
> Sent: Tuesday, April 28, 2015 4:19 PM
> To: James (Fei) Liu-SSI; dev@dpdk.org
> Cc: Liang, Cunming
> Subject: RE: [dpdk-dev] Issues of MSIx interrupt enable with user space driver
> 
> The V6 "Interrupt PMD" patch has been sent out to DPDK.org 2 months ago, and V7 is under development with changes to
> include more generic APIs that could support all kinds of interrupts (e.g. HW Rx interrupt and SW ring interrupt).
> 
> Stay tuned.
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of James (Fei)
> > Liu-SSI
> > Sent: Wednesday, April 29, 2015 5:58 AM
> > To: dev@dpdk.org
> > Subject: [dpdk-dev] Issues of MSIx interrupt enable with user space
> > driver
> >
> > Hi All,
> >     We are porting driver from kernel into user space driver with DPDK
> > for one of our devices. Polling thread works fine without any issue.
> > However, we want to switch from polling to interrupt based handler. While we enable interrupt with VFIO MSIx, we always get
> below errors after issuing one command:
> >
> > [  465.464186] dmar: DRHD: handling fault status reg 2 [  465.464194]
> > dmar: INTR-REMAP: Request device [[02:00.0] fault index 27 [
> > 465.464194] INTR-REMAP:[fault reason 34] Present field in the IRTE
> > entry is clear
> >
> > Here are steps we follow to enable interrupt:
> > 1. --vfio-intr msix   was passed in as parameter for rte_eal_init
> > 2. rte_intr_enable was called during the drive probe stage.
> > 3. rte_intr_callback_register was called after interrupt was enabled
> >
> >
> >
> > Is there anyone in the forum who worked on the interrupt handler to
> > share with us about your comments? thanks for your help in advance.
> >
> > Regards,
> > James
> >

Re: [dpdk-dev] Issues of MSIx interrupt enable with user space driver

[Stephen Hemminger]

On Tue, 28 Apr 2015 21:57:31 +0000
"James (Fei) Liu-SSI" <james.liu@ssi.samsung.com> wrote:

> Hi All,
>     We are porting driver from kernel into user space driver with DPDK for one of our devices. Polling thread works fine without any issue. However, we want to switch from polling to interrupt based handler. While we enable interrupt with VFIO MSIx, we always get below errors after issuing one command:
>   
> [  465.464186] dmar: DRHD: handling fault status reg 2
> [  465.464194] dmar: INTR-REMAP: Request device [[02:00.0] fault index 27
> [  465.464194] INTR-REMAP:[fault reason 34] Present field in the IRTE entry is clear
> 
> Here are steps we follow to enable interrupt:
> 1. --vfio-intr msix   was passed in as parameter for rte_eal_init
> 2. rte_intr_enable was called during the drive probe stage.
> 3. rte_intr_callback_register was called after interrupt was enabled
> 
> 
> 
> Is there anyone in the forum who worked on the interrupt handler to share with us about your comments? thanks for your help in advance.

Do you have kernel version in your build environment that has <linux/vfio.h> ?

Do you have iommu enabled on your target machine?

Did you program the MSI-X correctly?

Re: [dpdk-dev] Issues of MSIx interrupt enable with user space driver

[James (Fei) Liu-SSI]

Do you have kernel version in your build environment that has <linux/vfio.h> ?
[James] The kernel version is 3.17.2

Do you have iommu enabled on your target machine?
[James] Yes

Did you program the MSI-X correctly?
[James] We can issues several admin commands successfully with interrupt enabled , however. It is going to fail during IO traffic. By the way, we are using DPDK for PCIe storage driver development. Have you guys ever use dpdk for PCIe storage device?

Thanks so much.

Regards,
James

-----Original Message-----
From: Stephen Hemminger [mailto:stephen@networkplumber.org]
Sent: Tuesday, April 28, 2015 4:43 PM
To: James (Fei) Liu-SSI
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] Issues of MSIx interrupt enable with user space driver

On Tue, 28 Apr 2015 21:57:31 +0000
"James (Fei) Liu-SSI" <james.liu@ssi.samsung.com<mailto:james.liu@ssi.samsung.com>> wrote:

> Hi All,
>     We are porting driver from kernel into user space driver with DPDK for one of our devices. Polling thread works fine without any issue. However, we want to switch from polling to interrupt based handler. While we enable interrupt with VFIO MSIx, we always get below errors after issuing one command:
>
> [  465.464186] dmar: DRHD: handling fault status reg 2 [  465.464194]
> dmar: INTR-REMAP: Request device [[02:00.0] fault index 27 [
> 465.464194] INTR-REMAP:[fault reason 34] Present field in the IRTE
> entry is clear
>
> Here are steps we follow to enable interrupt:
> 1. --vfio-intr msix   was passed in as parameter for rte_eal_init
> 2. rte_intr_enable was called during the drive probe stage.
> 3. rte_intr_callback_register was called after interrupt was enabled
>
>
>
> Is there anyone in the forum who worked on the interrupt handler to share with us about your comments? thanks for your help in advance.

Do you have kernel version in your build environment that has <linux/vfio.h> ?

Do you have iommu enabled on your target machine?

Did you program the MSI-X correctly?

Re: [dpdk-dev] Issues of MSIx interrupt enable with user space driver

[James (Fei) Liu-SSI]

Hi ALL,
   Dump trace was added for more debugging information. thanks for your help in advance.

[  180.360229] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x180
[  180.486813] vfio-pci 0000:02:00.0: irq 27 for MSI/MSI-X
[  180.489332] dmar: DRHD: handling fault status reg 2
[  180.489354] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           OE  3.17.2 #4
[  180.489356] Hardware name: Dell Inc. OptiPlex 7010/0773VG, BIOS A05 07/31/2012
[  180.489358]  001f000000000000 ffff88011e203e68 ffffffff81ac42d7 0000000000000000
[  180.489362]  ffff88011e203ec8 ffffffff81997161 0000000000000019 ffff88011d807780
[  180.489365]  0000000000000200 0000000000000022 ffff88011d80692c ffff880117a80800
[  180.489368] Call Trace:
[  180.489370]  <IRQ>  [<ffffffff81ac42d7>] dump_stack+0x45/0x56
[  180.489383]  [<ffffffff81997161>] dmar_fault+0xc1/0x230
[  180.489388]  [<ffffffff8110469e>] handle_irq_event_percpu+0x3e/0x1a0
[  180.489391]  [<ffffffff8110483d>] handle_irq_event+0x3d/0x60
[  180.489394]  [<ffffffff811073e7>] handle_edge_irq+0x77/0x130
[  180.489399]  [<ffffffff810594ee>] handle_irq+0x1e/0x30
[  180.489402]  [<ffffffff81ad112f>] do_IRQ+0x4f/0xf0
[  180.489406]  [<ffffffff81acefad>] common_interrupt+0x6d/0x6d
[  180.489407]  <EOI>  [<ffffffff819611e0>] ? cpuidle_enter_state+0x70/0x170
[  180.489415]  [<ffffffff819611cd>] ? cpuidle_enter_state+0x5d/0x170
[  180.489419]  [<ffffffff81961397>] cpuidle_enter+0x17/0x20
[  180.489422]  [<ffffffff810eebad>] cpu_startup_entry+0x31d/0x340
[  180.489425]  [<ffffffff81ab4277>] rest_init+0x77/0x80
[  180.489430]  [<ffffffff8238d07f>] start_kernel+0x42a/0x435
[  180.489433]  [<ffffffff8238ca4e>] ? set_init_arg+0x53/0x53
[  180.489437]  [<ffffffff8238c120>] ? early_idt_handlers+0x120/0x120
[  180.489440]  [<ffffffff8238c5ee>] x86_64_start_reservations+0x2a/0x2c
[  180.489443]  [<ffffffff8238c733>] x86_64_start_kernel+0x143/0x152
[  180.489446] dmar: INTR-REMAP: Request device [[02:00.0] fault index 1f
[  180.489446] INTR-REMAP:[fault reason 34] Present field in the IRTE entry is clear


Regards,
James
________________________________________
From: dev [dev-bounces@dpdk.org] on behalf of James (Fei) Liu-SSI [james.liu@ssi.samsung.com]
Sent: Tuesday, April 28, 2015 2:57 PM
To: dev@dpdk.org
Subject: [dpdk-dev] Issues of MSIx interrupt enable with user space driver

Hi All,
    We are porting driver from kernel into user space driver with DPDK for one of our devices. Polling thread works fine without any issue. However, we want to switch from polling to interrupt based handler. While we enable interrupt with VFIO MSIx, we always get below errors after issuing one command:

[  465.464186] dmar: DRHD: handling fault status reg 2
[  465.464194] dmar: INTR-REMAP: Request device [[02:00.0] fault index 27
[  465.464194] INTR-REMAP:[fault reason 34] Present field in the IRTE entry is clear

Here are steps we follow to enable interrupt:
1. --vfio-intr msix   was passed in as parameter for rte_eal_init
2. rte_intr_enable was called during the drive probe stage.
3. rte_intr_callback_register was called after interrupt was enabled



Is there anyone in the forum who worked on the interrupt handler to share with us about your comments? thanks for your help in advance.

Regards,
James