[dpdk-dev] [PATCH] igb_uio: map dummy dma forcing iommu domain attachment

[dpdk-dev] [PATCH] igb_uio: map dummy dma forcing iommu domain attachment

[Alejandro Lucero]

For using a DPDK app when iommu is enabled, it requires to
add iommu=pt to the kernel command line. But using igb_uio driver
makes DMAR errors because the device has not an IOMMU domain.

Since kernel 3.15, iommu=pt requires to use the internal kernel
DMA API for attaching the device to the IOMMU 1:1 mapping, aka
si_domain. Previous versions did attach the device to that
domain when intel iommu notifier was called.

This is not a problem if the driver does later some call to the
DMA API because the mapping can be done then. But DPDK apps do
not use that DMA API at all.

Doing this dma map and unmap is harmless even when iommu is not
enabled at all.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
index df41e45..f79fe6b 100644
--- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
+++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
@@ -326,6 +326,8 @@ struct rte_uio_pci_dev {
 {
 	struct rte_uio_pci_dev *udev;
 	struct msix_entry msix_entry;
+	dma_addr_t map_dma_addr;
+	void *map_addr;
 	int err;
 
 	udev = kzalloc(sizeof(struct rte_uio_pci_dev), GFP_KERNEL);
@@ -423,6 +425,25 @@ struct rte_uio_pci_dev {
 	dev_info(&dev->dev, "uio device registered with irq %lx\n",
 		 udev->info.irq);
 
+	/*
+	 * Doing a harmless dma mapping for attaching the device to
+	 * the iommu identity mapping if kernel boots with iommu=pt.
+	 * Note this is not a problem if no IOMMU at all.
+	 */
+	map_addr = dma_zalloc_coherent(&dev->dev, 1024,
+				       &map_dma_addr, GFP_KERNEL);
+
+	if (!map_addr)
+		dev_info(&dev->dev, "dma mapping failed\n");
+	else {
+		dev_info(&dev->dev, "mapping 1K dma=%#llx host=%p\n",
+			 (unsigned long long)map_dma_addr, map_addr);
+
+		dma_free_coherent(&dev->dev, 1024, map_addr, map_dma_addr);
+		dev_info(&dev->dev, "unmapping 1K dma=%#llx host=%p\n",
+			 (unsigned long long)map_dma_addr, map_addr);
+	}
+
 	return 0;
 
 fail_remove_group:
-- 
1.9.1

Re: [dpdk-dev] [PATCH] igb_uio: map dummy dma forcing iommu domain attachment

[Ferruh Yigit]

Hi Alejandro,

On 1/18/2017 12:27 PM, Alejandro Lucero wrote:
> For using a DPDK app when iommu is enabled, it requires to
> add iommu=pt to the kernel command line. But using igb_uio driver
> makes DMAR errors because the device has not an IOMMU domain.

Please help to understand the scope of the problem,

1- How can you re-produce the problem?

2- What happens get DMAR errors, is it prevents device work or some
annoying error messages?

3- Can you please share the error messages?


> 
> Since kernel 3.15, iommu=pt requires to use the internal kernel
> DMA API for attaching the device to the IOMMU 1:1 mapping, aka
> si_domain. Previous versions did attach the device to that
> domain when intel iommu notifier was called.

Again, what is not working since 3.15?

> 
> This is not a problem if the driver does later some call to the
> DMA API because the mapping can be done then. But DPDK apps do
> not use that DMA API at all.

Is this same/similar with: http://dpdk.org/dev/patchwork/patch/12654/

> 
> Doing this dma map and unmap is harmless even when iommu is not
> enabled at all.
> 
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
<...>

Thanks,
ferruh

Re: [dpdk-dev] [PATCH] igb_uio: map dummy dma forcing iommu domain attachment

[Alejandro Lucero]

Hi Ferruh,

On Tue, Feb 7, 2017 at 3:59 PM, Ferruh Yigit <ferruh.yigit@intel.com> wrote:

> Hi Alejandro,
>
> On 1/18/2017 12:27 PM, Alejandro Lucero wrote:
> > For using a DPDK app when iommu is enabled, it requires to
> > add iommu=pt to the kernel command line. But using igb_uio driver
> > makes DMAR errors because the device has not an IOMMU domain.
>
> Please help to understand the scope of the problem,
>
>
After reading your reply, I realize I could have explained it better.
First of all, this is related to SRIOV, exactly when the VFs are created.


> 1- How can you re-produce the problem?
>
>
Using a VF from a Intel card by a DPDK app in the host and a kernel >=
3.15. Although usually VFs are assigned to VMs, it could also be an option
to use VFs by the host.

BTW, I did not try to reproduce the problem with an Intel card. I triggered
this problem with an NFP, but because the problem behind, I bet that is
going to happen for an Intel one as well.



> 2- What happens get DMAR errors, is it prevents device work or some
> annoying error messages?
>

A DMAR error implies the device can not access to the DMA address given by
the host. I have experienced several situations where it is just that
device not being able to work at all, but it also has more global
implications and you need to reboot the system because it is unreliable. I
think it depends on how these DMAR errors are handled, but in any case,
this is a bad thing.


>
> 3- Can you please share the error messages?
>

With this problem you can expect something like this:

 559.163874] DMAR: DRHD: handling fault status reg 2
[ 559.165427] DMAR: DMAR:[DMA Read] Request device [82:08.0] fault addr
e7b73b000
[ 559.165427] DMAR:[fault reason 02] Present bit in context entry is clear
[ 568.367417] DMAR: DRHD: handling fault status reg 102
[ 568.369025] DMAR: DMAR:[DMA Read] Request device [82:08.1] fault addr
ebb73b000
[ 568.369025] DMAR:[fault reason 02] Present bit in context entry is clear
[ 571.773944] DMAR: DRHD: handling fault status reg 202
[ 571.775550] DMAR: DMAR:[DMA Read] Request device [82:08.2] fault addr
efb73b000
[ 571.775550] DMAR:[fault reason 02] Present bit in context entry is clear
[ 575.039654] DMAR: DRHD: handling fault status reg 302
[ 575.041259] DMAR: DMAR:[DMA Read] Request device [82:08.3] fault addr
f3b73b000
[ 575.041259] DMAR:[fault reason 02] Present bit in context entry is clear

There are different DMAR errors, sometimes referring to a specific address
being wrong. In this case it is related to the device not having a context
or a IOMMU domain.

Also note we got these errors for different devices/VFs. This was with a
DPDK app using several VFs.


>
>
> >
> > Since kernel 3.15, iommu=pt requires to use the internal kernel
> > DMA API for attaching the device to the IOMMU 1:1 mapping, aka
> > si_domain. Previous versions did attach the device to that
> > domain when intel iommu notifier was called.
>
> Again, what is not working since 3.15?
>

This specific case, yes. With older kernels, when VFs are created, IOMMU
code is executed (notifier chain callback) and if iommu=pt, the VF is
attached to the si_domain, this is the 1:1 mapping. But this has changed
with newer kernels, and after VFs are created they have no IOMMU domain at
all. The kernel expects the driver to implicitly create such a domain when
the kernel DMA API is used.


>
> >
> > This is not a problem if the driver does later some call to the
> > DMA API because the mapping can be done then. But DPDK apps do
> > not use that DMA API at all.
>
> Is this same/similar with: http://dpdk.org/dev/patchwork/patch/12654/
>
>
That case was another issue regarding IOMMU and iommu=pt. The problem there
was when you detach a VF from a VM, but the VF was initially attached to
the si_domain because the kernel did so. The patch helped to attach the VF
again to that domain when binding to the UIO.

Looking at that patch now (I did comment on it then), it just solved the
problem if the VF was detach form the UIO, something that could be easily
forgotten or simply not done because, apparently, it is not needed.

What about to use VFIO?

With that previous patch, it was not enough. I do not remember the details
now, and I'm not sure if VFIO created another IOMMU domain if the device
had one, but it could leave the device without an IOMMU domain after the
first use.

In this particular case, VFIO would work, because the device gets its own
IOMMU domain. But there are two main problems if this is not fixed when
using UIO:

1) UIO is one of the two options for working with IOMMU. We all agree VFIO
is the right one for IOMMU, but as long as UIO is still an option, that
should be fixed.

2) Some installations need to work with and without IOMMU. Having same
module for both cases makes things simpler and therefore they use UIO
instead of VFIO.



> >
> > Doing this dma map and unmap is harmless even when iommu is not
> > enabled at all.
> >
> > Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> <...>
>
> Thanks,
> ferruh
>
>
>

Re: [dpdk-dev] [PATCH] igb_uio: map dummy dma forcing iommu domain attachment

[Ferruh Yigit]

On 2/8/2017 11:54 AM, Alejandro Lucero wrote:
> Hi Ferruh,
> 
> On Tue, Feb 7, 2017 at 3:59 PM, Ferruh Yigit <ferruh.yigit@intel.com
> <mailto:ferruh.yigit@intel.com>> wrote:
> 
>     Hi Alejandro,
> 
>     On 1/18/2017 12:27 PM, Alejandro Lucero wrote:
>     > For using a DPDK app when iommu is enabled, it requires to
>     > add iommu=pt to the kernel command line. But using igb_uio driver
>     > makes DMAR errors because the device has not an IOMMU domain.
> 
>     Please help to understand the scope of the problem,
> 
> 
> After reading your reply, I realize I could have explained it better.
> First of all, this is related to SRIOV, exactly when the VFs are created.
>  
> 
>     1- How can you re-produce the problem?
> 
> 
> Using a VF from a Intel card by a DPDK app in the host and a kernel >=
> 3.15. Although usually VFs are assigned to VMs, it could also be an
> option to use VFs by the host. 
> 
> BTW, I did not try to reproduce the problem with an Intel card. I
> triggered this problem with an NFP, but because the problem behind, I
> bet that is going to happen for an Intel one as well.

I can able to reproduce the problem with ixgbe, by using VF on the host.

And I verified your patch fixes it, it cause device attached to a vfio
group.

So, I believe good to get this patch, but it is already to late for
17.02 release.
I suggest getting this one early 17.05, so it gives more time to test.

> 
>  
> 
>     2- What happens get DMAR errors, is it prevents device work or some
>     annoying error messages?
> 
> 
> A DMAR error implies the device can not access to the DMA address given
> by the host. I have experienced several situations where it is just that
> device not being able to work at all, but it also has more global
> implications and you need to reboot the system because it is unreliable.
> I think it depends on how these DMAR errors are handled, but in any
> case, this is a bad thing.

In my test, implication was device is not working.

>  
> 
> 
>     3- Can you please share the error messages?
> 
> 
> With this problem you can expect something like this:
> 
>  559.163874] DMAR: DRHD: handling fault status reg 2
> [ 559.165427] DMAR: DMAR:[DMA Read] Request device [82:08.0] fault addr
> e7b73b000
> [ 559.165427] DMAR:[fault reason 02] Present bit in context entry is clear
> [ 568.367417] DMAR: DRHD: handling fault status reg 102
> [ 568.369025] DMAR: DMAR:[DMA Read] Request device [82:08.1] fault addr
> ebb73b000
> [ 568.369025] DMAR:[fault reason 02] Present bit in context entry is clear
> [ 571.773944] DMAR: DRHD: handling fault status reg 202
> [ 571.775550] DMAR: DMAR:[DMA Read] Request device [82:08.2] fault addr
> efb73b000
> [ 571.775550] DMAR:[fault reason 02] Present bit in context entry is clear
> [ 575.039654] DMAR: DRHD: handling fault status reg 302
> [ 575.041259] DMAR: DMAR:[DMA Read] Request device [82:08.3] fault addr
> f3b73b000
> [ 575.041259] DMAR:[fault reason 02] Present bit in context entry is clear
> 
> There are different DMAR errors, sometimes referring to a specific
> address being wrong. In this case it is related to the device not having
> a context or a IOMMU domain.
> 
> Also note we got these errors for different devices/VFs. This was with a
> DPDK app using several VFs.
>  
> 
> 
> 
>     >
>     > Since kernel 3.15, iommu=pt requires to use the internal kernel
>     > DMA API for attaching the device to the IOMMU 1:1 mapping, aka
>     > si_domain. Previous versions did attach the device to that
>     > domain when intel iommu notifier was called.
> 
>     Again, what is not working since 3.15?
> 
> 
> This specific case, yes. With older kernels, when VFs are created, IOMMU
> code is executed (notifier chain callback) and if iommu=pt, the VF is
> attached to the si_domain, this is the 1:1 mapping. But this has changed
> with newer kernels, and after VFs are created they have no IOMMU domain
> at all. The kernel expects the driver to implicitly create such a domain
> when the kernel DMA API is used.

Thanks again for clarification.
What will be the effect of your patch for kernel < 3.15, should your
update be protected with a kernel version check, or is it safe for all?

>  
> 
> 
>     >
>     > This is not a problem if the driver does later some call to the
>     > DMA API because the mapping can be done then. But DPDK apps do
>     > not use that DMA API at all.
> 
>     Is this same/similar with:
>     http://dpdk.org/dev/patchwork/patch/12654/
>     <http://dpdk.org/dev/patchwork/patch/12654/>
> 
>  
> That case was another issue regarding IOMMU and iommu=pt. The problem
> there was when you detach a VF from a VM, but the VF was initially
> attached to the si_domain because the kernel did so. The patch helped to
> attach the VF again to that domain when binding to the UIO.
> 
> Looking at that patch now (I did comment on it then), it just solved the
> problem if the VF was detach form the UIO, something that could be
> easily forgotten or simply not done because, apparently, it is not needed.

I also able to reproduce this case. When driver switched from igb_uio ->
vfio_pci -> igb_uio, it stops working, giving similar DMAR errors.

Your patch also fixing this, at least for my test. When unbind from
vfio_pci, iommu group removed, but binding igb_uio adds it back.

> 
> What about to use VFIO?
> 
> With that previous patch, it was not enough. I do not remember the
> details now, and I'm not sure if VFIO created another IOMMU domain if
> the device had one, but it could leave the device without an IOMMU
> domain after the first use.
> 
> In this particular case, VFIO would work, because the device gets its
> own IOMMU domain. But there are two main problems if this is not fixed
> when using UIO:
> 
> 1) UIO is one of the two options for working with IOMMU. We all agree
> VFIO is the right one for IOMMU, but as long as UIO is still an option,
> that should be fixed.
> 
> 2) Some installations need to work with and without IOMMU. Having same
> module for both cases makes things simpler and therefore they use UIO
> instead of VFIO.
> 
>  
> 
>     >
>     > Doing this dma map and unmap is harmless even when iommu is not
>     > enabled at all.
>     >
>     > Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com <mailto:alejandro.lucero@netronome.com>>

Tested-by: Ferruh Yigit <ferruh.yigit@intel.com>

>     <...>
> 
>     Thanks,
>     ferruh
> 
> 
> 

Re: [dpdk-dev] [PATCH] igb_uio: map dummy dma forcing iommu domain attachment

[Ferruh Yigit]

On 2/10/2017 7:03 PM, Ferruh Yigit wrote:
> On 2/8/2017 11:54 AM, Alejandro Lucero wrote:
>> Hi Ferruh,
>>
>> On Tue, Feb 7, 2017 at 3:59 PM, Ferruh Yigit <ferruh.yigit@intel.com
>> <mailto:ferruh.yigit@intel.com>> wrote:
>>
>>     Hi Alejandro,
>>
>>     On 1/18/2017 12:27 PM, Alejandro Lucero wrote:
>>     > For using a DPDK app when iommu is enabled, it requires to
>>     > add iommu=pt to the kernel command line. But using igb_uio driver
>>     > makes DMAR errors because the device has not an IOMMU domain.
>>
>>     Please help to understand the scope of the problem,
>>
>>
>> After reading your reply, I realize I could have explained it better.
>> First of all, this is related to SRIOV, exactly when the VFs are created.
>>  
>>
>>     1- How can you re-produce the problem?
>>
>>
>> Using a VF from a Intel card by a DPDK app in the host and a kernel >=
>> 3.15. Although usually VFs are assigned to VMs, it could also be an
>> option to use VFs by the host. 
>>
>> BTW, I did not try to reproduce the problem with an Intel card. I
>> triggered this problem with an NFP, but because the problem behind, I
>> bet that is going to happen for an Intel one as well.
> 
> I can able to reproduce the problem with ixgbe, by using VF on the host.
> 
> And I verified your patch fixes it, it cause device attached to a vfio
> group.

I want to send this in a separate mail, since not directly related to
your patch, but while I was testing with vfio-pci I get lower numbers
comparing to the igb_uio, which is unexpected AFAIK.

Most probably I am doing something wrong, but I would like to ask if are
you observing same behavior?

Thanks,
ferruh

Re: [dpdk-dev] [PATCH] igb_uio: map dummy dma forcing iommu domain attachment

[Alejandro Lucero]

On Fri, Feb 10, 2017 at 7:06 PM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 2/10/2017 7:03 PM, Ferruh Yigit wrote:
> > On 2/8/2017 11:54 AM, Alejandro Lucero wrote:
> >> Hi Ferruh,
> >>
> >> On Tue, Feb 7, 2017 at 3:59 PM, Ferruh Yigit <ferruh.yigit@intel.com
> >> <mailto:ferruh.yigit@intel.com>> wrote:
> >>
> >>     Hi Alejandro,
> >>
> >>     On 1/18/2017 12:27 PM, Alejandro Lucero wrote:
> >>     > For using a DPDK app when iommu is enabled, it requires to
> >>     > add iommu=pt to the kernel command line. But using igb_uio driver
> >>     > makes DMAR errors because the device has not an IOMMU domain.
> >>
> >>     Please help to understand the scope of the problem,
> >>
> >>
> >> After reading your reply, I realize I could have explained it better.
> >> First of all, this is related to SRIOV, exactly when the VFs are
> created.
> >>
> >>
> >>     1- How can you re-produce the problem?
> >>
> >>
> >> Using a VF from a Intel card by a DPDK app in the host and a kernel >=
> >> 3.15. Although usually VFs are assigned to VMs, it could also be an
> >> option to use VFs by the host.
> >>
> >> BTW, I did not try to reproduce the problem with an Intel card. I
> >> triggered this problem with an NFP, but because the problem behind, I
> >> bet that is going to happen for an Intel one as well.
> >
> > I can able to reproduce the problem with ixgbe, by using VF on the host.
> >
> > And I verified your patch fixes it, it cause device attached to a vfio
> > group.
>
> I want to send this in a separate mail, since not directly related to
> your patch, but while I was testing with vfio-pci I get lower numbers
> comparing to the igb_uio, which is unexpected AFAIK.
>
> Most probably I am doing something wrong, but I would like to ask if are
> you observing same behavior?
>

Can you tell me which test are you running?

Although both, igb_uio and vfio, allow to work with IOMMU, the first one
requires iommu=pt. It implies a single IOMMU domain already created by the
system with the 1:1 mapping being used.  With VFIO, a specific per device
IOMMU domain is created. Depending on how are you measuring performance,
that specific IOMMU domain creation by the DPDK app could have an impact,
but I do not think that should be really significant. But with IOMMU you
have the same problem than with MMU, there is a IOTLB for IOMMU as there is
a TLB for MMU. Depending on the app, some IOMMU/IOTLB contention is likely.
I have done some experiments and still investigating this during my spare
time. It would be worth a talk about this in the next DPDK meeting.


>
> Thanks,
> ferruh
>
>

Re: [dpdk-dev] [PATCH] igb_uio: map dummy dma forcing iommu domain attachment

[Ferruh Yigit]

On 2/13/2017 1:38 PM, Alejandro Lucero wrote:
> 
> 
> On Fri, Feb 10, 2017 at 7:06 PM, Ferruh Yigit <ferruh.yigit@intel.com
> <mailto:ferruh.yigit@intel.com>> wrote:
> 
>     On 2/10/2017 7:03 PM, Ferruh Yigit wrote:
>     > On 2/8/2017 11:54 AM, Alejandro Lucero wrote:
>     >> Hi Ferruh,
>     >>
>     >> On Tue, Feb 7, 2017 at 3:59 PM, Ferruh Yigit <ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>
>     >> <mailto:ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>>> wrote:
>     >>
>     >>     Hi Alejandro,
>     >>
>     >>     On 1/18/2017 12:27 PM, Alejandro Lucero wrote:
>     >>     > For using a DPDK app when iommu is enabled, it requires to
>     >>     > add iommu=pt to the kernel command line. But using igb_uio driver
>     >>     > makes DMAR errors because the device has not an IOMMU domain.
>     >>
>     >>     Please help to understand the scope of the problem,
>     >>
>     >>
>     >> After reading your reply, I realize I could have explained it better.
>     >> First of all, this is related to SRIOV, exactly when the VFs are created.
>     >>
>     >>
>     >>     1- How can you re-produce the problem?
>     >>
>     >>
>     >> Using a VF from a Intel card by a DPDK app in the host and a kernel >=
>     >> 3.15. Although usually VFs are assigned to VMs, it could also be an
>     >> option to use VFs by the host.
>     >>
>     >> BTW, I did not try to reproduce the problem with an Intel card. I
>     >> triggered this problem with an NFP, but because the problem behind, I
>     >> bet that is going to happen for an Intel one as well.
>     >
>     > I can able to reproduce the problem with ixgbe, by using VF on the host.
>     >
>     > And I verified your patch fixes it, it cause device attached to a vfio
>     > group.
> 
>     I want to send this in a separate mail, since not directly related to
>     your patch, but while I was testing with vfio-pci I get lower numbers
>     comparing to the igb_uio, which is unexpected AFAIK.
> 
>     Most probably I am doing something wrong, but I would like to ask if are
>     you observing same behavior?
> 
> 
> Can you tell me which test are you running?
> 
> Although both, igb_uio and vfio, allow to work with IOMMU, the first one
> requires iommu=pt. It implies a single IOMMU domain already created by
> the system with the 1:1 mapping being used.  With VFIO, a specific per
> device IOMMU domain is created. Depending on how are you measuring
> performance, that specific IOMMU domain creation by the DPDK app could
> have an impact, but I do not think that should be really significant.
> But with IOMMU you have the same problem than with MMU, there is a IOTLB
> for IOMMU as there is a TLB for MMU. Depending on the app, some
> IOMMU/IOTLB contention is likely. I have done some experiments and still
> investigating this during my spare time. It would be worth a talk about
> this in the next DPDK meeting.

After spending a few hours on it, still not sure about source of the
problem, and assume it is specific to my platform, otherwise we would
hear before.

The performance drop is huge to suspect from IOTLB.

With igb_uio, I am getting 10G line rate for 64bytes, ~14Mpps. When
switch to vfio-pci, it is ~1,5Mppps.

I am testing with "iommu=pt intel_iommu=on" kernel params.

Tried with kernel versions:
* 4.9.8
* 4.4.14
* 4.0.4

Tested with dpdk version:
* 17.02-rc3
* 16.11

Tested vfio-pci with kernel option "iommu=on intel_iommu=on" with 4.0.4
kernel.

All above gives low performance with vfio_pci, which does not make sense
to me, most probably doing a stupid mistake ...

Thanks,
ferruh

>  
> 
> 
>     Thanks,
>     ferruh
> 
> 

Re: [dpdk-dev] [PATCH] igb_uio: map dummy dma forcing iommu domain attachment

[Alejandro Lucero]

On Fri, Feb 10, 2017 at 7:03 PM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 2/8/2017 11:54 AM, Alejandro Lucero wrote:
> > Hi Ferruh,
> >
> > On Tue, Feb 7, 2017 at 3:59 PM, Ferruh Yigit <ferruh.yigit@intel.com
> > <mailto:ferruh.yigit@intel.com>> wrote:
> >
> >     Hi Alejandro,
> >
> >     On 1/18/2017 12:27 PM, Alejandro Lucero wrote:
> >     > For using a DPDK app when iommu is enabled, it requires to
> >     > add iommu=pt to the kernel command line. But using igb_uio driver
> >     > makes DMAR errors because the device has not an IOMMU domain.
> >
> >     Please help to understand the scope of the problem,
> >
> >
> > After reading your reply, I realize I could have explained it better.
> > First of all, this is related to SRIOV, exactly when the VFs are created.
> >
> >
> >     1- How can you re-produce the problem?
> >
> >
> > Using a VF from a Intel card by a DPDK app in the host and a kernel >=
> > 3.15. Although usually VFs are assigned to VMs, it could also be an
> > option to use VFs by the host.
> >
> > BTW, I did not try to reproduce the problem with an Intel card. I
> > triggered this problem with an NFP, but because the problem behind, I
> > bet that is going to happen for an Intel one as well.
>
> I can able to reproduce the problem with ixgbe, by using VF on the host.
>
> And I verified your patch fixes it, it cause device attached to a vfio
> group.
>
> So, I believe good to get this patch, but it is already to late for
> 17.02 release.
> I suggest getting this one early 17.05, so it gives more time to test.
>
>
Ok.


> >
> >
> >
> >     2- What happens get DMAR errors, is it prevents device work or some
> >     annoying error messages?
> >
> >
> > A DMAR error implies the device can not access to the DMA address given
> > by the host. I have experienced several situations where it is just that
> > device not being able to work at all, but it also has more global
> > implications and you need to reboot the system because it is unreliable.
> > I think it depends on how these DMAR errors are handled, but in any
> > case, this is a bad thing.
>
> In my test, implication was device is not working.
>
>
Yes, the device can not work for sure as it has not a IOMMU domain to work
with.
But sometimes the device is so badly after it, it does not help to create
one domain and attach the device to it, and just rebooting the system helps.


> >
> >
> >
> >     3- Can you please share the error messages?
> >
> >
> > With this problem you can expect something like this:
> >
> >  559.163874] DMAR: DRHD: handling fault status reg 2
> > [ 559.165427] DMAR: DMAR:[DMA Read] Request device [82:08.0] fault addr
> > e7b73b000
> > [ 559.165427] DMAR:[fault reason 02] Present bit in context entry is
> clear
> > [ 568.367417] DMAR: DRHD: handling fault status reg 102
> > [ 568.369025] DMAR: DMAR:[DMA Read] Request device [82:08.1] fault addr
> > ebb73b000
> > [ 568.369025] DMAR:[fault reason 02] Present bit in context entry is
> clear
> > [ 571.773944] DMAR: DRHD: handling fault status reg 202
> > [ 571.775550] DMAR: DMAR:[DMA Read] Request device [82:08.2] fault addr
> > efb73b000
> > [ 571.775550] DMAR:[fault reason 02] Present bit in context entry is
> clear
> > [ 575.039654] DMAR: DRHD: handling fault status reg 302
> > [ 575.041259] DMAR: DMAR:[DMA Read] Request device [82:08.3] fault addr
> > f3b73b000
> > [ 575.041259] DMAR:[fault reason 02] Present bit in context entry is
> clear
> >
> > There are different DMAR errors, sometimes referring to a specific
> > address being wrong. In this case it is related to the device not having
> > a context or a IOMMU domain.
> >
> > Also note we got these errors for different devices/VFs. This was with a
> > DPDK app using several VFs.
> >
> >
> >
> >
> >     >
> >     > Since kernel 3.15, iommu=pt requires to use the internal kernel
> >     > DMA API for attaching the device to the IOMMU 1:1 mapping, aka
> >     > si_domain. Previous versions did attach the device to that
> >     > domain when intel iommu notifier was called.
> >
> >     Again, what is not working since 3.15?
> >
> >
> > This specific case, yes. With older kernels, when VFs are created, IOMMU
> > code is executed (notifier chain callback) and if iommu=pt, the VF is
> > attached to the si_domain, this is the 1:1 mapping. But this has changed
> > with newer kernels, and after VFs are created they have no IOMMU domain
> > at all. The kernel expects the driver to implicitly create such a domain
> > when the kernel DMA API is used.
>
> Thanks again for clarification.
> What will be the effect of your patch for kernel < 3.15, should your
> update be protected with a kernel version check, or is it safe for all?
>
>
It is harmless. If the device got an IOMMU domain, like when iommu=pt in
older kernels,
the allocation does nothing. Note that we do not use kernel DMA API with
DPDK apps at all, and this allocation is just for forcing the binding to a
specific IOMMU domain if there is not such a binding yet.

If iommu is enabled, but iommu=pt is not set, that will create the iommu
domain, one specific for the device, but it will not avoid the DPDK app
failing afterwards, as that IOMMU domain will not be configured by anyone
properly. So, you need iommu=pt with igb_uio, or just to use VFIO.


> >
> >
> >
> >     >
> >     > This is not a problem if the driver does later some call to the
> >     > DMA API because the mapping can be done then. But DPDK apps do
> >     > not use that DMA API at all.
> >
> >     Is this same/similar with:
> >     http://dpdk.org/dev/patchwork/patch/12654/
> >     <http://dpdk.org/dev/patchwork/patch/12654/>
> >
> >
> > That case was another issue regarding IOMMU and iommu=pt. The problem
> > there was when you detach a VF from a VM, but the VF was initially
> > attached to the si_domain because the kernel did so. The patch helped to
> > attach the VF again to that domain when binding to the UIO.
> >
> > Looking at that patch now (I did comment on it then), it just solved the
> > problem if the VF was detach form the UIO, something that could be
> > easily forgotten or simply not done because, apparently, it is not
> needed.
>
> I also able to reproduce this case. When driver switched from igb_uio ->
> vfio_pci -> igb_uio, it stops working, giving similar DMAR errors.
>
> Your patch also fixing this, at least for my test. When unbind from
> vfio_pci, iommu group removed, but binding igb_uio adds it back.
>
>
> > What about to use VFIO?
> >
> > With that previous patch, it was not enough. I do not remember the
> > details now, and I'm not sure if VFIO created another IOMMU domain if
> > the device had one, but it could leave the device without an IOMMU
> > domain after the first use.
> >
> > In this particular case, VFIO would work, because the device gets its
> > own IOMMU domain. But there are two main problems if this is not fixed
> > when using UIO:
> >
> > 1) UIO is one of the two options for working with IOMMU. We all agree
> > VFIO is the right one for IOMMU, but as long as UIO is still an option,
> > that should be fixed.
> >
> > 2) Some installations need to work with and without IOMMU. Having same
> > module for both cases makes things simpler and therefore they use UIO
> > instead of VFIO.
> >
> >
> >
> >     >
> >     > Doing this dma map and unmap is harmless even when iommu is not
> >     > enabled at all.
> >     >
> >     > Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com
> <mailto:alejandro.lucero@netronome.com>>
>
> Tested-by: Ferruh Yigit <ferruh.yigit@intel.com>
>
> >     <...>
> >
> >     Thanks,
> >     ferruh
> >
> >
> >
>
>

Re: [dpdk-dev] [PATCH] igb_uio: map dummy dma forcing iommu domain attachment

[Ferruh Yigit]

On 1/18/2017 12:27 PM, Alejandro Lucero wrote:
> For using a DPDK app when iommu is enabled, it requires to
> add iommu=pt to the kernel command line. But using igb_uio driver
> makes DMAR errors because the device has not an IOMMU domain.
> 
> Since kernel 3.15, iommu=pt requires to use the internal kernel
> DMA API for attaching the device to the IOMMU 1:1 mapping, aka
> si_domain. Previous versions did attach the device to that
> domain when intel iommu notifier was called.
> 
> This is not a problem if the driver does later some call to the
> DMA API because the mapping can be done then. But DPDK apps do
> not use that DMA API at all.
> 
> Doing this dma map and unmap is harmless even when iommu is not
> enabled at all.
> 
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>

Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>

(I suggest getting this early in 17.05 release, so it can be tested more)

Re: [dpdk-dev] [PATCH] igb_uio: map dummy dma forcing iommu domain attachment

[Thomas Monjalon]

2017-02-17 12:29, Ferruh Yigit:
> On 1/18/2017 12:27 PM, Alejandro Lucero wrote:
> > For using a DPDK app when iommu is enabled, it requires to
> > add iommu=pt to the kernel command line. But using igb_uio driver
> > makes DMAR errors because the device has not an IOMMU domain.
> > 
> > Since kernel 3.15, iommu=pt requires to use the internal kernel
> > DMA API for attaching the device to the IOMMU 1:1 mapping, aka
> > si_domain. Previous versions did attach the device to that
> > domain when intel iommu notifier was called.
> > 
> > This is not a problem if the driver does later some call to the
> > DMA API because the mapping can be done then. But DPDK apps do
> > not use that DMA API at all.
> > 
> > Doing this dma map and unmap is harmless even when iommu is not
> > enabled at all.
> > 
> > Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> 
> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>
> 
> (I suggest getting this early in 17.05 release, so it can be tested more)

Sorry, I've missed it in early days of 17.05. Let's add it now.
Ferruh, Alejandro, do not hesitate to ping next time.

Applied, thanks