[dpdk-dev] Issue when the kernel parameter intel_iommu=on is being used

[dpdk-dev] Issue when the kernel parameter intel_iommu=on is being used

[Sridhar S]

Hello,



I am using DPDK 1.5 for development of host pmd for device “Connect X3”.



I am observing issue  while the ConnectX3 device DMA to a memory which is
allocated with rte_memzone_reserve_aligned() API .

The issue(please refer ERROR below) has been observed if the system runs
with the kernel parameter “intel_iommu=on”.



ERROR :

dmar: DRHD: handling fault status reg 302

dmar: DMAR:[DMA Write] Request device [01:00.0] fault addr 4f883000

DMAR:[fault reason 01] Present bit in root entry is clear


The reported "fault Addr" is the physical address which was returned by the
Above API.



I don’t see any issue with the same code when the system up with kernel
parameter intel_iommu=off.




Can you share your comments on this issue?


Thanks in advance

Sri

Re: [dpdk-dev] Issue when the kernel parameter intel_iommu=on is being used

[François-Frédéric Ozog]

Hi,

To understand the issue, you may have a look at:
http://www.intel.com/content/www/us/en/intelligent-systems/intel-technology/
vt-directed-io-spec.html

When you have no IOMMU, "physical" address space is accessed directly by
hardware, so your core works.

When VT-d is active, there is DMA/IRQ remapping hardware layer between the
device and the memory/cpu. If you look at §3.4.3 of the spec, you that for
each device of each bus there is a context (enumerated at boot time,
leveraging BIOS/ACPI). For each device, you may have address translation
programmed so that DMA produced by hardware is actually mapped to a physical
address.

When you use the Linux kernel API for mapping DMA memory, Linux takes care
of the "details". 

For DPDK, documentation §5.6 Using Linux IOMMU Pass-Through to Run Intel®
DPDK with Intel® VT-d says that you should have iommu=pt kernel parameter
on. Do you have it ?

FF


> -----Message d'origine-----
> De : dev [mailto:dev-bounces@dpdk.org] De la part de Sridhar S
> Envoyé : dimanche 5 janvier 2014 13:38
> À : dev@dpdk.org
> Objet : [dpdk-dev] Issue when the kernel parameter intel_iommu=on is being
> used
> 
> Hello,
> 
> 
> 
> I am using DPDK 1.5 for development of host pmd for device “Connect X3”.
> 
> 
> 
> I am observing issue  while the ConnectX3 device DMA to a memory which is
> allocated with rte_memzone_reserve_aligned() API .
> 
> The issue(please refer ERROR below) has been observed if the system runs
> with the kernel parameter “intel_iommu=on”.
> 
> 
> 
> ERROR :
> 
> dmar: DRHD: handling fault status reg 302
> 
> dmar: DMAR:[DMA Write] Request device [01:00.0] fault addr 4f883000
> 
> DMAR:[fault reason 01] Present bit in root entry is clear
> 
> 
> The reported "fault Addr" is the physical address which was returned by
the
> Above API.
> 
> 
> 
> I don’t see any issue with the same code when the system up with kernel
> parameter intel_iommu=off.
> 
> 
> 
> 
> Can you share your comments on this issue?
> 
> 
> Thanks in advance
> 
> Sri

Re: [dpdk-dev] Issue when the kernel parameter intel_iommu=on is being used

[Sridhar S]

Hi,

Thanks for information.


If I use kernel parameters intel_iommu=on and iommu=pt, then the following
error has been observed.

####ERROR REPORT########
dmar: DRHD: handling fault status reg 2
dmar: DMAR:[DMA Write] Request device [01:00.0] fault addr 4f883000
DMAR:[fault reason 02] Present bit in context entry is clear
##########################

Does this mean no context entry hasn't been created for the Connect X3
device?
But, as per my understanding , the intel iommu code (intel_iommu.c) creates
root entries and context entries as per DMAR table which provided by BIOS
to OS.Also, it should create entry for Connect X3 device as well(?).

Or  the created memory via DPDK API is not belongs to Domain(VT-d) to which
Connect X3 device is assigned to?
Or  does this error code is generic?

Can you share your knowledge on this issue.

Thanks
Sri


On Sun, Jan 5, 2014 at 10:58 PM, François-Frédéric Ozog <ff@ozog.com> wrote:

> Hi,
>
> To understand the issue, you may have a look at:
>
> http://www.intel.com/content/www/us/en/intelligent-systems/intel-technology/
> vt-directed-io-spec.html
>
> When you have no IOMMU, "physical" address space is accessed directly by
> hardware, so your core works.
>
> When VT-d is active, there is DMA/IRQ remapping hardware layer between the
> device and the memory/cpu. If you look at §3.4.3 of the spec, you that for
> each device of each bus there is a context (enumerated at boot time,
> leveraging BIOS/ACPI). For each device, you may have address translation
> programmed so that DMA produced by hardware is actually mapped to a
> physical
> address.
>
> When you use the Linux kernel API for mapping DMA memory, Linux takes care
> of the "details".
>
> For DPDK, documentation §5.6 Using Linux IOMMU Pass-Through to Run Intel®
> DPDK with Intel® VT-d says that you should have iommu=pt kernel parameter
> on. Do you have it ?
>
> FF
>
>
> > -----Message d'origine-----
> > De : dev [mailto:dev-bounces@dpdk.org] De la part de Sridhar S
> > Envoyé : dimanche 5 janvier 2014 13:38
> > À : dev@dpdk.org
> > Objet : [dpdk-dev] Issue when the kernel parameter intel_iommu=on is
> being
> > used
> >
> > Hello,
> >
> >
> >
> > I am using DPDK 1.5 for development of host pmd for device “Connect X3”.
> >
> >
> >
> > I am observing issue  while the ConnectX3 device DMA to a memory which is
> > allocated with rte_memzone_reserve_aligned() API .
> >
> > The issue(please refer ERROR below) has been observed if the system runs
> > with the kernel parameter “intel_iommu=on”.
> >
> >
> >
> > ERROR :
> >
> > dmar: DRHD: handling fault status reg 302
> >
> > dmar: DMAR:[DMA Write] Request device [01:00.0] fault addr 4f883000
> >
> > DMAR:[fault reason 01] Present bit in root entry is clear
> >
> >
> > The reported "fault Addr" is the physical address which was returned by
> the
> > Above API.
> >
> >
> >
> > I don’t see any issue with the same code when the system up with kernel
> > parameter intel_iommu=off.
> >
> >
> >
> >
> > Can you share your comments on this issue?
> >
> >
> > Thanks in advance
> >
> > Sri
>
>