[dpdk-dev] DMAR errors when running testpmd

[dpdk-dev] DMAR errors when running testpmd

[Tomasz K]

Hi All

We're trying to run testpmd application on HP Proliant DL380P Gen 8 server.
We've enabled SR-IOV in BIOS and set appropriate flags when booting kernel
(iommu=pt intel_iommu=on)
The NIC we are using is 82599EB (2 ports, 10Gb SFP+)

When running testpmd application we always encouter DMAR error in dmesg

[  186.302866] dmar: DRHD: handling fault status reg 2
[  186.302872] dmar: DMAR:[DMA Read] Request device [07:00.1] fault addr
1f73220000
[  186.302872] DMAR:[fault reason 06] PTE Read access is not set
[  186.302875] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr
1f73200000
[  186.302875] DMAR:[fault reason 06] PTE Read access is not set
[  324.759520] dmar: DRHD: handling fault status reg 202
[  324.759525] dmar: DMAR:[DMA Read] Request device [07:00.1] fault addr
1f73220000
[  324.759525] DMAR:[fault reason 06] PTE Read access is not set
[  324.759528] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr
1f73200000
[  324.759528] DMAR:[fault reason 06] PTE Read access is not set

Has anyone encountered this issue?
Tried to search through gmane and google and the only solution was to
disable SR-IOV which we cannot do.

Thanks in advance
Tomasz

Host:
HP Proliant DL380p Gen 8 Server
Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz

NIC:
82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Re: [dpdk-dev] DMAR errors when running testpmd

[Burakov, Anatoly]

Hi Tomasz

It looks like you have your kernel booted with iommu=on. Please check your /proc/cmdline to make sure you didn't accidentally selected the wrong bootloader entry.

> We're trying to run testpmd application on HP Proliant DL380P Gen 8 server.
> We've enabled SR-IOV in BIOS and set appropriate flags when booting kernel
> (iommu=pt intel_iommu=on) The NIC we are using is 82599EB (2 ports, 10Gb
> SFP+)
> 
> When running testpmd application we always encouter DMAR error in dmesg
> 
> [  186.302866] dmar: DRHD: handling fault status reg 2 [  186.302872] dmar:
> DMAR:[DMA Read] Request device [07:00.1] fault addr
> 1f73220000
> [  186.302872] DMAR:[fault reason 06] PTE Read access is not set [
> 186.302875] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr
> 1f73200000
> [  186.302875] DMAR:[fault reason 06] PTE Read access is not set [
> 324.759520] dmar: DRHD: handling fault status reg 202 [  324.759525] dmar:
> DMAR:[DMA Read] Request device [07:00.1] fault addr
> 1f73220000
> [  324.759525] DMAR:[fault reason 06] PTE Read access is not set [
> 324.759528] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr
> 1f73200000
> [  324.759528] DMAR:[fault reason 06] PTE Read access is not set

Best regards,
Anatoly Burakov
DPDK SW Engineer

Re: [dpdk-dev] DMAR errors when running testpmd

[Tomasz K]

Hi Anatoly.

Yes I do have iommu=on set when booting kernel. This is one of
prerequisites to have SR-IOV running.
Below is the cmdline output

cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.11.0-20-generic
root=UUID=c70fa456-ee10-43e5-9f07-4dbb372dcee3 ro quiet splash iommu=pt
intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=4 vt.handoff=7

Thanks
Tomasz


2014-05-07 16:07 GMT+02:00 Burakov, Anatoly <anatoly.burakov@intel.com>:

> Hi Tomasz
>
> It looks like you have your kernel booted with iommu=on. Please check your
> /proc/cmdline to make sure you didn't accidentally selected the wrong
> bootloader entry.
>
> > We're trying to run testpmd application on HP Proliant DL380P Gen 8
> server.
> > We've enabled SR-IOV in BIOS and set appropriate flags when booting
> kernel
> > (iommu=pt intel_iommu=on) The NIC we are using is 82599EB (2 ports, 10Gb
> > SFP+)
> >
> > When running testpmd application we always encouter DMAR error in dmesg
> >
> > [ 186.302866] dmar: DRHD: handling fault status reg 2 [  186.302872]
> dmar:
> > DMAR:[DMA Read] Request device [07:00.1] fault addr
> > 1f73220000
> > [  186.302872] DMAR:[fault reason 06] PTE Read access is not set [
> > 186.302875] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr
> > 1f73200000
> > [  186.302875] DMAR:[fault reason 06] PTE Read access is not set [
> > 324.759520] dmar: DRHD: handling fault status reg 202 [  324.759525]
> dmar:
> > DMAR:[DMA Read] Request device [07:00.1] fault addr
> > 1f73220000
> > [  324.759525] DMAR:[fault reason 06] PTE Read access is not set [
> > 324.759528] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr
> > 1f73200000
> > [  324.759528] DMAR:[fault reason 06] PTE Read access is not set
>
> Best regards,
> Anatoly Burakov
> DPDK SW Engineer
>
>
>
>

Re: [dpdk-dev] DMAR errors when running testpmd

[Burakov, Anatoly]

Hi Tomasz,

Your words:

> We've enabled SR-IOV in BIOS and set appropriate flags when booting kernel
> (iommu=pt intel_iommu=on)

Your other words:

> Yes I do have iommu=on set when booting kernel.

Here lies your mistake :-) Boot your kernel with iommu to "pt" (iommu=pt intel_iommu=on) and everything will work. Thje "pt" option enables IOMMU only for VM's while "on" sets up your whole system to work through IOMMU (including host devices). However, both of these options enable SR-IOV. 

Best regards,
Anatoly Burakov
DPDK SW Engineer

Re: [dpdk-dev] DMAR errors when running testpmd

[Burakov, Anatoly]

Hi Tomasz, 

> 
> Here lies your mistake :-) Boot your kernel with iommu to "pt" (iommu=pt
> intel_iommu=on) and everything will work. Thje "pt" option enables IOMMU
> only for VM's while "on" sets up your whole system to work through IOMMU
> (including host devices). However, both of these options enable SR-IOV.
> 

Apologies, I misread your command-line output. It seems like you have everything set up correctly. I'm not sure what is wrong here, but the messages you're seeing are usually related to incorrect IOMMU boot parameters. Which DPDK version are you using?

Best regards,
Anatoly Burakov
DPDK SW Engineer

Re: [dpdk-dev] DMAR errors when running testpmd

[Tomasz K]

Hi Anatoly

I'm using dpdk-1.6.0r2 on Ubuntu 12.04 LTS with kernel 3.11.0-20-generic
#35~precise1-Ubuntu SMP

As for further investigation:
1. It doesn't matter whether DPDK uses 1GB or 2MB hugepages.
2. dmesg messages appear only when I invoke "start tx_first" in testpmd app
(so only when I try to send some packets)

The setup is very easy. 2 NICs connected with each other (port0 on NIC1 to
port0 on NIC2, same for port1). Each NIC iis on different server

Thanks
Tomasz


2014-05-07 16:45 GMT+02:00 Burakov, Anatoly <anatoly.burakov@intel.com>:

> Hi Tomasz,
>
> >
> > Here lies your mistake :-) Boot your kernel with iommu to "pt" (iommu=pt
> > intel_iommu=on) and everything will work. Thje "pt" option enables IOMMU
> > only for VM's while "on" sets up your whole system to work through IOMMU
> > (including host devices). However, both of these options enable SR-IOV.
> >
>
> Apologies, I misread your command-line output. It seems like you have
> everything set up correctly. I'm not sure what is wrong here, but the
> messages you're seeing are usually related to incorrect IOMMU boot
> parameters. Which DPDK version are you using?
>
> Best regards,
> Anatoly Burakov
> DPDK SW Engineer
>
>
>
>

Re: [dpdk-dev] DMAR errors when running testpmd

[Burakov, Anatoly]

Hi Tomasz,

> 2. dmesg messages appear only when I invoke "start tx_first" in testpmd app (so only when I try to send some packets)

Does receiving packets work? I would assume it doesn't, but just making sure.

Best regards,
Anatoly Burakov
DPDK SW Engineer

Re: [dpdk-dev] DMAR errors when running testpmd

[Tomasz K]

Hi Anatoly

You guessed right...it doesn't
"show port stats all" always shows zeros.


Thanks
Tomasz


2014-05-07 17:08 GMT+02:00 Burakov, Anatoly <anatoly.burakov@intel.com>:

> Hi Tomasz,
>
> > 2. dmesg messages appear only when I invoke "start tx_first" in testpmd
> app (so only when I try to send some packets)
>
> Does receiving packets work? I would assume it doesn't, but just making
> sure.
>
> Best regards,
> Anatoly Burakov
> DPDK SW Engineer
>
>
>
>

Re: [dpdk-dev] DMAR errors when running testpmd

[Burakov, Anatoly]

Hi Tomasz,

> You guessed right...it doesn't
> "show port stats all" always shows zeros.

As I said earlier, such errors are usually related to errors in boot parameters, but your kernel cmdline looks perfectly fine, so unless there's something really odd happening, I can't see this being at fault.

Another (rather unlikely, but I'll still mention it just in case) reason could be that you're using igb_uio module that is really old (I don't remember when igb_uio gained IOMMU support, probably 1.4.x), which is why I asked about DPDK version, but since you're using 1.6.0 you should be fine.

Other than that, I'm afraid I can't think of any reasons why this could be happening. Did you try this on another board with the same OS? 

Best regards,
Anatoly Burakov
DPDK SW Engineer

Re: [dpdk-dev] DMAR errors when running testpmd

[Tomasz K]

Hi Anatoly

It seems that the problem was Ubuntu distribution.
We switched to Ubuntu 14.04 Server Edition and sending packets through NIC
works like charm

Thanks
Tomasz

PS.
This is for google to index in case anyone else has similar problems
If You are using:
HP ProLiant DL380p Gen8
HP SmartArray p420
Intel 82599EB

Use Ubuntu 14.04 as Your host OS !!!




2014-05-07 17:24 GMT+02:00 Burakov, Anatoly <anatoly.burakov@intel.com>:

> Hi Tomasz,
>
> > You guessed right...it doesn't
> > "show port stats all" always shows zeros.
>
> As I said earlier, such errors are usually related to errors in boot
> parameters, but your kernel cmdline looks perfectly fine, so unless there's
> something really odd happening, I can't see this being at fault.
>
> Another (rather unlikely, but I'll still mention it just in case) reason
> could be that you're using igb_uio module that is really old (I don't
> remember when igb_uio gained IOMMU support, probably 1.4.x), which is why I
> asked about DPDK version, but since you're using 1.6.0 you should be fine.
>
> Other than that, I'm afraid I can't think of any reasons why this could be
> happening. Did you try this on another board with the same OS?
>
> Best regards,
> Anatoly Burakov
> DPDK SW Engineer
>
>
>
>

Re: [dpdk-dev] DMAR errors when running testpmd

[Yang, Tao Y]

If you don't test the DPDK in virtualization environment.
The easy way to fix it is to disable the VT-d in the BIOS.




-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Tomasz K
Sent: Thursday, May 8, 2014 3:18 PM
To: Burakov, Anatoly
Cc: dev
Subject: Re: [dpdk-dev] DMAR errors when running testpmd

Hi Anatoly

It seems that the problem was Ubuntu distribution.
We switched to Ubuntu 14.04 Server Edition and sending packets through NIC works like charm

Thanks
Tomasz

PS.
This is for google to index in case anyone else has similar problems If You are using:
HP ProLiant DL380p Gen8
HP SmartArray p420
Intel 82599EB

Use Ubuntu 14.04 as Your host OS !!!




2014-05-07 17:24 GMT+02:00 Burakov, Anatoly <anatoly.burakov@intel.com>:

> Hi Tomasz,
>
> > You guessed right...it doesn't
> > "show port stats all" always shows zeros.
>
> As I said earlier, such errors are usually related to errors in boot 
> parameters, but your kernel cmdline looks perfectly fine, so unless 
> there's something really odd happening, I can't see this being at fault.
>
> Another (rather unlikely, but I'll still mention it just in case) 
> reason could be that you're using igb_uio module that is really old (I 
> don't remember when igb_uio gained IOMMU support, probably 1.4.x), 
> which is why I asked about DPDK version, but since you're using 1.6.0 you should be fine.
>
> Other than that, I'm afraid I can't think of any reasons why this 
> could be happening. Did you try this on another board with the same OS?
>
> Best regards,
> Anatoly Burakov
> DPDK SW Engineer
>
>
>
>