[dpdk-dev] DPDK doesn't work with iommu=pt

[Hiroshi Shimamoto <h-shimamoto@ct.jp.nec.com>]

I encountered an issue that DPDK doesn't work with "iommu=pt intel_iommu=on"
on HP ProLiant DL380p Gen8 server. I'm using the following environment;

  HW: ProLiant DL380p Gen8
  CPU: E5-2697 v2
  OS: RHEL7 
  kernel: kernel-3.10.0-123 and the latest kernel 3.17-rc6+
  DPDK: v1.7.1-53-gce5abac
  NIC: 82599ES

When boot with "iommu=pt intel_iommu=on", I got the below message and
no packets are handled.

  [  120.809611] dmar: DRHD: handling fault status reg 2
  [  120.809635] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr aa010000
  DMAR:[fault reason 02] Present bit in context entry is clear

How to reproduce;
just run testpmd
# ./testpmd -c 0xf -n 4 -- -i

Configuring Port 0 (socket 0)
PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7ffff54eafc0 hw_ring=0x7ffff4200000 dma_addr=0xaa000000
PMD: ixgbe_dev_tx_queue_setup(): Using full-featured tx code path
PMD: ixgbe_dev_tx_queue_setup():  - txq_flags = 0 [IXGBE_SIMPLE_FLAGS=f01]
PMD: ixgbe_dev_tx_queue_setup():  - tx_rs_thresh = 32 [RTE_PMD_IXGBE_TX_MAX_BURST=32]
PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7ffff54ea740 hw_ring=0x7ffff4210000 dma_addr=0xaa010000
PMD: check_rx_burst_bulk_alloc_preconditions(): Rx Burst Bulk Alloc Preconditions: rxq->rx_free_thresh=0, RTE_PMD_IXGBE_RX_MAX_BURST=32
PMD: ixgbe_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are not satisfied, Scattered Rx is requested, or RTE_LIBRTE_IXGBE_RX_ALLOW_BULK_ALLOC is not enabled (port=0, queue=0).
PMD: check_rx_burst_bulk_alloc_preconditions(): Rx Burst Bulk Alloc Preconditions: rxq->rx_free_thresh=0, RTE_PMD_IXGBE_RX_MAX_BURST=32

testpmd> start
  io packet forwarding - CRC stripping disabled - packets/burst=32
  nb forwarding cores=1 - nb forwarding ports=2
  RX queues=1 - RX desc=128 - RX free threshold=0
  RX threshold registers: pthresh=8 hthresh=8 wthresh=0
  TX queues=1 - TX desc=512 - TX free threshold=0
  TX threshold registers: pthresh=32 hthresh=0 wthresh=0
  TX RS bit threshold=0 - TXQ flags=0x0


and ping from another box to this server.
# ping6 -I eth2 ff02::1

I got the below error message and no packet is received.
I couldn't see any increase RX/TX count in testpmt statistics

testpmd> show port stats 0

  ######################## NIC statistics for port 0  ########################
  RX-packets: 6          RX-missed: 0          RX-bytes:  732
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 0
  RX-nombuf:  0
  TX-packets: 0          TX-errors: 0          TX-bytes:  0
  ############################################################################
testpmd> show port stats 0

  ######################## NIC statistics for port 0  ########################
  RX-packets: 6          RX-missed: 0          RX-bytes:  732
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 0
  RX-nombuf:  0
  TX-packets: 0          TX-errors: 0          TX-bytes:  0
  ############################################################################


The fault addr in error message must be RX DMA descriptor

error message
  [  120.809635] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr aa010000

log in testpmd
  PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7ffff54ea740 hw_ring=0x7ffff4210000 dma_addr=0xaa010000

I think the NIC received a packet in fifo and try to put into memory with DMA.
Before starting DMA, the NIC get the target address from RX descriptors in RDBA register.
But accessing RX descriptors failed in IOMMU unit and reported it to the kernel.

  DMAR:[fault reason 02] Present bit in context entry is clear

The error message looks there is no valid entry in IOMMU.

I think the following issue is very similar, but using Ubuntu14.04 couldn't fix in my case.
http://thread.gmane.org/gmane.comp.networking.dpdk.devel/2281

I tried Ubuntu14.04.1 and got the below error.

  [  199.710191] dmar: DRHD: handling fault status reg 2
  [  199.710896] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr 7c24df000
  [  199.710896] DMAR:[fault reason 06] PTE Read access is not set

Currently I could see this issue on HP ProLiant DL380p Gen8 only.
Is there any idea?
Has anyone noticed this issue?

Note: we're thinking to use SR-IOV and DPDK app in the same box.
The box has 2 NICs, one for SR-IOV and pass through to VM, one (no SR-IOV) for DPDK app in host.

thanks,
Hiroshi