Re: [dpdk-dev] DPDK doesn't work with iommu=pt

[Hiroshi Shimamoto <h-shimamoto@ct.jp.nec.com>]

Hi,

> Subject: RE: DPDK doesn't work with iommu=pt
> 
> Met the similar issue before.
> VT-d enabled? If so you may need to contact HP to upgrade the BIOS or you may disable VT-d and remove iommu=pt intel_iommu=on
> if you don't need VF function.

we need VT-d and it's enabled.
What we want to do is that use SR-IOV functionality and DPDK application concurrently on the same box.

thanks,
Hiroshi

> 
> >-----Original Message-----
> >From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Hiroshi Shimamoto
> >Sent: Friday, September 26, 2014 5:14 PM
> >To: dev@dpdk.org
> >Cc: Hayato Momma
> >Subject: [dpdk-dev] DPDK doesn't work with iommu=pt
> >
> >I encountered an issue that DPDK doesn't work with "iommu=pt
> >intel_iommu=on"
> >on HP ProLiant DL380p Gen8 server. I'm using the following environment;
> >
> >  HW: ProLiant DL380p Gen8
> >  CPU: E5-2697 v2
> >  OS: RHEL7
> >  kernel: kernel-3.10.0-123 and the latest kernel 3.17-rc6+
> >  DPDK: v1.7.1-53-gce5abac
> >  NIC: 82599ES
> >
> >When boot with "iommu=pt intel_iommu=on", I got the below message and no
> >packets are handled.
> >
> >  [  120.809611] dmar: DRHD: handling fault status reg 2
> >  [  120.809635] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr
> >aa010000
> >  DMAR:[fault reason 02] Present bit in context entry is clear
> >
> >How to reproduce;
> >just run testpmd
> ># ./testpmd -c 0xf -n 4 -- -i
> >
> >Configuring Port 0 (socket 0)
> >PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7ffff54eafc0
> >hw_ring=0x7ffff4200000 dma_addr=0xaa000000
> >PMD: ixgbe_dev_tx_queue_setup(): Using full-featured tx code path
> >PMD: ixgbe_dev_tx_queue_setup():  - txq_flags = 0
> >[IXGBE_SIMPLE_FLAGS=f01]
> >PMD: ixgbe_dev_tx_queue_setup():  - tx_rs_thresh = 32
> >[RTE_PMD_IXGBE_TX_MAX_BURST=32]
> >PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7ffff54ea740
> >hw_ring=0x7ffff4210000 dma_addr=0xaa010000
> >PMD: check_rx_burst_bulk_alloc_preconditions(): Rx Burst Bulk Alloc
> >Preconditions: rxq->rx_free_thresh=0, RTE_PMD_IXGBE_RX_MAX_BURST=32
> >PMD: ixgbe_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are not
> >satisfied, Scattered Rx is requested, or
> >RTE_LIBRTE_IXGBE_RX_ALLOW_BULK_ALLOC is not enabled (port=0, queue=0).
> >PMD: check_rx_burst_bulk_alloc_preconditions(): Rx Burst Bulk Alloc
> >Preconditions: rxq->rx_free_thresh=0, RTE_PMD_IXGBE_RX_MAX_BURST=32
> >
> >testpmd> start
> >  io packet forwarding - CRC stripping disabled - packets/burst=32
> >  nb forwarding cores=1 - nb forwarding ports=2
> >  RX queues=1 - RX desc=128 - RX free threshold=0
> >  RX threshold registers: pthresh=8 hthresh=8 wthresh=0
> >  TX queues=1 - TX desc=512 - TX free threshold=0
> >  TX threshold registers: pthresh=32 hthresh=0 wthresh=0
> >  TX RS bit threshold=0 - TXQ flags=0x0
> >
> >
> >and ping from another box to this server.
> ># ping6 -I eth2 ff02::1
> >
> >I got the below error message and no packet is received.
> >I couldn't see any increase RX/TX count in testpmt statistics
> >
> >testpmd> show port stats 0
> >
> >  ######################## NIC statistics for port 0
> >########################
> >  RX-packets: 6          RX-missed: 0          RX-bytes:  732
> >  RX-badcrc:  0          RX-badlen: 0          RX-errors: 0
> >  RX-nombuf:  0
> >  TX-packets: 0          TX-errors: 0          TX-bytes:  0
> >
> >#################################################################
> >###########
> >testpmd> show port stats 0
> >
> >  ######################## NIC statistics for port 0
> >########################
> >  RX-packets: 6          RX-missed: 0          RX-bytes:  732
> >  RX-badcrc:  0          RX-badlen: 0          RX-errors: 0
> >  RX-nombuf:  0
> >  TX-packets: 0          TX-errors: 0          TX-bytes:  0
> >
> >#################################################################
> >###########
> >
> >
> >The fault addr in error message must be RX DMA descriptor
> >
> >error message
> >  [  120.809635] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr
> >aa010000
> >
> >log in testpmd
> >  PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7ffff54ea740
> >hw_ring=0x7ffff4210000 dma_addr=0xaa010000
> >
> >I think the NIC received a packet in fifo and try to put into memory with DMA.
> >Before starting DMA, the NIC get the target address from RX descriptors in
> >RDBA register.
> >But accessing RX descriptors failed in IOMMU unit and reported it to the kernel.
> >
> >  DMAR:[fault reason 02] Present bit in context entry is clear
> >
> >The error message looks there is no valid entry in IOMMU.
> >
> >I think the following issue is very similar, but using Ubuntu14.04 couldn't fix in my
> >case.
> >http://thread.gmane.org/gmane.comp.networking.dpdk.devel/2281
> >
> >I tried Ubuntu14.04.1 and got the below error.
> >
> >  [  199.710191] dmar: DRHD: handling fault status reg 2
> >  [  199.710896] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr
> >7c24df000
> >  [  199.710896] DMAR:[fault reason 06] PTE Read access is not set
> >
> >Currently I could see this issue on HP ProLiant DL380p Gen8 only.
> >Is there any idea?
> >Has anyone noticed this issue?
> >
> >Note: we're thinking to use SR-IOV and DPDK app in the same box.
> >The box has 2 NICs, one for SR-IOV and pass through to VM, one (no SR-IOV) for
> >DPDK app in host.
> >
> >thanks,
> >Hiroshi