From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-f44.google.com (mail-lf0-f44.google.com [209.85.215.44]) by dpdk.org (Postfix) with ESMTP id 3670D1B7BB for ; Wed, 31 Jan 2018 22:51:27 +0100 (CET) Received: by mail-lf0-f44.google.com with SMTP id g72so23006986lfg.5 for ; Wed, 31 Jan 2018 13:51:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=pBOJzXSt7JTWwZPW5uhbzMKYcy3gQLk/CCWUob3RuHQ=; b=O8jwN03RJJK3WGAwUsYfhK4xDKTLSstQVAGMjJWbdPWUdXe6LKoTm+vrZkGHDkTXXq B5HJT3XJ/Y7yGvB03CxbCnMupSDGEnOYfl2oa5ndXI2PREVKgLta5Da6vS8WijKEEA63 kkSmeYApVbOSo4YI8mmkQ0MBIyjp3dx/U+MfUvwKptlZuujDtIx5g1nWTgCNN23IR9TT MNwE7R7u6wobmK3dza2MmgsBHV92ErGQi10/4MtNHhiehoTQ7gykCOFNvwE8NfLbuiYx XRoYl88acggfCQjaHKsYazKotg8yNrCbtX/v5LSykFUOfdh9Jea6q/D68/cgds1xl89T 34HA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=pBOJzXSt7JTWwZPW5uhbzMKYcy3gQLk/CCWUob3RuHQ=; b=Z7DuNQUwfGEp5WJyNIcg+aNOySy5Cp5PzMYttLhFqzNfBwL1oeyb5fJ7vY6LiwWm0p tqwaJ4CuUZy4zjiLWKpoHn19Q5GcNtAUiJ+V83P6mTiEI7iTghOXENO32uKflks1W/dH FGbODxSECVdXKxbM73wBjNjFEamIIcC4LaYmN9j0I2HD5L103RDFA5fiZRtS9rwy3RZs rffrgedcwIZfESm/bU+LPD3ErSz2SoAoz4gEjPtqQnxuVSwut2HE4iOCCGPIuYpaEQJW GFHtCH7IdVBpz5JVFoLVJ1iodjwCQcmJlmYvjQ4dcJ/bfNjiep+QQH0ngq/QU0vf6Bq1 5LDA== X-Gm-Message-State: AKwxytd7NXlxNU5GFzq+PnrgIB/xLGkFQryEVUzbz/8V/7J+pGoidDR2 facufe0hW+S/77elcJU1EUqJjVnyHEpeUtCdcMU= X-Google-Smtp-Source: AH8x225gmMMvJSv8nZ7cB6z074YUBwdGLpuNJscV4J8JkndswO6WsuJCe9dmDDgffCqC88wqv71pNP3MjPT3Fh/r9e0= X-Received: by 10.46.4.196 with SMTP id a65mr8176168ljf.22.1517435486542; Wed, 31 Jan 2018 13:51:26 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.196.17 with HTTP; Wed, 31 Jan 2018 13:51:25 -0800 (PST) In-Reply-To: <10911b54-57ee-370b-a4f7-f34accf4811e@intel.com> References: <8ddb30a3-1253-ff60-20bb-b735fef5a91c@intel.com> <10911b54-57ee-370b-a4f7-f34accf4811e@intel.com> From: Ravi Kerur Date: Wed, 31 Jan 2018 13:51:25 -0800 Message-ID: To: "Burakov, Anatoly" Cc: dev@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jan 2018 21:51:27 -0000 On Wed, Jan 31, 2018 at 1:59 AM, Burakov, Anatoly wrote: > On 29-Jan-18 10:35 PM, Ravi Kerur wrote: > >> Hi Burakov, >> >> When using vfio-pci on host both VF and PF interfaces works fine with >> dpdk i.e. I don't see DMAR fault messages anymore. However, when I attach a >> VF interface to a VM and start DPDK with vfio-pci inside VM I still see >> DMAR fault messages on host. Both host and VM are booted with >> 'intel-iommu=on' on GRUB. Ping from VM with DPDK/vfio-pci doesn't work (I >> think it's expected because of DMAR faults), however, when VF interface >> uses ixgbevf driver ping works. >> >> Following are some details >> >> /*****************On VM***************/ >> dpdk-devbind -s >> >> Network devices using DPDK-compatible driver >> ============================================ >> 0000:00:07.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci >> unused=ixgbevf >> >> Network devices using kernel driver >> =================================== >> 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active* >> 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci >> 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci >> >> Other network devices >> ===================== >> >> >> Crypto devices using DPDK-compatible driver >> =========================================== >> >> >> Crypto devices using kernel driver >> ================================== >> >> >> Other crypto devices >> ==================== >> >> >> >> 00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller >> Virtual Function (rev 01) >> Subsystem: Intel Corporation 82599 Ethernet Controller Virtual >> Function >> Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- >> ParErr- Stepping- SERR- FastB2B- DisINTx+ >> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- >> SERR- > Region 0: Memory at fda00000 (64-bit, prefetchable) [size=16K] >> Region 3: Memory at fda04000 (64-bit, prefetchable) [size=16K] >> Capabilities: [70] MSI-X: Enable+ Count=3 Masked- >> Vector table: BAR=3 offset=00000000 >> PBA: BAR=3 offset=00002000 >> Capabilities: [a0] Express (v1) Root Complex Integrated >> Endpoint, MSI 00 >> DevCap: MaxPayload 128 bytes, PhantFunc 0 >> ExtTag- RBE- >> DevCtl: Report errors: Correctable- Non-Fatal- Fatal- >> Unsupported- >> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- >> MaxPayload 128 bytes, MaxReadReq 128 bytes >> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- >> TransPend- >> Capabilities: [100 v1] Advanced Error Reporting >> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- >> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- >> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >> UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- >> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- >> NonFatalErr- >> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- >> NonFatalErr- >> AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- >> ChkEn- >> Kernel driver in use: vfio-pci >> Kernel modules: ixgbevf >> >> /***************on Host*************/ >> dmesg | grep DMAR >> ... >> [ 978.268143] DMAR: DRHD: handling fault status reg 2 >> [ 978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault addr >> 33a128000 [fault reason 06] PTE Read access is not set >> [ 1286.677726] DMAR: DRHD: handling fault status reg 102 >> [ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault addr >> fb663000 [fault reason 06] PTE Read access is not set >> [ 1676.436145] DMAR: DRHD: handling fault status reg 202 >> [ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault addr >> 33a128000 [fault reason 06] PTE Read access is not set >> [ 1734.433649] DMAR: DRHD: handling fault status reg 302 >> [ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault addr >> 33a128000 [fault reason 06] PTE Read access is not set >> [ 2324.428938] DMAR: DRHD: handling fault status reg 402 >> [ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault addr >> 7770c000 [fault reason 06] PTE Read access is not set >> [ 2388.553640] DMAR: DRHD: handling fault status reg 502 >> [ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault addr >> 33a128000 [fault reason 06] PTE Read access is not set >> >> >> VM is started with >> >> qemu-system-x86_64 -enable-kvm -M q35,accel=kvm,kernel-irqchip=split >> -object iothread,id=iothread0 -device intel-iommu,intremap=on,device-iotlb=on,caching-mode=on >> -cpu host -daemonize -m 16G -smp 14 -uuid 0fc91c66-f0b1-11e7-acf4-525400123456 >> -name 212748-sriov-ravi-smac-alpha-SMAC10 -device >> ioh3420,id=root.1,chassis=1 -device ioh3420,id=root.2,chassis=2 -netdev >> tap,vhost=on,queues=2,ifname=vn-vn2_1_,downscript=no,id=vn-vn2_1_,script=no >> -device ioh3420,id=root.3,chassis=3 -device virtio-net-pci,netdev=vn-vn2_1 >> _,bus=root.3,ats=on,mq=on,vectors=6,mac=DE:AD:02:88:10:37,id=vn-vn2_1__dev >> -netdev tap,vhost=on,queues=2,ifname=vn-vn92_1_,downscript=no,id=vn-vn92_1_,script=no >> -device ioh3420,id=root.4,chassis=4 -device virtio-net-pci,mac=DE:AD:02:88 >> :10:38,netdev=vn-vn92_1_,bus=root.4,ats=on,mq=on,vectors=6,id=vn-vn92_1__dev >> -netdev tap,vhost=on,queues=2,ifname=vn-vn93_1_,downscript=no,id=vn-vn93_1_,script=no >> -device ioh3420,id=root.5,chassis=5 -device virtio-net-pci,mac=DE:AD:02:88 >> :10:39,netdev=vn-vn93_1_,bus=root.5,ats=on,mq=on,vectors=6,id=vn-vn93_1__dev >> -vnc :16,websocket=15916 -qmp tcp:127.0.0.1:12001 > >,server,nowait -chardev socket,id=charmonitor,path=/tmp/mon.12001,server,nowait >> -mon chardev=charmonitor,id=monitor -cdrom /var/venom/cloud_init/0fc91c66 >> -f0b1-11e7-acf4-525400123456.iso -*device vfio-pci,host=04:10.0* -drive >> file=/var/venom/instance_repo/test.img,if=none,id=drive-virt >> io-disk0,format=raw,aio=native,cache=none -balloon none -device >> virtio-blk-pci,scsi=off,iothread=iothread0,drive=drive- >> virtio-disk0,id=virtio-disk0,bus=root.1,ats=on,bootindex=1 >> >> Thanks. >> >> >> On Thu, Jan 25, 2018 at 2:49 AM, Burakov, Anatoly < >> anatoly.burakov@intel.com > wrote: >> >> On 24-Jan-18 7:13 PM, Ravi Kerur wrote: >> >> Hi Burakov, Thank you. I will try with vfio-pci driver. I am >> assuming it will work for both PF and VF interfaces since I am >> using both in my setup? >> >> Thanks. >> >> >> Yes, it should work for both PF and VF devices. >> >> >> On Wed, Jan 24, 2018 at 2:31 AM, Burakov, Anatoly >> >> > >> >> wrote: >> >> On 23-Jan-18 5:25 PM, Ravi Kerur wrote: >> >> Hi, >> >> I am running into an issue when DPDK is started with >> iommu on >> via GRUB >> command. Problem is not seen with regular kernel >> driver, error >> messages >> show when DPDK is started and happens for both PF and >> VF interfaces. >> >> I am using DPDK 17.05 so the patch proposed in the >> following link is >> available >> http://dpdk.org/ml/archives/dev/2017-February/057048.html >> >> > ev/2017-February/057048.html >> > >> >> Workaround is to use "iommu=pt" but I want iommu >> enabled in my >> setup. I >> checked BIOS for reserved memory(DMA RMRR for IXGBE) >> didn't get >> any details >> on it. >> >> Kindly let me know how to resolve this issue. >> >> Following are the details >> >> (1) Linux kernel 4.9 >> (2) DPDK 17.05 >> >> (3) IXGBE details >> ethtool -i enp4s0f0 (PF driver) >> driver: ixgbe >> version: 5.3.3 >> firmware-version: 0x800007b8, 1.1018.0 >> bus-info: 0000:04:00.0 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: yes >> supports-register-dump: yes >> supports-priv-flags: yes >> >> ethtool -i enp4s16f2 (VF driver) >> driver: ixgbevf >> version: 4.3.2 >> firmware-version: >> bus-info: 0000:04:10.2 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: no >> supports-register-dump: yes >> supports-priv-flags: no >> >> Bus info Device Class Description >> ============================== >> =========================== >> pci@0000:01:00.0 ens11f0 network 82599ES >> 10-Gigabit >> SFI/SFP+ >> Network Connection >> pci@0000:01:00.1 ens11f1 network 82599ES >> 10-Gigabit >> SFI/SFP+ >> Network Connection >> pci@0000:04:00.0 enp4s0f0 network 82599ES >> 10-Gigabit >> SFI/SFP+ >> Network Connection >> pci@0000:04:00.1 enp4s0f1 network 82599ES >> 10-Gigabit >> SFI/SFP+ >> Network Connection >> pci@0000:04:10.0 enp4s16 network Illegal >> Vendor ID >> pci@0000:04:10.2 enp4s16f2 network Illegal >> Vendor ID >> >> (4) DPDK bind interfaces >> >> # dpdk-devbind -s >> >> Network devices using DPDK-compatible driver >> ============================================ >> 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network >> Connection 10fb' >> drv=igb_uio unused=vfio-pci >> 0000:04:10.2 '82599 Ethernet Controller Virtual >> Function 10ed' >> drv=igb_uio >> unused=vfio-pci >> >> Network devices using kernel driver >> =================================== >> 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network >> Connection 10fb' >> if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci >> 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network >> Connection 10fb' >> if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci >> 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network >> Connection 10fb' >> if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci >> 0000:04:10.0 '82599 Ethernet Controller Virtual >> Function 10ed' >> if=enp4s16 >> drv=ixgbevf unused=igb_uio,vfio-pci >> 0000:06:00.0 'I210 Gigabit Network Connection 1533' >> if=eno1 drv=igb >> unused=igb_uio,vfio-pci *Active* >> >> Other Network devices >> ===================== >> >> >> ... >> >> (5) Kernel dmesg >> >> # dmesg | grep -e DMAR >> [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 >> (v01 ALASKA >> A M I >> 00000001 INTL 20091013) >> [ 0.000000] DMAR: IOMMU enabled >> [ 0.518747] DMAR: Host address width 46 >> [ 0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: >> 0x0 >> [ 0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver >> 1:0 cap >> d2078c106f0466 ecap f020df >> [ 0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: >> 0x1 >> [ 0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver >> 1:0 cap >> d2078c106f0466 ecap f020df >> [ 0.580611] DMAR: RMRR base: 0x0000007bbc6000 end: >> 0x0000007bbd4fff >> [ 0.593344] DMAR: ATSR flags: 0x0 >> [ 0.600178] DMAR: RHSA base: 0x000000c7ffc000 >> proximity >> domain: 0x0 >> [ 0.612905] DMAR: RHSA base: 0x000000fbffc000 >> proximity >> domain: 0x1 >> [ 0.625632] DMAR-IR: IOAPIC id 3 under DRHD base >> 0xfbffc000 >> IOMMU 0 >> [ 0.638522] DMAR-IR: IOAPIC id 1 under DRHD base >> 0xc7ffc000 >> IOMMU 1 >> [ 0.651426] DMAR-IR: IOAPIC id 2 under DRHD base >> 0xc7ffc000 >> IOMMU 1 >> [ 0.664324] DMAR-IR: HPET id 0 under DRHD base >> 0xc7ffc000 >> [ 0.675326] DMAR-IR: Queued invalidation will be >> enabled to >> support >> x2apic and Intr-remapping. >> [ 0.693805] DMAR-IR: Enabled IRQ remapping in x2apic >> mode >> [ 9.395170] DMAR: dmar1: Using Queued invalidation >> [ 9.405011] DMAR: Setting RMRR: >> [ 9.412006] DMAR: Setting identity map for device >> 0000:00:1d.0 >> [0x7bbc6000 - 0x7bbd4fff] >> [ 9.428569] DMAR: Prepare 0-16MiB unity mapping for >> LPC >> [ 9.439712] DMAR: Setting identity map for device >> 0000:00:1f.0 [0x0 - >> 0xffffff] >> [ 9.454684] DMAR: Intel(R) Virtualization Technology >> for >> Directed I/O >> [ 287.023068] DMAR: DRHD: handling fault status reg 2 >> [ 287.023073] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 18a260a000 [fault reason 06] PTE Read access is not set >> [ 287.023180] DMAR: DRHD: handling fault status reg 102 >> [ 287.023183] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 18a3010000 [fault reason 06] PTE Read access is not set >> [ 287.038250] DMAR: DRHD: handling fault status reg 202 >> [ 287.038252] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 18a3010000 [fault reason 06] PTE Read access is not set >> [ 288.170165] DMAR: DRHD: handling fault status reg 302 >> [ 288.170170] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 288.694496] DMAR: DRHD: handling fault status reg 402 >> [ 288.694499] DMAR: [DMA Read] Request device >> [04:10.2] fault addr >> 189069c000 [fault reason 06] PTE Read access is not set >> [ 289.927113] DMAR: DRHD: handling fault status reg 502 >> [ 289.927116] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 290.174275] DMAR: DRHD: handling fault status reg 602 >> [ 290.174279] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 292.174247] DMAR: DRHD: handling fault status reg 702 >> [ 292.174251] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 294.174227] DMAR: DRHD: handling fault status reg 2 >> [ 294.174230] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 296.174216] DMAR: DRHD: handling fault status reg 102 >> [ 296.174219] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [root@infradev-comp006.naw02.infradev.viasat.io >> >> > > ~] >> # >> >> Thanks. >> >> >> Hi Ravi, >> >> The "iommu=pt" workaround applies only when you want to use >> igb_uio >> driver. VFIO driver is able to fully utilize IOMMU without >> the need >> for pass-through mode. From your log i can see that some >> devices are >> bound to igb_uio while others are bound to vfio-pci. Just >> bind all >> of the devices you want to use with DPDK to vfio-pci and >> these >> errors should go away. >> >> -- Thanks, >> Anatoly >> >> >> >> >> -- Thanks, >> Anatoly >> >> >> > Hi Ravi, > > Using vfio-pci in IOMMU mode will only work if your VM provides IOMMU > emulation (it's a fairly recent development, so your QEMU must be of an > appropriate version - can't recall which one off the top of my head). > Otherwise you'd have to treat your VM as if it was a machine without IOMMU, > i.e. use noiommu mode for VFIO, or igb_uio driver. > > -- > Thanks, > Anatoly > Hi Anatoly, Thanks. I am following wiki link below which uses vIOMMU with DPDK as a use-case and instantiate VM as specified with Q35 chipset in Qemu. https://wiki.qemu.org/Features/VT-d Qemu-version is 2.11 Host kernel 4.9 Guest kernel 4.4 I can only guess that guest kernel needs an upgrade in my setup to work correctly, if versions on my setup rings a bell on not having support kindly let me know. When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on guest I get following error ... vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored ... in guest. Thanks.