From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-f51.google.com (mail-lf0-f51.google.com [209.85.215.51]) by dpdk.org (Postfix) with ESMTP id 416D21B017; Tue, 23 Jan 2018 18:25:38 +0100 (CET) Received: by mail-lf0-f51.google.com with SMTP id g72so1591828lfg.5; Tue, 23 Jan 2018 09:25:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=wKZ1K4jYIhtRsa+aQjW4kl5Qnkvo6JMetOlQHf7T42I=; b=IBE31uDRWqc8+LezZYGxh9TJ51emFG8FgFKLtY0QZGEa0mgxHg0L0dATq63CQLyvzE 6W06QN98MJnu+Od5rNXbRJgaHyqUYUrCY836ENCPB8rBse63oGZQwy31tH+k6KKFQXBi qBWjbQecPdJsJqNK8pu1bS0kXbdaddtV6vrrpLK7FEUFx4nV1spMzz19gwzdemc0DDsO e1j6no9t5MnX7fvenq/GL4C9lU+cuVEwMWLjI8dK+yiY6gT5K2tPgAMHwWEs7UlROa1h dh/rICz1gj8soHpDphihInFNax+eVOs2LHP1RkKY49imYdR8N7qcNKA7KZ13HvwB1/CX 8I7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=wKZ1K4jYIhtRsa+aQjW4kl5Qnkvo6JMetOlQHf7T42I=; b=oeSfe+70WEwrO7Z2uQpMwn3D4t8aZO3YGGcHEuFjmj9L2cZ4VxItJRMemEcRh19TMq WIVrvs5WEaWlYB7RQ7UmEjiG5s/euIZwHw7+hZxkiShzyRixICvq8iSNA7XpTEqcEyN3 DZEjEGaGExEtHvrojM2bWvgBVojf0OeictlsxDZjas1O0E1LFaFYTPYnaq+8i+8zXBsu tBcWIyCkeQ39jWKIjiP8ucsl8SckhD562e8IZDf57HcLNU1bz7rGb+0OXNcufK2+IqgZ HngWTw83TxIt9c0LS0k0ts5FVpW+/ETMx2XLGgwjeVkk9CGbIzhlv2fb9ki46pF7hWSm piLw== X-Gm-Message-State: AKwxyteK1Z/ifKvUpJOC+GzaAQdsfzzA6mAHLF/j4vmER0hKnohLNlRn sUJj17L0dyM80dhWvtFpBon3CIhv6NOVHQ3L8m4aTw== X-Google-Smtp-Source: AH8x224DPal3oOasIgQAlNN7SoOPB1gpB04kRxCR+ImJzVCEdLAnZEiUBAXWJr9Tcn6vc4aVmbPeq4so10jlLsF8VZA= X-Received: by 10.25.113.2 with SMTP id m2mr1474231lfc.71.1516728337070; Tue, 23 Jan 2018 09:25:37 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.196.17 with HTTP; Tue, 23 Jan 2018 09:25:36 -0800 (PST) From: Ravi Kerur Date: Tue, 23 Jan 2018 09:25:36 -0800 Message-ID: To: dev@dpdk.org, users@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] IXGBE, IOMMU DMAR DRHD handling fault issue X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jan 2018 17:25:38 -0000 Hi, I am running into an issue when DPDK is started with iommu on via GRUB command. Problem is not seen with regular kernel driver, error messages show when DPDK is started and happens for both PF and VF interfaces. I am using DPDK 17.05 so the patch proposed in the following link is available http://dpdk.org/ml/archives/dev/2017-February/057048.html Workaround is to use "iommu=pt" but I want iommu enabled in my setup. I checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get any details on it. Kindly let me know how to resolve this issue. Following are the details (1) Linux kernel 4.9 (2) DPDK 17.05 (3) IXGBE details ethtool -i enp4s0f0 (PF driver) driver: ixgbe version: 5.3.3 firmware-version: 0x800007b8, 1.1018.0 bus-info: 0000:04:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes ethtool -i enp4s16f2 (VF driver) driver: ixgbevf version: 4.3.2 firmware-version: bus-info: 0000:04:10.2 supports-statistics: yes supports-test: yes supports-eeprom-access: no supports-register-dump: yes supports-priv-flags: no Bus info Device Class Description ========================================================= pci@0000:01:00.0 ens11f0 network 82599ES 10-Gigabit SFI/SFP+ Network Connection pci@0000:01:00.1 ens11f1 network 82599ES 10-Gigabit SFI/SFP+ Network Connection pci@0000:04:00.0 enp4s0f0 network 82599ES 10-Gigabit SFI/SFP+ Network Connection pci@0000:04:00.1 enp4s0f1 network 82599ES 10-Gigabit SFI/SFP+ Network Connection pci@0000:04:10.0 enp4s16 network Illegal Vendor ID pci@0000:04:10.2 enp4s16f2 network Illegal Vendor ID (4) DPDK bind interfaces # dpdk-devbind -s Network devices using DPDK-compatible driver ============================================ 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=igb_uio unused=vfio-pci 0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed' drv=igb_uio unused=vfio-pci Network devices using kernel driver =================================== 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci 0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed' if=enp4s16 drv=ixgbevf unused=igb_uio,vfio-pci 0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1 drv=igb unused=igb_uio,vfio-pci *Active* Other Network devices ===================== ... (5) Kernel dmesg # dmesg | grep -e DMAR [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I 00000001 INTL 20091013) [ 0.000000] DMAR: IOMMU enabled [ 0.518747] DMAR: Host address width 46 [ 0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 [ 0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df [ 0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 [ 0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap d2078c106f0466 ecap f020df [ 0.580611] DMAR: RMRR base: 0x0000007bbc6000 end: 0x0000007bbd4fff [ 0.593344] DMAR: ATSR flags: 0x0 [ 0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0 [ 0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1 [ 0.625632] DMAR-IR: IOAPIC id 3 under DRHD base 0xfbffc000 IOMMU 0 [ 0.638522] DMAR-IR: IOAPIC id 1 under DRHD base 0xc7ffc000 IOMMU 1 [ 0.651426] DMAR-IR: IOAPIC id 2 under DRHD base 0xc7ffc000 IOMMU 1 [ 0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 [ 0.675326] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode [ 9.395170] DMAR: dmar1: Using Queued invalidation [ 9.405011] DMAR: Setting RMRR: [ 9.412006] DMAR: Setting identity map for device 0000:00:1d.0 [0x7bbc6000 - 0x7bbd4fff] [ 9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC [ 9.439712] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 9.454684] DMAR: Intel(R) Virtualization Technology for Directed I/O [ 287.023068] DMAR: DRHD: handling fault status reg 2 [ 287.023073] DMAR: [DMA Read] Request device [01:00.0] fault addr 18a260a000 [fault reason 06] PTE Read access is not set [ 287.023180] DMAR: DRHD: handling fault status reg 102 [ 287.023183] DMAR: [DMA Read] Request device [01:00.0] fault addr 18a3010000 [fault reason 06] PTE Read access is not set [ 287.038250] DMAR: DRHD: handling fault status reg 202 [ 287.038252] DMAR: [DMA Read] Request device [01:00.0] fault addr 18a3010000 [fault reason 06] PTE Read access is not set [ 288.170165] DMAR: DRHD: handling fault status reg 302 [ 288.170170] DMAR: [DMA Read] Request device [01:00.0] fault addr 1890754000 [fault reason 06] PTE Read access is not set [ 288.694496] DMAR: DRHD: handling fault status reg 402 [ 288.694499] DMAR: [DMA Read] Request device [04:10.2] fault addr 189069c000 [fault reason 06] PTE Read access is not set [ 289.927113] DMAR: DRHD: handling fault status reg 502 [ 289.927116] DMAR: [DMA Read] Request device [01:00.0] fault addr 1890754000 [fault reason 06] PTE Read access is not set [ 290.174275] DMAR: DRHD: handling fault status reg 602 [ 290.174279] DMAR: [DMA Read] Request device [01:00.0] fault addr 1890754000 [fault reason 06] PTE Read access is not set [ 292.174247] DMAR: DRHD: handling fault status reg 702 [ 292.174251] DMAR: [DMA Read] Request device [01:00.0] fault addr 1890754000 [fault reason 06] PTE Read access is not set [ 294.174227] DMAR: DRHD: handling fault status reg 2 [ 294.174230] DMAR: [DMA Read] Request device [01:00.0] fault addr 1890754000 [fault reason 06] PTE Read access is not set [ 296.174216] DMAR: DRHD: handling fault status reg 102 [ 296.174219] DMAR: [DMA Read] Request device [01:00.0] fault addr 1890754000 [fault reason 06] PTE Read access is not set [root@infradev-comp006.naw02.infradev.viasat.io ~] # Thanks.