From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from g9t5008.houston.hpe.com (g9t5008.houston.hpe.com [15.241.48.72]) by dpdk.org (Postfix) with ESMTP id 909A42674 for ; Thu, 16 Aug 2018 11:50:58 +0200 (CEST) Received: from G4W9119.americas.hpqcorp.net (exchangepmrr1.us.hpecorp.net [16.210.20.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by g9t5008.houston.hpe.com (Postfix) with ESMTPS id A8A4C4F for ; Thu, 16 Aug 2018 09:50:57 +0000 (UTC) Received: from G4W9120.americas.hpqcorp.net (2002:10d2:150f::10d2:150f) by G4W9119.americas.hpqcorp.net (2002:10d2:14d6::10d2:14d6) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Thu, 16 Aug 2018 09:50:57 +0000 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (15.241.52.12) by G4W9120.americas.hpqcorp.net (16.210.21.15) with Microsoft SMTP Server (TLS) id 15.0.1367.3 via Frontend Transport; Thu, 16 Aug 2018 09:50:57 +0000 Received: from TU4PR8401MB1326.NAMPRD84.PROD.OUTLOOK.COM (10.169.51.18) by TU4PR8401MB1293.NAMPRD84.PROD.OUTLOOK.COM (10.169.51.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1038.25; Thu, 16 Aug 2018 09:50:56 +0000 Received: from TU4PR8401MB1326.NAMPRD84.PROD.OUTLOOK.COM ([fe80::e445:e812:1b4b:350c]) by TU4PR8401MB1326.NAMPRD84.PROD.OUTLOOK.COM ([fe80::e445:e812:1b4b:350c%10]) with mapi id 15.20.1038.025; Thu, 16 Aug 2018 09:50:56 +0000 From: "Raman, Sandeep" To: "users@dpdk.org" Thread-Topic: vfio-pci: probe of 0000:00:07.0 failed with error -22-KVM guest Thread-Index: AdQ1QoDYf02EsvKBTg+LJIoMoya8rQ== Date: Thu, 16 Aug 2018 09:50:55 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=sandeepr@hpe.com; x-originating-ip: [16.242.235.139] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; TU4PR8401MB1293; 6:TPWP5VJ6rFGeOJj6UwqQwam4w1Isz9yYrs3hquVb+3FKiiL5uSZsGKF5rLauRiOmGAjnI6NhkK4zQqaFNhihBZ1x+vbCEuq1XdTsZO6YSThv00TREhqzPr3D3TlKdDSlXCEXj4o3T5BAb28MLHTxjlGpmV6VM+HWc0jC71buOMLLpzBPZCLqJNgoFHhO2hMYp9fJDozWGroMHNxxgiVcg/K7/Urh/44BXZZGLCRtoOO+mLic0qJv64Boynay3Mk3pm5y1L5Zx1oqs7smdIQQujVjzVfQ/ENkiuYtHqmOnhzX/NvNNpVZB1LhBrL95ohD64ExJVYnfAIR8kL3p8pjNLVq+fjXjP6Evw5dVq5KyE9xt83ybTYQkNuI+YtTTi7vBd4fPFLYktfCitKBwJUDu/IB9Mq5N7O7seSmiy35A1yyESX0bMjqwbEVxiZT8ix3PPqMKih1uEcK5xqGEBZE+w==; 5:t1ig9lbLTXX3o9VMk1LbaNOECKJkwtVx8p9O4Gl9lL38Qz4/WUji+5FFN+/0nDupra6h1uBq0l5Sg5lxnnkci2YHYLLrzXxpMEz1ai75SikaDLBq4tbIxcQwH4ORiuXNFhsEbFd5uHSRHLSVdZTnkaOP8jEQrRWa8bOctEM+WxQ=; 7:18+gk82s6sSOZK0dtWj2QB3Z/MHpmP0pXH2Ywo9yaTb85I4KtJBlYu7KtP11pchWYNM5LZtIz5Tp4moLn7cyrqwN8NCzuYlbNy+kbIIS9Fb62bDQJRYksLM28e2BzeDyIIwcUpw0EoL8s+XHkOVxBd/WYERXJQvFmOMu6pvPubeFE58k42R0SV8/IgLwGVOu8DXutiCBjrvUAI+gSfT85uCFf48RU+593ocQi4v248dpiEC/1BakjzkwlmFlKTT7 x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 7bf3dd12-464f-41af-d36a-08d6035dc334 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989137)(4534165)(4627221)(201703031133081)(201702281549075)(8990107)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:TU4PR8401MB1293; x-ms-traffictypediagnostic: TU4PR8401MB1293: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(28532068793085)(108984395545644)(21748063052155)(228905959029699); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(3231311)(944501410)(52105095)(3002001)(6055026)(149027)(150027)(6041310)(20161123560045)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123558120)(201708071742011)(7699016); SRVR:TU4PR8401MB1293; BCL:0; PCL:0; RULEID:; SRVR:TU4PR8401MB1293; x-forefront-prvs: 07665BE9D1 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(366004)(376002)(396003)(346002)(39860400002)(136003)(189003)(199004)(7696005)(5250100002)(790700001)(9326002)(99286004)(25786009)(5630700001)(105586002)(2501003)(86362001)(316002)(8936002)(54896002)(102836004)(5640700003)(6116002)(3846002)(575784001)(6916009)(5660300001)(6506007)(6306002)(55016002)(236005)(26005)(68736007)(53936002)(6436002)(9686003)(186003)(2906002)(106356001)(8676002)(81156014)(256004)(14454004)(7736002)(478600001)(966005)(74316002)(81166006)(1730700003)(606006)(2351001)(21615005)(486006)(66066001)(2900100001)(476003)(33656002)(97736004); DIR:OUT; SFP:1102; SCL:1; SRVR:TU4PR8401MB1293; H:TU4PR8401MB1326.NAMPRD84.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: hpe.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: ph4vzaybZhvZupwrldycH5VQOikeL8xEcLgeWASgrZ6WG7Y9Q5+U7fGNtTLr0Y6p3u2eOfnIYkG+IjhOgcKNQ+Yx0aR+UqDUHcWoMg10C5qgBwPAje+IWrfALRiun923AF+DRn4+K7z5Kxjrd4U2uOQUvgK4H+hKq2K+HcKppFx/bXAIQLq9HYoCyxldDAddnquE1aH0STujJYy4ZV7eV2UgRlH0Kbx+JrCovEjnHGXmrtZd1bMnVKGgwtKzWDydn7bTtuThLtL3PS3TOknTqC0pX4WIS1bkFANjXHbixfboF6hzZQK5JzUpI+nxzfVAUhAwB9mSyU6GYXIB2hNHzS+d/VwCdCZcPCtOKY8LGCM= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 7bf3dd12-464f-41af-d36a-08d6035dc334 X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Aug 2018 09:50:55.9983 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 105b2061-b669-4b31-92ac-24d304d195dc X-MS-Exchange-Transport-CrossTenantHeadersStamped: TU4PR8401MB1293 X-OriginatorOrg: hpe.com X-Mailman-Approved-At: Tue, 21 Aug 2018 22:26:00 +0200 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] vfio-pci: probe of 0000:00:07.0 failed with error -22-KVM guest X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Aug 2018 09:50:59 -0000 Hi, I am trying to bind SRIOV VF to a kvm guest with vfio-pci module. DPDK vers= ion is 17.11. Both host and guest OS is RHEL 7.5. On KVM guest: [root@rh75vm ~]# cat /proc/cmdline BOOT_IMAGE=3D/vmlinuz-3.10.0-862.11.6.el7.x86_64 root=3D/dev/mapper/rhel-ro= ot ro crashkernel=3Dauto rd.lvm.lv=3Drhel/root rd.lvm.lv=3Drhel/swap LANG= =3Den_US.UTF-8 default_hugepagesz=3D1GB hugepagesz=3D1G hugepages=3D8 conso= le=3DttyS0,115200 [root@rh75vm ~]# dpdk-devbind --status-dev net Network devices using DPDK-compatible driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Network devices using kernel driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:00:03.0 'Virtio network device 1000' if=3Deth0 drv=3Dvirtio-pci unused= =3D *Active* 0000:00:07.0 'Ethernet Virtual Function 700 Series 154c' if=3Dens7 drv=3Di4= 0evf unused=3D 0000:00:08.0 'Ethernet Virtual Function 700 Series 154c' if=3Dens8 drv=3Di4= 0evf unused=3D Other Network devices =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D [root@rh75vm ~]# modprobe vfio-pci [root@rh75vm ~]# lsmod |grep vfio vfio_pci 41312 0 vfio_iommu_type1 22300 0 vfio 32695 2 vfio_iommu_type1,vfio_pci irqbypass 13503 1 vfio_pci [root@rh75vm ~]# dpdk-devbind -b vfio-pci 00:07.0 00:08.0 Error: bind failed for 0000:00:07.0 - Cannot bind to driver vfio-pci Error: unbind failed for 0000:00:07.0 - Cannot open /sys/bus/pci/drivers//u= nbind [root@rh75vm ~]# tailf -n3 /var/log/messages Aug 16 05:35:44 rh75vm systemd: Starting Session 1 of user root. Aug 16 05:36:55 rh75vm kernel: VFIO - User Level meta-driver version: 0.3 Aug 16 05:37:09 rh75vm kernel: vfio-pci: probe of 0000:00:07.0 failed with = error -22 On host: /proc/cmdline: BOOT_IMAGE=3D/vmlinuz-3.10.0-862.11.6.el7.x86_64 root=3D/dev/mapper/rhel-ro= ot ro crashkernel=3Dauto rd.lvm.lv=3Drhel/root rd.lvm.lv=3Drhel/swap rhgb q= uiet LANG=3Den_US.UTF-8 default_hugepagesz=3D1GB hugepagesz=3D1G hugepages= =3D20 isolcpus=3D1-15,17-31 rcu_nocbs=3D1-15,17-31 nohz_full=3D1-15,17-31 i= ntel_iommu=3Don iommu=3Dpt selinux=3D0 enforcing=3D0 processor.max_cstate= =3D0 intel_pstate=3Ddisable hpet=3Ddisable nosoftlockup intel_idle.max_csta= te=3D0 mce=3Dignore_ce audit=3D0 dpdk-devbind --status-dev net Network devices using DPDK-compatible driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:86:02.0 'Ethernet Virtual Function 700 Series 154c' drv=3Dvfio-pci unu= sed=3D 0000:86:0a.0 'Ethernet Virtual Function 700 Series 154c' drv=3Dvfio-pci unu= sed=3D Network devices using kernel driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:02:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=3Deno1 drv= =3Dtg3 unused=3Dvfio-pci 0000:02:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=3Deno2 drv= =3Dtg3 unused=3Dvfio-pci 0000:02:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=3Deno3 drv= =3Dtg3 unused=3Dvfio-pci 0000:02:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=3Deno4 drv= =3Dtg3 unused=3Dvfio-pci 0000:86:00.0 'Ethernet Controller XXV710 for 25GbE SFP28 158b' if=3Dens5f0 = drv=3Di40e unused=3Dvfio-pci 0000:86:00.1 'Ethernet Controller XXV710 for 25GbE SFP28 158b' if=3Dens5f1 = drv=3Di40e unused=3Dvfio-pci Other Network devices =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D dmesg: [ 0.000000] DMAR: IOMMU enabled [ 2.020878] DMAR: Hardware identity mapping for device 0000:86:00.0 [ 2.020881] DMAR: Hardware identity mapping for device 0000:86:00.1 [ 2.025617] DMAR: Intel(R) Virtualization Technology for Directed I/O [ 2.048734] iommu: Adding device 0000:86:00.0 to group 59 [ 2.048788] iommu: Adding device 0000:86:00.1 to group 60 [ 2.136560] pci 0000:86:00.0: Signaling PME through PCIe PME interrupt [ 2.136562] pci 0000:86:00.1: Signaling PME through PCIe PME interrupt [ 2.181458] DMAR: 32bit 0000:01:00.4 uses non-identity mapping [ 2.181839] DMAR: Setting identity map for device 0000:01:00.4 [0x8a6880= 00 - 0x8a688fff] [ 2.810190] DMAR: 32bit 0000:5c:00.0 uses non-identity mapping [ 2.841274] i40e 0000:86:00.0: fw 6.70.48807 api 1.7 nvm 6.00 0x800036cb= 1.1747.0 [ 3.082766] i40e 0000:86:00.0: MAC address: 48:df:37:36:1b:44 [ 3.092871] i40e 0000:86:00.0 eth3: NIC Link is Up, 25 Gbps Full Duplex,= Requested FEC: None, FEC: None, Autoneg: False, Flow Control: RX/TX [ 3.102039] i40e 0000:86:00.0: PCI-Express: Speed 8.0GT/s Width x8 [ 3.110709] i40e 0000:86:00.0: Features: PF-id[0] VFs: 64 VSIs: 66 QP: 3= 2 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA [ 3.136506] i40e 0000:86:00.1: fw 6.70.48807 api 1.7 nvm 6.00 0x800036cb= 1.1747.0 [ 3.373367] i40e 0000:86:00.1: MAC address: 48:df:37:36:1b:45 [ 1.510851] pci 0000:86:00.0: [8086:158b] type 00 class 0x020000 [ 1.510871] pci 0000:86:00.0: reg 0x10: [mem 0xf0000000-0xf0ffffff 64bit= pref] [ 1.510889] pci 0000:86:00.0: reg 0x1c: [mem 0xf2000000-0xf2007fff 64bit= pref] [ 1.510903] pci 0000:86:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref] [ 1.510955] pci 0000:86:00.0: PME# supported from D0 D3hot D3cold [ 1.510980] pci 0000:86:00.0: reg 0x184: [mem 0xd7fffa00000-0xd7fffa0fff= f 64bit pref] [ 1.510984] pci 0000:86:00.0: VF(n) BAR0 space: [mem 0xd7fffa00000-0xd7f= ffdfffff 64bit pref] (contains BAR0 for 64 VFs) [ 1.510998] pci 0000:86:00.0: reg 0x190: [mem 0xd7ffff00000-0xd7ffff03ff= f 64bit pref] [ 1.511000] pci 0000:86:00.0: VF(n) BAR3 space: [mem 0xd7ffff00000-0xd7f= ffffffff 64bit pref] (contains BAR3 for 64 VFs) [ 1.511193] pci 0000:86:00.1: [8086:158b] type 00 class 0x020000 [ 1.511212] pci 0000:86:00.1: reg 0x10: [mem 0xf1000000-0xf1ffffff 64bit= pref] [ 1.511229] pci 0000:86:00.1: reg 0x1c: [mem 0xf2008000-0xf200ffff 64bit= pref] [ 1.511243] pci 0000:86:00.1: reg 0x30: [mem 0x00000000-0x0007ffff pref] [ 1.511296] pci 0000:86:00.1: PME# supported from D0 D3hot D3cold [ 1.511317] pci 0000:86:00.1: reg 0x184: [mem 0xd7fff600000-0xd7fff60fff= f 64bit pref] [ 1.511320] pci 0000:86:00.1: VF(n) BAR0 space: [mem 0xd7fff600000-0xd7f= ff9fffff 64bit pref] (contains BAR0 for 64 VFs) [ 1.511334] pci 0000:86:00.1: reg 0x190: [mem 0xd7fffe00000-0xd7fffe03ff= f 64bit pref] [ 1.511337] pci 0000:86:00.1: VF(n) BAR3 space: [mem 0xd7fffe00000-0xd7f= ffefffff 64bit pref] (contains BAR3 for 64 VFs) [ 1.538878] pci 0000:86:00.0: BAR 6: assigned [mem 0xf2080000-0xf20fffff= pref] [ 1.538881] pci 0000:86:00.1: BAR 6: no space for [mem size 0x00080000 p= ref] [ 1.538884] pci 0000:86:00.1: BAR 6: failed to assign [mem size 0x000800= 00 pref] [ 1.538907] pci_bus 0000:86: resource 1 [mem 0xf0000000-0xf20fffff] [ 1.538910] pci_bus 0000:86: resource 2 [mem 0xd7fff600000-0xd7fffffffff= 64bit pref] I found a few threads reported with this error and the solution was to add = intel_iommu=3Don and iommu=3Dpt in the kernel. I have already added this. Any pointers what is causing the -22 error and how to solve this. http://mails.dpdk.org/archives/dev/2014-December/010455.html http://mails.dpdk.org/archives/users/2017-February/001544.html http://mails.dpdk.org/archives/users/2017-September/002433.html https://software.intel.com/en-us/forums/networking/topic/600159 Thanks, Sandeep.