From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-f65.google.com (mail-oi0-f65.google.com [209.85.218.65]) by dpdk.org (Postfix) with ESMTP id 19B012BF7 for ; Wed, 22 Aug 2018 15:20:20 +0200 (CEST) Received: by mail-oi0-f65.google.com with SMTP id c190-v6so3053379oig.6 for ; Wed, 22 Aug 2018 06:20:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=GM9771+9p5lumVrsSQf0PGZhrmyDVv9LtKSgXAruezs=; b=YcUhV3AT8tyHl41+nRVSqpF4bCWzoglr3dZMnYBH15k8HW/VjxaCwUlbphokV1agwk erZAgistsA0rl4kLRqNfVuD2m7UtI6Zclg0ErNNS/oe2aFw9fLQB+/x2EOkbeizmf/53 SUYSo+tUYjpxqL0OtWN4v0AxTZibySrmBFH7r6b75t93DjYUsYcasbd4Gt7UIGhinn0c TddwoXc4s7HQPqgs/3qv3qF5vaijQT/zx2ippik7ieEZjwlBZZt6dj/iM3lTR2ZZ8ieN apRs2NhqkuVH9yAERjSHf8jBf+zwP0ZmhtDrz00Baz4LK+H+zUVpEKrMJGkRrwycr1L3 ElVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=GM9771+9p5lumVrsSQf0PGZhrmyDVv9LtKSgXAruezs=; b=BEW1O3rukoxRAmzMckPzSkUduY0x6nZGn/ldGu+wYciMcLcuRUVvpm21xnsTwkJCJW SV8vVen/h6NJmwPxIy/u8rbTpBWVOCIzkR5ElX3769KLKygCWBYDbZZVfdRG6JI0qalL Z8xZQxUZsHWQRCXoW0sIcOLRQGH6vKwUSjz3zn1wcrium5lpcBClU2FfdK9INDMK1MRi xS992K+lA0WO9j22jzHovXJ0Bc8ZCxc3a7oIw7p0SdaH9yIjK7dhF2ygXwY+Upk3aiEV sW/J3Izh+yEXAA/rClgWQlt3sYwTTmT0/yIi2+GZ/ojiXmGjre/qpyWUtmfnUlLbAJr6 8uvw== X-Gm-Message-State: APzg51Avizjd5gfpQSoF84jaO1QybtDtOd6G7zMUFPs8n7/njO2nayBP kuInJRze45VTCuV+CtkWz5Yt2GnuwY1UyiYFnhw= X-Google-Smtp-Source: ANB0VdYAR0b/PQQ4v6pPB0yGhyesgdSy8RFKTTZfJ56CtAVZuf1qo+UukX+H9dKuKEczHAZymcdcSmfy7kiVM4VDK4w= X-Received: by 2002:aca:4083:: with SMTP id n125-v6mr3678698oia.167.1534944019289; Wed, 22 Aug 2018 06:20:19 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a4a:5cc9:0:0:0:0:0 with HTTP; Wed, 22 Aug 2018 06:20:18 -0700 (PDT) In-Reply-To: References: From: Rami Rosen Date: Wed, 22 Aug 2018 16:20:18 +0300 Message-ID: To: "Raman, Sandeep" Cc: "users@dpdk.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-users] vfio-pci: probe of 0000:00:07.0 failed with error -22-KVM guest X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 13:20:20 -0000 Hi Sandeep, How to you launch the guest ? if it by qemu, can you post the full command you are using? Regards, Rami Rosen On 16 August 2018 at 12:50, Raman, Sandeep wrote: > Hi, > > I am trying to bind SRIOV VF to a kvm guest with vfio-pci module. DPDK ve= rsion is 17.11. Both host and guest OS is RHEL 7.5. > > On KVM guest: > [root@rh75vm ~]# cat /proc/cmdline > BOOT_IMAGE=3D/vmlinuz-3.10.0-862.11.6.el7.x86_64 root=3D/dev/mapper/rhel-= root ro crashkernel=3Dauto rd.lvm.lv=3Drhel/root rd.lvm.lv=3Drhel/swap LANG= =3Den_US.UTF-8 default_hugepagesz=3D1GB hugepagesz=3D1G hugepages=3D8 conso= le=3DttyS0,115200 > > [root@rh75vm ~]# dpdk-devbind --status-dev net > > Network devices using DPDK-compatible driver > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > Network devices using kernel driver > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > 0000:00:03.0 'Virtio network device 1000' if=3Deth0 drv=3Dvirtio-pci unus= ed=3D *Active* > 0000:00:07.0 'Ethernet Virtual Function 700 Series 154c' if=3Dens7 drv=3D= i40evf unused=3D > 0000:00:08.0 'Ethernet Virtual Function 700 Series 154c' if=3Dens8 drv=3D= i40evf unused=3D > > Other Network devices > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > [root@rh75vm ~]# modprobe vfio-pci > [root@rh75vm ~]# lsmod |grep vfio > vfio_pci 41312 0 > vfio_iommu_type1 22300 0 > vfio 32695 2 vfio_iommu_type1,vfio_pci > irqbypass 13503 1 vfio_pci > [root@rh75vm ~]# dpdk-devbind -b vfio-pci 00:07.0 00:08.0 > Error: bind failed for 0000:00:07.0 - Cannot bind to driver vfio-pci > Error: unbind failed for 0000:00:07.0 - Cannot open /sys/bus/pci/drivers/= /unbind > [root@rh75vm ~]# tailf -n3 /var/log/messages > Aug 16 05:35:44 rh75vm systemd: Starting Session 1 of user root. > Aug 16 05:36:55 rh75vm kernel: VFIO - User Level meta-driver version: 0.3 > Aug 16 05:37:09 rh75vm kernel: vfio-pci: probe of 0000:00:07.0 failed wit= h error -22 > > On host: > > /proc/cmdline: > > BOOT_IMAGE=3D/vmlinuz-3.10.0-862.11.6.el7.x86_64 root=3D/dev/mapper/rhel-= root ro crashkernel=3Dauto rd.lvm.lv=3Drhel/root rd.lvm.lv=3Drhel/swap rhgb= quiet LANG=3Den_US.UTF-8 default_hugepagesz=3D1GB hugepagesz=3D1G hugepage= s=3D20 isolcpus=3D1-15,17-31 rcu_nocbs=3D1-15,17-31 nohz_full=3D1-15,17-31 = intel_iommu=3Don iommu=3Dpt selinux=3D0 enforcing=3D0 processor.max_cstate= =3D0 intel_pstate=3Ddisable hpet=3Ddisable nosoftlockup intel_idle.max_csta= te=3D0 mce=3Dignore_ce audit=3D0 > > dpdk-devbind --status-dev net > > Network devices using DPDK-compatible driver > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > 0000:86:02.0 'Ethernet Virtual Function 700 Series 154c' drv=3Dvfio-pci u= nused=3D > 0000:86:0a.0 'Ethernet Virtual Function 700 Series 154c' drv=3Dvfio-pci u= nused=3D > > Network devices using kernel driver > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > 0000:02:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=3Deno1 drv= =3Dtg3 unused=3Dvfio-pci > 0000:02:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=3Deno2 drv= =3Dtg3 unused=3Dvfio-pci > 0000:02:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=3Deno3 drv= =3Dtg3 unused=3Dvfio-pci > 0000:02:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=3Deno4 drv= =3Dtg3 unused=3Dvfio-pci > 0000:86:00.0 'Ethernet Controller XXV710 for 25GbE SFP28 158b' if=3Dens5f= 0 drv=3Di40e unused=3Dvfio-pci > 0000:86:00.1 'Ethernet Controller XXV710 for 25GbE SFP28 158b' if=3Dens5f= 1 drv=3Di40e unused=3Dvfio-pci > > Other Network devices > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > dmesg: > > [ 0.000000] DMAR: IOMMU enabled > [ 2.020878] DMAR: Hardware identity mapping for device 0000:86:00.0 > [ 2.020881] DMAR: Hardware identity mapping for device 0000:86:00.1 > [ 2.025617] DMAR: Intel(R) Virtualization Technology for Directed I/O > [ 2.048734] iommu: Adding device 0000:86:00.0 to group 59 > [ 2.048788] iommu: Adding device 0000:86:00.1 to group 60 > > [ 2.136560] pci 0000:86:00.0: Signaling PME through PCIe PME interrupt > [ 2.136562] pci 0000:86:00.1: Signaling PME through PCIe PME interrupt > [ 2.181458] DMAR: 32bit 0000:01:00.4 uses non-identity mapping > [ 2.181839] DMAR: Setting identity map for device 0000:01:00.4 [0x8a68= 8000 - 0x8a688fff] > [ 2.810190] DMAR: 32bit 0000:5c:00.0 uses non-identity mapping > [ 2.841274] i40e 0000:86:00.0: fw 6.70.48807 api 1.7 nvm 6.00 0x800036= cb 1.1747.0 > [ 3.082766] i40e 0000:86:00.0: MAC address: 48:df:37:36:1b:44 > [ 3.092871] i40e 0000:86:00.0 eth3: NIC Link is Up, 25 Gbps Full Duple= x, Requested FEC: None, FEC: None, Autoneg: False, Flow Control: RX/TX > [ 3.102039] i40e 0000:86:00.0: PCI-Express: Speed 8.0GT/s Width x8 > [ 3.110709] i40e 0000:86:00.0: Features: PF-id[0] VFs: 64 VSIs: 66 QP:= 32 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA > [ 3.136506] i40e 0000:86:00.1: fw 6.70.48807 api 1.7 nvm 6.00 0x800036= cb 1.1747.0 > [ 3.373367] i40e 0000:86:00.1: MAC address: 48:df:37:36:1b:45 > > [ 1.510851] pci 0000:86:00.0: [8086:158b] type 00 class 0x020000 > [ 1.510871] pci 0000:86:00.0: reg 0x10: [mem 0xf0000000-0xf0ffffff 64b= it pref] > [ 1.510889] pci 0000:86:00.0: reg 0x1c: [mem 0xf2000000-0xf2007fff 64b= it pref] > [ 1.510903] pci 0000:86:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pre= f] > [ 1.510955] pci 0000:86:00.0: PME# supported from D0 D3hot D3cold > [ 1.510980] pci 0000:86:00.0: reg 0x184: [mem 0xd7fffa00000-0xd7fffa0f= fff 64bit pref] > [ 1.510984] pci 0000:86:00.0: VF(n) BAR0 space: [mem 0xd7fffa00000-0xd= 7fffdfffff 64bit pref] (contains BAR0 for 64 VFs) > [ 1.510998] pci 0000:86:00.0: reg 0x190: [mem 0xd7ffff00000-0xd7ffff03= fff 64bit pref] > [ 1.511000] pci 0000:86:00.0: VF(n) BAR3 space: [mem 0xd7ffff00000-0xd= 7fffffffff 64bit pref] (contains BAR3 for 64 VFs) > [ 1.511193] pci 0000:86:00.1: [8086:158b] type 00 class 0x020000 > [ 1.511212] pci 0000:86:00.1: reg 0x10: [mem 0xf1000000-0xf1ffffff 64b= it pref] > [ 1.511229] pci 0000:86:00.1: reg 0x1c: [mem 0xf2008000-0xf200ffff 64b= it pref] > [ 1.511243] pci 0000:86:00.1: reg 0x30: [mem 0x00000000-0x0007ffff pre= f] > [ 1.511296] pci 0000:86:00.1: PME# supported from D0 D3hot D3cold > [ 1.511317] pci 0000:86:00.1: reg 0x184: [mem 0xd7fff600000-0xd7fff60f= fff 64bit pref] > [ 1.511320] pci 0000:86:00.1: VF(n) BAR0 space: [mem 0xd7fff600000-0xd= 7fff9fffff 64bit pref] (contains BAR0 for 64 VFs) > [ 1.511334] pci 0000:86:00.1: reg 0x190: [mem 0xd7fffe00000-0xd7fffe03= fff 64bit pref] > [ 1.511337] pci 0000:86:00.1: VF(n) BAR3 space: [mem 0xd7fffe00000-0xd= 7fffefffff 64bit pref] (contains BAR3 for 64 VFs) > [ 1.538878] pci 0000:86:00.0: BAR 6: assigned [mem 0xf2080000-0xf20fff= ff pref] > [ 1.538881] pci 0000:86:00.1: BAR 6: no space for [mem size 0x00080000= pref] > [ 1.538884] pci 0000:86:00.1: BAR 6: failed to assign [mem size 0x0008= 0000 pref] > [ 1.538907] pci_bus 0000:86: resource 1 [mem 0xf0000000-0xf20fffff] > [ 1.538910] pci_bus 0000:86: resource 2 [mem 0xd7fff600000-0xd7fffffff= ff 64bit pref] > > I found a few threads reported with this error and the solution was to ad= d intel_iommu=3Don and iommu=3Dpt in the kernel. I have already added this. > Any pointers what is causing the -22 error and how to solve this. > > http://mails.dpdk.org/archives/dev/2014-December/010455.html > http://mails.dpdk.org/archives/users/2017-February/001544.html > http://mails.dpdk.org/archives/users/2017-September/002433.html > https://software.intel.com/en-us/forums/networking/topic/600159 > > Thanks, > Sandeep.