From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 08E9341D4F; Thu, 23 Feb 2023 10:00:18 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6564C43143; Thu, 23 Feb 2023 10:00:09 +0100 (CET) Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by mails.dpdk.org (Postfix) with ESMTP id B207540E03 for ; Tue, 21 Feb 2023 18:27:44 +0100 (CET) Received: by mail-pj1-f50.google.com with SMTP id u10so5817244pjc.5 for ; Tue, 21 Feb 2023 09:27:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=/D0HpN10HwkkWr6ZthymXYXJjcB3ZjxEjr97HDaggGI=; b=OFXcY+w8wgR7yuNRlrK07PP4XSgrtcgbWVazcOaCX3EJrBhpm+pQnnPtjw+tsYgvSR USqoGzJ29UTBFptCYNVfapzeksoYq818q6dGLU9Eqgyc9bjvAe/dVDqEYe0h5nEFC6T7 2vjbRyoQI5gg9ns6qSCBcw1SBM+ndlVlmwCJAPUF+cUYFDx2YCDcYIQqT2w49YmvPwGe YqsQCZKmev+rFB/1ZQEe5142H1LE9WCWK1m/FSGiErqDzINpyWI1rN52OQKWu6Py9gEg FrbRJOaeVB0znE7Tv5Tw5b5yh4B/OR0ux0bO5XPZ4SN2d04eR0tEG/lhLXdA058Vfzi6 rz8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/D0HpN10HwkkWr6ZthymXYXJjcB3ZjxEjr97HDaggGI=; b=laQSRuWPG5K8OHkpVLvUJxcEPioHAAmXTvhAso3BqeKIW3iDLEKOreD8mb3Xa6UzgO vQsotOz14C8oD8TkdxSYH9L7TapIE8X58tHNFgnJtfnGb80/QIB4dn9QRrU2Z/BIeIA4 S1kNS/ytAN36J4bR5JgPNDvtitvO1JGygBPamT02OfjT6W9XED28exmfjxDgZRveMPvj Jt+VoHw5jbjFnUcIRU786AYITPiby7k5ED6lPnxzbN2PwLnTxgECzjoCn0DL/sx+BgfU +cZenwnU0vuFXqodq4dkbf2dryvNLwWv//jfoQNDGknqnrxVvMAixQD3LUuw9YK0RAwL aI0A== X-Gm-Message-State: AO0yUKXIsWor9QNOUbDYWoGZ7BIlS3ZJL0AvlXMr/Ooxq1kGibP8/TO2 P2ZMpjcPBubvEq9PZBxZY1zmQXN/slM5EBr0URgtpPryW/KJLg== X-Google-Smtp-Source: AK7set/jSM4GhVDs12qk+JHjkwwlGaEupSKCvp1R8G2t78Sr1EvaY89rJuLBamiPUQ2JQ7Q6vFrwP83w84CB9FGqlfo= X-Received: by 2002:a17:90a:e7cc:b0:236:6dea:8794 with SMTP id kb12-20020a17090ae7cc00b002366dea8794mr3144513pjb.22.1677000463795; Tue, 21 Feb 2023 09:27:43 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Nagendra Prabhu A T Date: Tue, 21 Feb 2023 22:57:32 +0530 Message-ID: Subject: Re: unable to bind to vfio-pci To: Bruce Richardson Cc: anatoly.burakov@intel.com, dev@dpdk.org Content-Type: multipart/alternative; boundary="000000000000eece0005f53918d2" X-Mailman-Approved-At: Thu, 23 Feb 2023 10:00:05 +0100 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --000000000000eece0005f53918d2 Content-Type: text/plain; charset="UTF-8" On Tue, Feb 21, 2023 at 9:57 PM Bruce Richardson wrote: > On Tue, Feb 21, 2023 at 09:46:30PM +0530, Nagendra Prabhu A T wrote: > > These are the messages i am seeing dmesg > > [16588.298660] vfio-pci: probe of 0000:8a:00.2 failed with error -22 > > [16591.574630] i40e 0000:8a:00.3: i40e_ptp_stop: removed PHC on > > enp138s0f3 > > [16593.138613] vfio-pci: probe of 0000:8a:00.3 failed with error -22 > > [16593.138674] vfio-pci: probe of 0000:8a:00.3 failed with error -22 > > [16627.551866] vfio-pci: probe of 0000:51:00.0 failed with error -22 > > [16627.551922] vfio-pci: probe of 0000:51:00.0 failed with error -22 > > [16716.344530] vfio-pci: probe of 0000:51:00.0 failed with error -22 > > [16716.344585] vfio-pci: probe of 0000:51:00.0 failed with error -22 > > [17859.876566] vfio-pci: probe of 0000:51:00.0 failed with error -22 > > [17859.876622] vfio-pci: probe of 0000:51:00.0 failed with error -22 > > root@nat-kvm:/home/nat# cat /boot/config-$(uname -r) | grep NOIOMMU > > CONFIG_VFIO_NOIOMMU=y > > root@nat-kvm:/home/nat# > > Interface type : > > Other Network devices > > ===================== > > 0000:51:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' > > unused=i40e,vfio-pci > > 0000:51:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' > > unused=i40e,vfio-pci > > 0000:51:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572' > > unused=i40e,vfio-pci > > 0000:51:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572' > > unused=i40e,vfio-pci > > 0000:8a:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' > > unused=i40e,vfio-pci > > 0000:8a:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' > > unused=i40e,vfio-pci > > 0000:8a:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572' > > unused=i40e,vfio-pci > > 0000:8a:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572' > > unused=i40e,vfio-pci > > > > Please try the following: > > echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode > dpdk-devbind.py -b vfio-pci 8a:00.* > > Does anything change then? > > /Bruce > > PS: For mailing list discussions, please don't top-post. Replies are easier > to follow if posted immediately below the relevant content. > Thanks Bruce, the above solution worked. root@nat-kvm:/usr/src/dpdk-22.11/usertools# echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode root@nat-kvm:/usr/src/dpdk-22.11/usertools# dpdk-devbind.py -b vfio-pci 8a:00.* root@nat-kvm:/usr/src/dpdk-22.11/usertools# dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:8a:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=i40e 0000:8a:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=i40e 0000:8a:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=i40e 0000:8a:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=i40e Thanks > > > On Tue, Feb 21, 2023 at 9:37 PM Bruce Richardson > > <[1]bruce.richardson@intel.com> wrote: > > > > On Tue, Feb 21, 2023 at 09:30:15PM +0530, Nagendra Prabhu A T wrote: > > > HI Bruce, > > > Thanks for your reply. > > > I have removed the binding of all other ports. So , i can > > confirm that > > > #2 is taken care of. > > > I tried using a different driver "uio_pci_generic". I still see > > the > > > same issue for that driver as well. > > > The problem seems to be because the write operation is not > > happening in > > > "bind" file. > > > Infact tried doing the following step manually. That also > > failed > > > echo sh -c 0000:51:00.0 > > > /sys/bus/pci/drivers/uio_pci_generic/bind > > > bash: echo: write error: No such device > > The "sh -c" doesn't look right in that command. It should just echo > > the > > device id to the bind file. However, it's generally recommended just > > to use > > the devbind script, as more writes to sysfs may be necessary first, > > e.g. to > > register the device ids first. > > > I have required permission for the file. > > > I tried doing with "sudo" > > > I dont have selinux running > > > So not sure whats blocking me. Infact , even > > > "cat /sys/bus/pci/drivers/uio_pci_generic/bind" is failing > > > Can you please help. > > To track down issues other than the obvious, I think we need more > > information. For example, what NIC card is this? Are there any > > useful error > > messages in dmesg? > > /Bruce > > > > References > > > > 1. mailto:bruce.richardson@intel.com > --000000000000eece0005f53918d2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Tue, Feb 21, 2023= at 9:57 PM Bruce Richardson <bruce.richardson@intel.com> wrote:
On Tue, Feb 21, 2023 at 09:46:30PM +0530, Nag= endra Prabhu A T wrote:
>=C2=A0 =C2=A0 These are the messages i am seeing dmesg
>=C2=A0 =C2=A0 [16588.298660] vfio-pci: probe of 0000:8a:00.2 failed wit= h error -22
>=C2=A0 =C2=A0 [16591.574630] i40e 0000:8a:00.3: i40e_ptp_stop: removed = PHC on
>=C2=A0 =C2=A0 enp138s0f3
>=C2=A0 =C2=A0 [16593.138613] vfio-pci: probe of 0000:8a:00.3 failed wit= h error -22
>=C2=A0 =C2=A0 [16593.138674] vfio-pci: probe of 0000:8a:00.3 failed wit= h error -22
>=C2=A0 =C2=A0 [16627.551866] vfio-pci: probe of 0000:51:00.0 failed wit= h error -22
>=C2=A0 =C2=A0 [16627.551922] vfio-pci: probe of 0000:51:00.0 failed wit= h error -22
>=C2=A0 =C2=A0 [16716.344530] vfio-pci: probe of 0000:51:00.0 failed wit= h error -22
>=C2=A0 =C2=A0 [16716.344585] vfio-pci: probe of 0000:51:00.0 failed wit= h error -22
>=C2=A0 =C2=A0 [17859.876566] vfio-pci: probe of 0000:51:00.0 failed wit= h error -22
>=C2=A0 =C2=A0 [17859.876622] vfio-pci: probe of 0000:51:00.0 failed wit= h error -22
>=C2=A0 =C2=A0 root@nat-kvm:/home/nat# cat /boot/config-$(uname -r) | gr= ep NOIOMMU
>=C2=A0 =C2=A0 CONFIG_VFIO_NOIOMMU=3Dy
>=C2=A0 =C2=A0 root@nat-kvm:/home/nat#
>=C2=A0 =C2=A0 Interface type :
>=C2=A0 =C2=A0 Other Network devices
>=C2=A0 =C2=A0 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D
>=C2=A0 =C2=A0 0000:51:00.0 'Ethernet Controller X710 for 10GbE SFP+= 1572'
>=C2=A0 =C2=A0 unused=3Di40e,vfio-pci
>=C2=A0 =C2=A0 0000:51:00.1 'Ethernet Controller X710 for 10GbE SFP+= 1572'
>=C2=A0 =C2=A0 unused=3Di40e,vfio-pci
>=C2=A0 =C2=A0 0000:51:00.2 'Ethernet Controller X710 for 10GbE SFP+= 1572'
>=C2=A0 =C2=A0 unused=3Di40e,vfio-pci
>=C2=A0 =C2=A0 0000:51:00.3 'Ethernet Controller X710 for 10GbE SFP+= 1572'
>=C2=A0 =C2=A0 unused=3Di40e,vfio-pci
>=C2=A0 =C2=A0 0000:8a:00.0 'Ethernet Controller X710 for 10GbE SFP+= 1572'
>=C2=A0 =C2=A0 unused=3Di40e,vfio-pci
>=C2=A0 =C2=A0 0000:8a:00.1 'Ethernet Controller X710 for 10GbE SFP+= 1572'
>=C2=A0 =C2=A0 unused=3Di40e,vfio-pci
>=C2=A0 =C2=A0 0000:8a:00.2 'Ethernet Controller X710 for 10GbE SFP+= 1572'
>=C2=A0 =C2=A0 unused=3Di40e,vfio-pci
>=C2=A0 =C2=A0 0000:8a:00.3 'Ethernet Controller X710 for 10GbE SFP+= 1572'
>=C2=A0 =C2=A0 unused=3Di40e,vfio-pci
>

Please try the following:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 echo 1 > /sys/module/vfio/parameters/enable_= unsafe_noiommu_mode
=C2=A0 =C2=A0 =C2=A0 =C2=A0 dpdk-devbind.py -b vfio-pci 8a:00.*

Does anything change then?

/Bruce

PS: For mailing list discussions, please don't top-post. Replies are ea= sier
to follow if posted immediately below the relevant content.

Thanks Bruce, the above solution worked.
root@nat-kvm:/usr/src/dpdk-22.11/usertools#=C2=A0 echo 1 > /= sys/module/vfio/parameters/enable_unsafe_noiommu_mode
root@nat-kv= m:/usr/src/dpdk-22.11/usertools#=C2=A0 dpdk-devbind.py -b vfio-pci 8a:00.*<= /div>
root@nat-kvm:/usr/src/dpdk-22.11/usertools# dpdk-devbind.py --sta= tus

Network devices using DPDK-compatible driver
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
0000:8a:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv= =3Dvfio-pci unused=3Di40e
0000:8a:00.1 'Ethernet Controller X= 710 for 10GbE SFP+ 1572' drv=3Dvfio-pci unused=3Di40e
0000:8a= :00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=3Dvfio-pci= unused=3Di40e
0000:8a:00.3 'Ethernet Controller X710 for 10G= bE SFP+ 1572' drv=3Dvfio-pci unused=3Di40e

Thanks=C2=A0

>=C2=A0 =C2=A0 On Tue, Feb 21, 2023 at 9:37 PM Bruce Richardson
>=C2=A0 =C2=A0 <[1]bruce.richardson@intel.com> wrote:
>
>=C2=A0 =C2=A0 =C2=A0 On Tue, Feb 21, 2023 at 09:30:15PM +0530, Nagendra= Prabhu A T wrote:
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 HI Bruce,
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 Thanks for your reply.
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 I have removed the binding of al= l other ports. So , i can
>=C2=A0 =C2=A0 =C2=A0 confirm that
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 #2 is taken care of.
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 I tried using a different driver= "uio_pci_generic". I still see
>=C2=A0 =C2=A0 =C2=A0 the
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 same issue for that driver as we= ll.
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 The problem seems to be because = the write operation is not
>=C2=A0 =C2=A0 =C2=A0 happening in
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 "bind" file.
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 Infact tried doing the following= step manually. That also
>=C2=A0 =C2=A0 =C2=A0 failed
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 echo sh -c 0000:51:00.0 >
>=C2=A0 =C2=A0 =C2=A0 /sys/bus/pci/drivers/uio_pci_generic/bind
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 bash: echo: write error: No such= device
>=C2=A0 =C2=A0 =C2=A0 The "sh -c" doesn't look right in th= at command. It should just echo
>=C2=A0 =C2=A0 =C2=A0 the
>=C2=A0 =C2=A0 =C2=A0 device id to the bind file. However, it's gene= rally recommended just
>=C2=A0 =C2=A0 =C2=A0 to use
>=C2=A0 =C2=A0 =C2=A0 the devbind script, as more writes to sysfs may be= necessary first,
>=C2=A0 =C2=A0 =C2=A0 e.g. to
>=C2=A0 =C2=A0 =C2=A0 register the device ids first.
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 I have required permission for t= he file.
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 I tried doing with "sudo&qu= ot;
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 I dont have selinux running
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 So not sure whats blocking me. I= nfact , even
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 "cat /sys/bus/pci/drivers/u= io_pci_generic/bind" is failing
>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 Can you please help.
>=C2=A0 =C2=A0 =C2=A0 To track down issues other than the obvious, I thi= nk we need more
>=C2=A0 =C2=A0 =C2=A0 information. For example, what NIC card is this? A= re there any
>=C2=A0 =C2=A0 =C2=A0 useful error
>=C2=A0 =C2=A0 =C2=A0 messages in dmesg?
>=C2=A0 =C2=A0 =C2=A0 /Bruce
>
> References
>
>=C2=A0 =C2=A0 1. mailto:bruce.richardson@intel.com
--000000000000eece0005f53918d2--