From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2EC3A41D31; Tue, 21 Feb 2023 18:03:56 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DDC0E4320E; Tue, 21 Feb 2023 18:03:55 +0100 (CET) Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by mails.dpdk.org (Postfix) with ESMTP id 36B4040E0F for ; Tue, 21 Feb 2023 18:03:55 +0100 (CET) Received: by mail-pl1-f181.google.com with SMTP id ko13so6387204plb.13 for ; Tue, 21 Feb 2023 09:03:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=GOl+pyXxW9MscWPhnMue4jEhMEIsSEOv9sMjD/3i7dE=; b=mX3BKxsmXdE2r6hvGqb4t8fwreN1nCUNO/HTD4kaX1VSl7Rk6/Mns4T+0W73cUv9Ba rIxBMuiGcx8b7qLOBWX/JfjKb83nn+7QSgtq6vPS647oRwcB+l1M0yOmXI0lIJoLoGu9 4wZGcGKV6KbP7MJ6en/9pAdSHLZLP/LmO3kxGMU3hhugVkOMzNWXJ2YIVX2P7HMAtWx8 mbXm4uM65Rq4CoSdPnpkPI3pthL9+qYRKp3cUWCJE7YGZm8ihgs3fAdtbnZUaM8uvvoa L/5+BW+y09z+takJlwjJg0qU01bwVBIucFmt7nLe+k0pxoVrEdRDMCfRI+BnHBt2WILT BVkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GOl+pyXxW9MscWPhnMue4jEhMEIsSEOv9sMjD/3i7dE=; b=rmn/T/M6pWOz8DSMNem2UT93/bdtr3vD5fBJnRz4v6HeFLi7LRCgjMTjXQfxEiwQpM qnTEQsxkJbiNqCRGhB1E3U+P52v96NATJYJuOCoUbcQePnUyHYag9OBEfFEEe7RU7eXb yw9Uyoe89lFDG1do+S0Xo4rHrPrMfdog/riCXSevc8rV6rO+ghLrBjI1T1eTjHk0a62u KoOgVtdDxXE8Dj0GQjvQ8NQ++/SN06hZOLf7186mjLkmULlvXFPlArODuvTn15bre5jB MPWgunuhH2y3LZFC/vcT4Gvrr+scjkU5+14aYcUzjxnGPg4Voi08nUP/Z0DsXI2dTvZ2 aQNg== X-Gm-Message-State: AO0yUKVC5S+v8jA1sRGO74oeQFIUT22Bh3A2YOM++V9JpVDkCoHaYEu1 EiVOHBVe/xupXbM0g3YB5Z+L+A== X-Google-Smtp-Source: AK7set+AgNFyYp2NkwIWIkzPEMBA7XHaxG80aXtdQV7hiYMzDVIhJBdrH2Q3mxLjWTuPb/rIiAMWSg== X-Received: by 2002:a05:6a20:9387:b0:c0:c905:7b09 with SMTP id x7-20020a056a20938700b000c0c9057b09mr16746767pzh.44.1676999034293; Tue, 21 Feb 2023 09:03:54 -0800 (PST) Received: from hermes.local (204-195-120-218.wavecable.com. [204.195.120.218]) by smtp.gmail.com with ESMTPSA id w35-20020a17090a6ba600b0022335f1dae2sm1923872pjj.22.2023.02.21.09.03.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Feb 2023 09:03:54 -0800 (PST) Date: Tue, 21 Feb 2023 09:03:52 -0800 From: Stephen Hemminger To: Bruce Richardson Cc: Nagendra Prabhu A T , , Subject: Re: unable to bind to vfio-pci Message-ID: <20230221090352.739881a0@hermes.local> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Tue, 21 Feb 2023 16:27:31 +0000 Bruce Richardson wrote: > On Tue, Feb 21, 2023 at 09:46:30PM +0530, Nagendra Prabhu A T wrote: > > These are the messages i am seeing dmesg > > [16588.298660] vfio-pci: probe of 0000:8a:00.2 failed with error -22 > > [16591.574630] i40e 0000:8a:00.3: i40e_ptp_stop: removed PHC on > > enp138s0f3 > > [16593.138613] vfio-pci: probe of 0000:8a:00.3 failed with error -22 > > [16593.138674] vfio-pci: probe of 0000:8a:00.3 failed with error -22 > > [16627.551866] vfio-pci: probe of 0000:51:00.0 failed with error -22 > > [16627.551922] vfio-pci: probe of 0000:51:00.0 failed with error -22 > > [16716.344530] vfio-pci: probe of 0000:51:00.0 failed with error -22 > > [16716.344585] vfio-pci: probe of 0000:51:00.0 failed with error -22 > > [17859.876566] vfio-pci: probe of 0000:51:00.0 failed with error -22 > > [17859.876622] vfio-pci: probe of 0000:51:00.0 failed with error -22 > > root@nat-kvm:/home/nat# cat /boot/config-$(uname -r) | grep NOIOMMU > > CONFIG_VFIO_NOIOMMU=y > > root@nat-kvm:/home/nat# > > Interface type : > > Other Network devices > > ===================== > > 0000:51:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' > > unused=i40e,vfio-pci > > 0000:51:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' > > unused=i40e,vfio-pci > > 0000:51:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572' > > unused=i40e,vfio-pci > > 0000:51:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572' > > unused=i40e,vfio-pci > > 0000:8a:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' > > unused=i40e,vfio-pci > > 0000:8a:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' > > unused=i40e,vfio-pci > > 0000:8a:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572' > > unused=i40e,vfio-pci > > 0000:8a:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572' > > unused=i40e,vfio-pci > > > > Please try the following: > > echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode > dpdk-devbind.py -b vfio-pci 8a:00.* > > Does anything change then? > > /Bruce > > PS: For mailing list discussions, please don't top-post. Replies are easier > to follow if posted immediately below the relevant content. > > > On Tue, Feb 21, 2023 at 9:37 PM Bruce Richardson > > <[1]bruce.richardson@intel.com> wrote: > > > > On Tue, Feb 21, 2023 at 09:30:15PM +0530, Nagendra Prabhu A T wrote: > > > HI Bruce, > > > Thanks for your reply. > > > I have removed the binding of all other ports. So , i can > > confirm that > > > #2 is taken care of. > > > I tried using a different driver "uio_pci_generic". I still see > > the > > > same issue for that driver as well. > > > The problem seems to be because the write operation is not > > happening in > > > "bind" file. > > > Infact tried doing the following step manually. That also > > failed > > > echo sh -c 0000:51:00.0 > > > /sys/bus/pci/drivers/uio_pci_generic/bind > > > bash: echo: write error: No such device > > The "sh -c" doesn't look right in that command. It should just echo > > the > > device id to the bind file. However, it's generally recommended just > > to use > > the devbind script, as more writes to sysfs may be necessary first, > > e.g. to > > register the device ids first. > > > I have required permission for the file. > > > I tried doing with "sudo" > > > I dont have selinux running > > > So not sure whats blocking me. Infact , even > > > "cat /sys/bus/pci/drivers/uio_pci_generic/bind" is failing > > > Can you please help. > > To track down issues other than the obvious, I think we need more > > information. For example, what NIC card is this? Are there any > > useful error > > messages in dmesg? > > /Bruce > > > > References > > > > 1. mailto:bruce.richardson@intel.com Another possibility is that the hardware shares IOMMU channels. One machine I was using had multiple PCI devices sharing same IOMMU channel and kernel would not let it work unless all of them were unbound from kernel.