From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BD879A0032 for ; Fri, 16 Sep 2022 17:11:03 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5637B4021D; Fri, 16 Sep 2022 17:11:03 +0200 (CEST) Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by mails.dpdk.org (Postfix) with ESMTP id C777040156 for ; Fri, 16 Sep 2022 17:11:02 +0200 (CEST) Received: by mail-pg1-f171.google.com with SMTP id h188so20615656pgc.12 for ; Fri, 16 Sep 2022 08:11:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date; bh=Oht2q58bdUjllFotpi4N6+cCAdG3GdlXTnbF37FSaFw=; b=GhaJBok2aOajSbL6BiHPDnaQz35Tla7fadtxeZend7qJ7G61TbfFqPxxjBV9Yh35KN oEErujHtIU0+gDwjsREwVP+izVzzGTqe23e1V1iwrNFGuzEIrmCwZLHMWReoB4hNU6Rn 2PhZ9JUBAUw8yEXUNLhrxenVF4gH2eZJ2MsQ4+gwuAXErlKdiOG9BljDRtnX0dv7cgz4 Reo0IN8fJf/NZAbYBxIzA4ABFsOvUOsT+PDdts4iubJaBzY9JfCF3riyQrVq0roqmBex d7oDLYMVnoiE+NLwOXvZ84bXpMXQtni8XlGjGedQW4W5BWTkvdOMgbAbfg29mEb8thpO z+mQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date; bh=Oht2q58bdUjllFotpi4N6+cCAdG3GdlXTnbF37FSaFw=; b=aWojDC2nOdTN54UjDelBJCccY/OnfMw5Gnl0/OsVCIJWeHbP7u3wnO5rh/WFZFHGDM eiF+SWRKEyhUT6E3zOOXpFIIr5q+DiBuhUq7M5CK2E+UQ4rJEuuYtuYF9JwLmtLb0mEA tRLEplf2nf5qROBzY1Al9KkQjKY4OMjb2nMlEOCV1TyF08UaUMykWZrVYa1cjgDDQDE4 F6dsITNrLAqbdO/QSYUUgFe44l3a9WxS52QV+G57axbB5Vz7wjUN2kPDuIlE0PJ5CNRV oaFWCjkjLAOWyHPqfvGDsoFn7BqDhgVJy0MimUXYrR6Ebtwckp6+eg0dctHAMCb6N63l PMwg== X-Gm-Message-State: ACrzQf3+FerAiwF4+r10xaoeg+1o8Nr889lTDlQX2SHwcZXogIW8yjPQ fb9eZvhJkjlr2IYvFyHI191an/no8ppe4A== X-Google-Smtp-Source: AMsMyM6TFw5ZqU27Jj7Zv6RN8Ba4JpEj6Z0Ds2x2nK+0UZj6r93NnfPUODAQJZpEu212ikAlnPCL3Q== X-Received: by 2002:a65:6d8f:0:b0:439:a5df:c8b9 with SMTP id bc15-20020a656d8f000000b00439a5dfc8b9mr4837859pgb.506.1663341061671; Fri, 16 Sep 2022 08:11:01 -0700 (PDT) Received: from hermes.local (204-195-120-218.wavecable.com. [204.195.120.218]) by smtp.gmail.com with ESMTPSA id m127-20020a625885000000b00546d875a944sm7456038pfb.214.2022.09.16.08.11.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Sep 2022 08:11:01 -0700 (PDT) Date: Fri, 16 Sep 2022 08:10:59 -0700 From: Stephen Hemminger To: F32 Cc: users@dpdk.org Subject: Re: l2fwd: Requested device 0000:02:00.0 cannot be used (Ubuntu 22.04, DPDK 22.07, N5095, I225-V) Message-ID: <20220916081059.5b9841a1@hermes.local> In-Reply-To: <5aed7250.25e2.1833a0110ef.Coremail.feng32@163.com> References: <5aed7250.25e2.1833a0110ef.Coremail.feng32@163.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Wed, 14 Sep 2022 11:18:11 +0800 (CST) F32 wrote: > Hi, > > > Recently I'm testing dpdk on a mini PC with N5095 CPU and 5 I225-V ethernet adapters under Ubuntu Server 22.04.1. > > > $ uname -r > 5.15.0-47-generic > > > $ cat /etc/default/grub | grep GRUB_CMDLINE_LINUX_DEFAULT > GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt" > $ update-grub > $ reboot > > > $ dmesg | grep DMAR > [ 0.016399] ACPI: DMAR 0x00000000787D6000 000088 (v02 INTEL EDK2 00000002 01000013) > [ 0.016443] ACPI: Reserving DMAR table memory at [mem 0x787d6000-0x787d6087] > [ 0.029273] DMAR: IOMMU enabled > [ 0.093267] DMAR: Host address width 39 > [ 0.093269] DMAR: DRHD base: 0x000000fed90000 flags: 0x0 > [ 0.093276] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 49e2ff0505e > [ 0.093281] DMAR: DRHD base: 0x000000fed91000 flags: 0x1 > [ 0.093287] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da > [ 0.093291] DMAR: RMRR base: 0x0000007b800000 end: 0x0000007fbfffff > [ 0.093296] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 > [ 0.093299] DMAR-IR: HPET id 0 under DRHD base 0xfed91000 > [ 0.093301] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. > [ 0.095158] DMAR-IR: Enabled IRQ remapping in x2apic mode > [ 0.288306] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics > [ 0.372113] DMAR: No ATSR found > [ 0.372115] DMAR: No SATC found > [ 0.372118] DMAR: IOMMU feature fl1gp_support inconsistent > [ 0.372120] DMAR: IOMMU feature pgsel_inv inconsistent > [ 0.372122] DMAR: IOMMU feature nwfs inconsistent > [ 0.372124] DMAR: IOMMU feature pds inconsistent > [ 0.372125] DMAR: IOMMU feature eafs inconsistent > [ 0.372127] DMAR: IOMMU feature prs inconsistent > [ 0.372129] DMAR: IOMMU feature nest inconsistent > [ 0.372130] DMAR: IOMMU feature mts inconsistent > [ 0.372132] DMAR: IOMMU feature sc_support inconsistent > [ 0.372134] DMAR: IOMMU feature dev_iotlb_support inconsistent > [ 0.372136] DMAR: dmar0: Using Queued invalidation > [ 0.372141] DMAR: dmar1: Using Queued invalidation > [ 0.373075] DMAR: Intel(R) Virtualization Technology for Directed I/O > > > $ lspci > lspci > 00:00.0 Host bridge: Intel Corporation Device 4e24 > 00:02.0 VGA compatible controller: Intel Corporation JasperLake [UHD Graphics] (rev 01) > 00:04.0 Signal processing controller: Intel Corporation Dynamic Tuning service > 00:08.0 System peripheral: Intel Corporation Device 4e11 > 00:14.0 USB controller: Intel Corporation Device 4ded (rev 01) > 00:14.2 RAM memory: Intel Corporation Device 4def (rev 01) > 00:15.0 Serial bus controller: Intel Corporation Serial IO I2C Host Controller (rev 01) > 00:15.2 Serial bus controller: Intel Corporation Device 4dea (rev 01) > 00:16.0 Communication controller: Intel Corporation Management Engine Interface (rev 01) > 00:17.0 SATA controller: Intel Corporation Device 4dd3 (rev 01) > 00:19.0 Serial bus controller: Intel Corporation Device 4dc5 (rev 01) > 00:19.1 Serial bus controller: Intel Corporation Device 4dc6 (rev 01) > 00:1c.0 PCI bridge: Intel Corporation Device 4db8 (rev 01) > 00:1c.4 PCI bridge: Intel Corporation Device 4dbc (rev 01) > 00:1c.5 PCI bridge: Intel Corporation Device 4dbd (rev 01) > 00:1c.6 PCI bridge: Intel Corporation Device 4dbe (rev 01) > 00:1c.7 PCI bridge: Intel Corporation Device 4dbf (rev 01) > 00:1e.0 Communication controller: Intel Corporation Device 4da8 (rev 01) > 00:1e.3 Serial bus controller: Intel Corporation Device 4dab (rev 01) > 00:1f.0 ISA bridge: Intel Corporation Device 4d87 (rev 01) > 00:1f.3 Audio device: Intel Corporation Device 4dc8 (rev 01) > 00:1f.4 SMBus: Intel Corporation JaserLake SMBus (rev 01) > 00:1f.5 Serial bus controller: Intel Corporation JaserLake SPI (flash) Controller (rev 01) > 01:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03) > 02:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03) > 03:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03) > 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03) > 05:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03) > > > (after huge page setup) > $ python3 dpdk-hugepages.py -s > Node Pages Size Total > 0 1607 2Mb 3Gb > > > Hugepages mounted on /dev/hugepages /mnt/huge > > > $ ifconfig enp2s0 down > $ ifconfig enp3s0 down > $ python3 dpdk-devbind.py --bind=vfio-pci 0000:02:00.0 > $ python3 dpdk-devbind.py --bind=vfio-pci 0000:03:00.0 > $ python3 dpdk-devbind.py -s > > > Network devices using DPDK-compatible driver > ============================================ > 0000:02:00.0 'Ethernet Controller I225-V 15f3' drv=vfio-pci unused=igc > 0000:03:00.0 'Ethernet Controller I225-V 15f3' drv=vfio-pci unused=igc > > > Network devices using kernel driver > =================================== > 0000:01:00.0 'Ethernet Controller I225-V 15f3' if=enp1s0 drv=igc unused=vfio-pci *Active* > 0000:04:00.0 'Ethernet Controller I225-V 15f3' if=enp4s0 drv=igc unused=vfio-pci > 0000:05:00.0 'Ethernet Controller I225-V 15f3' if=enp5s0 drv=igc unused=vfio-pci > > > No 'Baseband' devices detected > ============================== > > > No 'Crypto' devices detected > ============================ > > > No 'DMA' devices detected > ========================= > > > No 'Eventdev' devices detected > ============================== > > > No 'Mempool' devices detected > ============================= > > > No 'Compress' devices detected > ============================== > > > No 'Misc (rawdev)' devices detected > =================================== > > > No 'Regex' devices detected > =========================== > > > After the above configuration, dpck setup should be done. Then I tried to run l2fwd. It generates the following errror messages: > > > $ ./l2fwd > EAL: Detected CPU lcores: 4 > EAL: Detected NUMA nodes: 1 > EAL: Detected shared linkage of DPDK > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > EAL: Selected IOVA mode 'VA' > EAL: VFIO support initialized > EAL: Using IOMMU type 1 (Type 1) > EAL: Probe PCI driver: net_igc (8086:15f3) device: 0000:02:00.0 (socket 0) > EAL: Releasing PCI mapped resource for 0000:02:00.0 > EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x1100800000 > EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x1100900000 > EAL: Requested device 0000:02:00.0 cannot be used > EAL: Using IOMMU type 1 (Type 1) > EAL: Probe PCI driver: net_igc (8086:15f3) device: 0000:03:00.0 (socket 0) > EAL: Releasing PCI mapped resource for 0000:03:00.0 > EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1100904000 > EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1100a04000 > EAL: Requested device 0000:03:00.0 cannot be used > TELEMETRY: No legacy callbacks, legacy socket not created > MAC updating enabled > EAL: Error - exiting with code: 1 > Cause: No Ethernet ports - bye > > > It seems that the I225 adapters are not recognized by DPDK. I can not see any detailed reasons why the adapters cannot be used. > Any suggestions to fix this issue? > > > Regards, > F32 Check the IOMMU groups on that system. Often multiple PCI devices (especially multi-port NICs) share a single IOMMU group and VFIO does not allow using these devices unless all subdevices in the group are unbound from the kernel driver.