From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from delivery.mailspamprotection.com (delivery.mailspamprotection.com [108.178.13.117]) by dpdk.org (Postfix) with ESMTP id 4CB001BF40 for ; Fri, 15 Jun 2018 19:42:04 +0200 (CEST) Received: from ns1.es18.siteground.eu ([37.60.250.193] helo=es18.siteground.eu) by se11.mailspamprotection.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1fTsjY-0002HX-PJ; Fri, 15 Jun 2018 12:42:02 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=filipjaniszewski.com; s=default; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:MIME-Version:Date:Message-ID:References:To:From:Subject:Sender: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=mr+AqfEgRmJ3Un266YSV+8fYfDnJfi7KCd/r8SnVjHA=; b=kiG5hmygDLasiWJunUzeZshKqX P67+MSKNOkX3WDTTHVEpbuP6LIepLbX5tG5Mq3C7jduvlrkXUCxwFtyuiNSt3xR9z6CgYU4PAW8zZ Wjf5qbnaS+f4pT/+Mt3+Q6tkMaUK2v5WLU8mUJKhOoQMKrJ1KLUYx5xnfUzcK0v8tAqkFCAwyT18l lzQv+Uol8qWpzMTVVh3DzxnbK+rOY5qgw6yAg41b3o13ySLyTIl0rvUSa8i10CiMkfltq66/Usvgx rP0voRVMw8EGPr7hFOTNJ8p0Nt2TCgbMqhfJZTKOZ2p/0uVbg9HXa/Sq49whZ9KzLsFxjEV5xkHyX I3vEZqLQ==; Received: from [89.71.152.167] (port=47542 helo=localhost.localdomain) by es18.siteground.eu with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89_34-9f6032f-XX) (envelope-from ) id 1fTsjP-0001XF-Vs; Fri, 15 Jun 2018 19:41:52 +0200 From: Filip Janiszewski To: Andrew Rybchenko , users@dpdk.org, rami.rosen@intel.com References: <8e8c477b-066f-f909-d678-6623464b51c0@filipjaniszewski.com> <020f4f4c-c22d-7a2d-441e-925ebeeeff7e@filipjaniszewski.com> Message-ID: <4e819ee2-39dd-c34c-4420-bfbf9e0e0d81@filipjaniszewski.com> Date: Fri, 15 Jun 2018 19:41:50 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: <020f4f4c-c22d-7a2d-441e-925ebeeeff7e@filipjaniszewski.com> Content-Type: text/plain; charset=iso-8859-15 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: 37.60.250.193 X-SpamExperts-Domain: es18.siteground.eu X-SpamExperts-Username: 37.60.250.193 Authentication-Results: mailspamprotection.com; auth=pass smtp.auth=37.60.250.193@es18.siteground.eu X-SpamExperts-Outgoing-Class: ham X-SpamExperts-Outgoing-Evidence: Combined (0.17) X-Recommended-Action: accept X-Filter-ID: EX5BVjFpneJeBchSMxfU5lQFGjL2ngBApTUa7/Y/AmJ602E9L7XzfQH6nu9C/Fh9KJzpNe6xgvOx q3u0UDjvOxFOznhojlCJeKfXtJfmiKQwIjIqOBUSsZ2C2GILDR1a+6Zwcad1J4BiD9NbwFT6rkME SFy5HJPf1L25KcsMZ47HTKNRDXcIZdFLDI4l7EkIfWciSeMdQU/sZJLtZ1qvDFprjQPFk8m4tSTf ORUp3ylzmY/pt5I0dsoxBXVR9uIEN7BeU0Gon1KbjfkaTjjIaJx5dk0bQRRsQsDS6/1iF07Tf4rs m7+rFGG8cMBPkvJHBoaTaJ67iGJuQWO9rGKJb7SCaNgxA5caF+4FAVOw4Vy0xrdjrhzLfBLKhDQi A0OcQoePwaFK+dcbpqoA75ADnPlh2AH9eFNYt8A3EdPfL9LcItXNVtxTOVykeSEx5zSLyw0EuFNw JAjqQjAxOA6X67cfM+OalzYqHZ0KFjNC3PrGe4pLnol5Rd2cfJ0IHHd26rHbeQNJ53d+k7egmIeC iUXtQhr43M8eCrxQqJFh3M5ljKQKU7D2j8PE+9W3nKXRHuUCcH3gfGjK7M5vIGbB4Ck4xAm9D8KT eKJT7gNACPcEf0dCVvoFWxTNZM2Vz63tEG5Pt4eAPHm5IPX6bofproBHfBp2knUg9pkcNh5lHF9J 269tshdFaOIOBckGLTslqGMXClBiWwVY1SxYjBTILz2lj+fnFwrdsJvFB7jr7qCuNZRZZP8ZmbVO 9v5MPDfArMuKnlwkPaTIm/tpUSHIrpsvr9rbWosIgYISDY+QzglPiQAEhYeiQm8hO73XuhcTxGcQ 4ZJyS8OyCuiJmbqA9810xzT60uub84zZk03vFUq+kQ7VuUforTH+Y09jM8nG4X9uKUJLkSAabeW7 EoKLcI92X0pFPn+FwqLiYOBveZQfcKNcJQgyTdtQ3r5m41v+tuq3RB5g0sBZRfzJEI8ZbV69YZo2 H7LxjesuqUrmd/1eLF/RBstk10bmBVjIbly5dD60i6BzzGvTyOvlBQI6fWGc+45UkkR7+Qos4inp A1Obkruh8gEqZSdB9q/2W/unQW7usyTBTYjVD9C1j/ju1cBNTq854ZCQnNKeREq1NHWMyGnDIpSc hlco6RIDoNg/H/37sroqoygklCH4OjaKUKZf1jLPtbU4mg1rsFBKUm9DGu/60Hsy+w657hacozMi X-Report-Abuse-To: spam@quarantine1.mailspamprotection.com Subject: Re: [dpdk-users] Correct setup of sfc X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jun 2018 17:42:04 -0000 Hi, Apparently there was an additional option to click in the BIOS, now I'm able to bind to vfio-pci and IOMMU is working correctly: . root usertools : ./dpdk-devbind.py --status-dev net Network devices using DPDK-compatible driver ============================================ 0000:65:00.0 'SFC9140 10/40G Ethernet Controller 0923' drv=vfio-pci unused=sfc 0000:65:00.1 'SFC9140 10/40G Ethernet Controller 0923' drv=vfio-pci unused=sfc Network devices using kernel driver =================================== 0000:00:1f.6 'Ethernet Connection (2) I219-V 15b8' if=enp0s31f6 drv=e1000e unused=vfio-pci *Active* 0000:b3:00.0 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f0 drv=mlx5_core unused=vfio-pci 0000:b3:00.1 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f1 drv=mlx5_core unused=vfio-pci Other Network devices ===================== . also: . root build : ll /sys/class/iommu/ total 0 lrwxrwxrwx. 1 root root 0 Jun 15 2018 dmar0 -> ../../devices/virtual/iommu/dmar0 lrwxrwxrwx. 1 root root 0 Jun 15 2018 dmar1 -> ../../devices/virtual/iommu/dmar1 lrwxrwxrwx. 1 root root 0 Jun 15 2018 dmar2 -> ../../devices/virtual/iommu/dmar2 lrwxrwxrwx. 1 root root 0 Jun 15 2018 dmar3 -> ../../devices/virtual/iommu/dmar3 . Thanks everybody for the support. Il 15/06/18 19:02, Filip Janiszewski ha scritto: > Adding here also Rami Rosen to continue just one thread. > > First of all thanks for replying, now here's the current status: > > It seems that there might be some problems with IOMMU, according to > dmesg logs: > > . > root build : dmesg | grep IOMMU > [ 0.000000] DMAR: IOMMU enabled > [ 1179.652950] vboxpci: IOMMU not found (not registered) > . > > In particular the second line is a clear warning that IOMMU is not > enabled, will try to work this out through the bios options. Also, as > Andrew suggested I've had a look at /sys/class/iommu and it's empty: > > . > root build : ll /sys/class/iommu/ > total 0 > . > > Anyway the output of --status-dev net is: > > . > root usertools : ./dpdk-devbind.py --status-dev net > > Network devices using DPDK-compatible driver > ============================================ > > > > > > > Network devices using kernel driver > > > =================================== > > > 0000:00:1f.6 'Ethernet Connection (2) I219-V 15b8' if=enp0s31f6 > drv=e1000e unused= *Active* > > 0000:65:00.0 'SFC9140 10/40G Ethernet Controller 0923' if=enp101s0f0 > drv=sfc unused= > > 0000:65:00.1 'SFC9140 10/40G Ethernet Controller 0923' if=enp101s0f1 > drv=sfc unused= > > 0000:b3:00.0 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f0 > drv=mlx5_core unused= > > 0000:b3:00.1 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f1 > drv=mlx5_core unused= > > > > > Other Network devices > > > ===================== > > > > . > > I've been able to use the Solarflar device by binding it to the > uio_pci_generic driver: > > . > root usertools : ./dpdk-devbind.py --status-dev net > > Network devices using DPDK-compatible driver > ============================================ > 0000:65:00.0 'SFC9140 10/40G Ethernet Controller 0923' > drv=uio_pci_generic unused=sfc > 0000:65:00.1 'SFC9140 10/40G Ethernet Controller 0923' > drv=uio_pci_generic unused=sfc > > Network devices using kernel driver > =================================== > 0000:00:1f.6 'Ethernet Connection (2) I219-V 15b8' if=enp0s31f6 > drv=e1000e unused=uio_pci_generic *Active* > 0000:b3:00.0 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f0 > drv=mlx5_core unused=uio_pci_generic > 0000:b3:00.1 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f1 > drv=mlx5_core unused=uio_pci_generic > . > > and the card is properly recognized: > > . > EAL: PCI device 0000:00:1f.6 on NUMA socket 0 > EAL: probe driver: 8086:15b8 net_e1000_em > EAL: PCI device 0000:65:00.0 on NUMA socket 0 > EAL: probe driver: 1924:923 net_sfc_efx > PMD: sfc_efx 0000:65:00.0 #0: running FW variant is ultra-low-latency > PMD: sfc_efx 0000:65:00.0 #0: use ef10 Rx datapath > PMD: sfc_efx 0000:65:00.0 #0: use ef10 Tx datapath > EAL: PCI device 0000:65:00.1 on NUMA socket 0 > EAL: probe driver: 1924:923 net_sfc_efx > PMD: sfc_efx 0000:65:00.1 #1: running FW variant is ultra-low-latency > PMD: sfc_efx 0000:65:00.1 #1: use ef10 Rx datapath > PMD: sfc_efx 0000:65:00.1 #1: use ef10 Tx datapath > . > > On the other hand, attempting to bind to vfio-pci as Rami suggested is > not working: > > . > root usertools : ./dpdk-devbind.py --bind=vfio-pci 0000:65:00.0 0000:65:00.1 > Error: bind failed for 0000:65:00.0 - Cannot bind to driver vfio-pci > Error: bind failed for 0000:65:00.1 - Cannot bind to driver vfio-pci > . > > And as requested, those are the dmesg logs: > > . > [ 2146.354434] VFIO - User Level meta-driver version: 0.3 > [ 2159.418937] vfio-pci: probe of 0000:65:00.0 failed with error -22 > [ 2159.434537] vfio-pci: probe of 0000:65:00.1 failed with error -22 > [ 2199.773476] vfio-pci: probe of 0000:65:00.0 failed with error -22 > [ 2199.788973] vfio-pci: probe of 0000:65:00.1 failed with error -22 > . > > Which as far as I understand are referring to IOMMU not being enable on > the machine, I guess I'll have to find some specific option for that as > the BIOS suggest that "Intel Virtualization" is enabled, but that might > refer to what we need here. > > Filip > > Il 15/06/18 11:44, Andrew Rybchenko ha scritto: >> On 06/13/2018 10:14 PM, Filip Janiszewski wrote: >>> Hi Andrew, >>> >>> >>>> PCI devices of Solarflare NIC should be bound to vfio, >>>> uio-pci-generic or >>>> igb_uio (part of DPDK) module. In the case of Solarflare NICs, Linux >>>> driver is >>>> not required and not used in DPDK. >>>> >>>> So, you should load one of above modules (depending on your server >>>> IOMMU configuration), push already created interfaces down and rebind >>>> Solarflare PCI functions to the driver, something like: >>>> >>>> modprobe vfio-pci >>>> ip link set enp101s0f0 down >>>> ip link set enp101s0f1 down >>>> dpdk-devbind.py --bind=vfio-pci 0000:65:00.0 0000:65:00.1 >>>> >>>> The above assumes that dpdk-devbind.py script is in PATH. >>>> And start DPDK as you do before. >>>> >>> For some reason vfio-pci is refusing to bind the device: >>> >>> . >>> root usertools : ./dpdk-devbind.py --bind=vfio_pci 0000:65:00.0 >>> 0000:65:00.1 >>> Error: bind failed for 0000:65:00.0 - Cannot open >>> /sys/bus/pci/drivers/vfio_pci/bind >>> Error: bind failed for 0000:65:00.1 - Cannot open >>> /sys/bus/pci/drivers/vfio_pci/bind >>> . >>> >>> also: >>> >>> . >>> root usertools : echo 0000:65:00.0 > /sys/bus/pci/drivers/vfio-pci/bind >>> bash: echo: write error: No such device >>> . >>> >>> In the kernel command line I've included 'iommu=pt intel_iommu=on', but >>> still not working. Well I guess this is not a DPDK issue anymore. >> >> What does the following command shows? >> ./dpdk-devbind.py --status-dev net >> >> Is IOMMU (VT-d) enabled in BIOS/UEFI? You can check for files/symlinks >> in /sys/class/iommu. >> >> As an experiment I'd try uio-pci-generic as well. >> Also exact error logs from dmesg could be useful to understand what's >> going on. >> >> Andrew. > -- BR, Filip +48 666 369 823