From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from delivery.mailspamprotection.com (delivery.mailspamprotection.com [184.154.48.171]) by dpdk.org (Postfix) with ESMTP id 6A0F01BF52 for ; Fri, 15 Jun 2018 19:03:09 +0200 (CEST) Received: from ns1.es18.siteground.eu ([37.60.250.193] helo=es18.siteground.eu) by se5.mailspamprotection.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1fTs7n-0004eP-S2; Fri, 15 Jun 2018 12:03:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=filipjaniszewski.com; s=default; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:MIME-Version:Date:Message-ID:From:References:To:Subject:Sender: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=sekyjgaMcQDTaZfDKJ798h5V96rPyTlZXAsnoi5BXXw=; b=MGawFCXSl7RzV1Ynyu7GcFUwaZ B279Q6jJg8dXm/PKmJAOKD/O6c3YHeSsXMFRMw/r23JtaOTb57wQnREyLxNCJuHZ1fWEHnuaEGCdX 919Y/ylijGHHRyNsCioawIPkMFVKouS+9wCdOUXubCb/iUsTh313HdW4KrvoPxbPLWOU0ABb5W0zs r4O5nKOs/aDCjarV9EeELwqvg48ClovewxEk0wwBmTQjzITOcg4n0U81fX712Lr6UNr2qssxdf6Qs xiI/WZvXgEDk+f+nwrmEk9E6Q9pfK26aQebBbYeXEC6YOQ67p8adSSXevbrHfIMW+70M76dctbEmJ GH1RmlTg==; Received: from [89.71.152.167] (port=41888 helo=localhost.localdomain) by es18.siteground.eu with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89_34-9f6032f-XX) (envelope-from ) id 1fTs7g-00054I-RR; Fri, 15 Jun 2018 19:02:52 +0200 To: Andrew Rybchenko , users@dpdk.org, rami.rosen@intel.com References: <8e8c477b-066f-f909-d678-6623464b51c0@filipjaniszewski.com> From: Filip Janiszewski Message-ID: <020f4f4c-c22d-7a2d-441e-925ebeeeff7e@filipjaniszewski.com> Date: Fri, 15 Jun 2018 19:02:52 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=iso-8859-15 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: 37.60.250.193 X-SpamExperts-Domain: es18.siteground.eu X-SpamExperts-Username: 37.60.250.193 Authentication-Results: mailspamprotection.com; auth=pass smtp.auth=37.60.250.193@es18.siteground.eu X-SpamExperts-Outgoing-Class: ham X-SpamExperts-Outgoing-Evidence: Combined (0.15) X-Recommended-Action: accept X-Filter-ID: EX5BVjFpneJeBchSMxfU5il7cOLuUpiKRa1K3vtex6h602E9L7XzfQH6nu9C/Fh9KJzpNe6xgvOx q3u0UDjvOxFOznhojlCJeKfXtJfmiKQwIjIqOBUSsZ2C2GILDR1a+6Zwcad1J4BiD9NbwFT6rkME SFy5HJPf1L25KcsMZ47HTKNRDXcIZdFLDI4l7EkIPYuiyGOl5S+muOCC5zZ/4lprjQPFk8m4tSTf ORUp3ylzmY/pt5I0dsoxBXVR9uIEpELuUYcCJ4YswooZGFVYxozIacMilJyGVyjpEgOg2D8zsXsH 4fgE6CIAE+mIeFkhg71V+o4dEAc41OSjJZzKlo6Y0xynlrNX6Jdvj668dFfSXd33tMscXgH8tHBz ebMh0j+4sB4At/e8tafDQX/j5wf2oQ2QfvWxBOwRxetsuxJ/qqlYl1waI/651i0wJVD4HBb4rY1W FN3griV7KWznTx21S3bhuz7LR03u4f+/w+ZjkFqg9hIMSBLfLekpwharrKTaPeZ6EQyLCA8jPvID Dc7UZB5DjZsjTnYAKJBt0Qbo5JYLlRrHKrEumSmN3SwFhxr6c84pMp/Gxi6MUP8MEXoSKlHMxVnk urBPHtW+f1JHOGAZ/v25qeLC/keH8ZHPWhHMFnVfvyptTepxWmlNk5T39NDq+YBSGCoE+ashOdTY K4XVuojzasXjV+Ml3SnUB5hQ6nsDvccjqgmDvD9WhwcO86JRS1Xtpr6UHZZXs5k3YPTUB+QWPpiM Eqlb1C/6K6ebN50qxboHaunrOYs9PErvo4YFvOqHx6a22HhANGroU8qh52d3ffHksmxNhEFjCf2T gkapJGJZHrrgDYePmf9iqhwnXF68aSUABGh3xnDhc2sSCt5dmF+i7AekX0AHN7F7clRRV3XJetcB 321g7W9KSEN8VE2W6KpVFfeSpmtoIiOUYcmS05csdbrr3GVz26P6RK5Zu5c2nOvf9nC06ApsAAwy FZ7+sgOc+isPGJ7u1uSaKMtPzj/Cy7D4RDiwqEznkzBzeWls7BQQbR9vjD8RfsHpiiNAH+PncEsx QgmPdl9KRT5/hcKi4mDgb3mUZ555sRBBYx6x1CsjIE4pB+3EC7LEZVrjQ26HaiCus5h4DqxxlnCn OQKjhagEKuLSTahXS7PXCzjnTb3wDEEHFk7SQemK8vP1cspRRzrxdhJ4ugXNuP4w1MIDeQl5eW+H X-Report-Abuse-To: spam@quarantine1.mailspamprotection.com Subject: Re: [dpdk-users] Correct setup of sfc X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jun 2018 17:03:09 -0000 Adding here also Rami Rosen to continue just one thread. First of all thanks for replying, now here's the current status: It seems that there might be some problems with IOMMU, according to dmesg logs: . root build : dmesg | grep IOMMU [ 0.000000] DMAR: IOMMU enabled [ 1179.652950] vboxpci: IOMMU not found (not registered) . In particular the second line is a clear warning that IOMMU is not enabled, will try to work this out through the bios options. Also, as Andrew suggested I've had a look at /sys/class/iommu and it's empty: . root build : ll /sys/class/iommu/ total 0 . Anyway the output of --status-dev net is: . root usertools : ./dpdk-devbind.py --status-dev net Network devices using DPDK-compatible driver ============================================ Network devices using kernel driver =================================== 0000:00:1f.6 'Ethernet Connection (2) I219-V 15b8' if=enp0s31f6 drv=e1000e unused= *Active* 0000:65:00.0 'SFC9140 10/40G Ethernet Controller 0923' if=enp101s0f0 drv=sfc unused= 0000:65:00.1 'SFC9140 10/40G Ethernet Controller 0923' if=enp101s0f1 drv=sfc unused= 0000:b3:00.0 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f0 drv=mlx5_core unused= 0000:b3:00.1 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f1 drv=mlx5_core unused= Other Network devices ===================== . I've been able to use the Solarflar device by binding it to the uio_pci_generic driver: . root usertools : ./dpdk-devbind.py --status-dev net Network devices using DPDK-compatible driver ============================================ 0000:65:00.0 'SFC9140 10/40G Ethernet Controller 0923' drv=uio_pci_generic unused=sfc 0000:65:00.1 'SFC9140 10/40G Ethernet Controller 0923' drv=uio_pci_generic unused=sfc Network devices using kernel driver =================================== 0000:00:1f.6 'Ethernet Connection (2) I219-V 15b8' if=enp0s31f6 drv=e1000e unused=uio_pci_generic *Active* 0000:b3:00.0 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f0 drv=mlx5_core unused=uio_pci_generic 0000:b3:00.1 'MT27710 Family [ConnectX-4 Lx] 1015' if=enp179s0f1 drv=mlx5_core unused=uio_pci_generic . and the card is properly recognized: . EAL: PCI device 0000:00:1f.6 on NUMA socket 0 EAL: probe driver: 8086:15b8 net_e1000_em EAL: PCI device 0000:65:00.0 on NUMA socket 0 EAL: probe driver: 1924:923 net_sfc_efx PMD: sfc_efx 0000:65:00.0 #0: running FW variant is ultra-low-latency PMD: sfc_efx 0000:65:00.0 #0: use ef10 Rx datapath PMD: sfc_efx 0000:65:00.0 #0: use ef10 Tx datapath EAL: PCI device 0000:65:00.1 on NUMA socket 0 EAL: probe driver: 1924:923 net_sfc_efx PMD: sfc_efx 0000:65:00.1 #1: running FW variant is ultra-low-latency PMD: sfc_efx 0000:65:00.1 #1: use ef10 Rx datapath PMD: sfc_efx 0000:65:00.1 #1: use ef10 Tx datapath . On the other hand, attempting to bind to vfio-pci as Rami suggested is not working: . root usertools : ./dpdk-devbind.py --bind=vfio-pci 0000:65:00.0 0000:65:00.1 Error: bind failed for 0000:65:00.0 - Cannot bind to driver vfio-pci Error: bind failed for 0000:65:00.1 - Cannot bind to driver vfio-pci . And as requested, those are the dmesg logs: . [ 2146.354434] VFIO - User Level meta-driver version: 0.3 [ 2159.418937] vfio-pci: probe of 0000:65:00.0 failed with error -22 [ 2159.434537] vfio-pci: probe of 0000:65:00.1 failed with error -22 [ 2199.773476] vfio-pci: probe of 0000:65:00.0 failed with error -22 [ 2199.788973] vfio-pci: probe of 0000:65:00.1 failed with error -22 . Which as far as I understand are referring to IOMMU not being enable on the machine, I guess I'll have to find some specific option for that as the BIOS suggest that "Intel Virtualization" is enabled, but that might refer to what we need here. Filip Il 15/06/18 11:44, Andrew Rybchenko ha scritto: > On 06/13/2018 10:14 PM, Filip Janiszewski wrote: >> Hi Andrew, >> >> >>> PCI devices of Solarflare NIC should be bound to vfio, >>> uio-pci-generic or >>> igb_uio (part of DPDK) module. In the case of Solarflare NICs, Linux >>> driver is >>> not required and not used in DPDK. >>> >>> So, you should load one of above modules (depending on your server >>> IOMMU configuration), push already created interfaces down and rebind >>> Solarflare PCI functions to the driver, something like: >>> >>> modprobe vfio-pci >>> ip link set enp101s0f0 down >>> ip link set enp101s0f1 down >>> dpdk-devbind.py --bind=vfio-pci 0000:65:00.0 0000:65:00.1 >>> >>> The above assumes that dpdk-devbind.py script is in PATH. >>> And start DPDK as you do before. >>> >> For some reason vfio-pci is refusing to bind the device: >> >> . >> root usertools : ./dpdk-devbind.py --bind=vfio_pci 0000:65:00.0 >> 0000:65:00.1 >> Error: bind failed for 0000:65:00.0 - Cannot open >> /sys/bus/pci/drivers/vfio_pci/bind >> Error: bind failed for 0000:65:00.1 - Cannot open >> /sys/bus/pci/drivers/vfio_pci/bind >> . >> >> also: >> >> . >> root usertools : echo 0000:65:00.0 > /sys/bus/pci/drivers/vfio-pci/bind >> bash: echo: write error: No such device >> . >> >> In the kernel command line I've included 'iommu=pt intel_iommu=on', but >> still not working. Well I guess this is not a DPDK issue anymore. > > What does the following command shows? > ./dpdk-devbind.py --status-dev net > > Is IOMMU (VT-d) enabled in BIOS/UEFI? You can check for files/symlinks > in /sys/class/iommu. > > As an experiment I'd try uio-pci-generic as well. > Also exact error logs from dmesg could be useful to understand what's > going on. > > Andrew. -- BR, Filip +48 666 369 823