From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dispatch1-us1.ppe-hosted.com (dispatch1-us1.ppe-hosted.com [67.231.154.164]) by dpdk.org (Postfix) with ESMTP id DC1D8326C for ; Tue, 4 Apr 2017 17:29:43 +0200 (CEST) Received: from pure.maildistiller.com (unknown [10.110.50.29]) by dispatch1-us1.ppe-hosted.com (Proofpoint Essentials ESMTP Server) with ESMTP id 0DD07800AC; Tue, 4 Apr 2017 15:29:43 +0000 (UTC) X-Virus-Scanned: Proofpoint Essentials engine Received: from mx7-us1.ppe-hosted.com (unknown [10.110.49.251]) by pure.maildistiller.com (Proofpoint Essentials ESMTP Server) with ESMTPS id C862560051; Tue, 4 Apr 2017 15:29:42 +0000 (UTC) Received: from webmail.solarflare.com (uk.solarflare.com [193.34.186.16]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx7-us1.ppe-hosted.com (Proofpoint Essentials ESMTP Server) with ESMTPS id 7623554006C; Tue, 4 Apr 2017 15:29:40 +0000 (UTC) Received: from [192.168.38.17] (84.52.89.52) by ukex01.SolarFlarecom.com (10.17.10.4) with Microsoft SMTP Server (TLS) id 15.0.1044.25; Tue, 4 Apr 2017 16:29:35 +0100 To: "Burakov, Anatoly" , "dev@dpdk.org" , Alejandro Lucero References: <694e35cd-8c9d-715b-9b6b-482b5c802846@solarflare.com> From: Andrew Rybchenko Message-ID: Date: Tue, 4 Apr 2017 18:29:32 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: X-Originating-IP: [84.52.89.52] X-ClientProxiedBy: ocex03.SolarFlarecom.com (10.20.40.36) To ukex01.SolarFlarecom.com (10.17.10.4) X-TM-AS-Product-Ver: SMEX-11.0.0.1191-8.100.1062-22984.003 X-TM-AS-Result: No--19.011100-0.000000-31 X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-MDID: 1491319783-3dQikGdHG7Wg Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] vfio: failed to select IOMMU type X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Apr 2017 15:29:44 -0000 On 04/03/2017 07:11 PM, Burakov, Anatoly wrote: >> From: Andrew Rybchenko [mailto:arybchenko@solarflare.com] >> Sent: Saturday, April 1, 2017 11:47 AM >> To: dev@dpdk.org; Alejandro Lucero >> Cc: Burakov, Anatoly >> Subject: vfio: failed to select IOMMU type >> >> Hi, >> >> after the following commit (it was picked up by dpdk-next-net recently), I >> have problems with VFIO: >> === >> commit 94c0776b1badd1ee715d60f07391058f23494365 >> Author: Alejandro Lucero >> Date: Wed Mar 29 10:54:50 2017 +0100 >> >> vfio: support hotplug >> >> Current device hotplug is just supported by UIO managed devices. >> This patch adds same functionality with VFIO. >> >> It has been validated through tests using IOMMU and also with >> VFIO and no-iommu mode. >> >> Signed-off-by: Alejandro Lucero >> Acked-by: Anatoly Burakov === >> >> The second PCI function fails to bind: >> # testpmd -w 06:00.0 -w 06:00.1 -c 0xc -n 4 -- --rxd=512 --txd=512 --crc-strip - >> -disable-hw-vlan-filter --disable-hw-vlan-strip >> EAL: Detected 16 lcore(s) >> EAL: 2048 hugepages of size 2097152 reserved, but no mounted hugetlbfs >> found for that size >> EAL: Probing VFIO support... >> EAL: VFIO support initialized >> EAL: PCI device 0000:06:00.0 on NUMA socket 0 >> EAL: probe driver: 1924:a03 net_sfc_efx >> EAL: using IOMMU type 1 (Type 1) >> EAL: Ignore mapping IO port bar(0) addr: 2101 >> EAL: PCI device 0000:06:00.1 on NUMA socket 0 >> EAL: probe driver: 1924:a03 net_sfc_efx >> EAL: 0000:06:00.1 failed to select IOMMU type >> EAL: Requested device 0000:06:00.1 cannot be used >> EAL: Requested device 0000:7f:08.0 cannot be used >> EAL: Requested device 0000:7f:08.2 cannot be used >> EAL: Requested device 0000:7f:08.3 cannot be used ... >> >> Also I don't understand why it spams about many other PCI functions taking >> into account that just 2 are specified in whitelist. >> >> I've bisected to find commit when the problem appears, but has not found >> root cause yet. >> >> Andrew. > Hi Andrew, > > It would be interesting to know what was wrong there. The whitelist issue is surprising, and from the logs it seems like EAL is trying to set up DMA mappings multiple times. Posting a more detailed log would be very helpful in tracking down the issue as well. I have tested that code with ixgbe devices, so I'm not too sure what can go wrong there. Hi Anatoly, I've sent patch to fix whitelist issue. It the result of rte_exit substitution with just logging. I think a key to the main problem is the same IOMMU group used for both PCI functions. It tries to set IOMMU type using the same file descriptor twice. The second set is dummy, since the same value is set, but still fails, I guess, because it is already in use. See logs with debug enabled and few extra logs below: EAL: PCI device 0000:06:00.0 on NUMA socket 0 EAL: probe driver: 1924:a03 net_sfc_efx EAL: vfio_get_group_fd:75: group-no=53 EAL: vfio_get_group_fd:135: group-no=53 fd=16 filename=/dev/vfio/53 EAL: vfio_setup_device:319: ps-type=0 groups=1 EAL: using IOMMU type 1 (Type 1) EAL: Ignore mapping IO port bar(0) addr: 2101 EAL: PCI memory mapped at 0x7fffc0000000 EAL: Trying to map BAR 4 that contains the MSI-X table. Trying offsets: 0x40000000000:0x0000, 0x40000001000:0x3000 EAL: PCI memory mapped at 0x7fffc0801000 PMD: sfc_efx 0000:06:00.0 #0: use ef10 Rx datapath PMD: sfc_efx 0000:06:00.0 #0: use ef10_simple Tx datapath EAL: PCI device 0000:06:00.1 on NUMA socket 0 EAL: probe driver: 1924:a03 net_sfc_efx EAL: vfio_get_group_fd:75: group-no=53 EAL: vfio_setup_device:319: ps-type=0 groups=1 EAL: set IOMMU type 1 (Type 1) failed, error 22 (Invalid argument) EAL: set IOMMU type 7 (sPAPR) failed, error 22 (Invalid argument) EAL: set IOMMU type 8 (No-IOMMU) failed, error 22 (Invalid argument) EAL: 0000:06:00.1 failed to select IOMMU type EAL: Requested device 0000:06:00.1 cannot be used Andrew.