From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f175.google.com (mail-wi0-f175.google.com [209.85.212.175]) by dpdk.org (Postfix) with ESMTP id 7BCC93B5 for ; Tue, 29 Sep 2015 18:41:11 +0200 (CEST) Received: by wicfx3 with SMTP id fx3so159427528wic.1 for ; Tue, 29 Sep 2015 09:41:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=Y1gc+aowJb71T191HezlNRROqT42rOUofjLbp74/8ec=; b=I1XtVn3pUBo1azqZw3qr4pnnZNr1NLwx7WhVdn+IeW8FmObRdSjj6caBOcDPhQ57d9 tmHmMIil4Iyr43em56p/tx5i10MUNWRAYrgtkRUsq+y31RZsVptSFhd9ZRomSHmlGMEz CWnFWDWaNJ12aexHzQ4NHaNaZ0iuOOVyqT1VTgpYePpoEWQthNtJSj89sz8I5auGlnXm 9m61dC4XTINVa4yA7qzkPuKRqCPNB6cWegHFiO3GUXBMBFtGzCHOQaoK2ptgTsYCEP2r 1qiwP9REsUa4ezLOhMowXL9J60gMxsmh4tpkaP2eKxp6pJDch0bydjsKiGvEJBl0bcB8 w9YQ== X-Gm-Message-State: ALoCoQmKTWCeFUGS3Su+DgshGexBNtSNQHMgXx++Dz9/q4W5H++GdU2FXNIlT6/rRrNU0qmzT9qD X-Received: by 10.180.205.234 with SMTP id lj10mr24718587wic.49.1443544871272; Tue, 29 Sep 2015 09:41:11 -0700 (PDT) Received: from [10.0.0.165] ([37.142.229.250]) by smtp.googlemail.com with ESMTPSA id pu6sm24803594wjc.34.2015.09.29.09.41.09 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 29 Sep 2015 09:41:10 -0700 (PDT) To: "Michael S. Tsirkin" References: <56079527.3000802@cloudius-systems.com> <20150927123914-mutt-send-email-mst@redhat.com> From: Vlad Zolotarov Message-ID: <560ABF25.9030300@cloudius-systems.com> Date: Tue, 29 Sep 2015 19:41:09 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <20150927123914-mutt-send-email-mst@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] Having troubles binding an SR-IOV VF to uio_pci_generic on Amazon instance X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 16:41:11 -0000 On 09/27/15 12:43, Michael S. Tsirkin wrote: > On Sun, Sep 27, 2015 at 10:05:11AM +0300, Vlad Zolotarov wrote: >> Hi, >> I was trying to use uio_pci_generic with Intel's 10G SR-IOV devices on >> Amazon EC2 instances with Enhanced Networking enabled. >> The idea is to create a DPDK environment that doesn't require compiling >> kernel modules (igb_uio). >> However I was surprised to discover that uio_pci_generic refuses to work >> with EN device on AWS: >> >> $ lspci >> 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) >> 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] >> 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] >> 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) >> 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 >> 00:03.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) >> 00:04.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) >> 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) >> >> $ sudo ./dpdk/tools/dpdk_nic_bind.py -b uio_pci_generic 00:04.0 >> Error: bind failed for 0000:00:04.0 - Cannot bind to driver uio_pci_generic >> $dmesg >> >> --> snip <--- >> [ 816.655575] uio_pci_generic 0000:00:04.0: No IRQ assigned to device: no support for interrupts? >> >> $ sudo lspci -s 00:04.0 -vvv >> 00:04.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) >> Physical Slot: 4 >> Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- >> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- > Region 0: Memory at f3008000 (64-bit, prefetchable) [size=16K] >> Region 3: Memory at f300c000 (64-bit, prefetchable) [size=16K] >> Capabilities: [70] MSI-X: Enable- Count=3 Masked- >> Vector table: BAR=3 offset=00000000 >> PBA: BAR=3 offset=00002000 >> Kernel modules: ixgbevf >> >> So, as we may see the PCI device doesn't have an INTX interrupt line >> assigned indeed. It has an MSI-X capability however. >> Looking at the uio_pci_generic code it seems to require the INTX: >> >> uio_pci_generic.c: line 74: probe(): >> >> if (!pdev->irq) { >> dev_warn(&pdev->dev, "No IRQ assigned to device: " >> "no support for interrupts?\n"); >> pci_disable_device(pdev); >> return -ENODEV; >> } >> >> Is it a known limitation? Michael, could u, pls., comment on this? >> >> thanks, >> vlad Michael, I took a look at the pci_stub driver and the reason why DPDK uses uio the first place and I have some comments below. > This is expected. uio_pci_generic forwards INT#x interrupts from device > to userspace, but VF devices never assert INT#x. > > So it doesn't seem to make any sense to bind uio_pci_generic there. Well, it's not completely correct to put it this way. The thing is that DPDK (and it could be any other framework/developer) uses uio_pci_generic to actually get interrupts from the device and it makes a perfect sense to be able to do so in the SR-IOV devices too. The problem is, like u've described above, that the current implementation of uio_pci_generic wouldn't let them do so and it seems like a bogus behavior to me. There is no reason, why uio_pci_generic wouldn't be able to work the same way as it does today but with MSI-X interrupts. To make things simple forwarding just the first vector as an initial implementation. The security breach motivation u brought in "[RFC PATCH] uio: uio_pci_generic: Add support for MSI interrupts" thread seems a bit weak since one u let the userland access to the bar it may do any funny thing using the DMA engine of the device. This kind of stuff should be prevented using the iommu and if it's enabled then any funny tricks using MSI/MSI-X configuration will be prevented too. I'm about to send the patch to main Linux mailing list. Let's continue this discussion there. > > I think that DPDK should be fixed to not require uio_pci_generic > for VF devices (or any devices without INT#x). > > If DPDK requires a place-holder driver, the pci-stub driver should > do this adequately. See ./drivers/pci/pci-stub.c >