From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp.unipi.it (smtp3.unipi.it [131.114.21.153]) by dpdk.org (Postfix) with ESMTP id B5DE32B84 for ; Fri, 21 Apr 2017 12:53:15 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by smtp.unipi.it (Postfix) with ESMTP id 636FDC3321 for ; Fri, 21 Apr 2017 12:53:15 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at unipi.it Received: from mail-oi0-f47.google.com (mail-oi0-f47.google.com [209.85.218.47]) (Authenticated User) by smtp.unipi.it (Postfix) with ESMTPSA id 296FCC1BDD for ; Fri, 21 Apr 2017 12:53:13 +0200 (CEST) Received: by mail-oi0-f47.google.com with SMTP id s131so2963095oia.3 for ; Fri, 21 Apr 2017 03:53:13 -0700 (PDT) X-Gm-Message-State: AN3rC/6oPTqvHdRIg5dU+P4tMzrMvRqZVCwrb3NrTAKEdcXRiH95LgmQ dl2jZWgrorWVwFS4lZh6h+tpDo0aYg== X-Received: by 10.202.88.70 with SMTP id m67mr6528886oib.32.1492771991902; Fri, 21 Apr 2017 03:53:11 -0700 (PDT) MIME-Version: 1.0 Received: by 10.157.5.10 with HTTP; Fri, 21 Apr 2017 03:53:11 -0700 (PDT) In-Reply-To: References: From: Vincenzo Maffione Date: Fri, 21 Apr 2017 12:53:11 +0200 X-Gmail-Original-Message-ID: Message-ID: To: Shivaram Mysore Cc: users@dpdk.org, Giuseppe Lettieri Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] Cannot attach DPDK NIC port to OVS X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Apr 2017 10:53:16 -0000 Hi, I eventually managed to get it working by using "uio_pci_generic" as a DPDK-enabled driver for Intel 82599 ports. I had to disable IOMMU (e.g. intel_iommu=off in the kernel boot parameters), otherwise uio_pci_generic does not work. Is this documented somewhere? If not, I think it should be. In any case, I confirm that DPDK with OVS 2.7.0 does not work when vfio-pci is used as DPDK enabled driver; I keep getting "Error attaching device". This is weird, because DPDK pktgen works fine with vfio-pci DPDK driver (intel_iommu must be on in this case). Thanks, Vincenzo 2017-04-20 18:21 GMT+02:00 Vincenzo Maffione : > vfio-pci is loaded and bound to the NIC, as shown by dpdk-devbind. > > $ sudo tools/dpdk-devbind.py --status > > Network devices using DPDK-compatible driver > ============================================ > 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=vfio-pci > unused=ixgbe > > Network devices using kernel driver > =================================== > 0000:00:19.0 'Ethernet Connection (2) I218-V' if=eno1 drv=e1000e > unused=vfio-pci > 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp1s0f1 > drv=ixgbe unused=vfio-pci > 0000:06:00.0 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller' > if=enp6s0 drv=r8169 unused=vfio-pci > 0000:07:00.0 '82574L Gigabit Network Connection' if=enp7s0 drv=e1000e > unused=vfio-pci > > [...] > > And I get: > > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: PCI device > 0000:01:00.0 on NUMA socket -1 > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: probe driver: > 8086:10fb net_ixgbe > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: Requested device > 0000:01:00.0 cannot be used > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: Driver cannot attach > the device (0000:01:00.0) > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: 2017-04-20T16:19:22Z|00027|netdev_dpdk|WARN|Error > attaching device '0000:01:00.0' to DPDK > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: 2017-04-20T16:19:22Z|00028|netdev|WARN|dpdk-p1: > could not set configuration (Invalid argumen > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: PCI device > 0000:01:00.0 on NUMA socket -1 > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: probe driver: > 8086:10fb net_ixgbe > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: Requested device > 0000:01:00.0 cannot be used > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: Driver cannot attach > the device (0000:01:00.0) > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: ovs|00027|netdev_dpdk|WARN|Error > attaching device '0000:01:00.0' to DPDK > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: > ovs|00028|netdev|WARN|dpdk-p1: could not set configuration (Invalid > argument) > > > Thanks, > Vincenzo > > > 2017-04-20 18:13 GMT+02:00 Shivaram Mysore : > >> I have a feeling that vfio-pci is not installed. Note that you will have >> to edit /etc/modules to load on boot. >> >> You can also check by >> # lsmod | grep vfio >> >> To load on demand - use >> # modprobe vfio-pci >> >> >> >> On Thu, Apr 20, 2017 at 9:03 AM, Vincenzo Maffione >> wrote: >> >>> Hi, >>> I installed OVS 2.7.0 and use uio_pci_generic in place of vfio-pci, >>> and the error disappeared. >>> >>> Thanks, >>> Vincenzo >>> >>> 2017-04-15 17:56 GMT+02:00 Shivaram Mysore : >>> >>>> You should look at emails from -(Hint: search DPDK in subject) >>>> >>>> https://mail.openvswitch.org/pipermail/ovs-discuss/2017-Marc >>>> h/thread.html >>>> https://mail.openvswitch.org/pipermail/ovs-discuss/2017-Apri >>>> l/thread.html >>>> https://mail.openvswitch.org/pipermail/ovs-discuss/2017-Apri >>>> l/044155.html >>>> >>>> If possible, I would recommend using Ubuntu 17.04 with OVS 2.7 with >>>> DPDK packages from http://packages.wand.net.nz/ which can simplify >>>> your deployment. >>>> >>>> /Shivaram >>>> >>>> On Fri, Apr 14, 2017 at 4:04 AM, Vincenzo Maffione < >>>> vincenzo.maffione@ing.unipi.it> wrote: >>>> >>>>> Hi, >>>>> I would like to attach one of my ixgbe NIC ports to an OpenVSwitch >>>>> instance, but I keep getting an unexpected error, see below. >>>>> >>>>> I've successfully bound both my ixgbe NIC ports to DPDK, using VFIO: >>>>> >>>>> # tools/dpdk-devbind.py --status >>>>> >>>>> Network devices using DPDK-compatible driver >>>>> ============================================ >>>>> 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' >>>>> drv=vfio-pci >>>>> unused=ixgbe >>>>> 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' >>>>> drv=vfio-pci >>>>> unused=ixgbe >>>>> >>>>> Network devices using kernel driver >>>>> =================================== >>>>> 0000:00:19.0 'Ethernet Connection (2) I218-V' if=eno1 drv=e1000e >>>>> unused=vfio-pci >>>>> 0000:06:00.0 'RTL8111/8168/8411 PCI Express Gigabit Ethernet >>>>> Controller' >>>>> if=enp6s0 drv=r8169 unused=vfio-pci >>>>> 0000:07:00.0 '82574L Gigabit Network Connection' if=enp7s0 drv=e1000e >>>>> unused=vfio-pci >>>>> >>>>> Other network devices >>>>> ===================== >>>>> >>>>> >>>>> Crypto devices using DPDK-compatible driver >>>>> =========================================== >>>>> >>>>> >>>>> Crypto devices using kernel driver >>>>> ================================== >>>>> >>>>> >>>>> Other crypto devices >>>>> ==================== >>>>> >>>>> >>>>> >>>>> Then I created a bridge >>>>> >>>>> # ovs-vsctl add-br obr1 -- set bridge obr1 datapath_type=netdev >>>>> >>>>> But then when trying to add the dpdk NIC: >>>>> >>>>> # ovs-vsctl add-port obr1 dpdk-p0 -- set Interface dpdk-p0 type=dpdk >>>>> options:dpdk-devargs=0000:01:00.0 >>>>> >>>>> the command fails, and I get this in the log: >>>>> >>>>> ovs-vswitchd[353]: 2017-04-14T10:54:06Z|00026|bridge|WARN|could not >>>>> open >>>>> network device dpdk-p0 (No such device) >>>>> ovs-vswitchd[353]: ovs|00026|bridge|WARN|could not open network device >>>>> dpdk-p0 (No such device) >>>>> >>>>> Note that I've successfully managed to setup an OVS instance with two >>>>> DPDK >>>>> vhost-user ports (each one assigned to a QEMU VM): this is why I'm >>>>> confident my OVS is built and configured correctly to work with DPDK. >>>>> Also, >>>>> DPDK by itself is working as expected, I can use dpdk pktgen. >>>>> Moreover, OVS log shows that DPDK is recognized >>>>> >>>>> [...] >>>>> ovs|00005|dpdk|INFO|DPDK Enabled, initializing >>>>> [...] >>>>> >>>>> Note that I'm carefully following the documentation here >>>>> http://docs.openvswitch.org/en/latest/intro/install/dpdk/ and here >>>>> http://docs.openvswitch.org/en/latest/howto/dpdk/ >>>>> >>>>> More info: >>>>> >>>>> DPDK version: 16.11 >>>>> OVS version: 2.6.1, configured with --with-dpdk >>>>> Host: Archlinux >>>>> /proc/cmdline: BOOT_IMAGE=../vmlinuz-linux-vfio >>>>> root=UUID=1c777330-3f00-4ab8-8d5d-d4b2b5d860b0 rw intel_iommu=on >>>>> hugepages=2048 pci=assign-busses pcie_acs_override=downstream >>>>> initrd=../initramfs-linux-vfio.img >>>>> >>>>> Can anybody help? I'm at a loss here. >>>>> >>>>> Thanks, >>>>> Vincenzo >>>>> >>>> >>>> >>> >>> >>> -- >>> Vincenzo Maffione >>> >> >> >