From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yw0-f196.google.com (mail-yw0-f196.google.com [209.85.161.196]) by dpdk.org (Postfix) with ESMTP id E5458379B for ; Sun, 31 Jul 2016 14:28:17 +0200 (CEST) Received: by mail-yw0-f196.google.com with SMTP id r9so11187400ywg.2 for ; Sun, 31 Jul 2016 05:28:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=DAspC7Qe+eZI2P/cguiDbGacvx5Wwghf5uhFFUxoUs4=; b=CRdCNwwjT4/5ExA1BumzRoTZ2CenabIgT6liOgreVdWdQXBfwReiOb8/WB0Xgj0oZ6 pIf/V4xWVeJJkdDrl6bqTbAc6kmLtKCSIgnArU0Aubr5qbl/4MEAcyyer2z14BUMxPEv 4I07xYFw3woZ1fPDm/bxYgRDZQK6V012KNGLZmiN5untaeoLJ6Gutkn7zDNcwV4uTOsp CyXzVBWu5sztfAQyT86oRXhtRa20tvFL5cG9rKSOWUhnRSL3aIkM5VK7zIFdHD1uTDWi mRj/m07wiAwcVvqw0M+IRPDZ/mwK7+PEgYCR7Wby8Qp7s/hmaihVMIc0U4TTG8yU8v/D A/Dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=DAspC7Qe+eZI2P/cguiDbGacvx5Wwghf5uhFFUxoUs4=; b=jT9lB1mrHDnkKdU0HB3rp1YtdPTj1MEJH5/3QhxYLTEiphIQzzxnSR3CJmykpY13Tv g9XKbRhsn8J0fQQan3agYXd9t9J6E80v7yhrUeFzv/UVCV2gggqodyY7JvFQLpWN27Vj xo/FZrjpz2O8SC5nvocWeZJRbDzgYLDrOgHiMXhSWvrmyoao6IWi6pMVF5lFUHThw+oL o20oqsA/iUsnN9G3iTSeC+f61A1+th6cJOrZaJwhp1zk0tQw+eprbWfZI3dIyzD8oIdG yPnrLipqTRVmOc7hBhBRxPp9GDeDXqL1bweDxFdKk8u3ZtIizZ8V7OPdJhf4UUVkiq28 QsnA== X-Gm-Message-State: AEkoouugt4Qf0racTQCgoAM0xxHG+PrHu58N0rp+O3GsCDHAPKCAydPioYlAHWl/p4Qpnb4xOlYRMVd+ifWUNA== X-Received: by 10.13.201.199 with SMTP id l190mr42467327ywd.181.1469968096957; Sun, 31 Jul 2016 05:28:16 -0700 (PDT) MIME-Version: 1.0 Received: by 10.13.239.134 with HTTP; Sun, 31 Jul 2016 05:28:16 -0700 (PDT) From: Kapil Adhikesavalu Date: Sun, 31 Jul 2016 17:58:16 +0530 Message-ID: To: users@dpdk.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] OVS+DPDK: pci_map_resource(): cannot mmap error X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Jul 2016 12:28:18 -0000 Hello, i am getting "EAL: pci_map_resource(): cannot mmap(18, 0x7f5040000000, 0x80000, 0x0): Invalid argument (0xffffffffffffffff)" when i start ovs-vswitchd. Setup:DL360gen8 CPU:E5-2967 NIC:82599ES 10-Gigabit SFI/SFP+ (2 Port) (PCI: slot0: 04:00.0 04:00.1) Kernel: 4.6.4-201.fc23.x86_64 ixgbe driver: 4.2.1-k i have installed : DPDK-16.04 & OVS 2.5.90 - enabled with DPDK steps after installation: ===== 1. default_hugepagesz=1G hugepagesz=1G hugepages=16 hugepagesz=2M hugepages=2048 intel_iommu=off 2. mount -t hugetlbfs nodev /mnt/huge -o pagesize=1GB mount -t hugetlbfs nodev /mnt/huge_2mb -o pagesize=2MB 2. sudo modprobe uio [note, i am not using VFIO/IOMMU - i tried with that as well after enable iommu in grub - didnt help - got a different error] 3. tools/dpdk_nic_bind.py -b igb_uio 04:00.0 tools/dpdk_nic_bind.py -b igb_uio 04:00.1 4. mkdir -p $ovsdir/etc/openvswitch ovsdb-tool create $ovsdir/etc/openvswitch/conf.db $ovsdir/usr/share/openvswitch/vswitch.ovsschema # Bring up ovsdb-server daemon mkdir -p $ovsdir/var/run/openvswitch export OVS_DB_SOCK=${ovsdir}/var/run/openvswitch/db.sock $ovsdir/sbin/ovsdb-server --remote=punix:${OVS_DB_SOCK} \ --remote=db:Open_vSwitch,Open_vSwitch,manager_options \ --private-key=db:Open_vSwitch,SSL,private_key \ --certificate=db:Open_vSwitch,SSL,certificate \ --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \ --pidfile --detach --verbose=dbg # Intialize the ovs database $ovsdir/bin/ovs-vsctl --no-wait init 5. $ovsdir/bin/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true $ovsdir/bin/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="1024,1024" # Use Core 1 for user mode ovsvswitch $ovsdir/bin/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x2 # Number of memory channels on targeted platform $ovsdir/bin/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-extra="-n 4" 6. $ovsdir/sbin/ovs-vswitchd unix:${OVS_DB_SOCK} --pidfile --verbose=dbg --detach Note: i also have another setup with DL360gen9, with the same configuration(UIO) and it is working without any issues. i am not able to isolate to what is causing this issue. Appreciate any help. Error logs: ======== 2016-07-31T13:15:57Z|00001|reconnect|DBG|unix:/var/run/openvswitch/db.sock: entering BACKOFF 2016-07-31T13:15:57Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0 2016-07-31T13:15:57Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1 2016-07-31T13:15:57Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores 2016-07-31T13:15:57Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2016-07-31T13:15:57Z|00006|reconnect|DBG|unix:/var/run/openvswitch/db.sock: entering CONNECTING 2016-07-31T13:15:57Z|00007|poll_loop|DBG|wakeup due to [POLLOUT] on fd 10 (<->/var/run/openvswitch/db.sock) at lib/stream-fd.c:151 2016-07-31T13:15:57Z|00008|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2016-07-31T13:15:57Z|00009|reconnect|DBG|unix:/var/run/openvswitch/db.sock: entering ACTIVE 2016-07-31T13:15:57Z|00010|jsonrpc|DBG|unix:/var/run/openvswitch/db.sock: send request, method="get_schema", params=["Open_vSwitch"], id=0 2016-07-31T13:15:57Z|00015|jsonrpc|DBG|unix:/var/run/openvswitch/db.sock: received reply, result={"locked":true}, id=1 2016-07-31T13:15:57Z|00016|poll_loop|DBG|wakeup due to [POLLIN] on fd 10 (<->/var/run/openvswitch/db.sock) at lib/stream-fd.c:155 2016-07-31T13:15:57Z|00017|jsonrpc|DBG|unix:/var/run/openvswitch/db.sock: received reply, result={"Open_vSwitch":{"214ea68a-3cf4-4e8e-a319-4f0ba6205f17":{"initial":{"other_config":["map",[["dpdk-extra","-n 4"],["dpdk-init","true"],["dpdk-lcore-mask","0x2"],["dpdk-socket-mem","1024,1024"]]]}}}}, id=2 2016-07-31T13:15:57Z|00018|dpdk|INFO|DPDK Enabled, initializing 2016-07-31T13:15:57Z|00019|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2016-07-31T13:15:57Z|00020|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x2 --socket-mem 1024,1024 -n 4 EAL: Detected lcore 47 as core 13 on socket 1 EAL: Support maximum 128 logical core(s) by configuration. EAL: Detected 48 lcore(s) EAL: Probing VFIO support... EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory) EAL: VFIO modules not loaded, skipping VFIO support... EAL: Setting up physically contiguous memory... EAL: Ask a virtual area of 0x200000000 bytes EAL: Virtual area found at 0x7f5000000000 (size = 0x200000000) EAL: Ask a virtual area of 0x200000000 bytes EAL: Virtual area found at 0x7f4dc0000000 (size = 0x200000000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7f52bca00000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7f52bc600000 (size = 0x200000) EAL: Ask a virtual area of 0x8800000 bytes EAL: Virtual area found at 0x7f52b3c00000 (size = 0x8800000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7f52b3800000 (size = 0x200000) EAL: Ask a virtual area of 0x3fc00000 bytes EAL: Virtual area found at 0x7f5273a00000 (size = 0x3fc00000) EAL: Ask a virtual area of 0x37400000 bytes EAL: Virtual area found at 0x7f523c400000 (size = 0x37400000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7f523c000000 (size = 0x200000) EAL: Ask a virtual area of 0x48000000 bytes EAL: Virtual area found at 0x7f4d77e00000 (size = 0x48000000) EAL: Ask a virtual area of 0x38000000 bytes EAL: Virtual area found at 0x7f5203e00000 (size = 0x38000000) EAL: Requesting 1 pages of size 1024MB from socket 0 EAL: Requesting 1 pages of size 1024MB from socket 1 EAL: TSC frequency is ~2693520 KHz EAL: Master lcore 1 is ready (tid=c11bdbc0;cpuset=[1]) *EAL: PCI device 0000:04:00.0 on NUMA socket 0* *EAL: probe driver: 8086:10fb rte_ixgbe_pmd* *EAL: pci_map_resource(): cannot mmap(18, 0x7f5040000000, 0x80000, 0x0): Invalid argument (0xffffffffffffffff)* *EAL: Error - exiting with code: 1* * Cause: Requested device 0000:04:00.0 cannot be used* [root@localhost ~]# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-4.6.4-201.fc23.x86_64 root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb quiet default_hugepagesz=1G hugepagesz=1G hugepages=16 hugepagesz=2M hugepages=2048 intel_iommu=off [root@localhost ~]# cat /proc/meminfo | grep uge AnonHugePages: 0 kB HugePages_Total: 16 HugePages_Free: 14 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB [root@localhost ~]# lsmod | grep uio igb_uio 16384 0 uio 20480 1 igb_uio [root@localhost ~]# /localdisk/dpdk/dpdk-16.04/tools/dpdk_nic_bind.py --status Network devices using DPDK-compatible driver ============================================ 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe Network devices using kernel driver =================================== 0000:03:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno1 drv=tg3 unused=igb_uio *Active* 0000:03:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno2 drv=tg3 unused=igb_uio 0000:03:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno3 drv=tg3 unused=igb_uio 0000:03:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe' if=eno4 drv=tg3 unused=igb_uio [root@localhost ~]# dmesg | grep 04:00 [ 3.301717] pci 0000:04:00.0: [8086:10fb] type 00 class 0x020000 [ 3.301728] pci 0000:04:00.0: reg 0x10: [mem 0xf7f80000-0xf7ffffff 64bit] [ 3.301733] pci 0000:04:00.0: reg 0x18: [io 0x6000-0x601f] [ 3.301744] pci 0000:04:00.0: reg 0x20: [mem 0xf7f70000-0xf7f73fff 64bit] [ 3.301750] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref] [ 3.301781] pci 0000:04:00.0: PME# supported from D0 D3hot [ 3.301799] pci 0000:04:00.0: reg 0x184: [mem 0xf7e70000-0xf7e73fff 64bit] [ 3.301800] pci 0000:04:00.0: VF(n) BAR0 space: [mem 0xf7e70000-0xf7f6ffff 64bit] (contains BAR0 for 64 VFs) [ 3.301810] pci 0000:04:00.0: reg 0x190: [mem 0xf7d70000-0xf7d73fff 64bit] [ 3.301811] pci 0000:04:00.0: VF(n) BAR3 space: [mem 0xf7d70000-0xf7e6ffff 64bit] (contains BAR3 for 64 VFs) [ 3.301988] pci 0000:04:00.1: [8086:10fb] type 00 class 0x020000 [ 3.301999] pci 0000:04:00.1: reg 0x10: [mem 0xf7c80000-0xf7cfffff 64bit] [ 3.302004] pci 0000:04:00.1: reg 0x18: [io 0x6020-0x603f] [ 3.302015] pci 0000:04:00.1: reg 0x20: [mem 0xf7c70000-0xf7c73fff 64bit] [ 3.302021] pci 0000:04:00.1: reg 0x30: [mem 0x00000000-0x0007ffff pref] [ 3.302052] pci 0000:04:00.1: PME# supported from D0 D3hot [ 3.302066] pci 0000:04:00.1: reg 0x184: [mem 0xf7b70000-0xf7b73fff 64bit] [ 3.302067] pci 0000:04:00.1: VF(n) BAR0 space: [mem 0xf7b70000-0xf7c6ffff 64bit] (contains BAR0 for 64 VFs) [ 3.302076] pci 0000:04:00.1: reg 0x190: [mem 0xf7a70000-0xf7a73fff 64bit] [ 3.302078] pci 0000:04:00.1: VF(n) BAR3 space: [mem 0xf7a70000-0xf7b6ffff 64bit] (contains BAR3 for 64 VFs) [ 3.349170] pci 0000:04:00.0: BAR 6: no space for [mem size 0x00080000 pref] [ 3.349172] pci 0000:04:00.0: BAR 6: failed to assign [mem size 0x00080000 pref] [ 3.349173] pci 0000:04:00.1: BAR 6: no space for [mem size 0x00080000 pref] [ 3.349174] pci 0000:04:00.1: BAR 6: failed to assign [mem size 0x00080000 pref] [ 7.293595] ixgbe 0000:04:00.0: Multiqueue Enabled: Rx Queue count = 48, Tx Queue count = 48 [ 7.293735] ixgbe 0000:04:00.0: PCI Express bandwidth of 32GT/s available [ 7.293736] ixgbe 0000:04:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%) [ 7.293826] ixgbe 0000:04:00.0: MAC: 2, PHY: 1, PBA No: E66560-005 [ 7.293827] ixgbe 0000:04:00.0: 90:e2:ba:1d:18:50 [ 7.296841] ixgbe 0000:04:00.0: Intel(R) 10 Gigabit Network Connection [ 8.397128] ixgbe 0000:04:00.1: Multiqueue Enabled: Rx Queue count = 48, Tx Queue count = 48 [ 8.397254] ixgbe 0000:04:00.1: PCI Express bandwidth of 32GT/s available [ 8.397255] ixgbe 0000:04:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%) [ 8.397336] ixgbe 0000:04:00.1: MAC: 2, PHY: 1, PBA No: E66560-005 [ 8.397337] ixgbe 0000:04:00.1: 90:e2:ba:1d:18:51 [ 8.398676] ixgbe 0000:04:00.1: Intel(R) 10 Gigabit Network Connection [ 8.399637] ixgbe 0000:04:00.1 ens1f1: renamed from eth1 [ 8.409980] ixgbe 0000:04:00.0 ens1f0: renamed from eth0 [ 18.835386] ixgbe 0000:04:00.1: registered PHC device on ens1f1 [ 19.379900] ixgbe 0000:04:00.0: registered PHC device on ens1f0 [ 522.645300] ixgbe 0000:04:00.0: removed PHC on ens1f0 [ 522.645300] ixgbe 0000:04:00.0: removed PHC on ens1f0 [ 523.099810] ixgbe 0000:04:00.0: complete [ 523.100283] igb_uio 0000:04:00.0: uio device registered with irq 1b [ 523.291072] ixgbe 0000:04:00.1: removed PHC on ens1f1 [ 523.738970] ixgbe 0000:04:00.1: complete [ 523.739500] igb_uio 0000:04:00.1: uio device registered with irq 52 Thanks Kapil.