From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qg0-f53.google.com (mail-qg0-f53.google.com [209.85.192.53]) by dpdk.org (Postfix) with ESMTP id 753695590 for ; Fri, 18 Mar 2016 16:21:16 +0100 (CET) Received: by mail-qg0-f53.google.com with SMTP id w104so102531724qge.1 for ; Fri, 18 Mar 2016 08:21:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical-com.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=iMup07BEpt+HuEfw7ehWz9E+ChA9Yng/MwCvPF4VuZk=; b=JptvnZPuVs20MjghZApQ1HVxFYfDnVFuRwMFWU6EJ+M2pksJGA4rhrzVsSnSjFjIf9 /GsHS+nblrGF4cKZ3Jw92cqG+THjvkSFBNZD3Ip3h0Y5UBg27pbQ4TuEVMb4N7KeB6vh M01cG0aTtCKp413i87jpCascAz1JUeaW279oMAwIjbCVmrzfnfUyyDREtWBrdC71QzEi yyPsxiyY1BjpblWEmt9SrnfeNLb+9jphCaEVkr6vc17l55xuSQHmbeqBi40e0DytrE+x OY/VC0A8doTRgnPsbwZ9Ysx7Ybx5UKbgFg3r7gUX51VvrzhGveq1fM/M0aIBwiKAUvUN 8RWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=iMup07BEpt+HuEfw7ehWz9E+ChA9Yng/MwCvPF4VuZk=; b=XswBWdRRM0w/zGqaESyhuUFySdrhb/JnzztnqFipwmI5EHd/uO+ZJ+Muf0U9L58dY6 kyGNNCQkhNJryyP/rWbqr57qyIaMf0GCbonlAsvIvcbjeH1yvgSOXDI8z49zE1ykw1jd DyECs0F7nvI5D+yV/yRNv2WdOYhRRF+YyCwYCNgtTpYyI7Ny2pPOqzwg8s64wkN8Wjcm 55bIRghIL+TcJKic1qEURj1P8E6V4EU2fHd8DOBv0kJWrMcjyqTnFgz7NI3jhEtJ5x6h CkoOTpvdEmAXuD2wAJfAZikv4Jh+ocbijg4tbC4ALBTYN5rigZfXItM+yq/3mqcMqQR6 +YQA== X-Gm-Message-State: AD7BkJIwgfasL6UxL925KWA7nBscKvQvO1UquIWuAhqkcNgWQwTHmhtl7Qy6t0ZJKv0vPOY/cUDq2LFRfg5QV0nx X-Received: by 10.140.88.11 with SMTP id s11mr22288127qgd.93.1458314475866; Fri, 18 Mar 2016 08:21:15 -0700 (PDT) MIME-Version: 1.0 Received: by 10.55.169.6 with HTTP; Fri, 18 Mar 2016 08:20:56 -0700 (PDT) From: Christian Ehrhardt Date: Fri, 18 Mar 2016 16:20:56 +0100 Message-ID: To: dev , discuss@openvswitch.org, diproiettod@vmware.com, ciara.loftus@intel.com Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-dev] Issues with openvswitch2.5+dpdk2.2+kvm/virtio-pci X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Mar 2016 15:21:16 -0000 Hi, I was trying to replicate a setup that I have working on physical devices (ixgbe) under kvm since there is a virtio pmd driver. TL;DR: - under KVM with virtio-pci (working on baremetal with ixgbe cards) - adding dpdk port to ovs fails with memzone already exists and causes a segfault - I couldn't find a solution in similar mails that popped up here recently, any help or pointer appreciated. ## Details ## I thought I've read that others have it working I thought that would be a great way to gain more debug control of the environment, but something seems to be eluding me. There were quite some similar mails on the List recently, but none seemed to hit the same issue as I do. At least none of the tunings/workarounds seemed to apply to me. As versions I have Openvswitch 2.5, DPDK 2.2, Qemu 2.5, Kernel 4.4 - so a fairly recent software stack. The super-short repro summary is: 1. starting ovs-dpdk like ovs-vswitchd --dpdk -c 0x1 -n 4 --pci-blacklist 0000:00:03.0 -m 2048 -- unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor 2. add a bridge and a ovs dpdk port ovs-vsctl add-port ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk ovs-vsctl add-port ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk The log of the initialization after #1 looks good to me - I can see two of my three virtio devices recognized and one blacklisted. Memory allocation looks good, ... I'll attach the log at the end of the mail ## ISSUE ## But when I add a port and refer to one of the dpdk ports it fails with the following: ovs-vsctl[14023]: ovs|00001|vsctl|INFO|Called as ovs-vsctl add-port ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk ovs-vswitchd[13903]: EAL: memzone_reserve_aligned_thread_unsafe(): memzone already exists ovs-vswitchd[13903]: EAL: memzone_reserve_aligned_thread_unsafe(): memzone already exists ovs-vswitchd[13903]: EAL: memzone_reserve_aligned_thread_unsafe(): memzone already exists kernel: show_signal_msg: 18 callbacks suppressed kernel: pmd12[14025]: segfault at 2 ip 00007f3eb205eab2 sp 00007f3e3dffa590 error 4 in libdpdk.so.0[7f3eb1fdf000+1e9000] ovs-vswitchd[13902]: ovs|00003|daemon_unix(monitor)|ERR|1 crashes: pid 13903 died, killed (Segmentation fault), core dumped, restarting systemd-udevd[14040]: Could not generate persistent MAC address for ovs-netdev: No such file or directory kernel: device ovs-netdev entered promiscuous mode ovs-vswitchd[14036]: EAL: memzone_reserve_aligned_thread_unsafe(): memzone already exists ovs-vswitchd[14036]: RING: Cannot reserve memory kernel: device ovsdpdkbr0 entered promiscuous mode ovs-vswitchd[14036]: EAL: memzone_reserve_aligned_thread_unsafe(): memzone already exists ovs-vswitchd[14036]: RING: Cannot reserve memory ## Experiments (failed) ## I thought it could be related to all the multiqueue chances that recently going in. My usual setup has 4 vCPUs and 4 queues per virtio-net device. I tried them with only 1 of 4 queues, also with only 1 queue defined and only 1 CPU - but all fail the same way. I have testpmd and l2fwd on the same devices working, so I hope they are not totally set up badly. I also tried hilarious things like reassigning to uio_pci_generic before, but well its virtio_pmd eventually anyways - so it made no difference. >>From how it appears I felt that it could be related to the old discussions around [1] http://dpdk.org/ml/archives/dev/2015-May/017589.html [2] http://openvswitch.org/pipermail/dev/2015-March/052344.html But they are (partially) applied upstream already and the issue doesn't 100% match the old discussions. ## Logs ## [3] log of openvswitch start: systemd[1]: Starting Open vSwitch Internal Unit... ovs-ctl[13868]: * Starting ovsdb-server ovs-vsctl[13893]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.12.1 ovs-vsctl[13898]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=2.5.0 "external-ids:system-id=\"8ddb892c-53a5-410d-a765-0031ad6eb1be\"" "system-type=\"Ubuntu\"" "system-version=\"16.04-xenial\"" ovs-ctl[13868]: * Configuring Open vSwitch system IDs ovs-ctl[13868]: 2016-03-18T14:28:28Z|00001|dpdk|INFO|No -vhost_sock_dir provided - defaulting to /var/run/openvswitch ovs-vswitchd[13900]: ovs|00001|dpdk|INFO|No -vhost_sock_dir provided - defaulting to /var/run/openvswitch ovs-ctl[13868]: EAL: Detected lcore 0 as core 0 on socket 0 ovs-ctl[13868]: EAL: Detected lcore 1 as core 0 on socket 0 ovs-ctl[13868]: EAL: Detected lcore 2 as core 0 on socket 0 ovs-ctl[13868]: EAL: Detected lcore 3 as core 0 on socket 0 ovs-ctl[13868]: EAL: Support maximum 128 logical core(s) by configuration. ovs-ctl[13868]: EAL: Detected 4 lcore(s) ovs-ctl[13868]: EAL: No free hugepages reported in hugepages-1048576kB ovs-ctl[13868]: EAL: VFIO modules not all loaded, skip VFIO support... ovs-ctl[13868]: EAL: Setting up physically contiguous memory... ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3eaf800000 (size = 0x200000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x5ac00000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e54a00000 (size = 0x5ac00000) ovs-ctl[13868]: EAL: Ask a virtual area of 0xc00000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e53c00000 (size = 0xc00000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x1200000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e52800000 (size = 0x1200000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e52200000 (size = 0x400000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e51e00000 (size = 0x200000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e51800000 (size = 0x400000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e51400000 (size = 0x200000) ovs-ctl[13868]: EAL: Ask a virtual area of 0xc00000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e50600000 (size = 0xc00000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x1000000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4f400000 (size = 0x1000000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4f000000 (size = 0x200000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4ec00000 (size = 0x200000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4e800000 (size = 0x200000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4e400000 (size = 0x200000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4de00000 (size = 0x400000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4da00000 (size = 0x200000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x5ac00000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3df2c00000 (size = 0x5ac00000) ovs-ctl[13868]: EAL: Ask a virtual area of 0xc00000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3df1e00000 (size = 0xc00000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3df1800000 (size = 0x400000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x1000000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3df0600000 (size = 0x1000000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3df0200000 (size = 0x200000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3defc00000 (size = 0x400000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3def800000 (size = 0x200000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3def200000 (size = 0x400000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x600000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3deea00000 (size = 0x600000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x1800000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3ded000000 (size = 0x1800000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x600000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3dec800000 (size = 0x600000) ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes ovs-ctl[13868]: EAL: Virtual area found at 0x7f3dec400000 (size = 0x200000) ovs-ctl[13868]: EAL: Requesting 1024 pages of size 2MB from socket 0 ovs-ctl[13868]: EAL: TSC frequency is ~2397235 KHz ovs-vswitchd[13900]: EAL: TSC frequency is ~2397235 KHz ovs-ctl[13868]: EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles ! ovs-ctl[13868]: EAL: Master lcore 0 is ready (tid=b2f7ab00;cpuset=[0]) ovs-ctl[13868]: EAL: PCI device 0000:00:03.0 on NUMA socket -1 ovs-ctl[13868]: EAL: probe driver: 1af4:1000 rte_virtio_pmd ovs-ctl[13868]: EAL: Device is blacklisted, not initializing ovs-ctl[13868]: EAL: PCI device 0000:00:04.0 on NUMA socket -1 ovs-ctl[13868]: EAL: probe driver: 1af4:1000 rte_virtio_pmd ovs-vswitchd[13900]: EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles ! ovs-ctl[13868]: EAL: PCI device 0000:00:05.0 on NUMA socket -1 ovs-ctl[13868]: EAL: probe driver: 1af4:1000 rte_virtio_pmd ovs-vswitchd[13900]: EAL: Master lcore 0 is ready (tid=b2f7ab00;cpuset=[0]) ovs-vswitchd[13900]: EAL: PCI device 0000:00:03.0 on NUMA socket -1 ovs-vswitchd[13900]: EAL: probe driver: 1af4:1000 rte_virtio_pmd ovs-vswitchd[13900]: EAL: Device is blacklisted, not initializing ovs-vswitchd[13900]: EAL: PCI device 0000:00:04.0 on NUMA socket -1 ovs-vswitchd[13900]: EAL: probe driver: 1af4:1000 rte_virtio_pmd ovs-vswitchd[13900]: EAL: PCI device 0000:00:05.0 on NUMA socket -1 ovs-vswitchd[13900]: EAL: probe driver: 1af4:1000 rte_virtio_pmd ovs-ctl[13868]: Zone 0: name:, phys:0x1495fdec0, len:0x2080, virt:0x7f3e4dbfdec0, socket_id:0, flags:0 ovs-ctl[13868]: Zone 1: name:, phys:0xb7d75f00, len:0x28a0c0, virt:0x7f3e4df75f00, socket_id:0, flags:0 ovs-ctl[13868]: Zone 2: name:, phys:0xb7d467c0, len:0x2f700, virt:0x7f3e4df467c0, socket_id:0, flags:0 ovs-ctl[13868]: Zone 3: name:, phys:0xb7d43000, len:0x2000, virt:0x7f3e4df43000, socket_id:0, flags:0 ovs-ctl[13868]: Zone 4: name:, phys:0xb7d41fc0, len:0x1000, virt:0x7f3e4df41fc0, socket_id:0, flags:0 ovs-ctl[13868]: Zone 5: name:, phys:0xb7d3f000, len:0x2000, virt:0x7f3e4df3f000, socket_id:0, flags:0 ovs-ctl[13868]: Zone 6: name:, phys:0xb7d3dfc0, len:0x1000, virt:0x7f3e4df3dfc0, socket_id:0, flags:0 ovs-ctl[13868]: * Starting ovs-vswitchd ovs-ctl[13868]: * Enabling remote OVSDB managers systemd[1]: Started Open vSwitch Internal Unit. P.S. adding a few people I usually see replying on these topics directly. Thanks in advance for sharing your thoughts! Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd