Re: [dpdk-dev] Issues with openvswitch2.5+dpdk2.2+kvm/virtio-pci

[Christian Ehrhardt <christian.ehrhardt@canonical.com>]

FYI - a user also reported that the same issue also affects non-virtual
environments when using the BNX2X_PMD

Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

On Fri, Mar 18, 2016 at 4:20 PM, Christian Ehrhardt <
christian.ehrhardt@canonical.com> wrote:

> Hi,
> I was trying to replicate a setup that I have working on physical devices
> (ixgbe) under kvm since there is a virtio pmd driver.
>
> TL;DR:
> - under KVM with virtio-pci (working on baremetal with ixgbe cards)
> - adding dpdk port to ovs fails with memzone <port0_tvq0> already exists
> and causes a segfault
> - I couldn't find a solution in similar mails that popped up here
> recently, any help or pointer appreciated.
>
> ## Details ##
> I thought I've read that others have it working I thought that would be a
> great way to gain more debug control of the environment, but something
> seems to be eluding me.
>
> There were quite some similar mails on the List recently, but none seemed
> to hit the same issue as I do. At least none of the tunings/workarounds
> seemed to apply to me.
> As versions I have Openvswitch 2.5, DPDK 2.2, Qemu 2.5, Kernel 4.4 - so a
> fairly recent software stack.
>
> The super-short repro summary is:
> 1. starting ovs-dpdk like
> ovs-vswitchd --dpdk -c 0x1 -n 4 --pci-blacklist 0000:00:03.0 -m 2048 --
> unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info
> --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log
> --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
> 2. add a bridge and a ovs dpdk port
> ovs-vsctl add-port ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk
> ovs-vsctl add-port ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk
>
> The log of the initialization after #1 looks good to me - I can see two of
> my three virtio devices recognized and one blacklisted.
> Memory allocation looks good, ... I'll attach the log at the end of the
> mail
>
>
> ## ISSUE ##
> But when I add a port and refer to one of the dpdk ports it fails with the
> following:
> ovs-vsctl[14023]: ovs|00001|vsctl|INFO|Called as ovs-vsctl add-port
> ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk
> ovs-vswitchd[13903]: EAL: memzone_reserve_aligned_thread_unsafe(): memzone
> <port0_tvq0> already exists
> ovs-vswitchd[13903]: EAL: memzone_reserve_aligned_thread_unsafe(): memzone
> <port0_tvq0_hdrzone> already exists
> ovs-vswitchd[13903]: EAL: memzone_reserve_aligned_thread_unsafe(): memzone
> <port0_rvq0> already exists
> kernel: show_signal_msg: 18 callbacks suppressed
> kernel: pmd12[14025]: segfault at 2 ip 00007f3eb205eab2 sp
> 00007f3e3dffa590 error 4 in libdpdk.so.0[7f3eb1fdf000+1e9000]
> ovs-vswitchd[13902]: ovs|00003|daemon_unix(monitor)|ERR|1 crashes: pid
> 13903 died, killed (Segmentation fault), core dumped, restarting
> systemd-udevd[14040]: Could not generate persistent MAC address for
> ovs-netdev: No such file or directory
> kernel: device ovs-netdev entered promiscuous mode
> ovs-vswitchd[14036]: EAL: memzone_reserve_aligned_thread_unsafe(): memzone
> <RG_MP_ovs_mp_1500_0_262144> already exists
> ovs-vswitchd[14036]: RING: Cannot reserve memory
> kernel: device ovsdpdkbr0 entered promiscuous mode
> ovs-vswitchd[14036]: EAL: memzone_reserve_aligned_thread_unsafe(): memzone
> <RG_MP_ovs_mp_1500_0_262144> already exists
> ovs-vswitchd[14036]: RING: Cannot reserve memory
>
>
> ## Experiments (failed) ##
> I thought it could be related to all the multiqueue chances that recently
> going in.
> My usual setup has 4 vCPUs and 4 queues per virtio-net device.
> I tried them with only 1 of 4 queues, also with only 1 queue defined and
> only 1 CPU - but all fail the same way.
> I have testpmd and l2fwd on the same devices working, so I hope they are
> not totally set up badly.
>
> I also tried hilarious things like reassigning to uio_pci_generic before,
> but well its virtio_pmd eventually anyways - so it made no difference.
>
> From how it appears I felt that it could be related to the old discussions
> around
> [1] http://dpdk.org/ml/archives/dev/2015-May/017589.html
> [2] http://openvswitch.org/pipermail/dev/2015-March/052344.html
> But they are (partially) applied upstream already and the issue doesn't
> 100% match the old discussions.
>
>
> ## Logs ##
> [3] log of openvswitch start:
> systemd[1]: Starting Open vSwitch Internal Unit...
> ovs-ctl[13868]:  * Starting ovsdb-server
> ovs-vsctl[13893]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait --
> init -- set Open_vSwitch . db-version=7.12.1
> ovs-vsctl[13898]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set
> Open_vSwitch . ovs-version=2.5.0
> "external-ids:system-id=\"8ddb892c-53a5-410d-a765-0031ad6eb1be\""
> "system-type=\"Ubuntu\"" "system-version=\"16.04-xenial\""
> ovs-ctl[13868]:  * Configuring Open vSwitch system IDs
> ovs-ctl[13868]: 2016-03-18T14:28:28Z|00001|dpdk|INFO|No -vhost_sock_dir
> provided - defaulting to /var/run/openvswitch
> ovs-vswitchd[13900]: ovs|00001|dpdk|INFO|No -vhost_sock_dir provided -
> defaulting to /var/run/openvswitch
> ovs-ctl[13868]: EAL: Detected lcore 0 as core 0 on socket 0
> ovs-ctl[13868]: EAL: Detected lcore 1 as core 0 on socket 0
> ovs-ctl[13868]: EAL: Detected lcore 2 as core 0 on socket 0
> ovs-ctl[13868]: EAL: Detected lcore 3 as core 0 on socket 0
> ovs-ctl[13868]: EAL: Support maximum 128 logical core(s) by configuration.
> ovs-ctl[13868]: EAL: Detected 4 lcore(s)
> ovs-ctl[13868]: EAL: No free hugepages reported in hugepages-1048576kB
> ovs-ctl[13868]: EAL: VFIO modules not all loaded, skip VFIO support...
> ovs-ctl[13868]: EAL: Setting up physically contiguous memory...
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3eaf800000 (size = 0x200000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x5ac00000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e54a00000 (size =
> 0x5ac00000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0xc00000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e53c00000 (size = 0xc00000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x1200000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e52800000 (size =
> 0x1200000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e52200000 (size = 0x400000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e51e00000 (size = 0x200000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e51800000 (size = 0x400000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e51400000 (size = 0x200000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0xc00000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e50600000 (size = 0xc00000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x1000000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4f400000 (size =
> 0x1000000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4f000000 (size = 0x200000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4ec00000 (size = 0x200000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4e800000 (size = 0x200000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4e400000 (size = 0x200000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4de00000 (size = 0x400000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4da00000 (size = 0x200000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x5ac00000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3df2c00000 (size =
> 0x5ac00000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0xc00000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3df1e00000 (size = 0xc00000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3df1800000 (size = 0x400000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x1000000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3df0600000 (size =
> 0x1000000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3df0200000 (size = 0x200000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3defc00000 (size = 0x400000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3def800000 (size = 0x200000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3def200000 (size = 0x400000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x600000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3deea00000 (size = 0x600000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x1800000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3ded000000 (size =
> 0x1800000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x600000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3dec800000 (size = 0x600000)
> ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes
> ovs-ctl[13868]: EAL: Virtual area found at 0x7f3dec400000 (size = 0x200000)
> ovs-ctl[13868]: EAL: Requesting 1024 pages of size 2MB from socket 0
> ovs-ctl[13868]: EAL: TSC frequency is ~2397235 KHz
> ovs-vswitchd[13900]: EAL: TSC frequency is ~2397235 KHz
> ovs-ctl[13868]: EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no ->
> using unreliable clock cycles !
> ovs-ctl[13868]: EAL: Master lcore 0 is ready (tid=b2f7ab00;cpuset=[0])
> ovs-ctl[13868]: EAL: PCI device 0000:00:03.0 on NUMA socket -1
> ovs-ctl[13868]: EAL:   probe driver: 1af4:1000 rte_virtio_pmd
> ovs-ctl[13868]: EAL:   Device is blacklisted, not initializing
> ovs-ctl[13868]: EAL: PCI device 0000:00:04.0 on NUMA socket -1
> ovs-ctl[13868]: EAL:   probe driver: 1af4:1000 rte_virtio_pmd
> ovs-vswitchd[13900]: EAL: WARNING: cpu flags constant_tsc=yes
> nonstop_tsc=no -> using unreliable clock cycles !
> ovs-ctl[13868]: EAL: PCI device 0000:00:05.0 on NUMA socket -1
> ovs-ctl[13868]: EAL:   probe driver: 1af4:1000 rte_virtio_pmd
> ovs-vswitchd[13900]: EAL: Master lcore 0 is ready (tid=b2f7ab00;cpuset=[0])
> ovs-vswitchd[13900]: EAL: PCI device 0000:00:03.0 on NUMA socket -1
> ovs-vswitchd[13900]: EAL:   probe driver: 1af4:1000 rte_virtio_pmd
> ovs-vswitchd[13900]: EAL:   Device is blacklisted, not initializing
> ovs-vswitchd[13900]: EAL: PCI device 0000:00:04.0 on NUMA socket -1
> ovs-vswitchd[13900]: EAL:   probe driver: 1af4:1000 rte_virtio_pmd
> ovs-vswitchd[13900]: EAL: PCI device 0000:00:05.0 on NUMA socket -1
> ovs-vswitchd[13900]: EAL:   probe driver: 1af4:1000 rte_virtio_pmd
> ovs-ctl[13868]: Zone 0: name:<RG_MP_log_history>, phys:0x1495fdec0,
> len:0x2080, virt:0x7f3e4dbfdec0, socket_id:0, flags:0
> ovs-ctl[13868]: Zone 1: name:<MP_log_history>, phys:0xb7d75f00,
> len:0x28a0c0, virt:0x7f3e4df75f00, socket_id:0, flags:0
> ovs-ctl[13868]: Zone 2: name:<rte_eth_dev_data>, phys:0xb7d467c0,
> len:0x2f700, virt:0x7f3e4df467c0, socket_id:0, flags:0
> ovs-ctl[13868]: Zone 3: name:<port0_cvq>, phys:0xb7d43000, len:0x2000,
> virt:0x7f3e4df43000, socket_id:0, flags:0
> ovs-ctl[13868]: Zone 4: name:<port0_cvq_hdrzone>, phys:0xb7d41fc0,
> len:0x1000, virt:0x7f3e4df41fc0, socket_id:0, flags:0
> ovs-ctl[13868]: Zone 5: name:<port1_cvq>, phys:0xb7d3f000, len:0x2000,
> virt:0x7f3e4df3f000, socket_id:0, flags:0
> ovs-ctl[13868]: Zone 6: name:<port1_cvq_hdrzone>, phys:0xb7d3dfc0,
> len:0x1000, virt:0x7f3e4df3dfc0, socket_id:0, flags:0
> ovs-ctl[13868]:  * Starting ovs-vswitchd
> ovs-ctl[13868]:  * Enabling remote OVSDB managers
> systemd[1]: Started Open vSwitch Internal Unit.
>
> P.S. adding a few people I usually see replying on these topics directly.
> Thanks in advance for sharing your thoughts!
>
> Christian Ehrhardt
> Software Engineer, Ubuntu Server
> Canonical Ltd
>