DPDK usage discussions
 help / color / mirror / Atom feed
From: "Christian Graf" <chr.graf@gmx.de>
To: users@dpdk.org
Subject: [dpdk-users] dpdk vhost-user ovs unexpected low performance
Date: Mon, 14 Oct 2019 10:12:11 +0200	[thread overview]
Message-ID: <trinity-dcc4c5dc-3716-4261-a573-2de7a521285d-1571040731725@3c-app-gmx-bap74> (raw)

Dear all,

kindly asking for help.

I am observing packet-loss with dpdk-enabled OVS running vhost-user interfaces. As of now I am unclear if it is the dpdk-enabled VNF attached to OVS which is not capable of consuming the received packets, or if it is the OVS which does not properly send-out the packets.
I guess/hope that I just miss proper tuning of the VNF. So any further help or guidance whould be very much welcome.

As my troubleshooting-skills with such setup are limited, I am kindly asking for help and further guidance.

many thanks

christian


Topology
-----------
vnf (ge-0/0/4)--------[ vhost-user-vmx2 -ovs bridge br-dpdk--vhost-user-vmx1] -----vnf(ge-0/0/3)

 Issue
--------
ovs interface vhost-user-vmx1 claims to have egressed TX 1960035 packets towards attached vnf ge-0/0/3 interface.
But the vnf (ge-0/0/3) reports only 1599414 packets received (RX) - so somehow 400.000 packets get lost.
Packet-rate is about 200kpps


some general notes
-----------------------
The vnf and as well OVS run on numa node 0.
4 PMD's threads are pinned and as well l-core to dedicated cores. HT is enabled.
the vnf has all pinned cores as well
cores are isolated, hugepages enabled and HT enabled as well


ovs version
-------------
root@bn831x1a-node6-lab:~# sudo ovs-vsctl get Open_vSwitch . dpdk_initialized
true

root@bn831x1a-node6-lab:~# ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 2.11.0
DPDK 18.11.0

root@bn831x1a-node6-lab:~#  sudo ovs-vsctl get Open_vSwitch . dpdk_version
"DPDK 18.11.0"

root@bn831x1a-node6-lab:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 19.04
Release:        19.04
Codename:       disco

hugepages enabled
-----------------------
dcso@bn831x1a-node6-lab:~$ cat /sys/devices/system/node/node*/meminfo|grep Huge
Node 0 AnonHugePages:   1157120 kB
Node 0 ShmemHugePages:        0 kB
Node 0 HugePages_Total:    40
Node 0 HugePages_Free:     23
Node 0 HugePages_Surp:      0
Node 1 AnonHugePages:         0 kB
Node 1 ShmemHugePages:        0 kB
Node 1 HugePages_Total:    40
Node 1 HugePages_Free:     39
Node 1 HugePages_Surp:      0


vnf vhost-config
-------------------
virsh edit vfp-vmx1
...
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
...
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='16' threads='1'/>
    <numa>
      <cell id='0' cpus='0' memory='16777216' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
...
    <interface type='vhostuser'>
      <mac address='02:06:0a:00:00:01'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user-vmx1' mode='client'/>
      <model type='virtio'/>
      <driver name='vhost' rx_queue_size='1024' tx_queue_size='1024'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='02:06:0a:00:00:02'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user-vmx2' mode='client'/>
      <model type='virtio'/>
      <driver name='vhost' rx_queue_size='1024' tx_queue_size='1024'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </interface>


ovs config
-------------

export PATH=$PATH:/usr/share/openvswitch/scripts
export DB_SOCK=/var/run/openvswitch/db.sock
db=/var/lib/openvswitch/conf.db
schema="/usr/share/openvswitch/vswitch.ovsschema"

ovsdb-tool create $db $schema

sudo ovs-vsctl set Open_vSwitch . other_config:vhost-iommu-support=true

sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true

sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x100000
sudo ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=2

bridge="br-dpdk"
ovs-vsctl del-br $bridge
ovs-vsctl add-br $bridge -- set bridge $bridge datapath_type=netdev
ovs-vsctl add-port $bridge vhost-user-vmx1 -- set Interface vhost-user-vmx1 type=dpdkvhostuser
ovs-vsctl add-port $bridge vhost-user-vmx2 -- set Interface vhost-user-vmx2 type=dpdkvhostuser

ovs-vsctl set interface vhost-user-vmx1 options:n_rxq=2 \
    other_config:pmd-rxq-affinity="0:8,1:28"

ovs-vsctl set interface vhost-user-vmx2 options:n_rxq=2 \
    other_config:pmd-rxq-affinity="0:9,1:29"

ovs-vsctl set Interface vhost-user-vmx1 options:n_txq_desc=2
ovs-vsctl set Interface vhost-user-vmx2 options:n_txq_desc=2

some show cmd
--------------------
root@bn831x1a-node6-lab:~# sudo ovs-vsctl show
fff633a5-c11e-4b78-b1cb-dbd1ace240a1
    Bridge br-dpdk
        Port br-dpdk
            Interface br-dpdk
                type: internal
        Port "vhost-user-vmx2"
            trunks: [10, 11, 12]
            Interface "vhost-user-vmx2"
                type: dpdkvhostuser
                options: {n_rxq="2", n_txq_desc="2"}
        Port "vhost-user-vmx1"
            trunks: [10, 11, 12]
            Interface "vhost-user-vmx1"
                type: dpdkvhostuser
                options: {n_rxq="2", n_txq_desc="2"}
    ovs_version: "2.11.0"


port stats
------------
root@bn831x1a-node6-lab:~# sudo ovs-ofctl dump-ports br-dpdk
OFPST_PORT reply (xid=0x2): 3 ports
  port  "vhost-user-vmx2": rx pkts=1960035, bytes=678163420, drop=0, errs=0, frame=?, over=?, crc=?
           tx pkts=1960963, bytes=678484021, drop=0, errs=?, coll=?
  port  "vhost-user-vmx1": rx pkts=1960964, bytes=678484111, drop=0, errs=0, frame=?, over=?, crc=?
           tx pkts=1960035, bytes=678163420, drop=0, errs=?, coll=?
  port LOCAL: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=0, bytes=0, drop=5, errs=0, coll=0

interface-stats
--------------------
root@bn831x1a-node6-lab:~# ovs-vsctl list interface vhost-user-vmx1
_uuid               : 9e7d2537-b200-486f-bda1-d04d371c9765
..
mac                 : []
mac_in_use          : "00:00:00:00:00:00"
mtu                 : 1500
mtu_request         : []
name                : "vhost-user-vmx1"
..
options             : {n_rxq="2", n_txq_desc="2"}
other_config        : {pmd-rxq-affinity="0:8,1:28"}
statistics          : {"rx_1024_to_1522_packets"=0, "rx_128_to_255_packets"=1, "rx_1523_to_max_packets"=0, "rx_1_to_64_packets"=4, "rx_256_to_511_packets"=1960929, "rx_512_to_1023_packets"=0, "rx_65_to_127_packets"=30, rx_bytes=678484111, rx_dropped=0, rx_errors=0, rx_packets=1960964, tx_bytes=678163420, tx_dropped=0, tx_packets=1960035}
status              : {features="0x0000000050008000", mode=server, num_of_vrings="2", numa="0", socket="/var/run/openvswitch/vhost-user-vmx1", status=connected, "vring_0_size"="1024", "vring_1_size"="1024"}
type                : dpdkvhostuser


PMD-stats look good to me
-------------------------------
root@bn831x1a-node6-lab:~# sudo ovs-appctl dpif-netdev/pmd-stats-show
pmd thread numa_id 0 core_id 8:
  packets received: 1960964
  packet recirculations: 0
  avg. datapath passes per packet: 1.00
  emc hits: 1960909
  smc hits: 0
  megaflow hits: 50
  avg. subtable lookups per megaflow hit: 1.00
  miss with success upcall: 5
  miss with failed upcall: 0
  avg. packets per output batch: 1.03
  idle cycles: 2112185253324 (99.75%)
  processing cycles: 5244829908 (0.25%)
  avg cycles per packet: 1079790.39 (2117430083232/1960964)
  avg processing cycles per packet: 2674.62 (5244829908/1960964)
pmd thread numa_id 0 core_id 9:
  packets received: 1960035
  packet recirculations: 0
  avg. datapath passes per packet: 1.00
  emc hits: 1959755
  smc hits: 0
  megaflow hits: 276
  avg. subtable lookups per megaflow hit: 1.92
  miss with success upcall: 4
  miss with failed upcall: 0
  avg. packets per output batch: 1.08
  idle cycles: 2112413001990 (99.76%)
  processing cycles: 4990062087 (0.24%)
  avg cycles per packet: 1080288.39 (2117403064077/1960035)
  avg processing cycles per packet: 2545.90 (4990062087/1960035)
pmd thread numa_id 0 core_id 28:
  packets received: 0
  packet recirculations: 0
  avg. datapath passes per packet: 0.00
  emc hits: 0
  smc hits: 0
  megaflow hits: 0
  avg. subtable lookups per megaflow hit: 0.00
  miss with success upcall: 0
  miss with failed upcall: 0
  avg. packets per output batch: 0.00
pmd thread numa_id 0 core_id 29:
  packets received: 0
  packet recirculations: 0
  avg. datapath passes per packet: 0.00
  emc hits: 0
  smc hits: 0
  megaflow hits: 0
  avg. subtable lookups per megaflow hit: 0.00
  miss with success upcall: 0
  miss with failed upcall: 0
  avg. packets per output batch: 0.00
main thread:
  packets received: 0
  packet recirculations: 0
  avg. datapath passes per packet: 0.00
  emc hits: 0
  smc hits: 0
  megaflow hits: 0
  avg. subtable lookups per megaflow hit: 0.00
  miss with success upcall: 0
  miss with failed upcall: 0
  avg. packets per output batch: 0.00


PMD-cpu utilization looks rather low
-----------------------------------------
root@bn831x1a-node6-lab:~# ovs-appctl dpif-netdev/pmd-rxq-show
pmd thread numa_id 0 core_id 8:
  isolated : true
  port: vhost-user-vmx1   queue-id:  0  pmd usage:  2 %
pmd thread numa_id 0 core_id 9:
  isolated : true
  port: vhost-user-vmx2   queue-id:  0  pmd usage:  2 %
pmd thread numa_id 0 core_id 28:
  isolated : false
pmd thread numa_id 0 core_id 29:
  isolated : false


As said initially, OVS claims to have egressed 1.96mio packets on its interface vhost-user-vmx1, however the vnf attached to this interface is only seeing 1.58mio packets, so somewhere we are loosing 400.000 packets.
As the packet-rate is just about 200kpps, both the OVS and the VNF should have been able to process packets at given rate.
How can I please investigate further where the packets are getting lost?
What further tuning

thanks

christian

                 reply	other threads:[~2019-10-14  8:12 UTC|newest]

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=trinity-dcc4c5dc-3716-4261-a573-2de7a521285d-1571040731725@3c-app-gmx-bap74 \
    --to=chr.graf@gmx.de \
    --cc=users@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).