[dpdk-dev] outw() in virtio_ring_doorbell() in DPDK+virtio consume 40% of the CPU in oprofile

[James Yu <ypyu2011@gmail.com>]

I am using Spirent to send a 2Gbps traffic to a 10G port that are looped
back by l2fwd+DPDK+virtio in a CentOS 32-bit and receive on the other port
only at 700 Mbps.   The CentOS 32-bit is on a Fedora 18 KVM host. The
virtual interfaces are configured as virtio port type, not e1000. vhost-net
was automatically used in qemu-kvm when virtio ports are used in the guest.

The questions are
A. Why it can only reach 2Gbps
B. Why outw() is using 40% of the entire measurement when it only try to
write 2 bytes to the IO port using assembly outw command ? Is it a blocking
call ? or it wastes time is mapping from the IO address of the guest to the
physical address of the IO port on the host ?
C. any way to improve it ?
D. vmxnet PMD codes are using memory mapped IO address, not port IO
address. Will it be faster to use memory mapped IO address ?

Any pointers or feedback will help.
Thanks

James

---
While the traffic is on, I run a oprofile and oreport using the following
scripts on a seperate xterm window.
1. ./oprofile_start.sh
2. wait for 10 seconds
3. ./oprofile_stop.sh
::::::::::::::
oprofile_start.sh
::::::::::::::
#!/bin/bash
opcontrol --reset
opcontrol --deinit
modprobe oprofile timer=1
opcontrol --no-vmlinux --separate=cpu,thread --callgraph=10
--separate=kernel
opcontrol --session-dir=/root
opcontrol --start

::::::::::::::
oprofile_stop.sh
::::::::::::::
opcontrol --dump
opcontrol --stop
opcontrol --shutdown
opreport --session-dir=/root --details --merge tgid --symbols /root/dpdk/
dpdk-1.3.1r2/examples/l2fwd/build/l2fwd

Profiling through timer interrupt
vma      samples  %        image name               symbol name
00000d36 5445     40.1105  librte_pmd_virtio.so     outw
  00000d54 5442     99.9449
  00000d55 3         0.0551
00003032 3513     25.8785  librte_pmd_virtio.so     virtio_recv_buf

---
static void outw_jyu1(unsigned short int value, unsigned short int __port){
  __asm__ __volatile__ ("outw %w0,%w1": :"a" (value), "Nd" (__port));
}
---
This link
http://www.cs.nthu.edu.tw/~ychung/slides/Virtualization/VM-Lecture-2-3-IO%20Virtualization.pptx<http://www.cs.nthu.edu.tw/%7Eychung/slides/Virtualization/VM-Lecture-2-3-IO%20Virtualization.pptx>(page
17 – 22) described about the how IO ports can be accessed.