* [dpdk-dev] Vhost user no connection vm2vm
@ 2015-05-15 10:15 Maciej Grochowski
2015-05-17 14:41 ` Ouyang, Changchun
2015-05-21 9:12 ` [dpdk-dev] FW: " Gaohaifeng (A)
0 siblings, 2 replies; 13+ messages in thread
From: Maciej Grochowski @ 2015-05-15 10:15 UTC (permalink / raw)
To: dev
Hello, I have strange issue with example/vhost app.
I had compiled DPDK to run a vhost example app with followed flags
CONFIG_RTE_LIBRTE_VHOST=y
CONFIG_RTE_LIBRTE_VHOST_USER=y
CONFIG_RTE_LIBRTE_VHOST_DEBUG=n
then I run vhost app based on documentation:
./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge --socket-mem 3712
-- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9
-I use this strange --socket-mem 3712 because of physical limit of memoryon
device
-with this vhost user I run two KVM machines with followed parameters
kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 -cpu host
-smp 2
-hda /home/ubuntu/qemu/debian_squeeze2_amd64.qcow2 -m 1024 -mem-path
/mnt/huge -mem-prealloc
-chardev socket,id=char1,path=/home/ubuntu/dpdk/examples/vhost/usvhost
-netdev type=vhost-user,id=hostnet1,chardev=char1
-device virtio-net
pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off
-chardev socket,id=char2,path=/home/ubuntu/dpdk/examples/vhost/usvhost
-netdev type=vhost-user,id=hostnet2,chardev=char2
-device
virtio-net-pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off
After running KVM virtio correctly starting (below logs from vhost app)
...
VHOST_CONFIG: mapped region 0 fd:31 to 0x2aaabae00000 sz:0xa0000 off:0x0
VHOST_CONFIG: mapped region 1 fd:37 to 0x2aaabb000000 sz:0x10000000
off:0xc0000
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:0 file:38
VHOST_CONFIG: virtio isn't ready for processing.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:1 file:39
VHOST_CONFIG: virtio is now ready for processing.
VHOST_DATA: (1) Device has been added to data core 2
So everything looking good.
Maybe it is something trivial but using options: --vm2vm 1 (or) 2 --stats 9
it seems that I didn't have connection between VM2VM communication. I set
manually IP for eth0 and eth1:
on 1 VM
ifconfig eth0 192.168.0.100 netmask 255.255.255.0 up
ifconfig eth1 192.168.1.101 netmask 255.255.255.0 up
on 2 VM
ifconfig eth0 192.168.1.200 netmask 255.255.255.0 up
ifconfig eth1 192.168.0.202 netmask 255.255.255.0 up
I notice that in vhostapp are one directional rx/tx queue
so I tryied to ping between VM1 to VM2 using both interfaces
ping -I eth0 192.168.1.200
ping -I eth1 192.168.1.200
ping -I eth0 192.168.0.202
ping -I eth1 192.168.0.202
on VM2 using tcpdump on both interfaces I didn't see any ICMP requests or
traffic
And I cant ping between any IP/interfaces, moreover stats show me that:
Device statistics ====================================
Statistics for device 0 ------------------------------
TX total: 0
TX dropped: 0
TX successful: 0
RX total: 0
RX dropped: 0
RX successful: 0
Statistics for device 1 ------------------------------
TX total: 0
TX dropped: 0
TX successful: 0
RX total: 0
RX dropped: 0
RX successful: 0
Statistics for device 2 ------------------------------
TX total: 0
TX dropped: 0
TX successful: 0
RX total: 0
RX dropped: 0
RX successful: 0
Statistics for device 3 ------------------------------
TX total: 0
TX dropped: 0
TX successful: 0
RX total: 0
RX dropped: 0
RX successful: 0
======================================================
So it seems like any packet didn't leave my VM.
also arp table is empty on each VM.
ifconfig -a show that no packet come across eth0, eth1 which I used with
ping, but everything come across local loopback
eth0 Link encap:Ethernet HWaddr 52:54:00:12:34:56
inet addr:192.168.0.200 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth1 Link encap:Ethernet HWaddr 52:54:00:12:34:57
inet addr:192.168.1.202 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe12:3457/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:53 errors:0 dropped:0 overruns:0 frame:0
TX packets:53 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5936 (5.7 KiB) TX bytes:5936 (5.7 KiB)
Do You have any idea what can be wrong with configuration between VM2VM?
Any help would be appreciated.
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: [dpdk-dev] Vhost user no connection vm2vm 2015-05-15 10:15 [dpdk-dev] Vhost user no connection vm2vm Maciej Grochowski @ 2015-05-17 14:41 ` Ouyang, Changchun 2015-05-21 9:12 ` [dpdk-dev] FW: " Gaohaifeng (A) 1 sibling, 0 replies; 13+ messages in thread From: Ouyang, Changchun @ 2015-05-17 14:41 UTC (permalink / raw) To: Maciej Grochowski, dev > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Maciej Grochowski > Sent: Friday, May 15, 2015 6:15 PM > To: dev@dpdk.org > Subject: [dpdk-dev] Vhost user no connection vm2vm > > Hello, I have strange issue with example/vhost app. > > I had compiled DPDK to run a vhost example app with followed flags > > CONFIG_RTE_LIBRTE_VHOST=y > CONFIG_RTE_LIBRTE_VHOST_USER=y > CONFIG_RTE_LIBRTE_VHOST_DEBUG=n > > then I run vhost app based on documentation: > > ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge --socket-mem 3712 > -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9 > > -I use this strange --socket-mem 3712 because of physical limit of memoryon > device -with this vhost user I run two KVM machines with followed > parameters > > kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 -cpu > host -smp 2 -hda /home/ubuntu/qemu/debian_squeeze2_amd64.qcow2 -m > 1024 -mem-path /mnt/huge -mem-prealloc -chardev > socket,id=char1,path=/home/ubuntu/dpdk/examples/vhost/usvhost > -netdev type=vhost-user,id=hostnet1,chardev=char1 > -device virtio-net > pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6= > off,guest_ecn=off > -chardev > socket,id=char2,path=/home/ubuntu/dpdk/examples/vhost/usvhost > -netdev type=vhost-user,id=hostnet2,chardev=char2 > -device > virtio-net- > pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6= > off,guest_ecn=off > > After running KVM virtio correctly starting (below logs from vhost app) ... > VHOST_CONFIG: mapped region 0 fd:31 to 0x2aaabae00000 sz:0xa0000 > off:0x0 > VHOST_CONFIG: mapped region 1 fd:37 to 0x2aaabb000000 sz:0x10000000 > off:0xc0000 > VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > VHOST_CONFIG: vring kick idx:0 file:38 > VHOST_CONFIG: virtio isn't ready for processing. > VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > VHOST_CONFIG: vring kick idx:1 file:39 > VHOST_CONFIG: virtio is now ready for processing. > VHOST_DATA: (1) Device has been added to data core 2 > > So everything looking good. > > Maybe it is something trivial but using options: --vm2vm 1 (or) 2 --stats 9 it > seems that I didn't have connection between VM2VM communication. I set > manually IP for eth0 and eth1: > > on 1 VM > ifconfig eth0 192.168.0.100 netmask 255.255.255.0 up ifconfig eth1 > 192.168.1.101 netmask 255.255.255.0 up > > on 2 VM > ifconfig eth0 192.168.1.200 netmask 255.255.255.0 up ifconfig eth1 > 192.168.0.202 netmask 255.255.255.0 up > > I notice that in vhostapp are one directional rx/tx queue so I tryied to ping > between VM1 to VM2 using both interfaces ping -I eth0 192.168.1.200 ping -I > eth1 192.168.1.200 ping -I eth0 192.168.0.202 ping -I eth1 192.168.0.202 > > on VM2 using tcpdump on both interfaces I didn't see any ICMP requests or > traffic > > And I cant ping between any IP/interfaces, moreover stats show me that: > > Device statistics ==================================== > Statistics for device 0 ------------------------------ > TX total: 0 > TX dropped: 0 > TX successful: 0 > RX total: 0 > RX dropped: 0 > RX successful: 0 > Statistics for device 1 ------------------------------ > TX total: 0 > TX dropped: 0 > TX successful: 0 > RX total: 0 > RX dropped: 0 > RX successful: 0 > Statistics for device 2 ------------------------------ > TX total: 0 > TX dropped: 0 > TX successful: 0 > RX total: 0 > RX dropped: 0 > RX successful: 0 > Statistics for device 3 ------------------------------ > TX total: 0 > TX dropped: 0 > TX successful: 0 > RX total: 0 > RX dropped: 0 > RX successful: 0 > ====================================================== > > So it seems like any packet didn't leave my VM. > also arp table is empty on each VM. > I think you can try with this command line to config your ports, and make sure the arp table is NOT empty: ip addr add 192.168.1.2/24 dev eth0 ip neigh add 192.168.1.1 lladdr 00:01:04:00:01:00 dev eth0 ip link set dev eth0 up of course you need modify the ip, mac, and port according to your own platform. Thanks Changchun ^ permalink raw reply [flat|nested] 13+ messages in thread
* [dpdk-dev] FW: Vhost user no connection vm2vm 2015-05-15 10:15 [dpdk-dev] Vhost user no connection vm2vm Maciej Grochowski 2015-05-17 14:41 ` Ouyang, Changchun @ 2015-05-21 9:12 ` Gaohaifeng (A) 2015-05-22 8:05 ` Maciej Grochowski 1 sibling, 1 reply; 13+ messages in thread From: Gaohaifeng (A) @ 2015-05-21 9:12 UTC (permalink / raw) To: Maciej Grochowski; +Cc: dev Hi Maciej Did you solve your problem? I meet this problem as your case. And I found avail_idx(in rte_vhost_dequeue_burst function) is always zero although I do send packets in VM. Thanks. > Hello, I have strange issue with example/vhost app. > > I had compiled DPDK to run a vhost example app with followed flags > > CONFIG_RTE_LIBRTE_VHOST=y > CONFIG_RTE_LIBRTE_VHOST_USER=y > CONFIG_RTE_LIBRTE_VHOST_DEBUG=n > > then I run vhost app based on documentation: > > ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge --socket-mem > 3712 > -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9 > > -I use this strange --socket-mem 3712 because of physical limit of > memoryon device -with this vhost user I run two KVM machines with > followed parameters > > kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 -cpu > host -smp 2 -hda /home/ubuntu/qemu/debian_squeeze2_amd64.qcow2 -m > 1024 -mem-path /mnt/huge -mem-prealloc -chardev > socket,id=char1,path=/home/ubuntu/dpdk/examples/vhost/usvhost > -netdev type=vhost-user,id=hostnet1,chardev=char1 > -device virtio-net > pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6 > = > off,guest_ecn=off > -chardev > socket,id=char2,path=/home/ubuntu/dpdk/examples/vhost/usvhost > -netdev type=vhost-user,id=hostnet2,chardev=char2 > -device > virtio-net- > pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6 > = > off,guest_ecn=off > > After running KVM virtio correctly starting (below logs from vhost app) ... > VHOST_CONFIG: mapped region 0 fd:31 to 0x2aaabae00000 sz:0xa0000 > off:0x0 > VHOST_CONFIG: mapped region 1 fd:37 to 0x2aaabb000000 sz:0x10000000 > off:0xc0000 > VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > VHOST_CONFIG: vring kick idx:0 file:38 > VHOST_CONFIG: virtio isn't ready for processing. > VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > VHOST_CONFIG: vring kick idx:1 file:39 > VHOST_CONFIG: virtio is now ready for processing. > VHOST_DATA: (1) Device has been added to data core 2 > > So everything looking good. > > Maybe it is something trivial but using options: --vm2vm 1 (or) 2 > --stats 9 it seems that I didn't have connection between VM2VM > communication. I set manually IP for eth0 and eth1: > > on 1 VM > ifconfig eth0 192.168.0.100 netmask 255.255.255.0 up ifconfig eth1 > 192.168.1.101 netmask 255.255.255.0 up > > on 2 VM > ifconfig eth0 192.168.1.200 netmask 255.255.255.0 up ifconfig eth1 > 192.168.0.202 netmask 255.255.255.0 up > > I notice that in vhostapp are one directional rx/tx queue so I tryied > to ping between VM1 to VM2 using both interfaces ping -I eth0 > 192.168.1.200 ping -I > eth1 192.168.1.200 ping -I eth0 192.168.0.202 ping -I eth1 > 192.168.0.202 > > on VM2 using tcpdump on both interfaces I didn't see any ICMP requests > or traffic > > And I cant ping between any IP/interfaces, moreover stats show me that: > > Device statistics ==================================== > Statistics for device 0 ------------------------------ > TX total: 0 > TX dropped: 0 > TX successful: 0 > RX total: 0 > RX dropped: 0 > RX successful: 0 > Statistics for device 1 ------------------------------ > TX total: 0 > TX dropped: 0 > TX successful: 0 > RX total: 0 > RX dropped: 0 > RX successful: 0 > Statistics for device 2 ------------------------------ > TX total: 0 > TX dropped: 0 > TX successful: 0 > RX total: 0 > RX dropped: 0 > RX successful: 0 > Statistics for device 3 ------------------------------ > TX total: 0 > TX dropped: 0 > TX successful: 0 > RX total: 0 > RX dropped: 0 > RX successful: 0 > ====================================================== > > So it seems like any packet didn't leave my VM. > also arp table is empty on each VM. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [dpdk-dev] FW: Vhost user no connection vm2vm 2015-05-21 9:12 ` [dpdk-dev] FW: " Gaohaifeng (A) @ 2015-05-22 8:05 ` Maciej Grochowski 2015-05-22 8:26 ` Ouyang, Changchun ` (2 more replies) 0 siblings, 3 replies; 13+ messages in thread From: Maciej Grochowski @ 2015-05-22 8:05 UTC (permalink / raw) To: Gaohaifeng (A), dev Unfortunately not, I have the same issue in rte_vhost_dequeue_burst function. What kernel version are You using on host/guest? In my case on host I had 3.13.0 and on guests old 3.2 debian. I just looked deeper into virtio back-end (vhost) but at first glace it seems like nothing coming from virtio. What I'm going to do today is to compile newest kernel for vhost and guest and debug where packet flow stuck, I will report the result On Thu, May 21, 2015 at 11:12 AM, Gaohaifeng (A) <gaohaifeng.gao@huawei.com> wrote: > Hi Maciej > Did you solve your problem? I meet this problem as your case. And > I found avail_idx(in rte_vhost_dequeue_burst function) is always zero > although I do send packets in VM. > > Thanks. > > > > Hello, I have strange issue with example/vhost app. > > > > I had compiled DPDK to run a vhost example app with followed flags > > > > CONFIG_RTE_LIBRTE_VHOST=y > > CONFIG_RTE_LIBRTE_VHOST_USER=y > > CONFIG_RTE_LIBRTE_VHOST_DEBUG=n > > > > then I run vhost app based on documentation: > > > > ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge --socket-mem > > 3712 > > -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9 > > > > -I use this strange --socket-mem 3712 because of physical limit of > > memoryon device -with this vhost user I run two KVM machines with > > followed parameters > > > > kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 -cpu > > host -smp 2 -hda /home/ubuntu/qemu/debian_squeeze2_amd64.qcow2 -m > > 1024 -mem-path /mnt/huge -mem-prealloc -chardev > > socket,id=char1,path=/home/ubuntu/dpdk/examples/vhost/usvhost > > -netdev type=vhost-user,id=hostnet1,chardev=char1 > > -device virtio-net > > pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6 > > = > > off,guest_ecn=off > > -chardev > > socket,id=char2,path=/home/ubuntu/dpdk/examples/vhost/usvhost > > -netdev type=vhost-user,id=hostnet2,chardev=char2 > > -device > > virtio-net- > > pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6 > > = > > off,guest_ecn=off > > > > After running KVM virtio correctly starting (below logs from vhost app) > ... > > VHOST_CONFIG: mapped region 0 fd:31 to 0x2aaabae00000 sz:0xa0000 > > off:0x0 > > VHOST_CONFIG: mapped region 1 fd:37 to 0x2aaabb000000 sz:0x10000000 > > off:0xc0000 > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > > VHOST_CONFIG: vring kick idx:0 file:38 > > VHOST_CONFIG: virtio isn't ready for processing. > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > > VHOST_CONFIG: vring kick idx:1 file:39 > > VHOST_CONFIG: virtio is now ready for processing. > > VHOST_DATA: (1) Device has been added to data core 2 > > > > So everything looking good. > > > > Maybe it is something trivial but using options: --vm2vm 1 (or) 2 > > --stats 9 it seems that I didn't have connection between VM2VM > > communication. I set manually IP for eth0 and eth1: > > > > on 1 VM > > ifconfig eth0 192.168.0.100 netmask 255.255.255.0 up ifconfig eth1 > > 192.168.1.101 netmask 255.255.255.0 up > > > > on 2 VM > > ifconfig eth0 192.168.1.200 netmask 255.255.255.0 up ifconfig eth1 > > 192.168.0.202 netmask 255.255.255.0 up > > > > I notice that in vhostapp are one directional rx/tx queue so I tryied > > to ping between VM1 to VM2 using both interfaces ping -I eth0 > > 192.168.1.200 ping -I > > eth1 192.168.1.200 ping -I eth0 192.168.0.202 ping -I eth1 > > 192.168.0.202 > > > > on VM2 using tcpdump on both interfaces I didn't see any ICMP requests > > or traffic > > > > And I cant ping between any IP/interfaces, moreover stats show me that: > > > > Device statistics ==================================== > > Statistics for device 0 ------------------------------ > > TX total: 0 > > TX dropped: 0 > > TX successful: 0 > > RX total: 0 > > RX dropped: 0 > > RX successful: 0 > > Statistics for device 1 ------------------------------ > > TX total: 0 > > TX dropped: 0 > > TX successful: 0 > > RX total: 0 > > RX dropped: 0 > > RX successful: 0 > > Statistics for device 2 ------------------------------ > > TX total: 0 > > TX dropped: 0 > > TX successful: 0 > > RX total: 0 > > RX dropped: 0 > > RX successful: 0 > > Statistics for device 3 ------------------------------ > > TX total: 0 > > TX dropped: 0 > > TX successful: 0 > > RX total: 0 > > RX dropped: 0 > > RX successful: 0 > > ====================================================== > > > > So it seems like any packet didn't leave my VM. > > also arp table is empty on each VM. > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [dpdk-dev] FW: Vhost user no connection vm2vm 2015-05-22 8:05 ` Maciej Grochowski @ 2015-05-22 8:26 ` Ouyang, Changchun 2015-05-22 8:54 ` [dpdk-dev] " Gaohaifeng (A) 2015-05-22 9:27 ` [dpdk-dev] FW: " Luke Gorrie 2 siblings, 0 replies; 13+ messages in thread From: Ouyang, Changchun @ 2015-05-22 8:26 UTC (permalink / raw) To: Maciej Grochowski, Gaohaifeng (A), dev Hi, > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Maciej Grochowski > Sent: Friday, May 22, 2015 4:06 PM > To: Gaohaifeng (A); dev@dpdk.org > Subject: Re: [dpdk-dev] FW: Vhost user no connection vm2vm > > Unfortunately not, I have the same issue in rte_vhost_dequeue_burst > function. > > What kernel version are You using on host/guest? In my case on host I had > 3.13.0 and on guests old 3.2 debian. > > I just looked deeper into virtio back-end (vhost) but at first glace it seems > like nothing coming from virtio. > > What I'm going to do today is to compile newest kernel for vhost and guest > and debug where packet flow stuck, I will report the result > > On Thu, May 21, 2015 at 11:12 AM, Gaohaifeng (A) > <gaohaifeng.gao@huawei.com> > wrote: > Do you use some command I suggest before, In case of you miss the previous mail, just copy it again: " I think you can try with this command line to config your ports, and make sure the arp table is NOT empty: ip addr add 192.168.1.2/24 dev eth0 ip neigh add 192.168.1.1 lladdr 00:01:04:00:01:00 dev eth0 ip link set dev eth0 up of course you need modify the ip, mac, and port according to your own platform. " Thanks Changchun ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [dpdk-dev] Vhost user no connection vm2vm 2015-05-22 8:05 ` Maciej Grochowski 2015-05-22 8:26 ` Ouyang, Changchun @ 2015-05-22 8:54 ` Gaohaifeng (A) 2015-05-22 9:28 ` Maciej Grochowski 2015-05-22 9:27 ` [dpdk-dev] FW: " Luke Gorrie 2 siblings, 1 reply; 13+ messages in thread From: Gaohaifeng (A) @ 2015-05-22 8:54 UTC (permalink / raw) To: Maciej Grochowski, changchun.ouyang; +Cc: dev Hi What kernel version are You using on host/guest? >>I use ubuntu 12.04(3.11.0-15-generic) in host. In vm I use ubuntu 12.04 and ubuntu14.04 both, but the result is same. Do you use some command I suggest before, In case of you miss the previous mail, just copy it again: >> I try it but the result is same I use l2fwd in vm to do more test and found that virtio_xmit_pkts is called and avail_idx is increasing in vm, but in host avail_idx(in rte_vhost_dequeue_burst function) is always zero. It seems that the host see the different mem area. Init Logs below: VHOST_CONFIG: (0) Mergeable RX buffers disabled VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE VHOST_CONFIG: mapped region 0 fd:24 to 0x2aaaaac00000 sz:0xa0000 off:0x0 VHOST_CONFIG: REGION: 0 GPA: (nil) QEMU VA: 0x2aaaaac00000 SIZE (655360) VHOST_CONFIG: mapped region 1 fd:26 to 0x2aaaaae00000 sz:0x40000000 off:0xc0000 VHOST_CONFIG: REGION: 1 GPA: 0xc0000 QEMU VA: 0x2aaaaacc0000 SIZE (1072955392) VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR VHOST_CONFIG: (0) mapped address desc: 0x2aaae62f1000 VHOST_CONFIG: (0) mapped address avail: 0x2aaae62f2000 VHOST_CONFIG: (0) mapped address used: 0x2aaae62f3000 VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK VHOST_CONFIG: vring kick idx:0 file:23 VHOST_CONFIG: virtio isn't ready for processing. VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR VHOST_CONFIG: (0) mapped address desc: 0x2aaae62f4000 VHOST_CONFIG: (0) mapped address avail: 0x2aaae62f5000 VHOST_CONFIG: (0) mapped address used: 0x2aaae62f6000 VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK VHOST_CONFIG: vring kick idx:1 file:28 VHOST_CONFIG: virtio is now ready for processing. >Unfortunately not, I have the same issue in rte_vhost_dequeue_burst function. >What kernel version are You using on host/guest? In my case on host I had 3.13.0 and on guests old 3.2 debian. >I just looked deeper into virtio back-end (vhost) but at first glace it seems like nothing coming from virtio. >What I'm going to do today is to compile newest kernel for vhost and guest and debug where packet flow stuck, I will report the result >On Thu, May 21, 2015 at 11:12 AM, Gaohaifeng (A) <gaohaifeng.gao@huawei.com<mailto:gaohaifeng.gao@huawei.com>> wrote: >Hi Maciej >Did you solve your problem? I meet this problem as your case. And I found avail_idx(in rte_vhost_dequeue_burst function) is always zero although I do send packets in VM. >Thanks. > Hello, I have strange issue with example/vhost app. > > I had compiled DPDK to run a vhost example app with followed flags > > CONFIG_RTE_LIBRTE_VHOST=y > CONFIG_RTE_LIBRTE_VHOST_USER=y > CONFIG_RTE_LIBRTE_VHOST_DEBUG=n > > then I run vhost app based on documentation: > > ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge --socket-mem > 3712 > -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9 > > -I use this strange --socket-mem 3712 because of physical limit of > memoryon device -with this vhost user I run two KVM machines with > followed parameters > > kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 -cpu > host -smp 2 -hda /home/ubuntu/qemu/debian_squeeze2_amd64.qcow2 -m > 1024 -mem-path /mnt/huge -mem-prealloc -chardev > socket,id=char1,path=/home/ubuntu/dpdk/examples/vhost/usvhost > -netdev type=vhost-user,id=hostnet1,chardev=char1 > -device virtio-net > pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6 > = > off,guest_ecn=off > -chardev > socket,id=char2,path=/home/ubuntu/dpdk/examples/vhost/usvhost > -netdev type=vhost-user,id=hostnet2,chardev=char2 > -device > virtio-net- > pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6 > = > off,guest_ecn=off > > After running KVM virtio correctly starting (below logs from vhost app) ... > VHOST_CONFIG: mapped region 0 fd:31 to 0x2aaabae00000 sz:0xa0000 > off:0x0 > VHOST_CONFIG: mapped region 1 fd:37 to 0x2aaabb000000 sz:0x10000000 > off:0xc0000 > VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > VHOST_CONFIG: vring kick idx:0 file:38 > VHOST_CONFIG: virtio isn't ready for processing. > VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > VHOST_CONFIG: vring kick idx:1 file:39 > VHOST_CONFIG: virtio is now ready for processing. > VHOST_DATA: (1) Device has been added to data core 2 > > So everything looking good. > > Maybe it is something trivial but using options: --vm2vm 1 (or) 2 > --stats 9 it seems that I didn't have connection between VM2VM > communication. I set manually IP for eth0 and eth1: > > on 1 VM > ifconfig eth0 192.168.0.100 netmask 255.255.255.0 up ifconfig eth1 > 192.168.1.101 netmask 255.255.255.0 up > > on 2 VM > ifconfig eth0 192.168.1.200 netmask 255.255.255.0 up ifconfig eth1 > 192.168.0.202 netmask 255.255.255.0 up > > I notice that in vhostapp are one directional rx/tx queue so I tryied > to ping between VM1 to VM2 using both interfaces ping -I eth0 > 192.168.1.200 ping -I > eth1 192.168.1.200 ping -I eth0 192.168.0.202 ping -I eth1 > 192.168.0.202 > > on VM2 using tcpdump on both interfaces I didn't see any ICMP requests > or traffic > > And I cant ping between any IP/interfaces, moreover stats show me that: > > Device statistics ==================================== > Statistics for device 0 ------------------------------ > TX total: 0 > TX dropped: 0 > TX successful: 0 > RX total: 0 > RX dropped: 0 > RX successful: 0 > Statistics for device 1 ------------------------------ > TX total: 0 > TX dropped: 0 > TX successful: 0 > RX total: 0 > RX dropped: 0 > RX successful: 0 > Statistics for device 2 ------------------------------ > TX total: 0 > TX dropped: 0 > TX successful: 0 > RX total: 0 > RX dropped: 0 > RX successful: 0 > Statistics for device 3 ------------------------------ > TX total: 0 > TX dropped: 0 > TX successful: 0 > RX total: 0 > RX dropped: 0 > RX successful: 0 > ====================================================== > > So it seems like any packet didn't leave my VM. > also arp table is empty on each VM. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [dpdk-dev] Vhost user no connection vm2vm 2015-05-22 8:54 ` [dpdk-dev] " Gaohaifeng (A) @ 2015-05-22 9:28 ` Maciej Grochowski 2015-05-22 9:58 ` Tetsuya Mukawa 0 siblings, 1 reply; 13+ messages in thread From: Maciej Grochowski @ 2015-05-22 9:28 UTC (permalink / raw) To: Gaohaifeng (A); +Cc: dev "Do you use some command I suggest before, In case of you miss the previous mail, just copy it again:" -Yes but it didn't help me ;/ I will describe step by step to esure that configuration is made by right way I started vhost: ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge --socket-mem 3712 -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9 Now I run two vm machines, with followed configuration VM1 __ __ VM2 eth0 > \/ > eth0 eth1 >__/\__> eth1 So I will connect VM1.eth0 with VM2.eth1 and VM1.eth1 with VM2.eth0 Because it is test env and I didn't have other network connection on vhost I will create two networks 192.168.0.x and 192.168.1.x VM1.eth0 with VM2.eth1 will be placed in 192.168.0.x and VM1.eth1 with VM2.eth0 in 192.168.1.x ## I started first VM1 as follow kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm2 -cpu host -smp 1 \ -hda /home/ubuntu/esi_ee/qemu/debian_min_1.qcow2 -m 256 -mem-path /mnt/huge -mem-prealloc \ -chardev socket,id=char3,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ -netdev type=vhost-user,id=hostnet3,chardev=char3 \ -device virtio-net-pci,netdev=hostnet3,id=net3,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off \ -chardev socket,id=char4,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ -netdev type=vhost-user,id=hostnet4,chardev=char4 \ -device virtio-net-pci,netdev=hostnet4,id=net4,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off ## qemu give followed output qemu-system-x86_64: -netdev type=vhost-user,id=hostnet3,chardev=char3: chardev "char3" went up qemu-system-x86_64: -netdev type=vhost-user,id=hostnet4,chardev=char4: chardev "char4" went up ## second VM2 kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 -cpu host -smp 1 \ -hda /home/ubuntu/esi_ee/qemu/debian_min_2.qcow2 -m 256 -mem-path /mnt/huge -mem-prealloc \ -chardev socket,id=char1,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ -netdev type=vhost-user,id=hostnet1,chardev=char1 \ -device virtio-net-pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off \ -chardev socket,id=char2,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ -netdev type=vhost-user,id=hostnet2,chardev=char2 \ -device virtio-net-pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off ## second output qemu-system-x86_64: -netdev type=vhost-user,id=hostnet1,chardev=char1: chardev "char1" went up qemu-system-x86_64: -netdev type=vhost-user,id=hostnet2,chardev=char2: chardev "char2" went up After that I had MAC conflict between VM2 and VM1 VM1: -ifconfig -a eth0 Link encap:Ethernet HWaddr 52:54:00:12:34:56 inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eth1 Link encap:Ethernet HWaddr 52:54:00:12:34:57 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) VM2: -ifconfig -a eth0 Link encap:Ethernet HWaddr 52:54:00:12:34:56 inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eth1 Link encap:Ethernet HWaddr 52:54:00:12:34:57 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) In KNI example I had something similar and also no packet flow and solution was to change MAC addresses #VM1 /etc/init.d/networking stop ifconfig eth0 hw ether 00:01:04:00:01:00 ifconfig eth1 hw ether 00:01:04:00:01:01 /etc/init.d/networking start ifconfig eth0 ifconfig eth1 #VM2 /etc/init.d/networking stop ifconfig eth0 hw ether 00:01:04:00:02:00 ifconfig eth1 hw ether 00:01:04:00:02:01 /etc/init.d/networking start ifconfig eth0 ifconfig eth1 Then I make a configuration that You show: #VM1 ip addr add 192.168.0.100/24 dev eth0 ip addr add 192.168.1.100/24 dev eth1 ip neigh add 192.168.0.200 lladdr 00:01:04:00:02:01 dev eth0 ip link set dev eth0 up ip neigh add 192.168.1.200 lladdr 00:01:04:00:02:00 dev eth1 ip link set dev eth1 up eth0 Link encap:Ethernet HWaddr 00:01:04:00:01:00 inet addr:192.168.0.100 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::201:4ff:fe00:100/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eth1 Link encap:Ethernet HWaddr 00:01:04:00:01:01 inet addr:192.168.1.100 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::201:4ff:fe00:101/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) #VM2 ip addr add 192.168.1.200/24 dev eth0 ip addr add 192.168.0.200/24 dev eth1 ip neigh add 192.168.1.100 lladdr 00:01:04:00:01:01 dev eth0 ip link set dev eth0 up ip neigh add 192.168.0.100 lladdr 00:01:04:00:01:00 dev eth1 ip link set dev eth1 up eth0 Link encap:Ethernet HWaddr 00:01:04:00:02:00 inet addr:192.168.1.200 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::201:4ff:fe00:200/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eth1 Link encap:Ethernet HWaddr 00:01:04:00:02:01 inet addr:192.168.0.200 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::201:4ff:fe00:201/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) After that: VM1.eth0 ip=192.168.0.100-MAC=00:01:04:00:01:00 is connected to VM2.eth1 ip=192.168.0.200-MAC=00:01:04:00:02:01 VM1.eth1 ip=192.168.1.100-MAC=00:01:04:00:01:01 is connected to VM2.eth0 ip=192.168.1.200-MAC=00:01:04:00:02:00 That show my arp tables: #VM1 arp -a ? (192.168.0.200) at 00:01:04:00:02:01 [ether] PERM on eth0 ? (192.168.1.200) at 00:01:04:00:02:00 [ether] PERM on eth1 #VM2 arp -a ? (192.168.0.100) at 00:01:04:00:01:00 [ether] PERM on eth1 ? (192.168.1.100) at 00:01:04:00:01:01 [ether] PERM on eth0 #After this configuration I trying to ping from VM1 VM2 (both IP) root@debian-amd64:~# ping -I eth0 192.168.0.200 PING 192.168.0.200 (192.168.0.200) from 192.168.0.100 eth0: 56(84) bytes of data. ^C --- 192.168.0.200 ping statistics --- 5 packets transmitted, 0 received, 100% packet loss, time 4032ms root@debian-amd64:~# ping 192.168.0.200 PING 192.168.0.200 (192.168.0.200) 56(84) bytes of data. ^C --- 192.168.0.200 ping statistics --- 5 packets transmitted, 0 received, 100% packet loss, time 3999ms root@debian-amd64:~# ping -I eth1 192.168.1.200 PING 192.168.1.200 (192.168.1.200) from 192.168.1.100 eth1: 56(84) bytes of data. ^C --- 192.168.1.200 ping statistics --- 6 packets transmitted, 0 received, 100% packet loss, time 5040ms root@debian-amd64:~# ping 192.168.1.200 PING 192.168.1.200 (192.168.1.200) 56(84) bytes of data. ^C --- 192.168.1.200 ping statistics --- 5 packets transmitted, 0 received, 100% packet loss, time 4030ms #and from VM2 VM1 root@debian-amd64:~# ping 192.168.0.100 PING 192.168.0.100 (192.168.0.100) 56(84) bytes of data. ^C --- 192.168.0.100 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2015ms root@debian-amd64:~# ping -I eth1 192.168.0.100 PING 192.168.0.100 (192.168.0.100) from 192.168.0.200 eth1: 56(84) bytes of data. ^C --- 192.168.0.100 ping statistics --- 5 packets transmitted, 0 received, 100% packet loss, time 4032ms root@debian-amd64:~# ping -I eth0 192.168.1.100 PING 192.168.1.100 (192.168.1.100) from 192.168.1.200 eth0: 56(84) bytes of data. ^C --- 192.168.1.100 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 3024ms root@debian-amd64:~# ping 192.168.1.100 PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data. ^C --- 192.168.1.100 ping statistics --- 8 packets transmitted, 0 received, 100% packet loss, time 7055ms Also stats from vhost: Device statistics ==================================== Statistics for device 0 ------------------------------ TX total: 0 TX dropped: 0 TX successful: 0 RX total: 0 RX dropped: 0 RX successful: 0 Statistics for device 1 ------------------------------ TX total: 0 TX dropped: 0 TX successful: 0 RX total: 0 RX dropped: 0 RX successful: 0 Statistics for device 2 ------------------------------ TX total: 0 TX dropped: 0 TX successful: 0 RX total: 0 RX dropped: 0 RX successful: 0 Statistics for device 3 ------------------------------ TX total: 0 TX dropped: 0 TX successful: 0 RX total: 0 RX dropped: 0 RX successful: 0 ====================================================== My way of thinking was: "In vhost there are several function for L2 that learn MAC-s and links them so why I see no received packets?" Maybe I'm doing some silly bug in network configuration but for me its looking like data flow issue especially that no function on the vhost side did not see any packages. On Fri, May 22, 2015 at 10:54 AM, Gaohaifeng (A) <gaohaifeng.gao@huawei.com> wrote: > Hi > > What kernel version are You using on host/guest? > > >>I use ubuntu 12.04(3.11.0-15-generic) in host. In vm I use ubuntu 12.04 > and ubuntu14.04 both, but the result is same. > > > > Do you use some command I suggest before, In case of you miss the previous > mail, just copy it again: > > >> I try it but the result is same > > > > > > I use l2fwd in vm to do more test and found that virtio_xmit_pkts is > called and avail_idx is increasing in vm, but in host avail_idx(in > rte_vhost_dequeue_burst function) is always zero. It seems that the host > see the different mem area. > > > > Init Logs below: > > VHOST_CONFIG: (0) Mergeable RX buffers disabled > > VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE > > VHOST_CONFIG: mapped region 0 fd:24 to 0x2aaaaac00000 sz:0xa0000 off:0x0 > > VHOST_CONFIG: REGION: 0 GPA: (nil) QEMU VA: 0x2aaaaac00000 SIZE (655360) > > VHOST_CONFIG: mapped region 1 fd:26 to 0x2aaaaae00000 sz:0x40000000 > off:0xc0000 > > VHOST_CONFIG: REGION: 1 GPA: 0xc0000 QEMU VA: 0x2aaaaacc0000 SIZE > (1072955392) > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > > VHOST_CONFIG: (0) mapped address desc: 0x2aaae62f1000 > > VHOST_CONFIG: (0) mapped address avail: 0x2aaae62f2000 > > VHOST_CONFIG: (0) mapped address used: 0x2aaae62f3000 > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > > VHOST_CONFIG: vring kick idx:0 file:23 > > VHOST_CONFIG: virtio isn't ready for processing. > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > > VHOST_CONFIG: (0) mapped address desc: 0x2aaae62f4000 > > VHOST_CONFIG: (0) mapped address avail: 0x2aaae62f5000 > > VHOST_CONFIG: (0) mapped address used: 0x2aaae62f6000 > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > > VHOST_CONFIG: vring kick idx:1 file:28 > > VHOST_CONFIG: virtio is now ready for processing. > > > > > > >Unfortunately not, I have the same issue in rte_vhost_dequeue_burst > function. > > > > >What kernel version are You using on host/guest? In my case on host I > had 3.13.0 and on guests old 3.2 debian. > > > > >I just looked deeper into virtio back-end (vhost) but at first glace it > seems like nothing coming from virtio. > > > > >What I'm going to do today is to compile newest kernel for vhost and > guest and debug where packet flow stuck, I will report the result > > > > >On Thu, May 21, 2015 at 11:12 AM, Gaohaifeng (A) < > gaohaifeng.gao@huawei.com> wrote: > > >Hi Maciej > >Did you solve your problem? I meet this problem as your case. > And I found avail_idx(in rte_vhost_dequeue_burst function) is always zero > although I do send packets in VM. > > >Thanks. > > > > > Hello, I have strange issue with example/vhost app. > > > > I had compiled DPDK to run a vhost example app with followed flags > > > > CONFIG_RTE_LIBRTE_VHOST=y > > CONFIG_RTE_LIBRTE_VHOST_USER=y > > CONFIG_RTE_LIBRTE_VHOST_DEBUG=n > > > > then I run vhost app based on documentation: > > > > ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge --socket-mem > > 3712 > > -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9 > > > > -I use this strange --socket-mem 3712 because of physical limit of > > memoryon device -with this vhost user I run two KVM machines with > > followed parameters > > > > kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 -cpu > > host -smp 2 -hda /home/ubuntu/qemu/debian_squeeze2_amd64.qcow2 -m > > 1024 -mem-path /mnt/huge -mem-prealloc -chardev > > socket,id=char1,path=/home/ubuntu/dpdk/examples/vhost/usvhost > > -netdev type=vhost-user,id=hostnet1,chardev=char1 > > -device virtio-net > > pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6 > > = > > off,guest_ecn=off > > -chardev > > socket,id=char2,path=/home/ubuntu/dpdk/examples/vhost/usvhost > > -netdev type=vhost-user,id=hostnet2,chardev=char2 > > -device > > virtio-net- > > pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6 > > = > > off,guest_ecn=off > > > > After running KVM virtio correctly starting (below logs from vhost app) > ... > > VHOST_CONFIG: mapped region 0 fd:31 to 0x2aaabae00000 sz:0xa0000 > > off:0x0 > > VHOST_CONFIG: mapped region 1 fd:37 to 0x2aaabb000000 sz:0x10000000 > > off:0xc0000 > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > > VHOST_CONFIG: vring kick idx:0 file:38 > > VHOST_CONFIG: virtio isn't ready for processing. > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > > VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > > VHOST_CONFIG: vring kick idx:1 file:39 > > VHOST_CONFIG: virtio is now ready for processing. > > VHOST_DATA: (1) Device has been added to data core 2 > > > > So everything looking good. > > > > Maybe it is something trivial but using options: --vm2vm 1 (or) 2 > > --stats 9 it seems that I didn't have connection between VM2VM > > communication. I set manually IP for eth0 and eth1: > > > > on 1 VM > > ifconfig eth0 192.168.0.100 netmask 255.255.255.0 up ifconfig eth1 > > 192.168.1.101 netmask 255.255.255.0 up > > > > on 2 VM > > ifconfig eth0 192.168.1.200 netmask 255.255.255.0 up ifconfig eth1 > > 192.168.0.202 netmask 255.255.255.0 up > > > > I notice that in vhostapp are one directional rx/tx queue so I tryied > > to ping between VM1 to VM2 using both interfaces ping -I eth0 > > 192.168.1.200 ping -I > > eth1 192.168.1.200 ping -I eth0 192.168.0.202 ping -I eth1 > > 192.168.0.202 > > > > on VM2 using tcpdump on both interfaces I didn't see any ICMP requests > > or traffic > > > > And I cant ping between any IP/interfaces, moreover stats show me that: > > > > Device statistics ==================================== > > Statistics for device 0 ------------------------------ > > TX total: 0 > > TX dropped: 0 > > TX successful: 0 > > RX total: 0 > > RX dropped: 0 > > RX successful: 0 > > Statistics for device 1 ------------------------------ > > TX total: 0 > > TX dropped: 0 > > TX successful: 0 > > RX total: 0 > > RX dropped: 0 > > RX successful: 0 > > Statistics for device 2 ------------------------------ > > TX total: 0 > > TX dropped: 0 > > TX successful: 0 > > RX total: 0 > > RX dropped: 0 > > RX successful: 0 > > Statistics for device 3 ------------------------------ > > TX total: 0 > > TX dropped: 0 > > TX successful: 0 > > RX total: 0 > > RX dropped: 0 > > RX successful: 0 > > ====================================================== > > > > So it seems like any packet didn't leave my VM. > > also arp table is empty on each VM. > > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [dpdk-dev] Vhost user no connection vm2vm 2015-05-22 9:28 ` Maciej Grochowski @ 2015-05-22 9:58 ` Tetsuya Mukawa 2015-05-22 10:04 ` Maciej Grochowski 0 siblings, 1 reply; 13+ messages in thread From: Tetsuya Mukawa @ 2015-05-22 9:58 UTC (permalink / raw) To: Maciej Grochowski; +Cc: dev Hi Maciej, I guess it's nice to make sure guest memory is actually allocated by hugepages. So please check like below. $ cat /sys/kernel/mm/hugepage/xxxxx/free_hugepages $ ./start_qemu.sh $ cat /sys/kernel/mm/hugepage/xxxxx/free_hugepages If qemu guest allocates memory from hugepages, 2nd cat command will indicate it. Thanks, Tetsuya On 2015/05/22 18:28, Maciej Grochowski wrote: > "Do you use some command I suggest before, > In case of you miss the previous mail, just copy it again:" > > -Yes but it didn't help me ;/ > > I will describe step by step to esure that configuration is made by right > way > > > I started vhost: > > ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge --socket-mem 3712 > -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9 > > Now I run two vm machines, with followed configuration > > VM1 __ __ VM2 > eth0 > \/ > eth0 > eth1 >__/\__> eth1 > > So I will connect VM1.eth0 with VM2.eth1 and VM1.eth1 with VM2.eth0 > Because it is test env and I didn't have other network connection on vhost > I will create two networks 192.168.0.x and 192.168.1.x > VM1.eth0 with VM2.eth1 will be placed in 192.168.0.x and VM1.eth1 with > VM2.eth0 in 192.168.1.x > > ## I started first VM1 as follow > kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm2 -cpu host > -smp 1 \ > -hda /home/ubuntu/esi_ee/qemu/debian_min_1.qcow2 -m 256 -mem-path /mnt/huge > -mem-prealloc \ > -chardev > socket,id=char3,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ > -netdev type=vhost-user,id=hostnet3,chardev=char3 \ > -device > virtio-net-pci,netdev=hostnet3,id=net3,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off > \ > -chardev > socket,id=char4,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ > -netdev type=vhost-user,id=hostnet4,chardev=char4 \ > -device > virtio-net-pci,netdev=hostnet4,id=net4,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off > ## qemu give followed output > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet3,chardev=char3: > chardev "char3" went up > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet4,chardev=char4: > chardev "char4" went up > > ## second VM2 > kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 -cpu host > -smp 1 \ > -hda /home/ubuntu/esi_ee/qemu/debian_min_2.qcow2 -m 256 -mem-path /mnt/huge > -mem-prealloc \ > -chardev > socket,id=char1,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ > -netdev type=vhost-user,id=hostnet1,chardev=char1 \ > -device > virtio-net-pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off > \ > -chardev > socket,id=char2,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ > -netdev type=vhost-user,id=hostnet2,chardev=char2 \ > -device > virtio-net-pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off > ## second output > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet1,chardev=char1: > chardev "char1" went up > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet2,chardev=char2: > chardev "char2" went up > > > > After that I had MAC conflict between VM2 and VM1 > > VM1: -ifconfig -a > eth0 Link encap:Ethernet HWaddr 52:54:00:12:34:56 > inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > eth1 Link encap:Ethernet HWaddr 52:54:00:12:34:57 > BROADCAST MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > > VM2: -ifconfig -a > eth0 Link encap:Ethernet HWaddr 52:54:00:12:34:56 > inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > eth1 Link encap:Ethernet HWaddr 52:54:00:12:34:57 > BROADCAST MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > In KNI example I had something similar and also no packet flow and solution > was to change MAC addresses > > #VM1 > /etc/init.d/networking stop > ifconfig eth0 hw ether 00:01:04:00:01:00 > ifconfig eth1 hw ether 00:01:04:00:01:01 > /etc/init.d/networking start > ifconfig eth0 > ifconfig eth1 > > #VM2 > /etc/init.d/networking stop > ifconfig eth0 hw ether 00:01:04:00:02:00 > ifconfig eth1 hw ether 00:01:04:00:02:01 > /etc/init.d/networking start > ifconfig eth0 > ifconfig eth1 > > Then I make a configuration that You show: > > #VM1 > ip addr add 192.168.0.100/24 dev eth0 > ip addr add 192.168.1.100/24 dev eth1 > ip neigh add 192.168.0.200 lladdr 00:01:04:00:02:01 dev eth0 > ip link set dev eth0 up > ip neigh add 192.168.1.200 lladdr 00:01:04:00:02:00 dev eth1 > ip link set dev eth1 up > > eth0 Link encap:Ethernet HWaddr 00:01:04:00:01:00 > inet addr:192.168.0.100 Bcast:0.0.0.0 Mask:255.255.255.0 > inet6 addr: fe80::201:4ff:fe00:100/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > eth1 Link encap:Ethernet HWaddr 00:01:04:00:01:01 > inet addr:192.168.1.100 Bcast:0.0.0.0 Mask:255.255.255.0 > inet6 addr: fe80::201:4ff:fe00:101/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > > #VM2 > ip addr add 192.168.1.200/24 dev eth0 > ip addr add 192.168.0.200/24 dev eth1 > ip neigh add 192.168.1.100 lladdr 00:01:04:00:01:01 dev eth0 > ip link set dev eth0 up > ip neigh add 192.168.0.100 lladdr 00:01:04:00:01:00 dev eth1 > ip link set dev eth1 up > > eth0 Link encap:Ethernet HWaddr 00:01:04:00:02:00 > inet addr:192.168.1.200 Bcast:0.0.0.0 Mask:255.255.255.0 > inet6 addr: fe80::201:4ff:fe00:200/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > eth1 Link encap:Ethernet HWaddr 00:01:04:00:02:01 > inet addr:192.168.0.200 Bcast:0.0.0.0 Mask:255.255.255.0 > inet6 addr: fe80::201:4ff:fe00:201/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > After that: > > VM1.eth0 ip=192.168.0.100-MAC=00:01:04:00:01:00 is connected to VM2.eth1 > ip=192.168.0.200-MAC=00:01:04:00:02:01 > VM1.eth1 ip=192.168.1.100-MAC=00:01:04:00:01:01 is connected to VM2.eth0 > ip=192.168.1.200-MAC=00:01:04:00:02:00 > > That show my arp tables: > > #VM1 > arp -a > ? (192.168.0.200) at 00:01:04:00:02:01 [ether] PERM on eth0 > ? (192.168.1.200) at 00:01:04:00:02:00 [ether] PERM on eth1 > > > #VM2 > arp -a > ? (192.168.0.100) at 00:01:04:00:01:00 [ether] PERM on eth1 > ? (192.168.1.100) at 00:01:04:00:01:01 [ether] PERM on eth0 > > > #After this configuration I trying to ping from VM1 VM2 (both IP) > > root@debian-amd64:~# ping -I eth0 192.168.0.200 > PING 192.168.0.200 (192.168.0.200) from 192.168.0.100 eth0: 56(84) bytes of > data. > ^C > --- 192.168.0.200 ping statistics --- > 5 packets transmitted, 0 received, 100% packet loss, time 4032ms > > root@debian-amd64:~# ping 192.168.0.200 > PING 192.168.0.200 (192.168.0.200) 56(84) bytes of data. > ^C > --- 192.168.0.200 ping statistics --- > 5 packets transmitted, 0 received, 100% packet loss, time 3999ms > > root@debian-amd64:~# ping -I eth1 192.168.1.200 > PING 192.168.1.200 (192.168.1.200) from 192.168.1.100 eth1: 56(84) bytes of > data. > ^C > --- 192.168.1.200 ping statistics --- > 6 packets transmitted, 0 received, 100% packet loss, time 5040ms > > root@debian-amd64:~# ping 192.168.1.200 > PING 192.168.1.200 (192.168.1.200) 56(84) bytes of data. > ^C > --- 192.168.1.200 ping statistics --- > 5 packets transmitted, 0 received, 100% packet loss, time 4030ms > > #and from VM2 VM1 > root@debian-amd64:~# ping 192.168.0.100 > PING 192.168.0.100 (192.168.0.100) 56(84) bytes of data. > ^C > --- 192.168.0.100 ping statistics --- > 3 packets transmitted, 0 received, 100% packet loss, time 2015ms > > root@debian-amd64:~# ping -I eth1 192.168.0.100 > PING 192.168.0.100 (192.168.0.100) from 192.168.0.200 eth1: 56(84) bytes of > data. > ^C > --- 192.168.0.100 ping statistics --- > 5 packets transmitted, 0 received, 100% packet loss, time 4032ms > > root@debian-amd64:~# ping -I eth0 192.168.1.100 > PING 192.168.1.100 (192.168.1.100) from 192.168.1.200 eth0: 56(84) bytes of > data. > ^C > --- 192.168.1.100 ping statistics --- > 4 packets transmitted, 0 received, 100% packet loss, time 3024ms > > root@debian-amd64:~# ping 192.168.1.100 > PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data. > ^C > --- 192.168.1.100 ping statistics --- > 8 packets transmitted, 0 received, 100% packet loss, time 7055ms > > > Also stats from vhost: > Device statistics ==================================== > Statistics for device 0 ------------------------------ > TX total: 0 > TX dropped: 0 > TX successful: 0 > RX total: 0 > RX dropped: 0 > RX successful: 0 > Statistics for device 1 ------------------------------ > TX total: 0 > TX dropped: 0 > TX successful: 0 > RX total: 0 > RX dropped: 0 > RX successful: 0 > Statistics for device 2 ------------------------------ > TX total: 0 > TX dropped: 0 > TX successful: 0 > RX total: 0 > RX dropped: 0 > RX successful: 0 > Statistics for device 3 ------------------------------ > TX total: 0 > TX dropped: 0 > TX successful: 0 > RX total: 0 > RX dropped: 0 > RX successful: 0 > ====================================================== > > My way of thinking was: "In vhost there are several function for L2 that > learn MAC-s and links them so why I see no received packets?" > > Maybe I'm doing some silly bug in network configuration but for me its > looking like data flow issue especially that no function on the vhost side > did not see any packages. > > On Fri, May 22, 2015 at 10:54 AM, Gaohaifeng (A) <gaohaifeng.gao@huawei.com> > wrote: > >> Hi >> >> What kernel version are You using on host/guest? >> >>>> I use ubuntu 12.04(3.11.0-15-generic) in host. In vm I use ubuntu 12.04 >> and ubuntu14.04 both, but the result is same. >> >> >> >> Do you use some command I suggest before, In case of you miss the previous >> mail, just copy it again: >> >>>> I try it but the result is same >> >> >> >> >> I use l2fwd in vm to do more test and found that virtio_xmit_pkts is >> called and avail_idx is increasing in vm, but in host avail_idx(in >> rte_vhost_dequeue_burst function) is always zero. It seems that the host >> see the different mem area. >> >> >> >> Init Logs below: >> >> VHOST_CONFIG: (0) Mergeable RX buffers disabled >> >> VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE >> >> VHOST_CONFIG: mapped region 0 fd:24 to 0x2aaaaac00000 sz:0xa0000 off:0x0 >> >> VHOST_CONFIG: REGION: 0 GPA: (nil) QEMU VA: 0x2aaaaac00000 SIZE (655360) >> >> VHOST_CONFIG: mapped region 1 fd:26 to 0x2aaaaae00000 sz:0x40000000 >> off:0xc0000 >> >> VHOST_CONFIG: REGION: 1 GPA: 0xc0000 QEMU VA: 0x2aaaaacc0000 SIZE >> (1072955392) >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR >> >> VHOST_CONFIG: (0) mapped address desc: 0x2aaae62f1000 >> >> VHOST_CONFIG: (0) mapped address avail: 0x2aaae62f2000 >> >> VHOST_CONFIG: (0) mapped address used: 0x2aaae62f3000 >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK >> >> VHOST_CONFIG: vring kick idx:0 file:23 >> >> VHOST_CONFIG: virtio isn't ready for processing. >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR >> >> VHOST_CONFIG: (0) mapped address desc: 0x2aaae62f4000 >> >> VHOST_CONFIG: (0) mapped address avail: 0x2aaae62f5000 >> >> VHOST_CONFIG: (0) mapped address used: 0x2aaae62f6000 >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK >> >> VHOST_CONFIG: vring kick idx:1 file:28 >> >> VHOST_CONFIG: virtio is now ready for processing. >> >> >> >> >> >>> Unfortunately not, I have the same issue in rte_vhost_dequeue_burst >> function. >> >> >> >>> What kernel version are You using on host/guest? In my case on host I >> had 3.13.0 and on guests old 3.2 debian. >> >> >> >>> I just looked deeper into virtio back-end (vhost) but at first glace it >> seems like nothing coming from virtio. >> >> >> >>> What I'm going to do today is to compile newest kernel for vhost and >> guest and debug where packet flow stuck, I will report the result >> >> >> >>> On Thu, May 21, 2015 at 11:12 AM, Gaohaifeng (A) < >> gaohaifeng.gao@huawei.com> wrote: >> >>> Hi Maciej >> >Did you solve your problem? I meet this problem as your case. >> And I found avail_idx(in rte_vhost_dequeue_burst function) is always zero >> although I do send packets in VM. >> >>> Thanks. >> >> >>> Hello, I have strange issue with example/vhost app. >>> >>> I had compiled DPDK to run a vhost example app with followed flags >>> >>> CONFIG_RTE_LIBRTE_VHOST=y >>> CONFIG_RTE_LIBRTE_VHOST_USER=y >>> CONFIG_RTE_LIBRTE_VHOST_DEBUG=n >>> >>> then I run vhost app based on documentation: >>> >>> ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge --socket-mem >>> 3712 >>> -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9 >>> >>> -I use this strange --socket-mem 3712 because of physical limit of >>> memoryon device -with this vhost user I run two KVM machines with >>> followed parameters >>> >>> kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 -cpu >>> host -smp 2 -hda /home/ubuntu/qemu/debian_squeeze2_amd64.qcow2 -m >>> 1024 -mem-path /mnt/huge -mem-prealloc -chardev >>> socket,id=char1,path=/home/ubuntu/dpdk/examples/vhost/usvhost >>> -netdev type=vhost-user,id=hostnet1,chardev=char1 >>> -device virtio-net >>> pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6 >>> = >>> off,guest_ecn=off >>> -chardev >>> socket,id=char2,path=/home/ubuntu/dpdk/examples/vhost/usvhost >>> -netdev type=vhost-user,id=hostnet2,chardev=char2 >>> -device >>> virtio-net- >>> pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6 >>> = >>> off,guest_ecn=off >>> >>> After running KVM virtio correctly starting (below logs from vhost app) >> ... >>> VHOST_CONFIG: mapped region 0 fd:31 to 0x2aaabae00000 sz:0xa0000 >>> off:0x0 >>> VHOST_CONFIG: mapped region 1 fd:37 to 0x2aaabb000000 sz:0x10000000 >>> off:0xc0000 >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK >>> VHOST_CONFIG: vring kick idx:0 file:38 >>> VHOST_CONFIG: virtio isn't ready for processing. >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK >>> VHOST_CONFIG: vring kick idx:1 file:39 >>> VHOST_CONFIG: virtio is now ready for processing. >>> VHOST_DATA: (1) Device has been added to data core 2 >>> >>> So everything looking good. >>> >>> Maybe it is something trivial but using options: --vm2vm 1 (or) 2 >>> --stats 9 it seems that I didn't have connection between VM2VM >>> communication. I set manually IP for eth0 and eth1: >>> >>> on 1 VM >>> ifconfig eth0 192.168.0.100 netmask 255.255.255.0 up ifconfig eth1 >>> 192.168.1.101 netmask 255.255.255.0 up >>> >>> on 2 VM >>> ifconfig eth0 192.168.1.200 netmask 255.255.255.0 up ifconfig eth1 >>> 192.168.0.202 netmask 255.255.255.0 up >>> >>> I notice that in vhostapp are one directional rx/tx queue so I tryied >>> to ping between VM1 to VM2 using both interfaces ping -I eth0 >>> 192.168.1.200 ping -I >>> eth1 192.168.1.200 ping -I eth0 192.168.0.202 ping -I eth1 >>> 192.168.0.202 >>> >>> on VM2 using tcpdump on both interfaces I didn't see any ICMP requests >>> or traffic >>> >>> And I cant ping between any IP/interfaces, moreover stats show me that: >>> >>> Device statistics ==================================== >>> Statistics for device 0 ------------------------------ >>> TX total: 0 >>> TX dropped: 0 >>> TX successful: 0 >>> RX total: 0 >>> RX dropped: 0 >>> RX successful: 0 >>> Statistics for device 1 ------------------------------ >>> TX total: 0 >>> TX dropped: 0 >>> TX successful: 0 >>> RX total: 0 >>> RX dropped: 0 >>> RX successful: 0 >>> Statistics for device 2 ------------------------------ >>> TX total: 0 >>> TX dropped: 0 >>> TX successful: 0 >>> RX total: 0 >>> RX dropped: 0 >>> RX successful: 0 >>> Statistics for device 3 ------------------------------ >>> TX total: 0 >>> TX dropped: 0 >>> TX successful: 0 >>> RX total: 0 >>> RX dropped: 0 >>> RX successful: 0 >>> ====================================================== >>> >>> So it seems like any packet didn't leave my VM. >>> also arp table is empty on each VM. >> >> ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [dpdk-dev] Vhost user no connection vm2vm 2015-05-22 9:58 ` Tetsuya Mukawa @ 2015-05-22 10:04 ` Maciej Grochowski 2015-05-22 10:54 ` Andriy Berestovskyy 0 siblings, 1 reply; 13+ messages in thread From: Maciej Grochowski @ 2015-05-22 10:04 UTC (permalink / raw) To: Tetsuya Mukawa; +Cc: dev I checked this, results below #before script: root@# cat /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages 494 #after 1 qemu script root@# cat /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages 366 So qemu consume 262144k~262MB that is correct with script On Fri, May 22, 2015 at 11:58 AM, Tetsuya Mukawa <mukawa@igel.co.jp> wrote: > Hi Maciej, > > I guess it's nice to make sure guest memory is actually allocated by > hugepages. > So please check like below. > > $ cat /sys/kernel/mm/hugepage/xxxxx/free_hugepages > $ ./start_qemu.sh > $ cat /sys/kernel/mm/hugepage/xxxxx/free_hugepages > > If qemu guest allocates memory from hugepages, 2nd cat command will > indicate it. > > Thanks, > Tetsuya > > > On 2015/05/22 18:28, Maciej Grochowski wrote: > > "Do you use some command I suggest before, > > In case of you miss the previous mail, just copy it again:" > > > > -Yes but it didn't help me ;/ > > > > I will describe step by step to esure that configuration is made by right > > way > > > > > > I started vhost: > > > > ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge --socket-mem > 3712 > > -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9 > > > > Now I run two vm machines, with followed configuration > > > > VM1 __ __ VM2 > > eth0 > \/ > eth0 > > eth1 >__/\__> eth1 > > > > So I will connect VM1.eth0 with VM2.eth1 and VM1.eth1 with VM2.eth0 > > Because it is test env and I didn't have other network connection on > vhost > > I will create two networks 192.168.0.x and 192.168.1.x > > VM1.eth0 with VM2.eth1 will be placed in 192.168.0.x and VM1.eth1 with > > VM2.eth0 in 192.168.1.x > > > > ## I started first VM1 as follow > > kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm2 -cpu > host > > -smp 1 \ > > -hda /home/ubuntu/esi_ee/qemu/debian_min_1.qcow2 -m 256 -mem-path > /mnt/huge > > -mem-prealloc \ > > -chardev > > socket,id=char3,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ > > -netdev type=vhost-user,id=hostnet3,chardev=char3 \ > > -device > > > virtio-net-pci,netdev=hostnet3,id=net3,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off > > \ > > -chardev > > socket,id=char4,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ > > -netdev type=vhost-user,id=hostnet4,chardev=char4 \ > > -device > > > virtio-net-pci,netdev=hostnet4,id=net4,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off > > ## qemu give followed output > > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet3,chardev=char3: > > chardev "char3" went up > > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet4,chardev=char4: > > chardev "char4" went up > > > > ## second VM2 > > kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 -cpu > host > > -smp 1 \ > > -hda /home/ubuntu/esi_ee/qemu/debian_min_2.qcow2 -m 256 -mem-path > /mnt/huge > > -mem-prealloc \ > > -chardev > > socket,id=char1,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ > > -netdev type=vhost-user,id=hostnet1,chardev=char1 \ > > -device > > > virtio-net-pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off > > \ > > -chardev > > socket,id=char2,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ > > -netdev type=vhost-user,id=hostnet2,chardev=char2 \ > > -device > > > virtio-net-pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off > > ## second output > > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet1,chardev=char1: > > chardev "char1" went up > > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet2,chardev=char2: > > chardev "char2" went up > > > > > > > > After that I had MAC conflict between VM2 and VM1 > > > > VM1: -ifconfig -a > > eth0 Link encap:Ethernet HWaddr 52:54:00:12:34:56 > > inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:1000 > > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > > > eth1 Link encap:Ethernet HWaddr 52:54:00:12:34:57 > > BROADCAST MULTICAST MTU:1500 Metric:1 > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:1000 > > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > > > > > VM2: -ifconfig -a > > eth0 Link encap:Ethernet HWaddr 52:54:00:12:34:56 > > inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:1000 > > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > > > eth1 Link encap:Ethernet HWaddr 52:54:00:12:34:57 > > BROADCAST MULTICAST MTU:1500 Metric:1 > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:1000 > > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > > > In KNI example I had something similar and also no packet flow and > solution > > was to change MAC addresses > > > > #VM1 > > /etc/init.d/networking stop > > ifconfig eth0 hw ether 00:01:04:00:01:00 > > ifconfig eth1 hw ether 00:01:04:00:01:01 > > /etc/init.d/networking start > > ifconfig eth0 > > ifconfig eth1 > > > > #VM2 > > /etc/init.d/networking stop > > ifconfig eth0 hw ether 00:01:04:00:02:00 > > ifconfig eth1 hw ether 00:01:04:00:02:01 > > /etc/init.d/networking start > > ifconfig eth0 > > ifconfig eth1 > > > > Then I make a configuration that You show: > > > > #VM1 > > ip addr add 192.168.0.100/24 dev eth0 > > ip addr add 192.168.1.100/24 dev eth1 > > ip neigh add 192.168.0.200 lladdr 00:01:04:00:02:01 dev eth0 > > ip link set dev eth0 up > > ip neigh add 192.168.1.200 lladdr 00:01:04:00:02:00 dev eth1 > > ip link set dev eth1 up > > > > eth0 Link encap:Ethernet HWaddr 00:01:04:00:01:00 > > inet addr:192.168.0.100 Bcast:0.0.0.0 Mask:255.255.255.0 > > inet6 addr: fe80::201:4ff:fe00:100/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:1000 > > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > > > eth1 Link encap:Ethernet HWaddr 00:01:04:00:01:01 > > inet addr:192.168.1.100 Bcast:0.0.0.0 Mask:255.255.255.0 > > inet6 addr: fe80::201:4ff:fe00:101/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:1000 > > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > > > > > #VM2 > > ip addr add 192.168.1.200/24 dev eth0 > > ip addr add 192.168.0.200/24 dev eth1 > > ip neigh add 192.168.1.100 lladdr 00:01:04:00:01:01 dev eth0 > > ip link set dev eth0 up > > ip neigh add 192.168.0.100 lladdr 00:01:04:00:01:00 dev eth1 > > ip link set dev eth1 up > > > > eth0 Link encap:Ethernet HWaddr 00:01:04:00:02:00 > > inet addr:192.168.1.200 Bcast:0.0.0.0 Mask:255.255.255.0 > > inet6 addr: fe80::201:4ff:fe00:200/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:1000 > > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > > > eth1 Link encap:Ethernet HWaddr 00:01:04:00:02:01 > > inet addr:192.168.0.200 Bcast:0.0.0.0 Mask:255.255.255.0 > > inet6 addr: fe80::201:4ff:fe00:201/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:1000 > > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > > > After that: > > > > VM1.eth0 ip=192.168.0.100-MAC=00:01:04:00:01:00 is connected to VM2.eth1 > > ip=192.168.0.200-MAC=00:01:04:00:02:01 > > VM1.eth1 ip=192.168.1.100-MAC=00:01:04:00:01:01 is connected to VM2.eth0 > > ip=192.168.1.200-MAC=00:01:04:00:02:00 > > > > That show my arp tables: > > > > #VM1 > > arp -a > > ? (192.168.0.200) at 00:01:04:00:02:01 [ether] PERM on eth0 > > ? (192.168.1.200) at 00:01:04:00:02:00 [ether] PERM on eth1 > > > > > > #VM2 > > arp -a > > ? (192.168.0.100) at 00:01:04:00:01:00 [ether] PERM on eth1 > > ? (192.168.1.100) at 00:01:04:00:01:01 [ether] PERM on eth0 > > > > > > #After this configuration I trying to ping from VM1 VM2 (both IP) > > > > root@debian-amd64:~# ping -I eth0 192.168.0.200 > > PING 192.168.0.200 (192.168.0.200) from 192.168.0.100 eth0: 56(84) bytes > of > > data. > > ^C > > --- 192.168.0.200 ping statistics --- > > 5 packets transmitted, 0 received, 100% packet loss, time 4032ms > > > > root@debian-amd64:~# ping 192.168.0.200 > > PING 192.168.0.200 (192.168.0.200) 56(84) bytes of data. > > ^C > > --- 192.168.0.200 ping statistics --- > > 5 packets transmitted, 0 received, 100% packet loss, time 3999ms > > > > root@debian-amd64:~# ping -I eth1 192.168.1.200 > > PING 192.168.1.200 (192.168.1.200) from 192.168.1.100 eth1: 56(84) bytes > of > > data. > > ^C > > --- 192.168.1.200 ping statistics --- > > 6 packets transmitted, 0 received, 100% packet loss, time 5040ms > > > > root@debian-amd64:~# ping 192.168.1.200 > > PING 192.168.1.200 (192.168.1.200) 56(84) bytes of data. > > ^C > > --- 192.168.1.200 ping statistics --- > > 5 packets transmitted, 0 received, 100% packet loss, time 4030ms > > > > #and from VM2 VM1 > > root@debian-amd64:~# ping 192.168.0.100 > > PING 192.168.0.100 (192.168.0.100) 56(84) bytes of data. > > ^C > > --- 192.168.0.100 ping statistics --- > > 3 packets transmitted, 0 received, 100% packet loss, time 2015ms > > > > root@debian-amd64:~# ping -I eth1 192.168.0.100 > > PING 192.168.0.100 (192.168.0.100) from 192.168.0.200 eth1: 56(84) bytes > of > > data. > > ^C > > --- 192.168.0.100 ping statistics --- > > 5 packets transmitted, 0 received, 100% packet loss, time 4032ms > > > > root@debian-amd64:~# ping -I eth0 192.168.1.100 > > PING 192.168.1.100 (192.168.1.100) from 192.168.1.200 eth0: 56(84) bytes > of > > data. > > ^C > > --- 192.168.1.100 ping statistics --- > > 4 packets transmitted, 0 received, 100% packet loss, time 3024ms > > > > root@debian-amd64:~# ping 192.168.1.100 > > PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data. > > ^C > > --- 192.168.1.100 ping statistics --- > > 8 packets transmitted, 0 received, 100% packet loss, time 7055ms > > > > > > Also stats from vhost: > > Device statistics ==================================== > > Statistics for device 0 ------------------------------ > > TX total: 0 > > TX dropped: 0 > > TX successful: 0 > > RX total: 0 > > RX dropped: 0 > > RX successful: 0 > > Statistics for device 1 ------------------------------ > > TX total: 0 > > TX dropped: 0 > > TX successful: 0 > > RX total: 0 > > RX dropped: 0 > > RX successful: 0 > > Statistics for device 2 ------------------------------ > > TX total: 0 > > TX dropped: 0 > > TX successful: 0 > > RX total: 0 > > RX dropped: 0 > > RX successful: 0 > > Statistics for device 3 ------------------------------ > > TX total: 0 > > TX dropped: 0 > > TX successful: 0 > > RX total: 0 > > RX dropped: 0 > > RX successful: 0 > > ====================================================== > > > > My way of thinking was: "In vhost there are several function for L2 that > > learn MAC-s and links them so why I see no received packets?" > > > > Maybe I'm doing some silly bug in network configuration but for me its > > looking like data flow issue especially that no function on the vhost > side > > did not see any packages. > > > > On Fri, May 22, 2015 at 10:54 AM, Gaohaifeng (A) < > gaohaifeng.gao@huawei.com> > > wrote: > > > >> Hi > >> > >> What kernel version are You using on host/guest? > >> > >>>> I use ubuntu 12.04(3.11.0-15-generic) in host. In vm I use ubuntu > 12.04 > >> and ubuntu14.04 both, but the result is same. > >> > >> > >> > >> Do you use some command I suggest before, In case of you miss the > previous > >> mail, just copy it again: > >> > >>>> I try it but the result is same > >> > >> > >> > >> > >> I use l2fwd in vm to do more test and found that virtio_xmit_pkts is > >> called and avail_idx is increasing in vm, but in host avail_idx(in > >> rte_vhost_dequeue_burst function) is always zero. It seems that the host > >> see the different mem area. > >> > >> > >> > >> Init Logs below: > >> > >> VHOST_CONFIG: (0) Mergeable RX buffers disabled > >> > >> VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE > >> > >> VHOST_CONFIG: mapped region 0 fd:24 to 0x2aaaaac00000 sz:0xa0000 off:0x0 > >> > >> VHOST_CONFIG: REGION: 0 GPA: (nil) QEMU VA: 0x2aaaaac00000 SIZE (655360) > >> > >> VHOST_CONFIG: mapped region 1 fd:26 to 0x2aaaaae00000 sz:0x40000000 > >> off:0xc0000 > >> > >> VHOST_CONFIG: REGION: 1 GPA: 0xc0000 QEMU VA: 0x2aaaaacc0000 SIZE > >> (1072955392) > >> > >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > >> > >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > >> > >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > >> > >> VHOST_CONFIG: (0) mapped address desc: 0x2aaae62f1000 > >> > >> VHOST_CONFIG: (0) mapped address avail: 0x2aaae62f2000 > >> > >> VHOST_CONFIG: (0) mapped address used: 0x2aaae62f3000 > >> > >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > >> > >> VHOST_CONFIG: vring kick idx:0 file:23 > >> > >> VHOST_CONFIG: virtio isn't ready for processing. > >> > >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > >> > >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > >> > >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > >> > >> VHOST_CONFIG: (0) mapped address desc: 0x2aaae62f4000 > >> > >> VHOST_CONFIG: (0) mapped address avail: 0x2aaae62f5000 > >> > >> VHOST_CONFIG: (0) mapped address used: 0x2aaae62f6000 > >> > >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > >> > >> VHOST_CONFIG: vring kick idx:1 file:28 > >> > >> VHOST_CONFIG: virtio is now ready for processing. > >> > >> > >> > >> > >> > >>> Unfortunately not, I have the same issue in rte_vhost_dequeue_burst > >> function. > >> > >> > >> > >>> What kernel version are You using on host/guest? In my case on host I > >> had 3.13.0 and on guests old 3.2 debian. > >> > >> > >> > >>> I just looked deeper into virtio back-end (vhost) but at first glace > it > >> seems like nothing coming from virtio. > >> > >> > >> > >>> What I'm going to do today is to compile newest kernel for vhost and > >> guest and debug where packet flow stuck, I will report the result > >> > >> > >> > >>> On Thu, May 21, 2015 at 11:12 AM, Gaohaifeng (A) < > >> gaohaifeng.gao@huawei.com> wrote: > >> > >>> Hi Maciej > >> >Did you solve your problem? I meet this problem as your case. > >> And I found avail_idx(in rte_vhost_dequeue_burst function) is always > zero > >> although I do send packets in VM. > >> > >>> Thanks. > >> > >> > >>> Hello, I have strange issue with example/vhost app. > >>> > >>> I had compiled DPDK to run a vhost example app with followed flags > >>> > >>> CONFIG_RTE_LIBRTE_VHOST=y > >>> CONFIG_RTE_LIBRTE_VHOST_USER=y > >>> CONFIG_RTE_LIBRTE_VHOST_DEBUG=n > >>> > >>> then I run vhost app based on documentation: > >>> > >>> ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge --socket-mem > >>> 3712 > >>> -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9 > >>> > >>> -I use this strange --socket-mem 3712 because of physical limit of > >>> memoryon device -with this vhost user I run two KVM machines with > >>> followed parameters > >>> > >>> kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 -cpu > >>> host -smp 2 -hda /home/ubuntu/qemu/debian_squeeze2_amd64.qcow2 -m > >>> 1024 -mem-path /mnt/huge -mem-prealloc -chardev > >>> socket,id=char1,path=/home/ubuntu/dpdk/examples/vhost/usvhost > >>> -netdev type=vhost-user,id=hostnet1,chardev=char1 > >>> -device virtio-net > >>> pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6 > >>> = > >>> off,guest_ecn=off > >>> -chardev > >>> socket,id=char2,path=/home/ubuntu/dpdk/examples/vhost/usvhost > >>> -netdev type=vhost-user,id=hostnet2,chardev=char2 > >>> -device > >>> virtio-net- > >>> pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6 > >>> = > >>> off,guest_ecn=off > >>> > >>> After running KVM virtio correctly starting (below logs from vhost app) > >> ... > >>> VHOST_CONFIG: mapped region 0 fd:31 to 0x2aaabae00000 sz:0xa0000 > >>> off:0x0 > >>> VHOST_CONFIG: mapped region 1 fd:37 to 0x2aaabb000000 sz:0x10000000 > >>> off:0xc0000 > >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > >>> VHOST_CONFIG: vring kick idx:0 file:38 > >>> VHOST_CONFIG: virtio isn't ready for processing. > >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > >>> VHOST_CONFIG: vring kick idx:1 file:39 > >>> VHOST_CONFIG: virtio is now ready for processing. > >>> VHOST_DATA: (1) Device has been added to data core 2 > >>> > >>> So everything looking good. > >>> > >>> Maybe it is something trivial but using options: --vm2vm 1 (or) 2 > >>> --stats 9 it seems that I didn't have connection between VM2VM > >>> communication. I set manually IP for eth0 and eth1: > >>> > >>> on 1 VM > >>> ifconfig eth0 192.168.0.100 netmask 255.255.255.0 up ifconfig eth1 > >>> 192.168.1.101 netmask 255.255.255.0 up > >>> > >>> on 2 VM > >>> ifconfig eth0 192.168.1.200 netmask 255.255.255.0 up ifconfig eth1 > >>> 192.168.0.202 netmask 255.255.255.0 up > >>> > >>> I notice that in vhostapp are one directional rx/tx queue so I tryied > >>> to ping between VM1 to VM2 using both interfaces ping -I eth0 > >>> 192.168.1.200 ping -I > >>> eth1 192.168.1.200 ping -I eth0 192.168.0.202 ping -I eth1 > >>> 192.168.0.202 > >>> > >>> on VM2 using tcpdump on both interfaces I didn't see any ICMP requests > >>> or traffic > >>> > >>> And I cant ping between any IP/interfaces, moreover stats show me that: > >>> > >>> Device statistics ==================================== > >>> Statistics for device 0 ------------------------------ > >>> TX total: 0 > >>> TX dropped: 0 > >>> TX successful: 0 > >>> RX total: 0 > >>> RX dropped: 0 > >>> RX successful: 0 > >>> Statistics for device 1 ------------------------------ > >>> TX total: 0 > >>> TX dropped: 0 > >>> TX successful: 0 > >>> RX total: 0 > >>> RX dropped: 0 > >>> RX successful: 0 > >>> Statistics for device 2 ------------------------------ > >>> TX total: 0 > >>> TX dropped: 0 > >>> TX successful: 0 > >>> RX total: 0 > >>> RX dropped: 0 > >>> RX successful: 0 > >>> Statistics for device 3 ------------------------------ > >>> TX total: 0 > >>> TX dropped: 0 > >>> TX successful: 0 > >>> RX total: 0 > >>> RX dropped: 0 > >>> RX successful: 0 > >>> ====================================================== > >>> > >>> So it seems like any packet didn't leave my VM. > >>> also arp table is empty on each VM. > >> > >> > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [dpdk-dev] Vhost user no connection vm2vm 2015-05-22 10:04 ` Maciej Grochowski @ 2015-05-22 10:54 ` Andriy Berestovskyy 2015-05-22 17:59 ` Maciej Grochowski 0 siblings, 1 reply; 13+ messages in thread From: Andriy Berestovskyy @ 2015-05-22 10:54 UTC (permalink / raw) To: dev Hi guys, I guess you just miss the qemu flag to map the memory as shared, i.e.: -object memory-backend-file,id=mem,size=1024M,mem-path=/mnt/huge,share=on (the keyword is share=on) Here is an example script: https://github.com/srajag/contrail-vrouter/blob/dpdk-devel/examples/vms/VROUTER1/80.start-vm.sh Regards, Andriy On Fri, May 22, 2015 at 12:04 PM, Maciej Grochowski <maciej.grochowski@codilime.com> wrote: > I checked this, results below > > #before script: > root@# cat /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages > 494 > #after 1 qemu script > root@# cat /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages > 366 > > So qemu consume 262144k~262MB that is correct with script > > On Fri, May 22, 2015 at 11:58 AM, Tetsuya Mukawa <mukawa@igel.co.jp> wrote: > >> Hi Maciej, >> >> I guess it's nice to make sure guest memory is actually allocated by >> hugepages. >> So please check like below. >> >> $ cat /sys/kernel/mm/hugepage/xxxxx/free_hugepages >> $ ./start_qemu.sh >> $ cat /sys/kernel/mm/hugepage/xxxxx/free_hugepages >> >> If qemu guest allocates memory from hugepages, 2nd cat command will >> indicate it. >> >> Thanks, >> Tetsuya >> >> >> On 2015/05/22 18:28, Maciej Grochowski wrote: >> > "Do you use some command I suggest before, >> > In case of you miss the previous mail, just copy it again:" >> > >> > -Yes but it didn't help me ;/ >> > >> > I will describe step by step to esure that configuration is made by right >> > way >> > >> > >> > I started vhost: >> > >> > ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge --socket-mem >> 3712 >> > -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9 >> > >> > Now I run two vm machines, with followed configuration >> > >> > VM1 __ __ VM2 >> > eth0 > \/ > eth0 >> > eth1 >__/\__> eth1 >> > >> > So I will connect VM1.eth0 with VM2.eth1 and VM1.eth1 with VM2.eth0 >> > Because it is test env and I didn't have other network connection on >> vhost >> > I will create two networks 192.168.0.x and 192.168.1.x >> > VM1.eth0 with VM2.eth1 will be placed in 192.168.0.x and VM1.eth1 with >> > VM2.eth0 in 192.168.1.x >> > >> > ## I started first VM1 as follow >> > kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm2 -cpu >> host >> > -smp 1 \ >> > -hda /home/ubuntu/esi_ee/qemu/debian_min_1.qcow2 -m 256 -mem-path >> /mnt/huge >> > -mem-prealloc \ >> > -chardev >> > socket,id=char3,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ >> > -netdev type=vhost-user,id=hostnet3,chardev=char3 \ >> > -device >> > >> virtio-net-pci,netdev=hostnet3,id=net3,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off >> > \ >> > -chardev >> > socket,id=char4,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ >> > -netdev type=vhost-user,id=hostnet4,chardev=char4 \ >> > -device >> > >> virtio-net-pci,netdev=hostnet4,id=net4,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off >> > ## qemu give followed output >> > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet3,chardev=char3: >> > chardev "char3" went up >> > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet4,chardev=char4: >> > chardev "char4" went up >> > >> > ## second VM2 >> > kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 -cpu >> host >> > -smp 1 \ >> > -hda /home/ubuntu/esi_ee/qemu/debian_min_2.qcow2 -m 256 -mem-path >> /mnt/huge >> > -mem-prealloc \ >> > -chardev >> > socket,id=char1,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ >> > -netdev type=vhost-user,id=hostnet1,chardev=char1 \ >> > -device >> > >> virtio-net-pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off >> > \ >> > -chardev >> > socket,id=char2,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ >> > -netdev type=vhost-user,id=hostnet2,chardev=char2 \ >> > -device >> > >> virtio-net-pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off >> > ## second output >> > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet1,chardev=char1: >> > chardev "char1" went up >> > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet2,chardev=char2: >> > chardev "char2" went up >> > >> > >> > >> > After that I had MAC conflict between VM2 and VM1 >> > >> > VM1: -ifconfig -a >> > eth0 Link encap:Ethernet HWaddr 52:54:00:12:34:56 >> > inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> > collisions:0 txqueuelen:1000 >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >> > >> > eth1 Link encap:Ethernet HWaddr 52:54:00:12:34:57 >> > BROADCAST MULTICAST MTU:1500 Metric:1 >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> > collisions:0 txqueuelen:1000 >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >> > >> > >> > VM2: -ifconfig -a >> > eth0 Link encap:Ethernet HWaddr 52:54:00:12:34:56 >> > inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> > collisions:0 txqueuelen:1000 >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >> > >> > eth1 Link encap:Ethernet HWaddr 52:54:00:12:34:57 >> > BROADCAST MULTICAST MTU:1500 Metric:1 >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> > collisions:0 txqueuelen:1000 >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >> > >> > In KNI example I had something similar and also no packet flow and >> solution >> > was to change MAC addresses >> > >> > #VM1 >> > /etc/init.d/networking stop >> > ifconfig eth0 hw ether 00:01:04:00:01:00 >> > ifconfig eth1 hw ether 00:01:04:00:01:01 >> > /etc/init.d/networking start >> > ifconfig eth0 >> > ifconfig eth1 >> > >> > #VM2 >> > /etc/init.d/networking stop >> > ifconfig eth0 hw ether 00:01:04:00:02:00 >> > ifconfig eth1 hw ether 00:01:04:00:02:01 >> > /etc/init.d/networking start >> > ifconfig eth0 >> > ifconfig eth1 >> > >> > Then I make a configuration that You show: >> > >> > #VM1 >> > ip addr add 192.168.0.100/24 dev eth0 >> > ip addr add 192.168.1.100/24 dev eth1 >> > ip neigh add 192.168.0.200 lladdr 00:01:04:00:02:01 dev eth0 >> > ip link set dev eth0 up >> > ip neigh add 192.168.1.200 lladdr 00:01:04:00:02:00 dev eth1 >> > ip link set dev eth1 up >> > >> > eth0 Link encap:Ethernet HWaddr 00:01:04:00:01:00 >> > inet addr:192.168.0.100 Bcast:0.0.0.0 Mask:255.255.255.0 >> > inet6 addr: fe80::201:4ff:fe00:100/64 Scope:Link >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> > collisions:0 txqueuelen:1000 >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >> > >> > eth1 Link encap:Ethernet HWaddr 00:01:04:00:01:01 >> > inet addr:192.168.1.100 Bcast:0.0.0.0 Mask:255.255.255.0 >> > inet6 addr: fe80::201:4ff:fe00:101/64 Scope:Link >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> > collisions:0 txqueuelen:1000 >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >> > >> > >> > #VM2 >> > ip addr add 192.168.1.200/24 dev eth0 >> > ip addr add 192.168.0.200/24 dev eth1 >> > ip neigh add 192.168.1.100 lladdr 00:01:04:00:01:01 dev eth0 >> > ip link set dev eth0 up >> > ip neigh add 192.168.0.100 lladdr 00:01:04:00:01:00 dev eth1 >> > ip link set dev eth1 up >> > >> > eth0 Link encap:Ethernet HWaddr 00:01:04:00:02:00 >> > inet addr:192.168.1.200 Bcast:0.0.0.0 Mask:255.255.255.0 >> > inet6 addr: fe80::201:4ff:fe00:200/64 Scope:Link >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> > collisions:0 txqueuelen:1000 >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >> > >> > eth1 Link encap:Ethernet HWaddr 00:01:04:00:02:01 >> > inet addr:192.168.0.200 Bcast:0.0.0.0 Mask:255.255.255.0 >> > inet6 addr: fe80::201:4ff:fe00:201/64 Scope:Link >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> > collisions:0 txqueuelen:1000 >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >> > >> > After that: >> > >> > VM1.eth0 ip=192.168.0.100-MAC=00:01:04:00:01:00 is connected to VM2.eth1 >> > ip=192.168.0.200-MAC=00:01:04:00:02:01 >> > VM1.eth1 ip=192.168.1.100-MAC=00:01:04:00:01:01 is connected to VM2.eth0 >> > ip=192.168.1.200-MAC=00:01:04:00:02:00 >> > >> > That show my arp tables: >> > >> > #VM1 >> > arp -a >> > ? (192.168.0.200) at 00:01:04:00:02:01 [ether] PERM on eth0 >> > ? (192.168.1.200) at 00:01:04:00:02:00 [ether] PERM on eth1 >> > >> > >> > #VM2 >> > arp -a >> > ? (192.168.0.100) at 00:01:04:00:01:00 [ether] PERM on eth1 >> > ? (192.168.1.100) at 00:01:04:00:01:01 [ether] PERM on eth0 >> > >> > >> > #After this configuration I trying to ping from VM1 VM2 (both IP) >> > >> > root@debian-amd64:~# ping -I eth0 192.168.0.200 >> > PING 192.168.0.200 (192.168.0.200) from 192.168.0.100 eth0: 56(84) bytes >> of >> > data. >> > ^C >> > --- 192.168.0.200 ping statistics --- >> > 5 packets transmitted, 0 received, 100% packet loss, time 4032ms >> > >> > root@debian-amd64:~# ping 192.168.0.200 >> > PING 192.168.0.200 (192.168.0.200) 56(84) bytes of data. >> > ^C >> > --- 192.168.0.200 ping statistics --- >> > 5 packets transmitted, 0 received, 100% packet loss, time 3999ms >> > >> > root@debian-amd64:~# ping -I eth1 192.168.1.200 >> > PING 192.168.1.200 (192.168.1.200) from 192.168.1.100 eth1: 56(84) bytes >> of >> > data. >> > ^C >> > --- 192.168.1.200 ping statistics --- >> > 6 packets transmitted, 0 received, 100% packet loss, time 5040ms >> > >> > root@debian-amd64:~# ping 192.168.1.200 >> > PING 192.168.1.200 (192.168.1.200) 56(84) bytes of data. >> > ^C >> > --- 192.168.1.200 ping statistics --- >> > 5 packets transmitted, 0 received, 100% packet loss, time 4030ms >> > >> > #and from VM2 VM1 >> > root@debian-amd64:~# ping 192.168.0.100 >> > PING 192.168.0.100 (192.168.0.100) 56(84) bytes of data. >> > ^C >> > --- 192.168.0.100 ping statistics --- >> > 3 packets transmitted, 0 received, 100% packet loss, time 2015ms >> > >> > root@debian-amd64:~# ping -I eth1 192.168.0.100 >> > PING 192.168.0.100 (192.168.0.100) from 192.168.0.200 eth1: 56(84) bytes >> of >> > data. >> > ^C >> > --- 192.168.0.100 ping statistics --- >> > 5 packets transmitted, 0 received, 100% packet loss, time 4032ms >> > >> > root@debian-amd64:~# ping -I eth0 192.168.1.100 >> > PING 192.168.1.100 (192.168.1.100) from 192.168.1.200 eth0: 56(84) bytes >> of >> > data. >> > ^C >> > --- 192.168.1.100 ping statistics --- >> > 4 packets transmitted, 0 received, 100% packet loss, time 3024ms >> > >> > root@debian-amd64:~# ping 192.168.1.100 >> > PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data. >> > ^C >> > --- 192.168.1.100 ping statistics --- >> > 8 packets transmitted, 0 received, 100% packet loss, time 7055ms >> > >> > >> > Also stats from vhost: >> > Device statistics ==================================== >> > Statistics for device 0 ------------------------------ >> > TX total: 0 >> > TX dropped: 0 >> > TX successful: 0 >> > RX total: 0 >> > RX dropped: 0 >> > RX successful: 0 >> > Statistics for device 1 ------------------------------ >> > TX total: 0 >> > TX dropped: 0 >> > TX successful: 0 >> > RX total: 0 >> > RX dropped: 0 >> > RX successful: 0 >> > Statistics for device 2 ------------------------------ >> > TX total: 0 >> > TX dropped: 0 >> > TX successful: 0 >> > RX total: 0 >> > RX dropped: 0 >> > RX successful: 0 >> > Statistics for device 3 ------------------------------ >> > TX total: 0 >> > TX dropped: 0 >> > TX successful: 0 >> > RX total: 0 >> > RX dropped: 0 >> > RX successful: 0 >> > ====================================================== >> > >> > My way of thinking was: "In vhost there are several function for L2 that >> > learn MAC-s and links them so why I see no received packets?" >> > >> > Maybe I'm doing some silly bug in network configuration but for me its >> > looking like data flow issue especially that no function on the vhost >> side >> > did not see any packages. >> > >> > On Fri, May 22, 2015 at 10:54 AM, Gaohaifeng (A) < >> gaohaifeng.gao@huawei.com> >> > wrote: >> > >> >> Hi >> >> >> >> What kernel version are You using on host/guest? >> >> >> >>>> I use ubuntu 12.04(3.11.0-15-generic) in host. In vm I use ubuntu >> 12.04 >> >> and ubuntu14.04 both, but the result is same. >> >> >> >> >> >> >> >> Do you use some command I suggest before, In case of you miss the >> previous >> >> mail, just copy it again: >> >> >> >>>> I try it but the result is same >> >> >> >> >> >> >> >> >> >> I use l2fwd in vm to do more test and found that virtio_xmit_pkts is >> >> called and avail_idx is increasing in vm, but in host avail_idx(in >> >> rte_vhost_dequeue_burst function) is always zero. It seems that the host >> >> see the different mem area. >> >> >> >> >> >> >> >> Init Logs below: >> >> >> >> VHOST_CONFIG: (0) Mergeable RX buffers disabled >> >> >> >> VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE >> >> >> >> VHOST_CONFIG: mapped region 0 fd:24 to 0x2aaaaac00000 sz:0xa0000 off:0x0 >> >> >> >> VHOST_CONFIG: REGION: 0 GPA: (nil) QEMU VA: 0x2aaaaac00000 SIZE (655360) >> >> >> >> VHOST_CONFIG: mapped region 1 fd:26 to 0x2aaaaae00000 sz:0x40000000 >> >> off:0xc0000 >> >> >> >> VHOST_CONFIG: REGION: 1 GPA: 0xc0000 QEMU VA: 0x2aaaaacc0000 SIZE >> >> (1072955392) >> >> >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM >> >> >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE >> >> >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR >> >> >> >> VHOST_CONFIG: (0) mapped address desc: 0x2aaae62f1000 >> >> >> >> VHOST_CONFIG: (0) mapped address avail: 0x2aaae62f2000 >> >> >> >> VHOST_CONFIG: (0) mapped address used: 0x2aaae62f3000 >> >> >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK >> >> >> >> VHOST_CONFIG: vring kick idx:0 file:23 >> >> >> >> VHOST_CONFIG: virtio isn't ready for processing. >> >> >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM >> >> >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE >> >> >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR >> >> >> >> VHOST_CONFIG: (0) mapped address desc: 0x2aaae62f4000 >> >> >> >> VHOST_CONFIG: (0) mapped address avail: 0x2aaae62f5000 >> >> >> >> VHOST_CONFIG: (0) mapped address used: 0x2aaae62f6000 >> >> >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK >> >> >> >> VHOST_CONFIG: vring kick idx:1 file:28 >> >> >> >> VHOST_CONFIG: virtio is now ready for processing. >> >> >> >> >> >> >> >> >> >> >> >>> Unfortunately not, I have the same issue in rte_vhost_dequeue_burst >> >> function. >> >> >> >> >> >> >> >>> What kernel version are You using on host/guest? In my case on host I >> >> had 3.13.0 and on guests old 3.2 debian. >> >> >> >> >> >> >> >>> I just looked deeper into virtio back-end (vhost) but at first glace >> it >> >> seems like nothing coming from virtio. >> >> >> >> >> >> >> >>> What I'm going to do today is to compile newest kernel for vhost and >> >> guest and debug where packet flow stuck, I will report the result >> >> >> >> >> >> >> >>> On Thu, May 21, 2015 at 11:12 AM, Gaohaifeng (A) < >> >> gaohaifeng.gao@huawei.com> wrote: >> >> >> >>> Hi Maciej >> >> >Did you solve your problem? I meet this problem as your case. >> >> And I found avail_idx(in rte_vhost_dequeue_burst function) is always >> zero >> >> although I do send packets in VM. >> >> >> >>> Thanks. >> >> >> >> >> >>> Hello, I have strange issue with example/vhost app. >> >>> >> >>> I had compiled DPDK to run a vhost example app with followed flags >> >>> >> >>> CONFIG_RTE_LIBRTE_VHOST=y >> >>> CONFIG_RTE_LIBRTE_VHOST_USER=y >> >>> CONFIG_RTE_LIBRTE_VHOST_DEBUG=n >> >>> >> >>> then I run vhost app based on documentation: >> >>> >> >>> ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge --socket-mem >> >>> 3712 >> >>> -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9 >> >>> >> >>> -I use this strange --socket-mem 3712 because of physical limit of >> >>> memoryon device -with this vhost user I run two KVM machines with >> >>> followed parameters >> >>> >> >>> kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 -cpu >> >>> host -smp 2 -hda /home/ubuntu/qemu/debian_squeeze2_amd64.qcow2 -m >> >>> 1024 -mem-path /mnt/huge -mem-prealloc -chardev >> >>> socket,id=char1,path=/home/ubuntu/dpdk/examples/vhost/usvhost >> >>> -netdev type=vhost-user,id=hostnet1,chardev=char1 >> >>> -device virtio-net >> >>> pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6 >> >>> = >> >>> off,guest_ecn=off >> >>> -chardev >> >>> socket,id=char2,path=/home/ubuntu/dpdk/examples/vhost/usvhost >> >>> -netdev type=vhost-user,id=hostnet2,chardev=char2 >> >>> -device >> >>> virtio-net- >> >>> pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6 >> >>> = >> >>> off,guest_ecn=off >> >>> >> >>> After running KVM virtio correctly starting (below logs from vhost app) >> >> ... >> >>> VHOST_CONFIG: mapped region 0 fd:31 to 0x2aaabae00000 sz:0xa0000 >> >>> off:0x0 >> >>> VHOST_CONFIG: mapped region 1 fd:37 to 0x2aaabb000000 sz:0x10000000 >> >>> off:0xc0000 >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK >> >>> VHOST_CONFIG: vring kick idx:0 file:38 >> >>> VHOST_CONFIG: virtio isn't ready for processing. >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK >> >>> VHOST_CONFIG: vring kick idx:1 file:39 >> >>> VHOST_CONFIG: virtio is now ready for processing. >> >>> VHOST_DATA: (1) Device has been added to data core 2 >> >>> >> >>> So everything looking good. >> >>> >> >>> Maybe it is something trivial but using options: --vm2vm 1 (or) 2 >> >>> --stats 9 it seems that I didn't have connection between VM2VM >> >>> communication. I set manually IP for eth0 and eth1: >> >>> >> >>> on 1 VM >> >>> ifconfig eth0 192.168.0.100 netmask 255.255.255.0 up ifconfig eth1 >> >>> 192.168.1.101 netmask 255.255.255.0 up >> >>> >> >>> on 2 VM >> >>> ifconfig eth0 192.168.1.200 netmask 255.255.255.0 up ifconfig eth1 >> >>> 192.168.0.202 netmask 255.255.255.0 up >> >>> >> >>> I notice that in vhostapp are one directional rx/tx queue so I tryied >> >>> to ping between VM1 to VM2 using both interfaces ping -I eth0 >> >>> 192.168.1.200 ping -I >> >>> eth1 192.168.1.200 ping -I eth0 192.168.0.202 ping -I eth1 >> >>> 192.168.0.202 >> >>> >> >>> on VM2 using tcpdump on both interfaces I didn't see any ICMP requests >> >>> or traffic >> >>> >> >>> And I cant ping between any IP/interfaces, moreover stats show me that: >> >>> >> >>> Device statistics ==================================== >> >>> Statistics for device 0 ------------------------------ >> >>> TX total: 0 >> >>> TX dropped: 0 >> >>> TX successful: 0 >> >>> RX total: 0 >> >>> RX dropped: 0 >> >>> RX successful: 0 >> >>> Statistics for device 1 ------------------------------ >> >>> TX total: 0 >> >>> TX dropped: 0 >> >>> TX successful: 0 >> >>> RX total: 0 >> >>> RX dropped: 0 >> >>> RX successful: 0 >> >>> Statistics for device 2 ------------------------------ >> >>> TX total: 0 >> >>> TX dropped: 0 >> >>> TX successful: 0 >> >>> RX total: 0 >> >>> RX dropped: 0 >> >>> RX successful: 0 >> >>> Statistics for device 3 ------------------------------ >> >>> TX total: 0 >> >>> TX dropped: 0 >> >>> TX successful: 0 >> >>> RX total: 0 >> >>> RX dropped: 0 >> >>> RX successful: 0 >> >>> ====================================================== >> >>> >> >>> So it seems like any packet didn't leave my VM. >> >>> also arp table is empty on each VM. >> >> >> >> >> >> -- Andriy Berestovskyy ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [dpdk-dev] Vhost user no connection vm2vm 2015-05-22 10:54 ` Andriy Berestovskyy @ 2015-05-22 17:59 ` Maciej Grochowski 2015-05-25 4:15 ` Gaohaifeng (A) 0 siblings, 1 reply; 13+ messages in thread From: Maciej Grochowski @ 2015-05-22 17:59 UTC (permalink / raw) To: Andriy Berestovskyy; +Cc: dev Thank You Andriy, You are right Below I put tested KVM configuration that put packets into vhost dpdk plane space: export TLBFS_DIR=/mnt/huge export UVH_PREFIX=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost export VM1_MAC=00:01:04:00:01:00 kvm -cpu host -smp 2 -enable-kvm \ -drive if=virtio,file=debian_min_2.qcow2,cache=none \ -object memory-backend-file,id=mem,size=756M,mem-path=${TLBFS_DIR},share=on \ -numa node,memdev=mem \ -m 756 -nographic \ -chardev socket,id=charnet0,path=${UVH_PREFIX} \ -netdev type=vhost-user,id=hostnet0,chardev=charnet0 \ -device virtio-net-pci,netdev=hostnet0,mac=${VM1_MAC} Notice that: memory-backend-file must be with right -numa node, "memdev=mem" option So yeah right configuration is a essence :) On Fri, May 22, 2015 at 12:54 PM, Andriy Berestovskyy <aber@semihalf.com> wrote: > Hi guys, > I guess you just miss the qemu flag to map the memory as shared, i.e.: > -object memory-backend-file,id=mem,size=1024M,mem-path=/mnt/huge,share=on > (the keyword is share=on) > > Here is an example script: > > https://github.com/srajag/contrail-vrouter/blob/dpdk-devel/examples/vms/VROUTER1/80.start-vm.sh > > Regards, > Andriy > > On Fri, May 22, 2015 at 12:04 PM, Maciej Grochowski > <maciej.grochowski@codilime.com> wrote: > > I checked this, results below > > > > #before script: > > root@# cat /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages > > 494 > > #after 1 qemu script > > root@# cat /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages > > 366 > > > > So qemu consume 262144k~262MB that is correct with script > > > > On Fri, May 22, 2015 at 11:58 AM, Tetsuya Mukawa <mukawa@igel.co.jp> > wrote: > > > >> Hi Maciej, > >> > >> I guess it's nice to make sure guest memory is actually allocated by > >> hugepages. > >> So please check like below. > >> > >> $ cat /sys/kernel/mm/hugepage/xxxxx/free_hugepages > >> $ ./start_qemu.sh > >> $ cat /sys/kernel/mm/hugepage/xxxxx/free_hugepages > >> > >> If qemu guest allocates memory from hugepages, 2nd cat command will > >> indicate it. > >> > >> Thanks, > >> Tetsuya > >> > >> > >> On 2015/05/22 18:28, Maciej Grochowski wrote: > >> > "Do you use some command I suggest before, > >> > In case of you miss the previous mail, just copy it again:" > >> > > >> > -Yes but it didn't help me ;/ > >> > > >> > I will describe step by step to esure that configuration is made by > right > >> > way > >> > > >> > > >> > I started vhost: > >> > > >> > ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge --socket-mem > >> 3712 > >> > -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9 > >> > > >> > Now I run two vm machines, with followed configuration > >> > > >> > VM1 __ __ VM2 > >> > eth0 > \/ > eth0 > >> > eth1 >__/\__> eth1 > >> > > >> > So I will connect VM1.eth0 with VM2.eth1 and VM1.eth1 with VM2.eth0 > >> > Because it is test env and I didn't have other network connection on > >> vhost > >> > I will create two networks 192.168.0.x and 192.168.1.x > >> > VM1.eth0 with VM2.eth1 will be placed in 192.168.0.x and VM1.eth1 > with > >> > VM2.eth0 in 192.168.1.x > >> > > >> > ## I started first VM1 as follow > >> > kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm2 -cpu > >> host > >> > -smp 1 \ > >> > -hda /home/ubuntu/esi_ee/qemu/debian_min_1.qcow2 -m 256 -mem-path > >> /mnt/huge > >> > -mem-prealloc \ > >> > -chardev > >> > socket,id=char3,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ > >> > -netdev type=vhost-user,id=hostnet3,chardev=char3 \ > >> > -device > >> > > >> > virtio-net-pci,netdev=hostnet3,id=net3,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off > >> > \ > >> > -chardev > >> > socket,id=char4,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ > >> > -netdev type=vhost-user,id=hostnet4,chardev=char4 \ > >> > -device > >> > > >> > virtio-net-pci,netdev=hostnet4,id=net4,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off > >> > ## qemu give followed output > >> > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet3,chardev=char3: > >> > chardev "char3" went up > >> > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet4,chardev=char4: > >> > chardev "char4" went up > >> > > >> > ## second VM2 > >> > kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 -cpu > >> host > >> > -smp 1 \ > >> > -hda /home/ubuntu/esi_ee/qemu/debian_min_2.qcow2 -m 256 -mem-path > >> /mnt/huge > >> > -mem-prealloc \ > >> > -chardev > >> > socket,id=char1,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ > >> > -netdev type=vhost-user,id=hostnet1,chardev=char1 \ > >> > -device > >> > > >> > virtio-net-pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off > >> > \ > >> > -chardev > >> > socket,id=char2,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \ > >> > -netdev type=vhost-user,id=hostnet2,chardev=char2 \ > >> > -device > >> > > >> > virtio-net-pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off > >> > ## second output > >> > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet1,chardev=char1: > >> > chardev "char1" went up > >> > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet2,chardev=char2: > >> > chardev "char2" went up > >> > > >> > > >> > > >> > After that I had MAC conflict between VM2 and VM1 > >> > > >> > VM1: -ifconfig -a > >> > eth0 Link encap:Ethernet HWaddr 52:54:00:12:34:56 > >> > inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link > >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > > >> > eth1 Link encap:Ethernet HWaddr 52:54:00:12:34:57 > >> > BROADCAST MULTICAST MTU:1500 Metric:1 > >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > > >> > > >> > VM2: -ifconfig -a > >> > eth0 Link encap:Ethernet HWaddr 52:54:00:12:34:56 > >> > inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link > >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > > >> > eth1 Link encap:Ethernet HWaddr 52:54:00:12:34:57 > >> > BROADCAST MULTICAST MTU:1500 Metric:1 > >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > > >> > In KNI example I had something similar and also no packet flow and > >> solution > >> > was to change MAC addresses > >> > > >> > #VM1 > >> > /etc/init.d/networking stop > >> > ifconfig eth0 hw ether 00:01:04:00:01:00 > >> > ifconfig eth1 hw ether 00:01:04:00:01:01 > >> > /etc/init.d/networking start > >> > ifconfig eth0 > >> > ifconfig eth1 > >> > > >> > #VM2 > >> > /etc/init.d/networking stop > >> > ifconfig eth0 hw ether 00:01:04:00:02:00 > >> > ifconfig eth1 hw ether 00:01:04:00:02:01 > >> > /etc/init.d/networking start > >> > ifconfig eth0 > >> > ifconfig eth1 > >> > > >> > Then I make a configuration that You show: > >> > > >> > #VM1 > >> > ip addr add 192.168.0.100/24 dev eth0 > >> > ip addr add 192.168.1.100/24 dev eth1 > >> > ip neigh add 192.168.0.200 lladdr 00:01:04:00:02:01 dev eth0 > >> > ip link set dev eth0 up > >> > ip neigh add 192.168.1.200 lladdr 00:01:04:00:02:00 dev eth1 > >> > ip link set dev eth1 up > >> > > >> > eth0 Link encap:Ethernet HWaddr 00:01:04:00:01:00 > >> > inet addr:192.168.0.100 Bcast:0.0.0.0 Mask:255.255.255.0 > >> > inet6 addr: fe80::201:4ff:fe00:100/64 Scope:Link > >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > > >> > eth1 Link encap:Ethernet HWaddr 00:01:04:00:01:01 > >> > inet addr:192.168.1.100 Bcast:0.0.0.0 Mask:255.255.255.0 > >> > inet6 addr: fe80::201:4ff:fe00:101/64 Scope:Link > >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > > >> > > >> > #VM2 > >> > ip addr add 192.168.1.200/24 dev eth0 > >> > ip addr add 192.168.0.200/24 dev eth1 > >> > ip neigh add 192.168.1.100 lladdr 00:01:04:00:01:01 dev eth0 > >> > ip link set dev eth0 up > >> > ip neigh add 192.168.0.100 lladdr 00:01:04:00:01:00 dev eth1 > >> > ip link set dev eth1 up > >> > > >> > eth0 Link encap:Ethernet HWaddr 00:01:04:00:02:00 > >> > inet addr:192.168.1.200 Bcast:0.0.0.0 Mask:255.255.255.0 > >> > inet6 addr: fe80::201:4ff:fe00:200/64 Scope:Link > >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > > >> > eth1 Link encap:Ethernet HWaddr 00:01:04:00:02:01 > >> > inet addr:192.168.0.200 Bcast:0.0.0.0 Mask:255.255.255.0 > >> > inet6 addr: fe80::201:4ff:fe00:201/64 Scope:Link > >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > > >> > After that: > >> > > >> > VM1.eth0 ip=192.168.0.100-MAC=00:01:04:00:01:00 is connected to > VM2.eth1 > >> > ip=192.168.0.200-MAC=00:01:04:00:02:01 > >> > VM1.eth1 ip=192.168.1.100-MAC=00:01:04:00:01:01 is connected to > VM2.eth0 > >> > ip=192.168.1.200-MAC=00:01:04:00:02:00 > >> > > >> > That show my arp tables: > >> > > >> > #VM1 > >> > arp -a > >> > ? (192.168.0.200) at 00:01:04:00:02:01 [ether] PERM on eth0 > >> > ? (192.168.1.200) at 00:01:04:00:02:00 [ether] PERM on eth1 > >> > > >> > > >> > #VM2 > >> > arp -a > >> > ? (192.168.0.100) at 00:01:04:00:01:00 [ether] PERM on eth1 > >> > ? (192.168.1.100) at 00:01:04:00:01:01 [ether] PERM on eth0 > >> > > >> > > >> > #After this configuration I trying to ping from VM1 VM2 (both IP) > >> > > >> > root@debian-amd64:~# ping -I eth0 192.168.0.200 > >> > PING 192.168.0.200 (192.168.0.200) from 192.168.0.100 eth0: 56(84) > bytes > >> of > >> > data. > >> > ^C > >> > --- 192.168.0.200 ping statistics --- > >> > 5 packets transmitted, 0 received, 100% packet loss, time 4032ms > >> > > >> > root@debian-amd64:~# ping 192.168.0.200 > >> > PING 192.168.0.200 (192.168.0.200) 56(84) bytes of data. > >> > ^C > >> > --- 192.168.0.200 ping statistics --- > >> > 5 packets transmitted, 0 received, 100% packet loss, time 3999ms > >> > > >> > root@debian-amd64:~# ping -I eth1 192.168.1.200 > >> > PING 192.168.1.200 (192.168.1.200) from 192.168.1.100 eth1: 56(84) > bytes > >> of > >> > data. > >> > ^C > >> > --- 192.168.1.200 ping statistics --- > >> > 6 packets transmitted, 0 received, 100% packet loss, time 5040ms > >> > > >> > root@debian-amd64:~# ping 192.168.1.200 > >> > PING 192.168.1.200 (192.168.1.200) 56(84) bytes of data. > >> > ^C > >> > --- 192.168.1.200 ping statistics --- > >> > 5 packets transmitted, 0 received, 100% packet loss, time 4030ms > >> > > >> > #and from VM2 VM1 > >> > root@debian-amd64:~# ping 192.168.0.100 > >> > PING 192.168.0.100 (192.168.0.100) 56(84) bytes of data. > >> > ^C > >> > --- 192.168.0.100 ping statistics --- > >> > 3 packets transmitted, 0 received, 100% packet loss, time 2015ms > >> > > >> > root@debian-amd64:~# ping -I eth1 192.168.0.100 > >> > PING 192.168.0.100 (192.168.0.100) from 192.168.0.200 eth1: 56(84) > bytes > >> of > >> > data. > >> > ^C > >> > --- 192.168.0.100 ping statistics --- > >> > 5 packets transmitted, 0 received, 100% packet loss, time 4032ms > >> > > >> > root@debian-amd64:~# ping -I eth0 192.168.1.100 > >> > PING 192.168.1.100 (192.168.1.100) from 192.168.1.200 eth0: 56(84) > bytes > >> of > >> > data. > >> > ^C > >> > --- 192.168.1.100 ping statistics --- > >> > 4 packets transmitted, 0 received, 100% packet loss, time 3024ms > >> > > >> > root@debian-amd64:~# ping 192.168.1.100 > >> > PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data. > >> > ^C > >> > --- 192.168.1.100 ping statistics --- > >> > 8 packets transmitted, 0 received, 100% packet loss, time 7055ms > >> > > >> > > >> > Also stats from vhost: > >> > Device statistics ==================================== > >> > Statistics for device 0 ------------------------------ > >> > TX total: 0 > >> > TX dropped: 0 > >> > TX successful: 0 > >> > RX total: 0 > >> > RX dropped: 0 > >> > RX successful: 0 > >> > Statistics for device 1 ------------------------------ > >> > TX total: 0 > >> > TX dropped: 0 > >> > TX successful: 0 > >> > RX total: 0 > >> > RX dropped: 0 > >> > RX successful: 0 > >> > Statistics for device 2 ------------------------------ > >> > TX total: 0 > >> > TX dropped: 0 > >> > TX successful: 0 > >> > RX total: 0 > >> > RX dropped: 0 > >> > RX successful: 0 > >> > Statistics for device 3 ------------------------------ > >> > TX total: 0 > >> > TX dropped: 0 > >> > TX successful: 0 > >> > RX total: 0 > >> > RX dropped: 0 > >> > RX successful: 0 > >> > ====================================================== > >> > > >> > My way of thinking was: "In vhost there are several function for L2 > that > >> > learn MAC-s and links them so why I see no received packets?" > >> > > >> > Maybe I'm doing some silly bug in network configuration but for me its > >> > looking like data flow issue especially that no function on the vhost > >> side > >> > did not see any packages. > >> > > >> > On Fri, May 22, 2015 at 10:54 AM, Gaohaifeng (A) < > >> gaohaifeng.gao@huawei.com> > >> > wrote: > >> > > >> >> Hi > >> >> > >> >> What kernel version are You using on host/guest? > >> >> > >> >>>> I use ubuntu 12.04(3.11.0-15-generic) in host. In vm I use ubuntu > >> 12.04 > >> >> and ubuntu14.04 both, but the result is same. > >> >> > >> >> > >> >> > >> >> Do you use some command I suggest before, In case of you miss the > >> previous > >> >> mail, just copy it again: > >> >> > >> >>>> I try it but the result is same > >> >> > >> >> > >> >> > >> >> > >> >> I use l2fwd in vm to do more test and found that virtio_xmit_pkts is > >> >> called and avail_idx is increasing in vm, but in host avail_idx(in > >> >> rte_vhost_dequeue_burst function) is always zero. It seems that the > host > >> >> see the different mem area. > >> >> > >> >> > >> >> > >> >> Init Logs below: > >> >> > >> >> VHOST_CONFIG: (0) Mergeable RX buffers disabled > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE > >> >> > >> >> VHOST_CONFIG: mapped region 0 fd:24 to 0x2aaaaac00000 sz:0xa0000 > off:0x0 > >> >> > >> >> VHOST_CONFIG: REGION: 0 GPA: (nil) QEMU VA: 0x2aaaaac00000 SIZE > (655360) > >> >> > >> >> VHOST_CONFIG: mapped region 1 fd:26 to 0x2aaaaae00000 sz:0x40000000 > >> >> off:0xc0000 > >> >> > >> >> VHOST_CONFIG: REGION: 1 GPA: 0xc0000 QEMU VA: 0x2aaaaacc0000 SIZE > >> >> (1072955392) > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > >> >> > >> >> VHOST_CONFIG: (0) mapped address desc: 0x2aaae62f1000 > >> >> > >> >> VHOST_CONFIG: (0) mapped address avail: 0x2aaae62f2000 > >> >> > >> >> VHOST_CONFIG: (0) mapped address used: 0x2aaae62f3000 > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > >> >> > >> >> VHOST_CONFIG: vring kick idx:0 file:23 > >> >> > >> >> VHOST_CONFIG: virtio isn't ready for processing. > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > >> >> > >> >> VHOST_CONFIG: (0) mapped address desc: 0x2aaae62f4000 > >> >> > >> >> VHOST_CONFIG: (0) mapped address avail: 0x2aaae62f5000 > >> >> > >> >> VHOST_CONFIG: (0) mapped address used: 0x2aaae62f6000 > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > >> >> > >> >> VHOST_CONFIG: vring kick idx:1 file:28 > >> >> > >> >> VHOST_CONFIG: virtio is now ready for processing. > >> >> > >> >> > >> >> > >> >> > >> >> > >> >>> Unfortunately not, I have the same issue in rte_vhost_dequeue_burst > >> >> function. > >> >> > >> >> > >> >> > >> >>> What kernel version are You using on host/guest? In my case on host > I > >> >> had 3.13.0 and on guests old 3.2 debian. > >> >> > >> >> > >> >> > >> >>> I just looked deeper into virtio back-end (vhost) but at first > glace > >> it > >> >> seems like nothing coming from virtio. > >> >> > >> >> > >> >> > >> >>> What I'm going to do today is to compile newest kernel for vhost and > >> >> guest and debug where packet flow stuck, I will report the result > >> >> > >> >> > >> >> > >> >>> On Thu, May 21, 2015 at 11:12 AM, Gaohaifeng (A) < > >> >> gaohaifeng.gao@huawei.com> wrote: > >> >> > >> >>> Hi Maciej > >> >> >Did you solve your problem? I meet this problem as your > case. > >> >> And I found avail_idx(in rte_vhost_dequeue_burst function) is always > >> zero > >> >> although I do send packets in VM. > >> >> > >> >>> Thanks. > >> >> > >> >> > >> >>> Hello, I have strange issue with example/vhost app. > >> >>> > >> >>> I had compiled DPDK to run a vhost example app with followed flags > >> >>> > >> >>> CONFIG_RTE_LIBRTE_VHOST=y > >> >>> CONFIG_RTE_LIBRTE_VHOST_USER=y > >> >>> CONFIG_RTE_LIBRTE_VHOST_DEBUG=n > >> >>> > >> >>> then I run vhost app based on documentation: > >> >>> > >> >>> ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge > --socket-mem > >> >>> 3712 > >> >>> -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9 > >> >>> > >> >>> -I use this strange --socket-mem 3712 because of physical limit of > >> >>> memoryon device -with this vhost user I run two KVM machines with > >> >>> followed parameters > >> >>> > >> >>> kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 > -cpu > >> >>> host -smp 2 -hda /home/ubuntu/qemu/debian_squeeze2_amd64.qcow2 -m > >> >>> 1024 -mem-path /mnt/huge -mem-prealloc -chardev > >> >>> socket,id=char1,path=/home/ubuntu/dpdk/examples/vhost/usvhost > >> >>> -netdev type=vhost-user,id=hostnet1,chardev=char1 > >> >>> -device virtio-net > >> >>> > pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6 > >> >>> = > >> >>> off,guest_ecn=off > >> >>> -chardev > >> >>> socket,id=char2,path=/home/ubuntu/dpdk/examples/vhost/usvhost > >> >>> -netdev type=vhost-user,id=hostnet2,chardev=char2 > >> >>> -device > >> >>> virtio-net- > >> >>> > pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6 > >> >>> = > >> >>> off,guest_ecn=off > >> >>> > >> >>> After running KVM virtio correctly starting (below logs from vhost > app) > >> >> ... > >> >>> VHOST_CONFIG: mapped region 0 fd:31 to 0x2aaabae00000 sz:0xa0000 > >> >>> off:0x0 > >> >>> VHOST_CONFIG: mapped region 1 fd:37 to 0x2aaabb000000 sz:0x10000000 > >> >>> off:0xc0000 > >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > >> >>> VHOST_CONFIG: vring kick idx:0 file:38 > >> >>> VHOST_CONFIG: virtio isn't ready for processing. > >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > >> >>> VHOST_CONFIG: vring kick idx:1 file:39 > >> >>> VHOST_CONFIG: virtio is now ready for processing. > >> >>> VHOST_DATA: (1) Device has been added to data core 2 > >> >>> > >> >>> So everything looking good. > >> >>> > >> >>> Maybe it is something trivial but using options: --vm2vm 1 (or) 2 > >> >>> --stats 9 it seems that I didn't have connection between VM2VM > >> >>> communication. I set manually IP for eth0 and eth1: > >> >>> > >> >>> on 1 VM > >> >>> ifconfig eth0 192.168.0.100 netmask 255.255.255.0 up ifconfig eth1 > >> >>> 192.168.1.101 netmask 255.255.255.0 up > >> >>> > >> >>> on 2 VM > >> >>> ifconfig eth0 192.168.1.200 netmask 255.255.255.0 up ifconfig eth1 > >> >>> 192.168.0.202 netmask 255.255.255.0 up > >> >>> > >> >>> I notice that in vhostapp are one directional rx/tx queue so I > tryied > >> >>> to ping between VM1 to VM2 using both interfaces ping -I eth0 > >> >>> 192.168.1.200 ping -I > >> >>> eth1 192.168.1.200 ping -I eth0 192.168.0.202 ping -I eth1 > >> >>> 192.168.0.202 > >> >>> > >> >>> on VM2 using tcpdump on both interfaces I didn't see any ICMP > requests > >> >>> or traffic > >> >>> > >> >>> And I cant ping between any IP/interfaces, moreover stats show me > that: > >> >>> > >> >>> Device statistics ==================================== > >> >>> Statistics for device 0 ------------------------------ > >> >>> TX total: 0 > >> >>> TX dropped: 0 > >> >>> TX successful: 0 > >> >>> RX total: 0 > >> >>> RX dropped: 0 > >> >>> RX successful: 0 > >> >>> Statistics for device 1 ------------------------------ > >> >>> TX total: 0 > >> >>> TX dropped: 0 > >> >>> TX successful: 0 > >> >>> RX total: 0 > >> >>> RX dropped: 0 > >> >>> RX successful: 0 > >> >>> Statistics for device 2 ------------------------------ > >> >>> TX total: 0 > >> >>> TX dropped: 0 > >> >>> TX successful: 0 > >> >>> RX total: 0 > >> >>> RX dropped: 0 > >> >>> RX successful: 0 > >> >>> Statistics for device 3 ------------------------------ > >> >>> TX total: 0 > >> >>> TX dropped: 0 > >> >>> TX successful: 0 > >> >>> RX total: 0 > >> >>> RX dropped: 0 > >> >>> RX successful: 0 > >> >>> ====================================================== > >> >>> > >> >>> So it seems like any packet didn't leave my VM. > >> >>> also arp table is empty on each VM. > >> >> > >> >> > >> > >> > > > > -- > Andriy Berestovskyy > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [dpdk-dev] Vhost user no connection vm2vm 2015-05-22 17:59 ` Maciej Grochowski @ 2015-05-25 4:15 ` Gaohaifeng (A) 0 siblings, 0 replies; 13+ messages in thread From: Gaohaifeng (A) @ 2015-05-25 4:15 UTC (permalink / raw) To: Maciej Grochowski, Andriy Berestovskyy; +Cc: dev Much Thanks. >Thank You Andriy, You are right >Below I put tested KVM configuration that put packets into vhost dpdk plane >space: >export TLBFS_DIR=/mnt/huge >export UVH_PREFIX=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost >export VM1_MAC=00:01:04:00:01:00 >kvm -cpu host -smp 2 -enable-kvm \ > -drive if=virtio,file=debian_min_2.qcow2,cache=none \ > -object >memory-backend-file,id=mem,size=756M,mem-path=${TLBFS_DIR},share=on \ > -numa node,memdev=mem \ > -m 756 -nographic \ > -chardev socket,id=charnet0,path=${UVH_PREFIX} \ > -netdev type=vhost-user,id=hostnet0,chardev=charnet0 \ > -device virtio-net-pci,netdev=hostnet0,mac=${VM1_MAC} >Notice that: >memory-backend-file must be with right -numa node, "memdev=mem" option > So yeah right configuration is a essence :) > On Fri, May 22, 2015 at 12:54 PM, Andriy Berestovskyy <aber@semihalf.com> > wrote: > Hi guys, > I guess you just miss the qemu flag to map the memory as shared, i.e.: > -object > memory-backend-file,id=mem,size=1024M,mem-path=/mnt/huge,share=on > (the keyword is share=on) > > Here is an example script: > > https://github.com/srajag/contrail-vrouter/blob/dpdk-devel/examples/vm > s/VROUTER1/80.start-vm.sh > > Regards, > Andriy > > On Fri, May 22, 2015 at 12:04 PM, Maciej Grochowski > <maciej.grochowski@codilime.com> wrote: > > I checked this, results below > > > > #before script: > > root@# cat /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages > > 494 > > #after 1 qemu script > > root@# cat /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages > > 366 > > > > So qemu consume 262144k~262MB that is correct with script > > > > On Fri, May 22, 2015 at 11:58 AM, Tetsuya Mukawa <mukawa@igel.co.jp> > wrote: > > > >> Hi Maciej, > >> > >> I guess it's nice to make sure guest memory is actually allocated > >> by hugepages. > >> So please check like below. > >> > >> $ cat /sys/kernel/mm/hugepage/xxxxx/free_hugepages > >> $ ./start_qemu.sh > >> $ cat /sys/kernel/mm/hugepage/xxxxx/free_hugepages > >> > >> If qemu guest allocates memory from hugepages, 2nd cat command will > >> indicate it. > >> > >> Thanks, > >> Tetsuya > >> > >> > >> On 2015/05/22 18:28, Maciej Grochowski wrote: > >> > "Do you use some command I suggest before, In case of you miss > >> > the previous mail, just copy it again:" > >> > > >> > -Yes but it didn't help me ;/ > >> > > >> > I will describe step by step to esure that configuration is made > >> > by > right > >> > way > >> > > >> > > >> > I started vhost: > >> > > >> > ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge > >> > --socket-mem > >> 3712 > >> > -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9 > >> > > >> > Now I run two vm machines, with followed configuration > >> > > >> > VM1 __ __ VM2 > >> > eth0 > \/ > eth0 > >> > eth1 >__/\__> eth1 > >> > > >> > So I will connect VM1.eth0 with VM2.eth1 and VM1.eth1 with > >> > VM2.eth0 Because it is test env and I didn't have other network > >> > connection on > >> vhost > >> > I will create two networks 192.168.0.x and 192.168.1.x > >> > VM1.eth0 with VM2.eth1 will be placed in 192.168.0.x and > >> > VM1.eth1 > with > >> > VM2.eth0 in 192.168.1.x > >> > > >> > ## I started first VM1 as follow > >> > kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm2 > >> > -cpu > >> host > >> > -smp 1 \ > >> > -hda /home/ubuntu/esi_ee/qemu/debian_min_1.qcow2 -m 256 -mem-path > >> /mnt/huge > >> > -mem-prealloc \ > >> > -chardev > >> > socket,id=char3,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvh > >> > ost \ -netdev type=vhost-user,id=hostnet3,chardev=char3 \ -device > >> > > >> > virtio-net-pci,netdev=hostnet3,id=net3,csum=off,gso=off,guest_tso4=off > ,guest_tso6=off,guest_ecn=off > >> > \ > >> > -chardev > >> > socket,id=char4,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvh > >> > ost \ -netdev type=vhost-user,id=hostnet4,chardev=char4 \ -device > >> > > >> > virtio-net-pci,netdev=hostnet4,id=net4,csum=off,gso=off,guest_tso4=off > ,guest_tso6=off,guest_ecn=off > >> > ## qemu give followed output > >> > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet3,chardev=char3: > >> > chardev "char3" went up > >> > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet4,chardev=char4: > >> > chardev "char4" went up > >> > > >> > ## second VM2 > >> > kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 > >> > -cpu > >> host > >> > -smp 1 \ > >> > -hda /home/ubuntu/esi_ee/qemu/debian_min_2.qcow2 -m 256 -mem-path > >> /mnt/huge > >> > -mem-prealloc \ > >> > -chardev > >> > socket,id=char1,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvh > >> > ost \ -netdev type=vhost-user,id=hostnet1,chardev=char1 \ -device > >> > > >> > virtio-net-pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off > ,guest_tso6=off,guest_ecn=off > >> > \ > >> > -chardev > >> > socket,id=char2,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvh > >> > ost \ -netdev type=vhost-user,id=hostnet2,chardev=char2 \ -device > >> > > >> > virtio-net-pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off > ,guest_tso6=off,guest_ecn=off > >> > ## second output > >> > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet1,chardev=char1: > >> > chardev "char1" went up > >> > qemu-system-x86_64: -netdev type=vhost-user,id=hostnet2,chardev=char2: > >> > chardev "char2" went up > >> > > >> > > >> > > >> > After that I had MAC conflict between VM2 and VM1 > >> > > >> > VM1: -ifconfig -a > >> > eth0 Link encap:Ethernet HWaddr 52:54:00:12:34:56 > >> > inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link > >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > > >> > eth1 Link encap:Ethernet HWaddr 52:54:00:12:34:57 > >> > BROADCAST MULTICAST MTU:1500 Metric:1 > >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > > >> > > >> > VM2: -ifconfig -a > >> > eth0 Link encap:Ethernet HWaddr 52:54:00:12:34:56 > >> > inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link > >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > > >> > eth1 Link encap:Ethernet HWaddr 52:54:00:12:34:57 > >> > BROADCAST MULTICAST MTU:1500 Metric:1 > >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > > >> > In KNI example I had something similar and also no packet flow > >> > and > >> solution > >> > was to change MAC addresses > >> > > >> > #VM1 > >> > /etc/init.d/networking stop > >> > ifconfig eth0 hw ether 00:01:04:00:01:00 ifconfig eth1 hw ether > >> > 00:01:04:00:01:01 /etc/init.d/networking start ifconfig eth0 > >> > ifconfig eth1 > >> > > >> > #VM2 > >> > /etc/init.d/networking stop > >> > ifconfig eth0 hw ether 00:01:04:00:02:00 > >> > ifconfig eth1 hw ether 00:01:04:00:02:01 > >> > /etc/init.d/networking start > >> > ifconfig eth0 > >> > ifconfig eth1 > >> > > >> > Then I make a configuration that You show: > >> > > >> > #VM1 > >> > ip addr add 192.168.0.100/24 dev eth0 > >> > ip addr add 192.168.1.100/24 dev eth1 > >> > ip neigh add 192.168.0.200 lladdr 00:01:04:00:02:01 dev eth0 > >> > ip link set dev eth0 up > >> > ip neigh add 192.168.1.200 lladdr 00:01:04:00:02:00 dev eth1 > >> > ip link set dev eth1 up > >> > > >> > eth0 Link encap:Ethernet HWaddr 00:01:04:00:01:00 > >> > inet addr:192.168.0.100 Bcast:0.0.0.0 Mask:255.255.255.0 > >> > inet6 addr: fe80::201:4ff:fe00:100/64 Scope:Link > >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > > >> > eth1 Link encap:Ethernet HWaddr 00:01:04:00:01:01 > >> > inet addr:192.168.1.100 Bcast:0.0.0.0 Mask:255.255.255.0 > >> > inet6 addr: fe80::201:4ff:fe00:101/64 Scope:Link > >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > > >> > > >> > #VM2 > >> > ip addr add 192.168.1.200/24 dev eth0 > >> > ip addr add 192.168.0.200/24 dev eth1 > >> > ip neigh add 192.168.1.100 lladdr 00:01:04:00:01:01 dev eth0 > >> > ip link set dev eth0 up > >> > ip neigh add 192.168.0.100 lladdr 00:01:04:00:01:00 dev eth1 > >> > ip link set dev eth1 up > >> > > >> > eth0 Link encap:Ethernet HWaddr 00:01:04:00:02:00 > >> > inet addr:192.168.1.200 Bcast:0.0.0.0 Mask:255.255.255.0 > >> > inet6 addr: fe80::201:4ff:fe00:200/64 Scope:Link > >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > > >> > eth1 Link encap:Ethernet HWaddr 00:01:04:00:02:01 > >> > inet addr:192.168.0.200 Bcast:0.0.0.0 Mask:255.255.255.0 > >> > inet6 addr: fe80::201:4ff:fe00:201/64 Scope:Link > >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > >> > > >> > After that: > >> > > >> > VM1.eth0 ip=192.168.0.100-MAC=00:01:04:00:01:00 is connected to > VM2.eth1 > >> > ip=192.168.0.200-MAC=00:01:04:00:02:01 > >> > VM1.eth1 ip=192.168.1.100-MAC=00:01:04:00:01:01 is connected to > VM2.eth0 > >> > ip=192.168.1.200-MAC=00:01:04:00:02:00 > >> > > >> > That show my arp tables: > >> > > >> > #VM1 > >> > arp -a > >> > ? (192.168.0.200) at 00:01:04:00:02:01 [ether] PERM on eth0 > >> > ? (192.168.1.200) at 00:01:04:00:02:00 [ether] PERM on eth1 > >> > > >> > > >> > #VM2 > >> > arp -a > >> > ? (192.168.0.100) at 00:01:04:00:01:00 [ether] PERM on eth1 > >> > ? (192.168.1.100) at 00:01:04:00:01:01 [ether] PERM on eth0 > >> > > >> > > >> > #After this configuration I trying to ping from VM1 VM2 (both IP) > >> > > >> > root@debian-amd64:~# ping -I eth0 192.168.0.200 > >> > PING 192.168.0.200 (192.168.0.200) from 192.168.0.100 eth0: 56(84) > bytes > >> of > >> > data. > >> > ^C > >> > --- 192.168.0.200 ping statistics --- > >> > 5 packets transmitted, 0 received, 100% packet loss, time 4032ms > >> > > >> > root@debian-amd64:~# ping 192.168.0.200 > >> > PING 192.168.0.200 (192.168.0.200) 56(84) bytes of data. > >> > ^C > >> > --- 192.168.0.200 ping statistics --- > >> > 5 packets transmitted, 0 received, 100% packet loss, time 3999ms > >> > > >> > root@debian-amd64:~# ping -I eth1 192.168.1.200 > >> > PING 192.168.1.200 (192.168.1.200) from 192.168.1.100 eth1: 56(84) > bytes > >> of > >> > data. > >> > ^C > >> > --- 192.168.1.200 ping statistics --- > >> > 6 packets transmitted, 0 received, 100% packet loss, time 5040ms > >> > > >> > root@debian-amd64:~# ping 192.168.1.200 > >> > PING 192.168.1.200 (192.168.1.200) 56(84) bytes of data. > >> > ^C > >> > --- 192.168.1.200 ping statistics --- > >> > 5 packets transmitted, 0 received, 100% packet loss, time 4030ms > >> > > >> > #and from VM2 VM1 > >> > root@debian-amd64:~# ping 192.168.0.100 > >> > PING 192.168.0.100 (192.168.0.100) 56(84) bytes of data. > >> > ^C > >> > --- 192.168.0.100 ping statistics --- > >> > 3 packets transmitted, 0 received, 100% packet loss, time 2015ms > >> > > >> > root@debian-amd64:~# ping -I eth1 192.168.0.100 > >> > PING 192.168.0.100 (192.168.0.100) from 192.168.0.200 eth1: 56(84) > bytes > >> of > >> > data. > >> > ^C > >> > --- 192.168.0.100 ping statistics --- > >> > 5 packets transmitted, 0 received, 100% packet loss, time 4032ms > >> > > >> > root@debian-amd64:~# ping -I eth0 192.168.1.100 > >> > PING 192.168.1.100 (192.168.1.100) from 192.168.1.200 eth0: 56(84) > bytes > >> of > >> > data. > >> > ^C > >> > --- 192.168.1.100 ping statistics --- > >> > 4 packets transmitted, 0 received, 100% packet loss, time 3024ms > >> > > >> > root@debian-amd64:~# ping 192.168.1.100 > >> > PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data. > >> > ^C > >> > --- 192.168.1.100 ping statistics --- > >> > 8 packets transmitted, 0 received, 100% packet loss, time 7055ms > >> > > >> > > >> > Also stats from vhost: > >> > Device statistics ==================================== > >> > Statistics for device 0 ------------------------------ > >> > TX total: 0 > >> > TX dropped: 0 > >> > TX successful: 0 > >> > RX total: 0 > >> > RX dropped: 0 > >> > RX successful: 0 > >> > Statistics for device 1 ------------------------------ > >> > TX total: 0 > >> > TX dropped: 0 > >> > TX successful: 0 > >> > RX total: 0 > >> > RX dropped: 0 > >> > RX successful: 0 > >> > Statistics for device 2 ------------------------------ > >> > TX total: 0 > >> > TX dropped: 0 > >> > TX successful: 0 > >> > RX total: 0 > >> > RX dropped: 0 > >> > RX successful: 0 > >> > Statistics for device 3 ------------------------------ > >> > TX total: 0 > >> > TX dropped: 0 > >> > TX successful: 0 > >> > RX total: 0 > >> > RX dropped: 0 > >> > RX successful: 0 > >> > ====================================================== > >> > > >> > My way of thinking was: "In vhost there are several function for L2 > that > >> > learn MAC-s and links them so why I see no received packets?" > >> > > >> > Maybe I'm doing some silly bug in network configuration but for me its > >> > looking like data flow issue especially that no function on the vhost > >> side > >> > did not see any packages. > >> > > >> > On Fri, May 22, 2015 at 10:54 AM, Gaohaifeng (A) < > >> gaohaifeng.gao@huawei.com> > >> > wrote: > >> > > >> >> Hi > >> >> > >> >> What kernel version are You using on host/guest? > >> >> > >> >>>> I use ubuntu 12.04(3.11.0-15-generic) in host. In vm I use ubuntu > >> 12.04 > >> >> and ubuntu14.04 both, but the result is same. > >> >> > >> >> > >> >> > >> >> Do you use some command I suggest before, In case of you miss the > >> previous > >> >> mail, just copy it again: > >> >> > >> >>>> I try it but the result is same > >> >> > >> >> > >> >> > >> >> > >> >> I use l2fwd in vm to do more test and found that virtio_xmit_pkts is > >> >> called and avail_idx is increasing in vm, but in host avail_idx(in > >> >> rte_vhost_dequeue_burst function) is always zero. It seems that the > host > >> >> see the different mem area. > >> >> > >> >> > >> >> > >> >> Init Logs below: > >> >> > >> >> VHOST_CONFIG: (0) Mergeable RX buffers disabled > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE > >> >> > >> >> VHOST_CONFIG: mapped region 0 fd:24 to 0x2aaaaac00000 sz:0xa0000 > off:0x0 > >> >> > >> >> VHOST_CONFIG: REGION: 0 GPA: (nil) QEMU VA: 0x2aaaaac00000 SIZE > (655360) > >> >> > >> >> VHOST_CONFIG: mapped region 1 fd:26 to 0x2aaaaae00000 sz:0x40000000 > >> >> off:0xc0000 > >> >> > >> >> VHOST_CONFIG: REGION: 1 GPA: 0xc0000 QEMU VA: 0x2aaaaacc0000 SIZE > >> >> (1072955392) > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > >> >> > >> >> VHOST_CONFIG: (0) mapped address desc: 0x2aaae62f1000 > >> >> > >> >> VHOST_CONFIG: (0) mapped address avail: 0x2aaae62f2000 > >> >> > >> >> VHOST_CONFIG: (0) mapped address used: 0x2aaae62f3000 > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > >> >> > >> >> VHOST_CONFIG: vring kick idx:0 file:23 > >> >> > >> >> VHOST_CONFIG: virtio isn't ready for processing. > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > >> >> > >> >> VHOST_CONFIG: (0) mapped address desc: 0x2aaae62f4000 > >> >> > >> >> VHOST_CONFIG: (0) mapped address avail: 0x2aaae62f5000 > >> >> > >> >> VHOST_CONFIG: (0) mapped address used: 0x2aaae62f6000 > >> >> > >> >> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > >> >> > >> >> VHOST_CONFIG: vring kick idx:1 file:28 > >> >> > >> >> VHOST_CONFIG: virtio is now ready for processing. > >> >> > >> >> > >> >> > >> >> > >> >> > >> >>> Unfortunately not, I have the same issue in rte_vhost_dequeue_burst > >> >> function. > >> >> > >> >> > >> >> > >> >>> What kernel version are You using on host/guest? In my case on host > I > >> >> had 3.13.0 and on guests old 3.2 debian. > >> >> > >> >> > >> >> > >> >>> I just looked deeper into virtio back-end (vhost) but at first > glace > >> it > >> >> seems like nothing coming from virtio. > >> >> > >> >> > >> >> > >> >>> What I'm going to do today is to compile newest kernel for vhost and > >> >> guest and debug where packet flow stuck, I will report the result > >> >> > >> >> > >> >> > >> >>> On Thu, May 21, 2015 at 11:12 AM, Gaohaifeng (A) < > >> >> gaohaifeng.gao@huawei.com> wrote: > >> >> > >> >>> Hi Maciej > >> >> >Did you solve your problem? I meet this problem as your > case. > >> >> And I found avail_idx(in rte_vhost_dequeue_burst function) is always > >> zero > >> >> although I do send packets in VM. > >> >> > >> >>> Thanks. > >> >> > >> >> > >> >>> Hello, I have strange issue with example/vhost app. > >> >>> > >> >>> I had compiled DPDK to run a vhost example app with followed flags > >> >>> > >> >>> CONFIG_RTE_LIBRTE_VHOST=y > >> >>> CONFIG_RTE_LIBRTE_VHOST_USER=y > >> >>> CONFIG_RTE_LIBRTE_VHOST_DEBUG=n > >> >>> > >> >>> then I run vhost app based on documentation: > >> >>> > >> >>> ./build/app/vhost-switch -c f -n 4 --huge-dir /mnt/huge > --socket-mem > >> >>> 3712 > >> >>> -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9 > >> >>> > >> >>> -I use this strange --socket-mem 3712 because of physical limit of > >> >>> memoryon device -with this vhost user I run two KVM machines with > >> >>> followed parameters > >> >>> > >> >>> kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 > -cpu > >> >>> host -smp 2 -hda /home/ubuntu/qemu/debian_squeeze2_amd64.qcow2 -m > >> >>> 1024 -mem-path /mnt/huge -mem-prealloc -chardev > >> >>> socket,id=char1,path=/home/ubuntu/dpdk/examples/vhost/usvhost > >> >>> -netdev type=vhost-user,id=hostnet1,chardev=char1 > >> >>> -device virtio-net > >> >>> > pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6 > >> >>> = > >> >>> off,guest_ecn=off > >> >>> -chardev > >> >>> socket,id=char2,path=/home/ubuntu/dpdk/examples/vhost/usvhost > >> >>> -netdev type=vhost-user,id=hostnet2,chardev=char2 > >> >>> -device > >> >>> virtio-net- > >> >>> > pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6 > >> >>> = > >> >>> off,guest_ecn=off > >> >>> > >> >>> After running KVM virtio correctly starting (below logs from vhost > app) > >> >> ... > >> >>> VHOST_CONFIG: mapped region 0 fd:31 to 0x2aaabae00000 sz:0xa0000 > >> >>> off:0x0 > >> >>> VHOST_CONFIG: mapped region 1 fd:37 to 0x2aaabb000000 sz:0x10000000 > >> >>> off:0xc0000 > >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > >> >>> VHOST_CONFIG: vring kick idx:0 file:38 > >> >>> VHOST_CONFIG: virtio isn't ready for processing. > >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM > >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE > >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR > >> >>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK > >> >>> VHOST_CONFIG: vring kick idx:1 file:39 > >> >>> VHOST_CONFIG: virtio is now ready for processing. > >> >>> VHOST_DATA: (1) Device has been added to data core 2 > >> >>> > >> >>> So everything looking good. > >> >>> > >> >>> Maybe it is something trivial but using options: --vm2vm 1 (or) 2 > >> >>> --stats 9 it seems that I didn't have connection between VM2VM > >> >>> communication. I set manually IP for eth0 and eth1: > >> >>> > >> >>> on 1 VM > >> >>> ifconfig eth0 192.168.0.100 netmask 255.255.255.0 up ifconfig eth1 > >> >>> 192.168.1.101 netmask 255.255.255.0 up > >> >>> > >> >>> on 2 VM > >> >>> ifconfig eth0 192.168.1.200 netmask 255.255.255.0 up ifconfig eth1 > >> >>> 192.168.0.202 netmask 255.255.255.0 up > >> >>> > >> >>> I notice that in vhostapp are one directional rx/tx queue so I > tryied > >> >>> to ping between VM1 to VM2 using both interfaces ping -I eth0 > >> >>> 192.168.1.200 ping -I > >> >>> eth1 192.168.1.200 ping -I eth0 192.168.0.202 ping -I eth1 > >> >>> 192.168.0.202 > >> >>> > >> >>> on VM2 using tcpdump on both interfaces I didn't see any ICMP > requests > >> >>> or traffic > >> >>> > >> >>> And I cant ping between any IP/interfaces, moreover stats show me > that: > >> >>> > >> >>> Device statistics ==================================== > >> >>> Statistics for device 0 ------------------------------ > >> >>> TX total: 0 > >> >>> TX dropped: 0 > >> >>> TX successful: 0 > >> >>> RX total: 0 > >> >>> RX dropped: 0 > >> >>> RX successful: 0 > >> >>> Statistics for device 1 ------------------------------ > >> >>> TX total: 0 > >> >>> TX dropped: 0 > >> >>> TX successful: 0 > >> >>> RX total: 0 > >> >>> RX dropped: 0 > >> >>> RX successful: 0 > >> >>> Statistics for device 2 ------------------------------ > >> >>> TX total: 0 > >> >>> TX dropped: 0 > >> >>> TX successful: 0 > >> >>> RX total: 0 > >> >>> RX dropped: 0 > >> >>> RX successful: 0 > >> >>> Statistics for device 3 ------------------------------ > >> >>> TX total: 0 > >> >>> TX dropped: 0 > >> >>> TX successful: 0 > >> >>> RX total: 0 > >> >>> RX dropped: 0 > >> >>> RX successful: 0 > >> >>> ====================================================== > >> >>> > >> >>> So it seems like any packet didn't leave my VM. > >> >>> also arp table is empty on each VM. > >> >> > >> >> > >> > >> > > > > -- > Andriy Berestovskyy > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [dpdk-dev] FW: Vhost user no connection vm2vm 2015-05-22 8:05 ` Maciej Grochowski 2015-05-22 8:26 ` Ouyang, Changchun 2015-05-22 8:54 ` [dpdk-dev] " Gaohaifeng (A) @ 2015-05-22 9:27 ` Luke Gorrie 2 siblings, 0 replies; 13+ messages in thread From: Luke Gorrie @ 2015-05-22 9:27 UTC (permalink / raw) To: Maciej Grochowski; +Cc: dev On 22 May 2015 at 10:05, Maciej Grochowski <maciej.grochowski@codilime.com> wrote: > What I'm going to do today is to compile newest kernel for vhost and guest > and debug where packet flow stuck, I will report the result > Compiling the guest virtio-net driver with debug printouts enabled can be really helpful in these situations too. ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2015-05-25 4:16 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-05-15 10:15 [dpdk-dev] Vhost user no connection vm2vm Maciej Grochowski 2015-05-17 14:41 ` Ouyang, Changchun 2015-05-21 9:12 ` [dpdk-dev] FW: " Gaohaifeng (A) 2015-05-22 8:05 ` Maciej Grochowski 2015-05-22 8:26 ` Ouyang, Changchun 2015-05-22 8:54 ` [dpdk-dev] " Gaohaifeng (A) 2015-05-22 9:28 ` Maciej Grochowski 2015-05-22 9:58 ` Tetsuya Mukawa 2015-05-22 10:04 ` Maciej Grochowski 2015-05-22 10:54 ` Andriy Berestovskyy 2015-05-22 17:59 ` Maciej Grochowski 2015-05-25 4:15 ` Gaohaifeng (A) 2015-05-22 9:27 ` [dpdk-dev] FW: " Luke Gorrie
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).