Re: [dpdk-dev] [PATCH v5 resend 03/12] vhost: vring queue setup for multiple queue support

[Yuanhan Liu <yuanhan.liu@linux.intel.com>]

On Tue, Sep 22, 2015 at 11:47:34AM +0300, Marcel Apfelbaum wrote:
> On 09/22/2015 11:34 AM, Yuanhan Liu wrote:
> >On Tue, Sep 22, 2015 at 11:10:13AM +0300, Marcel Apfelbaum wrote:
> >>On 09/22/2015 10:31 AM, Yuanhan Liu wrote:
> >>>On Mon, Sep 21, 2015 at 08:56:30PM +0300, Marcel Apfelbaum wrote:
> >>[...]
> >>>>>
> >>>>>Hi,
> >>>>>
> >>>>>I have made 4 cleanup patches few weeks before, including the patch
> >>>>>to define kickfd and callfd as int type, and they have already got
> >>>>>the ACK from Huawei Xie, and Chuangchun Ouyang. It's likely that
> >>>>>they will be merged, hence I made this patchset based on them.
> >>>>>
> >>>>>This will also answer the question from your another email: can't
> >>>>>apply.
> >>>>
> >>>>Hi,
> >>>>Thank you for the response, it makes sense now.
> >>>>
> >>>>T have another issue, maybe you can help.
> >>>>I have some problems making it work with OVS/DPDK backend and virtio-net driver in guest.
> >>>>
> >>>>I am using a simple setup:
> >>>>     http://wiki.qemu.org/Features/vhost-user-ovs-dpdk
> >>>>that connects 2 VMs using OVS's dpdkvhostuser ports (regular virtio-net driver in guest, not the PMD driver).
> >>>>
> >>>>The setup worked fine with the prev DPDK MQ implementation (V4), however on this one the traffic stops
> >>>>once I set queues=n in guest.
> >>>
> >>>Hi,
> >>>
> >>>Could you be more specific about that? It also would be helpful if you
> >>>could tell me the steps, besides those setup steps you mentioned in the
> >>>qemu wiki and this email, you did for testing.
> >>>
> >>
> >>Hi,
> >>Thank you for your help.
> >>
> >>I am sorry the wiki is not enough, I'll be happy to add all the missing parts.
> >>In the meantime maybe you can tell me where the problem is, I also suggest to
> >>post here the output of journalctl command.
> >>
> >>We only need a regular machine and we want traffic between 2 VMs. I'll try to summarize the steps:
> >>
> >>1. Be sure you have enough hugepages enabled (2M pages are enough) and mounted.
> >>2. Configure and start OVS following the wiki
> >>    - we only want one bridge with 2 dpdkvhostuser ports.
> >>3. Start VMs using the wiki command line
> >>    - check journalctl for possible errors. You can use
> >>         journalctl  --since `date +%T --date="-10 minutes"`
> >>      to see only last 10 minutes.
> >>4. Configure the guests IPs.
> >>    - Disable the Network Manager as described bellow in the mail.
> >>5. At this point you should be able to ping between guests.
> >>
> >>Please let me know if you have any problem until this point.
> >>I'll be happy to help. Please point any special steps you made that
> >>are not in the WIKI. The journalctl logs would also help.
> >>
> >>Does the ping between VMS work now?
> >
> >Yes, it works, too. I can ping the other vm inside a vm.
> >
> >     [root@dpdk-kvm ~]# ethtool -l eth0
> >     Channel parameters for eth0:
> >     Pre-set maximums:
> >     RX:             0
> >     TX:             0
> >     Other:          0
> >     Combined:       2
> >     Current hardware settings:
> >     RX:             0
> >     TX:             0
> >     Other:          0
> >     Combined:       2
> >
> >     [root@dpdk-kvm ~]# ifconfig eth0
> >     eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> >             inet 192.168.100.11  netmask 255.255.255.0  broadcast 192.168.100.255
> >             inet6 fe80::5054:ff:fe12:3459  prefixlen 64  scopeid 0x20<link>
> >             ether 52:54:00:12:34:59  txqueuelen 1000  (Ethernet)
> >             RX packets 56  bytes 5166 (5.0 KiB)
> >             RX errors 0  dropped 0  overruns 0  frame 0
> >             TX packets 84  bytes 8303 (8.1 KiB)
> >             TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> >
> >     [root@dpdk-kvm ~]# ping 192.168.100.10
> >     PING 192.168.100.10 (192.168.100.10) 56(84) bytes of data.
> >     64 bytes from 192.168.100.10: icmp_seq=1 ttl=64 time=0.213 ms
> >     64 bytes from 192.168.100.10: icmp_seq=2 ttl=64 time=0.094 ms
> >     64 bytes from 192.168.100.10: icmp_seq=3 ttl=64 time=0.246 ms
> >     64 bytes from 192.168.100.10: icmp_seq=4 ttl=64 time=0.153 ms
> >     64 bytes from 192.168.100.10: icmp_seq=5 ttl=64 time=0.104 ms
> >     ^C
> >>
> >>If yes, please let me know and I'll go over MQ enabling.
> >
> >I'm just wondering why it doesn't work on your side.
> 
> Hi,
> 
> This is working also for me, but without enabling the MQ. (ethtool -L eth0 combined n (n>1) )
> The problem starts when I am applying the patches and I enable MQ. (Need a slightly different QEMU commandline)
> 
> >
> >>
> >>>I had a very rough testing based on your test guides, I indeed found
> >>>an issue: the IP address assigned by "ifconfig" disappears soon in the
> >>>first few times and after about 2 or 3 times reset, it never changes.
> >>>
> >>>(well, I saw that quite few times before while trying different QEMU
> >>>net devices. So, it might be a system configuration issue, or something
> >>>else?)
> >>>
> >>
> >>You are right, this is a guest config issue, I think you should disable NetworkManager
> >
> >Yeah, I figured it out by my self, and it worked when I hardcoded it at
> >/etc/sysconfig/network-scripts/ifcfg-eth0.
> >
> >>for static IP addresses. Please use only the virtio-net device.
> >>
> >>You cant try this:
> >>sudo systemctl stop NetworkManager
> >>sudo systemctl disable NetworkManager
> >
> >Thanks for the info and tip!
> >
> >>
> >>>Besides that, it works, say, I can wget a big file from host.
> >>>
> >>
> >>The target here is traffic between 2 VMs.
> >>We want to be able to ping (for example) between VMS when MQ > 1 is enabled on both guests:
> >>- ethtool -L eth0 combined <queues nr, the same as QEMU>
> >
> >As you can see from my command log, I did so and it worked :)
> >
> 
> Let me understand, it worked after applying MQ patches on all 3 projects (DPDK, QEMU and OVS)?
> It worked with MQ enabled? MQ >1 ?

Yes, however, I tried few more times this time, and found it sometimes
worked, and sometimes not. Sounds like there is a bug somewhere.

> 
> You can be sure by using the following command in one of the VMs:
>   cat /proc/interrupts | grep virtio
> and see that you have interrupts for all virtio0-input.0/1/...

     [root@dpdk-kvm ~]# cat /proc/interrupts | grep virtio
     24:        0        0    PCI-MSI-edge       virtio0-config
     25:      425        0    PCI-MSI-edge       virtio0-virtqueues


BTW, I have seen some warnings from ovs:

    2015-09-22T02:08:58Z|00003|ofproto_dpif_upcall(pmd45)|WARN|upcall_cb failure: ukey installation fails
    
    2015-09-22T02:11:05Z|00003|ofproto_dpif_upcall(pmd44)|WARN|Dropped 29 log messages in last 127 seconds (most recently, 82 seconds ago) due to excessive rate
    2015-09-22T02:11:05Z|00004|ofproto_dpif_upcall(pmd44)|WARN|upcall_cb failure: ukey installation fails
    2015-09-22T02:12:17Z|00005|ofproto_dpif_upcall(pmd44)|WARN|Dropped 11 log messages in last 32 seconds (most recently, 14 seconds ago) due to excessive rate
    2015-09-22T02:12:17Z|00006|ofproto_dpif_upcall(pmd44)|WARN|upcall_cb failure: ukey installation fails
    2015-09-22T02:14:59Z|00007|ofproto_dpif_upcall(pmd44)|WARN|Dropped 2 log messages in last 161 seconds (most recently, 161 seconds ago) due to excessive rate
    2015-09-22T02:14:59Z|00008|ofproto_dpif_upcall(pmd44)|WARN|upcall_cb failure: ukey installation fails
    

Does that look abnormal to you?

Anyway, I here check if there is anything I can fix.

	--yliu