[dpdk-dev] Bug in virtqueue_dequeue_burst_rx()

[dpdk-dev] Bug in virtqueue_dequeue_burst_rx()

[Gopakumar Choorakkot Edakkunni]

While I was testing virtio with ubuntu 14.04 kvm as host and dpdk16.07
linux as guest, quite often I have seen that I get into a situation where
virtio_recv_mergeable_pkts() gets into a forever loop, after sending
traffic for a while. In the below API,  I see that it clearly leads to a
while loop, I am not quite familiar with virtio or mergeable buffers, so
thought of checking with dpdk alias on the intent here.

I checked the linux kernel virtio_net.c file which does the similar
mergeable recieve, and the kernel code breaks out in case of error.
Shouldnt dpdk be breaking out of here on error instead of continue ?

virtio_recv_mergeable_pkts()
{
<snip>
    while (i < nb_used) {
        <snip>
*        num = virtqueue_dequeue_burst_rx(rxvq, rcv_pkts, len, 1);*
*        if (num != 1)*
*            continue;*
       <snip>
    }
<snip?
}

Rgds,
Gopa.

Re: [dpdk-dev] Bug in virtqueue_dequeue_burst_rx()

[Yuanhan Liu]

On Mon, Dec 19, 2016 at 09:59:33PM -0800, Gopakumar Choorakkot Edakkunni wrote:
> While I was testing virtio with ubuntu 14.04 kvm as host and dpdk16.07
> linux as guest, quite often I have seen that I get into a situation where
> virtio_recv_mergeable_pkts() gets into a forever loop, after sending
> traffic for a while. In the below API,  I see that it clearly leads to a
> while loop, I am not quite familiar with virtio or mergeable buffers, so
> thought of checking with dpdk alias on the intent here.
> 
> I checked the linux kernel virtio_net.c file which does the similar
> mergeable recieve, and the kernel code breaks out in case of error.
> Shouldnt dpdk be breaking out of here on error instead of continue ?

Yep, that looks buggy.

> 
> virtio_recv_mergeable_pkts()
> {
> <snip>
>     while (i < nb_used) {
>         <snip>
> *        num = virtqueue_dequeue_burst_rx(rxvq, rcv_pkts, len, 1);*
> *        if (num != 1)*
> *            continue;*

However, normally, virtqueue_dequeue_burst_rx() would be successful
here: the outer check (i < nb_used) somehow ascertains it.

Two options I can think of so far:

- will it go back working once you do "break" here?

- tell us how you managed to trigger this issue, so that I could
  also reproduce it that I can debug it.

	--yliu

Re: [dpdk-dev] Bug in virtqueue_dequeue_burst_rx()

[Gopakumar Choorakkot Edakkunni]

Hi Yuanahn,

Thanks for the response. Because of my filters, I  missed this email
completely, apologies for the late response ! I haven't tried breaking out
of the loop to see if it will go back to working state - I have two
testbeds, one using plain linux kvm + guest VM running 16.07 dpdk where I
see the problem. I have another testved where I use ovs-dpdk + the same
guest VM image running 16.07 dpdk. The issue happens only on the linux kvm
setup, so for now I just switched to using ovs-dpdk instead. I will go back
sometime to the linux kvm setup and try breaking of the loop and see what
happens.

Without hardly any knowledge of virtio myself, I am guessing this issue is
because the "host" said it has put one buffer into the virtio ring whereas
it really did not put the buffer, right ? So is the issue likely to be in
the host virtio layer rather than in guest dpdk ?

As for the linux-kvm setup, the trigger is very straightforward. Its just
sending traffic. The details of my setup are as below. If you have a
similar setup, I can possibly try and help you recreate the issue or maybe
even provide access to my testbed if you would like that.

1. The guest VM runs dpdk 16.07 and has 8 virtio interfaces mapping to 8
different bridges on the host. The VM has 8Gig memory with 1Gig hugepage
size and 1 hugepage (1gig memory) given to dpdk. And the VM is backed up by
16 x 1Gig hugepages on the host. I am attaching the guest XML file here

2. The host runs the below version. And all that I do is send iperf traffic
that comes in on one of the 8 interfaces in the guest VM and goes out of
another interface in the guest VM.

root@ubuntu:/home/vcadmin# uname -a
Linux ubuntu 3.16.0-77-generic #99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10
UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu:/home/vcadmin# lsb_release -a
No LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 14.04.5 LTS
Release:    14.04
Codename:    trusty

root@ubuntu:/home/vcadmin# cat /proc/meminfo  | grep Huge
AnonHugePages:    141312 kB
HugePages_Total:      16
HugePages_Free:       16
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB

Rgds,
Gopa.

On Wed, Dec 21, 2016 at 11:38 PM, Yuanhan Liu <yuanhan.liu@linux.intel.com>
wrote:

> On Mon, Dec 19, 2016 at 09:59:33PM -0800, Gopakumar Choorakkot Edakkunni
> wrote:
> > While I was testing virtio with ubuntu 14.04 kvm as host and dpdk16.07
> > linux as guest, quite often I have seen that I get into a situation where
> > virtio_recv_mergeable_pkts() gets into a forever loop, after sending
> > traffic for a while. In the below API,  I see that it clearly leads to a
> > while loop, I am not quite familiar with virtio or mergeable buffers, so
> > thought of checking with dpdk alias on the intent here.
> >
> > I checked the linux kernel virtio_net.c file which does the similar
> > mergeable recieve, and the kernel code breaks out in case of error.
> > Shouldnt dpdk be breaking out of here on error instead of continue ?
>
> Yep, that looks buggy.
>
> >
> > virtio_recv_mergeable_pkts()
> > {
> > <snip>
> >     while (i < nb_used) {
> >         <snip>
> > *        num = virtqueue_dequeue_burst_rx(rxvq, rcv_pkts, len, 1);*
> > *        if (num != 1)*
> > *            continue;*
>
> However, normally, virtqueue_dequeue_burst_rx() would be successful
> here: the outer check (i < nb_used) somehow ascertains it.
>
> Two options I can think of so far:
>
> - will it go back working once you do "break" here?
>
> - tell us how you managed to trigger this issue, so that I could
>   also reproduce it that I can debug it.
>
>         --yliu
>