[dpdk-dev] traffic no longer readable...

[dpdk-dev] traffic no longer readable...

[Jeff Weeks]

Hello all,

I've been developing with dpdk, using virtio, and I'm noticing that I can 
often get into a state whereby I no longer receive any packets.

The call to rte_eth_rx_burst always returns 0, indicating the rx queue is 
empty.

How is this queue filled?  I see there is a dpdk thread, which waits on an 
epoll fd for interrupts.  I set a breakpoint here, and it appeared as though 
we were waiting forever on the epoll fd.  This, to me, seemed to be the source 
of the issue I'm seeing (presumably interrupts cause us to read off the nic and 
fill the queue?)... but, oddly enough, I see the same behaviour even when I'm 
receiving packets.

Have I misinterpreted how this is supposed to work?  Or is gdb behaving poorly 
for me (wouldn't be surprising...).

What/how else can I further investigate this?

Thanks,
Jeff

Re: [dpdk-dev] traffic no longer readable...

[Thomas Monjalon]

2015-05-20 11:11, Jeff Weeks:
> Hello all,
> 
> I've been developing with dpdk, using virtio, and I'm noticing that I can 
> often get into a state whereby I no longer receive any packets.
> 
> The call to rte_eth_rx_burst always returns 0, indicating the rx queue is 
> empty.
> 
> How is this queue filled?  I see there is a dpdk thread, which waits on an 
> epoll fd for interrupts.  I set a breakpoint here, and it appeared as though 
> we were waiting forever on the epoll fd.  This, to me, seemed to be the source 
> of the issue I'm seeing (presumably interrupts cause us to read off the nic and 
> fill the queue?)... but, oddly enough, I see the same behaviour even when I'm 
> receiving packets.
> 
> Have I misinterpreted how this is supposed to work?  Or is gdb behaving poorly 
> for me (wouldn't be surprising...).
> 
> What/how else can I further investigate this?

Are you using the latest version (git HEAD)?
Are you using DPDK vhost?

Re: [dpdk-dev] traffic no longer readable...

[Jeff Weeks]

On Wednesday, May 20, 2015 06:31:33 PM Thomas Monjalon wrote:
> 2015-05-20 11:59, Jeff Weeks:
> > forgot to include the list...
> > 
> > ----------  Forwarded Message  ----------
> > 
> > Subject: Re: [dpdk-dev] traffic no longer readable...
> > Date: Wednesday, May 20, 2015, 11:56:47 AM
> > From: Jeff Weeks <jweeks@neuraldk.org>
> > To: Thomas Monjalon <thomas.monjalon@6wind.com>
> > 
> > On Wednesday, May 20, 2015 05:33:20 PM you wrote:
> > > 2015-05-20 11:11, Jeff Weeks:
> > > > Hello all,
> > > > 
> > > > I've been developing with dpdk, using virtio, and I'm noticing that I
> > > > can
> > > > often get into a state whereby I no longer receive any packets.
> > > > 
> > > > The call to rte_eth_rx_burst always returns 0, indicating the rx queue
> > > > is
> > > > empty.
> > > > 
> > > > How is this queue filled?  I see there is a dpdk thread, which waits
> > > > on an
> > > > epoll fd for interrupts.  I set a breakpoint here, and it appeared as
> > > > though we were waiting forever on the epoll fd.  This, to me, seemed
> > > > to
> > > > be the source of the issue I'm seeing (presumably interrupts cause us
> > > > to
> > > > read off the nic and fill the queue?)... but, oddly enough, I see the
> > > > same behaviour even when I'm receiving packets.
> > > > 
> > > > Have I misinterpreted how this is supposed to work?  Or is gdb
> > > > behaving
> > > > poorly for me (wouldn't be surprising...).
> > > > 
> > > > What/how else can I further investigate this?
> > > 
> > > Are you using the latest version (git HEAD)?
> > > Are you using DPDK vhost?
> > 
> > I am using vhost, yes.
> > I'm currently using 1.8.  Is this a known issue fixed in HEAD, or 2.0?
> 
> Maybe it is due to this (fixed) bug:
> 	http://dpdk.org/browse/dpdk/commit/?id=159793ac867

Hmm... I realize now that I've misspoke on *both* questions; my apologies.

I am, in fact, running dpdk 1.7.

And, while I am using the linux vhost kernel module, I am not using the dpdk 
vhost sample code, so the above bugfix doesn't apply.

While I'd like to upgrade to the latest dpdk code, the API changes make this 
less-than-trivial at the moment.

Are there specific areas I should be looking to debug this?  rte_eth_rx_burst 
believes the rx queue is empty.

I'm not sure whether this is indication that the "filler" of the queue is 
misbehaving, or if the dpdk side of reading the queue is misbehaving.

--Jeff

Re: [dpdk-dev] traffic no longer readable...

[Thomas Monjalon]

2015-05-20 11:59, Jeff Weeks:
> forgot to include the list...
> 
> ----------  Forwarded Message  ----------
> 
> Subject: Re: [dpdk-dev] traffic no longer readable...
> Date: Wednesday, May 20, 2015, 11:56:47 AM
> From: Jeff Weeks <jweeks@neuraldk.org>
> To: Thomas Monjalon <thomas.monjalon@6wind.com>
> 
> On Wednesday, May 20, 2015 05:33:20 PM you wrote:
> > 2015-05-20 11:11, Jeff Weeks:
> > > Hello all,
> > > 
> > > I've been developing with dpdk, using virtio, and I'm noticing that I can
> > > often get into a state whereby I no longer receive any packets.
> > > 
> > > The call to rte_eth_rx_burst always returns 0, indicating the rx queue is
> > > empty.
> > > 
> > > How is this queue filled?  I see there is a dpdk thread, which waits on an
> > > epoll fd for interrupts.  I set a breakpoint here, and it appeared as
> > > though we were waiting forever on the epoll fd.  This, to me, seemed to
> > > be the source of the issue I'm seeing (presumably interrupts cause us to
> > > read off the nic and fill the queue?)... but, oddly enough, I see the
> > > same behaviour even when I'm receiving packets.
> > > 
> > > Have I misinterpreted how this is supposed to work?  Or is gdb behaving
> > > poorly for me (wouldn't be surprising...).
> > > 
> > > What/how else can I further investigate this?
> > 
> > Are you using the latest version (git HEAD)?
> > Are you using DPDK vhost?
> 
> I am using vhost, yes.
> I'm currently using 1.8.  Is this a known issue fixed in HEAD, or 2.0?

Maybe it is due to this (fixed) bug:
	http://dpdk.org/browse/dpdk/commit/?id=159793ac867