[dpdk-users] Facing issues with Interrupt mode (IGB,VFIO)

[dpdk-users] Facing issues with Interrupt mode (IGB,VFIO)

[Prem Chaitanya]

Hi All,

We modified the l3fwd application to run in interrupt+poll mode. The driver
we used was IGB with Intel I350 NIC.
When we ran with 1 lcore and 1 queue, there were no issues. When we tried
to run with 2 lcores and 2 Rx-queues we are not getting interrupts.
In one of the documentation(DPDK-201.pdf), it was mentioned that IGB driver
doesn't support interrupt mode with multiple lcores since it supports only
1 epoll-fd.
Can you please confirm if this is the case?

Then we went ahead and loaded the VFIO driver and tried to run the same
application.
Now we get an interrupt once and after that when we go back to waiting on
epoll, we dont receive any further interrupts.
But when we print the stats for rx-packets we see that there are packets
waiting in the NIC Rx queue.
Same behavior is observed even when we run application with 1 lcore and 1
Rx queue.

We took the output of "/proc/interrupts", and we don't see the interrupt
count increasing there either.
Looks like interrupt is not getting raised eventhough packets arrive.

Would appreciate the help.

-- 
Regards
Prem

Re: [dpdk-users] Facing issues with Interrupt mode (IGB,VFIO)

[Stephen Hemminger]

On Tue, 13 Dec 2016 14:43:41 +0530
Prem Chaitanya <prem@telaverge.com> wrote:

> Hi All,
> 
> We modified the l3fwd application to run in interrupt+poll mode. The driver
> we used was IGB with Intel I350 NIC.
> When we ran with 1 lcore and 1 queue, there were no issues. When we tried
> to run with 2 lcores and 2 Rx-queues we are not getting interrupts.
> In one of the documentation(DPDK-201.pdf), it was mentioned that IGB driver
> doesn't support interrupt mode with multiple lcores since it supports only
> 1 epoll-fd.
> Can you please confirm if this is the case?

igb_uio device driver can not support an interrupt per queue (MSI).


> Then we went ahead and loaded the VFIO driver and tried to run the same
> application.
> Now we get an interrupt once and after that when we go back to waiting on
> epoll, we dont receive any further interrupts.
> But when we print the stats for rx-packets we see that there are packets
> waiting in the NIC Rx queue.
> Same behavior is observed even when we run application with 1 lcore and 1
> Rx queue.

You need to do some careful coding to handle interrupt+poll mode.
Look at l3fwd-power example, and even that isn't enough.

Basically, you need to dynamically go in and out of interrupt mode
and use the rx_queue_count function to check for races.

The application must have poll loop like:

while() {
          rte_epoll_wait()

          if (rx_event) {
readmore:
		rte_eth_dev_rx_intr_disable()

		while ( (n = rte_eth_rx_queue_burst()) > 0) {
			process n packets;
		}
	
		rte_eth_dev_rx_intr_enable();
		if (rte_eth_dev_rx_queue_count() > 0) {
			// lost race 
			goto readmore;
		}
	}
}

Re: [dpdk-users] Facing issues with Interrupt mode (IGB,VFIO)

[Prem Chaitanya]

Hi Stephen,

Thanks a lot for your response.
We wrote our application(by modifying l3fwd application) and by looking at
the l3fwd-power sample application. But didnt know about this race you are
talking about.
Question: If we enable interrupt and go into epoll_wait, wont the
interrupts be pending in eventfd ? Or am i missing something.

I applied the queue_count check and tried to read again in case there are
packets. But still i see there is a lot of inconsistency in receiving
interrupts.
After i did this change, first time i ran the application i saw interrupts
coming in, but from subsequent time i don't see them anymore.

Here is the our code flow.

1. Register for interrupts
2. Poll Rx-Queue.
3. If idle_poll crosses configured limit, turn-on interrupt for that queue
and goto epoll_wait
4. One interrupt is received then disable interrupt for that queue and poll.

Let me know if you need any specific information.


On Wed, Dec 14, 2016 at 3:52 AM, Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Tue, 13 Dec 2016 14:43:41 +0530
> Prem Chaitanya <prem@telaverge.com> wrote:
>
> > Hi All,
> >
> > We modified the l3fwd application to run in interrupt+poll mode. The
> driver
> > we used was IGB with Intel I350 NIC.
> > When we ran with 1 lcore and 1 queue, there were no issues. When we tried
> > to run with 2 lcores and 2 Rx-queues we are not getting interrupts.
> > In one of the documentation(DPDK-201.pdf), it was mentioned that IGB
> driver
> > doesn't support interrupt mode with multiple lcores since it supports
> only
> > 1 epoll-fd.
> > Can you please confirm if this is the case?
>
> igb_uio device driver can not support an interrupt per queue (MSI).
>
>
> > Then we went ahead and loaded the VFIO driver and tried to run the same
> > application.
> > Now we get an interrupt once and after that when we go back to waiting on
> > epoll, we dont receive any further interrupts.
> > But when we print the stats for rx-packets we see that there are packets
> > waiting in the NIC Rx queue.
> > Same behavior is observed even when we run application with 1 lcore and 1
> > Rx queue.
>
> You need to do some careful coding to handle interrupt+poll mode.
> Look at l3fwd-power example, and even that isn't enough.
>
> Basically, you need to dynamically go in and out of interrupt mode
> and use the rx_queue_count function to check for races.
>
> The application must have poll loop like:
>
> while() {
>           rte_epoll_wait()
>
>           if (rx_event) {
> readmore:
>                 rte_eth_dev_rx_intr_disable()
>
>                 while ( (n = rte_eth_rx_queue_burst()) > 0) {
>                         process n packets;
>                 }
>
>                 rte_eth_dev_rx_intr_enable();
>                 if (rte_eth_dev_rx_queue_count() > 0) {
>                         // lost race
>                         goto readmore;
>                 }
>         }
> }
>



-- 
Regards
Prem