Questions about interrupts

Questions about interrupts

[Alan Beadle]

Hello,

I am trying to use rte_epoll_wait() to avoid rx polling and reduce CPU
utilization while preserving low latency. I am using the Intel X550-T2
NIC with DPDK 23.11 and the ixgbe driver.

I was able to make interrupts work and my process is getting
interrupts when a packet arrives. I followed the example here:
https://stackoverflow.com/questions/69792978/using-interrupts-with-dpdk

Before calling rte_epoll_wait(), I must first arm the interrupt by
calling rte_eth_dev_rx_intr_enable() every time my thread is about to
wait. However, if a packet arrives between these calls, the interrupt
will not work and my process will only resume once the timeout is
reached. This causes huge latency spikes unless my timeout is very
short, and the minimum time of 1ms is still too long to tolerate this
problem, since my goal is to keep latency low.

Here are my questions:
1) Is there a way to prevent the thread from suspending at all if the
interrupt is not "armed" anymore?
2) Is it possible to set a timeout shorter than 1 ms? It looks like
this is not supported by rte_epoll_wait().

Thank you,
-Alan Beadle

Re: Questions about interrupts

[Stephen Hemminger]

On Thu, 13 Feb 2025 16:55:57 -0500
Alan Beadle <ab.beadle@gmail.com> wrote:

> Hello,
> 
> I am trying to use rte_epoll_wait() to avoid rx polling and reduce CPU
> utilization while preserving low latency. I am using the Intel X550-T2
> NIC with DPDK 23.11 and the ixgbe driver.
> 
> I was able to make interrupts work and my process is getting
> interrupts when a packet arrives. I followed the example here:
> https://stackoverflow.com/questions/69792978/using-interrupts-with-dpdk
> 
> Before calling rte_epoll_wait(), I must first arm the interrupt by
> calling rte_eth_dev_rx_intr_enable() every time my thread is about to
> wait. However, if a packet arrives between these calls, the interrupt
> will not work and my process will only resume once the timeout is
> reached. This causes huge latency spikes unless my timeout is very
> short, and the minimum time of 1ms is still too long to tolerate this
> problem, since my goal is to keep latency low.

After enabling interrupts, the application should do one more call to rx_burst
which will see if there any packets that raced in.
If it does get some packets, chances are you want to disable interrupts
and go back to polling mode.

See the l3fwd-power example application.


> Here are my questions:
> 1) Is there a way to prevent the thread from suspending at all if the
> interrupt is not "armed" anymore?
> 2) Is it possible to set a timeout shorter than 1 ms? It looks like
> this is not supported by rte_epoll_wait().

Internally rte_epoll_wait is just a wrapper around the epoll system
call. which takes a timeout in milliseconds.

You could make a version of rte_epoll_wait using the newer epoll_pwait2
system call which accepts a timeout in nanosecond resolution.

Read the source of EAL, it is much easier than trying to explain
all this.

Re: Questions about interrupts

[Alan Beadle]

> After enabling interrupts, the application should do one more call to rx_burst
> which will see if there any packets that raced in.
> If it does get some packets, chances are you want to disable interrupts
> and go back to polling mode.

As I'm sure you are aware, polling one more time would not change the
likelihood that additional packets arrive and disarm the interrupt
before the thread suspends. A packet could still arrive between that
polling round and the call to rte_epoll_wait(). Is there really no way
to atomically arm the interrupt and suspend, guaranteeing that my
thread will never suspend if packets have arrived prior to suspending?

Re: Questions about interrupts

[Stephen Hemminger]

On Sat, 15 Feb 2025 10:55:45 -0500
Alan Beadle <ab.beadle@gmail.com> wrote:

> > After enabling interrupts, the application should do one more call to rx_burst
> > which will see if there any packets that raced in.
> > If it does get some packets, chances are you want to disable interrupts
> > and go back to polling mode.  
> 
> As I'm sure you are aware, polling one more time would not change the
> likelihood that additional packets arrive and disarm the interrupt
> before the thread suspends. A packet could still arrive between that
> polling round and the call to rte_epoll_wait(). Is there really no way
> to atomically arm the interrupt and suspend, guaranteeing that my
> thread will never suspend if packets have arrived prior to suspending?

The interrupts are edge-triggered so the state of the interrupt is recorded.
Using epoll, the device is reading the state out of the MSI-x register,
and before going to sleep epoll does a "do you you have anything" check
for all sources.