Enable RX interrupts

Enable RX interrupts

[Jack Humphries]

Hi folks,

Hope all are well. I’m trying to enable interrupts in DPDK so that my network receive thread can sleep on an epoll until packets arrive. I am using the ixgbe kernel driver and igb_uio userspace driver with an Intel 82599ES 10Gbps NIC.

I'm doing roughly the following to enable the interrupts, but the epoll never indicates that packets have arrived. The thread only handles packets when the epoll times out. I don't even see interrupts arrive from the device when monitoring /proc/interrupts.

port_conf.intr_conf.rxq = 1;

...

CHECK_EQ(rte_eth_dev_rx_intr_ctl_q(kPort, kQueue, RTE_EPOLL_PER_THREAD,
                                   RTE_INTR_EVENT_ADD, nullptr),
         0); 
CHECK_EQ(rte_eth_dev_rx_intr_enable(kPort, kQueue), 0);

...

rte_epoll_event event;
while (true) {
  int n = rte_epoll_wait(RTE_EPOLL_PER_THREAD, &event, /*maxevents=*/1,
                         /*timeout=*/1000);
  if (n == 0) {
    // Timeout expired.
  } else {
    // Received RX interrupt.
  }
}

Given that I don't see anything coming through in /proc/interrupts, I am going to start digging through the ixgbe driver. However, I wanted to ask here first to see if my setup is missing anything obvious. I based it closely on the l3fwd-power example, though I haven't been able to get that example up and running yet since my NIC only has one port plugged in right now and the example requires two (COVID building restrictions).

Thanks,
Jack Humphries

Re: Enable RX interrupts

[Dmitry Kozlyuk]

2021-10-31 22:50 (UTC-0700), Jack Humphries:
> Hi folks,
> 
> Hope all are well. I’m trying to enable interrupts in DPDK so that my network receive thread can sleep on an epoll until packets arrive. I am using the ixgbe kernel driver and igb_uio userspace driver with an Intel 82599ES 10Gbps NIC.
> 
> I'm doing roughly the following to enable the interrupts, but the epoll never indicates that packets have arrived. The thread only handles packets when the epoll times out. I don't even see interrupts arrive from the device when monitoring /proc/interrupts.
> 
> port_conf.intr_conf.rxq = 1;
> 
> ...
> 
> CHECK_EQ(rte_eth_dev_rx_intr_ctl_q(kPort, kQueue, RTE_EPOLL_PER_THREAD,
>                                    RTE_INTR_EVENT_ADD, nullptr),
>          0); 
> CHECK_EQ(rte_eth_dev_rx_intr_enable(kPort, kQueue), 0);
> 
> ...
> 
> rte_epoll_event event;
> while (true) {
>   int n = rte_epoll_wait(RTE_EPOLL_PER_THREAD, &event, /*maxevents=*/1,
>                          /*timeout=*/1000);
>   if (n == 0) {
>     // Timeout expired.
>   } else {
>     // Received RX interrupt.
>   }
> }
> 
> Given that I don't see anything coming through in /proc/interrupts, I am going to start digging through the ixgbe driver. However, I wanted to ask here first to see if my setup is missing anything obvious. I based it closely on the l3fwd-power example, though I haven't been able to get that example up and running yet since my NIC only has one port plugged in right now and the example requires two (COVID building restrictions).
> 
> Thanks,
> Jack Humphries

Hi Jack,

Have you called rte_eth_dev_rx_intr_enable(port_id, queue_id)?

Not sure about ixgbe, but vmxnet3 also requires rearming interrupts with it
on each iteration before rte_epoll_wait().
This is what l3fwd-power does by the way (see turn_on_off_intr() function).

+ ixgbe maintainer just in case.

Re: Enable RX interrupts

[Jack Humphries]

Hi Dmitry,

Thanks for the quick response! Yes, I was calling rte_eth_dev_rx_intr_enable — I just tried re-arming interrupts before each call to epoll and my initial experiment shows that fixes the issue.

You are right that l3fwd-power does the re-arming as well. If possible, it would be helpful to have a comment in that example code about the re-arming since the impression I got from quickly looking at the l3fwd-power code is that interrupts are turned off because the app wants to poll for a threshold of time before “giving up” and turning interrupts back on again. I’m happy to open a pull request to add this comment, too, if you prefer.

Thanks again for your help!
Jack

> On Nov 1, 2021, at 12:46 AM, Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
> 
> 2021-10-31 22:50 (UTC-0700), Jack Humphries:
>> Hi folks,
>> 
>> Hope all are well. I’m trying to enable interrupts in DPDK so that my network receive thread can sleep on an epoll until packets arrive. I am using the ixgbe kernel driver and igb_uio userspace driver with an Intel 82599ES 10Gbps NIC.
>> 
>> I'm doing roughly the following to enable the interrupts, but the epoll never indicates that packets have arrived. The thread only handles packets when the epoll times out. I don't even see interrupts arrive from the device when monitoring /proc/interrupts.
>> 
>> port_conf.intr_conf.rxq = 1;
>> 
>> ...
>> 
>> CHECK_EQ(rte_eth_dev_rx_intr_ctl_q(kPort, kQueue, RTE_EPOLL_PER_THREAD,
>>                                   RTE_INTR_EVENT_ADD, nullptr),
>>         0); 
>> CHECK_EQ(rte_eth_dev_rx_intr_enable(kPort, kQueue), 0);
>> 
>> ...
>> 
>> rte_epoll_event event;
>> while (true) {
>>  int n = rte_epoll_wait(RTE_EPOLL_PER_THREAD, &event, /*maxevents=*/1,
>>                         /*timeout=*/1000);
>>  if (n == 0) {
>>    // Timeout expired.
>>  } else {
>>    // Received RX interrupt.
>>  }
>> }
>> 
>> Given that I don't see anything coming through in /proc/interrupts, I am going to start digging through the ixgbe driver. However, I wanted to ask here first to see if my setup is missing anything obvious. I based it closely on the l3fwd-power example, though I haven't been able to get that example up and running yet since my NIC only has one port plugged in right now and the example requires two (COVID building restrictions).
>> 
>> Thanks,
>> Jack Humphries
> 
> Hi Jack,
> 
> Have you called rte_eth_dev_rx_intr_enable(port_id, queue_id)?
> 
> Not sure about ixgbe, but vmxnet3 also requires rearming interrupts with it
> on each iteration before rte_epoll_wait().
> This is what l3fwd-power does by the way (see turn_on_off_intr() function).
> 
> + ixgbe maintainer just in case.

Re: Enable RX interrupts

[Stephen Hemminger]

On Sun, 31 Oct 2021 22:50:40 -0700
Jack Humphries <jack@chillysky.com> wrote:

> Hi folks,
> 
> Hope all are well. I’m trying to enable interrupts in DPDK so that my network receive thread can sleep on an epoll until packets arrive. I am using the ixgbe kernel driver and igb_uio userspace driver with an Intel 82599ES 10Gbps NIC.
> 
> I'm doing roughly the following to enable the interrupts, but the epoll never indicates that packets have arrived. The thread only handles packets when the epoll times out. I don't even see interrupts arrive from the device when monitoring /proc/interrupts.
> 
> port_conf.intr_conf.rxq = 1;
> 
> ...
> 
> CHECK_EQ(rte_eth_dev_rx_intr_ctl_q(kPort, kQueue, RTE_EPOLL_PER_THREAD,
>                                    RTE_INTR_EVENT_ADD, nullptr),
>          0); 
> CHECK_EQ(rte_eth_dev_rx_intr_enable(kPort, kQueue), 0);
> 
> ...
> 
> rte_epoll_event event;
> while (true) {
>   int n = rte_epoll_wait(RTE_EPOLL_PER_THREAD, &event, /*maxevents=*/1,
>                          /*timeout=*/1000);
>   if (n == 0) {
>     // Timeout expired.
>   } else {
>     // Received RX interrupt.
>   }
> }
> 
> Given that I don't see anything coming through in /proc/interrupts, I am going to start digging through the ixgbe driver. However, I wanted to ask here first to see if my setup is missing anything obvious. I based it closely on the l3fwd-power example, though I haven't been able to get that example up and running yet since my NIC only has one port plugged in right now and the example requires two (COVID building restrictions).

If you are going to use interrupts you need to have MSI-X to get per-queue interrupts.
The igb_uio driver doesn't support this; you need to use vfio-pci in kernel to get queue interrupts.

Re: Enable RX interrupts

[Jack Humphries]

Thanks Stephen. I am just using one RX queue from the NIC. I assume that since I don’t need per-queue interrupts here since there is only one queue, I should be good using ixgbe+igb_uio?

Thanks,
Jack

> On Nov 1, 2021, at 8:20 AM, Stephen Hemminger <stephen@networkplumber.org> wrote:
> 
> On Sun, 31 Oct 2021 22:50:40 -0700
> Jack Humphries <jack@chillysky.com> wrote:
> 
>> Hi folks,
>> 
>> Hope all are well. I’m trying to enable interrupts in DPDK so that my network receive thread can sleep on an epoll until packets arrive. I am using the ixgbe kernel driver and igb_uio userspace driver with an Intel 82599ES 10Gbps NIC.
>> 
>> I'm doing roughly the following to enable the interrupts, but the epoll never indicates that packets have arrived. The thread only handles packets when the epoll times out. I don't even see interrupts arrive from the device when monitoring /proc/interrupts.
>> 
>> port_conf.intr_conf.rxq = 1;
>> 
>> ...
>> 
>> CHECK_EQ(rte_eth_dev_rx_intr_ctl_q(kPort, kQueue, RTE_EPOLL_PER_THREAD,
>>                                   RTE_INTR_EVENT_ADD, nullptr),
>>         0); 
>> CHECK_EQ(rte_eth_dev_rx_intr_enable(kPort, kQueue), 0);
>> 
>> ...
>> 
>> rte_epoll_event event;
>> while (true) {
>>  int n = rte_epoll_wait(RTE_EPOLL_PER_THREAD, &event, /*maxevents=*/1,
>>                         /*timeout=*/1000);
>>  if (n == 0) {
>>    // Timeout expired.
>>  } else {
>>    // Received RX interrupt.
>>  }
>> }
>> 
>> Given that I don't see anything coming through in /proc/interrupts, I am going to start digging through the ixgbe driver. However, I wanted to ask here first to see if my setup is missing anything obvious. I based it closely on the l3fwd-power example, though I haven't been able to get that example up and running yet since my NIC only has one port plugged in right now and the example requires two (COVID building restrictions).
> 
> If you are going to use interrupts you need to have MSI-X to get per-queue interrupts.
> The igb_uio driver doesn't support this; you need to use vfio-pci in kernel to get queue interrupts.

Re: Enable RX interrupts

[Stephen Hemminger]

On Mon, 1 Nov 2021 12:16:14 -0700
Jack Humphries <jack@chillysky.com> wrote:

> Thanks Stephen. I am just using one RX queue from the NIC. I assume that since I don’t need per-queue interrupts here since there is only one queue, I should be good using ixgbe+igb_uio?
> 
> Thanks,
> Jack
> 

Even with one queue, it won't work because the interrupt is dedicated to link status.

Re: Enable RX interrupts

[Thomas Monjalon]

01/11/2021 09:56, Jack Humphries:
> Hi Dmitry,
> 
> Thanks for the quick response! Yes, I was calling rte_eth_dev_rx_intr_enable — I just tried re-arming interrupts before each call to epoll and my initial experiment shows that fixes the issue.
> 
> You are right that l3fwd-power does the re-arming as well. If possible, it would be helpful to have a comment in that example code about the re-arming since the impression I got from quickly looking at the l3fwd-power code is that interrupts are turned off because the app wants to poll for a threshold of time before “giving up” and turning interrupts back on again. I’m happy to open a pull request to add this comment, too, if you prefer.

The best is to submit a patch on the mailing list, Cc'in the relevant maintainers.
Thanks